CN108038225A - A kind of data processing method and system - Google Patents

A kind of data processing method and system Download PDF

Info

Publication number
CN108038225A
CN108038225A CN201711418696.XA CN201711418696A CN108038225A CN 108038225 A CN108038225 A CN 108038225A CN 201711418696 A CN201711418696 A CN 201711418696A CN 108038225 A CN108038225 A CN 108038225A
Authority
CN
China
Prior art keywords
data
keyword
data set
critical field
acquisition system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711418696.XA
Other languages
Chinese (zh)
Other versions
CN108038225B (en
Inventor
王清臣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nine Chapter Yunji Technology Co Ltd Beijing
Original Assignee
Nine Chapter Yunji Technology Co Ltd Beijing
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nine Chapter Yunji Technology Co Ltd Beijing filed Critical Nine Chapter Yunji Technology Co Ltd Beijing
Priority to CN201711418696.XA priority Critical patent/CN108038225B/en
Publication of CN108038225A publication Critical patent/CN108038225A/en
Application granted granted Critical
Publication of CN108038225B publication Critical patent/CN108038225B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/217—Database tuning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/235—Update request formulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention provides a kind of data processing method and system, receives the first data acquisition system of external system transmission；The second data set associated with target data set to be updated is generated in a data processing system；Empty the data in the target data set；Data update is carried out to the target data set using the data in first data acquisition system and the second data set.In this way, the first data acquisition system of external system transmission is being received, it is necessary to when carrying out data update, it is ensured that the stability of data handling system, without being scanned to all data, saves the plenty of time, and improve the efficiency of data update.

Description

A kind of data processing method and system
Technical field
The present invention relates to information technology field, more particularly to a kind of data processing method and data handling system.
Background technology
In recent years, big data processing has become global problem with analysis, as economic society is information-based and automation Level is continuously improved, and in many field face big data problems such as public administration, public service, scientific research, business application, needs There are various specific aims and cost-effective solution.Big data platform provides disposal ability for industry big data, collects data The functions such as access, data processing, data storage, query and search, analysis mining, application interface are integrated.
In data processing field, current environment increasingly payes attention to the accumulation of data, increasing with data volume, right Handle the ability of data and have the requirement of higher, it is necessary to faster processing speed, the data of bigger to the basic framework of system Storage capacity and ease for maintenance.
, it is necessary to the data variation historical information of recording key section under some business scenarios, to meet the needs of users, Need periodically to be updated the data in database.In some big data platforms, file system is based on distribution The storage of formula file, i.e., file has been stored in different nodes, and traditional data to such data platform carry out history more New processing mode to described, it is necessary to have data progressive scan, i.e., in storage region since the first row of first file Scanning, until the data for finding needs are modified, but in face of growing data volume and the increasingly business of complexity, especially It is the big data epoch of the huge increasing of data volume, so carries out the scanning of all data, efficiency is low, and time-consuming, and especially data volume is got over It is big, it is necessary to query time and feedback time it is longer, can not meet timeliness demand in the case of current data volume is increasing, Cause existing data handling system due to computationally intensive, and the time-consuming reason such as longer, data handling system stability is poor, Easily there is system interim card, or even stuck situation.
The content of the invention
The embodiment of the present invention provides a kind of data processing method and data handling system, to solve existing data processing system System is due to the efficiency of data processing is low and time-consuming etc. reason, the problem of causing data handling system stability poor.
In order to solve the above-mentioned technical problem, an embodiment of the present invention provides a kind of data processing method, the described method includes：
Receive the first data acquisition system of external system transmission；
The second data set associated with target data set to be updated is generated in a data processing system；
Empty the data in the target data set；
The target data set is carried out using the data in first data acquisition system and the second data set Data update.
Further, second associated with target data set to be updated is generated in a data processing system described Before the step of data acquisition system, the described method includes：
The first keyword or critical field are determined from first data acquisition system；
Inquired about using first keyword or critical field in the target data set；
Either critical field or inquired and institute if inquiring first keyword in the target data set State the first keyword or data that critical field matches, perform the generation in a data processing system and mesh to be updated The step of the second data set that mark data acquisition system is associated.
Further, carried out described using first keyword or critical field in the target data set After the step of inquiry, the described method includes：
If not inquiring first keyword or critical field in the target data set, and do not inquire The data to match with first keyword or critical field, by the data update of first data acquisition system to the mesh Mark in data acquisition system.
Further, the data using in first data acquisition system and the second data set are to the target Data acquisition system carries out the step of data update, including：
The second keyword or critical field are determined from the second data set；
Inquired about using second keyword or critical field in first data acquisition system；
If do not inquire second keyword or critical field in first data acquisition system, and described The data to match with second keyword or critical field are not inquired in one data acquisition system, by second data set In conjunction with the data update that second keyword or critical field match into the target data set；
By in first data acquisition system with the data update that first keyword or critical field match to institute State in target data set.
Further, it is described to use first data acquisition system when the target data set is combined into slide fastener data acquisition system The step of data update is carried out to the target data set with the data in the second data set, including：
The second keyword or critical field are determined from the second data set；
Inquired about using second keyword or critical field in first data acquisition system；
If do not inquire second keyword or critical field in first data acquisition system, and described The data to match with second keyword or critical field are not inquired in one data acquisition system, by second data set In conjunction with the data update that second keyword or critical field match into the target data set；
Determine the first slide fastener number to match in the second data set with first keyword or critical field According to；
The closed chain time for changing the first sub- slide fastener data in open chain state in the first slide fastener data is generation institute The time of the second data set is stated, and is based in first data acquisition system and first keyword or critical field phase The data matched somebody with somebody, generate the second sub- slide fastener data of the first slide fastener data, wherein, during the open chain of the second sub- slide fastener data Between to generate the time of the second data set, the closed chain time is empty or maximum；
If the number to match with first keyword or critical field is not inquired in the second data set According to based on data the second slide fastener of generation to match in first data acquisition system with first keyword or critical field Data, wherein, the open chain time of the second slide fastener data, the closed chain time was sky to generate the time of the second data set Or maximum；
By amended first slide fastener data and the second slide fastener data update into the target data set.
Further, after the step of data emptied in the target data set described, the described method includes：
Carry out occurring renewal mistake during data update to the target data set if detecting, use the second number of generation According to the data in target data set described in the data recovery in set；Or
If occurring data update mistake when detecting and carrying out data update to the target data set, backup in advance is obtained Backup Data set, use the data in target data set described in the data recovery in the Backup Data set.
Further, it is described to generate second number associated with target data set to be updated in a data processing system The step of according to set, including：
Obtain in the preset time period before receiving first data acquisition system, in updated target data set All data stored, or obtain receive first data acquisition system after, in this target data set to be updated Data, all data stored in the updated target data set of backup or this target data set to be updated Data in conjunction are to generate the second data set；Or
The acquisition last time receives the second data set generated during the first data acquisition system, by presently described target data set Data in conjunction be inserted into it is last receive in the second data set generated during the first data acquisition system, to generate this institute State the second data set.
The embodiment of the present invention also provides a kind of data handling system, and the data handling system includes：
Data memory module, for storing the internal data of the data handling system, and the data obtained from outside；
Business logic modules, for management and control service logic；
Data service module, for providing data service to the external system of data handling system；
Data processing engine module, for handling data.
Further, the data handling system includes：
Information exchange module, for receiving operational order input by user, the data handling system is managed and Set.
Further, the data memory module is distributed file storage system, data memory module storage from The data that outside obtains include direct extraction-type data and document form data.
Further, the business logic modules include：
Storage unit, for storing the service logic of the data handling system, the service logic include it is following at least One of：Scheduling rule, data genetic connection, model metadata and wscript.exe.
Further, the data service module includes：
Push unit, for the external system pushed information queue of data handling system and data；
Unit is achieved, for storage file form data；
Data transmission interface unit, is connected for the down-stream system or service system with data handling system, passes through institute State interface unit and provide data for the down-stream system or service system.
Further, the data handling system further includes automation tools module, and the automation tools module includes：
Parameter receiving unit, for receiving the parameter of input；
Script generation unit, for based on preset rules and the parameter, generating automation tools script.
Further, the data processing engine module includes：
Receiving unit, for receiving the first data acquisition system of external system transmission；
Generation unit, for generating second number associated with target data set to be updated in a data processing system According to set；
Clearing cell, for emptying the data in the target data set；
First updating block, for using the data in first data acquisition system and the second data set to described Target data set carries out data update.
Further, the data processing engine module further includes：
First determination unit, for determining the first keyword or critical field from first data acquisition system；
Query unit, for being looked into using first keyword or critical field in the target data set Ask；
Execution unit, if for inquiring first keyword or critical field in the target data set, The data to match with first keyword or critical field are either inquired, are given birth in a data processing system described in execution The step of into the second data set associated with target data set to be updated.
Further, the data processing engine module further includes：
Second updating block, if for not inquiring first keyword or key in the target data set Field, and the data to match with first keyword or critical field are not inquired, by first data acquisition system Data update into the target data set.
Further, first updating block includes：
First determination subelement, for determining the second keyword or critical field from the second data set；
First inquiry subelement, for using second keyword or critical field in first data acquisition system Inquired about；
First renewal subelement, if for not inquiring second keyword or pass in first data acquisition system Key field, and do not inquire the number to match with second keyword or critical field in first data acquisition system According to, by the second data set with the data update that second keyword or critical field match to the target In data acquisition system；
Second renewal subelement, for by first data acquisition system with first keyword or critical field phase Matched data update is into the target data set.
Further, when the target data set is combined into slide fastener data acquisition system, first updating block includes：
Second determination subelement, for determining the second keyword or critical field from the second data set；
Second inquiry subelement, for using second keyword or critical field in first data acquisition system Inquired about；
3rd renewal subelement, if for not inquiring second keyword or pass in first data acquisition system Key field, and do not inquire the number to match with second keyword or critical field in first data acquisition system According to, by the second data set with the data update that second keyword or critical field match to the target In data acquisition system；
3rd determination subelement, determine in the second data set with first keyword or critical field phase The the first slide fastener data matched somebody with somebody；
Subelement is changed, is closed for changing the first sub- slide fastener data in open chain state in the first slide fastener data The chain time to generate the time of the second data set, and based in first data acquisition system with first keyword or The data that person's critical field matches, generate the second sub- slide fastener data of the first slide fastener data, wherein, second son is drawn The open chain time of chain data, the closed chain time was empty or maximum to generate the time of the second data set；
Subelement is generated, if for not inquired in the second data set and first keyword or key The data that field matches, based on the number to match in first data acquisition system with first keyword or critical field According to the second slide fastener data of generation, wherein, open chain time of the second slide fastener data for generate the second data set when Between, the closed chain time is empty or maximum；
4th renewal subelement, for by amended first slide fastener data and the second slide fastener data update to described In target data set.
Further, the data processing engine module includes：
First recovery unit, if carrying out renewal mistake occur during data update to the target data set for detecting By mistake, the data in target data set described in the data recovery in the second data set of generation are used；Or
Second recovery unit, if there is data update when carrying out data update to the target data set for detecting Mistake, obtains the Backup Data set backed up in advance, uses target data described in the data recovery in the Backup Data set Data in set.
Further, the generation unit is additionally operable to obtain the preset time period received before first data acquisition system All data that are interior, having been stored in updated target data set, or after obtaining and receiving first data acquisition system, Data in this target data set to be updated, back up all data stored in updated target data set Or the data in this target data set to be updated are to generate the second data set；
Alternatively, the generation unit, which is additionally operable to the acquisition last time, receives the second data set generated during the first data acquisition system Close, by the data in presently described target data set be inserted into it is last receive the first data acquisition system when the second number for generating According in set, to generate this second data set.
Data processing method and data handling system provided in an embodiment of the present invention, receive the first number of external system transmission According to set；The second data set associated with target data set to be updated is generated in a data processing system；Empty institute State the data in target data set；Using the data in first data acquisition system and the second data set to the mesh Mark data acquisition system and carry out data update.In this way, the first data acquisition system of external system transmission is being received, it is necessary to carry out data more When new, it is ensured that the stability of data handling system, without being scanned to all data, saves the plenty of time, and improve The efficiency of data update.
Brief description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, needed in being described below to the embodiment of the present invention Attached drawing to be used is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, For those of ordinary skill in the art, without having to pay creative labor, can also be obtained according to these attached drawings Obtain other attached drawings.
Fig. 1 is the flow chart for the data processing method that one embodiment of the invention provides；
Fig. 2 is the flow chart for the data processing method that another embodiment of the present invention provides；
Fig. 3 is the service information list that the data before representing not update in target data set represent；
Fig. 4 is to represent the information table that the data in the first data acquisition system represent；
Fig. 5 is to represent the information table that the data in the second data set represent；
Fig. 6 and Fig. 7 is the process schematic for representing to be updated the information that the data in target data set represent；
Fig. 8 is to represent the service information list that the data in the target data set after updating represent；
Fig. 9 is the service information list that the slide fastener data before representing not update in target data set represent；
Figure 10 is to represent the information table that the slide fastener data in the second data set represent；
Figure 11 is to represent the information table that the data in the first data acquisition system represent；
Figure 12 and Figure 13 is the process schematic for representing to be updated the information that the data in target data set represent；
Figure 14 is to represent the service information list that the data in the target data set after updating represent；
Figure 15 is the structure chart for the data handling system that one embodiment of the invention provides；
Figure 16 is one of structure chart of data processing engine module of data handling system shown in Figure 15；
Figure 17 is two of the structure chart of the data processing engine module of data handling system shown in Figure 15；
Figure 18 is three of the structure chart of the data processing engine module of data handling system shown in Figure 15；
The four of the structure chart of the data processing engine module of data handling system shown in Figure 19 Figure 15；
Figure 20 is one of structure chart of the first updating block shown in Figure 16；
Figure 21 is the two of the structure chart of the first updating block shown in Figure 16.
Embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete Site preparation describes, it is clear that described embodiment is part of the embodiment of the present invention, instead of all the embodiments.Based on this hair Embodiment in bright, the every other implementation that those of ordinary skill in the art are obtained without creative efforts Example, belongs to the scope of protection of the invention.
Referring to Fig. 1, Fig. 1 is the flow chart for the data processing method that one embodiment of the invention provides.The method can answer For data handling system, as shown in Figure 1, the described method comprises the following steps：
Step 101, the first data acquisition system for receiving external system transmission.
, it is necessary to the data variation historical information of recording key section under some business scenarios, to meet customer need, example Such as in financial field, certain customer banking account remaining sum change histories information need to be recorded, to meet customer inquiries bank account balances Demand.It is therefore desirable to periodic data update is carried out to the data in database.
Therefore, in this step, the first data set that the reception of data handling system meeting periodicity is transmitted from external system Close.
Wherein, the reception of data handling system periodicity first data acquisition system, can be with time limit fixed cycle Receive first data acquisition system, such as 1 day receive once, or 12 it is small when receive once；For the timeliness of data, at data Reason system can also be real-time reception or approximate real time reception first data acquisition system, and such as 1 receives once when small, or Person's half an hour receives once, or even a few minutes reception one is inferior, does not do any restriction.Can be to include in first data acquisition system There are modification data in batch.
Wherein, first data acquisition system, can be the set of single data, such as the set of single type of service data, Such as only comprising deposit data either flowing water expenditure data financial transaction data or the number of single client or target According to, Zhang San is such as only included, or the associated traffic data of Li Si or the set of integrated data are only included, such as comprising not , such as can be at the same time comprising deposit data and flowing water expenditure data financial transaction data, and communication with the data of type of service Data etc., can also include the data of multiple clients or target at the same time, such as include the related service number of Zhang San and Li Si at the same time According to etc..
Wherein, the first data acquisition system of external system transmission is received, can directly receive the first data from external system Set or by the related memory module of data handling system after the first data acquisition system of external system is stored, The first data acquisition system of external system is obtained from memory module.
Step 102, generate second data set associated with target data set to be updated in a data processing system Close.
In the step, after the data handling system receives first data acquisition system, the data processing system System can be controlled in the data handling system, generate one associated with the target data set to be updated second Data acquisition system.
Wherein, the type of the second data set associated with the target data set, can be second data The data type included in set the either type represented by data and data type or data in the target data set Represented type is identical.
For example, for example, the data in the target data set be certain client Zhang San cash in banks data or industry Be engaged in pipelined data etc., then the data in the second data set of generation be also client Zhang San cash in banks data or Business pipelined data, and if the data in the target data set include the cash in banks data or industry of certain client Zhang San Business pipelined data, and the cash in banks data or business pipelined data of Li Si, then the second data set of generation In data be also client Zhang San cash in banks data either the cash in banks data of business pipelined data and Li Si or Business pipelined data.
Wherein, the data included in the second data set, can be most complete data, i.e., described second data set Conjunction is the data included in the data acquisition system of time span maximum, such as the second data set, can be from described in generation Target data set runs the beginning jointly, ends all data recorded in target data set described in current time, that is to say, that described The second data set records the time span of data, is until the current time, that is, most since producing the business datum The long time.
Wherein, data that are corresponding, being included in the target data set, can be most complete data, i.e., described Target data set is the data acquisition system of time span maximum；The data that are included in the target data set or only Comprising the partial data in most complete data, such as only comprising last renewal to the data of this reproducting periods, i.e., only include Data in one update cycle, or the data in several update cycles.
Preferably, the data in the target data set and the second data set, all be comprising maximum time across Data message in degree.
Wherein, the second data set is the time-domain snapshot data set of the target data set.
Can be usage history data i.e. this renewal by way of backup for generating the second data set The mode that preceding target data set is backed up generates the second data set.
Further, the second data set can be generated or updated based on set frequency, such as described The generation of the second data set or renewal frequency could be provided as once a day, it is preferred that the generation of the second data set Or renewal frequency, can be with batch change data transfer to data handling system frequency it is identical, i.e., with data processing system The frequency of reception first data acquisition system for periodicity of uniting is identical.
In this way, after data handling system receives the first data acquisition system, can be by controlling generation and target data set Associated the second data set is closed, target data set is determined without being scanned to all data in data handling system The position of middle data and back end, it is time saving and energy saving, the workload of data handling system can be effectively reduced, improves work effect Rate.
Step 103, empty data in the target data set.
In the step, when data handling system control generates second data in the data handling system After set, the data handling system, which can control, empties the data in the target data set, so as to subsequently to institute State and data update is carried out in target data set.
Step 104, using the data in first data acquisition system and the second data set to the target data Set carries out data update.
In the step, after the data handling system empties the data in the target data set, the data Processing system can extract the data for needing to update in first data acquisition system, and the dependency number in the second data set According to be inserted into, add or write in the target data set, so that the target data set is carried out data update.
Preferably, it is by first data acquisition system and described second by the way of inquiry is inserted into present embodiment Data update in data acquisition system carries out more the data in the target data set into the target data set Newly.
For example, for example, the data in the target data set be certain client Zhang San cash in banks data or industry Business pipelined data etc., then using the data in first data acquisition system and the second data set to the target data Set carries out data update, it is possible to is the new cash in banks data or business for using Zhang San in first data acquisition system The passing cash in banks data of Zhang San or business pipelined data in pipelined data, and the second data set, to store Into the target data set, data update, or such as described target data are carried out to the target data set Data in set are the cash in banks data or business pipelined data of certain client Zhang San, and the cash in banks of client Li Si Data or business pipelined data, and such as this is to need to be updated the data of Zhang San, i.e., described first data acquisition system In have the new cash in banks data or business pipelined data of Zhang San, then can use in first data acquisition system The passing bank of Zhang San deposits in the new cash in banks data or business pipelined data, and the second data set of Zhang San Amount of money is according to the either passing cash in banks data or business pipelined data of business pipelined data and client Li Si, to deposit Storage carries out data update into the target data set, to the target data set.
Wherein, data update or periodically renewal, such as renewal one in one day are carried out to the target data set It is secondary, or 12 it is small when renewal it is one inferior, it is preferred that the update cycle of the target data set, can be with the data The frequency that reason system receives first data acquisition system is identical.
In this way, after data handling system receives the first data acquisition system, can be by generating and number of targets to be updated According to the associated the second data set of set, and after emptying target data set, by the first data acquisition system and the second data set In data inquire about be inserted into by way of be inserted into target data set, to be updated to target data set, without right Data handling system carries out the scanning of total data, you can completes the renewal of data in target data set, can save totally The time of scanning, and then the workload of data handling system is effectively reduced, improve work efficiency.
In the embodiment of the present invention, above-mentioned data handling system, can be put down for developing and running the backstage of processing data Platform etc., realizes and carries out Distributed Calculation to mass data in the cluster of a large amount of computers composition, it is preferred that the data processing System is big data platform.
Above-mentioned data handling system, can be applied to the big data application of financial system, medical system and educational system etc. Scene, such as bank data system, hospital data system and school's data system.
Data processing method provided in an embodiment of the present invention, receives the first data acquisition system of external system transmission；In data The second data set associated with target data set to be updated is generated in processing system；Empty the target data set In data；The target data set is carried out using the data in first data acquisition system and the second data set Data update.In this way, it can pass through, it is necessary to when carrying out data update in the first data acquisition system for receiving external system transmission Extraction and the relevant data of data in the target data set in a data processing system, so as to generate and mesh to be updated The second data set that mark data acquisition system is associated, then by the first data acquisition system and the second data by way of inquiring about and being inserted into In data insertion target data set in set, to be updated to the data in target data set, without to all numbers It is scanned according to node, you can the renewal of data in target data set is completed, the plenty of time of scan full hard disk can be saved, And then the workload of data handling system is effectively reduced, improve the efficiency of data update.
Referring to Fig. 2, Fig. 2 is the flow chart for the data processing method that another embodiment of the present invention provides.The method application In data handling system, as shown in Fig. 2, the described method comprises the following steps：
Step 201, the first data acquisition system for receiving external system transmission.
Step 202, determine the first keyword or critical field from first data acquisition system.
In the step, after the data handling system receives the first data acquisition system of external system transmission, the number Can be according to the data for needing to store or updating in first data acquisition system, from first data set according to processing system Corresponding first keyword or critical field are determined in conjunction.
Wherein, first keyword or critical field, only refer to, for example, first data acquisition system include it is more The business datum of a type either the data of multiple clients when can be the business datum to each type or each visitor respectively The data at family are updated, every time the business datum to corresponding type or during the data update of client, corresponding type The data of business datum either client all have corresponding first keyword or critical field.
Wherein, the first keyword or critical field, can be set according to the actual requirements, such as use a keyword It can represent data to be updated, you can only definite keyword, otherwise, it is necessary to the critical field of multiple keywords composition It could represent to treat the data more gone, i.e., it needs to be determined that critical field.
Step 203, inquired about using first keyword or critical field in the target data set.
In the step, after the data handling system determines first keyword or critical field, the data Processing system can be controlled is inquired about using first keyword or critical field, i.e., using first keyword or Person's critical field is inquired about in the target data set, so that the data handling system can be learnt by inquiry, Whether there is the data match represented with first keyword or critical field in the target data set Data historical information or data record etc..
If step 204, inquire first keyword or critical field, Huo Zhecha in the target data set The data to match with first keyword or critical field are ask, generates and treats in a data processing system described in execution The step of the second data set that the target data set of renewal is associated.
In the step, when the data handling system using first keyword or critical field in the number of targets According to being inquired about in set, and first keyword or critical field are inquired in the target data set, or If person inquires the data for existing in the target data set and matching with first keyword or critical field, that The data handling system can think exist and first keyword or keyword in the target data set The passing information for the data that section matches, then the data handling system can control execution described in data handling system Middle the step of generating the second data set associated with target data set to be updated, so as to be completed by subsequent action Data in the target data set are updated.
Wherein, the data to match with first keyword or critical field are inquired, can be referred in the mesh When being inquired about in mark data acquisition system, since some data are the problems such as putting in order, it may be displayed in compared with rearward position, this Sample may expend the time if directly inquiring first keyword or critical field longer, at this moment, when inquiring and institute State the first keyword or data that critical field matches if, it is possible to be considered to have inquired first keyword or Person's critical field, in this way, the time can be saved, reduces data scanning amount.
Wherein, the data to match with first keyword or critical field, can be and first keyword Such as described first keyword of data that is associated of data or critical field that either critical field represents are, Zhang Sanhuo The ID of person Zhang San, then first keyword data that either critical field matches can be represent, Zhang San or The data of three ID or represent, Zhang San either Zhang San ID some date some deposits or Flow Record Data, can either represent, data of information such as the telephone number of the ID of Zhang San or Zhang San or identification card number etc..
Step 205, generate second data set associated with target data set to be updated in a data processing system Close.
Step 206, empty data in the target data set.
Step 207, using the data in first data acquisition system and the second data set to the target data Set carries out data update.
Wherein, the description of step 201 and step 205 to step 207 is referred to step 101 in above-described embodiment to step Rapid 104 description, this will not be repeated here.
Optionally, after step 203, the described method includes：
If not inquiring first keyword or critical field in the target data set, and do not inquire The data to match with first keyword or critical field, by the data update of first data acquisition system to the mesh Mark in data acquisition system.
In the step, when the data handling system using first keyword or critical field in the number of targets According to being inquired about in set, and do not inquire first keyword or critical field in the target data set, and And if not inquiring the data to match with first keyword or critical field in the target data set, that The data handling system can think, the first keyword or critical field described in the first object data acquisition system The data of expression, are brand-new data for the target data set, the data handling system can be direct By the data insertion in first data acquisition system, addition or write into the target data set, so as to described Data in target data set are updated.
Optionally, step 207 includes：
Determined from the second data set the second keyword either critical field using second keyword or Critical field is inquired about in first data acquisition system, is closed if not inquiring described second in first data acquisition system Key word critical field and does not inquire and second keyword or critical field either in first data acquisition system The data to match, the data update that will be matched in the second data set with second keyword or critical field Into the target data set；By what is matched in first data acquisition system with first keyword or critical field Data update is into the target data set.
In the step, after the data handling system empties the data in the target data set, the data Processing system can be according to the data included in the second data set, to determine the second keyword or critical field, so Inquired about afterwards using second keyword or critical field in first data acquisition system, to inquire about first number The data to match according to whether having in set with second keyword or critical field, if the data handling system is led to Inquiry is crossed, determines not inquiring second keyword or critical field in first data acquisition system, also, determine Do not inquired in first data acquisition system with second keyword or the matched data of critical field, the data Processing system can consider in the second data set and is not required to second keyword or the matched data of critical field Update, so, the data handling system can by the second data set with second keyword or critical field In the target data set of the data update to match to after emptying, then further according to first keyword or key Field, will determine to extract with the data that first keyword or critical field match from first data acquisition system Go out, and by the target data set of the data update extracted to after emptying, so as to complete to the target data set The data update of conjunction.
Wherein, the data update that will be matched in the second data set with second keyword or critical field To in the target data set, and by first data acquisition system with first keyword or critical field phase The data update matched somebody with somebody can be by after inquiry, by data by being inserted into, adding or write into the target data set The mode such as enter, be updated in the target data set.
For example, refer to Fig. 3 and represent that the data before not updating in target data set are represented into Fig. 5, such as Fig. 3 Service information list, the information table that the data in the first data acquisition system represent is represented in Fig. 4, the second data set is represented in Fig. 5 In the information table that represents of data, Fig. 6 and Fig. 7 represent the mistake being updated to the information that the data in target data set represent Journey schematic diagram, the service information list that the data in the target data set after renewal represent is represented in Fig. 8.Such as the institute before not updating State the credit balance information that data in target data set represent Zhang San and Li Si, the tables of data in first data acquisition system Show the related deposit business information that the personnel Zhang San of business handling, king five, Zhao six etc. are carried out in the past period, described the All business information of same personnel in the data expression target data set in two data acquisition systems, i.e. Zhang San and Li Si Credit balance information.
It is so when carrying out data update to the target data set, the data in the target data set are clear Sky, i.e., after the information in the tables of data in Fig. 3 is emptied, the blank letter shown in the Fig. 6 for the target data expression for obtaining blank Cease table；Then the data handling system can determine the second keyword or critical field from the second data set (such as ID of the ID of Zhang San either Li Sis) is then according to second keyword or critical field in first data set Inquired about in conjunction, whether inquire about in first data acquisition system has what is matched with second keyword or critical field Data, i.e., inquire about the data for whether having the related service information for representing Zhang San or Li Si, such as in first data acquisition system Fruit does not inquire the data to match with second keyword or critical field in first data acquisition system, as in institute The data for not inquiring in the first data acquisition system and matching with the keyword of Li Si or critical field are stated, mean that this data Renewal, the business datum of no Li Si needs to update, then can will be crucial with described second in the second data set The data that word or critical field match, i.e., with the relevant business datum of Li Si, be updated to the target data after emptying In set, so as to complete first step renewal, the related service information table of the Li Si shown in Fig. 7 is obtained, whereas if described the The data to match with second keyword or critical field are inquired in one data acquisition system, such as in first data set The data to match with the keyword or critical field for representing Zhang San are inquired in conjunction, mean that this data update, has and opens Three business datum needs to be updated, and is just not required to for the business datum of Zhang San in the second data set to be added to the institute after emptying State in target data set, i.e., will not match with second keyword or critical field in the second data set Data are added to the target data set after emptying；Then, the data handling system can be according to first data The first keyword or critical field in set, such as relevant first keyword of business datum with Zhang San, king five and Zhao six Or critical field (such as ID of Zhang San, king five and Zhao six), by first data acquisition system with first keyword or The data that person's critical field (such as ID of Zhang San, king five and Zhao six) matches are added directly to the target data after emptying In set, so as to complete the data update to the target data set so that obtain Fig. 8 shows renewal after the target The service information list that data in data acquisition system represent.
Optionally, when the target data set is combined into slide fastener data acquisition system, step 207 includes：
Inquired about using second keyword or critical field in first data acquisition system；If described Second keyword or critical field are not inquired in one data acquisition system, and is not inquired about in first data acquisition system , will be crucial with described second in the second data set to the data to match with second keyword or critical field The data update that word or critical field match is to the target data set；Determine in the second data set with it is described The first slide fastener data that first keyword or critical field match, change in the first slide fastener data and are in open chain state The first sub- slide fastener data the closed chain time to generate the time of the second data set, and be based on first data acquisition system In the data that match with first keyword or critical field, generate the second sub- slide fastener number of the first slide fastener data According to, wherein, the open chain time of the second sub- slide fastener data, the closed chain time was sky to generate the time of the second data set Or maximum；If the number to match with first keyword or critical field is not inquired in the second data set According to based on data the second slide fastener of generation to match in first data acquisition system with first keyword or critical field Data, wherein, the open chain time of the second slide fastener data, the closed chain time was sky to generate the time of the second data set Or maximum；By amended first slide fastener data, the second slide fastener data update into the target data set.
In the step, if the target data set is combined into slide fastener data acquisition system, i.e., the number in described target data set According to for slide fastener data, after the data handling system empties the data in the target data set, the data Processing system can be according to the data included in the second data set, to determine the second keyword or critical field, so Inquired about afterwards using second keyword or critical field in first data acquisition system, to inquire about first number The data to match according to whether having in set with second keyword or critical field, if the data handling system is led to Inquiry is crossed, determines not inquiring second keyword or critical field in first data acquisition system, also, determine Do not inquired in first data acquisition system with second keyword or the matched data of critical field, the data Processing is it is considered that need not be more with second keyword or the matched data of critical field in the second data set Newly, so, the data handling system can by the second data set with second keyword or critical field phase In the target data set of the data update matched somebody with somebody to after emptying.
Then, the data handling system can use first keyword or critical field in second data Inquired about in set, if there are the data to match with first keyword or critical field in the second data set Words, the data handling system can determine and first keyword or critical field in the second data set The the first slide fastener data to match, then the data handling system can modify to the first slide fastener data so that The closed chain time for the first sub- slide fastener data that open chain state is in the first slide fastener data is arranged to generate described second The time of data acquisition system, further, when the data handling system can reset the open chain of the first slide fastener data Between, that is, the sub- slide fastener data of second in open chain state of the first slide fastener data are generated, specifically, the data processing system System can obtain the data to match in first data acquisition system with first keyword or critical field, and according to institute The data to match in the first data acquisition system with first keyword or critical field are stated, to generate the first slide fastener number According to the second sub- slide fastener data, open chain time of the second sub- slide fastener data to generate the time of the second data set, The closed chain time is empty or maximum, and expression is in open chain state up to now.
If the data handling system do not inquired in the second data set with first keyword or If the data that critical field matches, then just illustrate in first data acquisition system with first keyword or key The data that field matches all are new data, the data handling system can according in first data acquisition system with institute State the first keyword or data that critical field matches, to generate new slide fastener data, i.e. the second slide fastener data, wherein, The open chain time of the second slide fastener data, the closed chain time was empty or maximum to generate the time of the second data set.
Finally, the data handling system is by amended first slide fastener data and newly-generated the second slide fastener number According to being updated in the target data set, the data update to the target data set is completed.
For example, the slide fastener number before not updating in target data set is represented please refer to Fig. 9 to Figure 11, in Fig. 9 Represent the information table that the slide fastener data in the second data set represent according to the service information list of expression, in Figure 10, Figure 11 represents the The information table that data in one data acquisition system represent, Figure 12 and Figure 13 represent the information represented the data in target data set The process schematic being updated, the service information list that the data in the target data set after renewal represent is represented in Figure 14. It is described such as the slide fastener data expression Zhang San in the target data set before not updating and the balance of deposits managing detailed catalogue of Li Si Data in first data acquisition system represent to carry out the personnel Zhang San, king five, Zhao six of business handling etc. in the past period Related deposit business information, the data in the second data set represent deposit personnel identical with the target data set All business information, i.e., the data in described the second data set represent the managing detailed catalogue of the balance of deposits of Zhang San and Li Si.
It is so when carrying out data update to the target data set, the data in the target data set are clear Sky, i.e., after the information in the tables of data in Fig. 9 is emptied, the blank letter shown in the Figure 12 for the target data expression for obtaining blank Cease table；Then the data handling system can determine the second keyword or critical field from the second data set (such as ID of the ID of Zhang San either Li Sis) is then according to second keyword or critical field in first data set Inquired about in conjunction, whether inquire about in first data acquisition system has what is matched with second keyword or critical field Data, i.e., inquire about the data for whether having the related service information for representing Zhang San or Li Si, such as in first data acquisition system Fruit does not inquire the data to match with second keyword or critical field in first data acquisition system, as in institute The data for not inquiring in the first data acquisition system and matching with the keyword of Li Si or critical field are stated, mean that this data Renewal, the business datum of no Li Si needs to update, then can will be crucial with described second in the second data set The data that word or critical field match, i.e., with the relevant business datum of Li Si, be updated to the target data after emptying In set, so as to complete first step renewal, the detail list of the related service information of Li Si shown in Figure 13 is obtained；Then, use First keyword or critical field are inquired about in the second data set, if in the second data set In inquire the data to match with first keyword or critical field, such as inquired in the second data set The data to match with the keyword or critical field for representing Zhang San, mean that this data update, the business number for having Zhang San According to needing to be updated, then, the data handling system can be crucial according to described first in the second data set Word or critical field determine to represent the first of the data of the related service information of Zhang San, i.e. the deposit information detail of expression Zhang San Slide fastener data, are then by the closed chain time modification for the first sub- slide fastener data that open chain state is in the first slide fastener data The time of the second data set is generated, i.e. the time of data update (carries out data update to the target data set Time), and according to the data to match in first data acquisition system with first keyword or critical field, that is, represent The data of the new business information of Zhang San, to generate the slide fastener data of the deposit information detail of a new expression Zhang San, i.e., second Sub- slide fastener data, set the open chain time of the second sub- slide fastener data as the time of the generation the second data set, i.e. data Renewal time, closed chain time are empty or maximum；Then by the second data set with first keyword or pass The data that key field matches, i.e., with the relevant business datum of Zhang San, be updated in the target data set after emptying, from And complete second step renewal；, whereas if do not inquired in the second data set and first keyword or pass The data that key field matches, do not inquire the data for representing the relevant information of king five and Zhao six such as, then the data processing system System can according in first data acquisition system with first keyword or critical field, i.e., in described first data acquisition system Represent the relevant data of king five and Zhao six, come generate in first data acquisition system with first keyword or keyword The second slide fastener data of data that section matches, to represent the detail list of the related service information of king five and Zhao six, and can be with The open chain time for setting the second slide fastener data is the data update time, and the closed chain time is empty or maximum；Then by described in First slide fastener data, that is, represent the data of the related service of Zhang San, and the second slide fastener data of generation, that is, represent king five and Zhao Six relevant business datum, is updated in the target data set after emptying, to complete to the target data set Data update so that obtain Figure 14 expression renewal after the target data set in data represent business information Table.
Optionally, after step 201, the described method includes：
Carry out occurring renewal mistake during data update to the target data set if detecting, use the second number of generation According to the data in target data set described in the data recovery in set.
In the step, complete data update in the target data set or carry out data in the target data set During renewal, the data handling system can monitor the data update of the target data set in real time, if prison Measure and carry out occurring renewal mistake during data update to the target data set, i.e., go out in step 206 and/or step 207 When now updating the situation of mistake, the data handling system can carry out data recovery to the target data set, specifically, The data handling system can obtain the second data set generated in step 205, then using the second data of generation Data in target data set described in data recovery in set.
After the data in recovering the target data set, the data handling system, which can control, stops data more Newly.
Here it is possible to directly use the second data set, i.e., the time-domain snapshot data set of target data set carries out Data recovery, simple and fast, the opposite data refresh mode shorter suitable for the data update cycle.
If alternatively, occurring data update mistake when detecting and carrying out data update to the target data set, obtain pre- The Backup Data set first backed up, uses the number in target data set described in the data recovery in the Backup Data set According to.
In the step, complete data update in the target data set or carry out data in the target data set During renewal, the data handling system can monitor the data update of the target data set in real time, if prison Measure and carry out occurring renewal mistake during data update to the target data set, the data handling system can be to the mesh Mark data acquisition system and carry out data recovery, specifically, the data handling system can obtain the backup data set backed up in advance Close, data recovery then is carried out to the target data set using the data in the Backup Data set.
Wherein, the backup cycle of the Backup Data set, can be the setting as needed for carrying out backup cycle, such as standby Part data volume of 1 month.
Wherein, the Backup Data set can be the data preserved in first data acquisition system, and to described second Data in data acquisition system, i.e. time-domain snapshot data set back up a full dose data according to default backup cycle.
After the data in recovering the target data set, the data handling system, which can control, stops data more Newly.
Here, using back mechanism, that is, data how long is backed up and just recover data how long, such as have been backed up one month Data just recover the data of one month, simple and fast, the opposite data refresh mode longer suitable for the data update cycle.
In present embodiment, there is data update mistake when monitoring and carrying out data update to the target data set When, data recovery can be carried out using the rollback of above two mode, but be not limited thereto, in other embodiments, Data update false alarm can be ignored, continue data update, can also be after rollback recovery data, re-start Data update.
Optionally, step 205 includes：
Obtain in the preset time period before receiving first data acquisition system, in updated target data set All data stored, or obtain receive first data acquisition system after, in this target data set to be updated Data, all data stored in the updated target data set of backup or this target data set to be updated Data in conjunction are to generate the second data set.
Can be the mode of usage history data backup by way of backup for generating the second data set Generate the second data set.
Therefore, in this step, after the data handling system receives first data acquisition system, at the data Reason system can be detected historical data, using after first data acquisition system is received, this target to be updated The mode that data acquisition system is backed up generates the second data set；Alternatively, obtaining first data are received at this In preset time period before set, all data for being stored in updated target data set, so that by more All data backups stored in the target data set newly crossed are into a set, so as to generate second data set Close.
Alternatively, obtaining the last time receives the second data set generated during the first data acquisition system, by presently described target Data in data acquisition system be inserted into it is last receive in the second data set generated during the first data acquisition system, to generate this The secondary the second data set.
Can be by way of to available data insertion renewal, with reference to existing for generating the second data set Historical data generate the second data set.
Therefore, in the step, after this described data handling system receives first data acquisition system, the data Processing system can be obtained before this receives first data acquisition system, when the last time receives the first data acquisition system The second data set of generation, then, then obtains the data in the target data set, and by the target data set Data be inserted into it is last receive in the second data set generated during the first data acquisition system so as to generate this institute State the second data set.
Data processing method provided in an embodiment of the present invention, receives the first data acquisition system of external system transmission；From described The first keyword or critical field are determined in first data acquisition system；Using first keyword or critical field described Inquired about in target data set；If first keyword or keyword are inquired in the target data set Section, either inquires the data to match with first keyword or critical field, performs described in data handling system Middle the step of generating the second data set associated with target data set to be updated；In a data processing system generation with The second data set that target data set to be updated is associated；Empty the data in the target data set；Using institute The data stated in the first data acquisition system and the second data set carry out data update to the target data set.In this way, The data in the first data acquisition system and the second data set are used by way of inquiring about and being inserted into the number in target data set According to being updated, without being scanned to all data and node, you can the renewal of data in target data set is completed, can be with The plenty of time of scan full hard disk is saved, and then effectively reduces the workload of data handling system, improves the efficiency of data update.
Referring to Figure 15 to Figure 21, Figure 15 be one embodiment of the invention provide data handling system structure chart, Tu16Wei One of structure chart of data processing engine module of data handling system shown in Figure 15, Figure 17 are data processing shown in Figure 15 Two, Figure 18 of the structure chart of the data processing engine module of system is the data processing engine of data handling system shown in Figure 15 The four of the structure chart of the data processing engine module of data handling system shown in three, Figure 19 Figure 15 of the structure chart of module, figure 20 be one of structure chart of the first updating block shown in Figure 16, and Figure 21 is the structure of the first updating block shown in Figure 16 The two of figure.As shown in figure 15, data handling system 1500 includes data memory module 1510, business logic modules 1520, data Service module 1530 and data processing engine modules 1540.
The data handling system 1500 can be a kind of data engineering platform (Data Engineering Platform, DEP)。
Wherein, the data memory module 1510 is used for the internal data for storing the data handling system 1500, and The data obtained from outside.
The data memory module 1510 can be distributed document storage (Hadoop Distributed File System, HDFS) system.HDFS systems are accumulation layer, for storing the internal data of DEP, and store DEP from external system The data of acquisition.DEP obtains data from external system, can be direct extract in data, such as system R DB2 Data, the data in database Cloud Server Oracle ExaData, the data of Excel forms, can also be document form Data, i.e., sent to the data of the data of DEP, such as textual form with document form, further includes unstructured data, such as Log daily records, audio/video multimedia file.
Wherein, the business logic modules 1520 are used for management and control service logic.The business logic modules 1520 can wrap The storage unit for the service logic for storing the data handling system is included, the service logic includes at least one following：Scheduling Rule, data genetic connection, model metadata and wscript.exe (such as automation tools) etc..
Wherein, the data service module 1530 is used to provide data service to the external system of data handling system, its Including：
Push unit 1531, for the queue of external system pushed information and data, such as PUSH message queue, push number According to database.
Unit 1532 is achieved, for storage file form data.
Data transmission interface (Representational State Transfer API, Rest API) unit 1533, is used In with the down-stream system of data handling system either service system be connected by the interface unit as the down-stream system or Service system provides data, such as reporting system, Analysis Service etc..
The data processing engine module 1540 is used to handle data, it can be structured query language (Structured Query Language) engine modules, abbreviation SQL engine modules, SQL engine modules can by Hive and/ Or the engine such as Spark is formed.
Optionally, the data handling system 1500 further includes：
Information exchange module 1550, for receiving operational order input by user, pipe is carried out to the data handling system Reason and setting.User can include business personnel (personnel on service line), operation maintenance personnel (personnel on technology line) etc., Yong Hujiao Mutual module can set corresponding UI user interfaces.
Optionally, the data handling system 1500 further includes automation tools module, it can be rule-based (such as logical Cross the method that the data processing method carries out data update) write automation tools (i.e. one section of program), it is only necessary to understand in DEP In which data acquisition system need by slide fastener method record change history, pass through the automation tools i.e. can be achieved the algorithm routine Automation generation, such as SQL statement is generated in Hive.
Wherein, the automation tools module can include：
Parameter receiving unit, for receiving the parameter of input.
Script generation unit, for based on preset rules and the parameter, generating automation tools script.
Specifically, the parameter receiving unit, the parameter of the input data processing system for receiving user, can be root According to the instruction write-in received parameter corresponding with described instruction.The parameter includes at least one following：The name of data acquisition system Title, field, data type.
For example, if wondering certain customer banking account remaining sum situation of change, i.e., it should be understood that the remaining sum (mesh of client The balance amount information table represented in mark data acquisition system) and revenue and expenditure detail (the balance detail information table represented in target data set), The carry out data update for generating and realizing in above-described embodiment can be automated by the corresponding automation tools of automation module Method (connection table inquiry compares and insertion algorithm) correlative code, real dynamic inquiry.Need to run based on business, put down in Hadoop Platform record data variation history operation, specifically can by Hadoop platform data variation historical record into HDFS.
Wherein, as shown in figure 16, the data processing engine module 1540 includes：
Receiving unit 1541, for receiving the first data acquisition system of external system transmission.
Generation unit 1542, for generating associated with target data set to be updated in a data processing system Two data acquisition systems.
Clearing cell 1543, for emptying the data in the target data set.
First updating block 1544, for using the data pair in first data acquisition system and the second data set The target data set carries out data update.
Wherein, the first data acquisition system of the external system transmission that the receiving unit 1541 receives, can be directly from outer Portion's system receives first data acquisition system or is stored in the number by the first data acquisition system that external system is transmitted After memory module 1510, first data acquisition system is obtained from the data memory module 1510.
Optionally, as shown in figure 17, the data processing engine module 1540 further includes：
First determination unit 1545, for determining the first keyword or critical field from first data acquisition system.
Query unit 1546, for using first keyword or critical field in the target data set into Row inquiry.
Execution unit 1547, if for inquiring first keyword or keyword in the target data set Section, either inquires the data to match with first keyword or critical field, performs described in data handling system Middle the step of generating the second data set associated with target data set to be updated.
Optionally, as shown in figure 17, the data processing engine module 1540 further includes：
Second updating block 1548, if for do not inquired in the target data set first keyword or Critical field, and the data to match with first keyword or critical field are not inquired, by first data The data update of set is into the target data set.
Optionally, as shown in figure 18, the data processing engine module 1540 further includes：
First recovery unit 1549, if being updated when carrying out data update to the target data set for detecting Mistake, uses the data in target data set described in the data recovery in the second data set of generation.
Alternatively, as shown in figure 19, the data processing engine module 1540 includes：
Second recovery unit 15410, if there is number when carrying out data update to the target data set for detecting According to renewal mistake, the Backup Data set backed up in advance is obtained, uses mesh described in the data recovery in the Backup Data set Mark the data in data acquisition system.
Optionally, as shown in figure 20, first updating block 1544 includes：
First determination subelement 15441, for determining the second keyword or keyword from the second data set Section.
First inquiry subelement 15442, for using second keyword or critical field in first data Inquired about in set.
First renewal subelement 15443, if for not inquiring second keyword in first data acquisition system Critical field and do not inquired in first data acquisition system and second keyword or critical field phase either The data matched somebody with somebody, by the second data set with the data update that second keyword or critical field match to institute State in target data set.
Second renewal subelement 15444, for by first data acquisition system with first keyword or key The data update that field matches is into the target data set.
Optionally, as shown in figure 21, when the target data set is combined into slide fastener data acquisition system, first updating block 1544 include：
Second determination subelement 15445, for determining the second keyword or keyword from the second data set Section.
Second inquiry subelement 15446, for using second keyword or critical field in first data Inquired about in set.
3rd renewal subelement 15447, if for not inquiring second keyword in first data acquisition system Critical field and do not inquired in first data acquisition system and second keyword or critical field phase either The data matched somebody with somebody, by the second data set with the data update that second keyword or critical field match to institute State in target data set.
3rd determination subelement 15448, for determine in the second data set with first keyword or pass The slide fastener data that key field matches.
Subelement 15449 is changed, is in the first sub- slide fastener number of open chain state in the first slide fastener data for changing According to the closed chain time to generate the time of the second data set, and based on being closed in first data acquisition system with described first The data that key word or critical field match, generate the second sub- slide fastener data of the first slide fastener data, wherein, described the The open chain time of two sub- slide fastener data, the closed chain time was empty or maximum to generate the time of the second data set.
Generate subelement 154410, if for do not inquired in the second data set with first keyword or The data that person's critical field matches, based in first data acquisition system with first keyword or critical field phase The data matched somebody with somebody generate the second slide fastener data, wherein, the open chain time of the second slide fastener data is generation second data set The time of conjunction, closed chain time are empty or maximum.
4th renewal subelement 154411, for by amended slide fastener data and first data acquisition system with first The data update that keyword or critical field match is into the target data set.
Optionally, the generation unit 1542 is additionally operable to obtain the preset time received before first data acquisition system In section, all data for having been stored in updated target data set, or obtain and receive first data acquisition system Afterwards, the data in this target data set to be updated, stored in the updated target data set of backup all Data in data or this target data set to be updated are to generate the second data set.
Alternatively, the generation unit 1542 is additionally operable to obtain last the second number for receiving and generating during the first data acquisition system According to set, by the data in presently described target data set be inserted into it is last receive the first data acquisition system when generate the In two data acquisition systems, to generate this second data set.
Data handling system 1500 provided in an embodiment of the present invention can realize data in the embodiment of the method for Fig. 1 to Fig. 2 Each process that processing system is realized, to avoid repeating, which is not described herein again.
Data handling system provided in an embodiment of the present invention, is receiving the first data acquisition system of external system transmission, is needing When carrying out data update, connection table inquiry mode can be used to carry out data update by inquiring about the means of insertion, to ensure number According to the stability of processing system, without being scanned to all data, the plenty of time is saved, and improve the efficiency of data update.
The embodiment of the present invention is described above in conjunction with attached drawing, but the invention is not limited in above-mentioned specific Embodiment, above-mentioned embodiment is only schematical, rather than restricted, those of ordinary skill in the art Under the enlightenment of the present invention, in the case of present inventive concept and scope of the claimed protection is not departed from, it can also make very much Form, belongs within the protection of the present invention.

Claims (11)

  1. A kind of 1. data processing method, it is characterised in that the described method includes：
    Receive the first data acquisition system of external system transmission；
    The second data set associated with target data set to be updated is generated in a data processing system；
    Empty the data in the target data set；
    Data are carried out to the target data set using the data in first data acquisition system and the second data set Renewal.
  2. 2. the method as described in claim 1, it is characterised in that in the generation in a data processing system and mesh to be updated Before the step of the second data set that mark data acquisition system is associated, the described method includes：
    The first keyword or critical field are determined from first data acquisition system；
    Inquired about using first keyword or critical field in the target data set；
    Either critical field or inquired and described if inquiring first keyword in the target data set The data that one keyword or critical field match, perform the generation in a data processing system and number of targets to be updated The step of according to set associated the second data set.
  3. 3. method as claimed in claim 2, it is characterised in that described to use first data acquisition system and second data The step of data in set carry out data update to the target data set, including：
    The second keyword or critical field are determined from the second data set；
    Inquired about using second keyword or critical field in first data acquisition system；
    If do not inquire second keyword or critical field in first data acquisition system, and in the described first number The data to match according to not inquired in set with second keyword or critical field, by the second data set With the data update that second keyword or critical field match into the target data set；
    By in first data acquisition system with the data update that first keyword or critical field match to the mesh Mark in data acquisition system.
  4. 4. method as claimed in claim 2, it is characterised in that when the target data set is combined into slide fastener data acquisition system, institute State and data are carried out more to the target data set using the data in first data acquisition system and the second data set New step, including：
    The second keyword or critical field are determined from the second data set；
    Inquired about using second keyword or critical field in first data acquisition system；
    If do not inquire second keyword or critical field in first data acquisition system, and in the described first number The data to match according to not inquired in set with second keyword or critical field, by the second data set With the data update that second keyword or critical field match into the target data set；
    Determine the first slide fastener data to match in the second data set with first keyword or critical field, repair The closed chain time for changing the first sub- slide fastener data in open chain state in the first slide fastener data is generation second data The time of set, and based on the data to match in first data acquisition system with first keyword or critical field, The second sub- slide fastener data of the first slide fastener data are generated, wherein, the open chain time of the second sub- slide fastener data is generation The time of the second data set, closed chain time are empty or maximum；
    If the data to match with first keyword or critical field, base are not inquired in the second data set The data to match in first data acquisition system with first keyword or critical field generate the second slide fastener data, Wherein, the open chain time of the second slide fastener data, the closed chain time was empty or pole to generate the time of the second data set Big value；
    By amended first slide fastener data and the second slide fastener data update into the target data set.
  5. 5. a kind of data handling system, it is characterised in that the data handling system includes：
    Data memory module, for storing the internal data of the data handling system, and the data obtained from outside；
    Business logic modules, for management and control service logic；
    Data service module, for providing data service to the external system of data handling system；
    Data processing engine module, for handling data.
  6. 6. data handling system as claimed in claim 5, it is characterised in that the data handling system includes：
    Information exchange module, for receiving operational order input by user, is managed and sets to the data handling system.
  7. 7. data handling system as claimed in claim 5, it is characterised in that the data handling system further includes automatic chemical industry Has module, the automation tools module includes：
    Parameter receiving unit, for receiving the parameter of input；
    Script generation unit, for based on preset rules and the parameter, generating automation tools script.
  8. 8. data handling system as claimed in claim 5, it is characterised in that the data processing engine module includes：
    Receiving unit, for receiving the first data acquisition system of external system transmission；
    Generation unit, for generating second data set associated with target data set to be updated in a data processing system Close；
    Clearing cell, for emptying the data in the target data set；
    First updating block, for using the data in first data acquisition system and the second data set to the target Data acquisition system carries out data update.
  9. 9. data handling system as claimed in claim 8, it is characterised in that the data processing engine module further includes：
    First determination unit, for determining the first keyword or critical field from first data acquisition system；
    Query unit, for being inquired about using first keyword or critical field in the target data set；
    Execution unit, if for inquired in the target data set first keyword either critical field or Inquire the data to match with first keyword or critical field, perform the generation in a data processing system and The step of the second data set that target data set to be updated is associated.
  10. 10. data handling system as claimed in claim 9, it is characterised in that first updating block includes：
    First determination subelement, for determining the second keyword or critical field from the second data set；
    First inquiry subelement, for being carried out using second keyword or critical field in first data acquisition system Inquiry；
    First renewal subelement, if for not inquiring second keyword or keyword in first data acquisition system Section, and do not inquire the data to match with second keyword or critical field in first data acquisition system, By in the second data set with the data update that second keyword or critical field match to the number of targets According in set；
    Second renewal subelement, for will match in first data acquisition system with first keyword or critical field Data update into the target data set.
  11. 11. data handling system as claimed in claim 9, it is characterised in that when the target data set is combined into slide fastener data During set, first updating block includes：
    Second determination subelement, for determining the second keyword or critical field from the second data set；
    Second inquiry subelement, for being carried out using second keyword or critical field in first data acquisition system Inquiry；
    3rd renewal subelement, if for not inquiring second keyword or keyword in first data acquisition system Section, and do not inquire the data to match with second keyword or critical field in first data acquisition system, By in the second data set with the data update that second keyword or critical field match to the number of targets According in set；
    3rd determination subelement, determines what is matched in the second data set with first keyword or critical field First slide fastener data；Subelement is changed, is in the first sub- slide fastener number of open chain state in the first slide fastener data for changing According to the closed chain time to generate the time of the second data set, and based on being closed in first data acquisition system with described first The data that key word or critical field match, generate the second sub- slide fastener data of the first slide fastener data, wherein, described the The open chain time of two sub- slide fastener data, the closed chain time was empty or maximum to generate the time of the second data set；
    Subelement is generated, if for not inquired in the second data set and first keyword or critical field The data to match, based on the data life to match in first data acquisition system with first keyword or critical field Into the second slide fastener data, wherein, the open chain time of the second slide fastener data closes to generate the time of the second data set The chain time is empty or maximum；
    4th renewal subelement, for by amended first slide fastener data and the second slide fastener data update to the target In data acquisition system.
CN201711418696.XA 2017-12-25 2017-12-25 A kind of data processing method and system Active CN108038225B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711418696.XA CN108038225B (en) 2017-12-25 2017-12-25 A kind of data processing method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711418696.XA CN108038225B (en) 2017-12-25 2017-12-25 A kind of data processing method and system

Publications (2)

Publication Number Publication Date
CN108038225A true CN108038225A (en) 2018-05-15
CN108038225B CN108038225B (en) 2019-02-12

Family

ID=62100949

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711418696.XA Active CN108038225B (en) 2017-12-25 2017-12-25 A kind of data processing method and system

Country Status (1)

Country Link
CN (1) CN108038225B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10754895B2 (en) 2018-10-17 2020-08-25 International Business Machines Corporation Efficient metadata destage during safe data commit operation
CN114564477A (en) * 2022-02-23 2022-05-31 中国农业银行股份有限公司 Data storage method and device, electronic equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7707219B1 (en) * 2005-05-31 2010-04-27 Unisys Corporation System and method for transforming a database state
CN102802056A (en) * 2012-09-12 2012-11-28 北京播思软件技术有限公司 Method used for inserting advertisement in digital broadcasting television program
CN103455338A (en) * 2013-09-22 2013-12-18 广州中国科学院软件应用技术研究所 Method and device for acquiring data
US20140025702A1 (en) * 2012-07-23 2014-01-23 Michael Curtiss Filtering Structured Search Queries Based on Privacy Settings
CN104394155A (en) * 2014-11-27 2015-03-04 暨南大学 Multi-user cloud encryption keyboard searching method capable of verifying integrity and completeness
CN105574404A (en) * 2015-12-14 2016-05-11 国家电网公司 Method and device for prompting to change password
CN105677307A (en) * 2014-11-19 2016-06-15 上海烟草集团有限责任公司 Big data processing method and system of mobile terminal
US9697235B2 (en) * 2014-07-16 2017-07-04 Verizon Patent And Licensing Inc. On device image keyword identification and content overlay

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7707219B1 (en) * 2005-05-31 2010-04-27 Unisys Corporation System and method for transforming a database state
US20140025702A1 (en) * 2012-07-23 2014-01-23 Michael Curtiss Filtering Structured Search Queries Based on Privacy Settings
CN102802056A (en) * 2012-09-12 2012-11-28 北京播思软件技术有限公司 Method used for inserting advertisement in digital broadcasting television program
CN103455338A (en) * 2013-09-22 2013-12-18 广州中国科学院软件应用技术研究所 Method and device for acquiring data
US9697235B2 (en) * 2014-07-16 2017-07-04 Verizon Patent And Licensing Inc. On device image keyword identification and content overlay
CN105677307A (en) * 2014-11-19 2016-06-15 上海烟草集团有限责任公司 Big data processing method and system of mobile terminal
CN104394155A (en) * 2014-11-27 2015-03-04 暨南大学 Multi-user cloud encryption keyboard searching method capable of verifying integrity and completeness
CN105574404A (en) * 2015-12-14 2016-05-11 国家电网公司 Method and device for prompting to change password

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10754895B2 (en) 2018-10-17 2020-08-25 International Business Machines Corporation Efficient metadata destage during safe data commit operation
CN114564477A (en) * 2022-02-23 2022-05-31 中国农业银行股份有限公司 Data storage method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN108038225B (en) 2019-02-12

Similar Documents

Publication Publication Date Title
CN103930888B (en) Selected based on the many grain size subpopulation polymerizations updating, storing and response constrains
Santos et al. Real-time data warehouse loading methodology
CN106663038A (en) Feature processing recipes for machine learning
CA3198484A1 (en) Feature processing tradeoff management
CN102930024A (en) A data quality solution architecture based on knowledge
CN102930023A (en) A data quality solution based on knowledge
CN115422173A (en) Data management method and system in financial credit field
CN111639121A (en) Big data platform and method for constructing customer portrait
US20220351002A1 (en) Hierarchical deep neural network forecasting of cashflows with linear algebraic constraints
CN111061679A (en) Method and system for rapid configuration of technological innovation policy based on rete and drools rules
CN108038225B (en) A kind of data processing method and system
CN101013426A (en) Information management system using connection relation
US20230129094A1 (en) Method and system for training a query ranking machine-learning model to provide an answer for a user query
CN114756685A (en) Complaint risk identification method and device for complaint sheet
US11775757B2 (en) Automated machine-learning dataset preparation
Li [Retracted] Research on the Social Security and Elderly Care System under the Background of Big Data
CN111061853B (en) Method for rapidly acquiring FAQ model training corpus
Renfro Economic database systems: further reflections on the state of the art
Kvet et al. Enhancing Analytical Select Statements Using Reference Aliases
Xiao Data Processing Model of Bank Credit Evaluation System.
US11960542B2 (en) Methods and systems for building and/or using a graph data structure
Filev Construction and application of Database architecture for integrated business software purposes.
US20240281473A1 (en) Storing and searching for data in data stores
Puspitaningrum Big Data in Smart Cities: Usage in Kota Jababeka for Customer Satisfaction
EP4193267A1 (en) Storing and searching for data in data stores

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant