CN110309133A - The treating method and apparatus of batch data - Google Patents
The treating method and apparatus of batch data Download PDFInfo
- Publication number
- CN110309133A CN110309133A CN201910440400.7A CN201910440400A CN110309133A CN 110309133 A CN110309133 A CN 110309133A CN 201910440400 A CN201910440400 A CN 201910440400A CN 110309133 A CN110309133 A CN 110309133A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- batch
- data information
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is safety detection technology field, and the present invention provides a kind for the treatment of method and apparatus of batch data, the method includes receiving several batch data files, obtains the attribute information of the batch data file；According to batch data file store matching rule, according to the attribute information of the batch data file by data information memory therein into each tables of data；Attribute byte is obtained according to the data information, the corresponding attribute byte corresponds to the corresponding data information of covering tables of data one by one.This method advantageously ensures that the treatment effeciency of data, the final processing quality for guaranteeing data.
Description
Technical field
The present invention relates to technical field of data processing, specifically, the present invention relates to a kind of processing methods of batch data
And device.
Background technique
With the development of internet big data technology, server data volume meeting explosive growth to be dealt with, meanwhile, tool
Also gradually development is the multistage processing system for dividing functional module to the server system of the functions such as data processing, storage, inquiry.How
Data synchronization processing how is carried out between the functional module of server system becomes particularly important.
In order to accomplish that this point, existing way are the idles in data processing, such as 12 points of morning, in server system
Data between each functional module synchronize processing.But during synchronization usually with synchronization in need number
According to processing is synchronized together, the synchronization process efficiency and the quality of data that are unprofitable between the functional module of server system.
Summary of the invention
To overcome the above technical problem, especially in the prior art by the data of synchronization in need synchronize place together
The problem of reason is unfavorable for synchronization process efficiency and the quality of data between the functional module of server system:
In a first aspect, the present invention provides a kind of processing method of batch data comprising following steps:
Several batch data files are received, the attribute information of the batch data file is obtained；
It, will be therein according to the attribute information of the batch data file according to the matching rule that batch data file stores
Data information memory is into each tables of data；
Attribute byte is obtained according to the data information, the corresponding attribute byte corresponds to the correspondence of covering tables of data one by one
Data information.
It is described in one of the embodiments, that attribute byte, the corresponding attribute byte are obtained according to the data information
The step of corresponding data information of corresponding covering tables of data one by one, comprising:
It is indexed according to data information, extracts corresponding attribute byte according to the data information；
The attribute byte of each data information is compared with the former data information in tables of data；
According to comparison as a result, by the corresponding data information of data information covering tables of data.
The attribute information is belonged to newly to the data information in the batch data file in one of the embodiments,
Increase data information increasing and update data information and is divided.
The processing method of the batch data in one of the embodiments, further include:
It is corresponding according to the data information search when detecting the data information stored in the tables of data is abnormality
Batch data file, and stop the batch data file carry out data processing；
Data correction is carried out according to the error log of the data information.
It is described when detecting the data information stored in the tables of data is abnormality in one of the embodiments,
According to the corresponding batch data file of the data information search, and stop the step that the batch data file carries out data processing
Suddenly, comprising:
It is indexed according to data information, when detecting that it is described that the data information that can not be stored from the tables of data parses to obtain
When data information indexes corresponding related content, then judge the data information for abnormality；
Its corresponding batch data file is obtained according to the data information search, and stops batch data file progress
Data processing.
The step of error log according to the data information is modified in one of the embodiments, include:
The abnormal byte of corresponding data information is obtained according to error log；
The abnormal byte is corresponded into the data information and indexes corresponding byte, the abnormal byte is repaired
Just.
The matching rule stored according to the batch data file in one of the embodiments, will be described in correspondence
Step of the attribute information of batch data file by data information memory therein into each tables of data further include:
Obtain the data information is actually formed the period；
If described be actually formed the period within the setting generation period of corresponding batch data file, by corresponding number
It is believed that breath is first stored into temporary data table；
After other batch data files complete storage, according to the matching rule that the batch data file stores, then will
Data information memory in the temporary data table is into corresponding tables of data.
Second aspect, the present invention also provides a kind of processing units of batch data comprising:
Receiving module obtains the attribute information of the batch data file for receiving several batch data files；
Memory module, the matching rule for being stored according to batch data file, according to the category of the batch data file
Property information is by data information memory therein into each tables of data；
Overlay module, for obtaining attribute byte according to the data information, the corresponding attribute byte one by one cover by correspondence
The corresponding data information of lid tables of data.
The third aspect, the present invention also provides a kind of servers comprising:
One or more processors；
Memory；
One or more computer programs, wherein one or more of computer programs are stored in the memory
And be configured as being executed by one or more of processors, one or more of computer programs are configured to carry out first
The processing method of batch data described in aspect embodiment.
Fourth aspect, the present invention also provides a kind of computer readable storage medium, on the computer readable storage medium
It is stored with computer program, batch data described in first aspect embodiment is realized when which is executed by processor
Processing method.
The treating method and apparatus of a kind of batch data provided by the present invention, according to several received batch datas of institute
The attribute information of file distributes respective data information into different tables of data, and according to the data information of different data table
The characteristics of, obtain the corresponding data information of its attribute byte covering tables of data.
On the basis of the above, the present invention also provides the treating method and apparatus of another batch data, when the detection number
When the data information stored according to table is abnormality, corresponding data processing stopped to the batch data file, and according to
The error log of data information is repaired accordingly.
Technical solution provided by the present invention will be in the need of data processing synchronous between functional module in server system
Want, defined attribute information and to the data information carry out classification form respective batch data file.The batch data text
It is independent from each other between part, server can simultaneously or separately be handled the batch data file.But it if deposits
There is batch data file abnormality occur, and/or is needing to carry out other data processings, amendment, stopping data such as data
When the measures such as processing, the normal data processing of other batch data files is not influenced.In this manner it is ensured that the process that data are synchronous
In, it can prevent the case where abnormal, other data processings is caused to stagnate appearance occur because of individual data, advantageously ensure that data
Treatment effeciency, the final processing quality for guaranteeing data.
The additional aspect of the present invention and advantage will be set forth in part in the description, these will become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the applied environment figure of the embodiment of the present invention；
Fig. 2 is the flow chart of the processing method of the batch data of one embodiment in the present invention；
Fig. 3 is the flow chart of the processing method of the batch data of another embodiment in the present invention；
Fig. 4 is the schematic diagram of the processing unit of the batch data of one embodiment in the present invention；
Fig. 5 is the structural schematic diagram of the server of one embodiment in the present invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and for explaining only the invention, and is not construed as limiting the claims.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singular " one " used herein, " one
It is a ", " described " and "the" may also comprise plural form.It is to be further understood that being arranged used in specification of the invention
Diction " comprising " refer to that there are the feature, integer, step, operation, element and/or component, but it is not excluded that in the presence of or addition
Other one or more features, integer, step, operation, element, component and/or their group.It should be understood that when we claim member
Part is " connected " or when " coupled " to another element, it can be directly connected or coupled to other elements, or there may also be
Intermediary element.In addition, " connection " used herein or " coupling " may include being wirelessly connected or wirelessly coupling.It is used herein to arrange
Diction "and/or" includes one or more associated wholes for listing item or any cell and all combinations.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (including technology art
Language and scientific term), there is meaning identical with the general understanding of those of ordinary skill in fields of the present invention.Should also
Understand, those terms such as defined in the general dictionary, it should be understood that have in the context of the prior art
The consistent meaning of meaning, and unless idealization or meaning too formal otherwise will not be used by specific definitions as here
To explain.
Those skilled in the art of the present technique are appreciated that remote network devices used herein above comprising but be not limited to count
The cloud that calculation machine, network host, single network server, multiple network server collection or multiple servers are constituted.Here, Yun Youji
It is constituted in a large number of computers or network servers of cloud computing (Cloud Computing), wherein cloud computing is distributed computing
One kind, a super virtual computer consisting of a loosely coupled set of computers.In the embodiment of the present invention, distal end
It can be realized and be communicated by any communication modes between the network equipment, terminal device and WNS server, including but not limited to, is based on
The mobile communication of 3GPP, LTE, WIMAX, based on TCP/IP, the computer network communication of udp protocol and based on bluetooth, infrared
The low coverage wireless transmission method of transmission standard.
Refering to what is shown in Fig. 1, Fig. 1 is the applied environment figure of the embodiment of the present invention；In the embodiment, the technology of the present invention side
Case can be based on realizing on server, and as shown in figure 1, host server 110 and inquiry system server 120 can pass through
Internet realizes data interaction.Host server 110 can carry out data processing business according to the service request of user,
And corresponding user data is sent in the database that the inquiry system server 120 is established and is stored.As the master
When machine server 110 receives the inquiry and information update instruction of user, Xiang Suoshu inquiry system server 120 issues relevant
Operational order calls relevant data, forms relevant query information and is sent to user interface.
To solve the above-mentioned problems, the present invention provides a kind of processing methods of batch data.It can refer to Fig. 2, Fig. 2 is one
The processing method flow chart of the batch data of a embodiment, method includes the following steps:
S210, several batch data files are received, obtains the attribute information of the batch data file.
The data volume as handled by server is big, the synchronization of data is carried out between the module of server system, especially
Set period of time (such as idle, concretely 12 points of morning), related data processing usually in the form of batch data into
Row synchronizes.
In the present embodiment, the data synchronized are related to the data information of related fields, and the data information can be
The product information of the customer information or manufacturing of such as service industry's (finance, internet).Each batch data file
In may include tens of thousands of, even tens million of data informations.
In data synchronization processing for the ease of being distinguished to different batch data files.Generating batch data text
When part, it can be classified according to the data information of different attribute, and similar data information is concentrated on into a bulk information text
In part, and corresponding attribute information is arranged to the bulk information file.
By taking the data of financial services are synchronous as an example, the data information is user information, and each batch documents are every
For the batch documents of the user information of same attribute information when data synchronization processing.In the present embodiment, the data
Synchronization process is using the newly-increased of data information and updates the foundation as same Time segments division batch data file, corresponding, described
Attribute information is newly-increased data information and update data information.
According to the above-mentioned Attribute transposition to data information, within the same period, the host server is to the inquiry
In the batch data file of system server synchronous driving, including at least the first batch data text only comprising the information that Adds User
Part and only comprising update user information the second batch data file.
Since the data volume that the batch data file is included is big, it is possible to press the batch data file
After contracting processing, then to the inquiry system server synchronize transmission.Accordingly, the inquiry system server correspondingly connects
It has received the batch data file and has decompressed, corresponding attribute information is obtained from the batch data file, be subsequent data
Differentiation processing is carried out classification and is prepared.
S220, the matching rule stored according to batch data file, will according to the attribute information of the batch data file
Data information memory therein is into each tables of data.
Since the batch data file has been extracted in above-mentioned steps S210, it can be directly from corresponding lot number
According to the corresponding data information of file acquisition.
According to the matching rule that relevant batch data file stores, according to the attribute information of the batch data file,
By data information memory therein into corresponding tables of data.
Corresponding above-described embodiment, the information that Adds User of first batch data file are stored into table 0, and described second
The update user information of batch data file is stored into table 1.And table 0 and table 1 also correspondingly carry corresponding batch data file
Attribute information.
S230, attribute byte is obtained according to the data information, the corresponding attribute byte is corresponding one by one to cover tables of data
Corresponding data information.
In the present embodiment, the data information is stored in the form of byte.The fixation serial number of all data informations
The information of byte stored is to belong to the content of same attribute.In the present embodiment, the content such as may include user's account
Family number, the name of user, date of opening an account etc..Wherein, the attribute byte is basic between the data information for distinguishing
Distinctive points, in the present embodiment, the attribute byte are the byte for storing user account information.
After obtaining attribute byte in the data information of each tables of data, data of the attribute byte to tables of data are believed
It ceases corresponding attribute byte to be traversed, newest data information is covered into corresponding data information in tables of data.
The present invention provides a kind of processing method of batch data, by obtaining the attribute information of the batch data file simultaneously
It stores into corresponding tables of data, and to attribute byte is obtained in the data information of the batch data file, according to the spy
It levies byte and the data information is covered into tables of data original data information column.The present invention according to the attribute information to batch
Amount data file carries out separating the storage of different data table, and carries out the processing of related data information, so that between different data table
It is mutually indepedent in data synchronization processing, not by the wrong or modified interference of other tables of data, be conducive to promote synchronization
Treatment effeciency and the quality of data.
For step S230, can further comprise:
A1, it is indexed according to data information, extracts corresponding attribute byte according to the data information.
For same type of batch data file handled by corresponding same server apparatus, the data information is deposited
Storage form be it is identical, which is according to certain treaty rule, in the present embodiment, with data information
Index is to provide the storage form of the data information.Specific index defines such as: the 1- of the data information
10 bytes are to store corresponding user account number, and 11-18 stores the name of corresponding user, and 19-25 corresponding users' opens an account
Date etc..In the present embodiment, the inquiry system server based on data information index extracts energy according to the data information
By the characteristic information of data information, it is handled differently so as to subsequent.
A2, the attribute byte of each data information is compared with the former data information in tables of data.
The attribute byte of the data information is the most basic information for distinguishing between data information, in data processing
In, no matter the data information lives through how many times change, and in the present embodiment, the attribute byte will not change.
In the present embodiment, due to containing newly-increased data information or more in the data information in the batch data file
The attribute information of new data information.
The attribute byte of each data information is traversed and compared to data information original in tables of data, such as
Data information described in fruit is newly-increased data information, then the data information with same characteristic features field cannot be obtained in tables of data；
If the data information is to update data information, the available data letter with same characteristic features field in tables of data
Breath.
A3, according to comparison as a result, by the data information covering tables of data corresponding data information.
For above-mentioned steps A3 compare as a result, data for that cannot obtain that there is same characteristic features field in tables of data
Information, relevant data information is increased newly according to corresponding byte align form into the tables of data；It corresponds in tables of data
In the available data information with same characteristic features field, the life of time posterior data information covering tables of data will be generated
At time preceding corresponding data information.
The judgement for generating the time, can be determined according to the time-labeling of data information.
It is the flow chart of the processing method of the batch data of another embodiment in the present invention with reference to Fig. 3, Fig. 3.At this
The processing method for inventing a kind of batch data provided can comprise the further steps of: on the basis of above-mentioned
S240, when detecting the data information stored in the tables of data is abnormality, looked into according to the data information
Corresponding batch data file is looked for, and stops the batch data file and carries out data processing.
In this step, when the data information for detecting the tables of data is deposited when abnormal, due to the batch data text
Part carries out the data information of same attribute information to be packaged synchronization process, so can find correspondence according to the data information
Batch data file at this moment in order to guarantee the accuracy of data synchronization processing, stop to the corresponding batch of the data information
The synchrodata of the data information of data file is handled, including the synchronization process to newly-increased data information or update data information.
S250, it is modified according to the error log of the data information.
Obtain the data information for abnormality occur according to error log, and according to the error code of the error log,
The property content for occurring mistake in the data information is obtained, and it is modified.
Until completing the error correction of corresponding batch data file, can restore to the data in the batch data file
The synchronization process of information.
In the present embodiment, due to being independent from each other between batch data file, when there are individual batch data files
When processing occurs abnormal, the processing progress of other batch documents processing is not influenced, in this way, can further avoid because of part
Data exception situation influences the processing progress of other data, so that the treatment effeciency of batch data is further promoted.
For step S240, can further comprise:
B11, it is indexed according to data information, when detecting that the data information that can not be stored from the tables of data parses to obtain
When the data information indexes corresponding related content, then judge the data information for abnormality.
It is indexed according to the data information, the byte of the total different serial numbers of the data information represents the interior of different attributes
Hold, according to the property content.The property content of corresponding byte, believes with according to the data obtained by parsing to data information
The property content that breath index obtains is different, then it is abnormal to determine that the data information occurs.
Such as in account information, the information of client is made of several bytes, wherein the 19-25 byte is to open an account day
Phase if corresponding byte is 88888888 in certain file, rather than is the format yyyymmdd on preset date, then inquiry system
Server can not be parsed from the field and be opened an account accordingly the date, and at this moment, then corresponding data information is got the bid in state table
Knowing is abnormality, and generates the error log about the error message.
B12, its corresponding batch data file is obtained according to the data information search, and stops the batch data file
Carry out data processing.
Due to the data information come from corresponding batch data file, can by the coding of the data information or
Person is that position marks to obtain corresponding batch data file,
By the attribute information of the data information, its corresponding batch data file is obtained, so that server stops to this
Batch data file processing.
The scheme of above-described embodiment can be indexed according to the data information, quickly obtain the data letter for abnormality occur
Breath, and it is simultaneously stopped the data processing of corresponding batch data file, to reduce the quantity to report an error during data synchronization processing,
Improve the quality of data of synchronization process.
For above content, the step S250 may also include that
B21, the abnormal byte that corresponding data information is obtained according to error log；
B22, the abnormal byte is corresponded into the corresponding byte of data information index, the abnormal byte is carried out
Amendment.
In above-mentioned steps B21-B22, error code is obtained according in the error log, is obtained from the error code
It obtains corresponding data information and abnormal property content occurs, then obtain corresponding abnormal byte from the property content, or directly rush
Corresponding error byte is obtained in error code.
The corresponding byte serial number of the exception byte is corresponded into the byte sequence in data information index, obtains institute
State the correct property content of byte serial number of abnormal byte.According to the correct property content, again to corresponding data information
Corresponding content is obtained in newest information, and the abnormal byte of corresponding data information is corrected accordingly.
Above-mentioned combination data information according to the abnormal byte volume that error log obtains index, which compares, to be modified,
Can quickly error message be positioned and is modified, improved the quality of data of synchronization process, also further increase
The synchronous efficiency of data.
For step S220, can further comprise:
C1, obtain the batch data file be actually formed the period；
If C2, it is described be actually formed the period corresponding batch data file setting generate the period in, will correspond to
Batch data file first store into temporary data table；
C3, after other batch data files complete storage after, according to the batch data file store matching rule, then
By the data information memory in the temporary data table into corresponding tables of data.
In above-mentioned steps C1-C3, when the inquiry system server receives the batch data file, according at that time
Between label obtain to obtain the batch data file in the formation time of host server.If the data information is actually formed
Time just generates the period in the setting of the batch data file, i.e., right during the described batch data file is formed
The data information answered just just is formed, but cannot be inserted into the batch data file for having same alike result information at this time.In order to
The integrality and timely synchronization process for guaranteeing data, by corresponding data information first with data packet or another batch data text
Part is synchronized to the inquiry system server, and is stored with temporary data table.It stays in setting and generates generation in the period
After batch data file decompression and with the corresponding data information of data information tables of data one by one, after completing corresponding data storage,
Further according to the matching rule of batch data file storage, to the data information in the temporary data table according to attribute information,
It correspondingly keeps into corresponding tables of data.
Above-mentioned example is continued to use to be specifically described:
The data information obtained during above-mentioned table 0 and the corresponding batch data file of table 1 generate is with data packet
Form is synchronized in the inquiry system server, and is temporarily stored in the form of table 2.But table 0 and table 1 complete newly-increased use
After the storage of family information and update user information, according to the corresponding attribute information of corresponding data information in table 2, i.e., according to newly-increased number
It is incorporated into the table 1 according to more new data.
In order to shorten the corresponding time of subsequent external query service, each tables of data is incorporated into the underlying table externally serviced
In.In the present embodiment, the data information of table 0 and table 1 is incorporated into underlying table.
For in the processing method of the batch data of above statement, the batch data file is according to generation time of setting
Section and section synchronization time, perform corresponding processing newly-increased and update recently.In order in time to generated data information into
Row processing, the data information generated outside the generation period of setting is first synchronous in the form of data packet and stores to the inquiry
In system server.When to the generation period of the setting of the batch data file, to the newest update of the data information
Information is synchronous in the form of batch data file again and stores into the inquiry system server.
Based on inventive concept identical with the processing method of above-mentioned batch data, the embodiment of the invention also provides one kind batch
The processing unit of data is measured, as shown in Figure 4, comprising:
Receiving module 410 obtains the attribute letter of the batch data file for receiving several batch data files
Breath；
Memory module 420, the matching rule for being stored according to batch data file, according to the batch data file
Attribute information is by data information memory therein into each tables of data；
Overlay module 430, for obtaining attribute byte according to the data information, the corresponding attribute byte corresponds to one by one
Cover the corresponding data information of tables of data.
Referring to FIG. 5, Fig. 5 is the schematic diagram of internal structure of server in one embodiment.As shown in figure 5, the server
Including processor 510, storage medium 520, memory 530 and the network interface 540 connected by system bus.Wherein, the clothes
The storage medium 520 of business device is stored with operating system, database and computer-readable instruction, and control letter can be stored in database
Cease sequence may make processor 510 to realize a kind of place of batch data when the computer-readable instruction is executed by processor 510
Reason method, processor 510 be able to achieve receiving module 410 in the processing unit of one of embodiment illustrated in fig. 4 batch data,
The function of memory module 420 and overlay module 430.The processor 510 of the server is for providing calculating and control ability, support
The operation of entire server.It can be stored with computer-readable instruction in the memory 530 of the server, the computer-readable instruction
When being executed by processor 510, processor 510 may make to execute a kind of processing method of batch data.The network of the server connects
Mouth 540 is used for and terminal connection communication.It will be understood by those skilled in the art that structure shown in Fig. 5, only and the application
The block diagram of the relevant part-structure of scheme, does not constitute the restriction for the server being applied thereon to application scheme, specifically
Server may include perhaps combining certain components or with different portions than more or fewer components as shown in the figure
Part arrangement.
In one embodiment, the invention also provides a kind of storage medium for being stored with computer-readable instruction, the meters
When calculation machine readable instruction is executed by one or more processors, so that one or more processors execute following steps: if receiving
Dry batch data file, obtains the attribute information of the batch data file；According to the matching rule of batch data file storage
Then, according to the attribute information of the batch data file by data information memory therein into each tables of data；According to the number
According to acquisition of information attribute byte, the corresponding attribute byte corresponds to the corresponding data information of covering tables of data one by one.
Based on the above embodiments it is found that the maximum beneficial effect of the present invention is:
The treating method and apparatus of a kind of batch data provided by the present invention, according to several received batch datas of institute
The attribute information of file distributes respective data information into different tables of data, and according to the data information of different data table
The characteristics of, obtain the corresponding data information of its attribute byte covering tables of data.
On the basis of the above, the present invention also provides the treating method and apparatus of another batch data, when the detection number
When the data information stored according to table is abnormality, corresponding data processing stopped to the batch data file, and according to
The error log of data information is repaired accordingly.
On the basis of the above, the present invention also provides the treating method and apparatus of another batch data, for that cannot set
The data information of batch data file is generated in section of fixing time and covers the data information in corresponding data table, it will be with ephemeral data
The form of table is first stored, after other batch data files complete storage, what is stored according to the batch data file
Matching rule, by data information memory into corresponding tables of data.In this manner it is ensured that the integrality of data and obtaining same in time
Step processing.
Technical solution provided by the present invention will be in the need of data processing synchronous between functional module in server system
Want, defined attribute information and to the data information carry out classification form respective batch data file.The batch data text
It is independent from each other between part, server can simultaneously or separately be handled the batch data file.But it if deposits
There is batch data file abnormality occur, and/or is needing to carry out other data processings, amendment, stopping data such as data
When the measures such as processing, the normal data processing of other batch data files is not influenced.In this manner it is ensured that the process that data are synchronous
In, it can prevent the case where abnormal, other data processings is caused to stagnate appearance occur because of individual data, advantageously ensure that data
Treatment effeciency, the final processing quality for guaranteeing data.
To sum up, the present invention is by the treating method and apparatus of batch data, by defined attribute information to batch data into
Row classifies and forms corresponding batch data file, covers data according to the attribute byte of the data information in batch data file
The technical solution of the data information of table, solve in the prior art by synchronization in need data together synchronize processing not
The problem of conducive to synchronization process efficiency and the quality of data between the functional module of server system.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, which can be stored in a computer-readable storage and be situated between
In matter, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, storage medium above-mentioned can be
Storage mediums or the random access memories such as magnetic disk, CD, read-only memory (Read-Only Memory, ROM)
(Random Access Memory, RAM) etc..
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality
It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.Therefore, the scope of protection of the patent of the invention shall be subject to the appended claims.
Claims (10)
1. a kind of processing method of batch data, which comprises the following steps:
Several batch data files are received, the attribute information of the batch data file is obtained；
According to the matching rule that batch data file stores, according to the attribute information of the batch data file by data therein
Information is stored into each tables of data；
Attribute byte is obtained according to the data information, the corresponding attribute byte corresponds to the corresponding data of covering tables of data one by one
Information.
2. the method according to claim 1, wherein
Described to obtain attribute byte according to the data information, the corresponding attribute byte corresponds to the correspondence of covering tables of data one by one
The step of data information, comprising:
It is indexed according to data information, extracts corresponding attribute byte according to the data information；
The attribute byte of each data information is compared with the former data information in tables of data；
According to comparison as a result, by the corresponding data information of data information covering tables of data.
3. according to the method described in claim 2, it is characterized in that,
The attribute information is to belong to newly-increased data information to the data information in the batch data file to increase and more new data
Information is divided.
4. the method according to claim 1, wherein further include:
When detecting the data information stored in the tables of data is abnormality, according to the data information search corresponding batch
Data file is measured, and stops the batch data file and carries out data processing；
Data correction is carried out according to the error log of the data information.
5. according to the method described in claim 4, it is characterized in that,
It is described when detecting the data information stored in the tables of data is abnormality, it is corresponding according to the data information search
Batch data file, and stop the batch data file carry out data processing the step of, comprising:
It is indexed according to data information, when detecting that the data information that can not be stored from the tables of data parses to obtain the data
When the corresponding related content of information index, then judge the data information for abnormality；
Its corresponding batch data file is obtained according to the data information search, and stops the batch data file and carries out data
Processing.
6. according to method described in claim 4 or 5 one of them, which is characterized in that
The step of error log according to the data information is modified include:
The abnormal byte of corresponding data information is obtained according to error log；
The abnormal byte is corresponded into the data information and indexes corresponding byte, the abnormal byte is modified.
7. the method according to claim 1, wherein
The matching rule stored according to the batch data file, will by the attribute information of the correspondence batch data file
Step of the data information memory therein into each tables of data further include:
Obtain the data information is actually formed the period；
If described be actually formed the period within the setting generation period of corresponding batch data file, corresponding data are believed
Breath is first stored into temporary data table；
After other batch data files complete storage, according to the matching rule that the batch data file stores, then will be described
Data information memory in temporary data table is into corresponding tables of data.
8. a kind of processing unit of batch data characterized by comprising
Receiving module obtains the attribute information of the batch data file for receiving several batch data files；
Memory module, the matching rule for being stored according to batch data file are believed according to the attribute of the batch data file
Breath is by data information memory therein into each tables of data；
Overlay module, for obtaining attribute byte according to the data information, the corresponding attribute byte is corresponding one by one to cover number
According to the corresponding data information of table.
9. a kind of server characterized by comprising
One or more processors；
Memory；
One or more computer programs, wherein one or more of computer programs are stored in the memory and quilt
It is configured to be executed by one or more of processors, one or more of computer programs are configured to carry out according to right
It is required that the processing method of 1 to 7 described in any item batch datas.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes the processing side of the described in any item batch datas of claim 1 to 7 when the computer program is executed by processor
Method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910440400.7A CN110309133B (en) | 2019-05-24 | 2019-05-24 | Batch data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910440400.7A CN110309133B (en) | 2019-05-24 | 2019-05-24 | Batch data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309133A true CN110309133A (en) | 2019-10-08 |
| CN110309133B CN110309133B (en) | 2023-08-22 |
Family
ID=68074928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910440400.7A Active CN110309133B (en) | 2019-05-24 | 2019-05-24 | Batch data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309133B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030172368A1 (en) * | 2001-12-26 | 2003-09-11 | Elizabeth Alumbaugh | System and method for autonomously generating heterogeneous data source interoperability bridges based on semantic modeling derived from self adapting ontology |
| US20100036796A1 (en) * | 2008-08-08 | 2010-02-11 | Takeshi Kajikawa | Image forming apparatus, log storing method, and computer program product |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
| CN106776131A (en) * | 2016-11-30 | 2017-05-31 | 杭州华为数字技术有限公司 | A kind of data back up method and server |
| US20190050441A1 (en) * | 2017-08-09 | 2019-02-14 | Vmware, Inc. | Event based analytics database synchronization |
-
2019
- 2019-05-24 CN CN201910440400.7A patent/CN110309133B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030172368A1 (en) * | 2001-12-26 | 2003-09-11 | Elizabeth Alumbaugh | System and method for autonomously generating heterogeneous data source interoperability bridges based on semantic modeling derived from self adapting ontology |
| US20100036796A1 (en) * | 2008-08-08 | 2010-02-11 | Takeshi Kajikawa | Image forming apparatus, log storing method, and computer program product |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
| CN106776131A (en) * | 2016-11-30 | 2017-05-31 | 杭州华为数字技术有限公司 | A kind of data back up method and server |
| US20190050441A1 (en) * | 2017-08-09 | 2019-02-14 | Vmware, Inc. | Event based analytics database synchronization |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309133B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11194828B2 (en) | Method and system for implementing a log parser in a log analytics system | |
| US8161495B2 (en) | Parameters passing of data structures where API and corresponding stored procedure are different versions/releases | |
| US6665674B1 (en) | Framework for open directory operation extensibility | |
| US20040230667A1 (en) | Loosely coupled intellectual capital processing engine | |
| US20220092062A1 (en) | Method and system for implementing a log parser in a log analytics system | |
| CN112800095B (en) | Data processing method, device, equipment and storage medium | |
| WO2016161381A1 (en) | Method and system for implementing a log parser in a log analytics system | |
| CN111857880B (en) | Dialogue configuration item information management method, device, equipment and storage medium | |
| CN114500690B (en) | Interface data processing method and device, electronic equipment and storage medium | |
| CN115858796A (en) | Fault knowledge graph construction method and device | |
| CN112685433A (en) | Metadata updating method and device, electronic equipment and computer-readable storage medium | |
| CN110430103B (en) | Message monitoring method | |
| US20040230442A1 (en) | Access control over dynamic intellectual capital content | |
| CN110222028A (en) | A kind of data managing method, device, equipment and storage medium | |
| CN110334147A (en) | A kind of method of data synchronization and device | |
| CN116308344A (en) | Transaction data authentication consensus method and system based on blockchain | |
| CN103139298B (en) | Method for transmitting network data and device | |
| CN110309133A (en) | The treating method and apparatus of batch data | |
| US20040230567A1 (en) | Integrating intellectual capital into an intellectual capital management system | |
| CN114610385B (en) | Running environment adaptation system and method | |
| CN114218256B (en) | Access statement processing method, device, equipment and storage medium | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| CN108509293A (en) | A kind of user journal timestamp fault-tolerance approach and system | |
| CN114116268A (en) | Method and device for checking Flink SQL statement, computer equipment and storage medium | |
| US20040230618A1 (en) | Business intelligence using intellectual capital |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |