CN110209891A - A kind of zipper table generating method, device, equipment and medium - Google Patents
A kind of zipper table generating method, device, equipment and medium Download PDFInfo
- Publication number
- CN110209891A CN110209891A CN201910532415.6A CN201910532415A CN110209891A CN 110209891 A CN110209891 A CN 110209891A CN 201910532415 A CN201910532415 A CN 201910532415A CN 110209891 A CN110209891 A CN 110209891A
- Authority
- CN
- China
- Prior art keywords
- data
- current
- historgraphic
- character string
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
Abstract
The invention discloses a kind of zipper table generating method, device, equipment and media.The step of this method includes: historgraphic data recording corresponding history feature value of the target matrix under the historical juncture read in data warehouse；Obtain target matrix current data record corresponding with historgraphic data recording under current time；By current data record in the data content of each field be spliced into the second character string, and hash algorithm operation is carried out to the second character string and generates current characteristic value；Judge history feature value and current characteristic value with the presence or absence of difference；If it is, generating the zipper table that record has historgraphic data recording.The whole of the calculation resources of cluster device is occupied in this method relative reduction zipper table generating process, and then ensures the overall operation stability of big data platform and reduces O&M pressure.In addition, the present invention also provides a kind of zipper table creating device, equipment and medium, beneficial effect are same as above.
Description
Technical field
The present invention relates to database fields, more particularly to a kind of zipper table generating method, device, equipment and medium.
Background technique
With the arrival of big data era, each large enterprises often require to build the big data platform of itself, and based on big
Data warehouse on data platform is one of application important under big data platform.Data warehouse is for all ranks of enterprise
Decision-making process the strategy set that all types data are supported is provided, which is subject-oriented, integrated, time-varying
, it is non-volatile.
It is recorded due to being often stored with magnanimity data in actual scene, in the tables of data of data warehouse, in tables of data
The content of data record will also tend to generate variation over time, and in the application process to tables of data, user
The data record in tables of data is inscribed when it is generally necessary to a certain before tracing, therefore is just needed to tables of data under different historical periods
In data record stored.
Biggish waste is caused to memory space in order to avoid the data record in full dose data table memory, currently often
Changed data record in tables of data is only saved under historical period by way of zipper table.The purpose of zipper table is to save
Data record is before content change in tables of data, until the information of all changes of current state, zipper table is usually reconciliation
The history of family information changes the result that content is retained.When being currently generated zipper table, the moment before generally requiring to get
The tables of data of tables of data and current time, and corresponding data between the tables of data at moment and the tables of data at current time before comparing
Data in record in each respective field, and then sent when certain data of tables of data under moment before and current time records
When content change, by before when the data record inscribed save to zipper table.Due to currently carrying out adjacent moment data
It is field progress content comparison one by one when the comparison that corresponding data records between table, and in actual scene, in tables of data
The field that data record is included is often more, therefore currently in the ratio for carrying out corresponding data record between adjacent moment tables of data
Clock synchronization needs to occupy a large amount of calculation resources of cluster device, it is difficult to ensure the overall operation stability of big data platform is easily made
At biggish O&M pressure.
It can be seen that provide a kind of zipper table generating method, in relative reduction zipper table generating process to cluster device
The whole of calculation resources occupy, and then ensure the overall operation stability of big data platform and reduce O&M pressure, be ability
Field technique personnel's technical issues that need to address.
Summary of the invention
The object of the present invention is to provide a kind of zipper table generating method, device, equipment and media, with relative reduction zipper table
The entirety of the calculation resources of cluster device is occupied in generating process, and then ensures the overall operation stability of big data platform simultaneously
Reduce O&M pressure.
In order to solve the above technical problems, the present invention provides a kind of zipper table generating method, comprising:
Read historgraphic data recording corresponding history feature value of the target matrix in data warehouse under the historical juncture；
Wherein, history feature value be by the way that the data content of field each in historgraphic data recording is spliced into the first character string in advance, and
Hash algorithm operation generation is carried out to the first character string；
Obtain target matrix current data record corresponding with historgraphic data recording under current time；
By current data record in the data content of each field be spliced into the second character string, and the second character string is carried out
Hash algorithm operation generates current characteristic value；
Judge history feature value and current characteristic value with the presence or absence of difference；
If it is, generating the zipper table that record has historgraphic data recording.
Preferably, historgraphic data recording corresponding history of the target matrix under the historical juncture in data warehouse is read
Characteristic value, comprising:
The corresponding history feature value of historgraphic data recording is read in preset middle table；Wherein, middle table is based on target
The field of tables of data has additional the feature value fields of log history characteristic value.
It preferably, include the data record effective date of storing data record effective date in the field of target matrix
Field, zipper table include data record effective date field and data record Expiration Date field.
Preferably, target matrix current data record corresponding with historgraphic data recording, packet under current time are obtained
It includes:
Obtain target matrix current data identical with the major key field content of historgraphic data recording under current time
Record.
Preferably, hash algorithm includes MD5 hash algorithm.
Preferably, data warehouse includes Hive data warehouse.
Preferably, the data type of the first character string and the second character string is character string type.
In addition, the present invention also provides a kind of zipper table creating devices, comprising:
History feature obtains module, for reading historical data of the target matrix in data warehouse under the historical juncture
Record corresponding history feature value；Wherein, history feature value is by advance will be in the data of field each in historgraphic data recording
Appearance is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；
Current data obtains module, and for obtaining, target matrix is corresponding with historgraphic data recording under current time to work as
Preceding data record；
Current signature computing module, the data content for each field in recording current data are spliced into the second character
String, and hash algorithm operation is carried out to the second character string and generates current characteristic value；
Diversity judgement module, for judging history feature value and current characteristic value with the presence or absence of difference, if it is, calling
Zipper table generation module；
Zipper table generation module, the zipper table for having historgraphic data recording for generating record.
In addition, the present invention also provides a kind of zipper table generating devices, comprising:
Memory, for storing computer program；
Processor is realized when for executing computer program such as the step of above-mentioned zipper table generating method.
In addition, being stored with meter on computer readable storage medium the present invention also provides a kind of computer readable storage medium
Calculation machine program is realized when computer program is executed by processor such as the step of above-mentioned zipper table generating method.
Zipper table generating method provided by the present invention, target matrix in data warehouse first under the reading historical juncture
The corresponding history feature value of historical data, wherein by advance by each field in historgraphic data recording when history feature value
Data content is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；And then obtain number of targets
According to the historgraphic data recording corresponding current data record of the table under current time, by current data record in each field
Data content is spliced into the second character string, and carries out identical hash algorithm operation to the second character string and generate current characteristic value,
History feature value and current characteristic value are finally judged with the presence or absence of difference, if it is, generating record has the historgraphic data recording
Zipper table.This method is that the history of hash algorithm operation generation is carried out by comparing the whole historgraphic data recording of target matrix
Whether deposited between characteristic value and the current characteristic value of the whole current data record progress hash algorithm operation generation of target matrix
In the mode of difference, the comparison that whether there is difference between the corresponding data record to target matrix under different moments is realized,
Since history feature value and current characteristic value can represent the data content of the record of the overall data in target matrix, nothing
The comparison that field one by one carries out content need to be carried out between data record, the consistent of history feature value and current characteristic value can be passed through
Property learns whether data record changes, to the whole of the calculation resources of cluster device in relative reduction zipper table generating process
Body occupies, and then ensures the overall operation stability of big data platform and reduce O&M pressure.In addition, the present invention also provides
A kind of zipper table creating device, equipment and medium, beneficial effect are same as above.
Detailed description of the invention
In order to illustrate the embodiments of the present invention more clearly, attached drawing needed in the embodiment will be done simply below
It introduces, it should be apparent that, drawings in the following description are only some embodiments of the invention, for ordinary skill people
For member, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart of zipper table generating method disclosed by the invention；
Fig. 2 is a kind of structural schematic diagram of zipper table creating device disclosed in the present application.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, rather than whole embodiments.Based on this
Embodiment in invention, those of ordinary skill in the art are without making creative work, obtained every other
Embodiment belongs to the scope of the present invention.
Biggish waste is caused to memory space in order to avoid the data record in full dose data table memory, currently often
Changed data record in tables of data is only saved under historical period by way of zipper table.The purpose of zipper table is to save
Data record is before content change in tables of data, until the information of all changes of current state, zipper table is usually reconciliation
The history of family information changes the result that content is retained.When being currently generated zipper table, the moment before generally requiring to get
The tables of data of tables of data and current time, and corresponding data between the tables of data at moment and the tables of data at current time before comparing
Data in record in each respective field, and then sent when certain data of tables of data under moment before and current time records
When content change, by before when the data record inscribed save to zipper table.Due to currently carrying out adjacent moment data
It is field progress content comparison one by one when the comparison that corresponding data records between table, and in actual scene, in tables of data
The field that data record is included is often more, therefore currently in the ratio for carrying out corresponding data record between adjacent moment tables of data
Clock synchronization needs to occupy a large amount of calculation resources of cluster device, it is difficult to ensure the overall operation stability of big data platform is easily made
At biggish O&M pressure.
For this purpose, core of the invention is to provide a kind of zipper table generating method, in relative reduction zipper table generating process
The whole of calculation resources of cluster device is occupied, and then ensures the overall operation stability of big data platform and reduces O&M pressure
Power.
In order to enable those skilled in the art to better understand the solution of the present invention, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.
Shown in Figure 1, the embodiment of the invention discloses a kind of zipper table generating methods, comprising:
Step S10: read that historgraphic data recording of the target matrix under the historical juncture in data warehouse is corresponding to be gone through
History characteristic value.
Wherein, history feature value is by the way that the data content of field each in historgraphic data recording is spliced into the first word in advance
Symbol string, and hash algorithm operation generation is carried out to the first character string.
It should be noted that this step is to read go through corresponding with the historgraphic data recording of target matrix under the historical juncture
History characteristic value, history feature value are spliced into the first character string by the data content of resource each in historgraphic data recording in advance, and right
First character string carries out hash algorithm operation generation.
Hash algorithm in this method is essentially hash function, is that the input of arbitrary data length is passed through hashing algorithm
It is transformed into the output of regular length, which is exactly hashed value.This conversion is a kind of compression mapping, it is, hashed value
Space is generally much less than the space inputted, and different inputs may hash to identical output, it is impossible to from hashed value
Determine unique input value, hash function is briefly exactly a kind of message compression by random length to a certain regular length
Eap-message digest function.The first character string in this step is the hashed value of complete history data record, can characterize one
Complete history data record, therefore when the data content of field any in historgraphic data recording changes, the first character string
Content can change therewith.
In addition, it is necessary to illustrate, historgraphic data recording and current data record in this method refer both in itself
Be data record in tables of data, so-called data record is exactly the data in tables of data, and each data record is equal
It include the preset field of tables of data.
Step S11: target matrix current data record corresponding with historgraphic data recording under current time is obtained.
It is emphasized that since whether this method is to be changed and determined according to the data record in target matrix
Whether the target matrix corresponding zipper table is generated, therefore the current data record being compared should be answered with historgraphic data recording
Obtained for corresponding same data record, and then in this step target matrix under current time with historgraphic data recording
Corresponding current data record, for being compared in the next steps with historgraphic data recording.
Step S12: by current data record in the data content of each field be spliced into the second character string, and to the second character
String carries out hash algorithm operation and generates current characteristic value.
Due to this method focus on by historgraphic data recording and current data record between characteristic value comparison with
Substitution historgraphic data recording and current data record the comparison between each corresponding field, therefore during this step records current data
The data content of each field is spliced into the second character string, and executes hash algorithm identical with the first character string to the second character string
Operation generates current characteristic value.
Step S13: history feature value and current characteristic value are judged with the presence or absence of difference, if so, thening follow the steps S14.
Step S14: the zipper table that record has historgraphic data recording is generated.
It should be noted that since history feature value can characterize respectively historgraphic data recording with current characteristic value and work as
Preceding data record, therefore historgraphic data recording can be learned with the presence or absence of difference by comparing history feature value and current characteristic value
And whether current data record occurs the variation of data content, and then when history feature value has differences with current characteristic value
When, then the zipper table that record has historgraphic data recording is generated, changed data record is recorded with this.
Zipper table generating method provided by the present invention, target matrix in data warehouse first under the reading historical juncture
The corresponding history feature value of historical data, wherein by advance by each field in historgraphic data recording when history feature value
Data content is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；And then obtain number of targets
According to the historgraphic data recording corresponding current data record of the table under current time, by current data record in each field
Data content is spliced into the second character string, and carries out identical hash algorithm operation to the second character string and generate current characteristic value,
History feature value and current characteristic value are finally judged with the presence or absence of difference, if it is, generating record has the historgraphic data recording
Zipper table.This method is that the history of hash algorithm operation generation is carried out by comparing the whole historgraphic data recording of target matrix
Whether deposited between characteristic value and the current characteristic value of the whole current data record progress hash algorithm operation generation of target matrix
In the mode of difference, the comparison that whether there is difference between the corresponding data record to target matrix under different moments is realized,
Since history feature value and current characteristic value can represent the data content of the record of the overall data in target matrix, nothing
The comparison that field one by one carries out content need to be carried out between data record, the consistent of history feature value and current characteristic value can be passed through
Property learns whether data record changes, to the whole of the calculation resources of cluster device in relative reduction zipper table generating process
Body occupies, and then ensures the overall operation stability of big data platform and reduce O&M pressure.
On the basis of the above embodiments, the present invention also provides a series of preferred embodiments.
As a preferred embodiment, reading history number of the target matrix under the historical juncture in data warehouse
According to recording corresponding history feature value, comprising:
The corresponding history feature value of historgraphic data recording is read in preset middle table.
Wherein, middle table has additional the feature value fields of log history characteristic value based on the field of target matrix.
It should be noted that present embodiment focuses on introducing middle table, middle table includes target matrix
In field also have additional the feature value fields of log history characteristic value and on the basis of the field of target matrix, it is special
Value indicative field is used for the corresponding history feature value of historgraphic data recording under the pre-recorded historical juncture.Due in present embodiment
Between table on the basis of record has target matrix historgraphic data recording under the historical juncture, also record have the historgraphic data recording
History feature value, and generate by operation and the history feature value that is recorded in middle table can be adjusted repeatedly and efficiently
With, therefore being capable of the opposite whole efficiency for improving zipper table generating process.
In addition, as a preferred embodiment, including that storing data record comes into force in the field of target matrix
The data record effective date field on date, zipper table include data record effective date field and data record expiry date
Phase field.
It should be noted that including the storing data record effective date in the target data literary name section of present embodiment
Data record effective date field, zipper table include data record effective date field and data record Expiration Date word
Section, it is therefore an objective to the date come into force definitely can be recorded by target matrix storing data, target matrix is worked as with this
In data record when changing and generating the new data record effective date, can be according to coming into force historgraphic data recording
Content of the date as the data record effective date field in zipper table, and using the effective date of current data record as drawing
The content of data record Expiration Date field in chained list, so it is more detailed by zipper table store historical data record once
Date section through coming into force, convenient for according to zipper table more efficiently to the data record of the tables of data in certain time period before
It is traced.
In addition, as a preferred embodiment, obtain target matrix under current time with historgraphic data recording
Corresponding current data record, comprising:
Obtain target matrix current data identical with the major key field content of historgraphic data recording under current time
Record.
It should be noted that due to consideration that in tables of data in the unique mark data table of major key field energy of data record
Every a line, by major key field can pressure data table entity integrity, therefore the major key field of tables of data can be passed through
Data content characterizes the data record locating for it, and due in practical applications, the data content of major key field in tables of data
It is often constant, therefore present embodiment is using the data content of the major key field of historgraphic data recording as acquisition current data
The foundation of record, i.e. acquisition target matrix are identical with the major key field content of historgraphic data recording current under current time
Data record, can be accurate corresponding between historgraphic data recording and current data record with respect to ensure, and then guarantees zipper table
Content reliability.
In addition, as a preferred embodiment, hash algorithm includes MD5 hash algorithm.
It should be noted that MD5 hash algorithm is a kind of One-way encryption algorithm, the information of input can be encrypted and be converted
The information of random length is inputted for the integrality in inspection data transmission process for the hashed value of 128 regular lengths, is passed through
Processing is crossed, output is all 128 values of information, therefore the history feature value that opposite can ensure to obtain based on MD5 hash algorithm
And current characteristic value can have less byte number, be compared between relative reduction history feature value and current characteristic value
Compared with when to overall overhead caused by hardware resource.In addition, MD5 hash algorithm has faster calculating speed, it can be relatively high
History feature value and current characteristic value is calculated in effect, further improves the formation efficiency of zipper table.
In addition, as a preferred embodiment, data warehouse includes Hive data warehouse.
Hive data warehouse is the data warehouse based on the building of Hadoop big data framework, can be collected simply by increasing
The mode extension storage magnitude of group node, therefore more data records can be stored by tables of data, it is mentioned so as to opposite
The overall utility of high zipper table.
On the basis of a series of above-mentioned embodiments, as a preferred embodiment, the first character string and
The data type of two character strings is character string type.
It should be noted that due to consideration that there may be words in the data record of tables of data in practical application scene
The data content of section is empty situation, in order to guarantee each row of data when the data content of field is empty, in target matrix
Record is unique by the characteristic value that hash algorithm operation generates, and present embodiment is by the data content of each field of target matrix
After carrying out the conversion of character string (String) type, and then by way of adding separator between the data content in adjacent fields
Or the mode of direct splicing splices the data content of all fields in the table of source, generates the first character string and the second character with this
String opposite can be avoided when content change occurs for corresponding historgraphic data recording and current data record, the first character string with
Second character string is identical to be happened, and then ensures the content reliability of zipper table.
Shown in Figure 2, the embodiment of the invention also discloses a kind of zipper table creating devices, comprising:
History feature obtains module 10, for reading history number of the target matrix in data warehouse under the historical juncture
According to recording corresponding history feature value；Wherein, history feature value is by advance by the data of field each in historgraphic data recording
Content is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；
Current data obtains module 11, corresponding with historgraphic data recording under current time for obtaining target matrix
Current data record；
Current signature computing module 12, the data content for each field in recording current data are spliced into the second character
String, and hash algorithm operation is carried out to the second character string and generates current characteristic value；
Diversity judgement module 13, for judging history feature value and current characteristic value with the presence or absence of difference, if it is, adjusting
With zipper table generation module 14；
Zipper table generation module 14, the zipper table for having historgraphic data recording for generating record.
Zipper table creating device provided by the present invention, target matrix in data warehouse first under the reading historical juncture
The corresponding history feature value of historical data, wherein by advance by each field in historgraphic data recording when history feature value
Data content is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；And then obtain number of targets
According to the historgraphic data recording corresponding current data record of the table under current time, by current data record in each field
Data content is spliced into the second character string, and carries out identical hash algorithm operation to the second character string and generate current characteristic value,
History feature value and current characteristic value are finally judged with the presence or absence of difference, if it is, generating record has the historgraphic data recording
Zipper table.The present apparatus is that the history of hash algorithm operation generation is carried out by comparing the whole historgraphic data recording of target matrix
Whether deposited between characteristic value and the current characteristic value of the whole current data record progress hash algorithm operation generation of target matrix
In the mode of difference, the comparison that whether there is difference between the corresponding data record to target matrix under different moments is realized,
Since history feature value and current characteristic value can represent the data content of the record of the overall data in target matrix, nothing
The comparison that field one by one carries out content need to be carried out between data record, the consistent of history feature value and current characteristic value can be passed through
Property learns whether data record changes, to the whole of the calculation resources of cluster device in relative reduction zipper table generating process
Body occupies, and then ensures the overall operation stability of big data platform and reduce O&M pressure.
In addition, the embodiment of the present invention also provides a kind of zipper table generating device, comprising:
Memory, for storing computer program；
Processor is realized when for executing computer program such as the step of above-mentioned zipper table generating method.
Zipper table generating device provided by the present invention, target matrix in data warehouse first under the reading historical juncture
The corresponding history feature value of historical data, wherein by advance by each field in historgraphic data recording when history feature value
Data content is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；And then obtain number of targets
According to the historgraphic data recording corresponding current data record of the table under current time, by current data record in each field
Data content is spliced into the second character string, and carries out identical hash algorithm operation to the second character string and generate current characteristic value,
History feature value and current characteristic value are finally judged with the presence or absence of difference, if it is, generating record has the historgraphic data recording
Zipper table.This equipment is that the history of hash algorithm operation generation is carried out by comparing the whole historgraphic data recording of target matrix
Whether deposited between characteristic value and the current characteristic value of the whole current data record progress hash algorithm operation generation of target matrix
In the mode of difference, the comparison that whether there is difference between the corresponding data record to target matrix under different moments is realized,
Since history feature value and current characteristic value can represent the data content of the record of the overall data in target matrix, nothing
The comparison that field one by one carries out content need to be carried out between data record, the consistent of history feature value and current characteristic value can be passed through
Property learns whether data record changes, to the whole of the calculation resources of cluster device in relative reduction zipper table generating process
Body occupies, and then ensures the overall operation stability of big data platform and reduce O&M pressure.
In addition, the embodiment of the present invention also provides a kind of computer readable storage medium, deposited on computer readable storage medium
Computer program is contained, is realized when computer program is executed by processor such as the step of above-mentioned zipper table generating method.
Computer readable storage medium provided by the present invention, target data in data warehouse first under the reading historical juncture
The corresponding history feature value of the historical data of table, wherein by advance by each word in historgraphic data recording when history feature value
The data content of section is spliced into the first character string, and carries out hash algorithm operation generation to the first character string；And then obtain mesh
Mark tables of data under current time current data corresponding with the historgraphic data recording record, by current data record in each word
The data content of section is spliced into the second character string, and carries out identical hash algorithm operation to the second character string and generate current signature
Value finally judges history feature value and current characteristic value with the presence or absence of difference, if it is, generating record has historical data note
The zipper table of record.This computer readable storage medium is to carry out Hash calculation by comparing the whole historgraphic data recording of target matrix
The history feature value that method operation generates carries out the current of hash algorithm operation generation with the whole current data record of target matrix
Between characteristic value whether there is difference mode, realize under different moments target matrix corresponding data record between whether
The comparison having differences, since history feature value and current characteristic value can represent the record of the overall data in target matrix
Data content carries out the comparison of content there is no need to carry out between data record field one by one, can by history feature value with
The consistency of current characteristic value learns whether data record changes, and sets in relative reduction zipper table generating process to cluster
The whole of standby calculation resources occupies, and then ensures the overall operation stability of big data platform and reduce O&M pressure.
A kind of zipper table generating method provided by the present invention, device, equipment and medium are described in detail above.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other embodiments not
Same place, the same or similar parts in each embodiment may refer to each other.For the device disclosed in the embodiment, due to it
It corresponds to the methods disclosed in the examples, so being described relatively simple, reference may be made to the description of the method.It should
It points out, it for those skilled in the art, without departing from the principle of the present invention, can also be to this hair
Bright some improvement and modification can also be carried out, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or apparatus that includes the element.
Claims (10)
1. a kind of zipper table generating method characterized by comprising
Read historgraphic data recording corresponding history feature value of the target matrix in data warehouse under the historical juncture；Its
In, the history feature value is by the way that the data content of field each in the historgraphic data recording is spliced into the first character in advance
String, and hash algorithm operation generation is carried out to first character string；
Obtain target matrix current data record corresponding with the historgraphic data recording under current time；
The data content of each field in current data record is spliced into the second character string, and to second character string into
The row hash algorithm operation generates current characteristic value；
Judge the history feature value and the current characteristic value with the presence or absence of difference；
If it is, generating the zipper table that record has the historgraphic data recording.
2. zipper table generating method according to claim 1, which is characterized in that the number of targets read in data warehouse
According to the corresponding history feature value of historgraphic data recording of the table under the historical juncture, comprising:
The corresponding history feature value of the historgraphic data recording is read in preset middle table；Wherein, the middle table
Field based on the target matrix has additional the feature value fields for recording the history feature value.
3. zipper table generating method according to claim 1, which is characterized in that include in the field of the target matrix
There is the data record effective date field of storing data record effective date, the zipper table includes the data record effective date
Field and data record Expiration Date field.
4. zipper table generating method according to claim 1, which is characterized in that the acquisition target matrix is being worked as
Current data record corresponding with the historgraphic data recording is inscribed when preceding, comprising:
It is identical with the major key field content of the historgraphic data recording described under current time to obtain the target matrix
Current data record.
5. zipper table generating method according to claim 1, which is characterized in that the hash algorithm includes that MD5 Hash is calculated
Method.
6. zipper table generating method according to claim 1, which is characterized in that the data warehouse includes Hive data bins
Library.
7. according to claim 1 to zipper table generating method described in 6 any one, which is characterized in that first character string
And the data type of second character string is character string type.
8. a kind of zipper table creating device characterized by comprising
History feature obtains module, for reading historgraphic data recording of the target matrix in data warehouse under the historical juncture
Corresponding history feature value；Wherein, the history feature value is by advance by the number of field each in the historgraphic data recording
It is spliced into the first character string according to content, and hash algorithm operation generation is carried out to first character string；
Current data obtains module, corresponding with the historgraphic data recording under current time for obtaining the target matrix
Current data record；
Current signature computing module, for the data content of each field in current data record to be spliced into the second character
String, and the hash algorithm operation is carried out to second character string and generates current characteristic value；
Diversity judgement module, for judging that the history feature value and the current characteristic value whether there is difference, if it is,
Call zipper table generation module；
The zipper table generation module, the zipper table for having the historgraphic data recording for generating record.
9. a kind of zipper table generating device characterized by comprising
Memory, for storing computer program；
Processor realizes zipper table as described in any one of claim 1 to 7 generation side when for executing the computer program
The step of method.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program, the computer program realize zipper table generating method as described in any one of claim 1 to 7 when being executed by processor
The step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910532415.6A CN110209891A (en) | 2019-06-19 | 2019-06-19 | A kind of zipper table generating method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910532415.6A CN110209891A (en) | 2019-06-19 | 2019-06-19 | A kind of zipper table generating method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110209891A true CN110209891A (en) | 2019-09-06 |
Family
ID=67793610
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910532415.6A Pending CN110209891A (en) | 2019-06-19 | 2019-06-19 | A kind of zipper table generating method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209891A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078672A (en) * | 2019-12-20 | 2020-04-28 | 中国建设银行股份有限公司 | Data comparison method and device for database |
| CN111143350A (en) * | 2019-11-27 | 2020-05-12 | 深圳壹账通智能科技有限公司 | Enterprise data monitoring method and device, computer equipment and storage medium |
| CN112735144A (en) * | 2020-12-28 | 2021-04-30 | 浙江大华技术股份有限公司 | Fake plate identification method and device, computer equipment and storage medium |
| CN112749167A (en) * | 2021-01-18 | 2021-05-04 | 中国邮政储蓄银行股份有限公司 | Method and device for determining broken link data and nonvolatile storage medium |
| CN112905805A (en) * | 2021-03-05 | 2021-06-04 | 北京中经惠众科技有限公司 | Knowledge graph construction method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916262A (en) * | 2010-07-29 | 2010-12-15 | 北京用友政务软件有限公司 | Acceleration method of financial element matching |
| US20170026356A1 (en) * | 2015-07-22 | 2017-01-26 | Here Global B.V. | Method and apparatus for generating an intelligent primary key facilitating faster object retrieval |
| CN107193985A (en) * | 2017-05-27 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of slide fastener table design method of record data change histories |
| CN109446205A (en) * | 2017-08-28 | 2019-03-08 | 中国电信股份有限公司 | Judge the device and method of data mode and the device and method that data update |
-
2019
- 2019-06-19 CN CN201910532415.6A patent/CN110209891A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916262A (en) * | 2010-07-29 | 2010-12-15 | 北京用友政务软件有限公司 | Acceleration method of financial element matching |
| US20170026356A1 (en) * | 2015-07-22 | 2017-01-26 | Here Global B.V. | Method and apparatus for generating an intelligent primary key facilitating faster object retrieval |
| CN107193985A (en) * | 2017-05-27 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of slide fastener table design method of record data change histories |
| CN109446205A (en) * | 2017-08-28 | 2019-03-08 | 中国电信股份有限公司 | Judge the device and method of data mode and the device and method that data update |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143350A (en) * | 2019-11-27 | 2020-05-12 | 深圳壹账通智能科技有限公司 | Enterprise data monitoring method and device, computer equipment and storage medium |
| CN111078672A (en) * | 2019-12-20 | 2020-04-28 | 中国建设银行股份有限公司 | Data comparison method and device for database |
| CN111078672B (en) * | 2019-12-20 | 2023-06-02 | 中国建设银行股份有限公司 | Data comparison method and device for database |
| CN112735144A (en) * | 2020-12-28 | 2021-04-30 | 浙江大华技术股份有限公司 | Fake plate identification method and device, computer equipment and storage medium |
| CN112749167A (en) * | 2021-01-18 | 2021-05-04 | 中国邮政储蓄银行股份有限公司 | Method and device for determining broken link data and nonvolatile storage medium |
| CN112905805A (en) * | 2021-03-05 | 2021-06-04 | 北京中经惠众科技有限公司 | Knowledge graph construction method and device, computer equipment and storage medium |
| CN112905805B (en) * | 2021-03-05 | 2023-09-15 | 北京中经惠众科技有限公司 | Knowledge graph construction method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209891A (en) | A kind of zipper table generating method, device, equipment and medium | |
| Bennett et al. | Malstone: towards a benchmark for analytics on large data clouds | |
| CN105787128B (en) | A method of restoring Java and serializes file data | |
| CN109919691B (en) | Data processing system, method and device | |
| CN106682077A (en) | Method for storing massive time series data on basis of Hadoop technologies | |
| CN106844682A (en) | Method for interchanging data, apparatus and system | |
| CN107037978A (en) | Data Migration bearing calibration and system | |
| WO2021057482A1 (en) | Method and device for generating bloom filter in blockchain | |
| CN106484734A (en) | A kind of data query caching method and system | |
| Eppstein et al. | Separator based sparsification: I. Planarity testing and minimum spanning trees | |
| CN104965835B (en) | A kind of file read/write method and device of distributed file system | |
| CN109344268A (en) | Method, electronic equipment and the computer readable storage medium of graphic data base write-in | |
| CN110119947B (en) | Method and apparatus for shared workload proof computing power generation of symbiotic blockchains | |
| CN110134511A (en) | A kind of shared storage optimization method of OpenTSDB | |
| CN103488755B (en) | A kind of file system access method and apparatus | |
| CN107368404A (en) | A kind of method of auditing administration and system | |
| TWI522827B (en) | Real-time storage and real-time reading of huge amounts of data for non-related databases | |
| He et al. | SLC-index: A scalable skip list-based index for cloud data processing | |
| CN112381583A (en) | Power consumption calculation method and device based on distributed memory calculation technology | |
| CN102521451B (en) | A kind of electric network model file, generation method and the system of supporting accelerated model to splice | |
| CN111538804A (en) | HBase-based graph data processing method and equipment | |
| CN110245148A (en) | A kind of date storage method, device, system and medium | |
| He | Construction of Teaching Management Platform for Universities Based on Big Data | |
| CN113626438B (en) | Data table management method, device, computer equipment and storage medium | |
| Liu et al. | Digital preservation and presentation of institution photo archives: the Anhui University Memory Project Experience |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190906 |