CN110457401B - Data storage method and device, computer equipment and storage medium - Google Patents
Data storage method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110457401B CN110457401B CN201910611455.XA CN201910611455A CN110457401B CN 110457401 B CN110457401 B CN 110457401B CN 201910611455 A CN201910611455 A CN 201910611455A CN 110457401 B CN110457401 B CN 110457401B
- Authority
- CN
- China
- Prior art keywords
- data
- data item
- date
- current period
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The application relates to a data storage method, a data storage device, computer equipment and a storage medium. The method comprises the following steps: generating a date data item relation list of each service identifier according to the historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier; generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content; and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier. By adopting the method, the development difficulty can be reduced while the storage resources are reduced.
Description
Technical Field
The present application relates to the field of data warehouse technologies, and in particular, to a data storage method and apparatus, a computer device, and a storage medium.
Background
A data warehouse is a theme-oriented, integrated, relatively stable collection of data that reflects historical changes. In the data warehouse, the proportion of the variation of partial service data to the whole quantity is small (such as information of members, organizations, employees and the like), the full data state of the current date or the appointed date needs to be traversed in each use in the data analysis process, and the means adopted by the data warehouse is a periodic snapshot table or slow change dimension solution, but the two modes have the following defects:
the first mode is a periodic snapshot table, that is, each period stores one current full amount of data; the method has the advantages of simple implementation and small data volume, and when billions of data volumes such as members and commodities exist, a full scale is stored every period, so that the storage resource consumption of a large data platform is huge.
The second method is to change dimensions slowly, add an effective start time and end time to data, and when data changes, add a data zipper to store, and when data does not change, add new data, however, it is tedious to use SQL (Structured Query Language) to generate a zipper.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data storage method, an apparatus, a computer device, and a storage medium capable of reducing development difficulty while reducing storage resources.
A method of data storage, the method comprising:
generating a date data item relation list of each service identifier according to historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data contents;
and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
In one embodiment, the method further includes:
acquiring current period data item identifications and historical data item identifications of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item identifications with the historical data item identifications;
if the target data item identification does not exist in the current period data item identifications and the target data item identification exists in the historical data item identifications, historical data of the target data item identification recorded in the data limit storage table are reserved.
In one embodiment, the method further includes:
when target data item identifications exist in all current period data item identifications and all historical data item identifications, acquiring current period data item contents and historical data item contents of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item content with the historical data item content;
when the same data item content exists in the data item content of the current period and the historical data item content, the historical data of the target data item identification recorded in the data limit storage table is reserved, and the corresponding relation between the target service identification and the date of the current period is added in a date data item relation list of the target service identification of the data limit storage table.
In one embodiment, the method further includes:
and when the content of the data item in the current period and the content of the historical data item do not have the same content, merging the current period data identified by the target data item into the data limit storage table.
In one embodiment, the method further includes:
and when the target data item identification exists in each current period data item identification and the target data item identification does not exist in each historical data item identification, converting the current period data of the target data item identification according to the data storage structure of the data limit storage table, and storing the converted data into the data limit storage table.
In one embodiment, the method further includes: associating the current periodic variation data with the data limit storage table according to the service identification to obtain an association result;
the above obtaining the current period data item identifier and the historical data item identifier of the target service identifier according to the current period change data and the data limit storage table includes: and inquiring the current period data item identification and the historical data item identification of the target service identification according to the correlation result.
In one embodiment, the method further includes:
acquiring a full-volume data request with a specified date, wherein the specified date of the full-volume data request is a first data date;
in each date data item relation list, the query date is less than or equal to the maximum date data item identification in the first data date;
inquiring corresponding data content in each data item content list according to the maximum date data item identification;
or/and
acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date;
in each date data item relation list, inquiring data item identifications with dates equal to the second data date to obtain a data identification item list;
and identifying the corresponding data content in the corresponding data item content list according to the data identification item list.
A data storage device, the device comprising:
the first generation module is used for generating a date data item relation list of each service identifier according to historical data, and the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
the second generation module is used for generating a data item content list of each service identifier according to the historical data, and the data item content list records the corresponding relation between the data item identifiers and the data content;
and the third generation module is used for generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
generating a date data item relation list of each service identifier according to historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content;
and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
generating a date data item relation list of each service identifier according to historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content;
and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
According to the data storage method, the data storage device, the computer equipment and the storage medium, the date data item relation list of each service identifier is generated according to historical data, the date data item relation list records the corresponding relation between the data date of the data and the data item identifier, the data item content list of each service identifier is generated according to the historical data, the data item content list records the corresponding relation between the data item identifier and the data content, and the data limit storage table is generated based on the date data item relation list of each service identifier and the data item content list of each service identifier, so that the full data and the change data specified by the date can be obtained without storing one full data in each period.
Drawings
FIG. 1 is a diagram of an exemplary data storage system;
FIG. 2 is a schematic flow chart diagram illustrating a data storage method according to one embodiment;
FIG. 3 is a diagram of a data storage structure in one embodiment;
FIG. 4 is a diagram that illustrates a date data item relationship list and a data item content relationship list, in one embodiment;
FIG. 5 is a flow chart illustrating the storing step of the current cycle change data in one embodiment;
FIG. 6 is a flowchart illustrating the storing step of the current period change data when the current period and the history have the identification of the target data item in one embodiment;
FIG. 7 is a flowchart illustrating a storing step of current cycle change data in another embodiment;
FIG. 8 is a block diagram of a data storage device in one embodiment;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client, without departing from the scope of the present application. Both the first client and the second client are clients, but they are not the same client. The term "or/and" describes the association relationship of the associated object, indicating that there may be three relationships, for example, a or/and B, which may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The data storage method provided by the application can be applied to the application environment shown in fig. 1. Wherein the server 104 communicates with the data warehouse (or data warehouse facility) 106 and the terminal 102 via a network. The server 104 may generate a data limit storage table from the historical data recorded by the data warehouse 106, where the data limit storage table includes each service identifier, and a date data item relation list and a data item content list corresponding to each service identifier, where the date data item relation list records a corresponding relation between a data date of the data and the data item identifier, and the data item content list records a corresponding relation between the data item identifier and the data content, so that the development difficulty can be reduced while reducing storage resources. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers.
In one embodiment, as shown in fig. 2, a data storage method is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:
step 202, generating a date data item relation list of each service identifier according to historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
here, the historical data is generally slow-changing data in a data warehouse, the slow-changing data refers to that the corresponding business data change amount is small in proportion to the overall data change amount (for example, lower than a preset threshold), and the slow-changing data can be elected according to actual needs, for example, the slow-changing data can be information of members, organizations, employees, and the like.
Here, the service identification means a service unique key code, which may be, for example, a commodity code, a member code, an organization code, a job number, and the like.
Here, the data item identification refers to identification information that can be used to distinguish data contents of different items, which may also be referred to as data item encoding, and the data item identifications of different data contents are generally different.
Here, the data date refers to the date of the data content, i.e., the date on which the corresponding service occurred.
Specifically, the server obtains each data item identifier and data date of each service identifier from the historical data, and respectively generates a date data item relation list of each service identifier according to each data item identifier and data date of each service identifier.
Step 204, generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data contents;
specifically, the server obtains the data item identifiers and the data contents of the service identifiers from the historical data, and respectively generates a data item content list of the service identifiers according to the data item identifiers and the data contents of the service identifiers.
Step 206, generating a data limit storage table, where the data limit storage table includes each service identifier, and a date data item relation list and a data item content list corresponding to each service identifier.
One service identifier may correspond to a data storage structure in fig. 3, and as shown in fig. 3, the data storage structure table includes a service identifier, and a date data item relation table and a data item content table of the service identifier. FIG. 4 is a diagram of a date data item relationship table and a data item content table, according to an embodiment.
In the data storage method, because the date data item relation list of each service identifier is generated according to the historical data, the date data item relation list records the corresponding relation between the data date of the data and the data item identifier, and because the data item content list of each service identifier is generated according to the historical data, the data item content list records the corresponding relation between the data item identifier and the data content, and because the data limit storage table is generated based on the date data item relation list of each service identifier and the data item content list of each service identifier, the full data and the change data of the specified date can be obtained without storing one full data in each period, and compared with the period snapshot table mode, the storage resources are reduced, and if the full data and the change data of the specified date are inquired, the reference only needs to be made on the specified data date, so that the development only needs simple SQL transfer and calling, the internal implementation details do not need to be concerned, and compared with the slow change dimension mode, the development complexity is simplified.
In one embodiment, the data storage method of the present application may further include a step of storing current period variation data, as shown in fig. 5, the step of storing the current period variation data may include the following steps:
step 502, acquiring current period data item identifications and historical data item identifications of target service identifications according to current period change data and a data limit storage table;
here, the current period change data refers to change data in the current period. The length of the period can be set according to actual needs, for example, it can be one day. The change data generally includes change data and addition data.
Here, the target service identifier may be any one of the service identifiers in the current period change data or the data limit storage table.
Specifically, the server obtains, according to the current period change data and the data limit storage table, each current period data item identifier of the target service identifier and each historical data item identifier of the target service identifier. The current period data item identification refers to the data item identification in the current period of the target service identification, and the historical data item identification refers to the data item identification of the target service identification in the data limit storage table.
Step 504, comparing the current period data item identifications with the historical data item identifications;
specifically, the server may compare the current period data item identifiers with the historical data item identifiers one by one.
Step 506, if the target data item identifier does not exist in the data item identifiers of the current period and the target data item identifier exists in the historical data item identifiers, the historical data of the target data item identifier recorded in the data limit storage table is reserved.
Here, to keep the history data of the target data item identifier of the record in the data limit storage table, and to directly change or add the current cycle data in the data limit storage table, it may be understood that the history data of the target data item identifier of the record in the data limit storage table is maintained, and to generate the second data limit storage table from the first data limit storage table and the current cycle data, it may be understood that the history data of the target data item identifier of the record in the first data limit storage table is returned to the second data limit storage table.
In the scheme of the embodiment, when only the target data item identifier exists in the history storage data (namely, the data limit storage table) and the target data item identifier does not exist in the current period, only the history data of the target data item identifier recorded in the data limit storage table is retained, and the data item content does not need to be newly added in the data item content list, and the additional storage is not added.
In one embodiment, as shown in fig. 6, the step of storing the current period variation data may further include the steps of:
step 602, when target data item identifiers exist in all current period data item identifiers and all historical data item identifiers, obtaining current period data item contents and historical data item contents of target service identifiers according to current period change data and a data limit storage table;
specifically, when the target data item identifier exists in each current period data item identifier and the target data item identifier also exists in each historical data item identifier, the server acquires the current period data item content and the historical data item content of the target service identifier according to the current period change data and the data limit storage table.
Step 604, comparing the content of the current period data item with the content of the historical data item;
specifically, the server may compare the current period data item content of the target service identifier with the historical data item content of the target service identifier one by one.
Step 606, when the data item content in the current period and the historical data item content have the same data item content, the historical data of the target data item identifier recorded in the data limit storage table is reserved, and the corresponding relation between the target service identifier and the date in the current period is added in the date data item relation list of the target service identifier in the data limit storage table.
In addition, the relationship between date and historical version, that is, the corresponding relationship between data date and data version, can be added to the date data item relationship list of the target service identifier.
By adopting the scheme of the embodiment, if a certain change data of the current period appears in the historical data, the data item content list does not need to be added with the data item content (because the historical version already exists), but only a relationship between the data date and the data item identifier is added in the date data item relationship list, so that the data storage capacity can be reduced. In addition, if the date data item relationship list includes the corresponding relationship between the data date and the data version, the date data item relationship list may also be compared with the corresponding data of the near N versions (not all the historical storage data) to determine whether the content of the historical data item has the changed data, so as to reduce the number of comparisons and improve the processing efficiency. The size of N can be set according to actual needs.
In one embodiment, the step of storing the current period variation data may further include the steps of: when the same data item content does not exist in the current period data item content and the historical data item content, the current period data identified by the target data item is merged into the data limit storage table.
Here, the merging the current period data identified by the target data item into the data limit storage table may specifically include: and adding a corresponding relation between a target data item identifier and the date of the current period in a date data item relation list of the target service identifier, and adding a corresponding relation between the target data item identifier and the data content of the current period of the target data item identifier in a data item content list of the target service identifier.
In the solution of this embodiment, when the content of the data item in the current period and the content of the history data item do not have the same content, only data merging needs to be performed, which is simple and easy to implement.
In one embodiment, the step of storing the current period variation data may further include the steps of: and when the target data item identification exists in each current period data item identification and the target data item identification does not exist in each historical data item identification, converting the current period data of the target data item identification according to the data storage structure of the data limit storage table, and storing the converted data into the data limit storage table.
In one embodiment, the data storage method of the present application may further include the steps of: associating the current periodic variation data with the data limit storage table according to the service identification to obtain an association result; the aforementioned obtaining, according to the current period change data and the data limit storage table, the current period data item identifiers and the historical data item identifiers of the target service identifier may include the steps of: and inquiring the current period data item identification and the historical data item identification of the target service identification according to the correlation result.
Specifically, the current period change data can be acquired, the current period change data and the data limit storage table are linked according to the service identifier, that is, join operation is performed on the two tables according to the primary key ID, so that data query can be facilitated.
A preferred embodiment of the step of storing the current period variation data is described below, taking the current period as the current day as an example. However, it should be noted that the current period is the current day and does not constitute a limitation to the scheme of the present invention. For convenience of explanation, a data limit storage table that does not include the current-day change data but includes only the history data is hereinafter referred to as a historical slow change data limit storage table, and a limit storage table that includes both the current-day change data and the history data is hereinafter referred to as a slow change data limit storage table.
As shown in fig. 7, the step of storing the current period change data in the present embodiment includes the following steps:
step 702, performing data association on the current-day change data and the historical slow change data limit storage table;
specifically, the daily change data and the historical slow change data limit storage table may be obtained, and data association broadening may be performed on the daily change data and the historical slow change data limit storage table according to the service identifier.
Step 704, comparing the data item identification of the current day with the data item identification in the history slow change data limit storage table, if the current day has no history, then entering step 706, if the current day has no history, then entering step 708, if the current day has both current day and history, then entering step 710;
specifically, the data item identifier of the current day of each service identifier and the data item identifier in the historical slow change data limit storage table (hereinafter referred to as historical data item identifier) may be compared according to the service identifier; the data item identifications of different service identifications can have three conditions of ' having no history on the same day ', having history on the same day ' and ' having both history on the same day ', and the three conditions can also occur for different data item identifications of the same service identification.
the limit storage format herein refers to the data storage structure of the data limit storage table;
specifically, the data of the current day (i.e., the data of the current day corresponding to the data item identifier having history on the current day) may be converted into the limit storage format.
Step 708, returning historical data;
specifically, the corresponding historical data (i.e. historical data corresponding to the data item identification which has no history in the current day) can be returned
Step 710, comparing the content of the data item of the current day with the content of the historical data item, judging whether the same data exists according to the comparison result, if not, entering step 712, and if so, entering step 714;
step 712, merging the data of the current day into the historical slow change data limit storage table;
specifically, the corresponding data of the current day (i.e., the data of the current day corresponding to the data item identifier of both the current day and the history and having no same data compared with the history data item) may be merged into the corresponding position in the history slow change data limit storage table.
Step 714, returning historical data;
specifically, the corresponding historical data (i.e., the historical data corresponding to the data item identifiers of both the current day and the history and having the same data compared with the historical data item) may be returned.
Step 716, adding the relationship between the date and the historical version in the date data item relationship list;
here, the data date and history version relationship may refer to a correspondence relationship of the data date and the data version identification.
Wherein, the data of the service identifier may be processed according to the above steps 704 to 716, and the slow change data limit storage table may include the data processed in steps 706, 708, 712, 714 and 716.
In one embodiment, the data storage method of the present application may further include a step of requesting full data corresponding to a specified date or/and a step of requesting changed data corresponding to a specified date.
Specifically, the step of requesting full data corresponding to the fixed date may include the steps of: acquiring a full-volume data request with a specified date, wherein the specified date of the full-volume data request is a first data date; in each date data item relation list, the query date is less than or equal to the maximum date data item identification in the first data date; and inquiring corresponding data content in each data item content list according to the maximum date data item identification.
The maximum date data item identification refers to a date data item identification (or referred to as a date data item relationship) corresponding to the maximum date in data dates smaller than or equal to the first data date of the same data item. For example, there are 20190101, 20190215, 20190501 and 20191701 in a data item relationship list of a row of data, if a full-size data request of specified date 20190601 is requested, all three items of data 20190101, 20190215 and 20190501 are searched in the row of data, and then the largest data item 20190501 is searched in the three items of data and returned.
Specifically, the date-fixed data request step may include the steps of: acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date; in each date data item relation list, inquiring data item identifications with dates equal to the second data date to obtain a data identification item list; and identifying the corresponding data content in the corresponding data item content list according to the data identification item list.
In the embodiment, the participation of the data request only needs one participation data date, and meanwhile, the query logic is simplified compared with a slowly-changing dimension mode, so that the implementation difficulty can be reduced.
It should be understood that although the various steps in the flowcharts of fig. 2, 5-7 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2, 5-7 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 8, there is provided a data storage device comprising: a first generation module 802, a second generation module 804, and a third generation module 806, wherein:
a first generating module 802, configured to generate a date data item relationship list of each service identifier according to historical data, where the date data item relationship list records a correspondence between a data date of data and a data item identifier;
a second generating module 804, configured to generate a data item content list of each service identifier according to the historical data, where the data item content list records a corresponding relationship between the data item identifier and the data content;
a third generating module 806, configured to generate a data limit storage table, where the data limit storage table includes each service identifier, and a corresponding date data item relation list and data item content list of each service identifier.
In one embodiment, the data storage device may further include an update module, where the update module includes a query unit, a comparison unit, and a first processing unit, where:
the query unit is used for acquiring the current period data item identifications and the historical data item identifications of the target service identification according to the current period change data and the data limit storage table;
the comparison unit is used for comparing the current period data item identifications and the historical data item identifications;
and the first processing unit is used for keeping the historical data of the target data item identification recorded in the data limit storage table when the target data item identification does not exist in each current period data item identification and the target data item identification exists in each historical data item identification.
In one embodiment, the update module may further include a second processing unit, where the second processing unit is configured to, when target data item identifiers exist in each current period data item identifier and each historical data item identifier, obtain a current period data item content and a historical data item content of the target service identifier according to the current period change data and the data limit storage table, compare the current period data item content and the historical data item content, when the current period data item content and the historical data item content have the same data item content, keep the historical data of the target data item identifier recorded in the data limit storage table, and add a corresponding relationship between the target service identifier and a current period date in a date data item relationship list of the target service identifier in the data limit storage table.
In one embodiment, the second processing unit may be further configured to merge the current period data identified by the target data item into the data limit storage table when the current period data item content and the history data item content do not have the same data item content.
In one embodiment, the update module may further include a third processing unit, where the third processing unit is configured to, when a target data item identifier exists in each current-period data item identifier and a target data item identifier does not exist in each historical data item identifier, convert current-period data of the target data item identifier according to a data storage structure of the data limit storage table, and store the converted data in the data limit storage table.
In one embodiment, the data storage device may further include a data association module, where the data association module is configured to associate current periodic variation data with a data limit storage table according to a service identifier to obtain an association result; the query unit may query, according to the association result, the current period data item identifiers and the historical data item identifiers of the target service identifier.
In one embodiment, the data storage device may further include a first request processing module or/and a second request processing module.
The first request processing module is used for acquiring a full data request with a specified date, the specified date of the full data request is a first data date, in each date data item relation list, the query date is less than or equal to the maximum date data item identifier in the first data date, and the corresponding data content is queried in each data item content list according to the maximum date data item identifier.
The second request processing module is used for acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date, inquiring data item identifications with dates equal to the second data date in each date data item relation list to obtain a data identification item list, and obtaining corresponding data contents in the corresponding data item content list according to the data identification item list.
For specific limitations of the data storage device, reference may be made to the above limitations of the data storage method, which are not described herein again. The various modules in the data storage device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface and database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal, a database device or a data warehouse device through network connection. The computer program when executed by a processor implements a data storage method.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
generating a date data item relation list of each service identifier according to the historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content;
and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring current period data item identifications and historical data item identifications of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item identification with the historical data item identification;
if the target data item identification does not exist in the current period data item identifications and the target data item identification exists in the historical data item identifications, historical data of the target data item identification recorded in the data limit storage table are reserved.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
when target data item identifications exist in all current period data item identifications and all historical data item identifications, acquiring current period data item contents and historical data item contents of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item content with the historical data item content;
when the same data item content exists in the data item content of the current period and the historical data item content, the historical data of the target data item identification recorded in the data limit storage table is reserved, and the corresponding relation between the target service identification and the date of the current period is added in a date data item relation list of the target service identification of the data limit storage table.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and when the content of the data item in the current period and the content of the historical data item do not have the same content, merging the current period data identified by the target data item into the data limit storage table.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and when the target data item identification exists in the current period data item identification and the target data item identification does not exist in the historical data item identification, converting the current period data of the target data item identification according to the data storage structure of the data limit storage table, and storing the converted data into the data limit storage table.
In one embodiment, the method further includes: correlating the current periodic variation data with the data limit storage table according to the service identification to obtain a correlation result;
when the processor executes the computer program to realize the step of acquiring the current period data item identifications and the historical data item identifications of the target service identification according to the current period change data and the data limit storage table, the following steps are specifically realized: and inquiring the current period data item identification and the historical data item identification of the target service identification according to the correlation result.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a full-volume data request with a specified date, wherein the specified date of the full-volume data request is a first data date;
in each date data item relation list, the query date is less than or equal to the maximum date data item identification in the first data date;
and inquiring corresponding data content in each data item content list according to the maximum date data item identification.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date;
in each date data item relation list, inquiring data item identifications with dates equal to the second data date to obtain a data identification item list;
and identifying the corresponding data content in the corresponding data item content list according to the data identification item list.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
generating a date data item relation list of each service identifier according to the historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier;
generating a data item content list of each service identifier according to the historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content;
and generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and a date data item relation list and a data item content list which correspond to each service identifier.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring current period data item identifications and historical data item identifications of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item identification with the historical data item identification;
if the target data item identification does not exist in the data item identification of each current period and the target data item identification exists in the historical data item identification, historical data of the target data item identification recorded in the data limit storage table is reserved.
In one embodiment, the computer program when executed by the processor further performs the steps of:
when target data item identifications exist in all current period data item identifications and all historical data item identifications, acquiring current period data item contents and historical data item contents of target service identifications according to current period change data and a data limit storage table;
comparing the current period data item content with the historical data item content;
when the same data item content exists in the data item content of the current period and the historical data item content, the historical data of the target data item identification recorded in the data limit storage table is reserved, and the corresponding relation between the target service identification and the date of the current period is added in a date data item relation list of the target service identification of the data limit storage table.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and when the content of the data item in the current period and the content of the historical data item do not have the same content, merging the current period data identified by the target data item into the data limit storage table.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and when the target data item identification exists in each current period data item identification and the target data item identification does not exist in each historical data item identification, converting the current period data of the target data item identification according to the data storage structure of the data limit storage table, and storing the converted data into the data limit storage table.
In one embodiment, the method further includes: associating the current periodic variation data with the data limit storage table according to the service identification to obtain an association result;
when the computer program is executed by the processor to realize the step of acquiring the current period data item identifications and the historical data item identifications of the target service identification according to the current period change data and the data limit storage table, the following steps are specifically realized: and inquiring the current period data item identification and the historical data item identification of the target service identification according to the correlation result.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a full-volume data request with a specified date, wherein the specified date of the full-volume data request is a first data date;
in each date data item relation list, the query date is less than or equal to the maximum date data item identification in the first data date;
and inquiring corresponding data content in each data item content list according to the maximum date data item identification.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date;
in each date data item relation list, inquiring data item identifications with dates equal to the second data date to obtain a data identification item list;
and identifying the corresponding data content in the corresponding data item content list according to the data identification item list.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (14)
1. A method of data storage, the method comprising:
generating a date data item relation list of each service identifier according to historical data, wherein the date data item relation list records the corresponding relation between the data date of the data and the data item identifier, and the historical data comprises slow change data in a data warehouse;
generating a data item content list of each service identifier according to historical data, wherein the data item content list records the corresponding relation between the data item identifiers and the data content;
generating a data limit storage table, wherein the data limit storage table comprises each service identifier, and the date data item relation list and the data item content list corresponding to each service identifier;
the method further comprises the following steps:
acquiring a full-volume data request with a specified date, wherein the specified date of the full-volume data request is a first data date;
in each date data item relation list, the query date is less than or equal to the maximum date data item identification in the first data date;
inquiring corresponding data content in each data item content list according to the maximum date data item identification;
or/and
acquiring a change data request with a specified date, wherein the specified date of the change data request is a second data date;
in each date data item relation list, inquiring data item identifications with dates equal to the second data date to obtain a data identification item list;
and according to the data identification item list, corresponding data content in the corresponding data item content list.
2. The method of claim 1, further comprising:
acquiring current period data item identifications and historical data item identifications of target service identifications according to current period change data and the data limit storage table;
comparing each current period data item identification with each historical data item identification;
if the target data item identification does not exist in each current period data item identification and the target data item identification exists in each historical data item identification, the historical data of the target data item identification recorded in the data limit storage table is reserved.
3. The method of claim 2, further comprising:
when the target data item identification exists in each current period data item identification and each historical data item identification, acquiring the current period data item content and the historical data item content of the target service identification according to the current period change data and the data limit storage table;
comparing the current period data item content with the historical data item content;
when the current period data item content and the historical data item content have the same data item content, the historical data of the target data item identification recorded in the data limit storage table is reserved, and the corresponding relation between the target service identification and the current period date is added in a date data item relation list of the target service identification of the data limit storage table.
4. The method of claim 3, further comprising:
merging the current period data identified by the target data item into the data limit storage table when the same data item content does not exist in the current period data item content and the historical data item content.
5. The method of any of claims 2 to 4, further comprising:
and when target data item identifications exist in each current period data item identification and no target data item identification exists in each historical data item identification, converting the current period data of the target data item identification according to the data storage structure of the data limit storage table, and storing the converted data into the data limit storage table.
6. The method of claim 5, further comprising: associating the current period change data with the data limit storage table according to a service identifier to obtain an association result;
the acquiring of the current period data item identifications and the historical data item identifications of the target service identification according to the current period change data and the data limit storage table comprises: and inquiring the current period data item identification and the historical data item identification of the target service identification according to the correlation result.
7. A data storage device, characterized in that the device comprises:
the system comprises a first generation module, a second generation module and a third generation module, wherein the first generation module is used for generating a date data item relation list of each service identifier according to historical data, and the date data item relation list records the corresponding relation between the data date of data and the data item identifier;
the second generation module is used for generating a data item content list of each service identifier according to historical data, and the data item content list records the corresponding relation between the data item identifiers and the data content;
a third generating module, configured to generate a data limit storage table, where the data limit storage table includes each service identifier, and the date data item relationship list and the data item content list corresponding to each service identifier;
the device also comprises a first request processing module or/and a second request processing module:
the first request processing module is used for acquiring a full data request with a specified date, the specified date of the full data request is a first data date, in each date data item relation list, a query date is less than or equal to a maximum date data item identifier in the first data date, and corresponding data content is queried in each data item content list according to the maximum date data item identifier;
the second request processing module is configured to acquire a change data request with a specified date, where the specified date of the change data request is a second data date, query, in each date-data item relationship list, a data item identifier whose date is equal to the second data date to obtain a data identifier item list, and obtain, according to a data content corresponding to the data identifier item list in the corresponding data item content list, a data identifier item list.
8. The apparatus of claim 7, further comprising an update module, the update module comprising:
the query unit is used for acquiring the current period data item identifications and the historical data item identifications of the target service identification according to the current period change data and the data limit storage table;
the comparison unit is used for comparing each current period data item identification with each historical data item identification;
a first processing unit, configured to, if there is no target data item identifier in each current-period data item identifier and there is a target data item identifier in each historical data item identifier, retain historical data of the target data item identifier recorded in the data limit storage table.
9. The apparatus of claim 8, wherein the update module further comprises:
a second processing unit, configured to, when the target data item identifier exists in each current period data item identifier and each historical data item identifier, obtain a current period data item content and a historical data item content of the target service identifier according to the current period change data and the data limit storage table, compare the current period data item content and the historical data item content, when the current period data item content and the historical data item content have the same data item content, keep the historical data of the target data item identifier recorded in the data limit storage table, and add a corresponding relationship between the target service identifier and a current period date in a date data item relationship list of the target service identifier in the data limit storage table.
10. The apparatus of claim 9, wherein:
the second processing unit is further configured to merge the current period data identified by the target data item into the data limit storage table when the current period data item content and the history data item content do not have the same data item content.
11. The apparatus of any one of claims 8 to 10, wherein the update module further comprises:
and the third processing unit is further configured to, when a target data item identifier exists in each current period data item identifier and a target data item identifier does not exist in each historical data item identifier, convert the current period data of the target data item identifier according to the data storage structure of the data limit storage table, and store the converted data in the data limit storage table.
12. The apparatus according to claim 11, further comprising a data association module, wherein the data association module is configured to associate the current period change data with the data limit storage table according to a service identifier, so as to obtain an association result;
and the query unit queries the current period data item identifications and the historical data item identifications of the target service identification according to the correlation result.
13. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 6 are implemented when the computer program is executed by the processor.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910611455.XA CN110457401B (en) | 2019-07-08 | 2019-07-08 | Data storage method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910611455.XA CN110457401B (en) | 2019-07-08 | 2019-07-08 | Data storage method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110457401A CN110457401A (en) | 2019-11-15 |
| CN110457401B true CN110457401B (en) | 2022-11-08 |
Family
ID=68482444
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910611455.XA Active CN110457401B (en) | 2019-07-08 | 2019-07-08 | Data storage method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110457401B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078714B (en) * | 2019-11-25 | 2023-08-15 | 泰康保险集团股份有限公司 | Data processing method and device |
| CN113495891B (en) * | 2020-03-19 | 2023-09-26 | 北京京东振世信息技术有限公司 | Data processing method and device |
| CN113094000A (en) * | 2021-05-10 | 2021-07-09 | 宝能(广州)汽车研究院有限公司 | Vehicle signal storage method and device, storage equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541952A (en) * | 2010-12-29 | 2012-07-04 | 北大方正集团有限公司 | Method and device for acquiring history data on basis of database |
| CN104834487A (en) * | 2015-05-24 | 2015-08-12 | 华东电网有限公司 | Remote signaling data compression storage and vacancy checking and filling method based on slow change dimension format |
| CN104899199A (en) * | 2014-03-04 | 2015-09-09 | 阿里巴巴集团控股有限公司 | Data processing method and system for data warehouse |
| CN107193985A (en) * | 2017-05-27 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of slide fastener table design method of record data change histories |
| CN107526733A (en) * | 2016-06-20 | 2017-12-29 | 咪咕互动娱乐有限公司 | A kind of slide fastener table date storage method and device |
| CN109408501A (en) * | 2018-11-07 | 2019-03-01 | 北京锐安科技有限公司 | A kind of processing method of position data, device, server and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180300362A1 (en) * | 2015-04-11 | 2018-10-18 | Entit Software Llc | Dimension data insertion into a dimension table |
| US10983972B2 (en) * | 2017-09-08 | 2021-04-20 | Oracle International Corporation | System and method for slowing changing dimension and metadata versioning in a multidimensional database environment |
-
2019
- 2019-07-08 CN CN201910611455.XA patent/CN110457401B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541952A (en) * | 2010-12-29 | 2012-07-04 | 北大方正集团有限公司 | Method and device for acquiring history data on basis of database |
| CN104899199A (en) * | 2014-03-04 | 2015-09-09 | 阿里巴巴集团控股有限公司 | Data processing method and system for data warehouse |
| CN104834487A (en) * | 2015-05-24 | 2015-08-12 | 华东电网有限公司 | Remote signaling data compression storage and vacancy checking and filling method based on slow change dimension format |
| CN107526733A (en) * | 2016-06-20 | 2017-12-29 | 咪咕互动娱乐有限公司 | A kind of slide fastener table date storage method and device |
| CN107193985A (en) * | 2017-05-27 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of slide fastener table design method of record data change histories |
| CN109408501A (en) * | 2018-11-07 | 2019-03-01 | 北京锐安科技有限公司 | A kind of processing method of position data, device, server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110457401A (en) | 2019-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020186786A1 (en) | File processing method and apparatus, computer device and storage medium | |
| CN109408746B (en) | Image information query method, image information query device, computer equipment and storage medium | |
| CN108573371B (en) | Data approval method, device, computer equipment and storage medium | |
| CN110457401B (en) | Data storage method and device, computer equipment and storage medium | |
| CN109492017B (en) | Service information query processing method, system, computer equipment and storage medium | |
| CN110659298B (en) | Financial data processing method and device, computer equipment and storage medium | |
| CN110781214A (en) | Database reading and writing method and device, computer equipment and storage medium | |
| US20130173509A1 (en) | Method and arrangement for processing data | |
| CN110555165B (en) | Information identification method and device, computer equipment and storage medium | |
| CN112000903A (en) | Data query method and device, computer equipment and storage medium | |
| CN109656947B (en) | Data query method and device, computer equipment and storage medium | |
| CN112613271A (en) | Data paging method and device, computer equipment and storage medium | |
| CN112380130A (en) | Application testing method and device based on call dependency relationship | |
| CN112948504B (en) | Data acquisition method and device, computer equipment and storage medium | |
| CN114201511A (en) | Project management and control method and device, computer equipment and storage medium | |
| CN109542962B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN115544007A (en) | Label preprocessing method and device, computer equipment and storage medium | |
| CN114691653A (en) | Account set migration method and device, computer equipment and storage medium | |
| WO2021139480A1 (en) | Gis service aggregation method and apparatus, and computer device and storage medium | |
| CN113379243A (en) | Service subsystem evaluation method and device based on central platform and computer equipment | |
| CN112818021A (en) | Data request processing method and device, computer equipment and storage medium | |
| CN109656948B (en) | Bitmap data processing method and device, computer equipment and storage medium | |
| CN112732819A (en) | ETL-based data processing method, device, equipment and storage medium | |
| CN112416785A (en) | Word cutting tool version difference testing method, device, equipment and storage medium | |
| CN112465461A (en) | Business object information changing method, system, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |