CN113626449A - Data storage method, data query method and related equipment - Google Patents
Data storage method, data query method and related equipment Download PDFInfo
- Publication number
- CN113626449A CN113626449A CN202110754600.7A CN202110754600A CN113626449A CN 113626449 A CN113626449 A CN 113626449A CN 202110754600 A CN202110754600 A CN 202110754600A CN 113626449 A CN113626449 A CN 113626449A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- type
- query
- input data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data storage method, a data query method and related equipment, wherein the data query method comprises the following steps: acquiring input data; storing the input data to a transactional database; and acquiring target data with a data type as a preset data type from input data stored in the transactional database, synchronizing the target data from the transactional database to the analytical database, and determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type required to be searched by the query request when the query request is received.
Description
Technical Field
The present application relates to the field of computer application technologies, and in particular, to a data storage method, a data query method, and a related device.
Background
The transactional database is a computer application system which takes transaction elements as units of data processing and is used for man-machine interaction. It can update or operate on the data instantly, and the data in the system is always kept in the latest state. In order to ensure the storage efficiency of data, transactional databases are generally used for data storage in the related art.
However, with the rapid development of computer technology, the data scale is rapidly increased, and although compared with other databases, the efficiency of storing data by using a transactional database can be improved when dealing with big data, the efficiency of querying data by using a transactional database is low, so that the pressure of querying the transactional database is increased by storing a large amount of data, and the efficiency and stability of querying are affected.
Disclosure of Invention
The present application aims to solve at least one of the technical problems in the prior art, and provides a data storage method, a data query method and related devices, so as to improve the efficiency of querying data.
In a first aspect, an embodiment of the present application provides a data storage method, including:
acquiring input data;
storing the input data to a transactional database;
and acquiring target data with a data type as a preset data type from input data stored in the transactional database, synchronizing the target data from the transactional database to the analytical database, and determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type required to be searched by the query request when the query request is received.
After the input data are stored in the transaction type database, the target data with the data type being the preset data type in the input data are synchronized to the analysis type database, so that the corresponding database can be selected for query according to the data type required to be searched by the query request during subsequent data query, query separation is realized, the query pressure of the database is shared, and the data query efficiency is improved.
In a second aspect, in an embodiment of the present application, there is further provided a data query method, including:
acquiring a query request sent by user equipment;
and determining a corresponding database from the transactional database and the analytic database to perform data query according to the data type required to be searched by the query request, wherein the transactional database stores input data acquired from a data source, the analytic database stores target data of which the data type is a preset data type in the input data, and the target data is obtained from the transactional database synchronously.
In a third aspect, in an embodiment of the present application, there is also provided a data storage device including:
the data acquisition module is used for acquiring input data;
the data storage module is used for storing the input data into a transactional database;
and the data synchronization module is used for acquiring target data with a data type as a preset data type from input data stored in the transactional database, synchronizing the target data from the transactional database to the analytical database, and determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type required to be searched by the query request when the query request is received.
In a fourth aspect, in an embodiment of the present application, there is further provided a data query apparatus, including: the request acquisition module is used for acquiring a query request sent by user equipment;
and the data query module is used for determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type to be searched by the query request, wherein the transactional database stores input data acquired from a data source, the analytical database stores target data of which the data type is a preset data type in the input data, and the target data is obtained from the transactional database synchronously.
In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the data storage method as described in the above embodiments when executing the program.
In a sixth aspect, an embodiment of the present application provides an electronic device, including: the data query method comprises the following steps of a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize the data query method according to the embodiment.
In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where computer-executable instructions are stored, and the computer-executable instructions are configured to cause a computer to execute the data storage method according to the foregoing embodiment.
In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, where the computer-executable instructions are configured to cause a computer to execute the data query method according to the foregoing embodiment.
Drawings
The present application is further described with reference to the following figures and examples;
FIG. 1 is a schematic illustration of an exemplary application environment of the present application;
FIG. 2 is a schematic flow chart diagram illustrating a data storage method according to one embodiment;
FIG. 3 is a flow diagram that illustrates a methodology for querying data in one embodiment;
FIG. 4 is a block diagram of a data storage device in one embodiment;
FIG. 5 is a block diagram showing the structure of a data query apparatus according to an embodiment;
FIG. 6 is a block diagram of a computer device in one embodiment.
Detailed Description
Reference will now be made in detail to the present embodiments of the present application, preferred embodiments of which are illustrated in the accompanying drawings, which are for the purpose of visually supplementing the description with figures and detailed description, so as to enable a person skilled in the art to visually and visually understand each and every feature and technical solution of the present application, but not to limit the scope of the present application.
The transactional database is a computer application system which takes transaction elements as units of data processing and is used for man-machine interaction. It can update or operate on the data instantly, and the data in the system is always kept in the latest state. In order to ensure the storage efficiency of data, transactional databases are generally used for data storage in the related art. However, with the rapid development of computer technology, the data scale is rapidly increased, and although compared with other databases, the efficiency of storing data by using a transactional database can be improved when dealing with big data, the efficiency of querying data by using a transactional database is low, so that the pressure of querying the transactional database is increased by storing a large amount of data, and the efficiency and stability of querying are affected.
For this reason, in the related art, data storage may be performed by replacing the transactional database with the analytical database to which the computation node is added. However, increasing the number of computing nodes to expand the storage capacity of the analytical database results in increased use cost. Meanwhile, a large amount of cold data may exist in the data file imported into the analytical database by the user, and the cold data not only occupies a local storage space, but also affects the calculation speed. While the transactional database is adopted for data storage, the storage efficiency can be improved when large data is handled, but when the data needs to be inquired from a large amount of data in the database, the inquiry efficiency of the transactional database is low.
In order to solve the above technical problem, in an embodiment, the present application provides the following technical solutions.
Fig. 1 is a schematic diagram of an exemplary application environment of the present application. The server 110 includes a transactional database 210 and an analytic database 220, and the transactional database 210 interfaces with an external data source. After the input data is obtained and stored by the external data source, the target data with the data type of the preset data type is obtained from the input data stored in the transaction database 210, and the target data is synchronized from the transaction database 210 to the analysis database 220. When the server receives the query request sent by the user equipment, according to the data type to be searched by the query request, the data query is performed on the corresponding database determined from the transaction database 210 and the analysis database 220.
After the input data are stored in the transaction database, the target data of which the data types are preset data types in the input data are synchronized to the analysis database, so that the input data can be quickly stored through the transaction database, the storage pressure of the analysis database is reduced, the storage space consumption of the analysis database is reduced, the stability and the calculation speed of the analysis database service are improved, meanwhile, the corresponding database can be selected for query according to the data types required to be searched by a query request during data query, query separation is realized, the query pressure of the database is shared, and the data query efficiency is improved.
The technical solution of the present application will be described and illustrated in detail by means of several specific examples.
As shown in FIG. 2, in one embodiment, a data storage method is provided. The embodiment is mainly illustrated by applying the method to computer equipment. The computer device may specifically be the server in fig. 1 described above.
Referring to fig. 2, the data storage method specifically includes the following steps:
and S11, acquiring input data.
In one embodiment, the server may obtain the input data from the data source through an external data interface, such as an API interface. Input data may also be obtained from a data source by simulating user operations. In order to save the acquisition cost of the input data and improve the reliability of the acquisition of the target data, for example, the input data may be acquired from the data source through the API interface in advance, and when it is detected that the API interface cannot acquire the input data from the data source, the input data generated by the data source according to the simulation operation is acquired by simulating the operation of the user on the browser or the mobile phone.
For example, if the data type of the input data is APP application data, the packet name of the input data, the application market for acquiring the APP application data, the ID in the application market, and the like may be combined as the input data label.
And S12, storing the input data into a transactional database.
In an embodiment, the server writes all input data into the transactional data storage, and can quickly store the input data acquired from the data source into the transactional database by utilizing the characteristics of more efficient writing and updating and more reliable storage of the transactional database.
S13, acquiring target data with a data type as a preset data type from input data stored in the transactional database, and synchronizing the target data from the transactional database to the analytical database, so that when a query request is received, according to the data type required to be searched by the query request, a corresponding database is determined from the transactional database and the analytical database for data query.
In an embodiment, the preset data type may be a data type corresponding to input data that may be frequently accessed and is preset according to a user requirement, or a data type of data in the input data, which is queried by the user more than a preset number of times, is used as the preset data type.
In one embodiment, the server may detect a data type of each input data from the input data stored in the transaction database when receiving the query request, and determine that the input data is data that the user needs to acquire frequently when the data type of the input data is consistent with a data type that is preset to be frequently accessed at the beginning of the data structure design, so as to determine the input data as target data, i.e., hot data, and synchronize the target data, i.e., the hot data, into the analysis database. Conversely, the input data is determined to be non-target data, i.e., cold data, without synchronization.
In an embodiment, the server may further detect, when receiving the query request, a number of times each input data is queried by the user from the input data stored in the transaction database, so as to determine whether the data type of the input data is the preset data type according to the number of times each input data is queried by the user. When the number of times that the input data is inquired by a user to be displayed on a display interface of user equipment exceeds a preset number of times, judging that the input data is the data which is frequently required to be acquired by the user, determining that the data type of the input data is the preset data type, further judging that the input data is the data which is frequently required to be acquired by the user, determining that the input data is the target data, and then synchronizing the target data into an analysis type database. Conversely, the input data is determined to be non-target data and no synchronization is performed. The preset times can be set according to actual requirements, for example, the preset times is preset to 5 times, and when the number of times that the input data is queried by the user is detected to be 6 times, the data type of the input data is determined to be the preset data type and is synchronized to the analysis database.
For example, the data synchronization process flow may be started by detecting whether there is unsynchronized target data in the transactional database when a predetermined synchronization time is reached, and starting the data synchronization process flow to synchronize the target data from the transactional database to the analytic database when the target data that is not synchronized is detected. The synchronization time can be set according to actual conditions, for example, 0.01S is used for one-time synchronization.
For example, the data synchronization process flow may be started, and whether unsynchronized target data exists in the transactional database may be detected in real time, and if yes, the data synchronization process flow is started immediately to synchronize the target data from the transactional database to the analytic database.
Because only the target data with higher user query rate in the input data is synchronized to the analysis database from the transaction database, and the data with lower user demand is stored in the transaction database, when the server receives a query request sent by user equipment, the server can determine the corresponding database from the transaction database and the analysis database according to the data type required to be searched by the query request to perform data query, and the analysis database and the transaction database can share query pressure, thereby improving query efficiency.
For example, when the type of data to be searched for in the query request is a preset type, that is, target data needs to be searched for, the query can be performed from the analytic database; when the data type required to be searched by the query request is a non-preset type, namely non-target data is required to be searched, the query can be carried out from the transaction database; and when the data types required to be searched by the query request comprise a preset type and a non-preset type, querying target data from the analysis type database and querying non-target data from the transaction type database.
For example, according to the type and the amount of data to be searched by the query request, the corresponding database is determined from the transactional database and the analytical database for data query. If the type of the data to be searched by the query request is a preset type and the data volume to be searched is large, if 7 target data are to be searched, the query can be performed from an analysis type database suitable for large-volume data query and statistics; if the type of the data to be searched in the query request is a preset type and the required quantity to be searched is small, for example, 1 target data needs to be searched, the data can be directly searched from a transactional database suitable for a small quantity of data query; when the data type required to be searched by the query request is a non-preset type, the data type can be queried from the transaction database.
After the input data are stored in the transaction type database, the target data of which the data types are preset data types in the input data are synchronized to the analysis type database, so that the input data can be quickly stored through the transaction type database, the storage pressure of the analysis type database and the storage space consumption of the analysis type database are reduced, the stability and the calculation speed of the service of the analysis type database are improved, meanwhile, the corresponding database can be selected for query according to the data types required to be searched by a query request in the follow-up data query, the query separation is realized, the query pressure of the database is shared, and the data query efficiency is improved.
To avoid the transactional database storing a large amount of duplicate data and wasting storage space, in an embodiment, for S12, the method includes:
searching historical data corresponding to the identification information of the input data in the transactional database according to the identification information of the input data; and when the historical data is not found, storing the input data in a transactional database. When the historical data is found, comparing the input data with the historical data, and acquiring difference data from the input data; storing the difference data to the transactional database.
In one embodiment, the identification information refers to a field capable of uniquely identifying the input data. For example, the input data is mobile communication data, each mobile communication data has a different telephone number field, the mobile communication data can be uniquely identified through the telephone number field, and the telephone number field is the identity of the mobile communication data. In another embodiment, the identification information refers to a plurality of fields that are capable of uniquely identifying the input data. For example, the input data is APP application data, and one APP application data can be distinguished from other APP application data by a packet name of an application, an application market of the APP application data, and an ID in the application market.
In one embodiment, the identification information of the input data is compared with the identification information of the historical data stored in the transactional database, and if the identification information of the historical data identical to the identification information of the input data does not exist, the input data is unique, so that the input data is stored in the transactional database in a full amount.
In one embodiment, if the transactional database has the identification information of the historical data which is the same as the identification information of the input data, the part of the input data which is different from the historical data is stored in the transactional database, so that the time for storing the part of the input data which is the same as the historical data in the transactional database is saved, the part of the input data which is the same as the historical data is prevented from occupying the storage space of the database, the time consumption of data updating can be shortened, and the storage space occupied by the data is reduced.
To further improve the efficiency of data storage and query, in an embodiment, after the input data is obtained, the method further includes: and acquiring input data with the data attribute being the temporary data attribute from the input data and storing the input data into a cache.
Considering that the cache is the fastest temporary data read-write scheme, in an embodiment, the cache is added to store temporary data, so that a three-layer storage structure is implemented, input data of temporary data attributes is stored in the cache, and query pressure of the analytical database can be shared by the cache when the data volume to be searched for by a query request is large or the attributes of queried data are temporary data attributes.
To further reduce the storage pressure, in an embodiment, after completing the storage of the input data, the method further includes: and according to a preset cleaning period, removing the input data with the data type of log type in the transaction type database and removing the target data with the data type of log type in the analysis type database. The preset cleaning period can be preset according to actual conditions, and is not described herein in detail.
On the basis of the above data storage method, as shown in fig. 3, in one embodiment, a data query method is provided. The embodiment is mainly illustrated by applying the method to computer equipment. The computer device may specifically be the server in fig. 1 described above.
Referring to fig. 3, the data query method specifically includes the following steps:
and S21, acquiring the query request sent by the user equipment.
And S22, determining a corresponding database from the transactional database and the analytic database to perform data query according to the data type to be searched by the query request, wherein the transactional database stores input data acquired from a data source, the analytic database stores target data of which the data type is a preset data type in the input data, and the target data is obtained from the transactional database synchronously.
In an embodiment, the target data may be synchronized by detecting, by the server, a data type of each input data from input data stored in the transaction database, and determining that the input data is data that the user needs to acquire frequently when the data type of the input data is consistent with a data type that is preset at the beginning of the data structure design and is likely to access frequently, so as to determine the input data as the target data, i.e., the hot data, and synchronize the input data with the analysis database. Conversely, the input data is determined to be non-target data, i.e., cold data, without synchronization.
In an embodiment, the target data may be synchronized by the server, and the number of times each input data is queried by the user is detected from the input data stored in the transaction database, so as to determine whether the data type of the input data is the preset data type according to the number of times each input data is queried by the user. When the number of times that the input data is inquired by a user to be displayed on a display interface of user equipment exceeds a preset number of times, judging that the input data is the data which is frequently required to be acquired by the user, determining that the data type of the input data is the preset data type, further judging that the input data is the data which is frequently required to be acquired by the user, determining that the input data is the target data, and then synchronizing the target data into an analysis type database. Conversely, the input data is determined to be non-target data and no synchronization is performed. The preset times can be set according to actual requirements, for example, the preset times is preset to 5 times, and when the number of times that the input data is queried by the user is detected to be 6 times, the data type of the input data is determined to be the preset data type and is synchronized to the analysis database.
After the input data are stored in the transaction database, the target data of which the data types are preset data types in the input data are synchronized to the analysis database, so that the input data can be quickly stored through the transaction database, the storage pressure of the analysis database is reduced, the storage space consumption of the analysis database is reduced, the stability and the calculation speed of the analysis database service are improved, meanwhile, the corresponding database can be selected for query according to the data types required to be searched by a query request during data query, query separation is realized, the query pressure of the database is shared, and the data query efficiency is improved.
In an embodiment, for S21, comprising: detecting the data type required to be searched by the query request, and inquiring target data stored in the analysis type database according to the query request when the data type required to be searched by the query request is detected to be a preset data type; and when detecting that the data type required to be searched by the query request is not a preset data type, querying the input data stored in the transaction type database.
In one embodiment, the query location of the data may be preset, for example, a preset type of input data may be preset to be queried through the analytic database, and a non-preset type of input data, i.e., non-target data, may be queried through the transactional database. And when the data type required to be searched by the query request is detected to be a preset data type, querying target data stored in the analytical database, otherwise, querying input data stored in the transactional database. When the data types required to be searched by the query request comprise a preset data type and a non-preset type, namely target data and non-target data are required to be simultaneously queried, querying the target data from the analytical database and querying the non-target data from the transactional database. Therefore, query separation is realized, so that the query pressure of the transaction database is shared, and the data query efficiency is improved.
In order to further improve the storage and query efficiency, in an embodiment, a cache is further added for storing and querying the target data. Specifically, the data attribute of the data queried according to the query request is detected, and when the data attribute is the temporary data attribute, the data queried according to the query request is stored in the cache, so that when the data queried subsequently is the data with the temporary data attribute, the data queried subsequently can be queried directly through the cache, query pressure is shared, and query efficiency is improved.
In one embodiment, as shown in fig. 4, there is provided a data storage device comprising:
a data obtaining module 101, configured to obtain input data.
A data storage module 102 for storing the input data to the transactional database.
The data synchronization module 103 is configured to acquire target data with a data type being a preset data type from input data stored in the transactional database, synchronize the target data from the transactional database to the analytic database, and determine a corresponding database from the transactional database and the analytic database to perform data query according to a data type to be searched by the query request when the query request is received.
In an embodiment, the data storage module 102 is specifically configured to: searching historical data corresponding to the identification information of the input data in the transactional database according to the identification information of the input data; and when the historical data is not found, storing the input data in a transactional database.
In one embodiment, the data storage module 102 is further configured to: when the historical data is found, comparing the input data with the historical data, and acquiring difference data from the input data; storing the difference data to the transactional database.
In an embodiment, the data synchronization module 103 is further configured to: after the input data is acquired, the input data with the data attribute being the temporary data attribute is acquired from the input data and stored in the cache.
In an embodiment, the data synchronization module 103 is further configured to: and according to a preset cleaning period, removing the input data with the data type of log type in the transaction type database and removing the target data with the data type of log type in the analysis type database.
In one embodiment, as shown in fig. 5, there is provided a data query apparatus including:
a request obtaining module 201, configured to obtain an inquiry request sent by a user equipment.
And the data query module 202 is configured to determine a corresponding database from the transactional database and the analytic database to perform data query according to a data type to be searched by the query request, where the transactional database stores input data acquired from a data source, the analytic database stores target data in the input data, where the data type of the target data is a preset data type, and the target data is obtained from the transactional database synchronously.
In an embodiment, the data query module 202 is specifically configured to: and detecting the data type required to be searched by the query request, and inquiring the target data stored in the analysis type database according to the query request when the data type required to be searched by the query request is detected to be a preset data type.
In one embodiment, the data query module 202 is further configured to: and when detecting that the data type required to be searched by the query request is not a preset data type, querying the input data stored in the transaction type database.
In one embodiment, the data query module 202 is further configured to: and detecting the data attribute of the data inquired according to the inquiry request, and storing the data inquired according to the inquiry request into a cache when the data attribute is a temporary data attribute.
In one embodiment, a computer apparatus is provided, as shown in fig. 6, which includes a processor, a memory, a network interface, an input device, and a display screen connected by a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program that, when executed by the processor, causes the processor to implement the data storage method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a data storage method. Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the data storage device provided herein may be implemented in the form of a computer program that is executable on a computer apparatus as shown in fig. 6. The memory of the computer device may store various program modules that make up the data storage device. The computer program constituted by the respective program modules causes the processor to execute the steps in the data storage method or the data query method of the embodiments of the present application described in the present specification.
In one embodiment, there is provided an electronic device including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to perform the steps of the data storage method described above. The steps of the data storage method herein may be steps in the data storage method or the data query method of the various embodiments described above.
In one embodiment, a computer-readable storage medium is provided, which stores computer-executable instructions for causing a computer to perform the steps of the above-described data storage method. Here, the steps of the data storage method or the data query method may be steps in the data storage method or the data query method of the above embodiments.
The foregoing is a preferred embodiment of the present application, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations are also regarded as the protection scope of the present application.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Claims (12)
1. A method of storing data, comprising:
acquiring input data;
storing the input data to a transactional database;
and acquiring target data with a data type as a preset data type from input data stored in the transactional database, synchronizing the target data from the transactional database to the analytical database, and determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type required to be searched by the query request when the query request is received.
2. The data storage method of claim 1, wherein storing the input data to a transactional database comprises:
searching historical data corresponding to the identification information of the input data in the transactional database according to the identification information of the input data;
and when the historical data is not found, storing the input data in a transactional database.
3. The data storage method of claim 2, further comprising:
when the historical data is found, comparing the input data with the historical data, and acquiring difference data from the input data;
storing the difference data to the transactional database.
4. The data storage method of claim 1, after obtaining the input data, further comprising:
and acquiring input data with the data attribute being the temporary data attribute from the input data and storing the input data into a cache.
5. The data storage method of any of claims 1-4, further comprising:
and according to a preset cleaning period, removing the input data with the data type of log type in the transaction type database and removing the target data with the data type of log type in the analysis type database.
6. A method for querying data, comprising:
acquiring a query request sent by user equipment;
and determining a corresponding database from the transactional database and the analytic database to perform data query according to the data type required to be searched by the query request, wherein the transactional database stores input data acquired from a data source, the analytic database stores target data of which the data type is a preset data type in the input data, and the target data is obtained from the transactional database synchronously.
7. The data query method of claim 6, wherein determining a corresponding database from the transactional database and the analytic database for performing data query according to the data type to be searched by the query request comprises:
and detecting the data type required to be searched by the query request, and inquiring the target data stored in the analysis type database according to the query request when the data type required to be searched by the query request is detected to be a preset data type.
8. The data query method of claim 7, further comprising:
and when detecting that the data type required to be searched by the query request is not a preset data type, querying the input data stored in the transaction type database.
9. The data query method of claim 7, further comprising:
and detecting the data attribute of the data inquired according to the inquiry request, and storing the data inquired according to the inquiry request into a cache when the data attribute is a temporary data attribute.
10. A data storage device, comprising:
the data acquisition module is used for acquiring input data;
the data storage module is used for storing the input data into a transactional database;
and the data synchronization module is used for acquiring target data with a data type as a preset data type from input data stored in the transactional database, synchronizing the target data from the transactional database to the analytical database, and determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type required to be searched by the query request when the query request is received.
11. A data query apparatus, comprising:
the request acquisition module is used for acquiring a query request sent by user equipment;
and the data query module is used for determining a corresponding database from the transactional database and the analytical database to perform data query according to the data type to be searched by the query request, wherein the transactional database stores input data acquired from a data source, the analytical database stores target data of which the data type is a preset data type in the input data, and the target data is obtained from the transactional database synchronously.
12. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the data storage method according to any of claims 1 to 5 or the data storage method according to any of claims 6 to 9 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110754600.7A CN113626449A (en) | 2021-07-02 | 2021-07-02 | Data storage method, data query method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110754600.7A CN113626449A (en) | 2021-07-02 | 2021-07-02 | Data storage method, data query method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113626449A true CN113626449A (en) | 2021-11-09 |
Family
ID=78379201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110754600.7A Pending CN113626449A (en) | 2021-07-02 | 2021-07-02 | Data storage method, data query method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113626449A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115827660A (en) * | 2023-02-14 | 2023-03-21 | 本原数据(北京)信息技术有限公司 | Data updating method and device, electronic equipment and nonvolatile storage medium |
| CN116861455A (en) * | 2023-06-25 | 2023-10-10 | 上海数禾信息科技有限公司 | Event data processing method, system, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104537030A (en) * | 2014-12-22 | 2015-04-22 | 华为技术有限公司 | Business data query method and device and database system |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN110874290A (en) * | 2019-10-09 | 2020-03-10 | 上海交通大学 | Transaction analysis hybrid processing method of distributed memory database and database |
-
2021
- 2021-07-02 CN CN202110754600.7A patent/CN113626449A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104537030A (en) * | 2014-12-22 | 2015-04-22 | 华为技术有限公司 | Business data query method and device and database system |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN110874290A (en) * | 2019-10-09 | 2020-03-10 | 上海交通大学 | Transaction analysis hybrid processing method of distributed memory database and database |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115827660A (en) * | 2023-02-14 | 2023-03-21 | 本原数据(北京)信息技术有限公司 | Data updating method and device, electronic equipment and nonvolatile storage medium |
| CN115827660B (en) * | 2023-02-14 | 2023-05-12 | 本原数据(北京)信息技术有限公司 | Data updating method and device, electronic equipment and nonvolatile storage medium |
| CN116861455A (en) * | 2023-06-25 | 2023-10-10 | 上海数禾信息科技有限公司 | Event data processing method, system, electronic device and storage medium |
| CN116861455B (en) * | 2023-06-25 | 2024-04-26 | 上海数禾信息科技有限公司 | Event data processing method, system, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10606806B2 (en) | Method and apparatus for storing time series data | |
| CN107872528B (en) | Message pushing method and device | |
| CN107291869B (en) | Distributed service system and data query method thereof | |
| CN109587271B (en) | Main chain parallel chain architecture system, block synchronization method, equipment and storage medium | |
| CN111339171B (en) | Data query method, device and equipment | |
| CN106407360B (en) | Data processing method and device | |
| CN113626449A (en) | Data storage method, data query method and related equipment | |
| CN106997431B (en) | Data processing method and device | |
| US11243921B2 (en) | Database expansion system, equipment, and method of expanding database | |
| CN104572845A (en) | File distribution method and device, equipment and system | |
| CN109783258A (en) | A kind of message treatment method, device and server | |
| CN110012049B (en) | Information push method, system, server and computer readable storage medium | |
| EP2622544A1 (en) | Method and arrangement for processing data | |
| CN104111957A (en) | Method and system for synchronizing distributed transaction | |
| CN111737564A (en) | Information query method, device, equipment and medium | |
| CN115438087B (en) | Data query method, device, storage medium and equipment based on cache library | |
| CN105939355A (en) | Data access method and system, as well as client and server | |
| CN112115200A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| CN111046106A (en) | Cache data synchronization method, device, equipment and medium | |
| US20140108367A1 (en) | Client apparatus and database server for resumable transaction and method thereof | |
| CN112699187A (en) | Associated data processing method, device, equipment, medium and product | |
| CN110874365B (en) | Information query method and related equipment thereof | |
| CN109189864B (en) | Method, device and equipment for determining data synchronization delay | |
| CN105610596B (en) | Resource directory management method and network terminal | |
| CN103152381A (en) | Method and server system of processing browser corrupted data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |