CN113064919A

CN113064919A - Data processing method, data storage system, computer device and storage medium

Info

Publication number: CN113064919A
Application number: CN202110349252.5A
Authority: CN
Inventors: 夏宝宁
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2021-07-02
Anticipated expiration: 2041-03-31
Also published as: CN113064919B

Abstract

The disclosure relates to a data processing method, a data storage system, computer equipment and a storage medium, and belongs to the technical field of computers. According to the data storage system, the row storage space and the column storage space are simultaneously arranged in the data storage system, the storage form of each field is configured by applying configuration information, the field with a large data writing requirement is directly written into the row storage space in the row storage form, so that the data writing speed can be ensured to be high, then the data synchronization function is called, the updated data in the row storage space is synchronized into the column storage space, the column storage space can store complete and latest data, so that the data query can be well supported, and the data query speed is ensured to be high.

Description

Data processing method, data storage system, computer device and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, a data storage system, a computer device, and a storage medium.

Background

With the development of internet technology, more and more data are required to be stored in a network, and the performance requirements of mass data storage on a data storage system are greatly improved. However, the existing data storage system is often difficult to have a fast data writing speed and a fast data query speed at the same time, and is difficult to meet the high-frequency data writing and query requirements. Therefore, how to make the data storage system compatible with the data query speed and the data write speed is an important research direction.

Disclosure of Invention

The present disclosure provides a data processing method, a data storage system, a computer device, and a storage medium, which can improve a data query speed and a data write speed of the data storage system. The technical scheme of the disclosure is as follows:

according to a first aspect of the embodiments of the present disclosure, there is provided a data processing method applied to a data storage system, including:

receiving a data writing request, wherein the data writing request comprises data to be written and a first field identifier of a field to which the data to be written belongs;

responding to the first field identification hit configuration information in the data writing request, writing the data to be written into a row storage space of the data storage system in a row storage form, wherein the configuration information stores at least one target field identification corresponding to a target field, and writing parameters and query parameters corresponding to the target field meet reference conditions;

and synchronizing the incremental data in the row storage space to a column storage space of the data storage system, wherein the data in the column storage space is stored in a column storage form.

According to the technical scheme provided by the embodiment of the disclosure, the row storage space and the column storage space are simultaneously arranged in the data storage system, the storage form of each field is configured by applying the configuration information, and for the field with a high data writing requirement, the field is directly written into the row storage space in the row storage form, so that the data writing speed can be ensured to be high, then the data synchronization function is called, the updated data in the row storage space is synchronized into the column storage space, and the column storage space can store complete and latest data, so that the data query can be well supported, and the data query speed is ensured to be high.

In one possible implementation, after synchronizing the incremental data in the row storage space to the column storage space of the data storage system, the method further includes:

acquiring historical operation data of the data storage system, wherein the historical operation data stores field identification of a field to which the operated data belongs and corresponding data operation;

determining write parameters and query parameters corresponding to fields in the data storage system based on the historical operating data;

determining a candidate field in the field based on the writing parameters and the query parameters corresponding to the field, wherein the writing parameters and the query parameters corresponding to the candidate field meet the reference condition;

updating the configuration information based on the field identification of the candidate field.

In the embodiment of the disclosure, the configuration information is dynamically adjusted based on the real historical operation data, that is, the field stored in the line storage space is dynamically adjusted, so that the flexibility of configuring the data storage system can be effectively improved, the data storage system can better meet the current business requirements, and the field configuration problem of the storage space under different writing and query proportions can be effectively solved.

In one possible implementation, the reference condition is that the ratio between the write parameter and the query parameter is greater than or equal to a reference threshold.

In a possible implementation manner, the updating the configuration information based on the field identification of the candidate field includes:

matching the field identification of the candidate field with the target field identification stored in the configuration information;

in response to the field identification of the candidate field being different from the target field identification, adding the field identification of the candidate field to the configuration information.

In the embodiment of the disclosure, subsequent configuration information adjustment is directly performed based on real log data, and a field with a high writing frequency in a current service scene can be accurately screened out, so that the adjusted configuration information is ensured to better meet the requirements of the current service.

deleting the target field identification stored in the configuration information;

adding the field identification of the candidate field to the configuration information.

In the embodiment of the present disclosure, the content in the configuration information is completely replaced based on the current read-write condition of each field, so that it is ensured that the latest configuration information more conforms to the current service requirement.

In a possible implementation manner, after the updating the configuration information based on the field identification of the candidate field, the method further includes:

and adjusting the data stored in the row of storage space based on the updated configuration information.

In the embodiment of the disclosure, the data stored in the line storage space is adjusted in time based on the new configuration information, so that the data storage system can adapt to new service requirements in time.

In one possible implementation, the column of storage space stores a full amount of data;

the adjusting the data stored in the row of memory space based on the updated configuration information includes:

reading the configuration information based on a reference period;

comparing the first configuration information read in the period with the second configuration information read in the previous period;

responding to the first configuration information and the second configuration information, wherein the first configuration information has a new field mark, reading data indicated by the new field mark from the row of storage space, and adding the read data to the row of storage space;

in response to the first configuration information having a truncated field identification compared to the second configuration information, deleting data indicated by the truncated field identification from the row of memory space.

In the embodiment of the disclosure, as the line storage space stores non-full data, the data processing pressure in the data synchronization process can be effectively reduced.

In one possible implementation, after receiving the data write request, the method further includes:

and writing the data to be written into a column storage space of the data storage system in a column storage form in response to the first field identification missing configuration information in the data write request.

In the embodiment of the application, data with extremely low writing frequency is directly written into the column storage space, and the column storage space stores the full data, so that the data integrity in the data storage system is ensured, and good support is provided for functions such as data query.

receiving a data query request, wherein the data query request comprises a second field identifier of a field to which data to be queried belongs;

responding to the second field identification missing configuration information in the data query request, and performing data query in the column of storage space;

a data query is performed in the row of memory space in response to the second field in the data query request identifying hit configuration information.

In the embodiment of the application, the column storage space is used for storing the full amount of data, so that support is provided for the data query function, and the higher efficiency of data query can be ensured; for the line storage space, the stored non-full data is small, that is, the stored data amount is small, so that when the line storage space is subjected to data query, the redundancy of data reading can be reduced, and the data query speed is improved.

In one possible implementation, the synchronizing the incremental data in the row storage space to the column storage space of the data storage system includes:

in response to detecting the incremental data in the row of memory space, and the incremental data belongs to the reference field, synchronizing the incremental data to the column of memory space;

in response to detecting the incremental data in the row of memory space that does not belong to the reference field, synchronizing the incremental data to the column of memory space upon arrival of a next data synchronization cycle.

In the embodiment of the disclosure, data of different fields are processed in a distinguishing manner, and data synchronization is performed at different computers, so that the data processing amount of the computer equipment in one data synchronization process can be effectively reduced, and the data processing pressure of the computer equipment is reduced.

According to a second aspect of embodiments of the present disclosure, there is provided a data storage system, comprising a storage entry module, a write engine, and a synchronization engine;

the storage entry module is used for receiving a data writing request, wherein the data writing request comprises data to be written and a first field identifier of a field to which the data to be written belongs;

the write-in engine is configured to, in response to hit configuration information of the first field identifier in the data write-in request, write the data to be written in a row storage space of the data storage system in a row storage form, where at least one target field identifier corresponding to a target field is stored in the configuration information, and write-in parameters and query parameters corresponding to the target field satisfy reference conditions;

the synchronization engine is used for synchronizing the incremental data in the row storage space to a column storage space of the data storage system, and the data in the column storage space is stored in a column storage form.

In one possible implementation, the synchronization engine is further configured to:

In one possible implementation, the synchronization engine is to:

the synchronization engine is used for reading the configuration information based on a reference period; comparing the first configuration information read in the period with the second configuration information read in the previous period; responding to the first configuration information and the second configuration information, wherein the first configuration information has a new field mark, reading data indicated by the new field mark from the row of storage space, and adding the read data to the row of storage space; in response to the first configuration information having a truncated field identification compared to the second configuration information, deleting data indicated by the truncated field identification from the row of memory space.

In one possible implementation, the write engine is further configured to:

In a possible implementation manner, the storage entry module is further configured to receive a data query request, where the data query request includes a second field identifier of a field to which data to be queried belongs;

the system also includes a read engine for performing a data query in the column of storage space in response to the second field identification in the data query request missing configuration information; a data query is performed in the row of memory space in response to the second field in the data query request identifying hit configuration information.

In one possible implementation, the synchronization engine is to:

According to a third aspect of embodiments of the present disclosure, there is provided a computer apparatus comprising:

one or more processors;

a memory for storing the processor executable program code;

wherein the processor is configured to execute the program code to implement the data processing method described above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium including: the program code in the computer readable storage medium, when executed by a processor of a computer device, enables the computer to readable perform the data processing method described above.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the data processing method described above.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.

FIG. 1 is a schematic diagram illustrating a data storage system in accordance with an exemplary embodiment;

FIG. 2 is a flow diagram illustrating a method of data processing in accordance with an exemplary embodiment;

FIG. 3 is a flow chart illustrating a method of data processing according to an exemplary embodiment;

FIG. 4 is a diagram illustrating the storage of data in a different storage space in accordance with an exemplary embodiment;

FIG. 5 is a flow diagram illustrating a method of data querying in a data storage system in accordance with an exemplary embodiment;

FIG. 6 is a flow chart illustrating a method of configuration information adjustment according to an example embodiment;

FIG. 7 is a data interaction diagram of a data storage system shown in accordance with an exemplary embodiment;

FIG. 8 is a block diagram illustrating a data processing apparatus in accordance with an exemplary embodiment;

FIG. 9 is a block diagram illustrating a terminal architecture according to an exemplary embodiment;

FIG. 10 is a block diagram illustrating a server in accordance with an example embodiment.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should be noted that the data referred to in the present disclosure is data authorized by a user or data sufficiently authorized by each party.

In order to facilitate understanding of technical processes of the embodiments of the present disclosure, some terms related to the embodiments of the present disclosure are explained below:

and (3) line storage: the data belonging to the same row are assembled together, the data of each row are respectively written into the storage space, the speed of the row storage mode is high when the data are written, but redundant data are generated in the data reading process, the data addressing time is long, and the data reading speed is low.

Column store: the data is split according to columns, a column of data and an ID corresponding to each data in the column are assembled together, the data in each column are written into a storage space respectively, the column storage form can be quickly positioned to a storage position of a certain column of data during data reading, the data reading speed is high, but the data needs to be split during data writing, and the data writing speed is low.

FIG. 1 is a schematic diagram illustrating a data storage system, which may include a storage entry module 101, a write engine 102, a configuration module 103, and a synchronization engine 104, as shown in FIG. 1, according to an example embodiment. In one possible implementation, the data storage system may be deployed on any device, for example, the data storage system is deployed on a backend server corresponding to a target application, and the data storage system stores running data, user data, and the like of the target application.

For example, when the storage entry module 101 receives a data write request, the storage entry module 101 may forward the data write request to the write engine 102, and the data write engine 102 performs a subsequent data write operation.

The configuration module 103 stores therein configuration information indicating storage forms of respective fields in the data storage system, the storage forms including a row storage form and a column storage form. In the embodiment of the present disclosure, the data storage system further includes two types of storage spaces, namely a row storage space 105 and a column storage space 106, where data is stored in the row storage space 105 in a row storage form and data is stored in the column storage space 106 in a column storage form. In a possible implementation manner, before forwarding the data operation request, the storage entry 101 may further read the configuration information in the configuration module 103, obtain a storage form corresponding to the data field of the requested operation, and send the data operation request and the obtained storage form to the write engine 102, so that the write engine 102 writes the data into a correct storage space.

The synchronization engine 104 is used to synchronize data between the row memory space 105 and the column memory space 106. In a possible implementation manner, the synchronization engine 104 may further obtain historical operation data of the data operating system, perform data analysis on the historical operation data, and obtain historical operation condition information of each field in the data storage system, so as to adjust a storage form of each field based on the historical operation condition information corresponding to each field.

In one possible implementation, the data storage system further includes a read engine 107, the read engine 107 is configured to provide a data query function, for example, the storage entry module 101 forwards the received data query request to the read engine 107, and the read engine 107 reads data from the row storage space 105 or the column storage space 106.

It should be noted that the above description of the data storage system is only an exemplary illustration, and the data storage system may further include other functional modules, and the structure of the data storage system is not limited in the embodiment of the present disclosure.

Fig. 2 is a flowchart illustrating a data processing method applied to the data storage system described above, as shown in fig. 1, according to an exemplary embodiment, which may be executed by a computer device, and in one possible implementation, the embodiment includes the following steps.

In step 201, a computer device receives a data write request, where the data write request includes data to be written and a first field identifier of a field to which the data to be written belongs.

In one possible implementation, the computer device may be a terminal or a server, and the computer device is deployed with a data storage system, and the data storage system stores data of a plurality of fields. An application or user having write permission to the data storage system can send a write request to the data storage system to write new data in the data storage system.

In step 202, in response to that the first field identifier in the data write request hits configuration information, the computer device writes data to be written into a line storage space of the data storage system in a line storage form, where at least one target field identifier corresponding to a target field is stored in the configuration information, and write parameters and query parameters corresponding to the target field satisfy reference conditions.

The writing parameter and the query parameter may be writing frequency and query frequency, or writing frequency and query frequency, respectively, which are not limited in the embodiments of the present disclosure; the reference condition may be set by a developer, for example, the reference condition is set to be that a ratio of the write parameter to the query parameter is greater than or equal to a reference threshold, and the like, which is not limited by the embodiment of the disclosure. In the embodiments of the present disclosure, the line storage space refers to a space for storing data in a line storage form in a data storage system, and it should be noted that, in some embodiments, data in the line storage space can also be stored in other forms, and a data storage form corresponding to the line storage space can ensure a faster writing speed when data is written in the line storage space.

In the embodiment of the disclosure, the computer device can distinguish the storage forms of the data by using the configuration information, and write the data into the field with higher writing frequency in the form of line storage, so that the data writing speed can be improved.

In step 203, the computer device synchronizes the incremental data in the row storage space to a column storage space of the data storage system, wherein the data in the column storage space is stored in a column storage form.

In the embodiment of the present disclosure, the column storage space refers to a space for storing data in a column storage manner in a data storage system, and it should be noted that, in some embodiments, data in the column storage space may also be stored in other forms, and a data storage form corresponding to the column storage space is capable of ensuring a faster reading speed when data is read in the column storage space. In one possible implementation, the computer device can acquire the incremental data in the row storage space according to a certain period, and synchronize the incremental data to the column storage space to ensure that the column storage space can store complete and up-to-date data so as to provide support for the data query function.

The above embodiments are merely brief descriptions of embodiments of the present disclosure, fig. 3 is a flowchart illustrating a data processing method according to an exemplary embodiment, which may be applied to the data storage system shown in fig. 1, and the scheme provided by the present disclosure is further explained below with reference to fig. 1 and 3, and referring to fig. 3, the method includes the following steps.

In step 301, a computer device receives a data write request.

The data writing request comprises data to be written and a first field identification of a field to which the data to be written belongs.

In the embodiment of the present disclosure, a data storage system as shown in fig. 1 is deployed on the computer device, and the data storage system may be a background database corresponding to a target application program, and is used for storing running data, user data, and the like of the target application program. For example, the computer device is a backend server of the target application, and a backend database of the target application, that is, the data storage system, is deployed on the backend server. The target application may be a social application, a video application, or the like, and the type of the target application is not limited in the embodiments of the present disclosure.

In one possible implementation, the user can send the data write request through the target application, and the computer device receives the data write request through a storage entry module in the data storage system. Optionally, different users in the target application program correspond to different user permissions, in a possible implementation manner, the data write request further carries a user identifier, the user indicated by the user identifier is a request sending user, after receiving the data write request, the computer device performs permission verification based on the user identifier, and determines whether the user indicated by the user identifier has a write permission for the data storage system. In response to the user having write permission to the data storage system, the computer device continuing to perform subsequent data writing steps; in response to the user not having write permission to the data storage system, the computer device may send a first prompt to the target application indicating that the user is not currently authorized to write data. It should be noted that, the method for verifying the user right of the computer device is not limited in the embodiment of the present disclosure.

In one possible implementation manner, in response to receiving the data write request, the storage entry module in the data storage system may further read configuration information from the configuration module, where the configuration information is used to indicate a storage form corresponding to the data belonging to each field. Illustratively, the configuration information stores a target field identifier corresponding to at least one target field, the write parameter and the query parameter corresponding to the target field satisfy a reference condition, and data belonging to the target field may be stored in the same storage space of the data storage system in the same storage form. For example, the reference condition may be set to be that a ratio of the number of writing times to the number of querying times is greater than or equal to a reference threshold, that is, the writing frequency corresponding to the target field is much greater than the querying frequency, in this case, the data belonging to the target field may be stored in a line storage space of the data storage system in a line storage form, so as to ensure a fast writing speed. Of course, the reference condition may also be set to other contents, and the computer device may also store the data belonging to the target field in other storage forms based on the configuration information, which is not limited in this embodiment of the disclosure, and in this embodiment of the disclosure, only the data belonging to the reference field is stored in a line storage form as an example for explanation. In the embodiment of the present disclosure, the computer device may continue to perform step 302 or step 303 described below based on the hit condition of the first field identification carried by the data write request to the configuration information.

In step 302, the computer device writes the data to be written into a line memory space of the data storage system in a line memory form in response to the first field identification in the data write request hitting the configuration information.

In the embodiment of the present disclosure, in response to that the first field identifier hits the configuration information, the computer device determines that the field to which the data to be written belongs is a field with a higher writing frequency, and the computer device stores the data to be written into a line storage space, thereby ensuring that the complexity of the data writing process is low and the time consumption is short.

In a possible implementation manner, the step of obtaining the configuration information and matching the first field identifier with the configuration information is performed by a storage entry module in the data storage system, the storage entry module determines a storage form of the data to be written, the first field identifier and the storage form corresponding to the data to be written are sent to a write-in engine, and the write-in engine performs a subsequent data writing step. In one possible implementation manner, in response to that the storage form is a row storage form, the write engine determines a target write position of the data to be written in a row storage space, and writes the data to be written in the target write position. Illustratively, the data write request includes an ID (Identity Document) corresponding to the data to be written, the data is stored in association with the corresponding ID in the storage space, and the write engine may locate the target write location in the line storage space based on the ID corresponding to the data to be written and the first field identification. Fig. 4 is a schematic diagram illustrating a data storage form in different storage spaces according to an exemplary embodiment, referring to fig. 4, an original data table is shown as 401, the data table 401 includes data of ID, name, age, gender, and location fields, taking an example that a row storage space 402 stores data of name and age fields, and a column storage space 403 stores full data, a write engine receives data to be written AAA, and if the data to be written belongs to the name field and corresponds to ID3, the write engine may determine a target write location of the data to be written AAA to be 404, and writes AAA to a location 404 in the row storage space.

In the embodiment of the application, the line storage space stores non-full data, the fields stored in the line storage space are configured through the configuration information, and only the data corresponding to the field with the high writing frequency is stored in the line storage space.

In step 303, the computer device writes the data to be written in a column storage form into a column storage space of the data storage system in response to the first field in the data write request identifying missing configuration information.

As shown in fig. 4, the column memory space 403 stores data of all fields in the data table 401, and stores data of the same column and an ID corresponding to each data together, as shown by data in 405.

The method for writing the data to be written into the column memory space in step 303 is the same as the process for writing the data to be written into the row memory space in step 302, and is not described herein again.

In step 304, the computer device synchronizes the delta data in the row memory space to the column memory space of the data storage system.

The incremental data refers to updated data, newly added data, and the like in the row memory space.

In some embodiments, the computer device may perform data synchronization based on a data synchronization period, and when a data synchronization period is reached, acquire incremental data in the row storage space in the period, and synchronize the acquired incremental data to the column storage space. The duration of the data synchronization period is set by a developer, which is not limited in the embodiment of the disclosure. In the embodiment of the application, the integrity and the effectiveness of the data stored in the column storage space can be ensured through the periodic data synchronization, so that a user can conveniently perform data query on the data storage system.

In some embodiments, the computer device may perform data synchronization upon detecting that there is incremental data in the line memory space, i.e., after the computer device performs a write operation to the line memory space. In the embodiment of the application, after new data is written in the row storage space, the data is immediately synchronized to the column storage space, so that the consistency of the data in the two storage spaces can be effectively maintained.

In some embodiments, the computer device may determine whether to immediately data synchronize the delta data based on a field to which the delta data belongs when detecting that the row storage space has the delta data. Illustratively, the computer device synchronizes the incremental data to the column of memory space in response to detecting the incremental data in the row of memory space and the incremental data belonging to the reference field; in response to detecting the incremental data in the row of memory space that does not belong to the reference field, synchronizing the incremental data to the column of memory space upon arrival of a next data synchronization cycle. The reference field is set by a developer, for example, the reference field is a field with a high requirement on data accuracy and a high data query frequency, and the like, which is not limited in the embodiment of the present disclosure. In the embodiment of the disclosure, data of different fields are processed in a distinguishing manner, and data synchronization is performed at different computers, so that the data processing amount of the computer equipment in one data synchronization process can be effectively reduced, and the data processing pressure of the computer equipment is reduced.

In the embodiment of the present disclosure, the data synchronization process may be performed by a synchronization engine in the data storage system, and optionally, a data synchronization period is set in the synchronization engine, and the step of incremental data synchronization is performed based on the data synchronization period, or the synchronization engine may be capable of monitoring a data write operation of a write engine to the line storage space and performing the step of incremental data synchronization in response to detecting the data write operation. It should be noted that the above description of the method for performing data synchronization between the row memory space and the column memory space is only an exemplary illustration of one possible implementation manner, and the embodiment of the present disclosure does not limit which method is specifically used for performing data synchronization. In the embodiment of the disclosure, since the line storage space stores non-full amount of data and the data amount is small, the data processing pressure of the synchronization engine is also small in the data synchronization process, and the synchronization engine can be ensured to operate well in the data synchronization process.

It should be noted that, in this embodiment of the application, the data write request may include multiple data to be written, each data to be written corresponds to one field identifier, and the computer device may perform the above steps 302 to 303 based on each data to be written and the corresponding field identifier, so as to write each data to be written into the data storage system. It should be noted that, in the embodiment of the present application, the number of data to be written included in the data write request is not limited.

The above embodiments describe the process of data writing and data synchronization in the data storage system, and in the embodiments of the present disclosure, data query may also be performed in the data storage system. FIG. 5 is a flowchart illustrating a method of data querying in a data storage system according to an exemplary embodiment, which includes the following steps in one possible implementation, referring to FIG. 5.

In step 501, a computer device receives a data query request.

The data query request includes the second field identifier of the field to which the data to be queried belongs, and optionally, the data query request may further include an ID corresponding to the data to be queried.

In one possible implementation, a user can send a data query request through a target application, which is received by a computer device through a storage entry module of a data storage system. Optionally, before performing the data query, the computer device may further verify whether the user has a query right to the data storage system. Illustratively, the data query request further carries a user identifier of a user requesting to send, and after receiving the data write request, the computer device performs permission verification based on the user identifier. In response to the permission verification being passed, i.e. it is determined that the user has the inquiry permission for the data storage system, the computer device continues to perform subsequent data inquiry steps; in response to the permission verification failing, i.e., it is determined that the user does not have permission to query the data storage system, the computer device may send a second prompt to the target application indicating that the user is not currently authorized to query for data. It should be noted that, the method for verifying the user right of the computer device is not limited in the embodiment of the present disclosure.

In a possible implementation manner, in response to receiving the data query request, the storage entry module in the data storage system may further read the configuration information from the configuration module, and determine a storage form of the data to be queried based on the configuration information, that is, determine a storage location of the data to be queried, so as to perform data query in the corresponding storage location. It should be noted that, in the embodiment of the present disclosure, only the target field identifier of at least one target field is stored in the configuration information, and the data of the target field is the data written into the line storage space.

In step 502, the computer device performs a data query in a column storage space of the data storage system in response to the second field in the data query request identifying the miss configuration information.

In a possible implementation manner, in response to that the second field identifier does not match any target field identifier in the configuration information, it may be determined that the data indicated by the second field identifier does not need to be written into the row storage space and is directly stored in the column storage space, and the computer device may directly perform data query from the column storage space. In a possible implementation manner, the step of obtaining the configuration information and matching the second field identifier with the configuration information is performed by a storage entry module in the data storage system, the storage entry module determines a storage form of the data to be queried, the second field identifier and the storage form corresponding to the data to be queried may be sent to a reading engine, and the reading engine performs a subsequent data reading step.

In step 503, the computer device performs a data query in a row storage space of the data storage system in response to the second field identification in the data query request hitting the configuration information.

In one possible implementation, in response to that the second field identification is the same as any target field identification in the configuration information, it may be determined that the data indicated by the second field identification needs to be written into the line memory space, and the computer device may perform a data query from the line memory space to query the latest data. In the embodiment of the present application, frequently updated data is stored in the line storage space, and if a field to which the data to be queried belongs is included in the line storage space, data query is directly performed from the line storage space, so that latest and accurate data can be obtained.

In step 504, the computer device generates a data query result.

In one possible implementation, the read engine generates query results based on data retrieved from the respective storage spaces. For example, in a case that the data query request may request to query data of a plurality of fields, in response to a hit of the field identifier of any one of the plurality of fields on the configuration information, the read engine queries data corresponding to the any one of the plurality of fields from the line storage space to obtain a first intermediate result; and for other fields except any field, the reading engine inquires data corresponding to other fields from the column storage space to obtain a second intermediate result. The reading engine may combine the first intermediate result and the second intermediate result to obtain a query result. The reading engine sends the query result to the storage entrance module, and the storage entrance module sends the query result to the target application program.

In the embodiment of the present disclosure, the computer device may dynamically adjust the configuration information based on a required amount of writing and querying each field in a current business scenario, that is, dynamically adjust the fields stored in the line storage space, so that the data storage system can meet the current business demand, fig. 6 is a flowchart of a configuration information adjustment method according to an exemplary embodiment, and referring to fig. 6, in a possible implementation manner, the embodiment includes the following steps.

In step 601, a computer device obtains historical operating data for a data storage system.

The historical operation data stores field identification of fields to which the operated data belongs and corresponding data operation, the data operation comprises number query operation, data write-in operation and the like, and the historical operation data can reflect the reading and writing conditions of users on the fields. Illustratively, the historical operation data is a historical data operation log of the database storage system, and the like.

In a possible implementation manner, the computer device may obtain, by a synchronization engine in the data storage system, historical operation data in a target time period according to a target period, where the target period and the target time period may both be set by a developer, for example, the target time period may be set to a time period corresponding to a latest target period, or time periods corresponding to latest two periods, which is not limited in this disclosure. In the embodiment of the present disclosure, subsequent configuration information adjustment is directly performed based on real log data, so that it can be ensured that the adjusted configuration information better meets the requirements of the current service.

In step 602, the computer device determines write parameters and query parameters corresponding to the fields in the data storage system based on the historical operating data.

In a possible implementation manner, the computer device may collect log data including the same field identifier and the same data operation through the synchronization engine, thereby counting the number of different write operations and the number of query operations performed on each field, or counting the frequency of write operations and the frequency of query operations performed on each field, and determine the write parameter and the query parameter based on the number of write operations and the number of query operations, or the write frequency and the query frequency. It should be noted that, the embodiments of the present disclosure do not limit the method for determining the write parameter and the query parameter by the synchronization engine.

In step 603, the computer device determines candidate fields in each field based on the write parameters and the query parameters corresponding to each field.

In one possible implementation manner, the computer device may determine a field in which the corresponding write parameter and query parameter satisfy the reference condition as a candidate field, and for example, the computer device determines a field with a higher write frequency as a candidate field. Of course, the computer device may also filter the candidate fields based on other conditions, which is not limited in the embodiment of the present application.

In step 604, the computer device updates the configuration information based on the field identification of the candidate field.

In one possible implementation, the computer device compares, by the synchronization engine, the field identification of the candidate field with a target field identification already stored in the configuration information, and adds, in response to the field identification of the candidate field being different from the target field identification, the field identification of the candidate field to the configuration information. In a possible implementation manner, in response to acquiring the field identifier of the candidate field, the synchronization engine may delete the target field identifier stored in the configuration information, and then add the field identifier of the candidate field to the configuration information. It should be noted that the above description of the configuration information updating method is only an exemplary description, and the embodiment of the present disclosure does not limit which specific manner is used to update the configuration information.

In step 605, the computer device adjusts the data stored in the row of memory space based on the updated configuration information.

In one possible implementation, the computer device may read the configuration information based on a reference period, compare a first configuration information read in the reference period with a second configuration information read in a previous period, in response to the first configuration information having a new field identifier compared with the second configuration information, read data indicated by the new field identifier from the column of the storage space, add the read data to the row of the storage space, in response to the first configuration information having a deleted field identifier compared with the second configuration information, delete data indicated by the deleted field identifier from the row of the storage space. The reference period is set by a developer, and the embodiment of the disclosure is not limited thereto. In a possible implementation manner, the reference period may not be set, and after the configuration information is updated, the step of updating the data stored in the line storage space is automatically triggered, which is not limited in this disclosure.

The configuration information in the data storage system may also be set by a developer, which is not limited in the embodiments of the present application.

According to the technical scheme provided by the embodiment of the disclosure, the configuration information is dynamically adjusted based on the real historical operation data, namely, the field stored in the line storage space is dynamically adjusted, so that the flexibility of configuring the data storage system can be effectively improved, the data storage system can better meet the current business requirements, and the field configuration problem of the storage space under different writing and query proportions can be effectively solved.

Fig. 7 is a data exchange diagram of a data storage system according to an exemplary embodiment, and the following describes the process of writing, querying, and updating configuration information in conjunction with fig. 7. As shown in fig. 7, in one possible implementation, the storage entry module of the data storage system can receive a data operation request sent by an external program, including a data write request and a data query request, and then read configuration information from the configuration module, determine a storage form of data to be processed, forward the data operation request and the obtained storage form to a corresponding data processing module, for example, forward the data write request and the storage form information to a write engine, and forward the data query request and the storage form information to a read engine. The writing engine or the reading engine can respond to different data operation requests and process data in different storage spaces. The data storage system also comprises a synchronization engine which can synchronize data among different data storage systems and can also periodically read the configuration in the configuration module and update the data stored in the row storage space based on the configuration information. In the embodiment of the present disclosure, the line storage space of the data storage system only needs to store data with a relatively high update frequency, the amount of the stored data is small, and the speed of querying data in the line storage space can be increased even when the data writing speed is ensured to be relatively high. And the data storage and the data reading are both line storage space and column storage space, so that the overlarge data difference in the two storage spaces is avoided, and the data processing amount and the data processing pressure of the synchronization engine are smaller during data synchronization.

FIG. 8 is a block diagram illustrating a data processing apparatus according to an example embodiment. Referring to fig. 8, the apparatus includes a receiving unit 801, a writing unit 802, and a synchronization unit 803.

A receiving unit 801 configured to receive a data write request, where the data write request includes data to be written and a first field identifier of a field to which the data to be written belongs;

a writing unit 802, configured to respond to that the first field identifier in the data writing request hits configuration information, write the data to be written into a row storage space of the data storage system in a row storage form, where the configuration information stores a target field identifier corresponding to at least one target field, and a writing parameter and a query parameter corresponding to the target field satisfy a reference condition;

a synchronization unit 803 configured to synchronize the incremental data in the row storage space to a column storage space of the data storage system, the data in the column storage space being stored in a column storage form.

In one possible implementation, the apparatus further includes:

the acquisition unit is configured to acquire historical operation data of the data storage system, and the historical operation data stores field identification of a field to which the operated data belongs and corresponding data operation;

a first determining unit configured to determine, based on the historical operation data, a write parameter and a query parameter corresponding to a field in the data storage system;

a second determining unit, configured to determine a candidate field in the field based on the write parameter and the query parameter corresponding to the field, where the write parameter and the query parameter corresponding to the candidate field satisfy the reference condition;

and the updating unit is configured to update the configuration information based on the field identification of the candidate field.

In one possible implementation, the updating unit is configured to:

In one possible implementation, the apparatus further includes:

an adjusting unit configured to adjust the data stored in the row of storage spaces based on the updated configuration information.

the adjusting unit is configured to read the configuration information based on a reference period; comparing the first configuration information read in the period with the second configuration information read in the previous period; responding to the first configuration information and the second configuration information, wherein the first configuration information has a new field mark, reading data indicated by the new field mark from the row of storage space, and adding the read data to the row of storage space; in response to the first configuration information having a truncated field identification compared to the second configuration information, deleting data indicated by the truncated field identification from the row of memory space.

In one possible implementation, the writing unit is configured to:

In one possible implementation manner, the receiving unit 801 is configured to receive a data query request, where the data query request includes a second field identifier of a field to which data to be queried belongs;

the apparatus also includes a query unit configured to perform a data query in the column of storage space in response to the second field in the data query request identifying missing configuration information; a data query is performed in the row of memory space in response to the second field in the data query request identifying hit configuration information.

In one possible implementation, the synchronization unit 803 is configured to:

The device provided by the embodiment of the disclosure simultaneously sets the row storage space and the column storage space in the data storage system, applies the configuration information to configure the storage form of each field, and directly writes the field with a large data writing demand into the row storage space in the row storage form, thereby ensuring that the writing speed of data is high, and then calls the data synchronization function to synchronize the updated data in the row storage space into the column storage space, so that the column storage space can store complete and latest data, thereby being capable of well supporting data query and ensuring that the speed of data query is high.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

The computer device may be implemented as a terminal. Fig. 9 is a block diagram illustrating a terminal structure according to an exemplary embodiment. The terminal 900 may be: a smartphone, a tablet, a laptop, or a desktop computer. Terminal 900 may also be referred to by other names such as user equipment, portable terminals, laptop terminals, desktop terminals, and the like.

In general, terminal 900 includes: a processor 901 and a memory 902.

Processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so forth. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

Memory 902 may include one or more computer-readable storage media, which may be non-transitory. In some embodiments, a non-transitory computer readable storage medium in the memory 902 is used to store at least one program code for execution by the processor 901 to implement the data processing methods provided by the method embodiments in the present disclosure.

In some embodiments, terminal 900 can also optionally include: a peripheral interface 903 and at least one peripheral. The processor 901, memory 902, and peripheral interface 903 may be connected by buses or signal lines. Various peripheral devices may be connected to the peripheral interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

The peripheral interface 903 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 901 and the memory 902.

The display screen 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to capture touch signals on or over the surface of the display screen 905. The touch signal may be input to the processor 901 as a control signal for processing. At this point, the display 905 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard.

In some embodiments, terminal 900 can also include one or more sensors 910. The one or more sensors 910 include, but are not limited to: acceleration sensor 911, gyro sensor 912, pressure sensor 913, fingerprint sensor 913, optical sensor 915, and proximity sensor 916.

Those skilled in the art will appreciate that the configuration shown in fig. 9 does not constitute a limitation of terminal 900, and may include more or fewer components than those shown, or may combine certain components, or may employ a different arrangement of components.

The computer device may be implemented as a server. Fig. 10 is a block diagram of a server, which may have a relatively large difference due to different configurations or performances, according to an exemplary embodiment, and the server 1000 may include one or more processors (CPUs) 1001 and one or more memories 1002, wherein at least one program code is stored in the one or more memories 1002, and the at least one program code is loaded and executed by the one or more processors 1001 to implement the processes executed by the server in the data Processing methods provided by the above-mentioned method embodiments. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input and output, and the server 1000 may also include other components for implementing the functions of the device, which are not described herein again.

In an exemplary embodiment, there is also provided a computer readable storage medium, such as a memory, including program code, which is executable by the processor 1001 of the server 1000 to perform the above-described data processing method. Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a Random Access Memory (RAM), a compact-disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the data processing method described above.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method is applied to a data storage system and comprises the following steps:

responding to the first field identification hit configuration information in the data writing request, writing the data to be written into a line storage space of the data storage system in a line storage form, wherein the configuration information stores a target field identification corresponding to at least one target field, and writing parameters and query parameters corresponding to the target field meet reference conditions;

2. The data processing method of claim 1, wherein after synchronizing the incremental data in the row memory space to a column memory space of the data storage system, the method further comprises:

obtaining historical operation data of the data storage system, wherein the historical operation data stores field identification of a field to which the operated data belongs and corresponding data operation;

determining candidate fields in the fields based on the writing parameters and the query parameters corresponding to the fields, wherein the writing parameters and the query parameters corresponding to the candidate fields meet the reference condition;

3. The data processing method of claim 2, wherein the updating the configuration information based on the field identification of the candidate field comprises:

4. The data processing method of claim 2, wherein the updating the configuration information based on the field identification of the candidate field comprises:

5. The data processing method of claim 2, wherein after updating the configuration information based on the field identification of the candidate field, the method further comprises:

adjusting the data stored in the row storage space based on the updated configuration information.

6. The data processing method of claim 5, wherein the column storage space stores a full amount of data;

the adjusting the data stored in the line storage space based on the updated configuration information includes:

reading the configuration information based on a reference period;

in response to that the first configuration information has a new field identifier compared with the second configuration information, reading data indicated by the new field identifier from the column storage space, and adding the read data to the row storage space;

in response to the first configuration information being compared to the second configuration information, with a truncated field identification, deleting data indicated by the truncated field identification from the row storage space.

7. A data storage system is characterized by comprising a storage entry module, a write engine and a synchronization engine;

the write-in engine is configured to, in response to hit configuration information of the first field identifier in the data write-in request, write the data to be written in a line storage space of the data storage system in a line storage form, where a target field identifier corresponding to at least one target field is stored in the configuration information, and write-in parameters and query parameters corresponding to the target field satisfy reference conditions;

8. A computer device, comprising:

one or more processors;

a memory for storing the processor executable program code;

wherein the processor is configured to execute the program code to implement the data processing method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that program code in the computer-readable storage medium, when executed by a processor of a computer device, enables the computer device to perform the data processing method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the data processing method of any one of claims 1 to 6 when executed by a processor.