CN116339626A

CN116339626A - Data processing method, device, computer equipment and storage medium

Info

Publication number: CN116339626A
Application number: CN202310246703.1A
Authority: CN
Inventors: 王鹏培; 赵叶红; 王炳辉; 易辛悦
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-03-15
Filing date: 2023-03-15
Publication date: 2023-06-27

Abstract

The application relates to a data processing method, a data processing device, computer equipment, a storage medium and a computer program product, and relates to the technical field of cloud computing. The method comprises the following steps: under the condition that a preset writing condition is reached, acquiring target transaction data in a writing data memory; writing the target transaction data into a segmented file of a file cache system; the segmented file is used for storing multi-label transaction data; when the file cache system meets preset submitting conditions, applying for a storage block to a database disk for each segmented file in the file cache system, and storing the segmented file into the storage block of the database disk. By adopting the method, the data writing efficiency can be improved.

Description

Data processing method, device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of cloud computing, and in particular, to a data processing method, apparatus, computer device, storage medium, and computer program product.

Background

The stability of data processing is the basis for the normal provision of services by the system. The related data processing method initiates data writing for the direct-connected database.

However, in distributed usage scenarios, etc., data is growing in bursts, which presents a significant challenge to storage systems. When the data volume reaches a certain degree, the writing performance of the data can be seriously affected, and the writing performance bottleneck (namely, writing efficiency is low) appears, so that the system stability and the user use experience are affected, and even the service cannot be normally provided.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve data writing efficiency.

In a first aspect, the present application provides a data processing method. The method comprises the following steps:

under the condition that a preset writing condition is reached, acquiring target transaction data in a writing data memory;

writing the target transaction data into a segmented file of a file cache system; the segmented file is used for storing multi-label transaction data;

when the file cache system meets preset submitting conditions, applying for a storage block to a database disk for each segmented file in the file cache system, and storing the segmented file into the storage block of the database disk.

In one embodiment, the method further comprises:

and writing the target transaction data into the pre-divided write data memory in response to the write data request of the target transaction data.

In one embodiment, the obtaining the target transaction data in the write data memory when the preset writing condition is reached includes:

and under the condition that a preset writing time interval is reached, acquiring target transaction data in a writing data memory.

In one embodiment, the method further comprises:

and when the number of the segmented files which are written in the file cache system is larger than a preset segmented file number threshold value or the uncommitted timing of the file cache system reaches a preset segmented file time threshold value, determining that the file cache system meets preset commit conditions.

In one embodiment, after the writing the target transaction data into the pre-divided write data memory in response to the write data request of the target transaction data, the method further includes:

writing the target transaction data in the write data memory into a transaction log file;

and under the condition that the transaction log file meets the preset synchronization condition, synchronizing target transaction data in the transaction log file to the database disk.

In one embodiment, the synchronizing the target transaction data in the transaction log file to the database disk when the transaction log file meets a preset synchronization condition includes:

acquiring a line identification of target transaction data in the transaction log file under the condition that the transaction log file meets a preset synchronization condition; the line identifier is used for identifying whether target transaction data in the transaction log file is synchronous to a database disk;

synchronizing target transaction data to be synchronized in the transaction log file to the database disk according to the line identification of the target transaction data in the transaction log file;

and deleting the transaction log file and establishing a new transaction log file when all target transaction data in the transaction log file are synchronized to a database disk.

In one embodiment, the method comprises:

reading data from a pre-divided read data memory in response to a read data request;

if the target transaction data corresponding to the read data request does not exist in the read data memory, reading data from a segmented file of a file cache system;

If the target transaction data corresponding to the read data request exists in the segmented file of the file cache system, loading the target transaction data into the read data memory, performing data reading processing on the target transaction data to obtain a data reading result corresponding to the read data request, and returning the data reading result corresponding to the read data request.

In one embodiment, the method further comprises:

if the target transaction data corresponding to the read data request does not exist in the segmented file of the file cache system, the read data from a database disk is loaded into the read data memory, the data reading processing is carried out on the target transaction data, a data reading result corresponding to the read data request is obtained, and the data reading result corresponding to the read data request is returned.

In one embodiment, if the target transaction data corresponding to the read data request does not exist in the read data memory, reading the data from the segmented file of the file cache system includes:

if the target transaction data corresponding to the read data request does not exist in the read data memory, reading the data from a read cache file;

Under the condition that target transaction data corresponding to the read data request exists in the read cache file, the query result of the read cache file is used as a data reading result corresponding to the read data request, and the data reading result corresponding to the read data request is returned;

and if the target transaction data corresponding to the read data request does not exist in the read cache file, reading the data from the segmented file of the file cache system.

In one embodiment, in a case where the target transaction data corresponding to the read data request does not exist in the read cache file, the method further includes:

and synchronously writing the data reading result corresponding to the data reading request into the reading cache file.

In a second aspect, the present application also provides a data processing system. The system comprises:

a write data memory unit for storing target transaction data;

the file cache system is used for acquiring target transaction data in the write data memory under the condition that a preset write condition is reached; writing the target transaction data into a segmented file of the file cache system; the segmented file is used for storing multi-label transaction data; when the file cache system meets preset submitting conditions, applying for a storage block to a database disk for each segmented file in the file cache system;

And the database disk is used for storing the segmented file into the storage block.

In a third aspect, the present application further provides a data processing apparatus. The device comprises:

the acquisition module is used for acquiring target transaction data in the write data memory under the condition that a preset write-in condition is reached;

the first writing module is used for writing the target transaction data into the segmented file of the file cache system; the segmented file is used for storing multi-label transaction data;

and the storage module is used for applying a storage block to the database disk for each segmented file in the file cache system when the file cache system meets preset submitting conditions, and storing the segmented file into the storage block of the database disk.

In one embodiment, the apparatus further comprises:

and the second writing module is used for responding to the data writing request of the target transaction data and writing the target transaction data into the pre-divided data writing memory.

In one embodiment, the acquiring module is specifically configured to:

In one embodiment, the apparatus further comprises:

the determining module is used for determining that the file cache system meets a preset submitting condition when the number of the segmented files which are written in the file cache system is larger than a preset segmented file number threshold or the unresubmitted timing of the file cache system reaches a preset segmented file time threshold.

In one embodiment, the apparatus further comprises:

the third writing module is used for writing the target transaction data in the write data memory into a transaction log file;

and the first synchronization module is used for synchronizing the target transaction data in the transaction log file to the database disk under the condition that the transaction log file meets the preset synchronization condition.

In one embodiment, the first synchronization module is specifically configured to:

In one embodiment, the apparatus comprises:

the first reading module is used for responding to the read data request and reading data from the pre-divided read data memory;

the second reading module is used for reading data from the segmented file of the file cache system if the target transaction data corresponding to the read data request does not exist in the read data memory;

and the third reading module is used for loading the target transaction data into the read data memory if the target transaction data corresponding to the read data request exists in the segmented file of the file cache system, carrying out data reading processing on the target transaction data to obtain a data reading result corresponding to the read data request, and returning the data reading result corresponding to the read data request.

In one embodiment, the apparatus further comprises:

and the fourth reading module is used for loading the target transaction data from the database disk into the read data memory if the target transaction data corresponding to the read data request does not exist in the segmented file of the file cache system, carrying out data reading processing on the target transaction data to obtain a data reading result corresponding to the read data request, and returning the data reading result corresponding to the read data request.

In one embodiment, the second reading module is specifically configured to:

In one embodiment, in a case where the target transaction data corresponding to the read data request does not exist in the read cache file, the apparatus further includes:

and the second synchronization module is used for synchronously writing the data reading result corresponding to the read data request in the read cache file.

In a fourth aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the first aspect described above when the processor executes the computer program.

In a fifth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the first aspect described above.

In a sixth aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, carries out the steps of the first aspect described above.

The data processing method, the data processing device, the computer equipment, the storage medium and the computer program product acquire target transaction data in the write data memory under the condition that the preset writing condition is reached; writing the target transaction data into a segmented file of a file cache system; the segmented file is used for storing multi-label transaction data; when the file cache system meets preset submitting conditions, applying for a storage block to a database disk for each segmented file in the file cache system, and storing the segmented file into the storage block of the database disk. When writing data into a database disk, the corresponding disk space is generally locked for safety so as to prevent the data from being tampered with by other data, if one transaction is submitted once, the subsequent transaction can wait for the lock to be released and then can continue writing inwards, which is equivalent to serial processing, and has quite low efficiency. Therefore, through the introduction of the segmented files of the file cache system, the interaction between each piece of data and the database disk is divided into one piece of segmented files and the interaction between the database disk, one segmented file can store tens of millions of pieces of data, a storage block can be applied to the disk at one time, and tens of thousands of pieces of data can be stored only once by locking, so that the data writing efficiency is greatly improved.

Drawings

FIG. 1 is a flow diagram of a data processing method in one embodiment;

FIG. 2 is a flowchart illustrating steps after writing target transaction data into a pre-partitioned write data memory according to one embodiment;

FIG. 3 is a flowchart illustrating steps for synchronizing target transaction data in a transaction log file to a database disk in one embodiment;

FIG. 4 is a flow chart of a data processing method according to another embodiment;

FIG. 5 is a flowchart illustrating a step of reading data from a segmented file of a file cache system according to one embodiment;

FIG. 6 is a block diagram of a data processing system in one embodiment;

FIG. 7 is a block diagram of a data processing apparatus in one embodiment;

fig. 8 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

In one embodiment, as shown in fig. 1, a data processing method is provided, where this embodiment is applied to a terminal to illustrate the method, and it is understood that the method may also be applied to a server, and may also be applied to a system including a terminal and a server, and implemented through interaction between the terminal and the server. The method may be applied to a distributed system. The distributed system may be implemented by a cloud computing platform. For example, the distributed system may be implemented by a platform as a service (Platform as a Service, paaS). The method can be applied to a monitoring system of a distributed system. The terminal can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things equipment and portable wearable equipment, and the internet of things equipment can be smart speakers, smart televisions, smart air conditioners, smart vehicle-mounted equipment and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server may be implemented as a stand-alone server or as a server cluster composed of a plurality of servers. In this embodiment, the method includes the steps of:

Step 101, under the condition that the preset writing condition is reached, acquiring target transaction data in a writing data memory.

In the embodiment of the application, under the condition that the preset writing condition is reached, the terminal acquires the target transaction data in the writing data memory. The writing condition is a condition for writing target transaction data in the data writing memory into the database. The target transaction data is data related to a transaction action. The write data memory is a portion of memory that supports only data write operations.

Step 102, writing the target transaction data into the segmented file of the file cache system.

The segmented file is used for storing multi-label transaction data.

In the embodiment of the application, the terminal writes the target transaction data into the segmented file of the file cache system. Specifically, the file cache system receives a write request. Then, the terminal searches for an idle segmented file in the file cache system. Then, the terminal determines a target segment file in the idle segment file. Then, the terminal writes the target transaction data into the target segment file of the file cache system. Wherein the file cache system comprises one or more segmented files. For example, a file cache system includes 10 segmented files.

In one example, the terminal determines a target segment file sequence in the free segment file. Wherein the target segment file sequence comprises a plurality of target segment files. And then, the terminal sequentially writes the target transaction data into the target segment files of the file cache system according to the sequence in the target segment file sequence. One is written fully, and the next target segment file is written continuously.

In one example, the terminal writes the target transaction data in parallel into a plurality of target segment files of the file cache system in the case where the data amount of the target transaction data satisfies a preset parallel writing condition. Therefore, when the data peak is encountered, writing is performed concurrently, and a plurality of segmented files work simultaneously, so that the data writing efficiency can be improved.

Step 103, when the file cache system meets the preset submitting condition, applying for a storage block to the database disk for each segmented file in the file cache system, and storing the segmented file into the storage block of the database disk.

In the embodiment of the present application, the commit condition is a condition that the segmented files in the file cache system are stored to the database disk. When the file cache system meets preset submitting conditions, aiming at each segmented file in the file cache system, the terminal applies for a storage block to a database disk. After the application passes, the terminal stores the segmented file into the storage block of the database disk.

In the data processing method, under the condition that a preset writing condition is met, target transaction data in a writing data memory is obtained; writing target transaction data into a segmented file of a file cache system; the segmented file is used for storing multi-label transaction data; when the file cache system meets preset submitting conditions, applying for a storage block to the database disk for each segmented file in the file cache system, and storing the segmented file into the storage block of the database disk. When writing data into a database disk, the corresponding disk space is generally locked for safety so as to prevent the data from being tampered with by other data, if one transaction is submitted once, the subsequent transaction can wait for the lock to be released and then can continue writing inwards, which is equivalent to serial processing, and has quite low efficiency. Therefore, through the introduction of the segmented files of the file cache system, the interaction between each data and the database disk is changed into the interaction between each segmented file and the database disk, and one segmented file can store tens of millions of data, so that a storage block can be applied to the disk at one time, tens of millions of data can be stored only by locking once, the data writing efficiency is greatly improved, the data writing efficiency is improved by tens of millions of times, and the method is particularly suitable for the monitoring fields of Internet scenes with particularly large flow peaks, such as twenty-one monitoring, and rush tickets in the peak period of spring transport 12306.

In one embodiment, the data processing method further comprises the steps of: and writing the target transaction data into the pre-divided write data memory in response to the write data request of the target transaction data.

In the embodiment of the application, the terminal pre-divides the memory of one device into a write data memory and a read data memory. And responding to the data writing request of the target transaction data, and writing the target transaction data into a data writing memory in real time by the terminal. The read data memory is a part of memory only supporting data read operation. The memory of a device includes: write data memory and read data memory. The size of the write data memory may be different from the size of the read data memory. The magnitude of the size of the write data memory may be different from the magnitude of the size of the read data memory. The size of the write data memory and the size of the read data memory can be set by parameters of the underlying operating system.

In the data processing method, the memory is divided into the write data memory and the read data memory in advance, and the terminal writes the target transaction data into the write data memory in real time in response to a write data request of the target transaction data. The written data is in an original data format, basically, no processing is carried out (some fields such as time consumption can be converted by simple units, namely nanoseconds are converted into milliseconds), the read data is processed according to query logic, the data has a certain data format, the read operation and the write operation are two independent operations, and a data sharing scene does not exist. The writing operation only needs to ensure that the database is continuously written in, in theory, the large memory space is not occupied, the reading operation has large fluctuation of the space range of the memory according to different query logics, if the selected query range is 1 hour and 1 week, the data volume of the query is different orders of magnitude, and the required memory space is also different orders of magnitude because the data can be loaded into the memory for processing. Therefore, the method divides the memory of one device into two parts, namely a data writing memory and a data reading memory, has independent running spaces and can differentially set the sizes of the memory spaces, so that the read-write separation of the memory dimension in one device is realized, the data reading and the data writing are not influenced, the problem that the data cannot be written due to the fact that the memory of the device is occupied completely by the data reading is avoided due to the large data query range, the data loss is avoided, the continuous writing of the data which can not be influenced and the integrity of the data are ensured, and the stability of the data writing is further improved. On the contrary, assuming that the total memory is 100 megabits, the query range is large, and the 100 megabits are completely used up, so that writing cannot be performed, data is lost, and the system operation is affected. In addition, the method realizes the read-write separation of the memory dimension in one device, does not need to adopt a plurality of nodes to execute data reading and data writing, can save resources, is beneficial to transversely expanding the resources, and has low maintenance cost.

In one embodiment, the specific process of obtaining the target transaction data in the write data memory when the preset writing condition is reached includes the following steps: and under the condition that a preset writing time interval is reached, acquiring target transaction data in a writing data memory.

In the embodiment of the application, under the condition that a preset writing time interval is reached, the terminal acquires target transaction data in the writing data memory. The write time interval is used to represent the time interval when the target transaction data in the write data memory is acquired every two times. The write time interval may be configured in a start-up configuration file of the database, which is validated after restarting the database. For example, the write time interval may be 1 second.

In the data processing method, the terminal acquires target transaction data in the write data memory at intervals of write time. Thus, the target transaction data in the write data memory is acquired at fixed time so as to be written into the segmented file of the file cache system, a large amount of transaction data can be written into the segmented file at one time, the interaction times between the write data memory and the file cache system are reduced, and the data writing efficiency is further improved.

In one embodiment, the specific process of obtaining the target transaction data in the write data memory when the preset writing condition is reached includes the following steps: and responding to the data writing request of the target transaction data, and reading the target transaction data in the data writing memory. Thus, the target transaction data in the write data memory is obtained in real time to be written into the segmented file of the file cache system, the target transaction data can be written into the segmented file in real time, and the data loss can be prevented.

In one embodiment, the data processing method further comprises the steps of: when the number of the segmented files which are written in the file cache system is larger than a preset segmented file number threshold value or the non-submitting timing of the file cache system reaches a preset segmented file time threshold value, determining that the file cache system meets preset submitting conditions.

In the embodiment of the application, the terminal compares the number of the segment files which are written in the file cache system with a preset segment file number threshold in real time. Meanwhile, the terminal judges whether the uncommitted timing of the file cache system reaches a preset segmented file time threshold in real time. When the number of the segmented files which are written in the file cache system is larger than a preset segmented file number threshold value or the non-submitting timing of the file cache system reaches a preset segmented file time threshold value, the terminal determines that the file cache system meets preset submitting conditions. Wherein the segment file number threshold is related to the total number of segment files in the file cache system and the size of each segment file. For example, if the file cache system includes 10 equal-sized segmented files, the threshold number of segmented files may be 5. The uncommitted time is the time since the last time the file cache system written the segmented file to the database disk. The segment file time threshold is the time interval between the file cache system writing the segment file to the database disk twice. For example, the segment file time threshold may be half an hour.

In the data processing method, the condition that the number of the segmented files which are written in the file cache system is larger than a preset segmented file number threshold value and the time when the non-submitting time of the file cache system reaches the preset segmented file time threshold value is used as the submitting condition that the segmented files in the file cache system are written in the database disk once. Therefore, the data is timely written into the database disk under the condition of large data volume, the data is prevented from being jammed in the file cache system, the data writing efficiency can be further improved, the data is timely written into the database disk under the condition of small data volume, the data loss caused by untimely persistence of the data is prevented, and the data writing stability and the data reliability can be improved.

In one embodiment, as shown in fig. 2, in response to a write data request of the target transaction data, the method further comprises the following steps after writing the target transaction data into the pre-divided write data memory:

step 201, writing target transaction data in the write data memory into a transaction log file.

In the embodiment of the application, the terminal writes the target transaction data in the write data memory into the transaction log file in real time. The terminal synchronously writes the target transaction data in the write data memory into the segmented file and the transaction log file in the file cache system. The transaction log file is used for storing target transaction data in real time.

Step 202, synchronizing target transaction data in the transaction log file to a database disk under the condition that the transaction log file meets a preset synchronization condition.

In the embodiment of the application, under the condition that the transaction log file meets the preset synchronization condition, the terminal synchronizes target transaction data in the transaction log file to the database disk. The synchronization condition is used for measuring whether target transaction data in the transaction log file are synchronized to a database disk.

In the data processing method, after the target transaction data is written into the pre-divided write data memory in response to the write data request of the target transaction data, the terminal synchronously writes the target transaction data into the segmented file and the transaction log file in the file cache system, and under the condition that the transaction log file meets the preset synchronous condition, the target transaction data in the transaction log file is synchronized into the database disk. In this way, the target transaction data in the write data memory are synchronized to the transaction log file in real time, and then the target transaction data in the transaction log file are synchronized to the database disk, so that the data loss under the extreme conditions such as power failure can be prevented, and the reliability of the data is further improved.

In one embodiment, the specific process of writing target transaction data into a segmented file of a file cache system includes the steps of: according to a preset first data table structure rule, performing file verification on target transaction data to obtain a first verification result of the target transaction data; if the first verification result indicates that the target transaction data passes the verification, writing the target transaction data into a segmented file of a file cache system; and if the first verification result indicates that the target transaction data does not pass the verification, writing the target transaction data into an abnormal data log file. The first data table structure rule comprises a first field number rule, a first field size rule and a first field format rule. For example, the first field format rule may include that the time consuming field can only be a numeric type.

In one embodiment, the specific process of writing target transaction data into a transaction log file includes the steps of: according to a preset second data table structure rule, performing file verification on target transaction data to obtain a second verification result of the target transaction data; if the second check result indicates that the target transaction data passes the check, writing the target transaction data into a transaction log file; and if the second check result indicates that the target transaction data does not pass the check, writing the target transaction data into an abnormal data log file. The second data table structure rule comprises a second field number rule, a second field size rule and a second field format rule.

In one embodiment, as shown in fig. 3, in the case that the transaction log file meets the preset synchronization condition, synchronizing the target transaction data in the transaction log file to the database disk includes the following steps:

step 301, acquiring a line identifier of target transaction data in the transaction log file under the condition that the transaction log file meets a preset synchronization condition.

Wherein the row identification is used to identify whether the target transaction data in the transaction log file is synchronized to the database disk.

In the embodiment of the application, under the condition that the transaction log file meets the preset synchronization condition, the terminal acquires the line identification of the target transaction data in the transaction log file. Wherein the row identification may be synchronized and unsynchronized. The row identification may be one column at the end of each row of the file.

Specifically, when the unsynchronized timing of the transaction log file reaches a preset log time threshold, or the file size of the transaction log file meets a preset log size threshold, the terminal determines that the transaction log file meets a preset synchronization condition. Wherein the unsynchronized timing is the time since the last transaction log file was written to the database disk for the target transaction data. The log time threshold is the time interval between the transaction log file writing the target transaction data to the database disk twice. The log size threshold may be 1G.

Step 302, synchronizing target transaction data to be synchronized in the transaction log file to a database disk according to the line identification of the target transaction data in the transaction log file.

In the embodiment of the application, the terminal synchronizes target transaction data to be synchronized in the transaction log file to the database disk according to the line identification of the target transaction data in the transaction log file. Specifically, the terminal only synchronizes the target transaction data in the transaction log file, which is marked as unsynchronized by the line, to the database disk.

In step 303, when all target transaction data in the transaction log file are synchronized to the database disk, the transaction log file is deleted, and a new transaction log file is created.

In the embodiment of the application, when all target transaction data in the transaction log file are synchronized to the database disk, the terminal deletes the transaction log file. The terminal then creates a new transaction log file.

In the data processing method, under the condition that the transaction log file meets the preset synchronization condition, the line identification of the target transaction data in the transaction log file is obtained, according to the line identification of the target transaction data in the transaction log file, only the line identification is synchronized to the database disk as the target transaction data in the unsynchronized transaction log file, and when all the target transaction data in the transaction log file are synchronized to the database disk, the transaction log file is deleted, and a new transaction log file is established. Therefore, according to the line identification of the target transaction data, only the unwritten target transaction data is synchronized to the database disk, repeated writing can be avoided, and the data writing efficiency is further improved. Moreover, the method not only synchronizes the target transaction data under the condition that the transaction log file reaches a fixed size, but also synchronizes the target transaction data in the transaction log file to the database disk at regular time, thereby realizing timely synchronization of the transaction log file, comprehensively avoiding data loss and further improving the data reliability of data writing.

In one embodiment, as shown in fig. 4, the data processing method further includes the steps of:

in step 401, data is read from a pre-partitioned read data memory in response to a read data request.

In the embodiment of the application, the terminal pre-divides the memory of one device into a write data memory and a read data memory. Then, in response to the read data request, the terminal reads data from the pre-divided read data memory. Wherein the read data request is used to indicate that the user needs to read data.

Step 402, if the target transaction data corresponding to the read data request does not exist in the read data memory, the data is read from the segmented file of the file cache system.

In the embodiment of the application, the terminal searches the target transaction data corresponding to the read data request in the read data memory. If the target transaction data corresponding to the data reading request exists in the data reading memory, the terminal reads the data from the data reading memory. If the target transaction data corresponding to the data reading request does not exist in the data reading memory, the terminal reads the data from the segmented file of the file cache system.

Step 403, if the target transaction data corresponding to the read data request exists in the segmented file of the file cache system, loading the target transaction data into the read data memory, performing data reading processing on the target transaction data to obtain a data reading result corresponding to the read data request, and returning the data reading result corresponding to the read data request.

In the embodiment of the application, the terminal searches the target transaction data corresponding to the read data request in the segmented file of the file cache system. If the target transaction data corresponding to the read data request exists in the segmented file of the file cache system, the terminal loads the target transaction data into the read data memory. And then, the terminal performs data reading processing on the target transaction data to obtain a data reading result corresponding to the data reading request. And then, the terminal returns a data reading result corresponding to the data reading request.

Specifically, the terminal performs data reading processing on target transaction data according to query logic of the data reading request to obtain a data reading result of a data format corresponding to the data reading request.

In the data processing method, in response to the read data request, data is read from the pre-divided read data memory, if the read data memory does not have the target transaction data corresponding to the read data request, the data is read from the segmented file of the file cache system, the read target transaction data is loaded into the read data memory, the data reading processing is performed on the target transaction data, a data reading result corresponding to the read data request is obtained, and the data reading result corresponding to the read data request is returned. Therefore, the sequential query of the data reading memory and the file caching system accords with the actual situation that a user generally only queries target transaction data in the last period of time, the data reading can directly read the data reading memory and the segmented file without directly interacting with a database disk, the full data query is avoided, the data query is converted from the data query of the T level to the data query of hundreds of megabits, and the efficiency of data reading and query is greatly improved. Moreover, in the real-time monitoring system, the possibility of inquiring the data in the last period of time is highest, and the method is particularly suitable for real-time monitoring inquiry, and the inquiry is real-time and efficient.

In one embodiment, the data processing method further comprises the steps of: if the target transaction data corresponding to the read data request does not exist in the segmented file of the file cache system, the read data in the database disk is loaded into the read data memory, the data reading processing is carried out on the target transaction data, a data reading result corresponding to the read data request is obtained, and the data reading result corresponding to the read data request is returned.

In this embodiment of the present application, if the target transaction data corresponding to the read data request does not exist in the segmented file of the file cache system, the terminal will read the data from the database disk. Then, the terminal loads the target transaction data into the read data memory. And then, the terminal performs data reading processing on the target transaction data to obtain a data reading result corresponding to the data reading request. And then, the terminal returns a data reading result corresponding to the data reading request. Specifically, the specific procedure of the terminal performing the data reading process on the target transaction data is similar to that in step 303.

In the above data processing method, if the target transaction data corresponding to the read data request does not exist in the read data memory and the segmented file of the file cache system, the read data from the database disk is loaded into the read data memory, the data reading processing is performed on the target transaction data, the data reading result corresponding to the read data request is obtained, and the data reading result corresponding to the read data request is returned. Therefore, through sequential multi-level query of the read data memory, the file cache system and the database disk, the data query effectiveness and the data reading accuracy are ensured while direct interaction between data reading and the database disk is avoided as much as possible.

In one embodiment, as shown in fig. 5, if the target transaction data corresponding to the read data request does not exist in the read data memory, the specific process of reading the data from the segmented file of the file cache system includes the following steps:

in step 501, if the target transaction data corresponding to the read data request does not exist in the read data memory, the data is read from the read cache file.

In this embodiment of the present application, if the target transaction data corresponding to the read data request does not exist in the read data memory, the terminal queries the target transaction data from the read cache file. The read cache file is used for storing data reading results corresponding to one or more read data requests.

In one example, the terminal queries the target transaction data from the read cache file according to a query condition corresponding to the read data request. The query conditions may include, among other things, a data format. For example, if the query term is "select time, trxCnt from table", the data format corresponding to the query logic is two columns, one column is time and one column is trxCnt.

Step 502, under the condition that target transaction data corresponding to the read data request exists in the read cache file, the query result of the read cache file is used as a data reading result corresponding to the read data request, and the data reading result corresponding to the read data request is returned.

In this embodiment of the present application, when target transaction data corresponding to a read data request exists in a read cache file, a terminal will use a query result of the read cache file as a data read result corresponding to the read data request. And then, the terminal returns a data reading result corresponding to the data reading request.

In step 503, if the target transaction data corresponding to the read data request does not exist in the read cache file, the data is read from the segmented file of the file cache system.

In the embodiment of the application, if the target transaction data corresponding to the read data request does not exist in the read cache file, the terminal queries the target transaction data from the segmented file of the file cache system.

In the above data processing method, if the target transaction data corresponding to the read data request does not exist in the read data memory, the read data in the read cache file is read first, if the target transaction data corresponding to the read data request does not exist in the read cache file, the data is read from the segmented file of the file cache system, and if the target transaction data corresponding to the read data request exists in the read cache file, the query result of the read cache file is used as the data read result corresponding to the read data request, and the data read result corresponding to the read data request is returned. Therefore, by introducing the read cache file, sequential multi-level inquiry of the read data memory, the read cache file, the file cache system and the database disk is realized, when target transaction data cannot be read in the read data memory, the data is preferentially read from the read cache file storing data reading results corresponding to one or more previous read data requests, so that the inquiry habit of a user is better met, the data processing in the data reading process is reduced, and the data reading efficiency is further improved.

In one embodiment, in the case that the target transaction data corresponding to the read data request does not exist in the read cache file, the data processing method further includes the following steps: and synchronously writing a data reading result corresponding to the data reading request in the reading cache file.

In the embodiment of the application, under the condition that target transaction data corresponding to the read data request does not exist in the read cache file, the terminal synchronously writes the data read result corresponding to the read data request in the read cache file.

Specifically, if the target transaction data corresponding to the read data request does not exist in the read cache file and the target transaction data corresponding to the read data request exists in the segmented file of the file cache system, the terminal synchronously writes the data read result corresponding to the read data request in the read cache file while the terminal returns the data read result corresponding to the read data request. If the target transaction data corresponding to the read data request does not exist in the read cache file, the target transaction data corresponding to the read data request does not exist in the segmented file of the file cache system, and the target transaction data corresponding to the read data request exists in the database disk, the terminal synchronously writes the data reading result corresponding to the read data request in the read cache file while returning the data reading result corresponding to the read data request.

In the data processing method, when the target transaction data corresponding to the read data request does not exist in the read cache file, the data read result corresponding to the read data request is synchronously written in the read cache file. In this way, the data reading results corresponding to the data reading requests are synchronized in the read cache file in real time, so that the data reading results corresponding to the current data reading requests can be obtained by inquiring the previous data reading results, the inquiry habit of a user is met, the data processing in the data reading process is reduced, and the data reading efficiency is further improved.

In one embodiment, the terminal returns the prompt message when the data amount of the data reading result corresponding to the data reading request is greater than a preset query data amount threshold. Then, the terminal empties the read cache file. The prompt information is used for indicating that the query data volume is too large. For example, the query data volume threshold may be 2G. Therefore, by setting the defense mechanism and checking the size of the node for reading the data, the system is not available due to too large data quantity of inquiry caused by improper inquiry statement can be prevented, and the stability of the data processing system is further improved.

In one embodiment, the terminal empties the read cache file when the un-updated timing of the read cache file reaches a preset cache file update time threshold. The time of no update is the time since the last time the cache file was updated. For example, the cache file update time threshold may be 2 hours. Therefore, the cache file is cleaned at regular time, and data preservation can be achieved.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiments of the present application also provide a data processing system for implementing the above-mentioned data processing method. The implementation of the solution provided by the system is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing system provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.

In one embodiment, as shown in FIG. 6, there is provided a data processing system 600 comprising: a write data memory unit 610, a file cache system 620, and a database disk 630, wherein:

a write data memory unit 610 for storing target transaction data;

the file cache system 620 is configured to obtain target transaction data in the write data memory when a preset writing condition is reached; writing the target transaction data into a segmented file of the file cache system; the segmented file is used for storing multi-label transaction data; when the file cache system meets preset submitting conditions, applying for a storage block to a database disk for each segmented file in the file cache system;

A database disk 630, configured to store the segmented file into the storage block.

Based on the same inventive concept, the embodiment of the application also provides a data processing device for realizing the above related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.

In one embodiment, as shown in FIG. 7, there is provided a data processing apparatus 700 comprising: an acquisition module 710, a first writing module 720, and a storage module 730, wherein:

an obtaining module 710, configured to obtain target transaction data in the write data memory when a preset writing condition is reached;

a first writing module 720, configured to write the target transaction data into a segmented file of a file cache system; the segmented file is used for storing multi-label transaction data;

and the storage module 730 is configured to apply, for each segmented file in the file cache system, a storage block to a database disk when the file cache system meets a preset commit condition, and store the segmented file in the storage block of the database disk.

Optionally, the apparatus 700 further includes:

Optionally, the acquiring module 710 is specifically configured to:

Optionally, the apparatus 700 further includes:

Optionally, the first synchronization module is specifically configured to:

Optionally, the apparatus 700 includes:

Optionally, the apparatus 700 further includes:

Optionally, the second reading module is specifically configured to:

Optionally, in the case that the target transaction data corresponding to the read data request does not exist in the read cache file, the apparatus 700 further includes:

Each of the modules in the above-described data processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 8. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data processing method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method of data processing, the method comprising:

2. The method according to claim 1, wherein the method further comprises:

3. The method of claim 1, wherein the obtaining the target transaction data in the write data memory if the predetermined write condition is reached comprises:

4. The method according to claim 1, wherein the method further comprises:

5. The method of claim 2, wherein after writing the target transaction data into the pre-partitioned write data memory in response to the write data request for the target transaction data, further comprising:

6. The method of claim 5, wherein synchronizing the target transaction data in the transaction log file to the database disk if the transaction log file satisfies a preset synchronization condition comprises:

7. The method according to claim 1, characterized in that the method comprises:

8. The method of claim 7, wherein the method further comprises:

9. The method of claim 8, wherein if the target transaction data corresponding to the read data request does not exist in the read data memory, reading the data from the segmented file of the file cache system comprises:

10. The method of claim 9, wherein in the absence of target transaction data corresponding to the read data request in the read cache file, the method further comprises:

11. A data processing system, the system comprising:

a write data memory unit for storing target transaction data;

12. A data processing apparatus, the apparatus comprising:

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 10 when the computer program is executed.

14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 10.

15. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 10.