CN113704274B - Data reading method and electronic equipment - Google Patents
Data reading method and electronic equipment Download PDFInfo
- Publication number
- CN113704274B CN113704274B CN202010429746.XA CN202010429746A CN113704274B CN 113704274 B CN113704274 B CN 113704274B CN 202010429746 A CN202010429746 A CN 202010429746A CN 113704274 B CN113704274 B CN 113704274B
- Authority
- CN
- China
- Prior art keywords
- server
- data
- merging
- baseline data
- target file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 230000015654 memory Effects 0.000 claims abstract description 115
- 230000004048 modification Effects 0.000 claims description 42
- 238000012986 modification Methods 0.000 claims description 42
- 230000008859 change Effects 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000008676 import Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/235—Update request formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data reading method and electronic equipment, which are used for solving the problem that a client cannot read data when a shared memory server updates the data in real time in the prior art. The method comprises the following steps: when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data; the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file; and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
Description
Technical Field
The present invention relates to the field of mobile communications technologies, and in particular, to a data reading method and an electronic device.
Background
Currently, a Business Operation Support System (BOSS) adopts a single machine to load a database data table to realize a shared memory, that is, a host in a shared memory server accesses the database in a read mode, and stores data in the database into a shared memory block of the host of the shared memory server so as to be accessed by all application programs on the host of the shared memory server, thereby improving the data access efficiency of the application programs of the host of the shared memory server.
With the widespread use of distributed systems, computer server hosts are increasingly replacing small computers. In the distributed system, a shared memory server program is deployed on each host, and the shared memory server program needs to directly interact with the database to load the data in the database into the shared memory server.
However, when the existing shared memory server updates data in real time, access processes of all clients are locked, and client programs are prohibited from accessing the shared memory server, so that the shared memory server updates data. In the period of time, all clients cannot process any task, and the processing efficiency of the clients is greatly reduced. Also, in scenarios where real-time requirements are high, such long client-side blocking will cause a large number of message timeouts.
Disclosure of Invention
The embodiment of the invention provides a data reading method and electronic equipment, which are used for solving the problem that a client cannot read data when a shared memory server in the prior art updates the data in real time.
In order to solve the technical problems, the invention is realized as follows: in a first aspect, an embodiment of the present invention provides a method for reading data, including:
When the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
In a second aspect, an embodiment of the present invention further provides a data reading apparatus, including:
a generation unit configured to generate a target file based on incremental update data in an update server when a data amount of the incremental update data in the update server is greater than or equal to a preset threshold;
the updating unit is used for sending a merging notification to the merging server by the updating server, so that the merging server updates the specified baseline data stored in the out-of-heap memory based on the merging notification and the target file;
and the sending unit is used for responding to the reading request for the specified baseline data sent by the client and sending the specified baseline data to the client by the configuration server.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
The update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
By adopting the data reading method provided by the invention, the update server can generate the target file based on the increment update data when the data quantity of the increment update data in the update server is larger than or equal to the preset threshold value; then, the updating server sends a merging notification to the merging server, so that the merging server updates the specified baseline data stored in the external memory based on the merging notification and the target file; finally, the configuration server responds to the read request for the specified baseline data sent by the client, and sends the specified baseline data to the client. Therefore, no matter whether the merging server finishes updating the specified baseline data or not, the configuration server can send the specified baseline data of the latest version to the client, so that the performance and efficiency of the updating server, the merging server and the configuration server are improved, and the performance and efficiency of the client are also greatly improved by timely processing the request of the client.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic implementation flow chart of a data reading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a shared memory cluster in a method for reading data according to an embodiment of the present invention;
fig. 3 is a schematic diagram of sending specified baseline data to a client in the data reading method provided in the embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data reading device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The following describes in detail the technical solutions provided by the embodiments of the present invention with reference to the accompanying drawings.
In order to solve the problem that the client cannot read data when the shared memory server updates the data in real time in the prior art, the invention provides a data reading method, and an execution subject of the method can be but is not limited to be configured to execute at least one of the user terminals provided by the embodiment of the invention.
For convenience of description, hereinafter, embodiments of the method will be described taking an execution subject of the method as an electronic device capable of executing the method as an example. It will be appreciated that the subject of execution of the method is an exemplary illustration of an electronic device and should not be construed as limiting the method.
Specifically, the method for reading data provided by the invention comprises the following steps: firstly, when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data; then, the updating server sends a merging notification to the merging server, so that the merging server updates the specified baseline data stored in the external memory based on the merging notification and the target file; finally, the configuration server responds to the read request for the specified baseline data sent by the client, and sends the specified baseline data to the client. Thus, no matter whether the merging server finishes updating the specified baseline data or not, the configuration server can send the specified baseline data of the latest version to the client, thereby improving the updating server. The performance and efficiency of the server and the configuration server are combined, and the performance and efficiency of the client are greatly improved by timely processing the request of the client.
The following describes the implementation process of the method in detail with reference to the implementation flow chart of the data reading method shown in fig. 1, including:
step 101, when the data volume of incremental update data in an update server is greater than or equal to a preset threshold, the update server generates a target file based on the incremental update data;
it should be appreciated that existing shared memory servers use a library-level lock to prohibit responding to all read requests from the client program when updating the data. When the shared server updates data, all clients cannot process services related to the shared memory server, which greatly reduces the processing performance of client application programs.
In one scenario for handling online charging tasks, to ensure processing performance of a client application, a shared memory server may respond to requests issued by clients within 200ms, otherwise the processing is considered to be timeout. In such a scenario, where real-time requirements are extremely high, a long client read request is not processed, which would cause a large number of requests and timeouts for the message.
In addition, as the number of hosts in a distributed system increases, the shared memory server deployed on each host needs to be able to interact with the database, which occupies a large amount of database resources, and frequent accesses will cause a decrease in the performance of the database, thereby affecting the overall performance of the shared memory database. Particularly, when the task of data reloading or batch restarting in the shared memory server is in progress, the resources of the central processor and the input and output resources on the database host are often fully utilized. Therefore, in the above-mentioned situations, vulnerabilities often occur in the database hosts, the shared memory server hosts or the clients, which greatly prevent the smooth running of the processes.
Therefore, the method provided by the embodiment of the invention can be realized, and the shared memory server can still respond to the read request sent by the client in the process of data updating, so that the efficiency of processing tasks by the shared memory server is improved.
Optionally, as shown in fig. 2, a schematic diagram of a shared memory cluster in the method for reading data according to an embodiment of the present invention is shown. In fig. 2, a shared memory cluster in a shared memory system may include a primary update server, a backup update server, a primary configuration server, a backup configuration server, and a plurality of merge servers.
Wherein the update server is operable to store recorded modification operations and the like for the baseline data, i.e., incremental update data for the baseline data; the configuration server can be used for centrally managing other servers in the shared memory cluster; the merging servers can be used for updating the baseline data based on the incremental updating data, and a whole baseline data part can be stored in the out-of-pile memory corresponding to each merging server, so that the client side is ensured not to delay the progress of the client side due to the busy merging servers when the client side accesses the shared memory system.
In addition, in order to save host resources, one update server and one configuration server may be deployed on the same host, and different modes may be set between the primary update server and the backup update server, and between the primary configuration server and the backup configuration server.
Optionally, in practical application, in order to timely satisfy a read request of a client, a plurality of shared memory clusters with the above structure may be deployed in one shared memory system. Each shared memory cluster may be responsible for management operations in that cluster by a respective configuration server.
Normally, the shared memory cluster can only serve the corresponding client, and if unexpected situations occur, the shared memory cluster can also serve other unspecified clients. If the need to ensure that each shared memory cluster serves the corresponding client, the shared memory cluster can only process offline read requests for other clients, so that the clients corresponding to the shared memory clusters can be preferentially served under the condition that the shared memory clusters help other clusters to work.
Optionally, a suitable preset threshold may be set, so that when the data amount of the incremental update data recorded in the update server is greater than or equal to the preset threshold, the update server, the merge server and the configuration server can cooperate to complete updating of the data, thereby implementing an operation of updating the data in real time when the condition is satisfied.
Step 102, the update server sends a merging notification to the merging server, so that the merging server updates the specified baseline data stored in the external memory based on the merging notification and the target file;
it should be appreciated that the amount of data that a client requests to read may be large at times, which may require that the read performance of the shared memory system be high, such as an average read latency of the shared memory system to several microseconds. Therefore, in order to improve the reading performance of the shared memory system, the method provided by the embodiment of the invention can store the baseline data in a local area such as an off-heap memory, that is, in a memory heap outside the update server, the merge server and the configuration server. Neither the input-output or remote procedure call (Remote Procedure Call, RPC) approach using disk can meet the higher read performance requirements.
In addition, to enable the update server, merge server, and configuration server to access data in the off-heap memory, a virtual memory space corresponding to the baseline data may be established in the merge server, enabling multiple merge servers to access data in the same off-heap memory, rather than storing data using respective in-heap memories. Therefore, the merging server does not need to transmit data in the traditional Inter-process communication Inter-Process Communication, IPC) or RPC mode, so that the data access performance is greatly improved, and the utilization rate of the memory is also greatly improved.
The merging server can store the memory mapping file corresponding to the baseline data, so that the merging server can realize the modification of the baseline data without reading the baseline data body.
Optionally, the configuration server may centrally manage other servers in the shared memory system and send a notification to the designated server according to the requirement, so that the designated server completes the required processing task, so when the combination server needs to be notified to update the baseline data, the update server may send a data change notification indicating that the version of the data changes to the configuration server, then determine, according to the requirement, the combination server that needs to perform the task of updating the baseline data, and send the combination notification indicating that the baseline data can be updated to the combination servers.
Specifically, in the method provided by the embodiment of the present invention, the update server sends a merge notification to the merge server, so that the merge server updates specified baseline data stored in the external memory based on the merge notification and the target file, including:
the update server sends a data change notification to the configuration server to notify the configuration server that the data version of the baseline data is changed;
The configuration server sends a merge notification to the merge server in response to the data change notification, such that the merge server updates the specified baseline data based on the merge notification and the target file.
Alternatively, to be able to obtain modification operations in the target file and update the baseline data based on these modification operations, the merge server may obtain the target file from the update server and be able to index a first sub-table of each modification operation.
Specifically, in the method provided by the embodiment of the present invention, the merging server updates the specified baseline data stored in the off-heap memory based on the merging notification and the target file, including:
the method comprises the steps that a merging server obtains a target file and a first sub-table corresponding to the target file from an updating server based on merging notification, the target file is stored in an external memory, and index information corresponding to at least one modification operation in the target file is contained in the first sub-table;
the merge server updates the specified baseline data based on the merge notification, the target file, and the first sub-table.
Optionally, the merging server may acquire the specified modification operation in the target file one by one based on the index information in the first sub-table, and determine the specific position of the data to be modified in the specified baseline data based on the index information in the second sub-table; the merge server may then correspondingly modify the data based on the specified modification operation and the determined data location that needs modification.
Specifically, in the method provided by the embodiment of the present invention, the merging server updates the specified baseline data based on the merging notification, the target file and the first sub-table, including:
the merging server obtains at least one appointed modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table;
the merge server updates the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicative of the specified baseline data.
Specifically, the target file may store a modification operation such as adding or deleting data to or from a specified location of the specified baseline data, so that the merging server may perform a corresponding modification on the specified baseline data based on the read modification operation, thereby achieving the purpose of updating the specified baseline data.
Alternatively, after the specified baseline data is updated, the second sub-table may not correspond to the updated specified baseline data, and thus the merge server may generate a third sub-table containing location index information indicating the updated specified baseline data.
Specifically, after the merging server updates the specified baseline data based on the merging notification, the target file and the first sub-table, the method provided by the embodiment of the invention further comprises the following steps:
The merge server generates a third sub-table containing location index information indicating the updated specified baseline data.
Alternatively, in order to enable the configuration server to record and save version information of the baseline data to determine whether the baseline data is updated in real time, the merge server may want the configuration server to send an update completion notification after completion of updating the specified baseline data.
Specifically, the method provided by the embodiment of the invention further comprises the following steps:
the merging server sends an update completion notification to the configuration server, wherein the update completion notification comprises routing information of the appointed baseline data for completing the update and data version information of the appointed baseline data for completing the update;
the configuration server updates index information of the specified baseline data stored in the configuration server based on the routing information of the specified baseline data for which the update is completed, and updates data version information of the specified baseline data in the configuration server based on the data version information of the specified baseline data for which the update is completed.
Alternatively, in one embodiment, the merge server may generate a new file based on the updated specified baseline data.
Specifically, the off-heap memory may be divided into at least one storage area, and the file storing the specified baseline data before the update and the file storing the specified baseline data after the update may be respectively in different storage areas. Therefore, after the merge server can generate a new file based on the updated specified baseline data, the file storing the specified baseline data before update may not be deleted immediately, and may not be deleted until the next time the specified baseline data is updated, and the storage area where the file is located needs to be used.
Alternatively, if the amount of incremental update data is too large, resulting in a significant amount of time being spent updating the specified baseline data, a portion of the update tasks that do not require immediate execution may be scheduled during the shared memory system idle period in order to avoid overloading the shared memory system.
Specifically, similar to the method of updating specified baseline data described above, the target file may be stored directly in the specified off-heap memory, but the specified baseline data is not immediately updated based on the target file.
Optionally, since the shared memory system is often required to import a large amount of data, the update server may directly interact with the database, acquire the data from the database and generate the file, thereby improving the data import performance of the shared memory system.
Specifically, firstly, the update server can directly interact with a database, acquire data to be imported from the database, generate a file to be imported based on the data to be imported, store the file to be imported in a network disk, and wait for importing; then, the update server may send an import notification to the configuration server, causing the configuration server to send a copy notification to the merge server; finally, the merge server may import the file to be imported into the specified off-heap memory as baseline data based on the copy notification, and send an import completion notification to the configuration server, so that the configuration server updates the data version information of the baseline data.
It should be appreciated that the storage is at the update server. The files in the merging server and the configuration server can be in the form of memory mapping files, so that the performance of the shared memory server is improved.
Step 103, the configuration server responds to the read request for the specified baseline data sent by the client, and sends the specified baseline data to the client.
Alternatively, when updating the specified baseline data, in order to enable the client to read the latest version of the specified baseline data, and the incremental update data corresponding to the specified baseline data, may be transmitted to the client.
Specifically, in the method provided by the implementation of the present invention, the configuration server sends the specified baseline data to the client in response to the read request for the specified baseline data sent by the client, and the method includes:
the configuration server responds to a read request for the specified baseline data sent by the client and respectively sends read notifications to the update server and the merging server;
the update server transmitting incremental update data for the specified baseline data to the configuration server based on the read notification;
the merging server sends the specified baseline data to the configuration server based on the read notification;
The configuration server sends incremental update data for the specified baseline data and the specified baseline data to the client.
Optionally, if the client sends a read request to the shared memory system before the operation of updating the specified baseline data is completed, the configuration server may send the new delta update data, the target file, and the specified baseline data to the client.
Fig. 3 is a schematic diagram illustrating sending specified baseline data to a client in the data reading method according to the embodiment of the present invention.
Specifically, as shown in fig. 3, the update server sends the first sub-table and new incremental update data to the configuration server based on the read notification sent by the configuration server; the merging server sends the second sub-table to the configuration server based on the reading notification sent by the configuration server; finally, the configuration server sends the new delta update data, the target file, and the specified baseline data to the client.
Wherein the specified baseline data is data for completing updating, and the new incremental updating data is obtained by recording a modifying operation of the specified baseline data for incomplete updating by the updating server after the target file is generated.
Optionally, if the client sends a read request to the shared memory system after the operation of updating the specified baseline data is completed, the configuration server may send new incremental update data and the specified baseline data to the client.
Specifically, as in fig. 3, the update server transmits new incremental update data to the configuration server based on the read notification transmitted by the configuration server; the merging server sends a third sub-table to the configuration server based on the reading notification sent by the configuration server; finally, the configuration server sends the new delta update data to the client with the specified baseline data.
The specified baseline operation is based on the updated data of the target file, and the new incremental update data is obtained by the update server recording the modification operation of the updated specified baseline data after the generation of the target file.
By adopting the data reading method provided by the invention, the update server can generate the target file based on the increment update data when the data quantity of the increment update data in the update server is larger than or equal to the preset threshold value; then, the updating server sends a merging notification to the merging server, so that the merging server updates the specified baseline data stored in the external memory based on the merging notification and the target file; finally, the configuration server responds to the read request for the specified baseline data sent by the client, and sends the specified baseline data to the client. Therefore, no matter whether the merging server finishes updating the specified baseline data or not, the configuration server can send the specified baseline data of the latest version to the client, so that the performance and efficiency of the updating server, the merging server and the configuration server are improved, and the performance and efficiency of the client are also greatly improved by timely processing the request of the client.
The embodiment of the present invention further provides a data reading apparatus 400, as shown in fig. 4, including:
a generating unit 401, configured to generate, when a data amount of incremental update data in an update server is greater than or equal to a preset threshold, a target file based on the incremental update data by the update server;
an updating unit 402, configured to send, by the updating server, a merge notification to a merge server, so that the merge server updates specified baseline data stored in an out-of-heap memory based on the merge notification and the target file;
a sending unit 403, configured to send the specified baseline data to a client in response to a read request for the specified baseline data sent by the client by using a configuration server.
Optionally, in an embodiment, the updating unit 402 is configured to:
the update server sends a data change notification to the configuration server to notify the configuration server that the data version of the baseline data is changed;
the configuration server sends the merge notification to the merge server in response to the data change notification, such that the merge server updates the specified baseline data based on the merge notification and the target file.
Optionally, in an embodiment, the updating unit 402 is configured to:
the merging server acquires the target file and a first sub-table corresponding to the target file from the updating server based on the merging notification, and stores the target file in an external memory, wherein the first sub-table contains index information corresponding to at least one modification operation in the target file;
the merge server updates the specified baseline data based on the merge notification, the target file, and the first sub-table.
Optionally, in an embodiment, the updating unit 402 is configured to:
the merging server obtains at least one designated modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table;
the merge server updates the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicating the specified baseline data.
Optionally, in one embodiment, after the merging server updates the specified baseline data based on the merging notification, the target file, and the first sub-table, the electronic device further includes:
A second generating unit 404, configured to generate a third sub-table by using the merging server, where the third sub-table includes location index information indicating the updated specified baseline data.
Optionally, in one embodiment, the electronic device further includes:
a sending unit 405, configured to send, to the configuration server, an update completion notification, where the update completion notification includes routing information of the updated specified baseline data, and data version information of the updated specified baseline data;
a second updating unit 406, configured to update, by the configuration server, index information of the specified baseline data stored in the configuration server based on the routing information of the updated specified baseline data, and update, by the configuration server, data version information of the specified baseline data based on the data version information of the updated specified baseline data.
Optionally, in an embodiment, the sending unit 403 is configured to:
the configuration server responds to the read request for the specified baseline data sent by the client and respectively sends read notifications to the update server and the merging server;
The update server sending incremental update data for the specified baseline data to the configuration server based on the read notification;
the merge server sending the specified baseline data to the configuration server based on the read notification;
the configuration server sends the delta update data for the specified baseline data and the specified baseline data to the client.
The data reading apparatus 400 can implement the method of the method embodiment of fig. 1 to 3, and specifically, the method for reading data in the embodiment shown in fig. 1 to 3 may be referred to, which is not described herein.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to fig. 5, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 5, but not only one bus or type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the biometric authentication, identification and detection device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
The method performed by the data reading device disclosed in the embodiment shown in fig. 5 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
The electronic device may also execute the method of fig. 1, and implement the functions of the embodiment shown in fig. 1 or the functions of the embodiments shown in fig. 2 to 3 by using the data reading device, which are not described herein.
Of course, other implementations, such as a logic device or a combination of hardware and software, are not excluded from the electronic device of the present application, that is, the execution subject of the following processing flow is not limited to each logic unit, but may be hardware or a logic device.
The present embodiments also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiment of fig. 1, and in particular to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file;
And the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (7)
1. A method of reading data, comprising:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file; wherein the merge server updates specified baseline data stored in the off-heap memory based on the merge notification and the target file, comprising: the merging server acquires the target file and a first sub-table corresponding to the target file from the updating server based on the merging notification, and stores the target file in an external memory, wherein the first sub-table contains index information corresponding to at least one modification operation in the target file; the merging server obtains at least one designated modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table; the merge server updating the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicating the specified baseline data;
The merging server generates a third sub-table, wherein the third sub-table contains position index information indicating the updated appointed baseline data;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
2. The method of claim 1, wherein the updating server sending a merge notification to a merge server such that the merge server updates specified baseline data stored in off-heap memory based on the merge notification and the target file, comprising:
the update server sends a data change notification to the configuration server to notify the configuration server that the data version of the baseline data is changed;
the configuration server sends the merge notification to the merge server in response to the data change notification, such that the merge server updates the specified baseline data based on the merge notification and the target file.
3. The method of claim 1, wherein the method further comprises:
the merging server sends an update completion notification to the configuration server, wherein the update completion notification comprises routing information of the updated specified baseline data and data version information of the updated specified baseline data;
The configuration server updates index information of the specified baseline data stored in the configuration server based on the routing information of the updated specified baseline data, and updates data version information of the specified baseline data in the configuration server based on the data version information of the updated specified baseline data.
4. The method of claim 3, wherein the configuration server sending the specified baseline data to a client in response to a read request sent by the client for the specified baseline data, comprising:
the configuration server responds to the read request for the specified baseline data sent by the client and respectively sends read notifications to the update server and the merging server;
the update server sending incremental update data for the specified baseline data to the configuration server based on the read notification;
the merge server sending the specified baseline data to the configuration server based on the read notification;
the configuration server sends the delta update data for the specified baseline data and the specified baseline data to the client.
5. A data reading apparatus, comprising:
a generation unit configured to generate a target file based on incremental update data in an update server when a data amount of the incremental update data in the update server is greater than or equal to a preset threshold;
the updating unit is used for sending a merging notification to the merging server by the updating server, so that the merging server updates the specified baseline data stored in the out-of-heap memory based on the merging notification and the target file; wherein the merge server updates specified baseline data stored in the off-heap memory based on the merge notification and the target file, comprising: the merging server acquires the target file and a first sub-table corresponding to the target file from the updating server based on the merging notification, and stores the target file in an external memory, wherein the first sub-table contains index information corresponding to at least one modification operation in the target file; the merging server obtains at least one designated modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table; the merge server updating the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicating the specified baseline data;
A second generation unit, configured to generate a third sub-table by using the merging server, where the third sub-table includes location index information indicating the updated specified baseline data;
and the sending unit is used for responding to the reading request for the specified baseline data sent by the client and sending the specified baseline data to the client by the configuration server.
6. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file; wherein the merge server updates specified baseline data stored in the off-heap memory based on the merge notification and the target file, comprising: the merging server acquires the target file and a first sub-table corresponding to the target file from the updating server based on the merging notification, and stores the target file in an external memory, wherein the first sub-table contains index information corresponding to at least one modification operation in the target file; the merging server obtains at least one designated modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table; the merge server updating the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicating the specified baseline data;
The merging server generates a third sub-table, wherein the third sub-table contains position index information indicating the updated appointed baseline data;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
7. A computer-readable storage medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:
when the data volume of incremental update data in an update server is greater than or equal to a preset threshold value, the update server generates a target file based on the incremental update data;
the update server sends a merging notification to a merging server, so that the merging server updates specified baseline data stored in an out-of-heap memory based on the merging notification and the target file; wherein the merge server updates specified baseline data stored in the off-heap memory based on the merge notification and the target file, comprising: the merging server acquires the target file and a first sub-table corresponding to the target file from the updating server based on the merging notification, and stores the target file in an external memory, wherein the first sub-table contains index information corresponding to at least one modification operation in the target file; the merging server obtains at least one designated modification operation from at least one modification operation based on the merging notification and index information corresponding to the at least one modification operation in the target file in the first sub-table; the merge server updating the specified baseline data based on the at least one specified modification operation and a second sub-table containing location index information indicating the specified baseline data;
The merging server generates a third sub-table, wherein the third sub-table contains position index information indicating the updated appointed baseline data;
and the configuration server responds to a read request for the specified baseline data sent by the client and sends the specified baseline data to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010429746.XA CN113704274B (en) | 2020-05-20 | 2020-05-20 | Data reading method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010429746.XA CN113704274B (en) | 2020-05-20 | 2020-05-20 | Data reading method and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113704274A CN113704274A (en) | 2021-11-26 |
| CN113704274B true CN113704274B (en) | 2024-03-19 |
Family
ID=78645524
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010429746.XA Active CN113704274B (en) | 2020-05-20 | 2020-05-20 | Data reading method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113704274B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101650741A (en) * | 2009-08-27 | 2010-02-17 | 中国电信股份有限公司 | Method and system for updating index of distributed full-text search in real time |
| CN102024051A (en) * | 2010-12-17 | 2011-04-20 | 北京世纪互联工程技术服务有限公司 | Data updating method for distributed memory database |
| JP2012008934A (en) * | 2010-06-28 | 2012-01-12 | Kddi Corp | Distributed file system and redundancy method in distributed file system |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
| US9886443B1 (en) * | 2014-12-15 | 2018-02-06 | Nutanix, Inc. | Distributed NFS metadata server |
| CN107943594A (en) * | 2016-10-13 | 2018-04-20 | 北京京东尚科信息技术有限公司 | Data capture method and device |
| CN109697075A (en) * | 2017-10-20 | 2019-04-30 | 北京京东尚科信息技术有限公司 | File updating method, system and device |
| CN110888889A (en) * | 2018-08-17 | 2020-03-17 | 阿里巴巴集团控股有限公司 | Data information updating method, device and equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10657123B2 (en) * | 2015-09-16 | 2020-05-19 | Sesame Software | Method and system for reducing time-out incidence by scoping date time stamp value ranges of succeeding record update requests in view of previous responses |
| CN107291710B (en) * | 2016-03-30 | 2020-07-03 | 杭州海康威视数字技术股份有限公司 | Method and device for updating data for distributed database system |
-
2020
- 2020-05-20 CN CN202010429746.XA patent/CN113704274B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101650741A (en) * | 2009-08-27 | 2010-02-17 | 中国电信股份有限公司 | Method and system for updating index of distributed full-text search in real time |
| JP2012008934A (en) * | 2010-06-28 | 2012-01-12 | Kddi Corp | Distributed file system and redundancy method in distributed file system |
| CN102024051A (en) * | 2010-12-17 | 2011-04-20 | 北京世纪互联工程技术服务有限公司 | Data updating method for distributed memory database |
| US9886443B1 (en) * | 2014-12-15 | 2018-02-06 | Nutanix, Inc. | Distributed NFS metadata server |
| CN107943594A (en) * | 2016-10-13 | 2018-04-20 | 北京京东尚科信息技术有限公司 | Data capture method and device |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
| CN109697075A (en) * | 2017-10-20 | 2019-04-30 | 北京京东尚科信息技术有限公司 | File updating method, system and device |
| CN110888889A (en) * | 2018-08-17 | 2020-03-17 | 阿里巴巴集团控股有限公司 | Data information updating method, device and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113704274A (en) | 2021-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113032099B (en) | Cloud computing node, file management method and device | |
| CN110532106B (en) | Inter-process communication method, device, equipment and storage medium | |
| US20170153909A1 (en) | Methods and Devices for Acquiring Data Using Virtual Machine and Host Machine | |
| CN112395107A (en) | Tax control equipment control method and device, storage medium and electronic equipment | |
| CN111885184A (en) | Method and device for processing hot spot access keywords in high concurrency scene | |
| CN108512930B (en) | Shared file management method, device, server and storage medium | |
| CN114595186A (en) | Inter-core communication method and communication device of multi-core processor | |
| CN108228842B (en) | Docker mirror image library file storage method, terminal, device and storage medium | |
| CN115114042A (en) | Storage data access method and device, electronic equipment and storage medium | |
| CN113704274B (en) | Data reading method and electronic equipment | |
| CN111596864A (en) | Method, device, server and storage medium for data delayed deletion | |
| CN110764930A (en) | Request or response processing method and device based on message mode | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN112764897B (en) | Task request processing method, device and system and computer readable storage medium | |
| CN114741165A (en) | Processing method of data processing platform, computer equipment and storage device | |
| CN110474954B (en) | Block chain-based file sharing method and device, computer equipment and readable storage medium | |
| CN110928944B (en) | Data processing method and device | |
| US20140280347A1 (en) | Managing Digital Files with Shared Locks | |
| CN115695454B (en) | Data storage method, device and equipment of MEC host and storage medium | |
| CN116820430B (en) | Asynchronous read-write method, device, computer equipment and storage medium | |
| CN116150116B (en) | File system sharing method and device, electronic equipment and storage medium | |
| CN116528300B (en) | Task issuing method, device, system and communication equipment | |
| US11163537B1 (en) | Tiered application pattern | |
| CN110784479B (en) | Data verification method and device, electronic equipment and storage medium | |
| CN112306896B (en) | Garbage recycling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |