CN115686881A - Data processing method and device and computer equipment - Google Patents
Data processing method and device and computer equipment Download PDFInfo
- Publication number
- CN115686881A CN115686881A CN202211232291.8A CN202211232291A CN115686881A CN 115686881 A CN115686881 A CN 115686881A CN 202211232291 A CN202211232291 A CN 202211232291A CN 115686881 A CN115686881 A CN 115686881A
- Authority
- CN
- China
- Prior art keywords
- shared memory
- page number
- pages
- type
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data processing method and device and computer equipment. The method comprises the following steps: and responding to a first type data request of a first type process, determining whether the current read page number and the current written page number of the shared memory meet preset data processing conditions, wherein the first type process is a writing process or a reading process, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory. By adopting the method, the current written page number can be updated through the writing process, the current read page number is determined according to the updated current page number, and the current read page number is updated through the reading process, so that the shared memory streaming message is realized, the lock-free message communication and the lock-free circulating streaming message among the processes are realized, the additional consumption of resources caused by the lock communication is avoided, and the efficiency of the inter-process communication and the data processing efficiency are greatly improved.
Description
Technical Field
The present application relates to the field of shared memory technologies, and in particular, to a data processing method and apparatus, and a computer device.
Background
The existing processes often adopt various ways such as pipelines, semaphores, message queues, sockets and the like for communication. The shared memory is also one of methods for communication between processes, the shared memory is created by one process, a plurality of processes can access the shared memory, and kernel or system call is not relied on, so that the communication efficiency is high. However, the shared memory needs to synchronize messages of multiple processes accessing the shared memory in a synchronous manner, so that the read process and the write process are prevented from simultaneously performing read-write operation on the shared memory.
In the related art, a semaphore synchronization method or a mutex lock synchronization method is generally used to perform message synchronization on a plurality of processes accessing the shared memory. However, the above-mentioned synchronization methods bring extra performance overhead, resulting in extra consumption of resources.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data processing method, an apparatus and a computer device that can avoid additional consumption of resources.
In a first aspect, the present application provides a data processing method. The method comprises the following steps:
responding to a first type data request of a first type process, and determining whether the current read page number and the current written page number of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process;
and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory, wherein the target page number is the page number corresponding to the first type process, and the target page number is the current read page number or the current written page number of the shared memory.
Based on the scheme, the current write page number can be updated through the write process, the current read page number can be determined according to the updated current page number, and the current read page number can be updated through the read process, so that the memory flow type message can be shared, the lock-free message communication and the lock-free circulation flow type message among the processes can be realized, the additional consumption of resources caused by the lock communication is avoided, and the communication efficiency and the data processing efficiency among the processes are greatly improved.
In one embodiment, the first type process is a write process, the first type data request is a data write request, the first type pointer is a write pointer, and the preset data processing condition includes a preset write condition; before the step in the case where it is determined that the shared memory satisfies the preset data processing condition, the method further includes:
calculating the current written page number according to the data to be written corresponding to the writing process and the preset single page capacity;
calculating the remaining pages of the shared memory according to the total pages, the written pages and the read pages of the shared memory;
and determining that the shared memory meets a preset writing condition under the condition that the residual page number is greater than or equal to the current writing page number.
In this embodiment, by comparing the remaining number of pages of the shared memory with the current number of pages written in, it can be quickly and accurately determined whether the shared memory meets the preset writing condition, so as to ensure the timeliness of the writing process.
In one embodiment, the first type process is a read process, the first type data request is a data read request, the first type pointer is a read pointer, and the preset data processing condition includes a preset read condition; before the step in the case where it is determined that the shared memory satisfies the preset data processing condition, the method further includes:
and under the condition that the number of the read pages is less than the number of written pages of the shared memory, determining that the shared memory meets the preset reading condition.
In this embodiment, by comparing the number of read pages and the number of written pages of the shared memory, it can be quickly and accurately determined whether the shared memory meets the preset reading condition, and the timeliness of the read process processing is ensured.
In one embodiment, the updating the target number of pages of the shared memory includes:
and under the condition that the shared memory meets the preset writing condition, writing the data to be written corresponding to the writing process in the shared memory through a writing pointer corresponding to the shared memory, and updating the written page number of the shared memory corresponding to the writing process.
In the embodiment, the written pages of the writing process can be timely and accurately updated, and timely communication between the writing process and one or more reading processes is ensured.
In one embodiment, the method further comprises:
acquiring a first position of a written page number and a second position of a read page number of the shared memory;
calculating the distance between the first position and the second position according to the writing sequence of the shared memory;
and if the distance is less than the total page number of the shared memory, determining that the shared memory has available pages, and under the condition that the shared memory has the available pages, executing the step of calculating the current written page number according to the data to be written corresponding to the writing process and the preset single page capacity.
In this embodiment, by performing a pre-determination on whether there is an available page number in the shared memory, a writing condition of the shared memory may be predetermined, and a specific condition of the available page number of the shared memory is determined under a condition of consuming less computing resources.
In one embodiment, the method further comprises:
and if the distance is greater than or equal to the total page number of the shared memory, determining that no available page exists in the shared memory, and ending the writing process under the condition that no available page exists in the shared memory.
In this embodiment, by performing a pre-determination on whether there is an available page number in the shared memory, a writing condition of the shared memory may be predetermined, and a specific condition of the available page number of the shared memory is determined under a condition of consuming less computing resources.
In one embodiment, the updating the written page number of the shared memory corresponding to the writing process includes:
and calculating a first sum of the written page number of the shared memory and the current written page number, and determining that the first sum is the updated written page number of the shared memory.
In this embodiment, by updating the written page number in real time, the validity of communication among multiple processes corresponding to the shared memory can be ensured without a lock mechanism.
In one embodiment, the updating the target number of pages of the shared memory includes:
determining the current reading page number of the reading pointer;
and calculating a second sum of the read page number of the shared memory and the current read page number, and determining the second sum as the updated read page number of the shared memory.
In this embodiment, the validity of communication among multiple processes corresponding to the shared memory can be ensured under a lock-free mechanism by updating the number of read pages in real time.
In a second aspect, the present application further provides a data processing apparatus. The device comprises:
the first determining module is used for responding to a first type data request of a first type process, and determining whether the number of read pages and the number of written pages of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are conditions corresponding to the first type process;
and the second determining module is used for performing the first type operation corresponding to the first type data request in the shared memory through the first type pointer corresponding to the shared memory under the condition that the shared memory is determined to meet the preset data processing condition, and updating at least one of the read page number and the written page number of the shared memory.
In a third aspect, the application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
responding to a first type data request of a first type process, and determining whether the current read page number and the current written page number of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process;
and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating a target page number of the shared memory, wherein the target page number is a page number corresponding to the first type process, and the target page number is a current read page number or a current written page number of the shared memory.
In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
responding to a first type data request of a first type process, and determining whether the current read page number and the current written page number of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process;
and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory, wherein the target page number is the page number corresponding to the first type process, and the target page number is the current read page number or the current written page number of the shared memory.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
responding to a first type data request of a first type process, and determining whether a current read page number and a current written page number of a shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process;
and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory, wherein the target page number is the page number corresponding to the first type process, and the target page number is the current read page number or the current written page number of the shared memory.
The data processing method, the data processing device and the computer equipment comprise the following steps: responding to a first type data request of a first type process, and determining whether the current read page number and the current written page number of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process; and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory, wherein the target page number is the page number corresponding to the first type process, and the target page number is the current read page number or the current written page number of the shared memory. By adopting the method, the current written page number can be updated through the writing process, the current read page number can be determined according to the updated current page number, and the current read page number can be updated through the reading process, so that the memory flow type message sharing can be realized, the lock-free message communication and the lock-free circulation flow type message among the processes can be realized, the additional consumption of resources caused by the lock communication is avoided, and the communication efficiency and the data processing efficiency among the processes are greatly improved.
Drawings
FIG. 1 is a flow diagram illustrating a data processing method according to one embodiment;
FIG. 2 is a flowchart illustrating steps of determining that a predetermined writing condition is satisfied according to an embodiment;
FIG. 3 is a flowchart of the step of determining if there are available pages in one embodiment;
FIG. 4 is a flowchart illustrating the step of updating the number of pages read in one embodiment;
FIG. 5 is a schematic diagram of a header block in one embodiment;
FIG. 6 is a block diagram illustrating the structure of a shared memory stream state block in one embodiment;
FIG. 7 is a flowchart illustrating the initialization steps of the write process in one embodiment;
FIG. 8 is a flowchart illustrating an initialization step of the read process in one embodiment;
FIG. 9 is a diagram illustrating the structure of a single piece of data in one embodiment;
FIG. 10 is a flow chart illustrating a data writing step according to an embodiment;
FIG. 11 is a flowchart illustrating the data reading step in one embodiment;
FIG. 12 is a block diagram showing the structure of a data processing apparatus according to an embodiment;
FIG. 13 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In an embodiment, as shown in fig. 1, a data processing method is provided, and this embodiment is illustrated by applying this method to a terminal, it is to be understood that this method may also be applied to a server, and may also be applied to a system including a terminal and a server, and is implemented through interaction between the terminal and the server, where the terminal may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like, and the server may be implemented by an independent server or a server cluster formed by multiple servers. In this embodiment, the data processing method includes the following steps:
step 102, in response to the first type data request of the first type process, determining whether the current read page number and the current written page number of the shared memory meet a preset data processing condition.
The preset data processing condition is a processing condition corresponding to a first type process, and the first type process is a writing process or a reading process.
In this embodiment, when the terminal receives a first type data request corresponding to a first type process, the terminal may obtain a current read page number and a current written page number of a preset shared memory based on the first type data request, and determine whether the read page number and the written page number of the preset shared memory satisfy a preset data processing condition corresponding to the first type process.
In one example, the first type process may be a write process, where the write process is configured to write data into the predetermined shared memory, and the first type process may also be a read process, where the read process is configured to read the written data in the shared memory after the data has been written in the predetermined shared memory.
The first type data request may be a data request initiated by a first type process, and in the case that the first type process is a write process, the write process may generate a data write request and send the data write request to the terminal; in the case where the first type of process is a read process, the read process may generate a data read request and transmit the data read request to the terminal. At the current moment, the read page number of the shared memory is the read page number of the shared memory obtained after the terminal processes the shared memory based on the data reading request of the reading process; similarly, the written page number is the written page number of the shared memory obtained after the terminal processes the shared memory based on the data write request of the write process.
And step 104, under the condition that the shared memory meets the preset data processing condition, performing a first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory.
The target page number is a page number corresponding to the first type process, and the target page number is a current read page number or a current written page number of the shared memory.
In this embodiment, in response to a first type data request corresponding to a first type process, a terminal may obtain the number of read pages and the number of written pages of a shared memory, and determine whether the number of read pages and the number of written pages of the shared memory satisfy a preset data processing condition corresponding to the first type process. Under the condition that the terminal determines that the preset data processing condition is met, the terminal updates the target page number of the shared memory based on the operation result of the first type operation on the first type operation corresponding to the first type data request in the shared memory through the first type pointer based on the first type pointer corresponding to the first type data request.
Alternatively, in the case where the first type process is a write process, the target number of pages may be the number of written pages; in the case where the first type process is a read process, the target number of pages may be the number of read pages.
In one example, the terminal may create a header shared memory block, a status memory block, and multiple data memory blocks based on a write process. The head shared memory block is an annular shared memory block, and the annular shared memory block comprises a plurality of memory pages connected end to end; the status memory block includes status data of a plurality of processes (threads), such as the written pages of a writing process and the read pages of a reading process. The terminal may read the state data corresponding to the first type process from the state memory block, and determine the current written number of pages and the read number of pages of the shared memory, and the like, in case that the terminal needs to obtain the written number of pages and the read number of pages of the shared memory in response to the first type data request of the first type process. The multiple data memory blocks may store data corresponding to each process, for example, the data memory block corresponding to the write process may store data that the write process needs to write into the shared memory, and similarly, the data memory block corresponding to the read process may store data that the read process reads from the shared memory. Since the read process reads the data written by the write process in the shared memory, the start condition of the read process is that the initialization processing of the write process is completed.
A possible implementation manner may be that the terminal determines, through the write process, whether the current head shared memory block is the shared memory block that has already been created, and after determining that the current head shared memory block is the shared memory block that has already been created, the terminal may determine, based on the data stored in the state memory block, a write pointer corresponding to a write process that satisfies a preset data processing condition in the shared memory (shared memory block), process the shared memory based on the write pointer, and update the number of written pages of the shared memory, that is, update the number of written pages corresponding to the write process in real time. Therefore, the terminal can respond to the data reading request corresponding to the reading process, read the data in the state memory block, determine the reading pointer corresponding to the reading process meeting the preset data processing conditions, process the shared memory based on the reading pointer, and update the read page number of the shared memory. Therefore, by updating the read page number and the written page number in the state memory block in real time, the communication of the circulating flow type message between the writing process and the reading process can be realized, the communication efficiency is ensured, and the additional consumption of resources is avoided.
In the data processing method, in response to a first type data request of a first type process, whether a current read page number and a current written page number of a shared memory meet a preset data processing condition is determined, wherein the preset data processing condition is a processing condition corresponding to the first type process, and the first type process is a writing process or a reading process. And under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating a target page number of the shared memory, wherein the target page number is a page number corresponding to the first type process, and the target page number is a current read page number or a current written page number of the shared memory. By adopting the method, the current write page number can be updated through the write process, the current read page number can be determined according to the updated current page number, and the current read page number can be updated through the read process, so that the memory flow type message can be shared, the lock-free message communication and the lock-free circulation flow type message among the processes can be realized, the additional consumption of resources caused by the lock communication can be avoided, and the communication efficiency and the data processing efficiency among the processes can be greatly improved.
In one embodiment, the first type of process is a write process, the first type of data request is a data write request, the first type of pointer is a write pointer, and the predetermined data processing condition includes a predetermined write condition.
Accordingly, as shown in fig. 2, before the step in the case that the shared memory is determined to satisfy the preset data processing condition, the data processing method further includes:
step 202, calculating the current write page number according to the data to be written corresponding to the write process and the preset single page capacity.
The data to be written corresponding to the write process is data that the write process needs to write into the shared memory, multiple pages can be configured in the shared memory, and the preset single-page capacity is the data capacity of each page.
In this embodiment, the terminal may obtain the data size of the data to be written corresponding to the write process, and obtain the corresponding preset single page capacity of the shared memory. Therefore, the terminal can calculate the ratio of the data to be written and the preset single page capacity, and determine that the ratio is the number of pages occupied by the writing process which needs to write the data to be written into the shared memory, namely the current written page number.
In step 204, the remaining number of pages of the shared memory is calculated according to the total number of pages of the shared memory, the written number of pages and the read number of pages.
The total number of pages of the shared memory may be configured in advance, the written number of pages is the number of pages of the shared memory to which data has been written, and the read number of pages is the number of pages of the shared memory to which data has been read by the read pointer corresponding to the read process.
In this embodiment, the terminal may calculate a difference between the total number of pages of the shared memory and the number of written pages, and determine the difference as the initial remaining number of pages. Since the number of pages read by the read pointer can be recycled, the terminal can calculate the sum of the initial remaining number of pages and the number of pages that have been read, and use the sum as the remaining number of pages of the shared memory.
In step 206, when the remaining page count is greater than or equal to the current write page count, it is determined that the shared memory meets the preset write condition.
In this embodiment, the terminal needs to compare the shared memory with the current number of pages written in, obtain a comparison result, and determine whether the current shared memory meets the preset data processing condition based on the comparison result. Based on this, the terminal may determine that the shared memory satisfies a preset writing condition (preset data processing condition) when determining that the number of remaining pages is greater than or equal to the current number of written pages.
In this embodiment, by comparing the remaining number of pages of the shared memory with the current number of pages written in, it can be quickly and accurately determined whether the shared memory meets the preset writing condition, so as to ensure the timeliness of the writing process.
In one embodiment, the first type process is a read process, the first type data request is a data read request, the first type pointer is a read pointer, and the preset data processing condition includes a preset read condition.
Accordingly, before the step "in the case that it is determined that the shared memory satisfies the preset data processing condition", the data processing method further includes:
and under the condition that the number of the read pages is less than the number of the written pages of the shared memory, determining that the shared memory meets a preset reading condition.
In this embodiment, the terminal may compare the number of read pages with the number of written pages of the shared memory to obtain a comparison result, and determine whether the current shared memory meets the preset data processing condition based on the comparison result. Based on this, in a case that the first type process is a read process, the terminal may determine that the preset data processing condition is a preset read condition, and in a case that the comparison result is that the number of read pages is smaller than the number of written pages of the shared memory, the terminal may determine that the current shared memory has a memory page that can be read, that is, it is determined that the current shared memory meets the preset read condition.
In one example, the shared memory may be configured with ten memory pages, each memory page may be identified according to an order of 0 to 9, and the preset writing order may be a writing order from 0 to 9. For example, the written page number of the shared memory may be 6, the current read page number of the shared memory may be 4, and since the current read page number of the shared memory is smaller than the written page number, the terminal may determine that the current shared memory meets the preset reading condition.
In this embodiment, by comparing the number of read pages and the number of written pages of the shared memory, whether the shared memory meets the preset reading condition can be determined quickly and accurately, and the timeliness of the read process processing is ensured.
In an embodiment, the specific process of "updating the target number of pages of the shared memory" in step 104 includes:
and under the condition that the shared memory meets the preset writing condition, writing the data to be written corresponding to the writing process in the shared memory through the writing pointer corresponding to the shared memory, and updating the written page number of the shared memory corresponding to the writing process.
In this embodiment, the terminal may determine whether the shared memory meets the preset write-in condition, and the specific process of determining may be that the terminal determines whether a write pointer of the shared memory meets the preset write-in condition. Therefore, under the condition that the terminal determines that the write pointer in the shared memory meets the preset write condition, the terminal can write the data to be written stored in the data memory block of the write process into the shared memory through the write pointer of the shared memory, so as to complete the data write task corresponding to the write process.
After the terminal writes the data to be written into the shared memory through the write pointer, the terminal can update the written page number of the write process, and the updated written page number of the write process is used as the updated written page number of the shared memory.
In the embodiment, the written pages of the writing process can be timely and accurately updated, and timely communication between the writing process and one or more reading processes is ensured.
In one embodiment, as shown in fig. 3, the data processing method includes:
in step 302, a first position of the written page number and a second position of the read page number of the shared memory are obtained.
In this embodiment, the terminal may obtain the written page number and the read page number of the shared memory at the current time. Because the shared memory is configured with a plurality of memory pages, and the plurality of memory pages are identified according to a preset writing sequence (according to a writing sequence from 0 to 9), the identification information of each memory page is obtained. Therefore, the terminal can acquire the first position of the written number of pages and the second position of the read number of pages.
Step 304, calculating the distance between the first position and the second position according to the writing sequence of the shared memory.
The writing sequence of the shared memory may be a preset writing sequence, the shared memory may be configured with a plurality of memory pages, and each memory page may be sequentially identified according to a sequence from small to large, so that the writing sequence of the shared memory may be a writing sequence from small identification to large identification.
In this embodiment, the terminal may calculate the distance between the first position of the written page number and the second position of the read page number based on the writing sequence of the shared memory, and the specific process of calculating the distance may be: and the terminal determines a second position of the read page number in the annular shared memory, determines the number of the memory pages existing between the second position and the first position of the written page number according to the writing direction corresponding to the writing sequence of the shared memory, and takes the number of the memory pages as the distance between the first position and the second position.
In one example, 10 memory pages may be configured in the shared memory, each memory page may be identified from 0 to 9, the identification information of the first memory page may be 0, the identification information of the second memory page may be 1, \ 8230, and the identification information of the tenth memory page may be 9, so that the writing order of the shared memory may be from small to large. For example, the first position of the written page number of the shared memory may be 7, and the second position of the read page number may be 6, so that the distance between the first position and the second position calculated by the terminal may be 0; the first position of the written page number of the shared memory may be 6, and the second position of the read page number may be 7, so that the distance between the first position and the second position calculated by the terminal may be 10.
Step 306, if the distance is less than the total number of pages of the shared memory, determining that the shared memory has available pages, and under the condition that the shared memory has available pages, performing a step of calculating the current number of pages to be written according to the data to be written corresponding to the writing process and the preset single page capacity.
In this embodiment, the terminal may compare the distance between the first location and the second location with the total number of pages of the shared memory, and in a case that the terminal determines that the distance between the first location and the second location is less than the total number of pages of the shared memory, the terminal may determine that there is an available page in the shared memory. Based on this, when the terminal determines that the shared memory has available pages, the terminal may execute the step of calculating the current number of pages to be written, according to the data to be written corresponding to the writing process and the preset single page capacity provided in the foregoing embodiment.
In this embodiment, by performing a preliminary determination on whether there is an available page number in the shared memory, a writing condition of the shared memory may be determined in advance, and a specific condition of the available page number of the shared memory is determined in the case of consuming less computing resources.
In one embodiment, the data processing method includes:
and if the distance is larger than or equal to the total page number of the shared memory, determining that no available page exists in the shared memory, and ending the writing process under the condition that the shared memory is determined not to have the available page.
In this embodiment, the terminal may compare the distance between the first location and the second location with the total number of pages of the shared memory, and when the terminal determines that the distance between the first location and the second location is greater than or equal to the total number of pages of the shared memory, the terminal may determine that no available page exists in the shared memory. Based on this, the terminal may end the write process, may generate a notification message of the end of the data read request, and output the notification message of the end of the data read request.
In this embodiment, by performing a preliminary determination on whether there is an available page number in the shared memory, a writing condition of the shared memory may be determined in advance, and a specific condition of the available page number of the shared memory is determined in the case of consuming less computing resources.
In one embodiment, the specific process of updating the written page number of the shared memory corresponding to the writing process includes:
and calculating a first sum of the written page number and the current written page number of the shared memory, and determining the first sum as the updated written page number of the shared memory.
In this embodiment, the terminal may obtain the written page number of the shared memory and the current written page number corresponding to the data to be written, calculate a first sum of the written page number and the current written page number, and use the first sum as the updated written page number.
In an example, the written page number of the shared memory may be 3, and the current written page number corresponding to the data to be written may also be 3, so that the first sum of the written page number and the current written page number may be 6, and the updated written page number is 6.
In this embodiment, by updating the written page number in real time, the validity of communication among multiple processes corresponding to the shared memory can be ensured without a lock mechanism.
In an embodiment, as shown in fig. 4, the specific process of the step "updating the target page number of the shared memory" includes:
at step 402, the current number of pages read by the read pointer is determined.
In step 404, a second sum of the read page number of the shared memory and the current read page number is calculated, and the second sum is determined to be the updated read page number of the shared memory.
In this embodiment, when it is determined that the first type process is a read process, the terminal may obtain a current read page number corresponding to the read pointer, where the current read page number is a number of memory pages read by the read pointer in a data reading operation process. In this way, the terminal may calculate a second sum of the read page number and the current read page number, and use the second sum as the updated read page number of the shared memory, that is, may also use the second sum as the read page number of the updated read pointer.
In an example, the number of read pages of the shared memory may be 3, and the number of memory pages read by the read process in the data reading process may be 3, so that the second sum of the number of read pages and the current number of read pages may be 6, and the updated number of read pages is 6.
In this embodiment, the validity of communication among multiple processes corresponding to the shared memory can be ensured under a lock-free mechanism by updating the number of read pages in real time.
The specific implementation of the data processing method is described in detail below with reference to a detailed embodiment:
as shown in fig. 5, the storage structure of the shared memory is schematically illustrated, where the storage structure includes a header block and header block information, the header block is a header shared memory block, and includes a plurality of memory pages; the header block information stores therein various information such as the number of pages information (number of pages, total _ pages), information of a preset single-page capacity (size of page, page _ size), the number of bottom pages (bottom _ page), and the number of shared memory streams (stream _ count). The page number information is the number of a plurality of memory pages included in the header block, the preset single-page capacity may be the capacity of each memory page, the bottom page is smaller than the configured number of pages, and the bottom page exists to avoid writing through the memory.
The terminal may calculate the number of pages total _ pages by the following formula:
total_pages＝2 page_power
wherein page _ power represents the total number of pages,
the terminal may calculate the size of the page _ size by the following formula:
page_size＝2 page_size_power
wherein page _ size _ power represents the total size of the page;
specifically, as shown in fig. 6, one write process and M read processes communicate with each other, and M shared memories (shared memory streams) may be created, where each shared memory stream may include a plurality of small memory blocks (memory pages), and the terminal may obtain identification information (stream ID) of each shared memory stream, where the stream ID is used to identify the shared memory blocks used by one write process and one read process. The shared memory stream state block comprises a plurality of shared streams and state information of each shared stream; the shared memory stream state may include a state of shared stream 1, a state of shared stream 2, \8230, and a state of shared stream M, each shared stream includes a plurality of memory pages, and the state information corresponding to each shared stream includes a written page number (written page number, tx _ pages) and a read page number (read page number, rw _ pages), and identification information of the memory block. In shared memory, the actual data memory is stored in pages, where each memory page can be recycled.
Before the step of performing data processing based on the data write request of the write process, the terminal needs to perform write process initialization, as shown in fig. 7, the process of write process initialization includes:
in an example, after receiving the write process, the terminal may determine whether the current write process is attached to the head shared memory block, that is, determine whether the current write process successfully creates the head shared memory block.
If the attachment fails, a head shared memory block and a shared memory stream status block are created, and then the step of determining whether the current write process is attached by the head shared memory block is executed again.
If the attachment is successful, determining that the write process successfully creates the head shared memory block, performing shared memory configuration consistency check, namely checking whether the configuration information in the head block is consistent with the configuration information of the current write process, if the configuration information in the head block is consistent with the configuration information of the current write process, executing a step of attaching a shared memory stream state block, creating a data shared memory block corresponding to the memory stream according to the stream ID of the shared stream, performing attachment (stream _ shm _ addr), and updating the data block ID in the corresponding memory stream state block.
If the configuration information in the header block is inconsistent with the configuration information of the current write process, a header shared memory block and a shared memory stream state block are created, and then the step of judging whether the current write process is attached by the header shared memory block is executed again.
After the write process initialization is completed, the terminal needs to perform a process of initializing the read process, as shown in fig. 8:
the terminal attaches the head part shared memory block through the reading process, and if the attachment of the head part shared memory block fails, the reading process is ended; if the attachment of the head shared memory block is successful, acquiring configuration information of the head shared memory block, and attaching a shared memory stream state block based on the configuration information;
if the read process fails to attach the shared memory stream state block, the read process is ended; if the attaching of the shared memory stream status block is successful, circularly attaching all memory pages (stream _ shm _ addr), that is, circularly attaching all data blocks included in the shared stream, to the plurality of memory pages included in each shared stream according to the stream IDs of the plurality of shared streams included in the shared memory stream status block.
In one example, as shown in fig. 9, the structure diagram of a single piece of data is shown, each piece of data includes two parts, namely a data header and actual content of the data, wherein the data header includes a data length and a number of pages occupied by the data (data _ page), and the data length is a length (data _ len) of the actual data.
In one example, as shown in fig. 10, it is a schematic flow chart of data writing, and the data writing process includes:
the terminal responds to a data write-in request corresponding to a write process, judges whether an available page exists, and under the condition that the available page exists in the current shared stream, the shared memory block acquires a writable memory pointer, acquires data filling with a specified length, and updates the number of occupied memory pages; and judging whether the occupied page number is less than the residual page number. Under the condition that the occupied page number is less than the residual page number, performing data writing operation, and judging whether the written data is submitted or not; if the already written data has been committed, the number of used pages (number of written pages) is updated, and after the step of updating the number of used pages, the writing process is ended.
If no available page exists, directly ending the writing process; if the occupied page number is larger than or equal to the residual page number, initiating a rollback flow and ending the writing process; if the written data is not submitted, the step of 'obtaining data filling with specified length' is executed again.
Specifically, the terminal, in response to a data write request in a write process, determines whether an available page exists in the current shared memory, and the specific determination process may be: if tx _ pages-rx _ pages > = total _ pages, it means that there is no memory page currently available, where tx _ pages is the number of written pages, rx _ pages is the number of read pages, and total _ pages is the total number of pages of the shared memory stream.
In the case where there are available pages, the step of performing writing and committing of the actual data comprises:
s1a, acquiring identification Information (ID) of a last write page, and calculating the identification information of the last write page by the following formula: the last written page ID (page _ ID) = total written page number (tx _ pages) & ((1 UL < < total index of page (page _ power)) -1). Where, & denotes an AND operation, < < denotes a left shift operation, and 1< < page _ power is equivalent to 2 to the power of page _ power.
S1b, a writable memory pointer (ptr) is obtained.
Memory pointer (ptr) = shared memory start address (stream _ shm _ addr) + last written page ID (page _ ID) < < total index of page size (page _ size _ power)
S1c, the number of occupied pages (pages) may be updated according to the total length of data written, wherein the total length of data written may be the sum of the length total _ len of uncommitted data written and the length len of data to be written, and the number of occupied pages (pages) may be calculated by the following formula:
the number of occupied pages = total data length (total _ len) > > total index of page size (page _ size _ power) + total data length (total _ len) & (1 < < total index of page size (page _ size _ power) -1). Where, & denotes an AND operation, < < < denotes a left shift operation.
S1d, calculating the current residual page number of the shared memory, judging whether the residual page number is enough to write current data, and if so, executing the step S1f; otherwise, performing a rollback flow and ending the writing process;
written page number (tx _ pages) + number of occupied pages (pages) -read page number (rx _ pages) < = total page number (total _ pages).
S1f, if data are required to be filled, executing the step S1c; otherwise, step S1g is performed.
S1g, updating the written page number, submitting data and ending the writing process.
In one example, as shown in fig. 11, it is a schematic flow chart of data reading, and the process of data reading includes:
the terminal responds to the reading process and traverses the shared memory block; the terminal circularly traverses all the shared memory stream state blocks. For each shared stream state block in the state blocks, the step "get readable data pointer (ptr)" is performed.
Judging whether an unread memory page exists or not; the terminal traverses all the shared memory blocks, and determines whether memory pages which have been written with data and are not read exist in a plurality of memory pages included in the shared memory blocks. And if the unread memory pages exist, acquiring a read memory pointer and processing the read data. If there are no memory pages that have not been read, the read process ends.
And judging whether all the memory blocks are traversed or not, and ending the reading process if all the memory blocks are traversed. And if all the memory blocks have not been traversed, re-executing the step of "traversing all the shared memory blocks".
Specifically, the specific process of obtaining the readable data pointer (ptr) may include:
s1, if a plurality of memory pages contained in a shared stream (a preset shared memory) have readable pages, executing S2; and if the readable page does not exist in the plurality of memory pages contained in the shared stream, ending the reading process.
In the case that the written page number (tx _ pages) is greater than the read page number (rx _ pages), the terminal may determine that the readable page exists in the plurality of memory pages included in the shared stream.
S2, acquiring the ID of the page read last time, and calculating the ID (page _ ID) of the page read last time through the following formula:
last read page ID = total read page number (rx _ pages) & ((1 UL < < total index of pages (page _ power)) -1). Wherein (1 UL < < total index of pages (page _ power)) -1 represents 2^ page _power-1.
S3, acquiring a readable data pointer (ptr), and calculating the readable data pointer (ptr) through the following formula:
pointer (ptr) = shared memory start address (stream _ shm _ addr) + last read page ID (page _ ID) < < total index of page size (page _ size _ power)
And S4, acquiring the length (len) of the current data, the number of pages (pages) occupied by the current data and the read number of pages from the data header, executing data reading operation through a preset relevant service processing function of data reading, and updating the read number of pages after the data reading operation is completed.
Compared with the traditional interprocess communication methods such as pipelines, message queues and sockets, the data processing method provided by the embodiment of the invention does not need copying of a kernel state, does not have various system calls, and greatly improves the interprocess communication efficiency. Compared with other locked shared memories, frequent locking operation is avoided, and the synchronous performance of the shared memories can be improved through the design of the lockless circulating flow type messages.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a data processing apparatus for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more embodiments of the data processing device provided below may refer to the limitations on the data processing method in the above description, and are not described herein again.
In one embodiment, as shown in fig. 12, there is provided a data processing apparatus 1200 comprising: a first determining module 1201 and a second determining module 1202, wherein:
a first determining module 1201, configured to determine, in response to a first type data request of a first type process, whether the number of read pages and the number of written pages of the shared memory meet a preset data processing condition, where the preset data processing condition is a condition corresponding to the first type process;
the second determining module 1202 is configured to, when it is determined that the shared memory meets the preset data processing condition, perform a first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and update at least one of a number of read pages and a number of written pages of the shared memory.
The various modules in the data processing apparatus 1200 described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 13. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The database of the computer device is used for storing the related data of the shared memory. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.
Those skilled in the art will appreciate that the architecture shown in fig. 13 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, carries out the steps in the method embodiments described above.
It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases involved in the embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A method of data processing, the method comprising:
responding to a first type data request of a first type process, and determining whether a current read page number and a current written page number of a shared memory meet preset data processing conditions, wherein the preset data processing conditions are processing conditions corresponding to the first type process, and the first type process is a writing process or a reading process;
and under the condition that the shared memory meets the preset data processing condition, performing first type operation corresponding to the first type data request in the shared memory through a first type pointer corresponding to the shared memory, and updating the target page number of the shared memory, wherein the target page number is the page number corresponding to the first type process, and the target page number is the current read page number or the current written page number of the shared memory.
2. The method of claim 1, wherein the first type process is a write process, the first type data request is a data write request, the first type pointer is a write pointer, and the predetermined data processing condition comprises a predetermined write condition; before the step in the case where it is determined that the shared memory satisfies the preset data processing condition, the method further includes:
calculating the current written page number according to the data to be written corresponding to the writing process and the preset single page capacity;
calculating the remaining pages of the shared memory according to the total pages, the written pages and the read pages of the shared memory;
and determining that the shared memory meets a preset writing condition under the condition that the residual page number is greater than or equal to the current writing page number.
3. The method of claim 1, wherein the first type of process is a read process, the first type of data request is a data read request, the first type of pointer is a read pointer, and the predetermined data processing condition comprises a predetermined read condition; before the step in the case where it is determined that the shared memory satisfies the preset data processing condition, the method further includes:
and under the condition that the number of the read pages is less than the number of the written pages of the shared memory, determining that the shared memory meets the preset reading condition.
4. The method according to claim 2, wherein the performing, by using the first type pointer corresponding to the shared memory, the first type operation corresponding to the first type data request in the shared memory and updating the target number of pages of the shared memory when it is determined that the shared memory meets the preset data processing condition includes:
and under the condition that the shared memory meets the preset writing condition, writing the data to be written corresponding to the writing process in the shared memory through a writing pointer corresponding to the shared memory, and updating the written page number of the shared memory corresponding to the writing process.
5. The method of claim 2, further comprising:
acquiring a first position of a written page number and a second position of a read page number of the shared memory;
calculating the distance between the first position and the second position according to the writing sequence of the shared memory;
and if the distance is less than the total page number of the shared memory, determining that the shared memory has available pages, and executing the step of calculating the current written page number according to the data to be written corresponding to the writing process and the preset single page capacity under the condition of determining that the shared memory has the available pages.
6. The method of claim 5, further comprising:
and if the distance is larger than or equal to the total page number of the shared memory, determining that no available page exists in the shared memory, and under the condition that the shared memory is determined not to have the available page, ending the writing process.
7. The method according to claim 4, wherein the updating the written page number of the shared memory corresponding to the writing process includes:
and calculating a first sum of the written page number of the shared memory and the current written page number, and determining that the first sum is the updated written page number of the shared memory.
8. The method of claim 3, wherein updating the target number of pages of the shared memory comprises:
determining the current reading page number of the reading pointer;
and calculating a second sum of the read page number of the shared memory and the current read page number, and determining the second sum as the updated read page number of the shared memory.
9. A data processing apparatus, characterized in that the apparatus comprises:
the first determining module is used for responding to a first type data request of a first type process, and determining whether the number of read pages and the number of written pages of the shared memory meet preset data processing conditions, wherein the preset data processing conditions are conditions corresponding to the first type process;
and a second determining module, configured to, when it is determined that the shared memory meets a preset data processing condition, perform, through a first type pointer corresponding to the shared memory, a first type operation corresponding to a first type data request in the shared memory, and update at least one of a number of read pages and a number of written pages of the shared memory.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211232291.8A CN115686881A (en) | 2022-10-10 | 2022-10-10 | Data processing method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211232291.8A CN115686881A (en) | 2022-10-10 | 2022-10-10 | Data processing method and device and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115686881A true CN115686881A (en) | 2023-02-03 |
Family
ID=85065500
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211232291.8A Pending CN115686881A (en) | 2022-10-10 | 2022-10-10 | Data processing method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115686881A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858421A (en) * | 2023-03-01 | 2023-03-28 | 浪潮电子信息产业股份有限公司 | Cache management method, device, equipment, readable storage medium and server |
| CN117407356A (en) * | 2023-12-14 | 2024-01-16 | 芯原科技(上海)有限公司 | Inter-core communication method and device based on shared memory, storage medium and terminal |
-
2022
- 2022-10-10 CN CN202211232291.8A patent/CN115686881A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858421A (en) * | 2023-03-01 | 2023-03-28 | 浪潮电子信息产业股份有限公司 | Cache management method, device, equipment, readable storage medium and server |
| CN117407356A (en) * | 2023-12-14 | 2024-01-16 | 芯原科技(上海)有限公司 | Inter-core communication method and device based on shared memory, storage medium and terminal |
| CN117407356B (en) * | 2023-12-14 | 2024-04-16 | 芯原科技(上海)有限公司 | Inter-core communication method and device based on shared memory, storage medium and terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021017421A1 (en) | Blockchain state data recovery method and device, and electronic device | |
| CN115686881A (en) | Data processing method and device and computer equipment | |
| US11386065B2 (en) | Database concurrency control through hash-bucket latching | |
| CN104572568B (en) | Read lock operation method, write lock operation method and system | |
| US20110185359A1 (en) | Determining A Conflict in Accessing Shared Resources Using a Reduced Number of Cycles | |
| CN110188110B (en) | Method and device for constructing distributed lock | |
| CN110888727A (en) | Method, device and storage medium for realizing concurrent lock-free queue | |
| CN107025130B (en) | Processing node, computer system and transaction conflict detection method | |
| CN115408411A (en) | Data writing method and device, electronic equipment and storage medium | |
| US10101999B2 (en) | Memory address collision detection of ordered parallel threads with bloom filters | |
| CN116719646A (en) | Hot spot data processing method, device, electronic device and storage medium | |
| WO2016049807A1 (en) | Cache directory processing method and directory controller of multi-core processor system | |
| US11132351B2 (en) | Executing transactions based on success or failure of the transactions | |
| CN116225314A (en) | Data writing method, device, computer equipment and storage medium | |
| US10678453B2 (en) | Method and device for checking false sharing in data block deletion using a mapping pointer and weight bits | |
| US8977814B1 (en) | Information lifecycle management for binding content | |
| CN111435331A (en) | Data writing method and device for storage volume, electronic equipment and machine-readable storage medium | |
| CN116257531B (en) | Database space recovery method | |
| CN115202884B (en) | Method for adding read write lock of high-performance system based on polling and application | |
| CN115543970B (en) | Data page processing method, data page processing device, electronic equipment and storage medium | |
| US10776344B2 (en) | Index management in a multi-process environment | |
| CN111435285B (en) | Data writing method and device for storage volume, electronic equipment and machine-readable storage medium | |
| CN116991317A (en) | Multi-core SSD non-4 k aligned writing method, system, equipment and storage medium | |
| CN118069700A (en) | Data verification method, device, computer equipment and storage medium | |
| CN116521072A (en) | Extensible shared memory chip and remote access method of shared memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |