CN111767281A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111767281A CN111767281A CN202010415047.XA CN202010415047A CN111767281A CN 111767281 A CN111767281 A CN 111767281A CN 202010415047 A CN202010415047 A CN 202010415047A CN 111767281 A CN111767281 A CN 111767281A
- Authority
- CN
- China
- Prior art keywords
- thread
- temporary storage
- data
- producer
- linked list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data processing method, a data processing device, an electronic device and a storage medium, wherein the method comprises the following steps: processing the task to be processed by utilizing a producer thread to obtain production data; searching an index position corresponding to the producer thread in a temporary storage index array; searching a temporary storage linked list corresponding to the index position according to the index position; and consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread. According to the embodiment of the invention, the index positions corresponding to the producer threads are arranged in the temporary storage index array, and the corresponding temporary storage linked list is arranged for each index position, so that independent index positions and temporary storage linked lists are arranged for each pair of producer threads and consumer threads, namely, a special data storage space is established for each pair of producer threads and consumer threads, and the concurrent computation efficiency of the system is greatly improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
Currently, in order to improve the system operation performance, it is a very common method to perform system construction through multithreading. To address the issue of data coherency security in a multithreaded architecture, it is common practice in the industry to use thread locks. The thread lock is mainly used for preventing a plurality of threads from reading and writing a certain memory area at the same time.
When a plurality of threads need to access the same memory area, only one thread can be executed at the same time after locking, so that other threads can read and write the memory area of the block only after the current thread is executed. The method for locking the threads guarantees safety, and meanwhile, the current locked thread can be switched to the next thread after being executed, so that thread switching operations on multiple operating system layers are increased, and the overall concurrent computing efficiency of the system is greatly influenced.
Disclosure of Invention
In order to solve the technical problems described above or at least partially solve the technical problems, the present application provides a data processing method, an apparatus, an electronic device, and a storage medium.
In a first aspect, the present application provides a data processing method, including:
processing the task to be processed by utilizing a producer thread to obtain production data;
searching an index position corresponding to the producer thread in a temporary storage index array;
searching a temporary storage linked list corresponding to the index position according to the index position;
and consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread.
Optionally, the step of searching the index position corresponding to the producer thread in the temporary storage index array includes:
acquiring a thread identifier of the producer thread;
determining whether the thread identifier is located in a preset routing table;
and if the thread identifier is located in a preset routing table, determining an index position corresponding to the thread identifier in the preset routing table as an index position corresponding to the producer thread.
Optionally, the step of searching the index position corresponding to the producer thread in the temporary storage index array further includes:
if the thread identifier is not located in the preset routing table, allocating a corresponding index position to the thread identifier;
determining the index position allocated to the thread identifier as the index position corresponding to the producer thread;
and storing the corresponding relation between the thread identification and the index position into the preset routing table.
Optionally, the method further includes:
judging whether the current data volume to be consumed in the temporary storage linked list reaches a preset allowed maximum data volume or not;
and if the current data volume to be consumed in the temporary storage linked list reaches the preset maximum data volume, waiting for the consumption of the production data by the consumer thread until the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum data volume.
Optionally, the method further includes:
if the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum data volume, executing a step of processing the task to be processed by utilizing a producer thread to obtain production data;
and after obtaining the production data, sending a data consumption notice to the consumer thread.
Optionally, the method further includes:
judging whether the temporary storage linked list has production data to be consumed or not;
and if the temporary storage linked list does not have the production data to be consumed, waiting for a producer thread to process the task to be processed until the production data to be consumed exist in the temporary storage linked list.
Optionally, the method further includes:
if the production data to be consumed exist in the temporary storage linked list, executing a step of consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread;
after consuming the production data is complete, a data production notification is sent to the producer thread.
In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:
the production module is used for processing the tasks to be processed by utilizing the producer threads to obtain production data;
the first searching module is used for searching an index position corresponding to the producer thread in a temporary storage index array;
the second searching module is used for searching a temporary storage linked list corresponding to the index position according to the index position;
and the consumption module is used for consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread.
Optionally, the first lookup module includes:
the obtaining unit is used for obtaining the thread identification of the producer thread;
a first determining unit, configured to determine whether the thread identifier is located in a preset routing table;
and a second determining unit, configured to determine, if the thread identifier is located in a preset routing table, an index position corresponding to the thread identifier in the preset routing table as an index position corresponding to the producer thread.
Optionally, the first lookup module further includes:
the distribution unit is used for distributing a corresponding index position for the thread identifier if the thread identifier is not located in the preset routing table;
a third determining unit, configured to determine an index position allocated to the thread identifier as an index position corresponding to the producer thread;
and the storage unit is used for storing the corresponding relation between the thread identifier and the index position into the preset routing table.
Optionally, the apparatus further comprises:
the first judging module is used for judging whether the current data volume to be consumed in the temporary storage linked list reaches the preset allowed maximum data volume or not;
the first waiting module is used for waiting for the consumption of the production data by the consumer thread if the current data volume to be consumed in the temporary storage linked list reaches the preset maximum allowable data volume until the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum allowable data volume.
Optionally, the method further includes:
the first execution module is used for executing the step of processing the task to be processed by utilizing a producer thread to obtain production data if the current data volume to be consumed in the temporary storage linked list is smaller than the preset allowed maximum data volume;
and the first notification module is used for sending a data consumption notification to the consumer thread after the production data is obtained.
Optionally, the method further includes:
the second judgment module is used for judging whether the temporary storage linked list has production data to be consumed or not;
and the second waiting module is used for waiting for the producer thread to process the task to be processed if the production data to be consumed does not exist in the temporary storage linked list until the production data to be consumed exists in the temporary storage linked list.
Optionally, the method further includes:
the second execution module is used for executing the step of consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread if the production data to be consumed exists in the temporary storage linked list;
and the second notification module is used for sending a data production notification to the producer thread after the production data is consumed.
In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete mutual communication through the communication bus;
a memory for storing a computer program;
a processor for implementing the data processing method of any one of the first aspect when executing the program stored in the memory.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a program of a data processing method, which when executed by a processor, implements the steps of the data processing method of any one of the first aspects.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:
according to the embodiment of the invention, a producer thread is used for processing a task to be processed to obtain production data, then an index position corresponding to the producer thread is searched in a temporary storage index array, a temporary storage linked list corresponding to the index position is searched according to the index position, and finally the production data in the temporary storage linked list can be consumed by a consumer thread corresponding to the producer thread.
The embodiment of the invention realizes that independent index positions and temporary linked lists are set for each pair of producer threads and consumer threads by setting the index positions corresponding to the producer threads in the temporary index array and setting the corresponding temporary linked lists for each index position, namely, a special data storage space is established for each pair of the producer threads and the consumer threads.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;
FIG. 2 is a flowchart of step S102 in FIG. 1;
fig. 3 is another flowchart of a data processing method according to an embodiment of the present application;
fig. 4 is another flowchart of a data processing method according to an embodiment of the present application;
fig. 5 is a block diagram of a data processing apparatus according to an embodiment of the present application;
fig. 6 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the prior art, when a plurality of threads need to access the same memory area, only one thread can be executed at the same time after locking, so that other threads can read and write the memory area in the block only after the current thread is executed. The method for locking the threads guarantees safety, and meanwhile, the current locked thread can be switched to the next thread after being executed, so that thread switching operations on multiple operating system layers are increased, and the overall concurrent computing efficiency of the system is greatly influenced. To this end, an embodiment of the present invention provides a data processing method, an apparatus, an electronic device, and a storage medium, where the data processing method may be applied in a server, as shown in fig. 1, and the data processing method may include the following steps:
step S101, processing a task to be processed by using a producer thread to obtain production data;
in the embodiment of the present invention, the number of threads in the thread pool may be specified in advance according to the server performance, each thread has a unique thread identifier, and illustratively, the number of threads in the thread pool may be specified according to the number of CPU cores, for example, the number of CPU cores of the server is 8 cores, and the number of threads in the thread pool is 8.
And then, creating a data production thread group and a data consumption thread group according to the performance of the server, wherein illustratively, if the number of the CPU cores is 8, the data production thread group may include 8 producer threads, the data consumption thread group may include 8 consumer threads, and the producer threads in the data production thread group correspond to the consumer threads in the data consumption thread group one to one.
In practical applications, producer threads and consumer threads may be created by:
finally, can establish the index array of keeping in storage according to server performance to for every index position in the index array of keeping in storage establishes the corresponding chain table of keeping in storage, exemplarily, if the CPU number of cores is 8, can establish the index array of keeping in storage that contains 8 index positions, every index position corresponds with a chain table of keeping in storage, is used for the first address of the chain table of keeping in storage in the index position, the length of the chain table of keeping in storage can be according to actual need and carry out the pre-configuration.
Since the number of index positions in the scratch index array is the same as the number of producer threads or consumer threads in the thread pool, that is, each pair of producer and consumer threads share an index position and a scratch list.
In practical applications, the maximum allowed scratch list length is imported by the developer at device initialization. The size of the temporary storage index array is initialized by default according to the number of CPU cores of the running server:
in this step, after receiving the task to be processed, any producer thread currently in an idle state in the thread pool may be used to process the task to be processed, so as to obtain production data.
If there are multiple producer threads currently in an idle state in the thread pool, the producer threads located at the head and in the idle state in the sequence can be used to process the tasks to be processed according to the sequence of the thread identifier, so as to obtain production data.
If no producer thread in the idle state exists in the thread pool, waiting can be carried out until any producer thread finishes processing the task being processed, and the producer thread which finishes processing the task and has changed into the idle state is used for processing the task to be processed to obtain the production data.
Step S102, searching an index position corresponding to the producer thread in a temporary storage index array;
in this step, the index position corresponding to the producer thread in the temporary storage index array may be searched according to the thread identifier.
Step S103, searching a temporary storage linked list corresponding to the index position according to the index position;
in this step, the linked list address of the temporary storage linked list can be read in the index space corresponding to the index position, and then the temporary storage linked list can be found according to the linked list address of the temporary storage linked list.
And step S104, consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread.
In this step, the index position corresponding to the consumer thread in the temporary storage index array can be searched according to the thread identifier, then the corresponding temporary storage linked list is searched according to the linked list address in the index position, and then the production data in the consumer thread consumption temporary storage linked list corresponding to the producer thread is utilized.
According to the embodiment of the invention, a producer thread is used for processing a task to be processed to obtain production data, then an index position corresponding to the producer thread is searched in a temporary storage index array, a temporary storage linked list corresponding to the index position is searched according to the index position, and finally the production data in the temporary storage linked list can be consumed by a consumer thread corresponding to the producer thread.
The embodiment of the invention realizes that independent index positions and temporary linked lists are set for each pair of producer threads and consumer threads by setting the index positions corresponding to the producer threads in the temporary index array and setting the corresponding temporary linked lists for each index position, namely, a special data storage space is established for each pair of the producer threads and the consumer threads.
In another embodiment of the present invention, the step S102 of searching the temporary index array for the index position corresponding to the producer thread may include the following steps as shown in fig. 2:
step S201, acquiring a thread identifier of the producer thread;
in this step, since each thread has a unique thread identification, the thread identification of the producer thread currently processing the pending task may be acquired.
Step S202, determining whether the thread identifier is located in a preset routing table;
in the embodiment of the present invention, a preset routing table may be preset to store a corresponding relationship between the thread identifier and the index position.
In this step, it may be searched in the preset routing table whether a thread identifier matching the thread identifier of the producer thread for processing the task to be processed exists, and if such a thread identifier exists in the preset routing table, it may be determined that the thread identifier is located in the preset routing table.
If the thread identifier is located in a preset routing table, in step S203, an index position corresponding to the thread identifier in the preset routing table is determined as an index position corresponding to the producer thread.
In practical application, it may be determined whether the thread identifier is located in a preset routing table by:
according to the embodiment of the invention, the index position corresponding to the thread identifier can be searched in the preset routing table, so that the index position corresponding to the producer thread can be more conveniently and rapidly determined, the searching process is simpler and more convenient, and the searching efficiency is improved.
In another embodiment of the present invention, the step S102 searches the temporary storage index array for the index position corresponding to the producer thread, as shown in fig. 2, and may further include the following steps:
if the thread identifier is not located in the preset routing table, step S204, allocating a corresponding index position to the thread identifier;
in this step, the index position that does not establish a correspondence relationship with the thread identifier may be first searched for in the temporary storage index array, and the first index position that does not establish a correspondence relationship with the thread identifier is allocated to the thread identifier according to the order of the index positions in the index array.
Step S205, determining the index position allocated to the thread identifier as the index position corresponding to the producer thread;
in practical application, each index position has a corresponding temporary storage linked list, so the first address of the corresponding temporary storage linked list has been stored in the index space corresponding to the index position.
Step S206, storing the corresponding relationship between the thread identifier and the index position in the preset routing table.
According to the embodiment of the invention, when the thread identifier is not located in the preset routing table, the corresponding index position is automatically allocated to the thread identifier, so that the index position allocated to the thread identifier can be conveniently determined as the index position corresponding to the producer thread; and storing the corresponding relation between the thread identifier and the index position into the preset routing table, so that the corresponding index position can be directly found when the same thread identifier is used for searching the preset routing table in the subsequent process, and the subsequent use is facilitated.
In practical applications, there may be a large amount of production data, which may cause a large amount of production data to be consumed in the scratch list to be consumed by a consumer thread, and at this time, if the production data continues to be written, a system crash may be caused, for this reason, in another embodiment of the present invention, as shown in fig. 3, the method further includes:
step S301, judging whether the current data volume to be consumed in the temporary storage linked list reaches a preset allowed maximum data volume;
in this step, the data size of the production data that is not consumed yet in the scratch list may be read, the data size may be compared with a preset allowable maximum data size, and if the data size to be consumed is equal to the preset allowable maximum data size, it is determined that the data size to be consumed in the scratch list reaches the preset allowable maximum data size.
If the current data volume to be consumed in the temporary storage linked list reaches the preset maximum data volume, step S302 is performed to wait for the consumption of the production data by the consumer thread until the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum data volume.
If the current data volume to be consumed in the temporary storage linked list is smaller than the preset allowed maximum data volume, step S303, executing a step of processing the task to be processed by using a producer thread to obtain production data;
and step S304, after the production data is obtained, sending a data consumption notice to the consumer thread.
In practical applications, consumption of production data may be performed in the following manner:
according to the embodiment of the invention, when the current data volume to be consumed in the temporary storage linked list reaches the preset allowed maximum data volume, the consumption production data of the consumer thread can be automatically waited until the current data volume to be consumed in the temporary storage linked list is smaller than the preset allowed maximum data volume, so that the system collapse caused by overload of the consumer thread is avoided; when the current data volume to be consumed in the temporary storage linked list is smaller than the preset allowed maximum data volume, the producer thread can be automatically utilized to process the task to be processed to obtain production data, and after the production data are obtained, a data consumption notice is sent to the consumer thread, so that the close matching between the consumer thread and the producer thread can be facilitated, the cooperation capability between the consumer thread and the producer thread is improved, and the working efficiency between the consumer thread and the producer thread is improved.
In practical applications, there may be a case where a certain production data has a small data size and a consumer thread is able to consume the production data soon, and at this time, there may be a case where there is no production data to be consumed in the scratch list, for this reason, in another embodiment of the present invention, as shown in fig. 4, the method further includes:
step S401, judging whether the temporary storage linked list has production data to be consumed or not;
if the temporary storage linked list does not contain the production data to be consumed, step S402, waiting for the producer thread to process the task to be processed until the production data to be consumed exists in the temporary storage linked list.
If the production data to be consumed exists in the temporary storage linked list, step S403, executing a step of consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread;
and S404, after the production data is consumed, sending a data production notice to the producer thread.
In practical applications, the production of production data may be performed in the following manner:
the embodiment of the invention can automatically wait for a producer thread to process the next task to be processed when the production data to be consumed does not exist in the temporary storage linked list until the production data to be consumed exists in the temporary storage linked list, thereby avoiding the situation that the consumer does not have production data but can consume and work to cause waste of system resources; when the production data to be consumed exist in the temporary storage linked list, the production data in the temporary storage linked list can be consumed by a consumer thread automatically, and after the production data is consumed, a data production notice is sent to the producer thread, so that the consumer thread and the producer thread can be matched closely, the cooperation capability between the consumer thread and the producer thread is improved, and the working efficiency between the consumer thread and the producer thread is improved.
In still another embodiment of the present invention, there is also provided a data processing apparatus, as shown in fig. 5, including:
the production module 11 is used for processing the tasks to be processed by utilizing the producer threads to obtain production data;
the first searching module 12 is configured to search an index position corresponding to the producer thread in a temporary storage index array;
the second searching module 13 is configured to search a temporary storage linked list corresponding to the index position according to the index position;
and the consumption module 14 is configured to consume the production data in the scratch list by using a consumer thread corresponding to the producer thread.
Optionally, the first lookup module includes:
the obtaining unit is used for obtaining the thread identification of the producer thread;
a first determining unit, configured to determine whether the thread identifier is located in a preset routing table;
and a second determining unit, configured to determine, if the thread identifier is located in a preset routing table, an index position corresponding to the thread identifier in the preset routing table as an index position corresponding to the producer thread.
Optionally, the first lookup module further includes:
the distribution unit is used for distributing a corresponding index position for the thread identifier if the thread identifier is not located in the preset routing table;
a third determining unit, configured to determine an index position allocated to the thread identifier as an index position corresponding to the producer thread;
and the storage unit is used for storing the corresponding relation between the thread identifier and the index position into the preset routing table.
Optionally, the apparatus further comprises:
the first judging module is used for judging whether the current data volume to be consumed in the temporary storage linked list reaches the preset allowed maximum data volume or not;
the first waiting module is used for waiting for the consumption of the production data by the consumer thread if the current data volume to be consumed in the temporary storage linked list reaches the preset maximum allowable data volume until the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum allowable data volume.
Optionally, the method further includes:
the first execution module is used for executing the step of processing the task to be processed by utilizing a producer thread to obtain production data if the current data volume to be consumed in the temporary storage linked list is smaller than the preset allowed maximum data volume;
and the first notification module is used for sending a data consumption notification to the consumer thread after the production data is obtained.
Optionally, the method further includes:
the second judgment module is used for judging whether the temporary storage linked list has production data to be consumed or not;
and the second waiting module is used for waiting for the producer thread to process the task to be processed if the production data to be consumed does not exist in the temporary storage linked list until the production data to be consumed exists in the temporary storage linked list.
Optionally, the method further includes:
the second execution module is used for executing the step of consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread if the production data to be consumed exists in the temporary storage linked list;
and the second notification module is used for sending a data production notification to the producer thread after the production data is consumed.
In another embodiment of the present invention, an electronic device is further provided, which includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the data processing method according to any one of claims 1 to 7 when executing a program stored in a memory.
In the electronic device provided by the embodiment of the invention, the processor implements the embodiment of the invention by executing the program stored in the memory, the task to be processed is processed by utilizing the producer thread to obtain the production data, then the index position corresponding to the producer thread is searched in the temporary storage index array, then the temporary storage linked list corresponding to the index position is searched according to the index position, finally the production data in the temporary storage linked list can be consumed by utilizing the consumer thread corresponding to the producer thread, the independent index position and the temporary storage linked list are ensured to be set for each pair of the producer thread and the consumer thread by setting the index position corresponding to the producer thread in the temporary storage index array and setting the corresponding temporary storage linked list for each index position, namely, a special data storage space is established for each pair of the producer thread and the consumer thread, because a plurality of pairs of producer threads and consumer threads respectively use different data storage spaces, the situation that different producer threads or consumer threads use the same data storage space does not exist, the situation that other producer threads or consumer threads accessing the same data storage space are waited for execution completion does not exist, and the concurrent computation efficiency of the system is greatly improved.
The communication bus 1140 mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
The communication interface 1120 is used for communication between the electronic device and other devices.
The memory 1130 may include a Random Access Memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The processor 1110 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the integrated circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
In still another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a program of a data processing method, which when executed by a processor, implements the steps of the data processing method described in the foregoing method embodiment.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of data processing, the method comprising:
processing the task to be processed by utilizing a producer thread to obtain production data;
searching an index position corresponding to the producer thread in a temporary storage index array;
searching a temporary storage linked list corresponding to the index position according to the index position;
and consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread.
2. The data processing method of claim 1, wherein the step of searching the temporary index array for the index location corresponding to the producer thread comprises:
acquiring a thread identifier of the producer thread;
determining whether the thread identifier is located in a preset routing table;
and if the thread identifier is located in a preset routing table, determining an index position corresponding to the thread identifier in the preset routing table as an index position corresponding to the producer thread.
3. The data processing method of claim 2, wherein the step of searching the temporary index array for the index location corresponding to the producer thread further comprises:
if the thread identifier is not located in the preset routing table, allocating a corresponding index position to the thread identifier;
determining the index position allocated to the thread identifier as the index position corresponding to the producer thread;
and storing the corresponding relation between the thread identification and the index position into the preset routing table.
4. The data processing method of claim 1, wherein the method further comprises:
judging whether the current data volume to be consumed in the temporary storage linked list reaches a preset allowed maximum data volume or not;
and if the current data volume to be consumed in the temporary storage linked list reaches the preset maximum data volume, waiting for the consumption of the production data by the consumer thread until the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum data volume.
5. The data processing method of claim 4, wherein the method further comprises:
if the current data volume to be consumed in the temporary storage linked list is smaller than the preset maximum data volume, executing a step of processing the task to be processed by utilizing a producer thread to obtain production data;
and after obtaining the production data, sending a data consumption notice to the consumer thread.
6. The data processing method of claim 1, wherein the method further comprises:
judging whether the temporary storage linked list has production data to be consumed or not;
and if the temporary storage linked list does not have the production data to be consumed, waiting for a producer thread to process the task to be processed until the production data to be consumed exist in the temporary storage linked list.
7. The data processing method of claim 6, wherein the method further comprises:
if the production data to be consumed exist in the temporary storage linked list, executing a step of consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread;
after consuming the production data is complete, a data production notification is sent to the producer thread.
8. A data processing apparatus, characterized in that the apparatus comprises:
the production module is used for processing the tasks to be processed by utilizing the producer threads to obtain production data;
the first searching module is used for searching an index position corresponding to the producer thread in a temporary storage index array;
the second searching module is used for searching a temporary storage linked list corresponding to the index position according to the index position;
and the consumption module is used for consuming the production data in the temporary storage linked list by using a consumer thread corresponding to the producer thread.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the data processing method according to any one of claims 1 to 7 when executing a program stored in a memory.
10. A computer-readable storage medium, characterized in that a program of a data processing method is stored on the computer-readable storage medium, which program, when executed by a processor, implements the steps of the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010415047.XA CN111767281A (en) | 2020-05-15 | 2020-05-15 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010415047.XA CN111767281A (en) | 2020-05-15 | 2020-05-15 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111767281A true CN111767281A (en) | 2020-10-13 |
Family
ID=72719414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010415047.XA Pending CN111767281A (en) | 2020-05-15 | 2020-05-15 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111767281A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858444A (en) * | 2022-11-22 | 2023-03-28 | 格兰菲智能科技有限公司 | Data access method, device and system for general computing |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294753A (en) * | 2012-01-30 | 2013-09-11 | 辉达公司 | Lock-free fifo |
| US20160092273A1 (en) * | 2014-09-25 | 2016-03-31 | Software Ag | System and method for managing the allocating and freeing of objects in a multi-threaded system |
| CN105868031A (en) * | 2016-03-24 | 2016-08-17 | 车智互联（北京）科技有限公司 | A data transmission device and method |
| CN106294731A (en) * | 2016-08-09 | 2017-01-04 | 四川网达科技有限公司 | Enter management method and the device of database data |
| CN107025142A (en) * | 2016-01-29 | 2017-08-08 | 华为技术有限公司 | A kind of cross-thread message delivery method, device and electronic equipment |
| CN110069557A (en) * | 2019-03-16 | 2019-07-30 | 平安科技（深圳）有限公司 | Data transmission method, device, equipment and storage medium |
| CN111143079A (en) * | 2019-12-24 | 2020-05-12 | 浪潮软件股份有限公司 | Method for realizing multi-read multi-write lock-free queue |
-
2020
- 2020-05-15 CN CN202010415047.XA patent/CN111767281A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294753A (en) * | 2012-01-30 | 2013-09-11 | 辉达公司 | Lock-free fifo |
| US20160092273A1 (en) * | 2014-09-25 | 2016-03-31 | Software Ag | System and method for managing the allocating and freeing of objects in a multi-threaded system |
| CN107025142A (en) * | 2016-01-29 | 2017-08-08 | 华为技术有限公司 | A kind of cross-thread message delivery method, device and electronic equipment |
| CN105868031A (en) * | 2016-03-24 | 2016-08-17 | 车智互联（北京）科技有限公司 | A data transmission device and method |
| CN106294731A (en) * | 2016-08-09 | 2017-01-04 | 四川网达科技有限公司 | Enter management method and the device of database data |
| CN110069557A (en) * | 2019-03-16 | 2019-07-30 | 平安科技（深圳）有限公司 | Data transmission method, device, equipment and storage medium |
| CN111143079A (en) * | 2019-12-24 | 2020-05-12 | 浪潮软件股份有限公司 | Method for realizing multi-read multi-write lock-free queue |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858444A (en) * | 2022-11-22 | 2023-03-28 | 格兰菲智能科技有限公司 | Data access method, device and system for general computing |
| CN115858444B (en) * | 2022-11-22 | 2023-12-12 | 格兰菲智能科技有限公司 | Data access method, device and system for general computing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106294190B (en) | Storage space management method and device | |
| US8381230B2 (en) | Message passing with queues and channels | |
| TWI694700B (en) | Data processing method and device, user terminal | |
| KR102354848B1 (en) | Cache Memory Device and Electronic System including the Same | |
| TW202034154A (en) | Data reading and writing method and device | |
| CN111767281A (en) | Data processing method and device, electronic equipment and storage medium | |
| US10990569B2 (en) | Sorting system | |
| US8543722B2 (en) | Message passing with queues and channels | |
| CN114327642A (en) | Data read-write control method and electronic equipment | |
| CN115114042A (en) | Storage data access method and device, electronic equipment and storage medium | |
| CN117271162A (en) | Request processing method, device, equipment and storage medium | |
| CN108920092B (en) | Data operation method and device of memory data and electronic equipment | |
| CN110618946A (en) | Stack memory allocation method, device, equipment and storage medium | |
| CN109254858B (en) | Data sharing method, device and equipment of active window and storage medium | |
| CN110554905A (en) | Starting method and device of container | |
| CN113076266B (en) | Memory management method and device, electronic equipment and storage medium | |
| US10310857B2 (en) | Systems and methods facilitating multi-word atomic operation support for system on chip environments | |
| CN116521608A (en) | Data migration method and computing device | |
| US9063858B2 (en) | Multi-core system and method for data consistency by memory mapping address (ADB) to hash table pattern associated with at least one core | |
| CN115114289A (en) | Data query method and device and electronic equipment | |
| CN113132261B (en) | Traffic data packet classification method and device and electronic equipment | |
| CN108733678B (en) | Data searching method, device and related equipment | |
| CN112667682A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN103885824A (en) | Interface control circuit, equipment and identification switching method | |
| CN115543970B (en) | Data page processing method, data page processing device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |