CN111158939A - Data processing method, data processing device, storage medium and electronic equipment - Google Patents
Data processing method, data processing device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN111158939A CN111158939A CN201911419917.4A CN201911419917A CN111158939A CN 111158939 A CN111158939 A CN 111158939A CN 201911419917 A CN201911419917 A CN 201911419917A CN 111158939 A CN111158939 A CN 111158939A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- piece
- message queue
- threads
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 34
- 230000003993 interaction Effects 0.000 claims description 8
- 230000001960 triggered effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 16
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012163 sequencing technique Methods 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a data processing device, a storage medium and electronic equipment. Wherein, the method comprises the following steps: receiving the uploaded spreadsheet file; storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing a processing result of each piece of data; and carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads. The method and the device solve the technical problems that due to the fact that the data volume is large and the storage space is limited, processing efficiency is low, the data processing device is stuck and a data processing result cannot be obtained in the data processing process.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data processing method, an apparatus, a storage medium, and an electronic device.
Background
At present, the application in the aspect of the internet of things needs to meet the requirement of a large number of interaction with equipment, large-batch equipment data needs to be imported into a database, data files need to be divided into a plurality of tasks for processing in the process of data batch import, and the complexity of use of a user is increased. When the data volume of the excell is too large and the memory is limited, all information needs to be loaded into the memory at one time because the imported data is generally processed by a single thread, if the traditional synchronization method is used for solving the problem, the processing speed is too low, the processing progress of the data cannot be acquired, and the situations of blocking and the like can occur.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device, a storage medium and electronic equipment, and aims to at least solve the technical problems of low processing efficiency, blocking and incapability of acquiring a data processing result in a data processing process due to large data volume and limited storage space.
According to an aspect of an embodiment of the present application, there is provided a data processing method, including: receiving the uploaded spreadsheet file; storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing a processing result of each piece of data; and carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
Optionally, the recording and storing the processing result of each piece of data includes: storing the processing result of each piece of data in a specified database; after recording and storing the processing result of each piece of data, the method further comprises: acquiring a processing result from a specified database; determining a processing progress of each piece of data based on the processing result; and pushing the processing progress to the front-end equipment for displaying.
Optionally, the parallel processing of each piece of data in the message queue by using multiple threads includes: determining the number of data to be processed in the message queue; determining a first number of threads according to a preset rule based on the number of data pieces, wherein the preset rule comprises the following steps: the maximum number of processing supported by each thread; each piece of data in the message queue is processed in parallel using a first number of threads.
Optionally, before performing parallel processing on each piece of data in the message queue by using a first number of threads, the method further includes: determining target memory resources required by a first number of threads; obtaining available memory resources of equipment where the message queue is located; comparing the size of the target memory resource with the size of the available memory resource; when the target memory resource is greater than the available memory resource, reducing the value of the first quantity until the target memory resource is less than or equal to the available memory resource.
Optionally, storing each piece of data in the spreadsheet file into a message queue respectively, including: reading each piece of data from the spreadsheet file to obtain a plurality of pieces of data; and asynchronously storing the plurality of pieces of data into the message queue.
According to another optional aspect of the present application, there is provided another data processing method, including: displaying a control for uploading the electronic form file in a human-computer interaction interface, and receiving the uploaded electronic form file when the control is triggered; displaying a message queue for storing each data score in the electronic form file in a human-computer interaction interface, and displaying a processing result of each data; multiple threads are shown for parallel processing of various pieces of data in a message queue.
According to yet another alternative aspect of the present application, there is provided a data processing apparatus comprising: the receiving module is used for receiving the uploaded spreadsheet files; the storage module is used for respectively storing each piece of data in the electronic form file into the message queue and recording and storing the processing result of each piece of data; and the processing module is used for carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
Optionally, the processing module is further configured to determine the number of data to be processed in the message queue; determining a first number of threads according to a preset rule based on the number of data pieces, wherein the preset rule comprises the following steps: the maximum number of processing supported by each thread; and processing each piece of data in the message queue in parallel by adopting a first number of threads.
According to still another aspect of the embodiments of the present application, there is provided a nonvolatile storage medium including a stored program, wherein the apparatus in which the nonvolatile storage medium is located is controlled to execute the data processing method above the right when the program runs.
According to still another aspect of embodiments of the present application, there is provided an electronic device including: a memory for storing program instructions; a processor for executing program instructions, wherein the program when executed performs the data processing method of any one of claims 1 to 5.
In the embodiment of the application, an asynchronous processing mechanism is adopted, all the read data information is stored in the message queue by reading each piece of data information in the table file, a plurality of threads are started to process the information in parallel and record and store the information, the purposes of processing the data in batches and timely obtaining the processing result of each piece of data are achieved, and the technical problems that the processing efficiency is low, the data is jammed and the data processing result cannot be obtained in the data processing process due to the large data amount and the limited storage space are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic flow chart diagram of a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an overall flow of a data processing method according to an embodiment of the present application;
FIG. 3 is a flow chart diagram of a data processing method according to an embodiment of the application;
FIG. 4 is a block diagram of a data processing apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For a better understanding of the embodiments of the present application, the related terms referred to in the embodiments of the present application are explained below:
RabbitMQ (RabbitMessage Queue, simply called RabbitMQ): the open source Message agent software (also called Message oriented middleware) realizes Advanced Message Queuing Protocol (AMQP).
Message Queue (MQ): is a method of communication between programs.
Remote Dictionary service (Remote Dictionary Server, Redis): the system is a high-performance Key-Value database storage system, and supports relatively more stored Value types including string, list, set and zset. These data types all support push/pop, add/remove, and intersect, union, and difference or richer operations, and these operations are all atomic. On the basis, Redis supports various different sorting modes, and periodically writes updated data into a disk or writes modifications into an additional recording file, so that master-slave synchronization is realized.
WebSocket: the Protocol is a Protocol for performing full duplex communication on a single Transmission Control Protocol (TCP), and the Protocol can make data exchange between the client and the server simpler and allow the server to actively push data to the client.
mySQL: it is a relational database management system that keeps data in different tables instead of putting all data in one large repository, thus increasing speed and flexibility.
In accordance with an embodiment of the present application, there is provided a method embodiment of data processing, it should be noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a data processing method according to an embodiment of the present application, as shown in fig. 1, the method includes the following steps:
step S102, receiving the uploaded electronic form file.
Specifically, the spreadsheet file contains a large amount of data that needs to be uploaded and stored in a designated database.
And step S104, storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing the processing result of each piece of data.
In some embodiments of the present application, in order to store each piece of data in the spreadsheet file into the message queue, each piece of data needs to be read from the spreadsheet file, and after a plurality of pieces of data are obtained by reading, the obtained plurality of pieces of data are asynchronously stored into the message queue; when the processing result of each piece of data is recorded and stored, the processing result of each piece of data is stored in a specified database; in order to determine the processing progress of each piece of data, the processing result may be obtained from the designated database, the processing progress of each piece of data is determined according to the processing result, and then the processing progress is pushed to the front-end device for display.
Specifically, striping all data into message queues may be implemented using a RabbitMQ; the above-mentioned database for storage may be Redis and mySQL. Wherein RabbitMQ, an open source message agent software (also known as message-oriented middleware) that implements advanced message queuing protocol AMQP; redis, a high-performance Key-Value database storage system, supports relatively more stored Value types, including string, list, set, and zset. These data types all support push/pop, add/remove, and intersect, union, and difference or richer operations, and these operations are all atomic. In addition, Redis supports various different sequencing modes, and periodically writes updated data into a disk or writes modifications into an additional recording file, thereby realizing master-slave synchronization on the basis; mySQL: it is a relational database management system that keeps data in different tables instead of putting all data in one large repository, thus increasing speed and flexibility.
It should be noted here that both Redis and mySQL can record and store the success and failure results, which is convenient for the front-end user to query, and export the results for easy viewing. However, as shown in fig. 2, each time a thread processes one piece of data, the Redis marks one piece of record, can acquire the progress of the currently executed task from the Redis at any time, and pushes the progress or percentage to the front-end user through WebSocket.
In summary, in some embodiments of the present application, when data is imported in a large batch, an asynchronous processing mechanism is adopted, each piece of data in an excell is read first, instead of loading all information into a memory at one time, so that the burden of the memory can be reduced. And respectively storing each read message into a RabbitMQ (queue), then processing the data stored in the RabbitMQ, and simultaneously recording and storing the result in Redis and mySQL (so as to conveniently record the processing result).
And step S106, carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
In some embodiments of the present application, when a plurality of threads are used to perform parallel processing on each piece of data in a message queue, first, the number of data to be processed in the message queue is determined, a first number of threads is determined according to a preset rule according to the determined number of data, and then, each piece of data in the message queue is performed parallel processing by using the threads of the first number. Wherein, the preset rule is as follows: the maximum number of processes supported by each thread.
For example, the number of data to be processed in the message queue is one hundred thousand, and the preset conditions are as follows: the maximum number of processing pieces supported by each thread is twenty thousand, and the first number of threads is determined to be five according to the determined number of data pieces and the preset rule.
In some embodiments of the present application, before parallel processing is performed on each piece of data in the message queue by using the first number of threads, in order to avoid a situation of memory overflow, first, a target memory resource required by the first number of threads needs to be determined, then, an available memory resource of a device where the message queue is located is obtained, and a value of the first number is determined by comparing sizes of the target memory resource and the available memory resource. When the target memory resource is greater than the available memory resource, the value of the first quantity is reduced until the target memory resource is less than or equal to the available memory resource.
For example: assuming that the maximum number of data processing supported by each thread is twenty thousand, the number of the threads of the first number is five, and the five threads can process one hundred thousand pieces of data, if the available memory of the device where the obtained message queue is located can store twelve thousand pieces of data, because the available memory resource is greater than the target memory resource, the value of the first number is determined to be five or six according to the number of the threads which can be processed; if the available memory of the device where the obtained message queue is located can store nine thousand pieces of data, because the available memory resource is smaller than the target memory resource, the value of the first number is determined to be four according to the number of pieces that can be processed by each thread.
Through the steps, the technical effects of quickly processing data and saving storage space can be achieved, and the technical problems that the processing efficiency is low, the data is blocked and the data processing result cannot be obtained in the data processing process due to the large data volume and the limited storage space are solved.
Fig. 3 is a data processing method according to an embodiment of the present application, and as shown in fig. 3, the method includes the following steps:
step S302, a control used for uploading the electronic form file is displayed in the man-machine interaction interface, and the uploaded electronic form file is received when the control is triggered.
Specifically, the human-computer interaction interface can be a computer display screen, a mobile phone screen and the like; the spreadsheet file contains a large amount of data.
And step S304, displaying a message queue for storing each data score in the electronic form file in the human-computer interaction interface, and displaying the processing result of each data.
In some embodiments of the present application, in order to store each piece of data in the spreadsheet file into the message queue, each piece of data needs to be read from the spreadsheet file, and after a plurality of pieces of data are obtained by reading, the obtained plurality of pieces of data are asynchronously stored into the message queue; when the processing result of each piece of data is recorded and stored, the processing result of each piece of data is stored in a specified database; in order to determine the processing progress of each piece of data, the processing result may be obtained from the designated database, the processing progress of each piece of data is determined according to the processing result, and then the processing progress is pushed to the front-end device for display.
Specifically, striping all data into message queues may be implemented using a RabbitMQ; the above-mentioned database for storage may be Redis and mySQL, where RabbitMQ, i.e., open source message agent software (also called message-oriented middleware) that implements advanced message queue protocol AMQP; redis, a high-performance Key-Value database storage system, supports relatively more stored Value types, including string, list, set, and zset. These data types all support push/pop, add/remove, and intersect, union, and difference or richer operations, and these operations are all atomic. In addition, Redis supports various different sequencing modes, and periodically writes updated data into a disk or writes modifications into an additional recording file, thereby realizing master-slave synchronization on the basis; mySQL: it is a relational database management system that keeps data in different tables instead of putting all data in one large repository, thus increasing speed and flexibility.
It should be noted here that both Redis and mySQL can record and store the success and failure results, which is convenient for the front-end user to query, and export the results for easy viewing. However, as shown in the figure, each time a thread processes one piece of data, the Redis marks and records one piece of data, can acquire the progress condition of the currently executed task from the Redis at any time, and pushes the progress or percentage to the front-end user through the WebSocket.
Step S306, showing a plurality of threads for performing parallel processing on each piece of data in the message queue.
In some embodiments of the present application, when a plurality of threads are used to perform parallel processing on each piece of data in a message queue, first, the number of data to be processed in the message queue is determined, a first number of threads is determined according to a preset rule according to the determined number of data, and then, each piece of data in the message queue is performed parallel processing by using the threads of the first number. Wherein, the preset rule is as follows: the maximum number of processes supported by each thread.
For example, the number of data to be processed in the message queue is one hundred thousand, and the preset conditions are as follows: the maximum number of processing pieces supported by each thread is twenty thousand, and the first number of threads is determined to be five according to the determined number of data pieces and the preset rule.
In some embodiments of the present application, before parallel processing is performed on each piece of data in the message queue by using the first number of threads, in order to avoid a situation of memory overflow, first, a target memory resource required by the first number of threads needs to be determined, then, an available memory resource of a device where the message queue is located is obtained, and a value of the first number is determined by comparing sizes of the target memory resource and the available memory resource. When the target memory resource is greater than the available memory resource, the value of the first quantity is reduced until the target memory resource is less than or equal to the available memory resource.
Fig. 4 is a data processing apparatus according to an embodiment of the present application, and as shown in fig. 4, the apparatus includes the following structure:
a receiving module 40, configured to receive the uploaded spreadsheet file.
Specifically, the spreadsheet file contains a large amount of data that needs to be uploaded and stored in a designated database.
The storage module 42 is configured to store each piece of data in the spreadsheet file into the message queue, and record and store a processing result of each piece of data.
In some embodiments of the present application, in order to store each piece of data in the spreadsheet file into the message queue, each piece of data needs to be read from the spreadsheet file, and after a plurality of pieces of data are obtained by reading, the obtained plurality of pieces of data are asynchronously stored into the message queue; when the processing result of each piece of data is recorded and stored, the processing result of each piece of data is stored in a specified database; in order to determine the processing progress of each piece of data, the processing result may be obtained from the designated database, the processing progress of each piece of data is determined according to the processing result, and then the processing progress is pushed to the front-end device for display.
Specifically, striping all data into message queues may be implemented using a RabbitMQ; the above-mentioned database for storage may be Redis and mySQL,. Wherein RabbitMQ, an open source message agent software (also known as message-oriented middleware) that implements advanced message queuing protocol AMQP; redis, a high-performance Key-Value database storage system, supports relatively more stored Value types, including string, list, set, and zset. These data types all support push/pop, add/remove, and intersect, union, and difference or richer operations, and these operations are all atomic. In addition, Redis supports various different sequencing modes, and periodically writes updated data into a disk or writes modifications into an additional recording file, thereby realizing master-slave synchronization on the basis; mySQL: it is a relational database management system that keeps data in different tables instead of putting all data in one large repository, thus increasing speed and flexibility.
It should be noted here that both Redis and mySQL can record and store the success and failure results, which is convenient for the front-end user to query, and export the results for easy viewing. However, as shown in the figure, each time a thread processes one piece of data, the Redis marks and records one piece of data, can acquire the progress condition of the currently executed task from the Redis at any time, and pushes the progress or percentage to the front-end user through the WebSocket.
And the processing module 44 is configured to perform parallel processing on each piece of data in the message queue by using multiple threads.
In some embodiments of the present application, when a plurality of threads are used to perform parallel processing on each piece of data in a message queue, first, the number of data to be processed in the message queue is determined, a first number of threads is determined according to a preset rule according to the determined number of data, and then, each piece of data in the message queue is performed parallel processing by using the threads of the first number. Wherein, the preset rule is as follows: the maximum number of processes supported by each thread.
For example, the number of data to be processed in the message queue is one hundred thousand, and the preset conditions are as follows: the maximum number of processing pieces supported by each thread is twenty thousand, and the first number of threads is determined to be five according to the determined number of data pieces and the preset rule.
In some embodiments of the present application, before parallel processing is performed on each piece of data in the message queue by using the first number of threads, in order to avoid a situation of memory overflow, first, a target memory resource required by the first number of threads needs to be determined, then, an available memory resource of a device where the message queue is located is obtained, and a value of the first number is determined by comparing sizes of the target memory resource and the available memory resource. When the target memory resource is greater than the available memory resource, the value of the first quantity is reduced until the target memory resource is less than or equal to the available memory resource.
In some embodiments of the present application, the processing module 46 is further configured to determine the number of data pieces to be processed in the message queue, that is, determine the first number of threads according to the number of data pieces and according to a preset rule, and perform parallel processing on each piece of data in the message queue by using the first number of threads. Wherein, the preset rule is as follows: the maximum number of processes supported by each thread.
According to still another aspect of an embodiment of the present application, there is provided a nonvolatile storage medium including a stored program, the nonvolatile storage medium being configured to run the program that realizes:
receiving the uploaded spreadsheet file; storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing a processing result of each piece of data; and carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
Fig. 5 is an electronic device according to an embodiment of the application, and as shown in fig. 5, the device includes:
a memory 50 for storing program instructions.
A processor 52 for executing program instructions, the program being operable to perform the following functions:
receiving the uploaded spreadsheet file; storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing a processing result of each piece of data; and carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, a division of a unit may be a division of a logic function, and an actual implementation may have another division, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or may not be executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (10)
1. A data processing method, comprising:
receiving the uploaded spreadsheet file;
storing each piece of data in the spreadsheet file into a message queue respectively, and recording and storing a processing result of each piece of data;
and processing each piece of data in the message queue in parallel by adopting a plurality of threads.
2. The method of claim 1,
the recording and storing of the processing result of each piece of data includes: storing the processing result of each piece of data into a specified database;
after recording and storing the processing result of each piece of data, the method further comprises: acquiring the processing result from the specified database; determining the processing progress of each piece of data based on the processing result; and pushing the processing progress to front-end equipment for displaying.
3. The method of claim 1, wherein processing each piece of data in the message queue in parallel with a plurality of threads comprises:
determining the number of data to be processed in the message queue;
determining a first number of the threads according to a preset rule based on the number of the data pieces, wherein the preset rule comprises: the maximum number of processing supported by each thread;
and performing parallel processing on each piece of data in the message queue by using the first number of threads.
4. The method of claim 3, wherein prior to processing each piece of data in the message queue in parallel using the first number of threads, the method further comprises:
determining target memory resources required by the first number of threads;
obtaining available memory resources of equipment where the message queue is located;
comparing the size of the target memory resource with the size of the available memory resource;
and when the target memory resource is larger than the available memory resource, reducing the value of the first quantity until the target memory resource is smaller than or equal to the available memory resource.
5. The method of claim 1, wherein storing each piece of data in the spreadsheet file in a message queue separately comprises:
reading each piece of data from the spreadsheet file to obtain a plurality of pieces of data;
and asynchronously storing the plurality of pieces of data into the message queue.
6. A data processing method, comprising:
displaying a control for uploading the electronic form file in a human-computer interaction interface, and receiving the uploaded electronic form file when the control is triggered;
displaying a message queue for storing each data score in the electronic form file in the human-computer interaction interface, and displaying a processing result of each data;
multiple threads are shown for parallel processing of pieces of data in the message queue.
7. A data processing apparatus, comprising:
the receiving module is used for receiving the uploaded spreadsheet files;
the storage module is used for respectively storing each piece of data in the electronic form file into a message queue and recording and storing the processing result of each piece of data;
and the processing module is used for carrying out parallel processing on each piece of data in the message queue by adopting a plurality of threads.
8. The apparatus of claim 7, wherein the processing module is further configured to determine a number of data pieces to be processed in the message queue; determining a first number of the threads according to a preset rule based on the number of the data pieces, wherein the preset rule comprises: the maximum number of processing supported by each thread; and performing parallel processing on each piece of data in the message queue by using the first number of threads.
9. A non-volatile storage medium, comprising a stored program, wherein when the program is executed, a device in which the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 5.
10. An electronic device, comprising:
a memory for storing program instructions;
a processor for executing the program instructions, wherein the program when executed performs the data processing method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911419917.4A CN111158939A (en) | 2019-12-31 | 2019-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911419917.4A CN111158939A (en) | 2019-12-31 | 2019-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111158939A true CN111158939A (en) | 2020-05-15 |
Family
ID=70560424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911419917.4A Pending CN111158939A (en) | 2019-12-31 | 2019-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111158939A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100127A (en) * | 2020-11-12 | 2020-12-18 | 广州市玄武无线科技股份有限公司 | Big data excel file import system and method |
| CN112870724A (en) * | 2021-03-11 | 2021-06-01 | 网易（杭州）网络有限公司 | Resource management method, device, storage medium and electronic equipment |
| CN113051279A (en) * | 2021-03-05 | 2021-06-29 | 北京顺达同行科技有限公司 | Data message storage method, storage device, electronic equipment and storage medium |
| CN113407324A (en) * | 2021-06-28 | 2021-09-17 | 浙江太美医疗科技股份有限公司 | Method and device for processing database operation data |
| CN113419824A (en) * | 2021-01-25 | 2021-09-21 | 阿里巴巴集团控股有限公司 | Data processing method, device, system and computer storage medium |
| CN113885802A (en) * | 2021-09-30 | 2022-01-04 | 杭州贝嘟科技有限公司 | Data transmission method, device, electronic device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927248A (en) * | 2014-04-01 | 2014-07-16 | 可牛网络技术（北京）有限公司 | Progress display control method and device and electronic device |
| CN104598574A (en) * | 2015-01-13 | 2015-05-06 | 北京中交兴路车联网科技有限公司 | Method and device for storing massive GPS (global positioning system) data |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
| CN107548490A (en) * | 2014-12-26 | 2018-01-05 | 英特尔公司 | Humidometer in parallel computation |
| CN109325002A (en) * | 2018-09-03 | 2019-02-12 | 北京京东金融科技控股有限公司 | Text file processing method, device, system, electronic equipment, storage medium |
| CN109375968A (en) * | 2018-08-28 | 2019-02-22 | 苏州浪潮智能软件有限公司 | A kind of file uploading method that can graphically interact |
| CN110020359A (en) * | 2017-11-08 | 2019-07-16 | 亿阳信通股份有限公司 | Apply the data processing method in webpage front-end, device and storage medium |
-
2019
- 2019-12-31 CN CN201911419917.4A patent/CN111158939A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927248A (en) * | 2014-04-01 | 2014-07-16 | 可牛网络技术（北京）有限公司 | Progress display control method and device and electronic device |
| CN107548490A (en) * | 2014-12-26 | 2018-01-05 | 英特尔公司 | Humidometer in parallel computation |
| CN104598574A (en) * | 2015-01-13 | 2015-05-06 | 北京中交兴路车联网科技有限公司 | Method and device for storing massive GPS (global positioning system) data |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
| CN110020359A (en) * | 2017-11-08 | 2019-07-16 | 亿阳信通股份有限公司 | Apply the data processing method in webpage front-end, device and storage medium |
| CN109375968A (en) * | 2018-08-28 | 2019-02-22 | 苏州浪潮智能软件有限公司 | A kind of file uploading method that can graphically interact |
| CN109325002A (en) * | 2018-09-03 | 2019-02-12 | 北京京东金融科技控股有限公司 | Text file processing method, device, system, electronic equipment, storage medium |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100127A (en) * | 2020-11-12 | 2020-12-18 | 广州市玄武无线科技股份有限公司 | Big data excel file import system and method |
| CN113419824A (en) * | 2021-01-25 | 2021-09-21 | 阿里巴巴集团控股有限公司 | Data processing method, device, system and computer storage medium |
| CN113051279A (en) * | 2021-03-05 | 2021-06-29 | 北京顺达同行科技有限公司 | Data message storage method, storage device, electronic equipment and storage medium |
| CN113051279B (en) * | 2021-03-05 | 2024-05-10 | 北京顺达同行科技有限公司 | Storage method, storage device, electronic equipment and storage medium for data message |
| CN112870724A (en) * | 2021-03-11 | 2021-06-01 | 网易（杭州）网络有限公司 | Resource management method, device, storage medium and electronic equipment |
| CN112870724B (en) * | 2021-03-11 | 2024-05-17 | 网易（杭州）网络有限公司 | Resource management method and device, storage medium and electronic equipment |
| CN113407324A (en) * | 2021-06-28 | 2021-09-17 | 浙江太美医疗科技股份有限公司 | Method and device for processing database operation data |
| CN113407324B (en) * | 2021-06-28 | 2024-03-29 | 浙江太美医疗科技股份有限公司 | Database operation data processing method and device |
| CN113885802A (en) * | 2021-09-30 | 2022-01-04 | 杭州贝嘟科技有限公司 | Data transmission method, device, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111158939A (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN106503020B (en) | Log data processing method and device | |
| US9811577B2 (en) | Asynchronous data replication using an external buffer table | |
| US9128615B2 (en) | Storage systems that create snapshot queues | |
| US9246836B2 (en) | Single producer, single consumer lockless FIFO/LIFO queue | |
| CN112035571A (en) | Data synchronization method, device, equipment and storage medium | |
| CN112925792B (en) | Data storage control method, device, computing equipment and medium | |
| US20140365429A1 (en) | Method for issuing multipart receipts in connection to extensive database operations | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN115525631A (en) | Database data migration method, device, equipment and storage medium | |
| US8600990B2 (en) | Interacting methods of data extraction | |
| US7958083B2 (en) | Interacting methods of data summarization | |
| US11099960B2 (en) | Dynamically adjusting statistics collection time in a database management system | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN114428688A (en) | Online rapidly-shared table content extracting and processing method | |
| CN107479960B (en) | Method and device for displaying unit view | |
| US9852171B2 (en) | Index masking based on insert contention ratio | |
| WO2023197865A1 (en) | Information storage method and apparatus | |
| CN109240995B (en) | Method and device for counting time delay of operation word | |
| US9798626B2 (en) | Implementing change data capture by interpreting published events as a database recovery log | |
| CN104750547B (en) | The input and output I/O request processing method and processing device of virtual machine | |
| CN115981893A (en) | Message queue task processing method and device, server and storage medium | |
| CN111831206B (en) | Storage space switching method and back-end monitoring system | |
| WO2017001900A1 (en) | A data processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200515 |