CN110069493A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110069493A CN110069493A CN201910151904.7A CN201910151904A CN110069493A CN 110069493 A CN110069493 A CN 110069493A CN 201910151904 A CN201910151904 A CN 201910151904A CN 110069493 A CN110069493 A CN 110069493A
- Authority
- CN
- China
- Prior art keywords
- data
- user request
- request data
- thread
- list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 33
- 238000000034 method Methods 0.000 claims description 25
- 238000003780 insertion Methods 0.000 claims description 23
- 230000037431 insertion Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 239000007858 starting material Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/244—Grouping and aggregation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application belongs to big data technical field, it is related to a kind of data processing method, it include: starting first thread, and user's request data is scanned by first thread, if scanning includes preset request data into user's request data and user's request data, user's request data is deposited into the list of the first thread, user's request data is inserted into database according still further to the sequence in list.The application also provides a kind of data processing equipment, computer equipment and storage medium.Using the application, it can be according to the match condition of user's request data and preset data, user's request is inserted into database using thread, the situation for linking no response is avoided to occur, and then effectively improve data processing speed, even if can also ensure that link operates normally in the case where big data quantity is concurrent simultaneously.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.
Background
With the rapid growth of mobile internet services and the number of users, the number of users and user-related information handled by various companies is increasing. When a large number of users send requests at the same time, corresponding user information is also inserted into the database, and a large number of links are occupied, so that no response is caused to the links sometimes, and the data processing efficiency is further influenced.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data processing method, an apparatus, a computer device, and a storage medium, which can solve the problem of link bursting caused by a large number of user requests.
In order to solve the foregoing technical problem, an embodiment of the present application provides a data processing method, which is applied to a computer device, and adopts the following technical solutions: the method comprises the following steps:
starting a first thread, and scanning user request data through the first thread;
if the user request data are scanned, judging whether the user request data comprise preset request data or not;
when the user request data is judged to comprise preset request data, storing the user request data into a list of the first thread;
and inserting the user request data into the database according to the sequence in the list.
Further, the step of scanning the user request data through the first thread specifically includes:
setting a time interval parameter;
and the first thread scans the user request data at intervals of the time interval parameter.
Further, before the step of storing the user request data in the list of the first thread, the method further includes:
when the user request data is judged to comprise preset request data, judging whether the data volume of the user request data reaches a preset data volume threshold value;
and when the data volume of the user request data is judged to reach a preset data volume threshold value, further judging whether the number of the first threads is one or more than one.
Further, the step of storing the user request data in the list of the first thread specifically includes:
and when the data volume of the user data reaches a preset data volume threshold value and the number of the first threads is more than one, storing the user request data into at least two lists of the first threads according to a preset rule.
Further, the step of storing the user request data in the list of the first thread specifically includes:
when the data volume of the user data is judged to reach a preset data volume threshold value and the number of the first threads is one, dividing the data volume of the user request data into at least two sub data packets, wherein the data volume of each sub data packet is smaller than or equal to the data volume threshold value;
storing a sub-packet in the list of the first thread.
Further, after storing the user request data in the list of the first thread, the method further comprises:
restarting at least one second thread;
and storing the data of the sub data packet subtracted from the user request data into the list of the at least one second thread.
Further, after the step of inserting the user request data into the database in the order in the list, the method further comprises:
and when an instruction for suspending insertion or closing the instruction is received, stopping inserting the data into the database at the moment of issuing the instruction until the instruction for resuming the insertion is received, and inserting the residual non-inserted preset data into the database.
In order to solve the above technical problem, an embodiment of the present application further provides a data processing apparatus, which adopts the following technical solution: the device comprises:
the starting module is used for starting a first thread and scanning user request data through the first thread;
the judging module is used for judging whether the user request data comprises preset request data or not if the user request data is scanned;
the storage module is used for storing the user request data into the list of the first thread when the user request data is judged to comprise preset request data;
and the data insertion module is used for inserting the user request data into the database according to the sequence in the list.
In order to solve the above technical problem, an embodiment of the present application provides a computer device, which adopts the following technical solutions: the computer device comprises a memory in which a computer program is stored and a processor which implements the steps of the data processing method described above when executing the computer program.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions: the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the data processing method as described above.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
and starting a first thread, scanning user request data through the first thread, storing the user request data into a list of the first thread if the user request data are scanned and include preset request data, and then inserting the user request data into a database according to the sequence in the list. By adopting the data processing method, the data processing device, the computer equipment and the storage medium, the user request can be inserted into the database by utilizing the thread according to the matching condition of the user request data and the preset data, the condition that the link has no response is avoided, the data processing speed is further effectively improved, and the normal operation of the link can be ensured even under the condition that a large data volume is simultaneously concurrent.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of a data processing method according to the present application;
FIG. 3 is a flowchart of one embodiment of step S201 in FIG. 2;
FIG. 4 is a flow diagram of another embodiment of a data processing method according to the present application;
FIG. 5 is a flowchart of one embodiment of step S203 in FIG. 2;
FIG. 6 is a schematic block diagram of one embodiment of a data processing apparatus according to the present application;
FIG. 7 is a schematic diagram of one embodiment of the starter module of FIG. 6;
FIG. 8 is a schematic block diagram of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that the data processing method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the data processing apparatus is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continuing reference to FIG. 2, a flow diagram of one embodiment of a data processing method according to the present application is shown. The data processing method comprises the following steps:
step 201, a first thread is started, and user request data is scanned through the first thread.
Specifically, in this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the data processing method operates may receive the request of the user through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
Continuing to refer to FIG. 3, which illustrates a sub-flowchart of one embodiment of a data processing method according to the present application, step 201 further comprises:
step 301, setting a time interval parameter.
Step 302, the first thread scans the user request data at intervals of the time interval parameter.
More specifically, a timer is started in the first thread, a user request is periodically scanned according to a time interval parameter set by the timer, whether the user request exists or not is detected, and if the user request exists, user request data are acquired. The user request may be a loading request for entering a web page, or a request for triggering a function key, for example: submit a policy, etc.
The number of threads may be one or more. If the number of threads is one, one thread is started as follows, and one thread is started by way of thread. If the number of threads is at least two, starting other first threads as follows: the other first thread is started by writing thread.
In other embodiments, it may also be determined whether the number of user requests at the current time is greater than a preset number, if so, the process proceeds to step 201, and if not, the data is processed according to a normal flow. The preset number may be one hundred, two hundred, three hundred, or any number. That is, the data processing method of the present invention can be applied to a case where many users simultaneously issue request data.
Step 202, if the user request data is scanned, determining whether the user request data includes preset request data. If yes, go to step 203, otherwise, the process ends.
Specifically, it is determined whether the user request data includes preset request data, where the preset request data is specified data content. For example, the preset request data may be a loading request of a web page.
Step 203, storing the user request data into the list of the first thread.
Specifically, when the User request data includes preset request data, the User data is stored in the list < User Date > of the first thread.
Step 204, inserting the user request data into the database according to the sequence in the list.
Specifically, the user data is inserted into the database in a piece-by-piece manner through a well-defined SQL (Structured Query Language) statement.
Referring to fig. 4, which shows a flowchart of another embodiment of the data processing method according to the present application, between step 202 and step 203, the data processing method may further include:
step 401, determining whether the data volume of the user request data reaches a preset data volume threshold.
Specifically, step 401 is different from the previous step in determining whether the data amount requested by the user reaches the preset amount, and the data amount threshold in step 401 is greater than the preset amount mentioned in step 201. In step 401, when the data amount requested by the user is greater than or equal to the data amount threshold, which indicates that the user requests too much data and the data amount is too large, a large number of links are occupied, and it is determined whether to start multithreading to process data by determining the relationship between the data amount of the user requested data and the data amount threshold.
If the data volume requested by the user is smaller than the data volume threshold, which indicates that the current data volume can be borne, processing the data according to a normal flow, namely: data requested by the user may be inserted into the database via a link.
Step 402, when it is determined that the data amount of the user request data reaches a preset data amount threshold, further determining whether the number of the first threads is one or more than one.
Specifically, when the data volume of the user request data reaches a preset data volume threshold, it is further determined whether the number of the first threads is one or more than one. When it is determined that the number of first threads is more than one, then the process of starting more than one first thread may be: more than one first thread is started by writing thread.
In this embodiment, the process of step 203 is also different according to the data amount of the user request data and the number of threads, specifically, referring to fig. 5, it shows that the step of storing the user request data into the list of the first thread in the data processing method according to the present application specifically includes:
step 501, when it is determined that the data size of the user data reaches a preset data size threshold and the number of the first threads is one, dividing the data size of the user request data into at least two sub-packets, wherein the data size of each sub-packet is smaller than or equal to the data size threshold;
step 503, storing a sub-packet in the list of the first thread.
Specifically, by comparing the user request data volume with the data volume threshold, a number interval corresponding to the user request data volume can be determined, the data volume of the preset data is determined to be divided into a plurality of sub-packets according to the corresponding number interval, and one sub-packet is first allocated to the list of the first thread for data processing.
Preferably, in the plurality of sub packets, the data volume of each sub packet may be the same or different, and it is sufficient to ensure that the data volume of each sub packet is less than or equal to the data volume threshold.
When the data amount of the sub-packet is less than or equal to the preset data amount threshold, step 503 is: and storing data which is equal to the data quantity threshold value in the user request data into the list of the first thread.
Further, after storing the user request data into the list of the first thread, the data processing method according to the embodiment of the present application further includes:
restarting at least one second thread;
and storing the data of the sub data packet subtracted from the user request data into the list of the at least one second thread.
When the data volume of the sub-data packet is greater than the preset data volume threshold, the method comprises the following steps: storing the data of the user request data minus the data amount threshold into the list of the at least one second thread.
The data processing method of the embodiment can divide the user request data into a plurality of sub-packets, and allocate the plurality of sub-packets to a plurality of threads for data processing.
In other embodiments, the step of dividing the sub-packets may be omitted, and the user requests may be randomly distributed to multiple threads for data processing.
In the embodiment of the application, the specific process of storing the user request into the multiple threads is as follows: when the number of the threads is multiple, namely the situation of multithread Data processing, the Data of each User is stored into the list of the threads, so that a list < User Data > set is obtained, one thread is restarted, and meanwhile, the list set is transmitted into the restarted thread list.
In other embodiments, the data packet requested by the user may be correspondingly allocated to multiple threads for data processing according to the carrying capacities of different threads.
Further, when it is determined that the data amount of the user data reaches a preset data amount threshold and the number of the first threads is more than one, the user request data is stored in a list of at least two first threads.
Further, when it is determined that the data amount of the user does not reach the preset data amount threshold, the user request may be stored in a current list of the first thread, where the first thread may be one thread or more than one thread, and the embodiment of the present application is not limited specifically.
Further, referring to fig. 2 again, step 204 in the data processing method of the embodiment of the present application specifically includes:
the preset data in the list is grouped, and the preset data can be grouped according to the number of the preset data, or the preset data can be grouped according to the size of the data size, and the preset data is divided into a first group of data, a second group of data and the like. The first group of data is inserted into the database, and then the second group of data is inserted into the database until all the groups of data are successfully inserted into the database.
In other embodiments, data insertion may not be per packet, but rather de-partitioned according to the amount of data. That is, dividing the data into a first amount of data and a second amount of data, inserting the first amount of data into the database, determining whether the list has the un-uploaded preset data, if so, inserting the second amount of data into the database (the second amount may be equal to the first amount, or may be greater than or less than the first amount), determining whether the list has the un-uploaded preset data, if so, inserting the third amount of data into the database (the third amount may be equal to the first amount, or may be greater than or less than the first amount), …, until determining that the to-be-processed data in the list is zero, thereby completing the uploading.
After data is inserted once, whether the inserted data is inserted successfully or not is judged, and if yes, the next data uploading is carried out. And if the uploading is not successful, the uploading is carried out again.
In an alternative embodiment of the present application, the corresponding set of data is deleted after each insertion of data into the database, e.g., the first set of data is deleted after the first set of data is inserted into the database, and so on. Those skilled in the art will appreciate that in other embodiments, the first set of data may not be deleted, and all buffered data may be periodically cleared according to the setting.
In another alternative embodiment of the present application, corresponding processing may be performed according to the importance level of the preset data. Specifically, data of low importance is deleted, and data of high importance is backed up.
For the case of multithreading, the newly started second thread inserts the preset data in the list set into the database, thereby realizing that the preset data can be inserted into the database through one connection.
In the embodiment of the present application, when it is determined that the user request data does not include the preset data, the process ends.
Further, after the step of inserting the user request data into the database in the order in the list, the data processing method of the present application further includes:
and when an instruction for suspending insertion or closing the instruction is received, stopping inserting the data into the database at the moment of issuing the instruction until the instruction for resuming the insertion is received, and inserting the residual non-inserted preset data into the database.
Specifically, when an instruction for suspending insertion or closing, which is initiated by a user, is received, the data is stopped from being inserted into the database at the time of issuing the instruction until an instruction for resuming the insertion, which is initiated by the user, is received, and the remaining non-inserted preset data is inserted into the database.
According to the data processing method, when the first thread is started, the user request data are scanned through the first thread, if the user request data are scanned and include preset request data, the user request data are stored in the list of the first thread, and then the user request data are inserted into the database according to the sequence in the list. By adopting the data processing method, the user request can be inserted into the database by utilizing the thread according to the matching condition of the user request data and the preset data, the condition that the link has no response is avoided, the data processing speed is further effectively improved, and the normal operation of the link can be ensured even under the condition that a large amount of data is concurrent at the same time.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 6, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a data processing apparatus, which corresponds to the embodiment of the method shown in fig. 2, and which can be applied in various electronic devices.
As shown in fig. 6, the data processing apparatus 600 according to the present embodiment includes: a starting module 601, a judging module 602, a storing module 603 and a data inserting module 604. Wherein:
the starting module 601 is configured to start a first thread and scan user request data through the first thread.
Referring to fig. 7, which is a schematic structural diagram of a specific embodiment of the starting module 601, the starting module 601 includes a setting sub-module 6011 and a scanning sub-module 6012. Wherein,
a setting submodule 6011 is configured to set the time interval parameter.
The scanning sub-module 6012 is configured to scan the user request data by the first thread every time of the time interval parameter.
More specifically, a timer is started in the first thread, the user request is periodically scanned according to a time interval parameter set by the timer, the scanning sub-module 6012 detects whether the user request exists, and if the user request exists, the user request data is acquired. The user request may be a loading request for entering a web page, or a request for triggering a function key, for example: submit a policy, etc.
The number of threads may be one or more. If the number of threads is one, one thread is started as follows, and one thread is started by way of thread. If the number of threads is at least two, starting other first threads as follows: the other first thread is started by writing thread.
In other embodiments, it may also be determined whether the number of user requests at the current time is greater than a preset number, if so, the starting module 601 is triggered, and if not, the data is processed according to a normal flow. The preset number may be one hundred, two hundred, three hundred, or any number. That is, the data processing method of the present invention can be applied to a case where many users simultaneously issue request data.
The determining module 602, if the user request data is scanned, is configured to determine whether the user request data includes preset request data.
Specifically, it is determined whether the user request data includes preset request data, where the preset request data is specified data content. For example, the preset request data may be a loading request of a web page.
The storage module 603 is configured to, when it is determined that the user request data includes preset request data, store the user request data in the list of the first thread.
Specifically, when the User request data includes the predetermined request data, the User data is stored in the list < User Date > of the first thread.
A data inserting module 604, configured to insert the user request data into the database according to the order in the list.
Specifically, the user data is inserted into the database in a piece-by-piece manner through a well-defined SQL (Structured Query Language) statement.
Further, the determining module 602 is further configured to determine whether a data amount of the user request data reaches a preset data amount threshold.
Specifically, the data amount threshold is different from the data amount threshold determined by the determining module 602 whether the data amount requested by the user reaches the preset amount, and the data amount threshold is greater than the preset amount. When the data volume requested by the user is larger than or equal to the data volume threshold, the fact that the user requests too much data volume and occupies a large number of links is indicated, and whether multithreading is to be started for processing the data or not is determined by judging the relation between the data volume of the data requested by the user and the data volume threshold.
If the data volume requested by the user is smaller than the data volume threshold, which indicates that the current data volume can be borne, processing the data according to a normal flow, namely: data requested by the user may be inserted into the database via a link.
When it is determined that the data amount of the user request data reaches the preset data amount threshold, the determining module 602 is further configured to further determine whether the number of the first threads is one or more than one.
Specifically, when the data volume of the user request data reaches a preset data volume threshold, it is further determined whether the number of the first threads is one or more than one. When it is determined that the number of first threads is more than one, then the process of starting more than one first thread may be: more than one first thread is started by writing thread.
In this embodiment, the storage module 603 also has different functions according to the data amount of the user request data and the number of threads. Specifically, the storage module 603 is specifically configured to:
when the data volume of the user data is judged to reach a preset data volume threshold value and the number of the first threads is one, dividing the data volume of the user request data into at least two sub data packets, wherein the data volume of each sub data packet is smaller than or equal to the data volume threshold value; storing a sub-packet in the list of the first thread.
Specifically, by comparing the user request data volume with the data volume threshold, a number interval corresponding to the user request data volume can be determined, the data volume of the preset data is determined to be divided into a plurality of sub-packets according to the corresponding number interval, and one sub-packet is first allocated to the list of the first thread for data processing.
Preferably, in the plurality of sub packets, the data volume of each sub packet may be the same or different, and it is sufficient to ensure that the data volume of each sub packet is less than or equal to the data volume threshold.
When the data amount of the sub data packet is less than or equal to the preset data amount threshold, the storage module 603 is specifically configured to: restarting at least one second thread, and storing data in the user request data, which is equal to the data amount threshold, into the list of the first thread.
Further, after storing the user request data into the list of the first thread, the storage module 603 is further configured to:
and storing the data of the sub data packet subtracted from the user request data into the list of the at least one second thread.
When the data amount of the sub data packet is greater than the preset data amount threshold, the storage module 603 is further configured to: storing the data of the user request data minus the data amount threshold into the list of the at least one second thread.
The data processing apparatus 600 of the present embodiment can divide the user request data into a plurality of sub packets, and allocate the plurality of sub packets to a plurality of threads to perform data processing.
In other embodiments, the step of dividing the sub-packets may be omitted, and the user requests may be randomly distributed to multiple threads for data processing.
In this embodiment of the application, the specific process of the storage module 603 storing the user request into the multiple threads is as follows: when the number of the threads is multiple, namely the situation of multithread Data processing, the Data of each User is stored into the list of the threads, so that a list < User Data > set is obtained, one thread is restarted, and meanwhile, the list set is transmitted into the restarted thread list.
In other embodiments, the data packet requested by the user may be correspondingly allocated to multiple threads for data processing according to the carrying capacities of different threads.
Further, when the determining module 602 determines that the data amount of the user data reaches a preset data amount threshold and the number of the first threads is more than one, the storing module 603 stores the user request data in a list of at least two first threads.
Further, when the determining module 602 determines that the data amount of the user does not reach the preset data amount threshold, the storing module 603 may store the user request in a current list of a first thread, where the first thread may be one thread or more than one thread, and the embodiment of the present application is not limited specifically.
Further, the data insertion module 604 is specifically configured to:
the preset data in the list is grouped, and the preset data can be grouped according to the number of the preset data, or the preset data can be grouped according to the size of the data size, and the preset data is divided into a first group of data, a second group of data and the like. The first group of data is inserted into the database, and then the second group of data is inserted into the database until all the groups of data are successfully inserted into the database.
In other embodiments, data insertion may not be per packet, but rather de-partitioned according to the amount of data. That is, dividing the data into a first amount of data and a second amount of data, inserting the first amount of data into the database, determining whether the list has the un-uploaded preset data, if so, inserting the second amount of data into the database (the second amount may be equal to the first amount, or may be greater than or less than the first amount), determining whether the list has the un-uploaded preset data, if so, inserting the third amount of data into the database (the third amount may be equal to the first amount, or may be greater than or less than the first amount), …, until determining that the to-be-processed data in the list is zero, thereby completing the uploading.
After data is inserted once, whether the inserted data is inserted successfully or not is judged, and if yes, the next data uploading is carried out. And if the uploading is not successful, the uploading is carried out again.
In an alternative embodiment of the present application, the corresponding set of data is deleted after each insertion of data into the database, e.g., the first set of data is deleted after the first set of data is inserted into the database, and so on. Those skilled in the art will appreciate that in other embodiments, the first set of data may not be deleted, and all buffered data may be periodically cleared according to the setting.
In another alternative embodiment of the present application, corresponding processing may be performed according to the importance level of the preset data. Specifically, data of low importance is deleted, and data of high importance is backed up.
For the case of multithreading, the newly started second thread inserts the preset data in the list set into the database, thereby realizing that the preset data can be inserted into the database through one connection.
When an instruction for suspending insertion or closing the instruction is received, the data insertion module 604 stops inserting data into the database at the time of issuing the instruction until the instruction for resuming the insertion is received, and the data insertion module 604 inserts the remaining non-inserted preset data into the database.
Specifically, when an instruction for suspending insertion or closing, which is initiated by a user, is received, the data insertion module 604 stops inserting data into the database at the time of issuing the instruction until an instruction for resuming insertion, which is initiated by the user, is received, and the data insertion module 604 inserts the remaining non-inserted preset data into the database.
In the data processing apparatus 600 according to the embodiment of the application, when the starting module 601 starts the first thread, scans the user request data through the first thread, and if the user request data is scanned and the determining module 602 determines that the user request data includes the preset request data, the storage module 603 stores the user request data in the list of the first thread, and the data inserting module 604 inserts the user request data into the database according to the sequence in the list. By adopting the data processing device, the user request can be inserted into the database by utilizing the thread according to the matching condition of the user request data and the preset data, the condition that the link has no response is avoided, the data processing speed is further effectively increased, and the normal operation of the link can be ensured even under the condition that a large data volume is concurrent at the same time.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 8, fig. 8 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 8 comprises a memory 81, a processor 82, a network interface 83 communicatively connected to each other via a system bus. It is noted that only computer device 8 having components 81-83 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 81 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 81 may be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a flash Card (FlashCard), and the like, which are provided on the computer device 8. Of course, the memory 81 may also comprise both an internal storage unit of the computer device 8 and an external storage device thereof. In this embodiment, the memory 81 is generally used for storing an operating system installed in the computer device 8 and various types of application software, such as program codes of a data processing method. Further, the memory 81 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 82 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is configured to execute the program code stored in the memory 81 or process data, for example, execute the program code of the data processing method.
The network interface 63 may comprise a wireless network interface or a wired network interface, and the network interface 63 is typically used to establish a communication connection between the computer device 8 and other electronic devices.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing a data processing program, which is executable by at least one processor to cause the at least one processor to perform the steps of the data processing method as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A data processing method applied to computer equipment is characterized by comprising the following steps:
starting a first thread, and scanning user request data through the first thread;
if the user request data are scanned, judging whether the user request data comprise preset request data or not;
when the user request data is judged to comprise preset request data, storing the user request data into a list of the first thread;
and inserting the user request data into the database according to the sequence in the list.
2. The data processing method according to claim 1, wherein the step of scanning the user request data by the first thread specifically comprises:
setting a time interval parameter;
and the first thread scans the user request data at intervals of the time interval parameter.
3. The data processing method of claim 1, wherein the step of storing the user request data in the list of first threads is preceded by the step of:
when the user request data is judged to comprise preset request data, judging whether the data volume of the user request data reaches a preset data volume threshold value;
and when the data volume of the user request data is judged to reach a preset data volume threshold value, judging whether the number of the first threads is one or more than one.
4. The data processing method according to claim 3, wherein the step of storing the user request data in the list of the first thread specifically comprises:
and when the data volume of the user data reaches a preset data volume threshold value and the number of the first threads is more than one, storing the user request data into at least two lists of the first threads.
5. The data processing method according to claim 3, wherein the step of storing the user request data in the list of the first thread specifically comprises:
when the data volume of the user data is judged to reach a preset data volume threshold value and the number of the first threads is one, dividing the data volume of the user request data into at least two sub data packets, wherein the data volume of each sub data packet is smaller than or equal to the data volume threshold value;
storing a sub-packet in the list of the first thread.
6. The data processing method of claim 5, wherein after storing the user request data in the list of first threads, the method further comprises:
starting at least one second thread;
and storing the data of the sub data packet subtracted from the user request data into the list of the at least one second thread.
7. The data processing method of claim 1, wherein after the step of inserting the user request data into the database in the order in the list, the method further comprises:
and when an instruction for suspending insertion or closing the instruction is received, stopping inserting the data into the database at the moment of issuing the instruction until the instruction for resuming the insertion is received, and inserting the residual non-inserted preset data into the database.
8. A data processing device applied to computer equipment is characterized by comprising:
the starting module is used for starting a first thread and scanning user request data through the first thread;
the judging module is used for judging whether the user request data comprises preset request data or not if the user request data is scanned;
the storage module is used for storing the user request data into the list of the first thread when the user request data is judged to comprise preset request data;
and the data insertion module is used for inserting the user request data into the database according to the sequence in the list.
9. A computer device comprising a memory in which a computer program is stored and a processor which, when executing the computer program, carries out the steps of the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910151904.7A CN110069493B (en) | 2019-02-28 | 2019-02-28 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910151904.7A CN110069493B (en) | 2019-02-28 | 2019-02-28 | Data processing method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110069493A true CN110069493A (en) | 2019-07-30 |
| CN110069493B CN110069493B (en) | 2024-05-07 |
Family
ID=67366006
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910151904.7A Active CN110069493B (en) | 2019-02-28 | 2019-02-28 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110069493B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201319A (en) * | 2022-02-17 | 2022-03-18 | 广东东华发思特软件有限公司 | Data scheduling method, device, terminal and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015060285A (en) * | 2013-09-17 | 2015-03-30 | 株式会社日立システムズ | Global transaction processing method in cloud computing |
| CN107729559A (en) * | 2017-11-08 | 2018-02-23 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of data base read-write asynchronous access |
| CN108897876A (en) * | 2018-06-29 | 2018-11-27 | 中科鼎富（北京）科技发展有限公司 | A kind of data cut-in method and device |
| CN109144862A (en) * | 2018-08-10 | 2019-01-04 | 深圳市小牛在线互联网信息咨询有限公司 | Statistical method, device, computer equipment and the storage medium of test data |
-
2019
- 2019-02-28 CN CN201910151904.7A patent/CN110069493B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015060285A (en) * | 2013-09-17 | 2015-03-30 | 株式会社日立システムズ | Global transaction processing method in cloud computing |
| CN107729559A (en) * | 2017-11-08 | 2018-02-23 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of data base read-write asynchronous access |
| CN108897876A (en) * | 2018-06-29 | 2018-11-27 | 中科鼎富（北京）科技发展有限公司 | A kind of data cut-in method and device |
| CN109144862A (en) * | 2018-08-10 | 2019-01-04 | 深圳市小牛在线互联网信息咨询有限公司 | Statistical method, device, computer equipment and the storage medium of test data |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201319A (en) * | 2022-02-17 | 2022-03-18 | 广东东华发思特软件有限公司 | Data scheduling method, device, terminal and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110069493B (en) | 2024-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110519401B (en) | Method, device, equipment and storage medium for improving network access success rate | |
| CN112765271B (en) | Block chain transaction index storage method and device, computer equipment and medium | |
| CN111414334A (en) | File fragment uploading method, device, equipment and storage medium based on cloud technology | |
| CN108810831B (en) | Short message verification code pushing method, electronic device and readable storage medium | |
| WO2019179027A1 (en) | Electronic device, firewall provisioning verification method, system and storage medium | |
| EP2696285A1 (en) | Method and device for allocating resource | |
| CN110620681B (en) | Network connection timeout time setting method, device, equipment and medium | |
| CN110633299A (en) | Business data caching method and device, computer equipment and storage medium | |
| CN111782304A (en) | Logic control method and device for paging loading data, computer equipment and medium | |
| CN113805962B (en) | Application page display method and device and electronic equipment | |
| CN109117153B (en) | Application processing method, device, terminal and storage medium | |
| CN113254445A (en) | Real-time data storage method and device, computer equipment and storage medium | |
| CN115455058A (en) | Cache data processing method and device, computer equipment and storage medium | |
| CN113254106B (en) | Task execution method and device based on Flink, computer equipment and storage medium | |
| CN110069493B (en) | Data processing method, device, computer equipment and storage medium | |
| CN108491466B (en) | Electronic device, access instruction information acquisition method, and storage medium | |
| CN110677506A (en) | Network access method, device, computer equipment and storage medium | |
| CN112559131A (en) | Method, device and equipment for updating container mirror image and computer readable storage medium | |
| CN107657155B (en) | Method and device for authenticating user operation authority | |
| CN112364187A (en) | Automobile accessory database building method, device and equipment based on big data | |
| CN111475388A (en) | Data push test method and device, computer equipment and storage medium | |
| CN108595262B (en) | Data processing method and device | |
| CN108428182B (en) | Development platform device, method for acquiring policy number, and computer-readable storage medium | |
| CN113703793B (en) | Application installation method, device, equipment and medium for cloud mobile phone | |
| CN114995914A (en) | Picture data processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |