CN112835945A - User data-based label processing method, system, device and storage medium - Google Patents
User data-based label processing method, system, device and storage medium Download PDFInfo
- Publication number
- CN112835945A CN112835945A CN202110213714.0A CN202110213714A CN112835945A CN 112835945 A CN112835945 A CN 112835945A CN 202110213714 A CN202110213714 A CN 202110213714A CN 112835945 A CN112835945 A CN 112835945A
- Authority
- CN
- China
- Prior art keywords
- user data
- target user
- execution server
- processing
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 117
- 238000002372 labelling Methods 0.000 claims abstract description 12
- 238000004364 calculation method Methods 0.000 claims description 35
- 238000000034 method Methods 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000010923 batch production Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Abstract
The invention discloses a label processing method based on user data, which comprises the following steps: the main control server receives the trigger operation of the timing task, generates a task processing instruction according to the trigger operation scheduling, and sends the task processing instruction to an execution server corresponding to the given time task; executing a server task processing instruction to obtain a command number and a query label corresponding to a timing task; the execution server acquires at least one target user data of the line number corresponding to the token number from a preset database according to the acquired token number and the query tag; the execution server groups at least one target user data according to a preset grouping strategy to obtain multiple groups of target user data; and the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets, and performs labeling processing on a group of target user data associated with the thread pools through each thread pool. The invention improves the efficiency of processing data by the timing task.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a label processing method, a system, equipment and a storage medium based on user data.
Background
The data of the client is changed in real time, in order to adapt to the change, the system needs to calculate various service tags of the client in real time, such as credit line, credit qualification, historical loan data and the like, but the timeliness of many service tags is not high, such as whether overdue exists or not, whether historical overdue days reach X days or not and the like, the tags can be kept unchanged in time dimensions of year, month, week, day, time and the like according to timeliness, and the frequent calculation of the tags in use is time-consuming and labor-consuming.
One type of optimization scheme for such data is generally to run batch at regular time, batch process is performed on time-sensitive labels of a certain time dimension of users, the data of all users are generally processed in a single machine in a circulating mode, the labels are marked one by one, after the processing is completed, result labels are placed in a database, and when the result labels are used, the result labels are directly inquired from label results of the database. The method has extremely low efficiency, and when tens of millions of user data need to be processed, the timeliness requirement can only meet the needs of a week or a day generally, and the applicable scene of the scheme is low. In addition, all data are processed by one execution server, so that the processing time is long, the resources of the execution server are greatly consumed, and the processing is not beneficial to the failure compensation processing.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a tag processing method, system, device and storage medium based on user data, which are used to solve the problem of low efficiency when a timing task processes tens of millions of data.
In order to achieve the above object, an embodiment of the present invention provides a tag processing method based on user data, which is applied to a distributed cluster, where the distributed cluster includes a master control server and a plurality of execution servers, and the master control server is configured to schedule the execution servers, and includes:
the main control server receives triggering operation of a timing task, generates a task processing instruction according to the triggering operation, and sends the task processing instruction to an execution server corresponding to the timing task;
the execution server side obtains a token number according to the task processing instruction;
the execution server side acquires a query tag corresponding to the timing task according to the task processing instruction;
the execution server side obtains at least one target user data of the line number corresponding to the token number from a preset database according to the obtained token number and the query label;
the execution server side groups the at least one target user data according to a preset grouping strategy to obtain a plurality of groups of target user data;
the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets, and performs labeling processing on a group of target user data associated with the thread pools through each thread pool.
Further, the method further comprises:
the master control server side obtains the number of users to be processed and the number of single batch processing in the timing task;
the master control server side generates a token stack queue according to the number of the users to be processed and the number of single batch processing;
and the master control server stores the token stack queue in a redis database, the token stack queue stores a plurality of token numbers in sequence, and each token number corresponds to user data of single batch processing number.
Further, the method further comprises:
the master control server side obtains trigger operations of a plurality of execution server sides for starting the timing task;
and the master control server acquires the token number from the redis database according to the received trigger operation and distributes the acquired token number to the corresponding execution server.
Further, the method further comprises:
the execution server side obtains a label result after each thread pool is subjected to label printing;
and the execution server side collects the label result of each thread pool and stores the label result in the database.
Further, the step of grouping the at least one target user data by the execution server according to a preset grouping policy to obtain multiple groups of target user data includes:
the execution server side obtains the identity card number data in the at least one target user data;
the execution server calculates the month and the date in the identity card number data to obtain a plurality of groups of target user data;
and the execution server carries out grouping marking on each group of target user data to obtain a plurality of grouping labels corresponding to each group of target user data.
Further, the method further comprises:
the execution server side marks each thread pool to obtain a thread number corresponding to each thread pool;
and the execution server associates the grouping labels of each group of target user data with the thread numbers of the corresponding thread pools.
Further, the step of creating a corresponding number of thread pools by the execution server according to the number of the target user data packets, and labeling a group of target user data associated with the thread pools by each thread pool includes:
the execution server stores each group of target user data in a client temporary table;
the execution server side acquires label calculation data corresponding to the query label from the preset database according to the client temporary table;
the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets;
and the execution server controls each thread pool to calculate the label calculation data corresponding to a group of target user data associated with the thread pool according to the query label, and performs label printing processing on each target user data according to the calculation result.
In order to achieve the above object, an embodiment of the present invention provides a tag processing system based on user data, including:
the main control server is used for receiving triggering operation of a timing task, generating a processing instruction according to the triggering operation, and sending the task processing instruction to an execution server corresponding to the timing task;
the execution server is used for acquiring a token number;
the execution server is further configured to obtain a query tag corresponding to the timing task;
the execution server is further used for acquiring at least one target user data of the line number corresponding to the token number from a preset database according to the acquired token number and the query tag;
the execution server is further configured to group the at least one target user data according to a preset grouping policy to obtain multiple groups of target user data;
the execution server is further configured to create thread pools of corresponding quantities according to the quantity of the target user data packets, and perform labeling processing on a set of target user data associated with the thread pools through each thread pool.
To achieve the above object, an embodiment of the present invention provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the computer program, when executed by the processor, implements the steps of the tag processing method based on user data as described above.
To achieve the above object, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, the computer program being executable by at least one processor to cause the at least one processor to execute the steps of the tag processing method based on user data as described above.
According to the user data-based tag processing method, the system, the equipment and the storage medium provided by the embodiment of the invention, when the execution server side starts a timing task, each execution server side obtains the corresponding token number, so that the execution server side obtains the target user data corresponding to the line number from the database through the token number and the query tag, then the target user data are grouped, and the thread pool with the corresponding grouping number is established after the grouping to mark the target user data, so that the data processing efficiency is improved.
Drawings
Fig. 1 is a flowchart of a first embodiment of a tag processing method based on user data according to the present invention.
Fig. 2 is a schematic diagram of program modules of a second embodiment of a tag processing system based on user data according to the present invention.
Fig. 3 is a schematic diagram of a hardware structure of a third embodiment of the computer device according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Referring to fig. 1, a flowchart illustrating steps of a tag processing method based on user data according to a first embodiment of the present invention is shown. It is to be understood that the flow charts in the embodiments of the present method are not intended to limit the order in which the steps are performed. In the following, an exemplary description is given by taking a computer device 2 as an execution subject, where the computer device 2 may be a distributed cluster system, where the distributed cluster includes a master server and a plurality of execution servers, and the master server is used to schedule the execution servers. The details are as follows.
Step S100, a main control server receives a trigger operation of a timing task, generates a task processing instruction according to the trigger operation, and sends the task processing instruction to an execution server corresponding to the timing task.
Specifically, when the amount of user data processed by the timing task is too large, the user data is processed by a distributed cluster, where the distributed cluster includes a main control server and a plurality of designated servers, and the main control server controls and schedules the execution servers, for example, distributes the timing task to the execution servers. A corresponding timing task can be set in each execution server of the distributed clusters respectively, and the execution server in each distributed cluster triggers the timing task; or the timing tasks of each execution server are managed by the master control server in a unified way, and the master control server triggers the timing tasks of each execution server respectively or in a unified way. After the timing task of each execution server is triggered, the master control execution server performs unified scheduling on each execution server to execute the timing task. Wherein the timing task includes marking for each target user data. When the main control server schedules the execution server, a task processing instruction is generated according to the timing task and is distributed to the execution server, so that the execution server completes marking processing on target user data according to task processing execution. All the execution server sides participate in data processing, so that the utilization rate of the execution server sides is improved, the single machine processing time is reduced, and data processing abnormity caused by the problem of the execution server sides is avoided.
Furthermore, the query tag can be frequently refreshed according to the requirement, and the trigger time of the timing task is set according to the query tag. For example, the requirement on the timeliness of a part of query labels is not high, and the query labels can be in a [ day ] level. For example, if there is overdue record in the last 12 months, whether there is a repayment day currently, whether the overdue days have passed the expiration date, etc., the timing task may be set to be timed to be processed once every morning; the timeliness of some labels is required to be high, and can be in a [ time ] level, for example, whether the current time is out of date, whether a client has a piece in transit, and the like, and the timing task can be set to be processed once every integral point.
And step S102, the execution server side obtains a token number according to the task processing instruction.
Specifically, after the execution server in the distributed cluster performs timing task triggering, each execution server can directly take the token number from the token stack queue in the redis database according to the task processing instruction.
And step S104, the execution server acquires the query tag corresponding to the timing task according to the task processing instruction.
Specifically, the execution server obtains a query tag corresponding to the timing task according to the task processing instruction, where the query tag is a tag that needs to perform marking processing on target user data, for example: and if the time is expired, label calculation data related to the target user data is acquired according to the query label for calculation, and marking processing is performed on the target user data after calculation. The query tags are preset, and a corresponding query tag is set in each timing task.
And step S106, the execution server acquires at least one target user data of the line number corresponding to the token number from a preset database according to the acquired token number and the query label.
Specifically, each execution server acquires user data of a corresponding line number according to a token number, where the line number may be understood as a single batch processing number, and the single batch processing number acquired by each token is consistent, but the line numbers of the user data acquired by each token number are different, so that when the user data of each user is stored, the user data of each user is numbered correspondingly, so as to acquire the user data of the corresponding line number according to the token number. For example: if the token number is 1, user data with the number of 0-1000 is acquired; if the token number is 2, the user data with the 1000-2000 number is obtained. The target user data may be basic information for each user, such as name, identification number, tag calculation parameters required to query the tag, etc. The preset database may be a redis database.
Step S108, the execution server groups the at least one target user data according to a preset grouping strategy to obtain a plurality of groups of target user data.
Specifically, the grouping policy may be set according to requirements, and this embodiment is exemplified by an identification number in the user data.
Exemplarily, the step S108 further includes:
step S108A, the execution server obtains the identification number data in the at least one target user data. Step S108B, the execution service end calculates the month and the date in the identification number data to obtain a plurality of groups of target user data. Step S108C, the execution server performs grouping and marking on each group of target user data, and obtains a plurality of grouping labels corresponding to each group of target user data.
Specifically, the execution server groups the target user data by month and date in the identity card number, and the grouping policy is as follows: the grouping index is set to M _ N, where M is the month of the birthday and N is the number of days that the birthday is to be left over for 3. For example, the birthday of the user data is 20200817, the obtained grouping number is 08_2, and the user data with the consistent grouping number are grouped into one group for marking.
Step S110, the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets, and performs labeling processing on a group of target user data associated with the thread pools through each thread pool.
Specifically, as the grouping strategy is described above, because there are 12 months, and the remainder of 3 is 0, 1, 2, the target user data is divided into 12 × 3 — 36 groups, and the execution server creates 36 thread pools from the 36 groups, each thread pool processing a group of target user data associated therewith.
For example, if the execution server memory and the CPU are powerful enough, the number of the thread pools may be an integer divisor of the number of the query tags, for example, 50 query tags, and the number of the thread pools such as 5, 10, 25, and 50 may be set for processing, which facilitates statistics.
Exemplarily, the step S110 further includes:
step S110A, the execution server stores each set of the target user data in a client temporary table.
Specifically, each target user data includes data associated with a query tag, and storing the target user data in a client temporary table facilitates data query.
Illustratively, the client temporary table is used for temporarily storing each group of target user data, including storing identification card information, name, address, etc. of the target user, and when the execution of the timed task is completed, the data in the client temporary table can be released. When the target user data is stored in the client temporary table C1, a column of fields [ executive server IP ] is added, the executive server record IP1 of the 1 token, and the executive server record IP2 of the 2 token. Each execution server only processes corresponding target user data according to the execution server IP, and when an application table, a borrow table and a client temporary table are associated, because the target user data processed by different execution servers are different, the problem of database resource competition is avoided, and the processing performance is greatly improved.
Step S110B, the execution server obtains the tag computation data corresponding to the query tag from the preset database according to the temporary client table.
Specifically, the tag calculation data is data that needs to be calculated corresponding to the query tag when marking processing is performed on each target user data according to the query tag. And marking the target user data after correspondingly calculating the label calculation data. For example: and when the query tag is overdue, the required tag calculation data is the user information and the repayment data in the target user data, and the repayment data acquires specific tag calculation data, such as the loan amount detail of 1-5 months, from the database through the user borrowing table.
Step S110C, the execution server creates a thread pool with a corresponding number according to the number of the target user data packets.
Specifically, the multithread processing can shorten the processing time for processing the user data corresponding to the whole target tag, the applicable scenes of the timed batch processing task are more, and the implementation sets the thread pools with the number corresponding to the number of the target user data packets so as to save the processing time.
Step S110D, the execution server controls each thread pool to calculate tag calculation data corresponding to a group of target user data associated with the thread pool according to the query tag, and performs tagging processing on each target user data according to a calculation result.
Specifically, the tag calculation data is calculated according to a calculation formula of whether the tag is overdue or not to obtain a result of whether the tag is overdue or not, and the expected result is marked on the corresponding target user data. The thread pool can set a plurality of threads to perform calculation and marking processing according to the size of a group of corresponding target user data, so that the effect of concurrent processing is achieved, and the processing time of the timing task is saved.
Illustratively, the method further comprises:
step S121, the main control server side obtains the number of the users to be processed and the number of single batch processing in the timing task.
Specifically, in order to execute the timing task more quickly, a single batch processing number is preset, and the single batch processing number is the number of user data for each execution server to perform the timing task processing.
And step S122, the master control server side generates a token stack queue according to the number of the users to be processed and the number of single batch processing.
Specifically, a token stack queue storing a plurality of token numbers is generated according to the number of users to be processed and the number of single batch processing. For example, when the total number of 10000 users to be processed is 1000, 10 token numbers are generated.
Step S123, the master control server stores the token stack queue in a redis database, the token stack queue stores a plurality of token numbers in sequence, and each token number corresponds to user data of a single batch processing number.
Specifically, each token number is stored in the token stack queue according to a first-in first-out principle, and the pre-stored token number is taken out first. For example, the above-mentioned 10 token numbers store token number 1, then token number 2, and finally token number 10. The push method of the redis is circularly used, numbers are put into a queue set, when a timing task is started, each execution server side can use a pull interface of the redis to obtain a number, namely a token number, from the list, after the numbers are obtained, the numbers leave the queue, different execution servers are ensured to obtain different numbers, and target user data of single batch processing numbers in different intervals are obtained according to the numbers.
Illustratively, the method further comprises:
step S131, the main control server side obtains a plurality of trigger operations of executing the server side to start the timing task. And step S132, the master control server acquires the token number from the redis database according to the received trigger operation, and distributes the acquired token number to the corresponding execution server.
Specifically, when each execution server triggers a timing task, token numbers of a corresponding number may be obtained from the token stack queue according to the number of the started execution servers, and then distributed to the corresponding execution servers.
Illustratively, the method further comprises:
step S141, the execution server obtains the labeling result after the labeling process is performed on each thread pool. And step S142, the execution server summarizes and stores the tag result of each thread pool in the database.
Specifically, after marking processing is carried out on target user data, label results are uniformly obtained and are landed, namely, the label results are stored in a database so as to be inquired in real time according to the label results when the label data are used. The java thread pool and the multi-thread waiting technique can be used, if 10 threads are processing tags, but 50 tags are needed to be processed by one user, the main thread will wait until 50 tags are completely processed by the 10 thread pool, and then summarize the tag results, and then update the database table.
Illustratively, the method further comprises:
and step S151, the execution server marks the number of each thread pool to obtain the thread number corresponding to each thread pool. Step S152, the execution server associates the grouping label of each group of target user data with the thread number of the corresponding thread pool.
Specifically, when each thread pool processes a corresponding set of target user data, the target user data corresponding to the thread number is processed according to the thread number. For example, the thread pool processes the target user data with packet number 12_0, the thread number of the corresponding thread pool may be set to 12.
Example two
Referring to fig. 2, a program module diagram of a second embodiment of the tag processing system based on user data according to the present invention is shown. In the present embodiment, the tag processing system 20 based on user data may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to implement the present invention and implement the above-described tag processing method based on user data. The program modules referred to in the embodiments of the present invention refer to a series of computer program instruction segments capable of performing specific functions, and are more suitable than the program itself for describing the execution process of the tag processing system 20 in the storage medium based on the user data. The following description will specifically describe the functions of the program modules of the present embodiment:
the main control server 200 is configured to receive a trigger operation of a timing task, generate a processing instruction according to the trigger operation, and send the task processing instruction to an execution server corresponding to the timing task.
Specifically, when the data amount of the user data processed by the timed task is excessively large, the processing is performed by the distributed cluster. A corresponding timing task can be set in each execution server 202 of the distributed cluster, and the execution server 202 in each distributed cluster performs timing task triggering; or the main control server 200 manages the timing task of each execution server 202 in a unified manner, and the main control server 200 triggers the timing task of each execution server 202 in a separate or unified manner. After the timing task of each execution server 202 is triggered, the master control server 200 performs unified scheduling on each execution server 202 to execute the timing task. Wherein the timing task includes marking for each target user data. When the main control server 200 schedules the execution server 202, a task processing instruction is generated according to a timing task and is distributed to the execution server 202, so that the execution server 202 completes marking processing on target user data according to task processing execution. All the execution server 202 participate in data processing, so that the utilization rate of the execution server 202 is improved, the stand-alone processing time is reduced, and data processing abnormity caused by the problem of the execution server 202 is avoided.
Furthermore, the query tag can be frequently refreshed according to the requirement, and the trigger time of the timing task is set according to the query tag. For example, the requirement on the timeliness of a part of query labels is not high, and the query labels can be in a [ day ] level. For example, if there is overdue record in the last 12 months, whether there is a repayment day currently, whether the overdue days have passed the expiration date, etc., the timing task may be set to be timed to be processed once every morning; the timeliness of some labels is required to be high, and can be in a [ time ] level, for example, whether the current time is out of date, whether a client has a piece in transit, and the like, and the timing task can be set to be processed once every integral point.
The execution server 202 is configured to obtain a token number.
Specifically, after the execution server 202 in the distributed cluster performs timing task triggering, each execution server 202 may directly take the token number from the token stack queue in the redis database according to the task processing instruction.
The execution server 202 is further configured to obtain a query tag corresponding to the timing task.
Specifically, each execution server 202 will obtain the user data of the corresponding line number according to the token number, where the line number may be understood as the number of single batch processing, and the number of single batch processing obtained by each token is consistent, but the line numbers of the user data obtained by each token number are different, so that when the user data of each user is stored, the user data of each user is numbered correspondingly, so as to obtain the user data of the corresponding line number according to the token number. For example: if the token number is 1, user data with the number of 0-1000 is acquired; if the token number is 2, the user data with the 1000-2000 number is obtained. The target user data may be basic information for each user, such as name, identification number, tag calculation parameters required to query the tag, etc.
The execution server 202 is further configured to obtain at least one target user data of the line number corresponding to the token number from a preset database according to the obtained token number and the query tag.
Specifically, the query tag is a tag that needs to perform marking processing on target user data, for example: and if the time is expired, label calculation data related to the target user data is acquired according to the query label for calculation, and marking processing is performed on the target user data after calculation. The query tags are preset, and a corresponding query tag is set in each timing task.
The execution server 202 is further configured to group the at least one target user data according to a preset grouping policy to obtain multiple groups of target user data.
Specifically, the grouping policy may be set according to requirements, and this embodiment is exemplified by an identification number in the user data.
Illustratively, the execution server 202 is further configured to:
acquiring identity card number data in the at least one target user data; the execution server 202 calculates the month and the date in the identification number data to obtain a plurality of groups of target user data; the execution server 202 performs grouping marking on each group of target user data, and obtains a plurality of grouping labels corresponding to each group of target user data.
Specifically, the execution server 202 groups the target user data by month and date in the identity card number, and the grouping policy is: the grouping index is set to M _ N, where M is the month of the birthday and N is the number of days that the birthday is to be left over for 3. For example, the birthday of the user data is 20200817, the obtained grouping number is 08_2, and the user data with the consistent grouping number are grouped into one group for marking.
And creating thread pools with corresponding quantity according to the quantity of the target user data packets, and labeling a group of target user data associated with each thread pool through each thread pool.
Specifically, as the grouping policy mentioned above, since there are 12 in a month and the remainder of 3 is 0, 1, 2, the target user data is divided into 12 × 3 — 36 groups, and the execution server 202 creates 36 thread pools from the 36 groups, each thread pool processing a set of target user data associated therewith.
Illustratively, the execution server 202 is further configured to:
and storing each group of the target user data in a client temporary table.
Specifically, each target user data includes data associated with a query tag, and storing the target user data in a client temporary table facilitates data query.
Illustratively, the client temporary table is used for temporarily storing each group of target user data, including storing identification card information, name, address, etc. of the target user, and when the execution of the timed task is completed, the data in the client temporary table can be released. When the target user data is stored in the client temporary table C1, a column of fields [ executive server IP ] is added, the executive server record IP1 of the 1 token, and the executive server record IP2 of the 2 token. Each execution server only processes corresponding target user data according to the execution server IP, and when the application table, the borrow table and the client temporary table are associated, because the target user data processed by different execution servers 202 are different, the problem of database resource competition cannot exist, and the processing performance can be greatly improved.
And acquiring label calculation data corresponding to the query label from the preset database according to the client temporary table.
Specifically, the tag calculation data is data that needs to be calculated corresponding to the query tag when marking processing is performed on each target user data according to the query tag. And marking the target user data after correspondingly calculating the label calculation data. For example: and when the query tag is overdue, the required tag calculation data is the user information and the repayment data in the target user data, and the repayment data acquires specific tag calculation data, such as the loan amount detail of 1-5 months, from the database through the user borrowing table.
And creating a thread pool with a corresponding quantity according to the quantity of the target user data packets.
Specifically, the multithread processing can shorten the processing time for processing the user data corresponding to the whole target tag, the applicable scenes of the timed batch processing task are more, and the implementation sets the thread pools with the number corresponding to the number of the target user data packets so as to save the processing time.
And controlling each thread pool to calculate the label calculation data corresponding to a group of target user data associated with the thread pool according to the query label, and labeling each target user data according to the calculation result.
Specifically, the tag calculation data is calculated according to a calculation formula of whether the tag is overdue or not to obtain a result of whether the tag is overdue or not, and the expected result is marked on the corresponding target user data. The thread pool can set a plurality of threads to perform calculation and marking processing according to the size of a group of corresponding target user data, so that the effect of concurrent processing is achieved, and the processing time of the timing task is saved.
EXAMPLE III
Fig. 3 is a schematic diagram of a hardware architecture of a computer device according to a third embodiment of the present invention. In the present embodiment, the computer device 2 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a preset or stored instruction. The computer device 2 may be a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers), and the like. As shown in FIG. 3, the computer device 2 includes, but is not limited to, at least a memory 21, a processor 22, a network interface 23, and a tag processing system 20 based on user data, communicatively coupled to each other via a system bus. Wherein:
in this embodiment, the memory 21 includes at least one type of computer-readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 2, such as a hard disk or a memory of the computer device 2. In other embodiments, the memory 21 may also be an external storage device of the computer device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like provided on the computer device 2. Of course, the memory 21 may also comprise both internal and external memory units of the computer device 2. In this embodiment, the memory 21 is generally used for storing an operating system installed on the computer device 2 and various application software, such as the program code of the tag processing system 20 based on user data in the second embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The network interface 23 may comprise a wireless network interface or a wired network interface, and the network interface 23 is generally used for establishing communication connection between the server 2 and other electronic devices. For example, the network interface 23 is used to connect the server 2 to an external terminal via a network, establish a data transmission channel and a communication connection between the server 2 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like. It is noted that fig. 3 only shows the computer device 2 with components 20-23, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
In this embodiment, the tag processing system 20 based on user data stored in the memory 21 can be further divided into one or more program modules, and the one or more program modules are stored in the memory 21 and executed by one or more processors (in this embodiment, the processor 22) to complete the present invention.
For example, fig. 2 is a schematic diagram of program modules of a second embodiment of implementing the user data based tag processing system 20, in which the user data based tag processing system 20 may be divided into the main control server 200 and the execution server 202. The program modules referred to herein are a series of computer program instruction segments that can perform specific functions, and are more suitable than programs for describing the execution of the user data based tag processing system 20 in the computer device 2. The specific functions of the program modules 200 and 202 have been described in detail in the second embodiment, and are not described herein again.
Example four
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment is used in a computer program, and when executed by a processor, implements the tag processing method based on user data of embodiment one.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A label processing method based on user data is applied to a distributed cluster, the distributed cluster comprises a main control server and a plurality of execution servers, the main control server is used for scheduling the execution servers, and the label processing method is characterized by comprising the following steps:
the master control server receives trigger operation of a timing task, generates a task processing instruction according to the trigger operation scheduling, and sends the task processing instruction to an execution server corresponding to the timing task;
the execution server side obtains a token number according to the task processing instruction;
the execution server side acquires a query tag corresponding to the timing task according to the task processing instruction;
the execution server side obtains at least one target user data of the line number corresponding to the token number from a preset database according to the obtained token number and the query label;
the execution server side groups the at least one target user data according to a preset grouping strategy to obtain a plurality of groups of target user data;
the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets, and performs labeling processing on a group of target user data associated with the thread pools through each thread pool.
2. The method of claim 1, wherein the method further comprises:
the master control server side obtains the number of users to be processed and the number of single batch processing in the timing task;
the master control server side generates a token stack queue according to the number of the users to be processed and the number of single batch processing;
and the master control server stores the token stack queue in a redis database, the token stack queue stores a plurality of token numbers in sequence, and each token number corresponds to user data of single batch processing number.
3. The method of claim 2, wherein the method further comprises:
the master control server side obtains trigger operations of a plurality of execution server sides for starting the timing task;
and the master control server acquires the token number from the redis database according to the received trigger operation and distributes the acquired token number to the corresponding execution server.
4. The method of claim 1, wherein the method further comprises:
the execution server side obtains a label result after each thread pool is subjected to label printing;
and the execution server side collects the label result of each thread pool and stores the label result in the database.
5. The tag processing method based on user data according to claim 1, wherein the step of grouping the at least one target user data by the execution server according to a preset grouping policy to obtain multiple groups of target user data comprises:
the execution server side obtains the identity card number data in the at least one target user data;
the execution server calculates the month and the date in the identity card number data to obtain a plurality of groups of target user data;
and the execution server carries out grouping marking on each group of target user data to obtain a plurality of grouping labels corresponding to each group of target user data.
6. The method of claim 5, wherein the method further comprises:
the execution server side marks each thread pool to obtain a thread number corresponding to each thread pool;
and the execution server associates the grouping labels of each group of target user data with the thread numbers of the corresponding thread pools.
7. The tag processing method based on user data according to claim 1, wherein the step of creating a corresponding number of thread pools by the execution server according to the number of target user data packets, and performing tagging processing on a set of target user data associated with the thread pools through each thread pool comprises:
the execution server stores each group of target user data in a client temporary table;
the execution server side acquires label calculation data corresponding to the query label from the preset database according to the client temporary table;
the execution server creates thread pools with corresponding quantity according to the quantity of the target user data packets;
and the execution server controls each thread pool to calculate the label calculation data corresponding to a group of target user data associated with the thread pool according to the query label, and performs label printing processing on each target user data according to the calculation result.
8. A tag processing system based on user data, comprising:
the main control server is used for receiving triggering operation of a timing task, generating a processing instruction according to the triggering operation, and sending the task processing instruction to an execution server corresponding to the timing task;
the execution server is used for acquiring a token number;
the execution server is further configured to obtain a query tag corresponding to the timing task;
the execution server is further used for acquiring at least one target user data of the line number corresponding to the token number from a preset database according to the acquired token number and the query tag;
the execution server is further configured to group the at least one target user data according to a preset grouping policy to obtain multiple groups of target user data;
the execution server is further configured to create thread pools of corresponding quantities according to the quantity of the target user data packets, and perform labeling processing on a set of target user data associated with the thread pools through each thread pool.
9. A computer arrangement, characterized in that the computer arrangement comprises a memory, a processor, the memory having stored thereon a computer program being executable on the processor, the computer program, when being executed by the processor, realizing the steps of the user data based tag processing method according to any of the claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which is executable by at least one processor to cause the at least one processor to perform the steps of the method for tag processing based on user data according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110213714.0A CN112835945A (en) | 2021-02-25 | 2021-02-25 | User data-based label processing method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110213714.0A CN112835945A (en) | 2021-02-25 | 2021-02-25 | User data-based label processing method, system, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112835945A true CN112835945A (en) | 2021-05-25 |
Family
ID=75933476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110213714.0A Pending CN112835945A (en) | 2021-02-25 | 2021-02-25 | User data-based label processing method, system, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112835945A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008257A (en) * | 2019-04-10 | 2019-07-12 | 深圳市腾讯计算机系统有限公司 | Data processing method, device, system, computer equipment and storage medium |
| CN110765179A (en) * | 2019-10-18 | 2020-02-07 | 京东数字科技控股有限公司 | Distributed account checking processing method, device, equipment and storage medium |
| CN111061762A (en) * | 2019-11-08 | 2020-04-24 | 京东数字科技控股有限公司 | Distributed task processing method, related device, system and storage medium |
| CN112099958A (en) * | 2020-11-17 | 2020-12-18 | 深圳壹账通智能科技有限公司 | Distributed multi-task management method and device, computer equipment and storage medium |
-
2021
- 2021-02-25 CN CN202110213714.0A patent/CN112835945A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008257A (en) * | 2019-04-10 | 2019-07-12 | 深圳市腾讯计算机系统有限公司 | Data processing method, device, system, computer equipment and storage medium |
| CN110765179A (en) * | 2019-10-18 | 2020-02-07 | 京东数字科技控股有限公司 | Distributed account checking processing method, device, equipment and storage medium |
| CN111061762A (en) * | 2019-11-08 | 2020-04-24 | 京东数字科技控股有限公司 | Distributed task processing method, related device, system and storage medium |
| CN112099958A (en) * | 2020-11-17 | 2020-12-18 | 深圳壹账通智能科技有限公司 | Distributed multi-task management method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110611707B (en) | Task scheduling method and device | |
| CN111400011B (en) | Real-time task scheduling method, system, equipment and readable storage medium | |
| WO2020082611A1 (en) | Method for carrying out deep learning on basis of blockchain platform and electronic device | |
| CN111258774A (en) | Flow processing method and device, computer equipment and storage medium | |
| CN109426550A (en) | The dispatching method and equipment of resource | |
| CN108564345B (en) | Insurance platform data processing method, electronic device and computer readable storage medium | |
| CN111208992A (en) | System scheduling workflow generation method and system | |
| CN112817710A (en) | Timed task processing method, timed task processing system, computer equipment and computer storage medium | |
| CN116541134A (en) | Method and device for deploying containers in multi-architecture cluster | |
| CN113742109A (en) | Third-party service docking method, device, equipment and storage medium | |
| CN110471780B (en) | Distributed event processing apparatus, terminal, and computer storage medium | |
| CN108255735A (en) | Associated environment test method, electronic device and computer readable storage medium | |
| CN113179304A (en) | Message issuing method, system, device and storage medium | |
| CN112835945A (en) | User data-based label processing method, system, device and storage medium | |
| CN113220434B (en) | Task scheduling method based on dependency model and dependency model configuration method | |
| CN114531374A (en) | Network monitoring method, device, equipment and storage medium | |
| CN111338775B (en) | Method and equipment for executing timing task | |
| CN114169306A (en) | Method, device and equipment for generating electronic receipt and readable storage medium | |
| CN110674214B (en) | Big data synchronization method, device, computer equipment and storage medium | |
| CN112559155A (en) | Batch job processing method, system, computer device and readable storage medium | |
| CN111400058A (en) | Method and device for calling message, computer equipment and storage medium | |
| CN112084014A (en) | Data processing method, device, equipment and medium | |
| WO2019169696A1 (en) | Platform client data backflow method, electronic apparatus, device, and storage medium | |
| CN110673931A (en) | Distributed calculation method for document synthesis, document synthesis system and control device thereof | |
| CN111258728A (en) | Task execution method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210525 |