CN110069481B - Unified cleaning method and related equipment for production library data based on big data - Google Patents
Unified cleaning method and related equipment for production library data based on big data Download PDFInfo
- Publication number
- CN110069481B CN110069481B CN201910182538.1A CN201910182538A CN110069481B CN 110069481 B CN110069481 B CN 110069481B CN 201910182538 A CN201910182538 A CN 201910182538A CN 110069481 B CN110069481 B CN 110069481B
- Authority
- CN
- China
- Prior art keywords
- cleaning
- service system
- configuration
- information
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data, in particular to a unified cleaning method for production library data based on big data and related equipment. The method comprises the following steps: receiving configuration information sent by a user, and recording the configuration information into a configuration data table; generating a plurality of cleaning tasks in batches according to the configuration information; and starting a plurality of sub-threads, and executing the data cleaning script through the sub-threads. The invention uniformly manages all data cleaning tasks, completes the data cleaning work of each business system, has relatively independent data cleaning work, and has simpler and more convenient maintenance of the data cleaning script.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a unified cleaning method for production library data based on big data and related equipment.
Background
In the existing projects such as a telephone sales system, due to the fact that historical demand designs are numerous, the quantity of project production data is increased increasingly, so that when table data is queried, the query speed is gradually slow, and therefore the historical data is cleaned. The existing service system is provided with a plurality of own data cleaning programs in the system, and when the data cleaning programs need to be changed, version updating can only be completed by self, so that the purpose of instant use cannot be achieved.
Disclosure of Invention
In view of the above, it is necessary to provide a unified cleaning method and related equipment for database data based on big data, which aims at the problems of big data volume, slow data query speed and inconvenient cleaning of some systems.
A unified cleaning method for production library data based on big data comprises the following steps:
receiving configuration information sent by a user, and recording the configuration information into a configuration data table, wherein the configuration information comprises service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script;
reading the configuration data table, and generating a plurality of cleaning tasks in batches according to configuration information in the configuration data table, wherein one cleaning task corresponds to one item of configuration information;
and starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the business system corresponding to the business system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the business system database connection pool.
In one possible design, the receiving the configuration information sent by the user records the configuration information into a configuration data table, where the configuration information includes a service system code, a service system database connection pool, a cleaning task timing time, and a data cleaning script, and the method includes:
receiving a configuration request sent by a user, displaying a login page, and acquiring account information input by the user through the login page;
verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a configuration page if the account information has the configuration authority, and otherwise returning the error prompt;
and acquiring configuration information input by a user through the configuration page, and recording the configuration information into the configuration data table.
In one possible design, the method further comprises:
receiving a change request sent by a user, displaying a login page, and acquiring account information input by the user through the login page;
verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a change page if the account information has the change authority, and otherwise returning the error prompt;
acquiring modification information input by a user through the modification page, wherein the modification information comprises at least one of execution update time, service system codes and service system database connection pools corresponding to the service system codes, cleaning task timing time or data cleaning scripts, judging whether the account information corresponds to the permission of modifying the service system corresponding to the service system codes, and returning an error prompt if the account information does not have the modification permission;
and if the service system code has the modification authority, adding the modification information into the configuration information corresponding to the service system code in the configuration data table.
In one possible design, if the service system code has the modification authority, the adding the modification information to the configuration information corresponding to the service system code in the configuration data table includes:
reading the cleaning task timing time in the configuration information corresponding to the service system code in the configuration data table, judging whether the time difference between the execution updating time and the cleaning task timing time exceeds a preset time threshold, and if not, returning an error prompt;
if yes, adding the change information into the configuration information corresponding to the service system codes in the configuration data table.
In one possible design, the reading the configuration data table generates a plurality of cleaning tasks in batches according to configuration information in the configuration data table, where one cleaning task corresponds to one item of configuration information, and the method includes:
reading configuration information in the configuration data table one by one, and acquiring the service system codes, the service system database connection pool, the cleaning task timing time and the data cleaning script in the configuration information;
calling a preset task template, respectively assigning the service system codes, the service system database connection pool, the cleaning task timing time and the data cleaning script in the configuration information to variables in the task template, generating a cleaning task, and placing the cleaning task in a task pool;
traversing the configuration data table, generating a corresponding cleaning task by each piece of configuration information, and placing the generated cleaning task into a task pool.
In one possible design, the method further comprises:
searching the configuration data table at regular time, sequentially acquiring each piece of change information, and judging whether the current time reaches the execution update time in the change information;
if the configuration information is reached or exceeded, reading a cleaning task corresponding to the configuration information, changing at least one of a service system database connection pool, the cleaning task timing time or the data cleaning script in the cleaning task according to changing information, generating a new cleaning task, and placing the cleaning task in a task pool;
and deleting the change information after covering the corresponding configuration information with the change information in the configuration data table.
In one possible design, the starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the service system corresponding to the service system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the service system database connection pool, including:
starting a plurality of sub-threads, and putting the plurality of sub-threads into a working thread pool;
reading a sub-thread in the working thread pool, reading a cleaning task in the task pool, and distributing the cleaning task to the sub-thread for execution;
the sub-thread is connected with the service system corresponding to the service system code at regular time according to the cleaning task timing time in the cleaning task, and the data cleaning script is executed on the data in the service system database connection pool;
and after the sub-thread finishes the cleaning task, putting the sub-thread into a working thread pool again.
Production library data unified cleaning device based on big data includes:
the information acquisition module is used for receiving configuration information sent by a user and recording the configuration information into a configuration data table, wherein the configuration information comprises a service system code, a service system database connection pool, a cleaning task timing time and a data cleaning script;
the cleaning task generating module is used for reading the configuration data table, generating a plurality of cleaning tasks in batches according to the configuration information in the configuration data table, wherein one cleaning task corresponds to one piece of configuration information;
and the cleaning data module is used for starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the service system corresponding to the service system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the service system database connection pool.
A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the big data based production library data unified cleaning method described above.
A storage medium storing computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the big data based production library data unified cleaning method described above.
The method for uniformly cleaning the production library data based on big data and related equipment comprise the steps of receiving configuration information sent by a user, and recording the configuration information into a configuration data table, wherein the configuration information comprises service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script; reading the configuration data table, and generating a plurality of cleaning tasks in batches according to configuration information in the configuration data table, wherein one cleaning task corresponds to one item of configuration information; and starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the business system corresponding to the business system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the business system database connection pool. The invention manages all data cleaning tasks in a unified way to finish the data cleaning work of each service system, the service system does not carry out the data cleaning task, if the data cleaning script is changed, version update of the service system is not needed, only the data cleaning script corresponding to the cleaning task is updated, the data cleaning work of the invention is relatively independent, and the maintenance of the data cleaning script is simpler and more convenient.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
FIG. 1 is a flow chart of a unified cleaning method for big data based production library data in one embodiment of the invention;
fig. 2 is a block diagram of a unified cleaning device for database data based on big data in an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Fig. 1 is a flowchart of a method for cleaning up unified database data of a production library based on big data according to an embodiment of the present invention, as shown in fig. 1, and the method for cleaning up unified database data of a production library based on big data includes the following steps:
step S1, information is acquired: and receiving configuration information sent by a user, and recording the configuration information into a configuration data table, wherein the configuration information comprises service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script.
The main body of the step can be a page management project data platform, and the platform is pre-provided with a configuration data table for storing various configuration information. The platform receives configuration information sent by a user through a web front-end webpage, adds the configuration information into a preset configuration data table, and adopts the following modes when the platform acquires the configuration information:
in one embodiment, step S1 includes:
step S101, receiving a configuration request: and receiving a configuration request sent by a user, displaying a login page, and acquiring account information input by the user through the login page.
A user of the external service system can access the platform through the web front-end page, and the platform can determine whether the user has configuration permission by displaying the login page to acquire account information of the user because the configuration of the cleaning task is not all the users can operate.
Step S102, verifying account information: and verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a configuration page if the account information has the configuration authority, and otherwise returning the error prompt.
Before this step is performed, the user of the service system may register the operation roles on the platform, each operation role may bind a plurality of accounts, and the platform gives the operation roles and the plurality of accounts the authority of this service system configuration or modification.
The configuration page in this step may include information such as a service system code field, a service system database connection pool field, a cleaning task timing time field, and a data cleaning script upload entry. The user of the service system inputs configuration information through the configuration page, wherein the service system code is a specific code of the service system, is a code preset by the service system and a platform, such as a service system code TM001 of a telemarketing system. The service system database connection pool is a database path which needs to be cleaned by the service system. The cleaning task timing time is the time that the business system needs to clean, for example, cleaning is started in the early morning 1 day, or cleaning is started in the evening 11 days of the week. The data cleaning script is a specific cleaning program which needs to be cleaned by the service system, and can be realized by adopting an SQL script.
Step S103, recording configuration information: and acquiring configuration information input by a user through a configuration page, and recording the configuration information into a configuration data table.
After the configuration information input by the user is obtained, the step can also judge whether the account information corresponds to the service system or not has the authority for configuring the service system corresponding to the service system code in the configuration information. If the configuration permission is not available, an error prompt is returned, and if the configuration permission is available, the step is continued. Thus, the user corresponding to the service system can be limited to only configure configuration information of the system.
According to the embodiment, the configuration information of the user request corresponding to the external service system is obtained through the front-end page, the configuration information can be remotely and flexibly recorded, any user corresponding to the service system registered on the platform can access the platform through the webpage, and the purpose of remote cleaning and configuration of the service system is achieved.
In one embodiment, further comprising:
step S111, receiving a change request: and receiving a change request sent by a user, displaying a login page, and acquiring account information input by the user through the login page.
The step not only obtains the configuration request sent by the user and records the configuration information, but also has the function of changing the configuration so as to facilitate the data cleaning work of the independent updating service system.
Step S122, verifying account information: and verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a change page if the account information has the change authority, and otherwise returning the error prompt.
The verification method in this step is similar to step S102, and it is determined whether the user has the modification right by verifying the account information. The change page in the step is similar to the configuration page, and comprises a business system coding field of a necessary filling field, an execution update time field, a selected filling business system database connection pool field, a cleaning task timing time field or a data cleaning script uploading inlet and other information. The update time is the operation time for updating the change information, for example, the data cleaning work corresponding to the change information is executed in the early morning 1 of the 2 nd month 1 of 2019, and the data cleaning work corresponding to the original configuration information is still executed before the early morning 1 of the 1 nd month 2 of 2019.
Step S123, judging the modification right: and acquiring modification information input by a user through a modification page, wherein the modification information comprises at least one of execution update time, service system codes, a service system database connection pool corresponding to the service system codes, cleaning task timing time or data cleaning script, judging whether account information corresponds to the authority of the service system corresponding to the modification service system codes, and returning an error prompt if the account information does not have the modification authority.
The judging manner of this step is similar to step S103, and only the authority of the service system corresponding to the code of the service system is changed, the service system can be changed.
Step S124, adding modification information: if the service system has the modification authority, the modification information is added into the configuration information corresponding to the service system code in the configuration data table.
In this step, since the update time is different when the change information is stored, the change information is added to the configuration data table, not in a manner of overwriting the configuration information. Before adding, this step also makes a judgment on the execution update time:
reading the timing time of the cleaning task in the configuration information corresponding to the service system code in the configuration data table, judging whether the time difference between the execution updating time and the timing time of the cleaning task exceeds a preset time threshold, and if not, returning an error prompt; if yes, adding the change information into the configuration information corresponding to the service system codes in the configuration data table.
In order to avoid conflict between the execution update time and the configured cleaning task timing time, if the execution update time is the same as the cleaning task timing time or the difference between the execution update time and the cleaning task timing time is within a preset time threshold, a user is prompted to change the execution update time through a change page so as to update configuration information stably and reliably.
According to the embodiment, the front-end page is used for acquiring the change information of the user request corresponding to the external service system, so that the change information can be added remotely and flexibly, any user corresponding to the service system registered in the platform can access the platform through the webpage, the purpose of remotely changing the configuration information of the service system is achieved, the change information is independent of the running program of the service system, and the service system can realize the rapid and stable update of the cleaning data script without updating a new version.
Step S2, generating a cleaning task: and reading the configuration data table, and generating a plurality of cleaning tasks in batches according to the configuration information in the configuration data table, wherein one cleaning task corresponds to one item of configuration information.
The step can be to package corresponding cleaning tasks according to configuration information by starting preset sub-thread management software, such as a spring technical framework.
In one embodiment, step S2 includes:
step S201, reading configuration information: and reading configuration information in the configuration data table one by one, and acquiring service system codes, service system database connection pools, cleaning task timing time and data cleaning scripts in the configuration information.
Before generating a cleaning task, firstly, reading configuration information from a configuration data table one by one, and correspondingly generating a cleaning task by one configuration information.
Step S202, generating a task: calling a preset task template, respectively assigning the service system codes, the service system database connection pool, the cleaning task timing time and the data cleaning script in the configuration information to variables in the task template, generating a cleaning task, and placing the cleaning task in the task pool.
Because the main differences among the cleaning tasks are four parameters, namely service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script, the step is to configure different cleaning tasks according to different configuration information through a task template, and the purpose that one configuration information corresponds to the generation of one cleaning task is obtained.
In order to avoid the simultaneous execution of the same cleaning tasks at the same moment, when the cleaning tasks are generated, task types can be designated through the current attribute of the spring technical framework, and the cleaning tasks are stateless in default.
Step S203, generating a task: traversing the configuration data table, generating a corresponding cleaning task by each configuration information, and placing the generated cleaning task into a task pool.
And placing the cleaning tasks into the task pool, and sequentially reading the cleaning task execution cleaning data script according to the first-in first-out characteristics of the task pool.
According to the embodiment, the configuration data table is traversed, the spring technology framework is utilized to generate the stateful cleaning tasks in batches, the cleaning tasks are uniformly managed by the independent platform, and the cleaning tasks are independent of the service system, so that the data cleaning script can be better maintained and updated.
In one embodiment, step S2 further comprises:
step S211, timing retrieval: and searching the configuration data table at regular time, sequentially acquiring each piece of change information, and judging whether the current time reaches the execution update time in the change information.
If the user changes the configuration information, the configuration data table contains corresponding change information. Table 1 below is a configuration data table that can be written according to a row header or a column header when writing configuration information and modification information into the configuration data table. When the configuration data table is retrieved, whether the corresponding cell contains modification information can be searched according to the title.
TABLE 1
Step S212, update the cleaning task: if the cleaning task reaches or exceeds the cleaning task, reading the cleaning task corresponding to the configuration information, changing at least one of a service system database connection pool, cleaning task timing time or data cleaning script in the cleaning task according to the changing information, generating a new cleaning task, and placing the cleaning task in the task pool.
When the update execution time arrives, it is considered that the configuration information can be updated, and all information in the change information is covered on the corresponding configuration information. For example, the change information only changes the data cleaning script, and then the data cleaning script in the configuration information is replaced. And (3) re-calling the preset task template with the replaced configuration information to generate a new cleaning task and putting the new cleaning task into a task pool.
If the cleaning task is set to be a cleaning task with a status in step S202, the current status of the cleaning task is further determined before updating the configuration information, and if the status is executing, the cleaning task is updated after waiting for the sub-thread to complete executing the task. When the update configuration information is processed, the current state of the cleaning task is set to the state in the update. If a new cleaning task is generated, the current state of the cleaning task is set to be an idle state, and the current state is put into a task pool.
Step S213, updating the configuration data table: and deleting the change information after covering the corresponding configuration information with the change information in the configuration data table.
The step not only needs to update the cleaning task, but also correspondingly updates the configuration data table, so that the follow-up continuous configuration information changing can be ensured.
The embodiment provides a changing mode of the configuration information, only the changing information reaching the execution updating time can be updated, and only the independent cleaning task is updated when the updating is performed, so that the cleaning data process of other cleaning tasks is not influenced.
Step S3, executing a cleaning task: starting a plurality of sub-threads, reading cleaning tasks through the sub-threads, connecting service systems corresponding to service system codes according to timing time of the cleaning tasks in the cleaning tasks, and executing data cleaning scripts on data in a service system database connection pool.
In one embodiment, step S3 includes:
step S301, starting a sub-thread: and starting a plurality of sub-threads, and putting the plurality of sub-threads into a working thread pool.
The number of the sub-threads opened in the step can be the same as or less than the number of the cleaning tasks. When the platform is started, a plurality of sub-threads are automatically started, and the plurality of sub-threads are automatically put into a working thread pool.
Step S302, task allocation: and reading one sub-thread in the working thread pool, reading one cleaning task in the task pool, and distributing the cleaning task to the sub-thread for execution.
All idle sub-threads are placed in a working thread pool in the step, whether the idle sub-threads are contained or not is firstly determined from the working thread pool, and a cleaning task is taken out from a task pool and is distributed to the idle sub-threads only when the idle sub-threads are contained.
Step S303, cleaning up data: and executing a data cleaning script on the data in the service system database connection pool by the sub-thread according to the timing time of the cleaning task in the cleaning task and the timing connection service system code corresponding to the service system.
According to the method, the sub-thread waits for triggering of the cleaning task timing time, the service system corresponding to the service system code is connected to the cleaning task timing time through a preset API interface, a service system database connection pool is found, a data cleaning script is executed, and data cleaning work of external service data is achieved.
Step S304, replay sub-threads: and after the sub-thread finishes the cleaning task, putting the sub-thread into the working thread pool again.
After the sub-thread completes the cleaning task, in order to execute other cleaning tasks, the sub-thread which has completed the cleaning task is put back into the working thread pool for being distributed to other cleaning tasks.
According to the method, the data cleaning work of each service system is completed in a mode that the plurality of sub-threads respectively execute cleaning tasks at regular time, the service systems are mutually independent when cleaning data, and the cleaning of the data is simpler and faster.
According to the unified cleaning method for the production library data based on the big data, the cleaning task is generated according to the configuration information of different service systems, independent sub-threads are allocated to the cleaning task to execute data cleaning work, and version update of the service systems is not needed when the cleaning task is updated, so that the cleaning task is updated conveniently and rapidly. And each service system independently works when cleaning data, the cleaning work is free from any blockage, and the data cleaning efficiency is high.
In one embodiment, a unified cleaning device for production library data based on big data is provided, as shown in fig. 2, including the following modules:
the information acquisition module is used for receiving configuration information sent by a user, recording the configuration information into a configuration data table, wherein the configuration information comprises service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script;
the cleaning task generating module is used for reading the configuration data table, generating a plurality of cleaning tasks in batches according to the configuration information in the configuration data table, wherein one cleaning task corresponds to one piece of configuration information;
the cleaning data module is used for starting a plurality of sub-threads, reading cleaning tasks through the sub-threads, connecting service systems corresponding to service system codes according to timing time of the cleaning tasks in the cleaning tasks, and executing data cleaning scripts on data in a service system database connection pool.
In one embodiment, a computer device is provided, including a memory and a processor, where the memory stores computer readable instructions, where the computer readable instructions, when executed by the processor, cause the processor to implement the steps in the big data based production library data unified cleaning method of the above embodiments when executing the computer readable instructions.
In one embodiment, a storage medium storing computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps in the big data based production library data unified cleaning method of the above embodiments is presented. Wherein the storage medium may be a non-volatile storage medium.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above-described embodiments represent only some exemplary embodiments of the invention, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (8)
1. A unified cleaning method for production library data based on big data is characterized by comprising the following steps:
receiving configuration information sent by a user, and recording the configuration information into a configuration data table, wherein the configuration information comprises service system codes, a service system database connection pool, cleaning task timing time and a data cleaning script;
reading the configuration data table, and generating a plurality of cleaning tasks in batches according to configuration information in the configuration data table, wherein one cleaning task corresponds to one item of configuration information;
starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the service system corresponding to the service system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the service system database connection pool;
the method further comprises the steps of:
receiving a change request sent by a user, displaying a login page, and acquiring account information input by the user through the login page;
verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a change page if the account information has the change authority, and otherwise returning the error prompt;
acquiring modification information input by a user through the modification page, wherein the modification information comprises at least one of execution update time, service system codes and service system database connection pools corresponding to the service system codes, cleaning task timing time or data cleaning scripts, judging whether the account information corresponds to the permission of modifying the service system corresponding to the service system codes, and returning an error prompt if the account information does not have the modification permission; the execution update time is the operation time for updating the change information;
if the service system code has the modification authority, adding the modification information into configuration information corresponding to the service system code in the configuration data table;
if the service system code has the modification authority, adding the modification information into the configuration information corresponding to the service system code in the configuration data table, including:
reading the cleaning task timing time in the configuration information corresponding to the service system code in the configuration data table, judging whether the time difference between the execution updating time and the cleaning task timing time exceeds a preset time threshold, and if not, returning an error prompt;
if yes, adding the change information into the configuration information corresponding to the service system codes in the configuration data table.
2. The unified cleaning method for big data based production library data according to claim 1, wherein the receiving the configuration information sent by the user records the configuration information into a configuration data table, the configuration information includes service system codes, service system database connection pools, cleaning task timing time, and data cleaning scripts, and the method comprises the following steps:
receiving a configuration request sent by a user, displaying a login page, and acquiring account information input by the user through the login page;
verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a configuration page if the account information has the configuration authority, and otherwise returning the error prompt;
and acquiring configuration information input by a user through the configuration page, and recording the configuration information into the configuration data table.
3. The unified cleaning method for big data based production library data according to claim 1, wherein the reading the configuration data table generates a plurality of cleaning tasks in batch according to configuration information in the configuration data table, one cleaning task corresponding to one item of configuration information, and the method comprises the steps of:
reading configuration information in the configuration data table one by one, and acquiring the service system codes, the service system database connection pool, the cleaning task timing time and the data cleaning script in the configuration information;
calling a preset task template, respectively assigning the service system codes, the service system database connection pool, the cleaning task timing time and the data cleaning script in the configuration information to variables in the task template, generating a cleaning task, and placing the cleaning task in a task pool;
traversing the configuration data table, generating a corresponding cleaning task by each piece of configuration information, and placing the generated cleaning task into a task pool.
4. The unified cleaning method for big data based production library data according to claim 1, further comprising:
searching the configuration data table at regular time, sequentially acquiring each piece of change information, and judging whether the current time reaches the execution update time in the change information;
if the configuration information is reached or exceeded, reading a cleaning task corresponding to the configuration information, changing at least one of a service system database connection pool, the cleaning task timing time or the data cleaning script in the cleaning task according to changing information, generating a new cleaning task, and placing the cleaning task in a task pool;
and deleting the change information after covering the corresponding configuration information with the change information in the configuration data table.
5. The unified cleaning method for big data based production library data according to claim 3 or 4, wherein the starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the service system corresponding to the service system code according to the cleaning task timing time in the cleaning task, and executing the data cleaning script on the data in the service system database connection pool comprises:
starting a plurality of sub-threads, and putting the plurality of sub-threads into a working thread pool;
reading a sub-thread in the working thread pool, reading a cleaning task in the task pool, and distributing the cleaning task to the sub-thread for execution;
the sub-thread is connected with the service system corresponding to the service system code at regular time according to the cleaning task timing time in the cleaning task, and the data cleaning script is executed on the data in the service system database connection pool;
and after the sub-thread finishes the cleaning task, putting the sub-thread into a working thread pool again.
6. Production library data unified cleaning device based on big data, characterized by comprising:
the information acquisition module is used for receiving configuration information sent by a user and recording the configuration information into a configuration data table, wherein the configuration information comprises a service system code, a service system database connection pool, a cleaning task timing time and a data cleaning script;
the cleaning task generating module is used for reading the configuration data table, generating a plurality of cleaning tasks in batches according to the configuration information in the configuration data table, wherein one cleaning task corresponds to one piece of configuration information;
the cleaning data module is used for starting a plurality of sub-threads, reading the cleaning task through the sub-threads, connecting the service system corresponding to the service system code according to the timing time of the cleaning task in the cleaning task, and executing the data cleaning script on the data in the service system database connection pool;
the data adding module is used for receiving a change request sent by a user, displaying a login page and acquiring account information input by the user through the login page; verifying the account information, returning an error prompt if the account information fails to pass the verification, judging the authority corresponding to the account information if the account information passes the verification, displaying a change page if the account information has the change authority, and otherwise returning the error prompt; acquiring modification information input by a user through the modification page, wherein the modification information comprises at least one of execution update time, service system codes and service system database connection pools corresponding to the service system codes, cleaning task timing time or data cleaning scripts, judging whether the account information corresponds to the permission of modifying the service system corresponding to the service system codes, and returning an error prompt if the account information does not have the modification permission; the execution update time is the operation time for updating the change information; if the service system code has the modification authority, adding the modification information into configuration information corresponding to the service system code in the configuration data table;
the data adding module is further used for reading the cleaning task timing time in the configuration information corresponding to the service system code in the configuration data table, judging whether the time difference between the execution updating time and the cleaning task timing time exceeds a preset time threshold, and if not, returning an error prompt; if yes, adding the change information into the configuration information corresponding to the service system codes in the configuration data table.
7. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the big data based production library data unified cleaning method of any of claims 1 to 5.
8. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the big data based production library data unified cleaning method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910182538.1A CN110069481B (en) | 2019-03-12 | 2019-03-12 | Unified cleaning method and related equipment for production library data based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910182538.1A CN110069481B (en) | 2019-03-12 | 2019-03-12 | Unified cleaning method and related equipment for production library data based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110069481A CN110069481A (en) | 2019-07-30 |
| CN110069481B true CN110069481B (en) | 2023-07-28 |
Family
ID=67366162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910182538.1A Active CN110069481B (en) | 2019-03-12 | 2019-03-12 | Unified cleaning method and related equipment for production library data based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110069481B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140380105A1 (en) * | 2013-06-21 | 2014-12-25 | Dell Products, Lp | Integration Process Management Console With Error Resolution Interface |
| CN107360251A (en) * | 2017-08-16 | 2017-11-17 | 中国工商银行股份有限公司 | Method, system and the load balancing apparatus that session is kept |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731859B (en) * | 2015-02-02 | 2018-10-12 | 厦门市美亚柏科信息股份有限公司 | Data processing method and device |
| CN105760487B (en) * | 2016-02-17 | 2019-11-08 | 中国工商银行股份有限公司 | Data purge method and device |
| CN106528840A (en) * | 2016-11-11 | 2017-03-22 | 中国银行股份有限公司 | Service data clearing method and system based on banking system |
| CN109359157A (en) * | 2018-08-21 | 2019-02-19 | 中国平安人寿保险股份有限公司 | Data synchronize generation method, device, computer equipment and storage medium |
| CN109359107A (en) * | 2018-11-05 | 2019-02-19 | 中国建设银行股份有限公司 | Database method for cleaning, system, device and storage medium |
-
2019
- 2019-03-12 CN CN201910182538.1A patent/CN110069481B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140380105A1 (en) * | 2013-06-21 | 2014-12-25 | Dell Products, Lp | Integration Process Management Console With Error Resolution Interface |
| CN107360251A (en) * | 2017-08-16 | 2017-11-17 | 中国工商银行股份有限公司 | Method, system and the load balancing apparatus that session is kept |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110069481A (en) | 2019-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4611830B2 (en) | Database management system and method | |
| EP3428811A1 (en) | Database interface agent for a tenant-based upgrade system | |
| CN104781812A (en) | Policy driven data placement and information lifecycle management | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN104769583A (en) | Updating statistics in distributed databases | |
| CN104423982B (en) | The processing method and processing equipment of request | |
| CN105868216B (en) | A kind of method, apparatus and equipment for realizing the expired operation of object | |
| CN113656157A (en) | Distributed task scheduling method and device, storage medium and electronic equipment | |
| CN112685091A (en) | Service request processing method, device, equipment and medium based on big data | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN110046100B (en) | Packet testing method, electronic device and medium | |
| CN114416324A (en) | Task triggering method and device, computer equipment and storage medium | |
| US9747328B2 (en) | Method and apparatus for modifying a row in a database table to include meta-data | |
| CN107885543B (en) | Function management method and device of application program and intelligent terminal | |
| CN110069481B (en) | Unified cleaning method and related equipment for production library data based on big data | |
| CN113190576A (en) | Data processing method and device, computer equipment and readable storage medium | |
| CN104123303A (en) | Method and device for providing data | |
| KR20120082176A (en) | Data processing method of database management system and system thereof | |
| CN104133678A (en) | Method and device for building and operating SIDIS | |
| CN109660587B (en) | Data pushing method and device based on random number, storage medium and server | |
| CN115002187B (en) | Binding relation processing method and related equipment | |
| WO2019179252A1 (en) | Sample playback data access method and device | |
| CN106648550B (en) | Method and device for concurrently executing tasks | |
| US20190087238A1 (en) | Incremental workflow execution | |
| US11580128B2 (en) | Preventing DBMS deadlock by eliminating shared locking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |