CN113064717A - Method and device for uniformly managing data export function of multiple subsystems in system - Google Patents
Method and device for uniformly managing data export function of multiple subsystems in system Download PDFInfo
- Publication number
- CN113064717A CN113064717A CN202110235415.7A CN202110235415A CN113064717A CN 113064717 A CN113064717 A CN 113064717A CN 202110235415 A CN202110235415 A CN 202110235415A CN 113064717 A CN113064717 A CN 113064717A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- task
- executed
- subsystem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 94
- 230000006870 function Effects 0.000 claims description 27
- 238000013075 data extraction Methods 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 238000013500 data storage Methods 0.000 claims description 13
- 238000005457 optimization Methods 0.000 abstract description 3
- 238000007726 management method Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000009795 derivation Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to the field of computers, and in particular, to a method, an apparatus, a computer device, and a storage medium for uniformly managing a data export function of a plurality of subsystems in a system. The method comprises the following steps: when a data export request is received, generating a task to be executed according to data export requirement information carried by the data export request, and adding the generated task to be executed into a task set to be executed; the method comprises the steps of obtaining machine resource data of each subsystem in a timing mode, and obtaining target tasks to be executed which meet preset task obtaining conditions from the task to be executed in a centralized mode according to the machine resource data of each subsystem; and determining a target subsystem needing to export data according to the task parameters of the target task to be executed, extracting the target data needing to be exported from the target subsystem, and sending the target data to a file storage server for downloading by a user. The embodiment of the invention can realize centralized management and optimization of the resources of the system and improve the utilization rate of the resources.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method, an apparatus, a computer device, and a storage medium for uniformly managing a data export function of a plurality of subsystems in a system.
Background
The following statements are merely provided to provide background information related to the present disclosure and may not necessarily constitute prior art.
In the daily operation of the system, although each subsystem is divided according to the service division, each subsystem bears the pressure of exporting data, including the pressure in the aspects of I/O, memory, network and the like, along with the daily requirement iteration process, and the process of generating an export file by each subsystem occupies the resources serving the subsystem, and may also affect the risk of the subsystem during the service operation peak period.
Disclosure of Invention
The embodiment of the invention can realize centralized management and optimization of the system resources and improve the utilization rate of the resources.
The present invention provides, according to a first aspect, a method of unified management of data export functions of a plurality of subsystems in a system, which in one embodiment provides a download service platform for providing a data export service for each accessed subsystem; the method comprises the following steps:
when a data export request is received, generating a task to be executed according to data export requirement information carried by the data export request, and adding the generated task to be executed into a task set to be executed;
the method comprises the steps of obtaining machine resource data of each subsystem in a timing mode, and obtaining target tasks to be executed which meet preset task obtaining conditions from the task to be executed in a centralized mode according to the machine resource data of each subsystem;
and determining a target subsystem needing to export data according to the task parameters of the target task to be executed, extracting the target data needing to be exported from the target subsystem, and sending the target data to a file storage server for downloading by a user.
In one embodiment, the step of acquiring a target task to be executed meeting a preset task acquisition condition from a task set to be executed according to machine resource data of each subsystem includes:
determining a subsystem corresponding to each task to be executed according to the system code of each task to be executed in the task set to be executed; the data derived when the task to be executed is executed comes from the corresponding subsystem;
evaluating the machine resource occupation information of each task to be executed, wherein the machine resource occupation information of the task to be executed represents the machine resources occupied by the data to be exported of the task to be executed;
and judging whether each task to be executed can be executed according to the machine resource occupation information of each task to be executed and the machine resource data of the corresponding subsystem, and determining the task to be executed which is judged to be executable as a target task to be executed according with a preset task obtaining condition.
In one embodiment, the steps of determining a target subsystem needing to export data according to task parameters of a target task to be executed, extracting the target data needing to be exported from the target subsystem, and sending the target data to a file storage server include:
acquiring task parameters of a target task to be executed, wherein the task parameters comprise system codes, target method path information, target method parameter information and derived data storage position information;
determining a subsystem corresponding to the system code in the plurality of subsystems as a target subsystem needing to derive data;
calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain target data needing to be exported;
generating an export file according to the target data;
and uploading the generated export file to a directory of the file storage server corresponding to the export data storage position information.
In one embodiment, the task parameters further include a number of single-pass counts, the number of single-pass counts being a threshold value characterizing a maximum amount of data to generate an export file;
the step of generating an export file from the target data comprises:
when the data volume of the obtained target data is determined not to exceed the number of single processing pieces, generating an export file according to the target data;
and when the data amount of the obtained target data is determined to exceed the number of single processing pieces, generating a plurality of export files according to the target data, wherein the data amount of each generated export file does not exceed the number of single processing pieces.
In one embodiment, after the step of uploading the generated export file to a directory of the file storage server corresponding to the export data storage location information, the method further includes:
determining a user terminal corresponding to a target task to be executed;
and returning the download link of the exported file to the user terminal.
In one embodiment, the step of calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain target data to be exported includes:
calling a preset data extraction interface of a target subsystem according to the target method path information and the target method parameter information to obtain a data list, wherein data in the data list are target data to be exported;
and acquiring the data in the data list by adopting a one-time query mode or a paging query mode.
In one embodiment, the machine resource data for each subsystem is retrieved from a cache; the method further comprises the following steps: when target data needing to be exported are extracted from the target subsystem, machine resource data of the target subsystem are inquired in real time, and the inquired machine resource data are stored into a cache.
The present invention provides, according to a second aspect, an apparatus for unified management of data export functionality of a plurality of subsystems in a system, the apparatus comprising, in one embodiment:
the task generation module is used for generating a task to be executed according to the data export requirement information carried by the data export request when the data export request is received, and adding the generated task to be executed into a task set to be executed;
the target task acquisition module is used for acquiring the machine resource data of each subsystem at regular time and acquiring target tasks to be executed which accord with preset task acquisition conditions from the task set to be executed according to the machine resource data of each subsystem;
and the data export module is used for determining a target subsystem needing data export according to the task parameters of the target task to be executed, extracting the target data needing data export from the target subsystem, and sending the target data to the file storage server for downloading by the user.
The present invention provides according to a third aspect a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of an embodiment of any of the methods described above when executing the computer program.
The present invention provides according to a fourth aspect a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the embodiments of the method of any one of the above.
In the embodiment of the invention, the resource consumption of a plurality of subsystems of the system for generating and uploading files to the file storage server is extracted, and the part of resources are concentrated in a single service application, so that each subsystem only needs to bear the interface pressure of data extraction, the resource consumption of each subsystem can be reduced, and the index stability of each subsystem can be improved. The export functions of the subsystems in the system are managed in a centralized and unified mode, so that resource consumption of the subsystems in executing tasks such as file writing and file transmission can be reduced, centralized management and optimization of the resources of the system can be realized, the utilization rate of the resources of the system is improved, and centralized management of data export authorities is facilitated by centralized service of the export functions of the subsystems.
Drawings
FIG. 1 is a diagram of an application environment for a system in which data export functions are managed by respective subsystems;
FIG. 2 is a diagram of an application environment for a method for unified management of data export functionality for multiple subsystems in a system, according to one embodiment;
FIG. 3 is a flow diagram that illustrates a method for unified management of data export functionality for multiple subsystems in a system, under an embodiment;
FIG. 4 is a flow diagram that illustrates obtaining a target executable task in one embodiment;
FIG. 5 is a schematic flow chart illustrating the derivation of target d data in one embodiment;
FIG. 6 is a block diagram of an apparatus that may be used to uniformly manage data export functions for multiple subsystems in a system, according to one embodiment;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Many subsystems in the system have a data export requirement, and each subsystem is developed with a data export function at this time, as shown in fig. 1, relevant persons, such as test persons, operators, developers and the like, of each subsystem having the data export requirement can export relevant data from the subsystem through the data export function of the subsystem, and at this time, each subsystem bears the pressure of exporting data. In addition, each subsystem develops a data export function, and thus, the problem of repeated development is also caused.
In view of the above disadvantages, the present invention provides a method for uniformly managing data export functions of multiple subsystems in a system, which can transfer and reduce resource pressure brought by each subsystem when exporting data, and can be applied in a distributed environment or a single machine environment. Because each subsystem in the distributed environment is usually physically isolated, namely each subsystem is deployed in different server environments, the method can bring more obvious beneficial effects when being applied to the distributed environment, namely the resource pressure brought by each subsystem when exporting data can be more obviously reduced compared with the method when being applied to a single-machine environment; when the method is applied to a single machine environment, the advantage of code separation can be generated, namely the service of each subsystem is decoupled, and the maintainability is higher.
In one embodiment, the method for uniformly managing data export functions of multiple subsystems in a system can be applied to an application environment as shown in fig. 2.
In the application environment, a data extraction interface is established for each subsystem in the system in advance and is provided for the download service platform, so that the download service platform can extract data which a user wants to export from each subsystem through each data extraction interface. All the relevant persons (hereinafter referred to as users) having data export requirements in each subsystem may send data export requests to the download service platform through their user terminals (10 shown in fig. 2), and the download service platform may extract data that the users want to query from the relevant subsystem in response to the data export requests, and then generate an export file from the extracted data and provide the export file to the user terminals.
It should be noted that, a user can export data in any subsystem through the download service platform, and when each subsystem manages its own data export function in the past, the user can only export data in the subsystem. It should be further noted that in some scenarios, it is not necessary to centralize the data export functions of all subsystems in the system into the download service platform, and therefore, the data export functions of a plurality of subsystems in the system may be centralized into the download service platform, and at this time, the download service platform only provides the data export services for the plurality of subsystems, but it can be understood that at this time, the user can only export data in all subsystems accessing the download service platform through the download service platform.
The user terminal (e.g., 1-4 in fig. 1, 10 in fig. 2) may include, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, desktop computers, and the like. Any one of the subsystems in fig. 1 and fig. 2 may be implemented by an independent server or a server cluster composed of a plurality of servers, and of course, all the subsystems may also be deployed in an independent server.
The method for uniformly managing the data export function of the subsystems in the system provided by the embodiment provides a download service platform, and the download service platform can provide data export service for all the subsystems accessed to the download service platform. The following description is given by taking the method as an example applied to the download service platform in fig. 2, and as shown in fig. 3, the method includes:
s110: and when a data export request is received, generating a task to be executed according to the data export requirement information carried by the data export request, and adding the generated task to be executed into a task set to be executed.
In this embodiment, when a user needs to export data in a certain subsystem or some subsystems, a data export request may be sent to the download service platform through the user terminal thereof, where the data export request sent by the user includes information provided to the download service platform for determining data to be exported by the user, that is, data export requirement information, and after receiving the data export request sent by each user, the download service platform generates a task to be executed based on the data export requirement information and places the task into a task set to be executed.
When the download service platform generates the to-be-executed task according to the data export requirement information, the information contained in the to-be-executed task is stored as a task parameter, so that when the to-be-executed task needs to be executed, operations of relevant data extraction, generation of an export file from the extracted data and the like are completed according to the corresponding task parameter. The data export request information may include a system code (each subsystem has a unique system code, and information in this field of the system code is used to let the platform, i.e., the download service platform, know which subsystem to call), a method path (used to let the platform know which method to call the subsystem), and a method parameter (used to let the platform know how to call this method of the subsystem, i.e., a method corresponding to the method path). Which information is specifically included in the data derivation requirement information can be configured according to the needs of a particular scenario. For example, in one scenario, the data export request information may further include the number of single processing pieces (for the platform to know the number of single file bearing records expected by the service, for example, set to 100, the platform generates one export file from 100 pieces of data every time the platform fetches 100 pieces of data, and if 200 pieces of data are fetched in total, generates 2 export files), whether to perform multithreading (for the platform to know which way to select to call the subsystem, such as single-thread call or multithreading call), an upload file directory (for the platform to know where to put the generated export file for saving), and so on. The field information included in the task parameter is the same as the field information included in the data export request information, that is, the information includes the fields such as the system code and the method path described above.
S120: and acquiring machine resource data of each subsystem at regular time, and acquiring target tasks to be executed which accord with preset task acquisition conditions from the task set to be executed according to the machine resource data of each subsystem.
In this embodiment, the set of tasks to be executed includes a plurality of tasks to be executed, the download service platform may periodically retrieve a certain number of tasks to be executed from the set of tasks to be executed and execute the tasks, and the tasks to be executed retrieved by the download service platform are referred to as target tasks to be executed. The tasks to be executed, which are fished by the download service platform, need to meet some preset conditions (i.e., preset task obtaining conditions), and how many tasks to be executed are fished from the tasks to be executed by the download service platform each time for execution, which may be configured by relevant managers of the download service platform according to actual needs, which is not limited in this embodiment.
In order to avoid the influence on the service of the subsystem caused by excessive pressure on the subsystem when the data is extracted from the subsystem, therefore, when the download service platform fetches the to-be-executed task to be executed from the to-be-executed task set, it is first determined whether each subsystem related to the to-be-executed task currently has the capability of providing the data to be derived from the to-be-executed task (referred to as the subsystem) to the download service platform, for example, the current resource of the a subsystem can only process 100 pieces of data, while the B task needs to derive 150 pieces of data from the a subsystem, and then it can be determined that the a subsystem does not have the capability of providing the data required to be derived by the B task.
In an embodiment, the step of acquiring, from the set of tasks to be executed, a target task to be executed that meets a preset task acquisition condition according to the machine resource data of each subsystem may be as shown in fig. 4, and includes:
s121: determining a subsystem corresponding to each task to be executed according to the system code of each task to be executed in the task set to be executed;
s122: evaluating the machine resource occupation information of each task to be executed, wherein the machine resource occupation information of the task to be executed represents the machine resources occupied by the data to be exported of the task to be executed;
s123: and judging whether each task to be executed can be executed according to the machine resource occupation information of each task to be executed and the machine resource data of the corresponding subsystem, and determining the task to be executed which is judged to be executable as a target task to be executed according with a preset task obtaining condition.
In this embodiment, when the download service platform fetches a to-be-executed task, a subsystem corresponding to each to-be-executed task is determined according to a system code in a task parameter of each to-be-executed task in the to-be-executed task set, where the subsystem corresponding to the system code of the to-be-executed task is a subsystem corresponding to the system code of the to-be-executed task, and data derived by each to-be-executed task during execution is from the corresponding subsystem.
The downloading service platform firstly evaluates the machine resource occupation information of each task to be executed, the information represents machine resources which are probably occupied by a corresponding subsystem of the task to be executed when data which are derived from the task to be executed are provided, then machine resource information of the corresponding subsystem is obtained, whether each task to be executed can be executed or not is judged according to the machine resource occupation information of each task to be executed and the machine resource data of the corresponding subsystem, and the task to be executed which is judged to be executable is taken out by the downloading service platform. The machine resource data of the subsystem may include, but is not limited to, a remaining percentage of a machine memory, a remaining percentage of a CPU resource, a remaining percentage of an I/O (i.e., Input/Output), a configuration of an export data amount (for limiting an export data amount per task to be executed), and a machine connectivity. The machine resource occupation information of the task to be executed may be evaluated based on the size of the data amount to be derived of the task to be executed, and may be evaluated based on the data record in the past task extraction, for example, 100 pieces of data are extracted, how much proportion of machine memory, I/O, CPU resources are occupied, and the like. If the data size to be exported by the task to be executed does not exceed the export data size configuration of the corresponding subsystem, the evaluated proportion of machine memory, CPU resource, I/O and the like which need to be occupied does not exceed the obtained residual proportion of the corresponding field of the corresponding subsystem, and then the task execution state of the task to be executed can be judged to be executable.
In another embodiment, priority weights may be configured for different subsystems, and the higher the weight value is, the higher the priority is, when the download service platform catches the task to be executed, the tasks to be executed can be firstly sequenced according to the priority weights of the subsystems corresponding to the tasks to be executed in the task set to be executed, the task to be executed which is ranked at the top (with the highest priority weight) is sequentially acquired after the sequencing is finished, then judging whether the acquired task to be executed can be executed according to the machine resource occupation information of the task to be executed and the machine resource data of the corresponding subsystem, if the task is judged to be executable, the task is taken as a target task to be executed, if the task is judged to be non-executable, the next task to be executed is acquired again for judgment, and by analogy, stopping the current target executable task fishing operation until the obtained number of the target tasks to be executed meets a preset number threshold.
Further, the downloading service platform can periodically acquire the accessed machine resource data of each subsystem from the cache. The machine resource data in the cache may be obtained by querying the machine resource data of each subsystem in real time when the data is extracted from the subsystem, and then storing the queried machine resource data of the subsystem in the cache for a certain time period, for example, ten minutes. If the download service platform can not obtain the machine resource data of the subsystem in the cache, the relevant subsystem is queried in real time, and the queried latest machine resource data is updated into the cache.
S130: and determining a target subsystem needing to export data according to the task parameters of the target task to be executed, extracting the target data needing to be exported from the target subsystem, and sending the target data to a file storage server for downloading by a user.
In this embodiment, after the download service platform obtains the target to-be-executed task from the to-be-executed task set, the task parameter of the target to-be-executed task is obtained, and corresponding operations are performed according to the task parameter, for example, determining a target subsystem that needs to export data, extracting the target data that needs to be exported from the target subsystem, sending the target data to a file storage server, and the like, and the data export requirement of the user can be met only when the target to-be-executed task is executed based on the task parameter.
According to the embodiment of the invention, the resource consumption of a plurality of subsystems of the system for generating and uploading the file to the file storage server is extracted, and the part of resources are concentrated in a single service application, so that each subsystem only needs to bear the interface pressure of data extraction, the resource consumption of each subsystem can be reduced, and the index stability of each subsystem can be improved. Moreover, the export functions of the subsystems in the system are managed in a centralized and unified manner, so that the resource consumption of each subsystem when executing tasks such as writing files, file transmission and the like can be reduced, the resources of the system can be managed and optimized in a centralized manner, the utilization rate of the resources of the system is improved, and the centralized management of data export permission is facilitated by centralized servicing of the export functions of the subsystems.
In an embodiment, the step of determining a target subsystem requiring data export according to task parameters of a target task to be executed in the above embodiment, extracting target data requiring data export from the target subsystem, and sending the target data to the file storage server in the above embodiment, as shown in fig. 5, includes:
s131: acquiring task parameters of a target task to be executed, wherein the task parameters comprise system codes, target method path information, target method parameter information and derived data storage position information;
s132: determining a subsystem corresponding to the system code in the plurality of subsystems as a target subsystem needing to derive data;
s133: calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain target data needing to be exported;
s134: generating an export file according to the target data;
s135: and uploading the generated export file to a directory of the file storage server corresponding to the export data storage position information.
In this embodiment, the task parameters of the task to be executed include system code, target method path information, target method parameter information, and derived data storage location information. At this time, when the download service platform executes the target to-be-executed task, the corresponding subsystem (i.e., the target subsystem) may be determined according to the system code in the task parameter of the target to-be-executed task, and then the data extraction interface provided by the target subsystem to the download service platform is called according to the information of the method path and the method parameter field in the task parameter, i.e., the target method path information and the target method parameter information, so as to obtain the target data to be exported.
For the obtained target data, the download service platform generates an export file from the target data according to a preset file format of the user, for example, generates an export file in an excel format or the like, and uploads the generated export file to a directory corresponding to export data storage location information in the file storage server, so that the relevant user can download the export file containing the data to be exported from the file storage server. Specifically, after the generated export file is uploaded to a directory of the file storage server corresponding to the export data storage location information, the download service platform determines a user terminal corresponding to the target task to be executed, and then returns a download link of the export file to the user terminal, so that the user can download the export file from the file storage server through the download link.
In one embodiment, the step of calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain target data to be exported includes: calling a preset data extraction interface of a target subsystem according to the target method path information and the target method parameter information to obtain a data list, wherein data in the data list are target data to be exported; and acquiring the data in the data list by adopting a one-time query mode or a paging query mode.
If the data amount of the target data is small, for example, does not exceed a preset threshold, a one-time query mode may be adopted, that is, data is extracted from the relevant subsystem at one time. If the data size of the target data is large, the data in the data list may be obtained by a paging query method, specifically, before starting data extraction, a total data size of the target data is queried first, paging calculation is performed according to the total data size, and then rolling query is performed according to paging on the paging list data. And the corresponding data extraction mode is adopted according to the data quantity of the target data, so that the data extraction efficiency can be improved.
In another embodiment, the task parameters further include a number of single passes, the number of single passes being a threshold value characterizing a maximum amount of data to generate an export file. And the step of generating an export file from the target data may comprise: when the data volume of the obtained target data is determined not to exceed the number of single processing pieces, generating an export file according to the target data; and when the data amount of the obtained target data is determined to exceed the number of single processing pieces, generating a plurality of export files according to the target data, wherein the data amount of each generated export file does not exceed the number of single processing pieces. If the data amount of the target data exceeds the number of pieces of single processing, several export files need to be generated, for example, the data amount of the target data is 100 pieces of data, and the number of pieces of single processing is set to 40, the target data is generated as an export file in such a way that one export file is generated for every 40 pieces of data, so that 3 export files can be generated.
Fig. 3, 4, and 5 are flow diagrams illustrating a method for unified management of data export functionality of multiple subsystems in a system, according to an embodiment. It should be understood that, although the steps in the flowcharts of fig. 3, 4 and 5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 3, 4, and 5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least some of the sub-steps or stages of other steps.
Based on the same inventive concept, the invention also provides a device for uniformly managing the data export function of the subsystems in the system. In one embodiment, as shown in fig. 6, the apparatus for unifying data export functions of a plurality of subsystems in the management system comprises the following modules:
the task generation module is used for generating a task to be executed according to the data export requirement information carried by the data export request when the data export request is received, and adding the generated task to be executed into a task set to be executed;
the target task acquisition module is used for acquiring the machine resource data of each subsystem at regular time and acquiring target tasks to be executed which accord with preset task acquisition conditions from the task set to be executed according to the machine resource data of each subsystem;
and the data export module is used for determining a target subsystem needing data export according to the task parameters of the target task to be executed, extracting the target data needing data export from the target subsystem, and sending the target data to the file storage server for downloading by the user.
In one embodiment, the target task obtaining module includes:
the corresponding subsystem determining submodule is used for determining a corresponding subsystem of each task to be executed according to the system code of each task to be executed in the task set to be executed; the data derived when the task to be executed is executed comes from the corresponding subsystem;
the evaluation submodule is used for evaluating the machine resource occupation information of each task to be executed, and the machine resource occupation information of the task to be executed represents the machine resources occupied by the data to be exported of the task to be executed;
and the target task obtaining submodule is used for judging whether each task to be executed can be executed according to the machine resource occupation information of each task to be executed and the machine resource data of the corresponding subsystem of the task to be executed, and determining the task to be executed which is judged to be executable as the target task to be executed which meets the preset task obtaining condition.
In one embodiment, a data export module comprises:
the task parameter acquisition submodule is used for acquiring task parameters of a target task to be executed, and the task parameters comprise system codes, target method path information, target method parameter information and derived data storage position information;
the target subsystem determining submodule is used for determining a subsystem corresponding to the system code in the subsystems as a target subsystem needing to derive data;
the data extraction submodule is used for calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information so as to obtain target data needing to be exported;
the file generation submodule is used for generating an export file according to the target data;
and the file export submodule is used for uploading the generated export file to a directory of the file storage server corresponding to the export data storage position information.
In one embodiment, the task parameters further include a number of single-pass counts, the number of single-pass counts being a threshold value characterizing a maximum amount of data to generate an export file; correspondingly, the file generation submodule is also used for generating an export file according to the target data when the data volume of the obtained target data is determined not to exceed the number of single processing pieces; and generating a plurality of export files according to the target data when the data amount of the obtained target data is determined to exceed the number of single processing pieces, wherein the data amount of each generated export file does not exceed the number of single processing pieces.
In one embodiment, the apparatus for unifying data export functionality for a plurality of subsystems in a management system further comprises:
the terminal determining module is used for determining a user terminal corresponding to the target task to be executed;
and the download link returning module is used for returning the download link of the exported file to the user terminal.
In one embodiment, the data extraction submodule includes:
the data list acquisition unit is used for calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to acquire a data list, and data in the data list is target data needing to be exported;
and the data extraction unit is used for acquiring the data in the data list by adopting a one-time query mode or a paging query mode.
In one embodiment, the machine resource data for each subsystem is retrieved from a cache. Correspondingly, the device for uniformly managing the data export function of the subsystems in the system also comprises a machine resource acquisition module.
And the machine resource acquisition module is used for inquiring the machine resource data of the target subsystem in real time when the target data needing to be exported is extracted from the target subsystem, and storing the inquired machine resource data into the cache.
For specific limitations of the means for data export function of multiple subsystems in the unified management system, reference may be made to the above limitations of the method for data export function of multiple subsystems in the unified management system, and details are not described here. The modules in the device for uniformly managing the data export function of the subsystems in the system can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, the internal structure of which may be as shown in FIG. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data such as machine resource data of the plurality of subsystems, and the specific stored data can be referred to as the definition in the above method embodiment. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of unifying data export functionality of multiple subsystems in a management system.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the method for unified management of data export functions of multiple subsystems in a system provided in any of the above method embodiments.
In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method of centrally managing data export functionality of a plurality of subsystems in a system as provided in any one of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for uniformly managing data export functions of a plurality of subsystems in a system is characterized by providing a download service platform, wherein the download service platform provides data export service for each subsystem; the method comprises the following steps:
when a data export request is received, generating a task to be executed according to data export requirement information carried by the data export request, and adding the generated task to be executed into a task set to be executed;
the method comprises the steps of obtaining machine resource data of each subsystem at regular time, and obtaining target tasks to be executed which accord with preset task obtaining conditions from the task set to be executed according to the machine resource data of each subsystem;
and determining a target subsystem needing to export data according to the task parameters of the target task to be executed, extracting the target data needing to be exported from the target subsystem, and sending the target data to a file storage server for a user to download.
2. The method according to claim 1, wherein the step of acquiring the target tasks to be executed from the set of tasks to be executed according to the machine resource data of each subsystem, wherein the target tasks to be executed meet preset task acquisition conditions, comprises:
determining a subsystem corresponding to each task to be executed according to the system code of each task to be executed in the task set to be executed; the data derived when the task to be executed is executed comes from the corresponding subsystem;
evaluating the machine resource occupation information of each task to be executed, wherein the machine resource occupation information of the task to be executed provides machine resources occupied by data to be exported by the task to be executed;
and judging whether each task to be executed can be executed according to the machine resource occupation information of each task to be executed and the machine resource data of the corresponding subsystem of the task to be executed, and determining the task to be executed which is judged to be executable as a target task to be executed according with a preset task obtaining condition.
3. The method of claim 1, wherein the steps of determining a target subsystem requiring data export according to the task parameters of the target task to be executed, extracting the target data requiring data export from the target subsystem, and sending the target data to a file storage server comprise:
acquiring task parameters of the target task to be executed, wherein the task parameters comprise system codes, target method path information, target method parameter information and derived data storage position information;
determining a subsystem corresponding to the system code in the plurality of subsystems as a target subsystem needing to derive data;
calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain target data needing to be exported;
generating an export file according to the target data;
and uploading the generated export file to a directory of a file storage server corresponding to the export data storage position information.
4. The method of claim 3, wherein the task parameters further include a number of single passes, the number of single passes being a threshold value characterizing a maximum amount of data to generate an export file; the step of generating an export file from the target data comprises:
when the data volume of the obtained target data is determined not to exceed the single processing number, generating an export file according to the target data;
and when the data amount of the obtained target data is determined to exceed the single processing number, generating a plurality of export files according to the target data, wherein the data amount of each generated export file does not exceed the single processing number.
5. The method of claim 3, wherein the step of uploading the generated export file to a directory of a file storage server corresponding to the export data storage location information is followed by:
determining a user terminal corresponding to the target task to be executed;
and returning the download link of the export file to the user terminal.
6. The method of claim 3, wherein the step of calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain the target data to be exported comprises:
calling a preset data extraction interface of the target subsystem according to the target method path information and the target method parameter information to obtain a data list, wherein data in the data list are target data needing to be exported;
and acquiring the data in the data list by adopting a one-time query mode or a paging query mode.
7. The method of claim 1, wherein the machine resource data for each of the subsystems is retrieved from a cache;
the method further comprises the following steps: when target data needing to be exported are extracted from the target subsystem, machine resource data of the target subsystem are inquired in real time, and the inquired machine resource data are stored in a cache.
8. An apparatus for uniformly managing data export functions of a plurality of subsystems in a system, the apparatus comprising:
the task generation module is used for generating a task to be executed according to data export requirement information carried by the data export request when the data export request is received, and adding the generated task to be executed into a task set to be executed;
the target task acquisition module is used for acquiring the machine resource data of each subsystem at regular time and acquiring a target task to be executed which meets a preset task acquisition condition from the task set to be executed according to the machine resource data of each subsystem;
and the data export module is used for determining a target subsystem needing data export according to the task parameters of the target task to be executed, extracting the target data needing data export from the target subsystem, and sending the target data to a file storage server for downloading by a user.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110235415.7A CN113064717A (en) | 2021-03-03 | 2021-03-03 | Method and device for uniformly managing data export function of multiple subsystems in system |
| CA3151001A CA3151001A1 (en) | 2021-03-03 | 2022-03-03 | Method and device for uniformly managing data export functions of multiple subsystems in system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110235415.7A CN113064717A (en) | 2021-03-03 | 2021-03-03 | Method and device for uniformly managing data export function of multiple subsystems in system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113064717A true CN113064717A (en) | 2021-07-02 |
Family
ID=76559582
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110235415.7A Pending CN113064717A (en) | 2021-03-03 | 2021-03-03 | Method and device for uniformly managing data export function of multiple subsystems in system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113064717A (en) |
| CA (1) | CA3151001A1 (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947789A (en) * | 2019-01-28 | 2019-06-28 | 平安科技（深圳）有限公司 | A kind of method, apparatus, computer equipment and the storage medium of the data processing of multiple database |
| CN111290745A (en) * | 2020-02-11 | 2020-06-16 | 车配齐(珠海横琴)科技有限公司 | Device and method for realizing general data export |
| CN111352951A (en) * | 2020-02-26 | 2020-06-30 | 苏宁云计算有限公司 | Data export method, device and system |
| CN112256417A (en) * | 2020-10-23 | 2021-01-22 | 上海豹云网络信息服务有限公司 | Data request processing method and device and computer readable storage medium |
-
2021
- 2021-03-03 CN CN202110235415.7A patent/CN113064717A/en active Pending
-
2022
- 2022-03-03 CA CA3151001A patent/CA3151001A1/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947789A (en) * | 2019-01-28 | 2019-06-28 | 平安科技（深圳）有限公司 | A kind of method, apparatus, computer equipment and the storage medium of the data processing of multiple database |
| CN111290745A (en) * | 2020-02-11 | 2020-06-16 | 车配齐(珠海横琴)科技有限公司 | Device and method for realizing general data export |
| CN111352951A (en) * | 2020-02-26 | 2020-06-30 | 苏宁云计算有限公司 | Data export method, device and system |
| CN112256417A (en) * | 2020-10-23 | 2021-01-22 | 上海豹云网络信息服务有限公司 | Data request processing method and device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CA3151001A1 (en) | 2022-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108965381B (en) | Nginx-based load balancing implementation method and device, computer equipment and medium | |
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN109787908B (en) | Server current limiting method, system, computer equipment and storage medium | |
| CN108256114B (en) | Document online preview method and device, computer equipment and storage medium | |
| CN111741112B (en) | File downloading method, device, equipment and storage medium based on artificial intelligence | |
| CN110928653B (en) | Cross-cluster task execution method and device, computer equipment and storage medium | |
| CN110851159B (en) | Business rule updating method and device, computer equipment and storage medium | |
| CN110704177B (en) | Computing task processing method and device, computer equipment and storage medium | |
| CN110213392B (en) | Data distribution method and device, computer equipment and storage medium | |
| CN110716989A (en) | Dimension data processing method and device, computer equipment and storage medium | |
| CN111738404B (en) | Model training task processing method and device, electronic equipment and storage medium | |
| CN108845869B (en) | Concurrent request control method, device, computer equipment and storage medium | |
| CN112698952A (en) | Unified management method and device for computing resources, computer equipment and storage medium | |
| CN111224939B (en) | Task request intercepting method and device, computer equipment and storage medium | |
| CN111885184A (en) | Method and device for processing hot spot access keywords in high concurrency scene | |
| CN112422450A (en) | Computer equipment, and flow control method and device for service request | |
| CN110659272A (en) | Data cleaning method and system | |
| CN111897843B (en) | Configuration method and device of data flow strategy of Internet of things and computer equipment | |
| CN112181423B (en) | Project construction method and device based on configuration file, computer equipment and medium | |
| CN113419950A (en) | Method and device for generating UI automation script, computer equipment and storage medium | |
| CN110648052B (en) | Wind control decision method and device, computer equipment and storage medium | |
| CN109857344B (en) | Heartbeat state judgment method and device based on shared memory and computer equipment | |
| CN113064717A (en) | Method and device for uniformly managing data export function of multiple subsystems in system | |
| CN114090268B (en) | Container management method and container management system | |
| CN115017538A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210702 |