CN115168302A - Business data export method and device and electronic equipment - Google Patents
Business data export method and device and electronic equipment Download PDFInfo
- Publication number
- CN115168302A CN115168302A CN202210744002.6A CN202210744002A CN115168302A CN 115168302 A CN115168302 A CN 115168302A CN 202210744002 A CN202210744002 A CN 202210744002A CN 115168302 A CN115168302 A CN 115168302A
- Authority
- CN
- China
- Prior art keywords
- data
- service data
- sub
- service
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a business data exporting method, a business data exporting device and electronic equipment, wherein the method comprises the following steps: acquiring service data to be exported; under the condition that the data volume of the service data is larger than the target data volume in the preset configuration information, splitting the service data to obtain multiple copies of sub-service data; generating a template file in a single target data format for each part of sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data; and exporting the service file corresponding to each sub-service data. Compared with the loading of files storing all service data, the loading of one service file alone is certainly quicker and the time consumption is shorter. Therefore, under the condition that the data volume of the service data is overlarge, the data in the service file can be loaded and displayed to the user in a short time, so that the waiting time of the user is shortened, and the use experience of the user is improved.
Description
Technical Field
The present invention relates to the field of computer and software development, and in particular, to a method and an apparatus for exporting service data, and an electronic device.
Background
It is well known that the productive life of people cannot keep information interaction. When information is interacted on a computer, a large amount of data can be generated, and people can acquire the data through a report-type service program and export the data to the local.
In the current report service program, firstly, service data is acquired, and then the service data is exported to a local disk and stored as a file in an excel (table) document format.
However, when the data amount of the service data is too large, it usually takes a long time for the user to load the file completely and display the data when viewing the file containing the service data, so that the user needs to wait for a long time to see the service data, thereby degrading the user experience.
Disclosure of Invention
The embodiment of the invention aims to provide a business data exporting method, a business data exporting device and electronic equipment, which can solve the problem that a loading process takes longer time when a file exported in the related technology contains a large amount of business data.
In a first aspect, an embodiment of the present invention provides a method for exporting service data, where the method includes:
acquiring service data to be exported;
under the condition that the data volume of the service data is larger than the target data volume in preset configuration information, splitting the service data to obtain multiple sub-service data, wherein the data volume of each sub-service data is smaller than or equal to the target data volume;
generating a template file in a single target data format for each sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data;
and exporting the service file corresponding to each sub-service data.
Optionally, the splitting the service data to obtain multiple copies of sub-service data includes:
rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
and splitting the service data according to the target data volume to obtain a second number of sub-service data, wherein the second number is equal to the first number when the data volume of the service data is an integral multiple of the target data volume, and the second number is equal to the sum of the first number and one when the data volume of the service data is not the integral multiple of the target data volume.
Optionally, the splitting the service data to obtain multiple copies of sub-service data includes:
calculating a quotient of dividing the data volume of the service data by the target data volume to obtain a target value;
determining a third number based on the target value, wherein the third number is equal to the target value if the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
and averagely splitting the service data into the sub-service data of the third quantity.
Optionally, the method further comprises:
receiving a configuration request carrying a target value;
and updating the target data volume in the preset configuration information based on the target numerical value.
Optionally, the exporting the service file corresponding to each sub-service data includes:
compressing the service file corresponding to each sub-service data into a compressed file;
exporting the compressed file to the local of the electronic equipment.
Optionally, the generating a template file in a single target data format for each piece of the sub-service data, storing the sub-service data in the template file, and generating a service file corresponding to the sub-service data includes:
and creating a thread aiming at each sub-service data, generating a template file in a single target data format in the thread, storing the sub-service data to the template file in the thread, and generating a service file corresponding to the sub-service data.
Optionally, the preset configuration information further includes: thread pool configuration information, after said creating a thread for each of said sub-service data, said method further comprising:
submitting the thread to a thread pool created based on the thread pool configuration information.
Optionally, the method further comprises:
monitoring the running state of the thread pool;
and closing the thread pool under the condition that the running state represents that all threads in the thread pool are executed completely.
In a second aspect, an embodiment of the present invention provides an apparatus for exporting service data, where the apparatus includes:
the acquisition module is used for acquiring the service data to be exported;
the splitting module is used for splitting the service data to obtain multiple sub-service data under the condition that the data volume of the service data is larger than the target data volume in preset configuration information, wherein the data volume of each sub-service data is smaller than or equal to the target data volume;
the service file module is used for generating a template file in a single target data format aiming at each sub-service data, storing the sub-service data into the template file and generating a service file corresponding to the sub-service data;
and the export module is used for exporting the service file corresponding to each sub-service data.
Optionally, the splitting module includes:
the first calculation unit is used for rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
a first splitting unit, configured to split the service data according to the target data volume to obtain a second number of sub-service data, where the second number is equal to the first number when the data volume of the service data is an integer multiple of the target data volume, and the second number is equal to a sum of the first number and one when the data volume of the service data is not an integer multiple of the target data volume.
Optionally, the splitting module includes:
the second calculation unit is used for calculating the quotient of dividing the data volume of the service data by the target data volume to obtain a target value;
a third calculation unit configured to determine a third number based on the target value, wherein the third number is equal to the target value when the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
a second splitting unit, configured to split the service data into the third number of sub-service data on average.
Optionally, the apparatus further comprises:
the receiving module is used for receiving a configuration request carrying a target numerical value;
and the configuration module is used for updating the target data volume in the preset configuration information based on the target value.
Optionally, the deriving module includes:
a compressing unit, configured to compress the service file corresponding to each sub-service data into a compressed file;
and the exporting unit is used for exporting the compressed file to the local part of the electronic equipment.
Optionally, the service file module is specifically configured to create a thread for each piece of the sub-service data, generate a template file in a single target data format in the thread, store the sub-service data in the template file in the thread, and generate a service file corresponding to the sub-service data.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory and a computer program stored on the memory and operable on the processor, the processor implementing the method for deriving service data as described above when executing the program.
In a fourth aspect, an embodiment of the present invention provides a readable storage medium, where instructions, when executed by a processor of an electronic device, enable the electronic device to perform the method for deriving service data as described in any one of the above.
In the embodiment of the present invention, after the service data to be exported is obtained, the data volume of the service data is measured, so that the service data is split to obtain multiple sub-service data under the condition that the data volume of the service data is greater than the target data volume in the preset configuration information, so that the data volume of each sub-service data does not exceed the target data volume. And then generating a template file in a single target data format for each part of sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data. Therefore, under the condition that the data volume of the service data is overlarge, the data in the service file can be loaded and displayed to the user in a short time, so that the waiting time of the user is shortened, and the use experience of the user is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
Fig. 1 is a flowchart illustrating steps of a method for exporting service data according to an embodiment of the present invention;
fig. 2 is a flowchart of a step of splitting service data to obtain multiple copies of sub-service data in a method for exporting service data according to an embodiment of the present invention;
fig. 3 is a second flowchart illustrating steps of a business data exporting method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a method for exporting service data according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a device for exporting service data according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In various embodiments of the present invention, it should be understood that the sequence numbers of the following processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Referring to fig. 1, an embodiment of the present invention provides a method for exporting service data, where the method includes:
step 101: and acquiring the service data to be exported.
It should be noted that the service data is data generated by the server in the process of performing service processing. For example, the service data related to the transaction service may be transaction data, and the service data related to the call service may be call data, and the like, which is not limited herein. Of course, the business data can be classified according to different uses of the business data, and specifically, the business data used for making the report can be classified into report business data. Taking the call data as an example, a table is usually set, and a header part of the table may include, but is not limited to, a user, a contact, a call duration, a call device number, and the like, and then corresponding contents are filled in according to the header part, so as to generate report-type service data in which the call data is stored. The service data to be exported in the present invention may be report-type service data, but is not limited thereto.
It can be understood that, for different service requirements, the user needs to derive different service data from the server. That is to say, for different service requirements, a user needs to query service data required by a corresponding service from a local database of the server and determine the service data as service data to be exported. For example, the server receives an inquiry request related to the call service sent by the electronic device, which indicates that the user has a need to inquire the service data related to the call service, at this time, the server inquires the service data related to the call service in the database based on the inquiry request, and determines the inquired service data as the service data to be derived, but is not limited thereto.
Step 102: and under the condition that the data volume of the service data is larger than the target data volume in the preset configuration information, splitting the service data to obtain multiple copies of sub-service data.
It should be noted that the preset configuration information is information associated with the export task, which is configured in the server in advance. And the export task is a task of exporting the business data. Taking the configuration file configured in the server as an example, the preset configuration information may include the target data size. The target data size may be regarded as the maximum data size of a single file when the single file is loaded. In order to avoid that the time required for loading a single file is longer and the data stored in the single file is less, a user can determine a data size with a proper size as a target data size in advance according to the requirement, so that the time required for loading the file with the target data size is shorter, and the number of the service data stored in the file is not less. Specifically, the target data amount is a threshold number, and may be, for example, 100 pieces of data, but is not limited thereto.
It will be appreciated that the target amount of data is a threshold that measures whether the time it takes to load a file falls within a longer or shorter time. When the data size of the service data is larger than the target data size in the preset configuration information, it indicates that the time required for loading the file is long, that is, if all the service data are stored in the same file, the time required for loading the file is too long. At this time, all the service data is split into a plurality of shares so that the data amount of each share is less than or equal to the target data amount to solve the problem. And each part of split data is each part of sub-service data, and the data volume of each part of sub-service data is less than or equal to the target data volume.
Step 103: and generating a template file in a single target data format for each part of sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data.
It should be noted that the target data format is a data format related to the business requirements, which determines in which data format the business data is exposed after it is derived. Preferably, the server stores template files of multiple data formats in advance, each template file of a data format corresponds to a different service, and after determining a service corresponding to the service data, the corresponding data format is selected as the target data format. The data format includes, but is not limited to, a table (excel), a text format (txt), and the like.
It can be understood that the generated template file can be regarded as a file without data padding, and therefore, after the template file corresponding to each piece of sub-service data is generated, the sub-service data needs to be stored in the template file, so that the service file corresponding to the sub-service data can be generated. Here, the sub-service data is stored in the template file, and data transmission may be implemented based on a preset channel. For example, data transmission may be realized by an io Stream (Input Output Stream), which is Input and Output in a streaming manner and can be regarded as a communication channel established between the local server and the template file. Specifically, after generating the template file corresponding to each sub-service data, an io stream between the local server and each template file is established, and each sub-service data is input into the corresponding template file in a stream manner, so as to generate the service file corresponding to each sub-service data.
Step 104: and exporting the service file corresponding to each sub-service data.
It should be noted that the service file corresponding to each sub-service data includes all the service data to be exported in step 101, and each service file carries a different part therein. The business file can be exported asynchronously in batches when being exported, so that the time required for executing an export task is reduced, but not limited to.
In the embodiment of the invention, after the business data to be exported is obtained, the data volume of the business data is measured, so that the business data is split under the condition that the data volume of the business data is larger than the target data volume in the preset configuration information, and a plurality of sub-business data are obtained, so that the data volume of each sub-business data does not exceed the target data volume. And then generating a template file in a single target data format for each part of sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data. Therefore, under the condition that the data volume of the service data is overlarge, the data in the service file can be loaded and displayed to the user in a short time, so that the waiting time of the user is shortened, and the use experience of the user is improved.
Optionally, splitting the service data to obtain multiple copies of sub-service data includes:
rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
and splitting the service data according to the target data volume to obtain a second number of sub-service data.
It should be noted that, in the case where the data amount of the service data is an integer multiple of the target data amount, the second number is equal to the first number, and in the case where the data amount of the service data is not an integer multiple of the target data amount, the second number is equal to the sum of the first number plus one. That is, in the case where the data volume of the service data is an integer multiple of the target data volume, the second number of sub-service data includes: the first number is equal to the sub-service data of the target data quantity, and the second number is equal to the first number. When the data volume of the service data is not an integer multiple of the target data volume, the second number of sub-service data includes: the data processing method comprises the steps of a first amount of sub-service data with the data quantity equal to the target data quantity and one sub-service data containing residual data, wherein the residual data are data except the first amount of sub-service data in the service data. That is, in the case that the data volume of the service data is not an integer multiple of the target data volume, the second number of sub-service data includes a first type of sub-service data and a second type of sub-service data, wherein the data volume of the first type of sub-service data is equal to the target data volume, and the number of the first type of sub-service data is equal to the first number; the data amount of the second type of sub-service data is less than the target data amount and the number thereof is equal to one share.
Under the condition that the service data is split into multiple sub-service data, and the data volume of each sub-service data is less than or equal to the target data volume, in order to avoid the excessive number of sub-service data, the service data can be split according to the target data volume, so that the data volume of each split sub-service data is equal to the target data volume as much as possible. It can be understood that, when the data volume of the service data is not an integer multiple of the target data volume, and the service data is split to the end, the remaining data volume is smaller than the target data volume, and at this time, all the remaining data is regarded as one sub-service data, so that the data volume of only one of the split sub-service data is smaller than the target data volume, and the remaining data volumes are equal to the target data volume. Under the condition that the data volume of the service data is integral multiple of the target data volume, when the service data is split to the end, the residual data volume is zero, namely, no residual data exists, so that the data volume of the split multiple sub-service data is equal to the target data volume. Taking the data amount of the traffic data as 100 pieces as an example, in the case that the target data amount is 20 pieces, the data amount of the traffic data ÷ target data amount = the first number, i.e., 100 ÷ 20=5, that is, the first number is 5, and since the data amount of the traffic data is an integer multiple of the target data amount, the second number is equal to the first number, i.e., the second number is 5. In the case that the target data amount is 30 pieces, the data amount of the service data ÷ target data amount = the remaining amount of the first number … …, that is, 100 ÷ 30=3 … …, that is, the first number is 3 and the remaining amount is 10, since the data amount of the service data is not an integral multiple of the target data amount, there are 10 pieces of remaining data, and these 10 pieces of data are separately stored as one piece of sub-service data, so the second number is the sum of the first number and the sub-service data where the remaining data is located, that is, the second number is 3+1= -4.
In the embodiment of the invention, in the process of splitting the service data into multiple sub-service data, the data volume of the sub-service data is preferentially split into the target data volume, and when the remaining data volume is smaller than or equal to the target data volume, all the remaining data is used as one sub-service data, so that the number of the sub-service data obtained by splitting is not excessive, and the problem that the split sub-service data occupies a large amount of resources due to the excessive number of the sub-service data is avoided.
As shown in fig. 2, an embodiment of the present invention provides a flowchart of a step of splitting service data to obtain multiple copies of sub-service data, where the flowchart includes:
step 201: calculating the total segment number k of data to be exported, wherein the total segment number k = (tds/fd), wherein tds represents the total data amount of the data to be exported (corresponding to the service data to be exported in the embodiment of the present invention); fd denotes the maximum amount of data that the derived file can store (corresponding to the target amount of data in the above-described embodiment of the invention). Here, the total data amount tds of the data to be exported is 1000 pieces, and the maximum data amount fd storable by the export file is 200 pieces, but specific numerical values of the total data amount tds of the data to be exported and the maximum data amount fd storable by the export file are not limited thereto.
Step 202: setting a parameter i, initially assigning the parameter i to 0, and setting a loop condition so that the value of i satisfies the condition, executing the loop method from step 203 to step 206, i =0,i < = k, i + +. Here, k = (tds/fd) =5, that is, the total number of segments k =5, which can be obtained from the above step 201. If i =0, i < k =5, the loop condition is satisfied.
Step 203: setting whether a judgment condition i is equal to k, wherein the step 205 is executed when i = k, otherwise, executing the step 204, wherein i is not equal to k because i is equal to 0,k is equal to 5, and executing the step 204.
Step 204: sublist = td (i × fd, i × fd + fd), where the dataArrayList represents a split data group (corresponding to the sub-service data in the above-described embodiment of the invention). When i =0, dataArrayList = td. Sublist (0, fd), that is, 0 th to 199 th pieces of data are selected from service data to be derived, and a first piece of sub-service data with the data size equal to 200 is obtained. And so on, each time step 204 is executed, a sub-service data with the data volume equal to 200 will be obtained.
Step 205: sublist (i × fd, tds), wherein dataArrayList represents a split data set. After a number of cycles, i =5, then i = k, step 205 will be executed. Sublist = td (5 fd, tds), that is, the 1000 th to 1000 th pieces of data are selected from the service data to be derived, and at this time, since no such data exists in the service data to be derived, no data exists in the obtained split data group. Here, it may be determined whether there is data in the split data group, and if there is no data, step 206 may be skipped to directly perform step 202, but is not limited thereto.
Step 206: the data set is passed to the export class.
After the cycle is ended, five parts of sub-service data or data groups are obtained through splitting, namely, a first part of sub-service data from 0 to 199 th data, a second part of sub-service data from 200 to 399 th data, a third part of sub-service data from 400 to 599 th data, a fourth part of sub-service data from 600 to 799 th data and a fifth part of sub-service data from 800 to 999 th data in the service data to be derived. After each sub-service data is transferred to the export class, the splitting process is finished, and the export class further processes the sub-service data, that is, the service file is generated and exported based on the sub-service data, which is not described herein again.
Optionally, splitting the service data to obtain multiple copies of sub-service data includes:
calculating the quotient of the data volume of the service data divided by the target data volume to obtain a target value;
determining a third number based on the target value, wherein the third number is equal to the target value in case the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
and averagely splitting the service data into a third number of sub-service data.
It should be noted that, in the case that the service data is split into multiple sub-service data, and the data volume of each sub-service data is less than or equal to the target data volume, to avoid the excess number of sub-service data and the excess data volume of a single sub-service data, a quotient of the data volume of the service data divided by the target data volume may be calculated first, the number of the split sub-service data is determined based on the quotient, and then the service data is split into the above number of sub-service data on average. Taking the data volume of the traffic data as 100 pieces as an example, in the case that the target data volume is 20 pieces, the data volume of the traffic data ÷ target data volume = target value, i.e., 100 ÷ 20=5, that is, the target value is an integer, the third number is 5, and therefore, the traffic data is divided into 5 pieces of sub-traffic data on average. In the case that the target data amount is 30 pieces, the data amount of the traffic data ÷ target data amount ≈ target value, i.e., 100 ÷ 30 ≈ 3.3, that is, the target value is not an integer, and thus, adding one to the target value results in the third quantity being 3+1=4. Namely, the service data is averagely divided into 4 parts of sub-service data.
In the embodiment of the invention, the quotient of the data volume of the service data divided by the target data volume is calculated, the number of the split sub-service data is determined based on the quotient, and then the service data is split into the sub-service data with the number averagely, so that the number of the split sub-service data is not excessive, and the data volume of each part of sub-service data is relatively even, thereby not only avoiding the problem that the split sub-service data occupies a large amount of resources due to the excessive number of the sub-service data, but also avoiding the problem that the data volume of a single part of sub-service data is excessive or insufficient.
Optionally, the method further comprises:
receiving a configuration request carrying a target value;
and updating the target data volume in the preset configuration information based on the target value.
It should be noted that the target value is a value generated based on user input, which is associated with the derived requirements of the user. Here, a configuration file is adopted, and the data size of the split sub-service data is adjusted by modifying preset configuration information in the configuration file according to different export requirements of users. That is, the user can dynamically set the target amount of data required to perform the export task as needed.
In the embodiment of the invention, the effect of configurable storage data volume of the export file is achieved by dynamically setting the target data volume in the preset configuration information, so that the flexibility of the execution of the export task is improved.
Optionally, deriving a service file corresponding to each sub-service data includes:
compressing the service file corresponding to each sub-service data into a compressed file;
exporting the compressed file to the local of the electronic equipment.
It should be noted that the compressed file is a file generated by compressing a general file, and the purpose of the compressed file is to reduce the storage space occupied by the file. That is, compressed files take up less storage space than ordinary files. Therefore, the service file corresponding to each sub-service data is compressed into one compressed file, and the storage space occupied by the service file can be reduced. It is understood that the compressed file can be compressed in a manner of selecting the byte output stream, but is not limited thereto. Specifically, the service file corresponding to each sub-service data is written into a byte output stream, the file in the byte output stream is compressed into a compressed file, and the compressed file is exported to the local electronic device.
In the embodiment of the invention, the service file corresponding to each sub-service data is compressed into one compressed file, so that the storage space occupied by the service file can be reduced.
Optionally, for each piece of sub-service data, generating a template file in a single target data format, storing the sub-service data in the template file, and generating a service file corresponding to the sub-service data, including:
and aiming at each part of sub-service data, creating a thread, generating a template file in a single target data format in the thread, storing the sub-service data to the template file in the thread, and generating a service file corresponding to the sub-service data.
It should be noted that a thread refers to a scheduling unit of a running program. The running program we generally refer to as a process. Multiple threads may be included in each process, may be executed in parallel, and may each perform a different task. Here, for each piece of sub-service data, a thread is created, that is, each thread performs a task of deriving the corresponding sub-service data.
In the embodiment of the invention, different threads are utilized to execute the tasks of exporting different sub-service data in parallel, so that the time required for exporting each sub-service data is reduced.
Optionally, the preset configuration information further includes: thread pool configuration information, after creating a thread for each piece of sub-service data, the method further comprising:
the thread is submitted to a thread pool created based on the thread pool configuration information.
It should be noted that thread pools are tools for managing threads. Operations that it may perform include, but are not limited to, listening for the running state of a thread, reclaiming a thread that has completed execution, and the like. It is understood that the thread pool configuration information is the basis for the server to create the thread pool, including parameter information of the thread pool, and the like. It should be noted that, here, the configuration information of the thread pool includes configuration parameters of the thread pool, and may also include creation parameters of the thread, and the like, but is not limited thereto.
In the embodiment of the invention, the threads are submitted to the thread pool created based on the thread pool configuration information, so that the management of the threads is realized, and the situation that the executed threads continuously occupy resources is avoided.
Optionally, the method further comprises:
monitoring the running state of the thread pool;
and closing the thread pool under the condition that all threads in the running state representation thread pool are executed.
It should be noted that while executing the export task, the thread pool is continuously occupying resources in the program. Therefore, in order to save resources, the server is required to monitor the running state of the thread pool, and then, when the running state indicates that all threads in the thread pool have been executed, the thread pool is closed and resources are released.
Regarding that all threads in the thread pool have been executed, it can be regarded that all export tasks have been executed, that is, all the service data to be exported have been exported to the local disk. At this time, the server does not have business data to be exported, that is, there is no export task in the server, so the thread pool is closed and the resources are released, so that the part of resources can be used for executing other tasks in the server.
In the embodiment of the invention, the execution state of the export task can be checked in time by monitoring the running state of the thread pool; and under the condition that all threads in the running state representation thread pool are completely executed, closing the thread pool and releasing resources, thereby achieving the purpose of saving resources.
Referring to fig. 3, an embodiment of the present invention provides a flowchart of steps of a method for exporting service data, where the flowchart includes:
step 301: and receiving service data.
It should be noted that, the receiving of the service data here is similar to step 101 in the embodiment shown in fig. 1: the process of obtaining the service data to be exported is similar, and is not described herein again to avoid repetition.
Step 302: and inputting configuration file information in the configuration file, wherein the configuration file information comprises the maximum storage capacity of a single export file and the configuration information of the thread pool. Step 302 is located in any time period before step 301, where the process of exporting the service data can be divided into a configuration phase and an export phase located after the configuration phase; step 302 is the operation of the configuration phase. Step 301 is a step of the derivation phase, and all steps for the derivation phase are started from step 303.
It should be noted that the maximum storage amount in the configuration file information is similar to the target data amount in the preset configuration information in the embodiment shown in fig. 1, and is not described herein again to avoid repetition.
Step 303: configuration information is obtained in the data processing class and a thread pool is created.
It should be noted that, the configuration information here is the configuration file information in the above step, a thread pool is created based on the configuration information of the thread pool in the configuration file information, and the maximum storage amount of a single export file in the configuration file information is obtained.
Step 304: whether the traffic data is empty.
It should be noted that, under the condition that the service data is empty, if the program does not receive the service data to be exported, the export task is directly ended, that is, the thread pool is closed and the resources are released, so as to avoid affecting the execution of other tasks; in case the traffic data is not empty, step 305 is executed.
Step 305: whether the traffic data is larger than the maximum storage of a single derived file.
It should be noted that if not, step 306 is executed; if yes, go to step 307.
Step 306: and transmitting the service data.
It should be noted that, here, the service data is delivered to the file export class, so as to perform data export.
Step 307: and calculating the quotient of dividing the data volume of the service data by the maximum storage volume and rounding to obtain the number of the split segments.
It should be noted that the process of calculating the number of split segments is similar to the data splitting method in the embodiment shown in fig. 2, and is not described herein again.
Step 308: the amount of derived data for each segment is calculated.
It should be noted that the amount of derived data for each segment may be the same or different. For example, the service data is split according to the maximum storage amount to obtain service data with a multi-segment derived data amount equal to the maximum storage amount; or, the service data is averagely split into multiple segments of service data according to the number of the split segments, but is not limited thereto.
Step 309: each piece of data is transferred.
It should be noted that, here, transferring each piece of data is to transfer each piece of data after splitting to a file export class, so as to export the data.
Step 310: and calling the file export class to export data.
It should be noted that the file export class is a thread class, which is used to perform data export tasks. Specifically, for received service data, an export thread is created, a file template is created in the thread, the service data is written into the file template, and the file template with the written data is exported.
Step 311: and judging whether active threads exist in the thread pool or not.
It should be noted that if yes, the step 311 is continued; otherwise, go to step 312.
Step 312: and closing the thread pool and releasing the resources.
In the embodiment of the invention, under the condition of executing the export task, firstly, business data is received, then, configuration file information is obtained, and a thread pool is established to prepare for executing the export task; then judging whether the service data is empty or not, and under the condition that the service data is not empty, judging whether the service data is larger than the maximum storage capacity of a single export file or not so as to judge whether the service data needs to be split or not; under the condition that the service data is larger than the maximum storage capacity of a single export file, calculating the number of the split segments and the data volume of each segment of the split service data, and then transmitting the service data to a file export class for data export; and finally, under the condition that no active thread exists in the thread pool, closing the thread pool and releasing resources, thereby reducing the time required for executing the export task and improving the efficiency of executing the export task.
Referring to fig. 4, an embodiment of the present invention provides an architecture diagram of a method for exporting service data, where the architecture includes:
and the service layer 41 is used for storing modules of service data. It will be appreciated that the stored service data may vary from service to service. Therefore, under the condition that an inquiry request carrying a service requirement and sent by a user through electronic equipment is received, service data meeting the service requirement is determined in a service layer based on information in the inquiry request, the service data is determined to be data to be exported, and the data to be exported is sent to data classification.
A data splitting class 42, a module for splitting the received data to be exported. That is to say, under the condition that the data to be exported sent by the service layer is received, the data to be exported is split, and the split data group is sent to the file export class for data export. It should be noted that, in the case of receiving data to be exported, the data breakdown and classification first reads configuration file information pre-stored in the server to obtain the maximum data amount storable by a single export file and parameter configuration information of the thread pool; then, a thread pool is created based on the configuration file information, and the file name and the export path of the file are defined according to different service requirements; and finally, comparing the maximum data volume which can be stored in a single export file with the total data volume of the data to be exported, judging that the data to be exported needs to be split into a plurality of sections of data groups, and sending the split data groups to a file export class for exporting the data. In addition, the data breakdown category also includes a submodule for monitoring the running state of the thread pool. The submodule checks whether an active thread exists in the thread pool every preset time, namely checks whether an unexecuted thread exists in the thread pool, if so, the submodule continues to execute the thread and continues to monitor the running state of the thread pool; and if the active thread does not exist, closing the thread pool and releasing the resources.
And a file export class 43, which is a module for exporting the received data group. That is, when the split data group sent by the data split class is received, the data of the split data group is exported to the local disk. Here, the file export class is a thread class (thread class), and when the split data groups are received, a file export class object (i.e., an export thread) is created for each data group, and the export data is imported into a construction method of the thread, and the thread is submitted using a thread pool, and a data export task is executed. In the construction method for importing export data into the thread, it can be understood that in the export thread corresponding to each group of data groups, a template is created, data in the data groups are written into the template, and finally the template with the data written is exported to the local disk. It should be noted that the template used herein is an excel template, and may also be a template in other formats, for example, but not limited to, a txt format template, etc.
It is understood that, in order to reduce the disk space occupied by the template into which data has been written, after the data is written into the template, each template may be written into the byte output stream, and the byte output stream is read, and finally the read stream information is exported to the local disk in the form of a compressed file through zipoutputstream.
A local disk 44 for receiving modules to be exported.
In the embodiment of the invention, the data to be exported is firstly acquired from the service layer, the data to be exported is split into a plurality of data groups in the data splitting and classifying, and then the split data groups are sent to the file exporting class for data export. The data to be exported is split through the pre-configured configuration file information, so that a user can dynamically configure the storage quantity of export files according to actual requirements, and the flexibility of an export program is improved; and a mode of combining multithreading and compressed flow is adopted in the file export class, so that the occupancy rate of the disk space of the exported file is further reduced on the basis of accelerating the export speed.
Referring to fig. 5, an embodiment of the present invention provides an apparatus for exporting service data, where the apparatus includes:
an obtaining module 51, configured to obtain service data to be exported;
the splitting module 52 is configured to split the service data to obtain multiple sub-service data when the data amount of the service data is greater than a target data amount in the preset configuration information, where the data amount of each sub-service data is less than or equal to the target data amount;
the service file module 53 is configured to generate a template file in a single target data format for each piece of sub-service data, store the sub-service data in the template file, and generate a service file corresponding to the sub-service data;
and the exporting module 54 is configured to export the service file corresponding to each sub-service data.
Optionally, the splitting module 52 includes:
the first calculation unit is used for rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
the first splitting unit is configured to split the service data according to the target data volume to obtain a second number of sub-service data, where the second number is equal to the first number when the data volume of the service data is an integer multiple of the target data volume, and the second number is equal to a sum of the first number and one when the data volume of the service data is not an integer multiple of the target data volume.
Optionally, the splitting module 52 includes:
the second calculation unit is used for calculating the quotient of the data volume of the service data divided by the target data volume to obtain a target value;
a third calculation unit configured to determine a third number based on the target value, wherein the third number is equal to the target value in a case where the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
and the second splitting unit is used for averagely splitting the service data into a third number of sub-service data.
Optionally, the apparatus further comprises:
the receiving module is used for receiving a configuration request carrying a target numerical value;
and the configuration module is used for updating the target data volume in the preset configuration information based on the target value.
Optionally, the deriving module 54 includes:
the compression unit is used for compressing the service file corresponding to each sub-service data into a compressed file;
and the exporting unit is used for exporting the compressed file to the local part of the electronic equipment.
Optionally, the service file module 53 is specifically configured to create a thread for each piece of sub-service data, generate a template file in a single target data format in the thread, store the sub-service data in the template file in the thread, and generate a service file corresponding to the sub-service data.
Optionally, the preset configuration information further includes: the thread pool configuration information, the apparatus further comprises:
and the submitting module is used for submitting the thread to the thread pool created based on the thread pool configuration information.
Optionally, the apparatus further comprises:
the monitoring module is used for monitoring the running state of the thread pool;
and the closing module is used for closing the thread pool under the condition that all threads in the running state representation thread pool are executed completely.
In the embodiment of the present invention, after the service data to be exported is obtained, the data volume of the service data is measured, so that the service data is split to obtain multiple sub-service data under the condition that the data volume of the service data is greater than the target data volume in the preset configuration information, so that the data volume of each sub-service data does not exceed the target data volume. And then generating a template file in a single target data format for each part of sub-service data, storing the sub-service data in the template file, and generating a service file corresponding to the sub-service data. Therefore, under the condition that the data volume of the service data is overlarge, the data in the service file can be loaded and displayed to the user in a short time, so that the waiting time of the user is shortened, and the use experience of the user is improved.
On the other hand, an embodiment of the present application further provides an electronic device, which includes a processor, a memory, and a computer program that is stored in the memory and is executable on the processor, and when the processor executes the program, the method for exporting the service data provided by the embodiments of the present invention is implemented.
In still another aspect, the present invention further provides a readable storage medium, and when instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for exporting service data, provided by the above embodiments of the present invention.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (16)
1. A method for exporting service data, the method comprising:
acquiring service data to be exported;
under the condition that the data volume of the service data is larger than the target data volume in preset configuration information, splitting the service data to obtain multiple sub-service data, wherein the data volume of each sub-service data is smaller than or equal to the target data volume;
generating a template file in a single target data format for each sub-service data, storing the sub-service data into the template file, and generating a service file corresponding to the sub-service data;
and exporting the service file corresponding to each sub-service data.
2. The method according to claim 1, wherein the splitting the service data to obtain multiple copies of sub-service data includes:
rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
and splitting the service data according to the target data volume to obtain a second number of sub-service data, wherein the second number is equal to the first number when the data volume of the service data is an integral multiple of the target data volume, and the second number is equal to the sum of the first number and one when the data volume of the service data is not the integral multiple of the target data volume.
3. The method of claim 1, wherein the splitting the service data to obtain multiple copies of sub-service data comprises:
calculating a quotient of dividing the data volume of the service data by the target data volume to obtain a target value;
determining a third number based on the target value, wherein the third number is equal to the target value if the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
and averagely splitting the service data into the third number of sub-service data.
4. The method of claim 1, further comprising:
receiving a configuration request carrying a target value;
and updating the target data volume in the preset configuration information based on the target value.
5. The method according to claim 1, wherein said deriving the service file corresponding to each of the sub-service data comprises:
compressing the service file corresponding to each sub-service data into a compressed file;
exporting the compressed file to the local of the electronic equipment.
6. The method according to claim 1, wherein the generating a template file in a single target data format for each piece of the sub-service data, storing the sub-service data in the template file, and generating a service file corresponding to the sub-service data comprises:
and creating a thread aiming at each sub-service data, generating a template file in a single target data format in the thread, storing the sub-service data to the template file in the thread, and generating a service file corresponding to the sub-service data.
7. The method of claim 6, wherein the preset configuration information further comprises: thread pool configuration information, after said creating a thread for each of said sub-service data, said method further comprising:
submitting the thread to a thread pool created based on the thread pool configuration information.
8. The method of claim 7, further comprising:
monitoring the running state of the thread pool;
and closing the thread pool under the condition that the running state represents that all threads in the thread pool are executed completely.
9. An apparatus for deriving service data, the apparatus comprising:
the acquisition module is used for acquiring the service data to be exported;
the splitting module is used for splitting the service data to obtain multiple sub-service data under the condition that the data volume of the service data is larger than the target data volume in preset configuration information, wherein the data volume of each sub-service data is smaller than or equal to the target data volume;
the business file module is used for generating a template file in a single target data format aiming at each sub-business data, storing the sub-business data into the template file and generating a business file corresponding to the sub-business data;
and the export module is used for exporting the service file corresponding to each sub-service data.
10. The apparatus of claim 9, wherein the splitting module comprises:
the first calculation unit is used for rounding a quotient obtained by dividing the data volume of the service data by the target data volume to obtain a first quantity;
a first splitting unit, configured to split the service data according to the target data volume to obtain a second number of sub-service data, where the second number is equal to the first number when the data volume of the service data is an integer multiple of the target data volume, and the second number is equal to a sum of the first number and one when the data volume of the service data is not an integer multiple of the target data volume.
11. The apparatus of claim 9, wherein the splitting module comprises:
the second calculating unit is used for calculating the quotient of dividing the data volume of the service data by the target data volume to obtain a target value;
a third calculation unit configured to determine a third number based on the target value, wherein the third number is equal to the target value when the target value is an integer; in the case where the target value is not an integer, the third number is equal to the sum of the result of rounding the target value plus one;
a second splitting unit, configured to split the service data into the third number of sub-service data on average.
12. The apparatus of claim 9, further comprising:
the receiving module is used for receiving a configuration request carrying a target numerical value;
and the configuration module is used for updating the target data volume in the preset configuration information based on the target value.
13. The apparatus of claim 9, wherein the derivation module comprises:
a compressing unit, configured to compress the service file corresponding to each sub-service data into a compressed file;
and the exporting unit is used for exporting the compressed file to the local electronic equipment.
14. The apparatus according to claim 9, wherein the service file module is specifically configured to: and creating a thread aiming at each sub-service data, generating a template file in a single target data format in the thread, storing the sub-service data to the template file in the thread, and generating a service file corresponding to the sub-service data.
15. An electronic device, comprising: processor, memory and computer program stored on the memory and executable on the processor, which when executed by the processor implements a method of deriving traffic data according to any of claims 1-8.
16. A readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method of deriving traffic data according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744002.6A CN115168302A (en) | 2022-06-28 | 2022-06-28 | Business data export method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744002.6A CN115168302A (en) | 2022-06-28 | 2022-06-28 | Business data export method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115168302A true CN115168302A (en) | 2022-10-11 |
Family
ID=83489546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210744002.6A Pending CN115168302A (en) | 2022-06-28 | 2022-06-28 | Business data export method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115168302A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991933A (en) * | 2023-09-28 | 2023-11-03 | 北京谷器数据科技有限公司 | Offline export method and system of system data |
-
2022
- 2022-06-28 CN CN202210744002.6A patent/CN115168302A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991933A (en) * | 2023-09-28 | 2023-11-03 | 北京谷器数据科技有限公司 | Offline export method and system of system data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR20220079958A (en) | Blockchain message processing method and device, computer and readable storage medium | |
| CN106548262B (en) | Scheduling method, device and system for resources for processing tasks | |
| CN107241281B (en) | Data processing method and device | |
| CN106445675B (en) | B2B platform distributed application scheduling and resource allocation method | |
| CN108965884B (en) | Distribution method of transcoding tasks, scheduling device and transcoding device | |
| CN110347515B (en) | Resource optimization allocation method suitable for edge computing environment | |
| CN110231992A (en) | Distributed resource allocation method, device, equipment and readable storage medium storing program for executing | |
| WO2019096046A1 (en) | Data processing system and method, and token management method | |
| WO2021159831A1 (en) | Programming platform user code running method, platform, node, device and medium | |
| CN111435354A (en) | Data export method and device, storage medium and electronic equipment | |
| CN114155026A (en) | Resource allocation method, device, server and storage medium | |
| CN115168302A (en) | Business data export method and device and electronic equipment | |
| CN111586140A (en) | Data interaction method and server | |
| CN101667139A (en) | Service program invocation method based on aggregate type object, device and system therefor | |
| WO2021237630A1 (en) | Multi-key-value command processing method and apparatus, and electronic device and storage medium | |
| CN108023938B (en) | Message sending method and server | |
| CN106534247B (en) | The method and apparatus of list downloading | |
| CN103299298A (en) | Service processing method and system | |
| CN116226178A (en) | Data query method and device, storage medium and electronic device | |
| CN110109865A (en) | A kind of date storage method, device, equipment and readable storage medium storing program for executing | |
| CN112286623A (en) | Information processing method and device and storage medium | |
| CN112433855A (en) | Resource allocation method, device and server | |
| CN113342526A (en) | Dynamic management and control method, system, terminal and medium for cloud computing mobile network resources | |
| CN112995704B (en) | Cache management method and device, electronic equipment and storage medium | |
| CN113254476B (en) | Request processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |