CN114090524A

CN114090524A - Excel file distributed exporting method and device

Info

Publication number: CN114090524A
Application number: CN202110019654.9A
Authority: CN
Inventors: 胡海龙
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-01-07
Filing date: 2021-01-07
Publication date: 2022-02-25

Abstract

The invention discloses a distributed export method and device for Excel files, and relates to the technical field of computers. One embodiment of the method comprises: splitting an export total task of an Excel file to be exported into a plurality of main tasks, and identifying task attributes of each main task; processing a plurality of primary tasks using a server cluster comprising a plurality of servers such that each server processes any one primary task; and restoring the position of the main task in the Excel file according to the task attribute of each main task, so that the processed main tasks are combined to generate the Excel file. The embodiment reduces the processing time for exporting the Excel file with a large amount of data, improves the processing efficiency and stability and reduces the processing cost.

Description

Excel file distributed exporting method and device

Technical Field

The invention relates to the technical field of computers, in particular to a distributed export method and device for Excel files.

Background

In the prior art, an Excel file is exported through a single server, when the Excel file comprises a plurality of sheet (worksheet) pages, the export efficiency by using one server is usually low, and when each sheet page comprises a large amount of data, the export takes a large amount of time, and server failure can be caused.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for exporting an Excel file in a distributed manner, which can greatly improve the export efficiency of an Excel file including a plurality of sheet pages with a large amount of data, and significantly shorten the export time.

In order to achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a method for distributed export of an Excel file, including:

splitting an export total task of an Excel file to be exported into a plurality of main tasks, and identifying task attributes of each main task;

processing the plurality of primary tasks using a server cluster comprising a plurality of servers such that each of the servers processes any one of the primary tasks; and

and restoring the position of the main task in the Excel file according to the task attribute of each main task, so that the plurality of processed main tasks are combined to generate the Excel file.

According to an aspect of an embodiment of the present invention, preferably, wherein,

splitting an export total task of an Excel file to be exported into a plurality of main tasks specifically comprises the following steps: splitting the export total task of the Excel file into a plurality of main tasks respectively corresponding to a plurality of sheet pages included in the Excel file, and

the task attribute comprises a task ID, and the task ID is an index of the sheet page corresponding to the main task in the Excel file.

splitting an export total task of the Excel file to be exported into a plurality of main tasks, wherein the process further comprises the following steps:

splitting any one or more of the main tasks into a plurality of subtasks, respectively, an

And identifying the subtask attribute of each subtask, so that the position of the subtask in the corresponding main task can be restored according to the subtask attribute.

the main task attribute comprises the number of subtasks and the state of the subtasks; and is

The subtask attribute includes a task ID, a start data ID, and an end data ID of the main task to which the subtask attribute belongs.

and calling the server cluster according to the total number of the main tasks and the subtasks, so that each server processes any one main task or any one subtask.

and merging the plurality of subtasks and the plurality of main tasks according to the subtask attribute of each subtask and the task attribute of each main task to generate the Excel file.

and combining the subtasks belonging to one main task according to the subtask attributes of the subtasks to generate an Excel subfile, and combining a plurality of Excel subfiles according to the task attributes of the main task to generate the Excel file.

According to another aspect of the embodiments of the present invention, there is provided an Excel file distributed export apparatus, including:

the task splitting unit is used for splitting a total export task of the Excel file to be exported into a plurality of main tasks and identifying task attributes of each main task;

a task processing unit that processes the plurality of main tasks using a server cluster including a plurality of servers such that each of the servers processes any one of the main tasks; and

and the data merging unit restores the positions of the main tasks in the Excel file according to the task attributes of the main tasks, so that the processed main tasks are merged to generate the Excel file.

According to another aspect of embodiments of the present invention, preferably, wherein,

the task splitting unit splits the export total task of the Excel file into a plurality of main tasks respectively corresponding to a plurality of sheet pages included in the Excel file, and

the task splitting unit splits any more than one main task into a plurality of subtasks respectively, and identifies the subtask attribute of each subtask, so that the position of the subtask in the corresponding main task can be restored according to the subtask attribute.

the main task attribute comprises the number of subtasks and the state of the subtasks; and the subtask attribute includes a task ID, a start data ID, and an end data ID of the main task to which it belongs.

and the task processing unit calls the server cluster according to the total number of the main tasks and the subtasks, so that each server processes any one main task or any one subtask.

and the data merging unit performs merging processing on the plurality of subtasks and the plurality of main tasks according to the task attributes of each subtask and each main task to generate the Excel file.

the data merging unit merges the subtasks belonging to one main task according to the subtask attributes of the subtasks to generate an Excel subfile, and merges a plurality of Excel subfiles according to the task attributes of the main task to generate the Excel file.

According to another aspect of the embodiment of the invention, an electronic device for distributed export of an Excel file is provided, which is characterized by comprising:

one or more processors; and

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a method according to any one of aspects of embodiments of the invention.

According to a further aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, characterized in that the program, when executed by a processor, implements a method according to any one of the aspects of embodiments of the present invention.

One embodiment of the above invention has the following advantages or benefits: the method has the advantages that the Excel file to be exported is split into a plurality of tasks, including a main task split according to a sheet page and sub-tasks obtained by further splitting the main task, and the technical means of processing the sub-tasks in a distributed mode through a server cluster are utilized, so that the technical problems of long export time and low export efficiency when the export data size is large are solved, and a plurality of stateless tasks are obtained by identifying a plurality of main tasks and sub-tasks and are distributed to any server for processing, so that the stability and reliability of the export process are ensured, the technical effects of saving processing time, improving processing efficiency and stability and reducing processing cost are achieved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a main flow of an Excel file distributed export method according to an embodiment of the present invention;

FIG. 2 shows an example of a partial split result of an Excel file;

FIG. 3 is a schematic diagram of the main modules of an Excel file distributed export apparatus according to an embodiment of the present invention;

FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a main flow of an Excel file distributed export method according to an embodiment of the present invention, and as shown in fig. 1, the Excel file distributed export method according to an embodiment of the present invention mainly includes: step S101, splitting an export total task of an Excel file to be exported into a plurality of main tasks, and identifying task attributes of each main task; step S102, processing the main tasks by using a server cluster comprising a plurality of servers, so that each server processes any one main task; and step S103, restoring the position of the main task in the Excel file according to the task attribute of each main task, so as to merge the plurality of processed main tasks to generate the Excel file.

The individual steps are explained in detail below with reference to fig. 1-2.

Step S101: task splitting

The method provided by the embodiment of the invention supports the multi-dimensional and multi-granularity splitting of the data of the Excel file. The multi-dimension means that the Excel file can be split according to different attributes, for example, one Excel file is split into a plurality of main tasks according to the number and types of sheet pages; the multi-granularity means that the Excel file can be split according to different data volumes as required.

When the task is split, the total number of main tasks needing to be split is calculated at first, wherein the total number of the main tasks can be the number of sheet pages contained in an Excel file to be exported. The primary tasks are then split according to their total number and primary task data is generated. It should be noted here that the attribute of the main task needs to be set, for example, the index of the sheet page to which each main task belongs, that is, the index of the sheet page, needs to be clearly identified, so that the specific position of the main task in the Excel file can be restored according to the attribute of the main task, and thus any one main task can be allocated to any one server for processing without affecting subsequent data merging. In other words, by identifying attributes of the primary tasks, each primary task is enabled to be stateless.

When the data volume of the main task is large, the main task can be continuously split into a plurality of subtasks. Here too, attributes identifying each subtask, such as task ID, task start data ID, task end data ID, and an index of the sheet page of the main task to which it belongs, are required to further identify the position of the atomic task in the main task to which it belongs, depending on the attributes of the subtask. By identifying such key attributes for each subtask, the subtasks can also be stateless, so that any one subtask can be allocated to any one server for processing without affecting data merging of the subtasks.

In addition, by statelessly assigning the main task and the subtask to any server for execution as described above, when a certain task fails to be executed, any server can be automatically replaced to re-execute the task, thereby ensuring high usability of the system, ensuring each task to be successfully executed, and ensuring stable and reliable data export.

For the task start data ID and the task end data ID, for example, the split main task includes 100 pieces of data and is further split into 3 sub-tasks, each of which may range from 1 to 33, 34 to 66, and 67 to 100, respectively, where "1" and "33" may be the task start data ID and the task end data ID of the first sub-task. It is also possible to mark with time, such as a start time and an end time, if such an ID is not satisfactory in actual operation. That is, the task start data ID and the task end data ID are set as necessary.

Note that the attributes of the main task may also include, for example, user information, number of subtasks, subtask status, source of request, etc., as may be split into one or more subtasks. In addition, the attributes of the subtasks may also include data query conditions, and the like.

The subtasks can also be broken up into secondary subtasks and further broken up of the secondary subtasks is also possible. And taking the size of the subtask with the minimum unit as a split fine granularity, and splitting the Excel file by setting the split fine granularity according to needs to obtain a proper number of subtasks with proper size for processing. The fine splitting granularity can be set artificially according to actual needs, and can also be set as specific data comprehensively evaluated according to the performance of the server.

Splitting results as shown in fig. 2, fig. 2 shows an example of partial splitting results of an Excel file, wherein the Excel file is split into N main tasks according to the number of sheet pages, such as sheet2-1 of sheet2, each main task is further split into a plurality of subtasks, such as sheet2-1 subtask 1 to sheet2-1 subtask N, and wherein two subtasks sheet1-1 and sheet1-2 of sheet1 are further split into a plurality of secondary subtasks, such as sheet1-1 subtask 1 to sheet1-1 subtask N.

Step S102: task processing

And after the task splitting is completed, distributing all split tasks to a rear-end server cluster for processing. The server cluster is a set of a plurality of servers which process the same service together, can process subtasks in a large scale, and can ensure the processing efficiency of the tasks even if the task amount is huge. In the embodiment of the invention, each server can process any one subtask because the subtasks are stateless, and if the number of the servers is not enough, the dynamic adjustment can be carried out, namely the servers are called according to the total number of the main tasks and the subtasks, so that theoretically, how many servers can be called for simultaneous processing by how many tasks to be processed.

Step S103: data merging

After the sub-task processing is completed, the later-described data merging unit receives a message sent by the later-described task processing unit in real time and performs data merging processing. Specifically, the data merging unit reads the attribute of each subtask, obtains the main task to which the subtask belongs and the attribute of the main task, and calculates whether the returned subtasks of the main task are equal to the total number of the subtasks thereof, when the number of the returned subtasks of one main task is equal to the total number of the subtasks of the main task, all the subtasks of the main task are processed and completed, and when the number of the returned subtasks is equal to the total number of all the subtasks of all the main tasks, all the subtasks are processed and completed.

And then, combining the subtasks of the same main task according to the attributes of the subtasks to generate Excel subfiles corresponding to the same sheet page, and after the combination of the subtasks under all the sheet pages corresponding to the main task is completed, combining the Excel subfiles according to the attributes of the main task, and finally generating a complete Excel file.

And finally, uploading the finally generated Excel file to a cloud storage to support a user to download.

The Excel file distributed export method according to one embodiment of the invention is described above. The present invention is not limited to the above-described embodiments. For example, in the above embodiment, an example of splitting an Excel file according to a sheet page and a split fine granularity is described, but the present invention is not limited thereto. For example, an Excel file can be split using big data technology.

An Excel file distributed export apparatus 200 according to an embodiment of the present invention for executing the Excel file distributed export method according to an embodiment of the present invention is described below with reference to fig. 3. The Excel file distributed export apparatus 200 according to the embodiment of the present invention includes:

a task splitting unit 201, where the task splitting unit 201 executes the above-mentioned task splitting process, that is, step S101, splits an export total task of an Excel file to be exported into a plurality of main tasks, and identifies a task attribute of each main task;

a task processing unit 202, the task processing unit 202 performing the above-described process of task processing, that is, step S102, processing the plurality of main tasks using a server cluster including a plurality of servers such that each of the servers processes any one of the main tasks; and

the data merging unit 203 executes the data merging process described above, that is, step S103, and restores the position of the main task in the Excel file according to the task attribute of each main task, so as to merge the processed main tasks to generate the Excel file.

The Excel file distributed export method and the Excel file distributed export device according to the embodiments of the present invention are described above. By using the Excel file distributed export method and the Excel file distributed export device according to the embodiment of the invention, the Excel file to be exported is divided into a plurality of tasks, including the main task divided according to the sheet page and the subtasks obtained by further dividing the main task, and the technical means of server cluster distributed processing of the subtasks is utilized, so that the technical problems of long export time and low export efficiency when the export data amount is large are solved, and a plurality of stateless tasks are obtained by identifying the key attributes of the main tasks and the subtasks, so that any task can be allocated to any server for processing, the export process is stable and reliable, and the technical effects of saving the processing time, improving the processing efficiency and stability and reducing the processing cost are achieved.

Fig. 4 shows an exemplary system architecture 400 of an Excel file distributed export method or an Excel file distributed export apparatus to which an embodiment of the present invention may be applied.

As shown in fig. 4, the system architecture 400 may include

terminal devices

401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the

terminal devices

401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use

terminal devices

401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The

terminal devices

401, 402, 403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 405 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

401, 402, 403. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the Excel file distributed export method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the Excel file distributed export apparatus is generally disposed in the server 405.

It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a task splitting unit, a task processing unit, and a data merging unit. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: splitting an export total task of an Excel file to be exported into a plurality of main tasks, and identifying task attributes of each main task; processing the plurality of primary tasks using a server cluster comprising a plurality of servers such that each of the servers processes any one of the primary tasks; and restoring the position of the main task in the Excel file according to the task attribute of each main task, so that the processed main tasks are combined to generate the Excel file.

According to the technical scheme of the embodiment of the invention, the Excel file to be exported is divided into a plurality of tasks, including the main task which is divided according to the sheet page and the subtasks which are obtained by further dividing the main task, and the technical means of processing the subtasks in a distributed manner by using the server cluster is utilized, so that the technical problems of long export time and low export efficiency when the export data amount is large are solved, and a plurality of stateless tasks can be obtained by identifying the key attributes of the main tasks and the subtasks, so that any task can be allocated to any server for processing, the export process is ensured to be stable and reliable, and the technical effects of saving the processing time, improving the processing efficiency and stability and reducing the processing cost are further achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed export method of Excel files is characterized by comprising the following steps:

2. The method of claim 1, wherein,

3. The method of claim 1, wherein,

splitting an export total task of an Excel file to be exported into a plurality of main tasks specifically comprises the following steps:

4. The method of claim 2, wherein,

5. The method of claim 3 or 4, wherein,

6. The method of claim 3 or 4, wherein,

7. The method of claim 3 or 4, wherein,

8. The method of claim 7, wherein,

9. An Excel file distributed exporting device is characterized by comprising:

10. An electronic device for distributed export of Excel files, comprising:

one or more processors; and

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.