CN113722321A - Data export method and device and electronic equipment - Google Patents
Data export method and device and electronic equipment Download PDFInfo
- Publication number
- CN113722321A CN113722321A CN202110966592.2A CN202110966592A CN113722321A CN 113722321 A CN113722321 A CN 113722321A CN 202110966592 A CN202110966592 A CN 202110966592A CN 113722321 A CN113722321 A CN 113722321A
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- batches
- exported
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the specification discloses a data export method, a data export device and electronic equipment. The method comprises the following steps: acquiring a data export request; constructing a query operation instruction according to the data export request; and circularly executing the following steps until the stop condition is met: reading data to be exported in batches from a data table according to the query operation instruction; and writing the data read in batches into a data file with a preset file type. The embodiment of the specification can improve the user experience in data export.
Description
Technical Field
The embodiment of the specification relates to the technical field of computers, in particular to a data export method, a data export device and electronic equipment.
Background
To facilitate the circulation of data, it is often necessary to derive the data from a data table. For example, a part or all of the data in the data table is exported as an Excel file. When a scene with small exported data volume and simple service is faced, the existing data export method can meet most data export requirements. However, when a scene with a large amount of data to be exported is faced, the performance of the device may be reduced, and even errors such as memory overflow may occur, which may reduce user experience.
Disclosure of Invention
The embodiment of the specification provides a data export method, a data export device and electronic equipment, and improves user experience during data export.
In a first aspect of embodiments of the present specification, there is provided a data export method, including:
acquiring a data export request;
constructing a query operation instruction according to the data export request;
and circularly executing the following steps until the stop condition is met:
reading data to be exported in batches from a data table according to the query operation instruction;
and writing the data read in batches into a data file with a preset file type.
In a second aspect of embodiments of the present specification, there is provided a data derivation apparatus, including:
an acquisition unit configured to acquire a data export request;
the construction unit is used for constructing a query operation instruction according to the data export request;
a training unit for executing the following steps in a loop until a stop condition is satisfied:
reading data to be exported in batches from a data table according to the query operation instruction;
and writing the data read in batches into a data file with a preset file type.
In a third aspect of embodiments of the present specification, there is provided an electronic apparatus including:
at least one processor;
a memory storing program instructions configured to be suitable for execution by the at least one processor, the program instructions comprising instructions for performing the method of the first aspect.
According to the technical scheme provided by the embodiment of the specification, the data to be exported can be read in batches from the data table; the data read in batches may be written to a data file having a preset file type. Therefore, by reading in batches and writing in batches, the problem that when the quantity of exported data is large, the performance of equipment is reduced due to the fact that a large amount of data is read at one time is avoided, and user experience in data exporting is improved.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a data export method in an embodiment of the present disclosure;
FIG. 2 is a diagram illustrating a data export process in an embodiment of the present disclosure;
FIG. 3 is a schematic structural diagram of a data export apparatus in an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an electronic device in an embodiment of this specification.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step should fall within the scope of protection of the present specification.
In a conventional data export method, when the export data is read from the data table, all the export data is generally read at once. Thus, when the amount of the derived data is large, the performance of the device may be degraded, and even an error such as memory overflow may occur.
The embodiment of the specification provides a data export method. The data export method may be applied to a server. The server may be one server, or may be a server cluster including a plurality of servers.
Please refer to fig. 1 and fig. 2. The data derivation method may include the following steps.
Step S11: a data export request is obtained.
In some embodiments, the data export request may be to instruct a server to export data. The data export request may be generated by the server upon receiving a data export instruction. In particular, a user may enter data export instructions in the server. The server may generate a corresponding data export request after receiving the data export instruction. Alternatively, the data export request may also be sent by the terminal device. Specifically, the terminal device may send a data export request to the server. The server can receive a data export request sent by the terminal device. The terminal device may include a personal computer, a notebook computer, a smart phone, a tablet computer, a portable wearable device, and the like.
In some embodiments, the data export request may carry a data table identifier and a query condition. The data table identification is used for identifying a data table. The data table identification may include a name, number, etc. of the data table. The query may be a constraint. The server-derived data may satisfy the query condition. For example, the query condition may be that the validation date is earlier than 20210619. As another example, the query condition may be transaction data of Alice in the past year.
Step S13: and constructing a query operation instruction according to the data export request.
In some embodiments, the query operation instructions may be for querying derived data. For the SQL field, the query operation instruction may be an SQL statement. For example, the SQL statement may be as follows:
SELECT
the name of the field is 1, and,
the name of the field 2 is set to,
the name of the field 3 is set to,
…
FROM
name of data table
WHERE
Query conditions
ORDER BY
Field name 1, field name 2 …
Of course, for other fields, the query operation instruction may be in other forms.
In some embodiments, the server may parse the data export request to obtain a data table identifier and a query condition; it may be detected whether there is data table configuration information corresponding to the data table identification. If the data export request does not exist, data export cannot be achieved, and the server can ignore the data export request. The ignoring process may include discarding the data export request, deleting the data export request, and so forth. Of course, in some cases, the data table corresponding to the data table identifier may be a newly created data table, so that there is no data table configuration information corresponding to the data table identifier. To this end, the user may enter data table configuration information corresponding to the data table identification in the server. The server may receive data table configuration information corresponding to the data table identification; data table configuration information corresponding to the data table identification may be stored. Thus, after receiving a data export request next time, the server can respond to the data export request. If the query operation instruction exists, the server can construct the query operation instruction according to the configuration information and the query condition of the data table. For example, the server may determine fields associated with the query condition from data table configuration information, and may assemble an SQL statement from the fields associated with the query condition.
Wherein, the data table configuration information may be derived configuration information of the data table. The deriving configuration information may include: the data table contains fields, the identification of the fields, the meaning of the fields, the type of data in the fields, and the like. The data table configuration information may be located in a table, a text document, or other type of file.
Step S15: and circularly executing the following steps until the stop condition is met: and reading the data to be derived in batches from the data table according to the query operation instruction, and writing the data read in batches into a data file with a preset file type.
In some embodiments, the data to be exported may include oplog data, operational parameter data, business data, and the like. The data to be exported may be located in one or more data tables. The server may read the data to be exported in batches from one or more data tables. The data to be exported may include sub-data for one or more fields in the data table. The data file with the preset file type may include an excel file, a word file, and the like.
In some embodiments, the server may determine the amount of data to be exported according to the query operation instruction. When the number of the data to be exported is larger than the preset value, the number of the data to be exported is too large and exceeds the upper limit, and the server can ignore the data export request. Of course, when the number of the data to be exported is greater than the preset value, the server may also send a prompt message for prompting that the number of the data to be exported exceeds the upper limit. When the number of data to be exported is less than or equal to a preset value, the server may perform the following steps in a loop: and reading the data to be derived in batches from the data table according to the query operation instruction, and writing the data read in batches into a data file with a preset file type. The preset value may be preset according to the performance of the electronic device (e.g., a memory, a CPU, etc.).
In some embodiments, the preset condition may be: and finishing reading the data to be exported. Or, the server may determine the amount of data to be exported according to the query operation instruction; the amount of data read per batch can be determined; the maximum number of cycles may be calculated based on the amount of data to be exported and the amount of data read per batch. Thus, the preset condition may be: the number of cycles reaches the maximum number of cycles. Wherein the amount of data read per batch may be preset. For example, the amount of data read per batch may be set in the data table configuration information in advance. The server may obtain the data amount read for each batch from the data table configuration information. As another example, the user may also enter the amount of data read per batch. The server may receive the user input of the amount of data read per batch.
In some embodiments, the server may read the data to be exported from the data table in batches; the data read from the sub-batches can be subjected to escape processing; the escaped data can be written to a data file having a preset file type. For example, data "1" may be turned to "yes" and data "0" may be turned to "no". Through the escape processing, the readability of the data file can be improved, and the user experience is improved. The data to be exported may include subdata of one or more fields in the data table. The server may perform escaping processing on one or more sub-data. Specific escape rules may be located in the data table configuration information. The server can acquire escape rules from the data table configuration information; the data read from the sub-batches can be subjected to escaping processing according to escaping rules, and the escaped data can be written into a data file with a preset file type.
In the data export method in the embodiment of the present specification, data to be exported may be read in batches from a data table; the data read in batches may be written to a data file having a preset file type. Therefore, by reading in batches and writing in batches, the problem that when the quantity of exported data is large, the performance of equipment is reduced due to the fact that a large amount of data is read at one time is avoided, and user experience in data exporting is improved.
Please refer to fig. 3. This specification provides one embodiment of a data export apparatus.
The data deriving means may comprise the following modular units.
An acquisition unit 31 configured to acquire a data export request;
a construction unit 33, configured to construct a query operation instruction according to the data export request;
a circulation unit 35 for circulating the following steps until the stop condition is satisfied:
reading data to be exported in batches from a data table according to the query operation instruction;
and writing the data read in batches into a data file with a preset file type.
An embodiment of an electronic device of the present description is described below. Fig. 4 is a schematic diagram of a hardware configuration of the electronic apparatus in this embodiment. As shown in fig. 4, the electronic device may include one or more processors (only one of which is shown), memory, and a transmission module. Of course, it is understood by those skilled in the art that the hardware structure shown in fig. 4 is only an illustration, and does not limit the hardware structure of the electronic device. In practice the electronic device may also comprise more or fewer component elements than those shown in fig. 4; or have a different configuration than that shown in figure 4.
The memory may comprise high speed random access memory; alternatively, non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory may also be included. Of course, the memory may also comprise a remotely located network memory. The memory may be used to store program instructions or modules of application software, such as the program instructions or modules of the corresponding embodiments of fig. 1 of this specification.
The processor may be implemented in any suitable way. For example, the processor may take the form of, for example, a microprocessor or processor and a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, an embedded microcontroller, and so forth. The processor may read and execute the program instructions or modules in the memory.
The transmission module may be used for data transmission via a network, for example via a network such as the internet, an intranet, a local area network, a mobile communication network, etc.
This specification also provides one embodiment of a computer storage medium. The computer storage medium includes, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Cache (Cache), a Hard Disk (HDD), a Memory Card (Memory Card), and the like. The computer storage medium stores computer program instructions. The computer program instructions when executed implement: the program instructions or modules of the embodiments corresponding to fig. 1 in this description.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and the same or similar parts in each embodiment may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, apparatus embodiments, electronic device embodiments, and computer storage medium embodiments are substantially similar to method embodiments and therefore are described with relative ease, where reference may be made to some descriptions of method embodiments. In addition, it is understood that one skilled in the art, after reading this specification document, may conceive of any combination of some or all of the embodiments listed in this specification without the need for inventive faculty, which combinations are also within the scope of the disclosure and protection of this specification.
It will be understood by those skilled in the art that the "first stage" and "rollback stage" are just one nomenclature used for convenience of description, and any implementation having other names but whose functions are equivalent to those in the embodiments of the present specification is within the scope of the present application.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
From the above description of the embodiments, it is clear to those skilled in the art that the present specification can be implemented by software plus a necessary general hardware platform. Based on such understanding, the technical solutions of the present specification may be essentially or partially implemented in the form of software products, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present specification.
The description is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
While the specification has been described with examples, those skilled in the art will appreciate that there are numerous variations and permutations of the specification that do not depart from the spirit of the specification, and it is intended that the appended claims include such variations and modifications that do not depart from the spirit of the specification.
Claims (15)
1. A method of data export, comprising:
acquiring a data export request;
constructing a query operation instruction according to the data export request;
and circularly executing the following steps until the stop condition is met:
reading data to be exported in batches from a data table according to the query operation instruction;
and writing the data read in batches into a data file with a preset file type.
2. The method of claim 1, the constructing query operation instructions comprising:
analyzing the data export request to obtain a data table identifier and a query condition;
detecting whether data table configuration information corresponding to the data table identification exists or not;
if yes, constructing a query operation instruction according to the configuration information and the query condition of the data table.
3. The method of claim 2, the query operation instructions comprising SQL statements.
4. The method of claim 2, further comprising:
and if not, performing ignoring processing on the data export request.
5. The method of claim 3, further comprising:
receiving input data table configuration information corresponding to the data table identification;
and storing the data table configuration information corresponding to the data table identification.
6. The method of claim 1, said loop performing steps comprising:
determining the quantity of data to be exported according to the query operation instruction;
and when the quantity of the data to be exported is less than or equal to a preset value, circularly executing the following steps.
7. The method of claim 6, further comprising:
and when the quantity of the data to be exported is greater than a preset value, ignoring the data export request.
8. The method of claim 1, further comprising:
determining the quantity of data to be exported according to the query operation instruction;
determining the data amount read in each batch;
calculating the maximum cycle number according to the number of data to be exported and the data amount read in each batch;
the stop condition includes: the number of cycles reaches the maximum number of cycles.
9. The method of claim 1, further comprising:
determining the data amount read in each batch;
the reading of the data to be exported in batches from the data table comprises:
and reading the data to be exported in batches from the data table according to the data amount read in each batch.
10. The method according to claim 8 or 9, wherein the data export request carries a data table identifier;
the determining the data amount read in each batch comprises the following steps:
and acquiring the data volume read in each batch from the data table configuration information corresponding to the data table identification.
11. The method of claim 1, wherein writing the data read in batches to a data file having a preset file type comprises:
performing escape processing on the data read in the sub-batches;
and writing the escaped data into a data file with a preset file type.
12. The method of claim 11, wherein the data export request carries a data table identifier;
acquiring escape rules from the data table configuration information corresponding to the data table identification;
and performing escaping processing on the data read from the sub-batches according to the escaping rules.
13. The method of claim 1, wherein the data file having a preset file type comprises an excel file.
14. A data derivation apparatus, comprising:
an acquisition unit configured to acquire a data export request;
the construction unit is used for constructing a query operation instruction according to the data export request;
a circulation unit for circularly executing the following steps until the stop condition is satisfied:
reading data to be exported in batches from a data table according to the query operation instruction;
and writing the data read in batches into a data file with a preset file type.
15. An electronic device, comprising:
at least one processor;
a memory having stored thereon program instructions configured to be adapted to be executed by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 1-13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110966592.2A CN113722321A (en) | 2021-08-23 | 2021-08-23 | Data export method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110966592.2A CN113722321A (en) | 2021-08-23 | 2021-08-23 | Data export method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113722321A true CN113722321A (en) | 2021-11-30 |
Family
ID=78677297
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110966592.2A Pending CN113722321A (en) | 2021-08-23 | 2021-08-23 | Data export method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113722321A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138714A (en) * | 2022-02-08 | 2022-03-04 | 上海金仕达软件科技有限公司 | Data migration method and device, storage medium and electronic equipment |
-
2021
- 2021-08-23 CN CN202110966592.2A patent/CN113722321A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138714A (en) * | 2022-02-08 | 2022-03-04 | 上海金仕达软件科技有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN114138714B (en) * | 2022-02-08 | 2022-05-24 | 上海金仕达软件科技有限公司 | Data migration method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108334609B (en) | Method, device, equipment and storage medium for realizing JSON format data access in Oracle | |
| CN108959279B (en) | Data processing method, data processing device, readable medium and electronic equipment | |
| CN114090671A (en) | Data import method and device, electronic equipment and storage medium | |
| CN111159040A (en) | Test data generation method, device, equipment and storage medium | |
| CN112115105A (en) | Service processing method, device and equipment | |
| CN113408254A (en) | Page form information filling method, device, equipment and readable medium | |
| CN110688096A (en) | Method, device, medium and electronic equipment for constructing application program containing plug-in | |
| CN110727425A (en) | Electronic device, form data verification method and computer-readable storage medium | |
| CN115599359A (en) | Code generation method, device, equipment and medium | |
| CN113722321A (en) | Data export method and device and electronic equipment | |
| CN113609128A (en) | Method and device for generating database entity class, terminal equipment and storage medium | |
| CN112363814A (en) | Task scheduling method and device, computer equipment and storage medium | |
| CN117171030A (en) | Method, device, equipment and storage medium for detecting software running environment | |
| CN109324838B (en) | Execution method and execution device of single chip microcomputer program and terminal | |
| CN107451050B (en) | Function acquisition method and device and server | |
| CN112948400A (en) | Database management method, database management device and terminal equipment | |
| CN111143461A (en) | Mapping relation processing system and method and electronic equipment | |
| CN109408035B (en) | Flow configuration method, storage medium and server of business system | |
| CN116431481A (en) | Code parameter verification method and device based on multi-code condition | |
| CN116185389A (en) | Code generation method and device, electronic equipment and medium | |
| CN112579061B (en) | Method, device, system and storage medium for data transfer between android components | |
| CN114817707A (en) | Method and device for creating node and problem, electronic equipment and storage medium | |
| CN115018569A (en) | Invoice issuing method, invoice reimbursement device and equipment based on block chain | |
| CN110515653B (en) | Document generation method and device, electronic equipment and computer readable storage medium | |
| CN109145084B (en) | Data processing method, data processing device and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |