CN112650805A - Data export method and device, computer equipment and readable storage medium - Google Patents
Data export method and device, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN112650805A CN112650805A CN202011630210.0A CN202011630210A CN112650805A CN 112650805 A CN112650805 A CN 112650805A CN 202011630210 A CN202011630210 A CN 202011630210A CN 112650805 A CN112650805 A CN 112650805A
- Authority
- CN
- China
- Prior art keywords
- file
- exported
- data
- database
- read data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000004590 computer program Methods 0.000 claims description 16
- 238000009795 derivation Methods 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 7
- 230000005012 migration Effects 0.000 description 3
- 238000013508 migration Methods 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data export method, a device, a computer device and a readable storage medium, wherein the data export method comprises the following steps: the method comprises the steps of obtaining read data of a file to be exported in a server, determining the data volume of the read data according to the read data of the file to be exported, splitting the read data if the data volume is larger than an export upper limit, exporting the read data for multiple times, completing exporting the file to be exported, and otherwise, exporting the file to be exported through a single time, solving the problems that in the related technology, a large-scale data export mode of one-time export and continuous waiting is adopted, the occupied memory resource is too large, even the server crashes, the occupied memory resource of the server is reduced when the large-scale data are exported, and the server is prevented from crashing.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data exporting method, an apparatus, a computer device, and a readable storage medium.
Background
In many application scenarios, data in the information system background database needs to be exported as EXCEL tables. With the development of information-based big data, large-scale data is often required to be processed in data export, for example, for the garment customization industry, manufacturers regularly need to export the order quantity of one year or one quarter.
In the related art, a large-scale data export service is processed, generally, one-time export and continuous waiting are adopted, the export mode occupies too large memory resources, so that the system performance is reduced, and in the process of large-scale data export, the export time is very long and even a server crashes.
At present, an effective solution is not provided aiming at the problems that a large-scale data export mode of one-time export and continuous waiting is adopted, the occupied memory resource is too large, and even a server is crashed.
Disclosure of Invention
The embodiment of the application provides a data export method, a data export device, computer equipment and a readable storage medium, and solves the problem that in the related technology, a large-scale data export mode of one-time export and continuous waiting is adopted, so that the memory resource occupation is overlarge and even a server crashes.
In a first aspect, an embodiment of the present application provides a data export method, which is applied to a server, and the method includes:
acquiring read data of a file to be exported in the server;
determining the data volume of the read data according to the read data of the file to be exported;
and if the data volume is larger than the export upper limit, splitting the read data, exporting the file to be exported through exporting the read data for multiple times, and otherwise, exporting the file to be exported through exporting the read data for a single time.
In some embodiments, obtaining read data of a file to be exported includes:
acquiring storage attributes of a file to be exported, wherein the storage attributes at least comprise a storage database;
calling a file reading type matched with the storage attribute from a first database according to the storage attribute, wherein at least two different file reading types are stored in the first database;
and reading the file to be exported according to the file reading type matched with the storage attribute of the file to be exported, and generating the read data of the file to be exported.
In some embodiments, the storage database of the file to be exported is at least one of the following types: the elasticsearch database, the mysql database.
In some embodiments, the first database includes a file read type of snapshot read and a file read type of split read; and if the storage database of the file to be exported is an elastic search database, reading the file to be exported by adopting a file reading type read by the snapshot, and if the storage database of the file to be exported is a mysql database, reading the file to be exported by adopting a file reading type read by splitting.
In some embodiments, if the storage database of the file to be exported is an elastic search database, reading the file to be exported according to a file reading type matched with the storage attribute of the file to be exported, and generating read data of the file to be exported includes:
setting a query parameter based on the elastic search database;
according to the query parameters, files to be exported in the elastic search database are queried in batches;
and storing the inquired file to be exported into the snapshot copy to generate read data of the file to be exported.
In some embodiments, the batch querying, according to the query parameter, the file to be exported in the elasticsearch database includes:
and querying a file to be exported in the elastic search database in a cursor query mode according to the query parameters.
In some embodiments, if the storage database of the file to be exported is a mysql database, reading the file to be exported according to a file reading type matched with the storage attribute of the file to be exported, and generating read data of the file to be exported includes:
acquiring the storage quantity of files to be exported in the mysql database;
determining the reading times and the reading unit quantity of the file to be exported according to the storage quantity;
and reading the file to be exported according to the reading times and the reading unit quantity to generate read data of the file to be exported.
In a second aspect, an embodiment of the present application provides a data exporting apparatus, including: the device comprises a first acquisition module, a second acquisition module and a data export module;
the first obtaining module is used for obtaining read data of a file to be exported in the server;
the second obtaining module is configured to determine a data amount of the read data according to the read data of the file to be exported;
and the data export module is used for splitting the read data if the data volume is larger than the export upper limit, exporting the file to be exported by exporting the read data for multiple times, and exporting the file to be exported by exporting the read data once.
In a third aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor, when executing the computer program, implements the data export method according to the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the data export method as described in the first aspect above.
Compared with the prior art, the data export method, the data export device, the computer equipment and the readable storage medium provided by the application read data of files to be exported through obtaining, according to the read data of files to be exported determines the data volume of read data, if the data volume is greater than export the upper limit then will read data and split, and export through a lot of read data, accomplish the export of files to be exported, otherwise, export through the single read data, accomplish the export of files to be exported has solved the massive data export mode of once exporting, continuously waiting, has the problem that occupies too big memory resource or even server breakdown, has reduced the memory resource of the server that occupies when massive data exports, avoids the problem of server breakdown.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow diagram of a data export method according to an embodiment of the present application;
FIG. 2 is a flowchart of a first method for obtaining read data of a file to be exported according to an embodiment of the present application;
FIG. 3 is a flowchart II of a method for obtaining read data of a file to be exported according to an embodiment of the present application;
FIG. 4 is a flowchart of a method for obtaining read data of a file to be exported, according to an embodiment of the present application;
FIG. 5 is a flow chart of a data export method according to a preferred embodiment of the present application;
FIG. 6 is a block diagram of a data export apparatus according to an embodiment of the present application;
fig. 7 is a schematic diagram of an internal structure of a computer device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
In the related art, for the clothing customization industry, during a year or a quarter, a manufacturer or a merchant stores the order amount in the period, for example, the order amount may be stored in an elasticsearch database, a mysql database, an api database and the like which are deployed on the same server; later vendors or companies need to export the order quantity of a year or a quarter to facilitate analysis and summarization of the order quantity; however, at present, the export mode often adopted by a manufacturer or a company is "export once and wait continuously", the export mode occupies too large memory resources of a server, so that the system performance is reduced, and in the process of exporting large-scale data, the export time is very long, and even the server crashes. The data export method is applied to a server, the read data of a file to be exported are obtained, the data volume of the read data is determined according to the read data of the file to be exported, further, the read data is split if the data volume is larger than the export upper limit, the export of the file to be exported is completed by exporting the read data for multiple times, otherwise, the export of the file to be exported is completed by exporting the read data once, and the problem that the server occupies too large memory resources and even collapses due to the adoption of a large-scale data export mode of 'one-time export and continuous waiting' in the related technology is solved, the memory resources of the server occupied during the large-scale data export are reduced, and the server crash problem is avoided.
The present embodiment provides a data export method, and fig. 1 is a flowchart of a data export method according to an embodiment of the present application, and as shown in fig. 1, the data export method includes the following steps:
step S101, reading data of a file to be exported in a server is obtained;
specifically, before exporting a file to be exported in a server, reading data of the file to be exported in the server is obtained or pulled;
step S102, determining the data volume of read data according to the read data of the file to be exported;
for example, the data amount of the read data of the file to be exported may be determined according to the memory amount occupied by the read data;
step S103, splitting the read data if the data volume is larger than the export upper limit, exporting the read data for multiple times to finish the export of the file to be exported, and conversely, exporting the read data for a single time to finish the export of the file to be exported;
specifically, when reading data of a file to be exported is exported, further judgment is needed, that is, whether the data volume of the reading data of the file to be exported is larger than an export upper limit is judged, and if the data volume of the reading data is larger than the export upper limit, the reading data is exported for multiple times, wherein Excel can be exported, or an xml form of an Excel file can be exported and then stored as an xls file;
through the steps S101 to S103, firstly, the read data of the file to be exported is obtained from the corresponding server, and the data size of the read data is determined according to the read data of the file to be exported, further, if the data size is greater than the export upper limit, the read data is split, and the export of the file to be exported is completed by exporting the read data for multiple times, otherwise, the export of the file to be exported is completed by exporting the read data once, thereby solving the problem that the large-scale data export mode of "export once and wait continuously" is adopted in the related art, which occupies too large memory resources of the server, even the server crashes, reduces the memory resources of the server occupied during the export of the large-scale data, and avoids the server crashing.
In some embodiments, fig. 2 is a flowchart of a first method for obtaining read data of a file to be exported, where as shown in fig. 2, obtaining read data of a file to be exported includes the following steps:
step S201, obtaining the storage attribute of the file to be exported, wherein the storage attribute at least comprises a storage database;
it should be noted that during a year or a quarter, a manufacturer or a merchant may store the order amount in a certain server, but there are databases such as an elastic search database, a mysql database, and an api deployed at the same time on the same server, so there are cases where different files to be exported are stored in different database types;
step S202, according to the storage attribute, calling a file reading type matched with the storage attribute from a first database, wherein at least two different file reading types are stored in the first database;
for example, if the file to be exported is stored in the elastic search database, a first file reading type may be adopted, and if the file to be exported is stored in the mysql database, a second file reading type may be adopted, where the first file reading type and the second file reading type are different;
step S203, reading the file to be exported according to the file reading type matched with the storage attribute of the file to be exported, and generating reading data of the file to be exported;
through the steps S201 to S203, the storage attribute of the file to be exported is obtained in advance, then the corresponding file reading type is matched according to the storage attribute of the file to be exported, further, the file to be exported is read according to the file reading type matched with the storage attribute of the file to be exported, the read data of the file to be exported is generated, the file to be exported in the server is read by adopting the file reading type matched with the storage attribute of the file to be exported, and the reading efficiency is further improved.
In some embodiments, the storage database of the file to be exported is at least one of the following types: an elasticsearch database, a mysql database; in some embodiments, the first database includes a file read type of snapshot read and a file read type of split read; if the storage database of the file to be exported is an elastic search database, reading the file to be exported by adopting a file reading type read by the snapshot, and if the storage database of the file to be exported is a mysql database, reading the file to be exported by adopting a file reading type read by splitting;
it should be noted that, for a multi-cluster enterprise-level complex architecture, data migration is performed on data in an elastic search database in a server in a snapshot manner, so that data migration is stable, and the integrity of data migration is high.
In some embodiments, fig. 3 is a flowchart of a second method for acquiring read data of a file to be exported according to the embodiment of the present application, and as shown in fig. 3, if a storage database of the file to be exported is an elastic search database, reading the file to be exported according to a file read type matched with a storage attribute of the file to be exported, and generating the read data of the file to be exported includes the following steps:
step S301, setting query parameters based on the elasticsearch database;
in particular, the query parameter may be one minute per query, i.e. one minute per maintaining a query window in the server's elastic search database; the query time is refreshed each time the query is made, so that the time is only enough to process the current batch of data;
step S302, inquiring files to be exported in the elastic search database in batch according to the inquiry parameters;
step S303, storing the inquired file to be exported into the snapshot copy, and generating read data of the file to be exported.
In some embodiments, the batch querying, according to the query parameter, the file to be exported in the elasticsearch database includes: inquiring a file to be exported in the elastic search database in a cursor inquiry mode according to the inquiry parameters;
it should be noted that, the cursor query allows to perform query initialization first, that is, to set query parameters based on the elastic search database, and then to pull the results in batch; the vernier query takes snapshot data at a certain time point, any change on the index after the query is initialized can be ignored by the vernier query, the vernier query realizes the characteristic by saving an old data file, and the result is just like the retention of an index view during initialization; another way to query is deep paging, where the root of deep paging is result set global ordering, and where a cursor query is ordered by field _ doc compared to deep paging, this instruction lets the elastic search return the next batch of results only from the slice that has the result.
In some embodiments, fig. 4 is a flowchart of a third method for acquiring read data of a file to be exported according to the embodiment of the present application, where as shown in fig. 4, if a storage database of the file to be exported is a mysql database, reading the file to be exported according to a file read type matched with a storage attribute of the file to be exported, and generating the read data of the file to be exported includes:
step S401, acquiring the storage quantity of files to be exported in the mysql database;
step S402, determining the reading times and the reading unit quantity of the file to be exported according to the storage quantity;
for example, if the storage quantity of the current file to be exported in the mysql database is 10 ten thousand files, 1 ten thousand files can be read each time;
step S403, reading the file to be exported according to the reading times and the reading unit quantity, and generating reading data of the file to be exported;
through steps S401 to S403, according to the storage number of the files to be exported in the mysql database of the server, the consumed server memory is reduced to a certain extent by reading data or pulling data in batches.
The embodiments of the present application are described and illustrated below by means of preferred embodiments.
Fig. 5 is a flowchart of a data export method according to the preferred embodiment of the present application, and as shown in fig. 5, the data export method includes the following steps:
step S501, acquiring the type of a storage database of a file to be exported in a server;
step S502, according to the type of the storage database, calling a file reading type matched with the type of the storage database from a first database, wherein the first database comprises a file reading type of snapshot reading and a file reading type of split reading;
step S503, if the storage database of the file to be exported is an elastic search database, reading the file to be exported by adopting the file reading type read by the snapshot, and if the storage database of the file to be exported is a mysql database, reading the file to be exported by adopting the file reading type read by splitting;
step S504, according to the read data of the file to be exported, determining the data volume of the read data;
and step S505, if the data volume is larger than the export upper limit, splitting the read data, exporting the read data for multiple times, and otherwise, exporting the file to be exported by exporting the read data for a single time.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
The present embodiment further provides a data exporting apparatus, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted here. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
In some embodiments, fig. 6 is a block diagram of a data export apparatus according to an embodiment of the present application, and as shown in fig. 6, the data export apparatus includes: a first acquisition module 61, a second acquisition module 62 and a data derivation module 63;
a first obtaining module 61, configured to obtain read data of a file to be exported;
a second obtaining module 62, configured to determine a data amount of read data according to the read data of the file to be exported;
the data export module 63 is configured to split the read data if the data amount is greater than the export upper limit, and export the file to be exported by exporting the read data for multiple times, or, export the read data for a single time to complete export of the file to be exported;
through the data export device, the read data of the file to be exported is firstly obtained from the corresponding server, the data volume of the read data is determined according to the read data of the file to be exported, further, if the data volume is larger than the export upper limit, the read data is split, the export of the file to be exported is completed by exporting the read data for multiple times, otherwise, the export of the file to be exported is completed by exporting the read data once, so that the problem that the server occupies too large memory resources and even crashes due to the adoption of a large-scale data export mode of 'one-time export and continuous waiting' in the related technology is solved, the occupied memory resources of the server are reduced during the large-scale data export, and the problem that the server crashes is avoided.
In some embodiments, the first obtaining module 61 and the second obtaining module 62 are further configured to implement steps in the data exporting method provided in the foregoing embodiments, and are not described herein again.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data export method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
In an embodiment, fig. 7 is a schematic diagram of an internal structure of a computer device according to an embodiment of the present application, and as shown in fig. 7, a computer device is provided, where the computer device may be a server, and the internal structure diagram may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data derivation.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the steps in the data export method provided by the above embodiments are implemented.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of data derivation provided by the various embodiments described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A data export method, applied to a server, the method comprising:
acquiring read data of a file to be exported in the server;
determining the data volume of the read data according to the read data of the file to be exported;
and if the data volume is larger than the export upper limit, splitting the read data, exporting the file to be exported through exporting the read data for multiple times, and otherwise, exporting the file to be exported through exporting the read data for a single time.
2. The data export method of claim 1, wherein obtaining read data of a file to be exported comprises:
acquiring storage attributes of a file to be exported, wherein the storage attributes at least comprise a storage database;
calling a file reading type matched with the storage attribute from a first database according to the storage attribute, wherein at least two different file reading types are stored in the first database;
and reading the file to be exported according to the file reading type matched with the storage attribute of the file to be exported, and generating the read data of the file to be exported.
3. The data export method of claim 2, wherein the storage database of the file to be exported is at least one of the following types: the elasticsearch database, the mysql database.
4. The data export method according to claim 3, wherein the first database comprises a file read type of snapshot read and a file read type of split read; and if the storage database of the file to be exported is an elastic search database, reading the file to be exported by adopting a file reading type read by the snapshot, and if the storage database of the file to be exported is a mysql database, reading the file to be exported by adopting a file reading type read by splitting.
5. The data export method of claim 4, wherein if the storage database of the file to be exported is an elastic search database, reading the file to be exported according to a file reading type matching the storage attribute of the file to be exported, and generating the read data of the file to be exported comprises:
setting a query parameter based on the elastic search database;
according to the query parameters, files to be exported in the elastic search database are queried in batches;
and storing the inquired file to be exported into the snapshot copy to generate read data of the file to be exported.
6. The data export method of claim 5, wherein the batch query of the file to be exported in the elastic search database according to the query parameters comprises:
and querying a file to be exported in the elastic search database in a cursor query mode according to the query parameters.
7. The data exporting method according to claim 4, wherein if the storage database of the file to be exported is a mysql database, reading the file to be exported according to a file reading type matching the storage attribute of the file to be exported, and generating the read data of the file to be exported includes:
acquiring the storage quantity of files to be exported in the mysql database;
determining the reading times and the reading unit quantity of the file to be exported according to the storage quantity;
and reading the file to be exported according to the reading times and the reading unit quantity to generate read data of the file to be exported.
8. A data export apparatus, applied to a server, the data export apparatus comprising: the device comprises a first acquisition module, a second acquisition module and a data export module;
the first acquisition module is used for acquiring read data of a file to be exported in the server;
the second obtaining module is configured to determine a data amount of the read data according to the read data of the file to be exported;
and the data export module is used for splitting the read data if the data volume is larger than the export upper limit, exporting the file to be exported by exporting the read data for multiple times, and exporting the file to be exported by exporting the read data once.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data derivation method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out a data derivation method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011630210.0A CN112650805A (en) | 2020-12-30 | 2020-12-30 | Data export method and device, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011630210.0A CN112650805A (en) | 2020-12-30 | 2020-12-30 | Data export method and device, computer equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112650805A true CN112650805A (en) | 2021-04-13 |
Family
ID=75366991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011630210.0A Pending CN112650805A (en) | 2020-12-30 | 2020-12-30 | Data export method and device, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650805A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656968A (en) * | 2017-08-31 | 2018-02-02 | 武汉斗鱼网络科技有限公司 | High-volume business datum deriving method and system |
| CN107784085A (en) * | 2017-09-30 | 2018-03-09 | 平安科技（深圳）有限公司 | The deriving method and its terminal of a kind of data list |
| CN109299157A (en) * | 2018-08-27 | 2019-02-01 | 杭州安恒信息技术股份有限公司 | A kind of data export method and device of distributed big single table |
| CN110046191A (en) * | 2019-04-22 | 2019-07-23 | 成都四方伟业软件股份有限公司 | A kind of data export configuration method and device |
| CN110119418A (en) * | 2018-01-16 | 2019-08-13 | 北京京东尚科信息技术有限公司 | Data export method and device |
| CN110555077A (en) * | 2019-08-27 | 2019-12-10 | 上海易点时空网络有限公司 | automatic question bank synchronization method and device |
| CN112148513A (en) * | 2019-06-28 | 2020-12-29 | 北京百度网讯科技有限公司 | Dump method and device for single point service data and storage medium |
-
2020
- 2020-12-30 CN CN202011630210.0A patent/CN112650805A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656968A (en) * | 2017-08-31 | 2018-02-02 | 武汉斗鱼网络科技有限公司 | High-volume business datum deriving method and system |
| CN107784085A (en) * | 2017-09-30 | 2018-03-09 | 平安科技（深圳）有限公司 | The deriving method and its terminal of a kind of data list |
| CN110119418A (en) * | 2018-01-16 | 2019-08-13 | 北京京东尚科信息技术有限公司 | Data export method and device |
| CN109299157A (en) * | 2018-08-27 | 2019-02-01 | 杭州安恒信息技术股份有限公司 | A kind of data export method and device of distributed big single table |
| CN110046191A (en) * | 2019-04-22 | 2019-07-23 | 成都四方伟业软件股份有限公司 | A kind of data export configuration method and device |
| CN112148513A (en) * | 2019-06-28 | 2020-12-29 | 北京百度网讯科技有限公司 | Dump method and device for single point service data and storage medium |
| CN110555077A (en) * | 2019-08-27 | 2019-12-10 | 上海易点时空网络有限公司 | automatic question bank synchronization method and device |
Non-Patent Citations (2)
| Title |
|---|
| 城北涂公: "Elasticsearch的数据导入导出", 《CSDN，HTTPS://BLOG.CSDN.NET/WEIXIN_37785562/ARTICLE/DETAILS/76255718》 * |
| 豆不女: "elasticsearch-利用游标查询 "Scroll"来做分页查询", 《CSDN，HTTPS://BLOG.CSDN.NET/CHUAN442616909/ARTICLE/DETAILS/55195024》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10509772B1 (en) | Efficient locking of large data collections | |
| CN108462760B (en) | Electronic device, automatic cluster access domain name generation method and storage medium | |
| CN109766349B (en) | Task duplicate prevention method, device, computer equipment and storage medium | |
| CN108763572B (en) | Method and device for realizing Apache Solr read-write separation | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| CA3128540C (en) | Cache system hotspot data access method, apparatus, computer device and storage medium | |
| CN110427386B (en) | Data processing method, device and computer storage medium | |
| CN110943934A (en) | Service request processing method, system, terminal and readable storage medium | |
| CN110781196A (en) | Block chain transaction processing method and device, computer equipment and storage medium | |
| CN113094076A (en) | Version iteration method, device, equipment and medium based on version control | |
| CN110866011B (en) | Data table synchronization method and device, computer equipment and storage medium | |
| CN111400578B (en) | Cargo data query method, cargo data query device, computer equipment and storage medium | |
| CN110990132B (en) | Asynchronous task processing method and device, computer equipment and storage medium | |
| CN109542962B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN115525631A (en) | Database data migration method, device, equipment and storage medium | |
| CN115454512A (en) | Chip drive loading configuration method, device, equipment and storage medium | |
| CN112052144B (en) | Information management method, device, electronic equipment and storage medium | |
| CN112650805A (en) | Data export method and device, computer equipment and readable storage medium | |
| CN109857759B (en) | Concurrent data processing method and device, computer equipment and storage medium | |
| CN111309463B (en) | Method and device for determining task execution time and readable storage medium | |
| CN109462543B (en) | Mail downloading method, device, computer equipment and storage medium | |
| CN112463783A (en) | Index data monitoring method and device, computer equipment and storage medium | |
| CN109284260B (en) | Big data file reading method and device, computer equipment and storage medium | |
| CN113849195A (en) | Application program loading method and device, storage medium and terminal | |
| CN112783866A (en) | Data reading method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210413 |
|
| RJ01 | Rejection of invention patent application after publication |