CN114443705A

CN114443705A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN114443705A
Application number: CN202111566847.2A
Authority: CN
Inventors: 王习民; 郭威; 皮浩; 苏妍
Original assignee: Beijing Wangshen Dongjian Technology Co ltd
Current assignee: Beijing Wangshen Dongjian Technology Co ltd
Priority date: 2021-12-20
Filing date: 2021-12-20
Publication date: 2022-05-06

Abstract

The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data; and writing the target data of the insert sentence into a table file every time one insert sentence is read. The data processing method, the data processing device, the electronic equipment and the storage medium provided by the invention realize the parallel operation of reading data and writing data, thereby shortening the data processing time.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

In some fields, such as the authentication field, larger Structured Query Language (SQL) files are often encountered, which are files generated by backing up a large data table in a relational database management system (MySQL).

In the related art, when a user needs to use data of an SQL file, the SQL file is usually imported into a MySQL database, the MySQL database opens the SQL file, and then the data of the opened SQL file is written into a table file, that is, the data is exported into the table file for the user to use.

However, in the related art, if the SQL file is too large, the SQL file is first imported into the MySQL database and then exported into the table file, which results in a long data processing time.

Disclosure of Invention

The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, which are used for solving the defect of long data processing time in the prior art.

The invention provides a data processing method, which comprises the following steps:

reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

and writing the target data of the insert sentence into a table file when reading one insert sentence.

According to a data processing method provided by the invention, reading each insert statement in a database backup file through a script comprises the following steps:

reading each insert statement in the database backup file through a loop body of the pre-created script;

and the loop body is used for indicating that each insert statement in the database backup file is traversed in sequence until the loop is finished when all data in the database backup file is traversed.

According to the data processing method provided by the invention, the insertion statement further comprises a table name;

writing the target data of the insert statement into a table file every time one insert statement is read, including:

acquiring a table file corresponding to the table name when reading one insertion statement;

and writing the target data of the insertion statement into a table file corresponding to the table name.

According to a data processing method provided by the present invention, the writing the target data of the insertion statement into a table file every time one insertion statement is read includes:

determining the current line number written in the table file when reading one insert statement;

and when the current line number is determined to be smaller than a preset value, writing the target data of the insertion statement into the table file.

The data processing method provided by the invention further comprises the following steps:

when the current line number is determined to be larger than or equal to the preset value, a new table file is created;

writing the target data of the insertion statement into the new table file.

According to a data processing method provided by the present invention, when the target data of the insertion statement is written into a table file every time one insertion statement is read, the writing of the target data of the insertion statement into the table file includes:

when the current coding mode of the target data is determined to be not the same as the target coding mode of the table file every time one insertion statement is read, converting the current coding mode of the target data into the target coding mode;

and writing the target data converted into the target coding mode into the table file.

and writing the target data into the table file when the current coding mode is determined to be the same as the target coding mode.

According to a data processing method provided by the invention, before reading each insert statement in the database backup file through the script, the method further comprises the following steps:

and opening the database backup file through the preset statement of the script.

The present invention also provides a data processing apparatus comprising:

the reading unit is used for reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

and the writing unit is used for writing the target data of the insertion statement into a table file when one insertion statement is read.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the data processing methods described above when executing the program.

The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data processing method as described in any of the above.

The invention also provides a computer program product comprising a computer program which, when executed by a processor, carries out the steps of the data processing method as described in any one of the above.

According to the data processing method, the data processing device, the electronic equipment and the storage medium, each insert statement in the database backup file is read through the script, and the target data in the insert statement is written into the table file when one insert statement is read, so that parallel operation of reading data and writing data is realized, and the data processing time is shortened.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a data processing method provided by the present invention;

FIG. 2 is a second schematic flow chart of a data processing method according to the present invention;

FIG. 3 is a third schematic flow chart of a data processing method provided by the present invention;

FIG. 4 is a fourth flowchart of a data processing method provided by the present invention;

FIG. 5 is a fifth flowchart illustrating a data processing method according to the present invention;

FIG. 6 is a schematic diagram of a data processing apparatus according to the present invention;

fig. 7 is a schematic physical structure diagram of an electronic device provided in the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The data processing method of the present invention is described below with reference to fig. 1 to 5.

Fig. 1 is a schematic flow chart of a data processing method provided by the present invention, and as shown in fig. 1, the data processing method includes the following steps:

step 101, reading each insert statement in the database backup file through a script.

The inserted statement comprises target data, the script can be a Python script, and the database backup file can be an sql file; the database backup file generally includes database basic information, at least one data table structure, and at least one INSERT statement corresponding to each data table structure, where the INSERT statement is inserted in the corresponding data table structure, the INSERT statement may be an INSERT statement, each INSERT statement is inserted in the corresponding data table structure, and the target data is a "value in the INSERT statement.

The basic information of the database comprises the version information of the database, annotation information such as the type of the database and the like, and the aim of the invention is to read the target data in the backup file of the database, so that the basic information of the database can be skipped through a script, and the target data in each insert statement can be read based on a data table structure.

For example, a pre-created script first opens a database backup file to be read, and then reads each insert statement in the database backup file through the script, where reading each insert statement may be to read target data in each insert statement until each insert statement in the database backup file is read.

And 102, writing the target data of the insert statement into a table file every time one insert statement is read.

Wherein the table file may be a ". xlsx" file.

Illustratively, while the target data is written into the table file every time the target data is read into one insert statement by the script, it is emphasized here that in the process of writing the target data into the table file, the script is also reading the next insert statement in the database backup file at the same time.

According to the data processing method provided by the invention, each insert statement in the database backup file is read through the script, and the target data in the insert statement is written into the table file when each insert statement is read, so that the parallel operation of reading data and writing data is realized, the data processing time is shortened, and the method is suitable for the data reading scenes of small database backup files and large database backup files.

Optionally, step 101 in fig. 1 may be specifically implemented by the following steps:

reading each insert statement in the database backup file through a loop body of the script which is created in advance.

Illustratively, a loop body of the script is created in advance, and the loop body can be represented by a function for line in f, wherein the for line in f can be regarded as an iterator and can automatically acquire a buffer input/output port; wherein f represents a database backup file to be read, line represents that an insertion statement is read each time, that is, a line of data in the database backup file is read each time, and the execution flow of the for line in f is as follows: and each time one insert statement (a row of data) in the database backup file is traversed and read, and the cycle is ended when all the insert statements in the database backup file are traversed.

According to the data processing method provided by the invention, each insert statement in the database backup file is read by adopting the loop body of the pre-created script, all the insert statements cannot be read at one time, and the target data of one insert statement is read at one time, so that the occupied memory is small, and the normal operation of the script can be ensured. Particularly, for a large database backup file, the problem that a script is flashed back when all data are read at one time can be solved.

Optionally, the insertion statement further includes a table name, fig. 2 is a second flowchart of the data processing method provided by the present invention, and as shown in fig. 2, step 102 in fig. 1 may be specifically implemented by the following steps:

step 1021, when each insertion statement is read, acquiring a table file corresponding to the table name.

Illustratively, the insertion statement further comprises a table name for indicating that the target data in the insertion statement is data in a table file named as the table name; therefore, when each insert statement is read, the table file corresponding to the table name can be acquired before the target data is written into the table file.

And 1022, writing the target data of the insertion statement into a table file corresponding to the table name every time one insertion statement is read.

Illustratively, when the table file corresponding to the table name is acquired, the target data in the insertion statement is written into the table file corresponding to the table name. For example, the insert sentence includes table 1, zhang san and siegan, and it is understood that the table name is table 1, the target data is zhang san and siegan, the table file named table 1 is obtained, and zhang san and siegan are written into the table file named table 1.

It should be noted that the target data in an inserted statement may include multiple pieces of information, for example, 2 pieces of information, and the first piece of information includes sequence number 1, zhangsan, and sienna; the second piece of information comprises serial number 2, Liquan and Beijing; when reading an insertion statement, reading each piece of information and writing the information into a corresponding table file. For example, when there are multiple pieces of information, each piece of information occupies one row of the form file, that is, sequence number 1, zhangsan, and sika are written in the first row of the form file, and sequence number 2, liquad, and beijing are written in the second row of the form file.

It should be noted that, when creating the table file, a related data structure may be added in the table file in advance; for example, if the target data to be read includes the contents of the serial number, the name and the address, data structures such as the serial number in the first column, the name in the second column and the address in the third column are added in advance in the table file, so that the target data can be written into the corresponding position when the target data is read.

It should be noted that, because one database backup file may include data information of a plurality of data tables, when target data in the database backup file is written into a table file, a corresponding table file may be searched according to a table name in an insertion statement, and then the target data is written into the corresponding table file; for example, if the database backup file includes data information of two data tables, that is, data information of table 1 and data information of table 2, respectively, the data information of table 1 is written into the table file named table 1, and the data information of table 2 is written into the table file named table 2.

When the table name of each insertion term is acquired, the table name may be written in the corresponding table file, which is not limited to this.

According to the data processing method provided by the invention, the target data is written into the table file corresponding to the table name, so that the accuracy of data writing is improved.

Optionally, fig. 3 is a third schematic flow chart of the data processing method provided by the present invention, and as shown in fig. 3, step 102 in fig. 1 may be specifically implemented by the following steps:

and step 1023, when one insert statement is read, determining the current line number written into the table file.

For example, a counter may be added, and when the target data needs to be written into the table file, the counter is started, and the counter records the number of rows written into the table file, so as to determine the current number of rows written into the table file.

It should be noted that, when it is determined that the current row number is smaller than the preset value, step 1024 is executed; when it is determined that the current number of rows is greater than or equal to the preset value,

steps

1025 and 1026 are performed.

Step 1024, writing the target data of the insertion statement into the table file when the current line number is determined to be smaller than a preset value.

The preset value is the maximum number of rows that the table file can write into, for example, the preset value is 104 ten thousand rows.

Illustratively, the determined current line number is compared with a preset value, and when the current line number is determined to be smaller than the preset value, the table file is not full, and data can be continuously written; therefore, the target data in the insert sentence is written into the table file at this time.

The data processing method provided by the invention records the current line number written in the table file, and writes the target data into the table file only when the current line number is determined to be smaller than the preset value, so as to ensure that the data written into the table file can be normally displayed.

Optionally, as shown in fig. 3, step 102 in fig. 1 may be further specifically implemented by the following steps:

and 1025, creating a new table file when the current line number is determined to be larger than or equal to the preset value.

Exemplarily, when the current line number is determined to be greater than or equal to the preset value, the table file is full and data cannot be written; therefore, a new form file needs to be created at this time.

And step 1026, writing the target data of the inserted statement into the new form file.

Exemplarily, when a new table file is created, target data of the insertion statement is written into the new table file; and meanwhile, a counter is started to record the written current line number in the new table file, the current line number of the new table file is compared with a preset value, when the current line number of the new table file is larger than the preset value, the new table file is created again, and the process is circulated until all the data in the database backup file are completely read.

It will be appreciated that the name of the new table file created needs to be the same as the table name in the insert statement to be written.

According to the data processing method provided by the invention, when the current line number is determined to be larger than or equal to the preset value, the target data is written into a new table file, so that the data written into the table file can be normally displayed.

Optionally, fig. 4 is a fourth schematic flowchart of the data processing method provided by the present invention, as shown in fig. 4, step 102 in fig. 1 may be specifically implemented by the following steps:

it should be noted that, when it is determined that the current encoding mode of the target data is not the same as the target encoding mode of the table file, step 1027 and step 1028 are executed; when it is determined that the current encoding mode of the target data is the same as the target encoding mode of the table file, step 1029 is performed.

Step 1027, converting the current encoding mode of the target data into the target encoding mode when the current encoding mode of the target data is determined not to be the same as the target encoding mode of the table file.

For example, when target data of each insertion statement is acquired, a current encoding mode of the target data needs to be acquired first, whether the current encoding mode of the target data is the target encoding mode is determined, and when it is determined that the current encoding mode of the target data is not the target encoding mode, the current encoding mode of the target data needs to be converted into the target encoding mode.

Step 1028, writing the target data converted into the target coding mode into the table file.

For example, after the current encoding mode of the target data is converted into the target encoding mode, the target data converted into the target encoding mode is written into the table file, so that the table file can accurately identify the written target data.

And step 1029, writing the target data into the table file when the current coding mode is determined to be the same as the target coding mode.

For example, when the current encoding mode of the target data is determined to be the target encoding mode, the target data is directly written into the table file without converting the encoding mode of the target data.

According to the data processing method provided by the invention, before the target data is written into the table file, the coding mode of the target data needs to be converted into the coding mode which can be identified by the table file, so that the table file can accurately identify the written data.

Optionally, fig. 5 is a fifth schematic flowchart of the data processing method provided by the present invention, and as shown in fig. 5, before executing step 101 in fig. 1, the method further includes the following steps:

and 103, opening the database backup file through a preset statement of the script.

For example, the preset statement may be a with statement, and the with statement is used to open the database backup file to be read, and the with statement may also be capable of handling an abnormal situation occurring during reading the database backup file.

It should be noted that after all the data in the database backup file are completely read, the database backup file needs to be closed through a preset statement, that is, the database backup file needs to be closed through a with statement.

According to the data processing method provided by the invention, the database backup file is opened through the preset statement of the script, so that the target data in the database backup file can be conveniently read through the script in the follow-up process.

The data processing device provided by the invention is described below, and the data processing device described below and the data processing method described above can be referred to correspondingly.

Fig. 6 is a schematic structural diagram of a data processing apparatus provided in the present invention, and as shown in fig. 6, the data processing apparatus includes a reading unit 601 and a writing unit 602; wherein:

a reading unit 601, configured to read each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

a writing unit 602, configured to write the target data of the insertion statement into a table file every time one insertion statement is read. According to the data processing device provided by the invention, each insert statement in the database backup file is read through the script, and the target data in the insert statement is written into the table file when one insert statement is read, so that the parallel operation of reading and writing data is realized, and the data processing time is shortened.

Based on any of the above embodiments, the reading unit 601 is specifically configured to:

According to any of the above embodiments, the insertion statement further includes a table name; the writing unit 602 is specifically configured to:

Based on any of the above embodiments, the writing unit 602 is further specifically configured to:

determining a current number of rows written in the table file;

and writing the target data of the inserted statements into the new table file, and returning to the step of reading each inserted statement in the database backup file through the script until all the target data in the database backup file are read.

According to any of the above embodiments, the apparatus further comprises an opening unit;

and the opening unit is used for opening the database backup file through the preset statement of the script.

Fig. 7 is a schematic physical structure diagram of an electronic device provided in the present invention, and as shown in fig. 7, the electronic device may include: a processor (processor)710, a communication Interface (Communications Interface)720, a memory (memory)730, and a communication bus 740, wherein the processor 710, the communication Interface 720, and the memory 730 communicate with each other via the communication bus 740. Processor 710 may call logical instructions in memory 730 to perform a data processing method comprising: reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

and writing the target data of the insert sentence into a table file every time one insert sentence is read.

In addition, the logic instructions in the memory 730 can be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program, when executed by a processor, being capable of executing the data processing method provided by the above methods, the method comprising: reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the data processing method provided by the above methods, the method comprising: reading each insert statement in the database backup file through a script; wherein the insert statement comprises target data;

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, comprising:

2. The data processing method of claim 1, wherein the reading each insert statement in the database backup file by the script comprises:

3. The data processing method according to claim 1, wherein the insertion statement further includes a table name;

when each inserting statement is read, acquiring a table file corresponding to the table name;

4. The data processing method according to claim 1, wherein writing the target data of the insertion sentence into a table file every time one insertion sentence is read, comprises:

5. The data processing method of claim 4, further comprising:

writing the target data of the insert statement into the new table file.

6. The data processing method according to claim 1, wherein writing the target data of the insertion sentence into a table file every time one insertion sentence is read comprises:

7. The data processing method of claim 6, further comprising:

8. The data processing method of any of claims 1-7, wherein prior to the reading each insert statement in the database-backed file by the script, the method further comprises:

and opening the database backup file through a preset statement of the script.

9. A data processing apparatus, comprising:

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the data processing method according to any of claims 1 to 8 are implemented when the processor executes the program.

11. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the data processing method according to any one of claims 1 to 8 when executed by a processor.