CN115238010A - Data processing method and device based on Gaussian database - Google Patents
Data processing method and device based on Gaussian database Download PDFInfo
- Publication number
- CN115238010A CN115238010A CN202210977097.6A CN202210977097A CN115238010A CN 115238010 A CN115238010 A CN 115238010A CN 202210977097 A CN202210977097 A CN 202210977097A CN 115238010 A CN115238010 A CN 115238010A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- database
- gaussian
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and device based on a Gaussian database, which can be used in the financial field or other technical fields. The method comprises the following steps: analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset; connecting a Gaussian database based on the Gaussian database connection configuration information; and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file. The device is used for executing the method. The data processing method and device based on the Gaussian database provided by the embodiment of the invention realize the import and export of the data of the Gaussian database.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data processing method and device based on a Gaussian database.
Background
The Gaussian database is a relational database and has the four characteristics of strong consistency, second capacity expansion, low cost and super availability. However, because the gaussian database has a short appearance time, the related software of the gaussian database is relatively short, and how to import and export the data of the gaussian database becomes an important issue to be solved urgently in the field.
Disclosure of Invention
To solve the problems in the prior art, embodiments of the present invention provide a data processing method and apparatus based on a gaussian database, which can at least partially solve the problems in the prior art.
In a first aspect, the present invention provides a data processing method based on a gaussian database, including:
analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset;
connecting a Gaussian database based on the Gaussian database connection configuration information;
and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file.
In a second aspect, the present invention provides a data processing apparatus based on a gaussian database, comprising:
the analysis module is used for analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset;
the connection module is used for connecting the Gaussian database based on the Gaussian database connection configuration information;
and the data processing module is used for importing external data into the Gaussian database or exporting data from the Gaussian database to an external file according to the data processing configuration information.
In a third aspect, the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the data processing method based on the gaussian database according to any of the above embodiments.
In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data processing method based on the gaussian database according to any one of the above embodiments.
In a fifth aspect, the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the data processing method based on the gaussian database according to any one of the above embodiments.
The data processing method and device based on the Gaussian database provided by the embodiment of the invention analyze the data processing configuration file to obtain the data processing configuration information, connect the Gaussian database based on the Gaussian database connection configuration information, and import or export external data into or from the Gaussian database to the external file according to the data processing configuration information, thereby realizing the import and export of the data of the Gaussian database.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic flowchart of a data processing method based on a gaussian database according to a first embodiment of the present invention.
Fig. 2 is a schematic flowchart of a data processing method based on a gaussian database according to a second embodiment of the present invention.
Fig. 3 is a schematic flowchart of a data processing method based on a gaussian database according to a third embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a fourth embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a fifth embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a sixth embodiment of the present invention.
Fig. 7 is a schematic physical structure diagram of an electronic device according to a seventh embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
The following describes a specific implementation process of the data processing method based on the gaussian database according to the embodiment of the present invention, taking a server as an execution subject.
Fig. 1 is a schematic flowchart of a data processing method based on a gaussian database according to a first embodiment of the present invention, and as shown in fig. 1, the data processing method based on a gaussian database according to the embodiment of the present invention includes:
s101, analyzing a data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset;
specifically, the server may parse the data processing configuration file to obtain data processing configuration information, where the data processing configuration information includes information that needs to be used when importing data or exporting data from the gaussian database. Wherein the data processing profile is preset. The data processing configuration file is a data import configuration file or a data export configuration file. In order to import external data into the gaussian database, a data import configuration file needs to be preset; in order to export data in the gaussian database to an external file, a data export configuration file needs to be set in advance.
For example, the data processing configuration file is in an eXtensible Markup Language (XML) format. The method ParseXml () can be constructed, SAXReader types are used for analyzing XML files, SAXReader. Wherein, before using SAXReader class, MAVEN dependence of corresponding dom4j needs to be introduced to support the use of SAXReader class.
S102, connecting a Gaussian database based on the Gaussian database connection configuration information;
specifically, the server may establish a connection with the gaussian database based on the gaussian database connection configuration information, so as to facilitate subsequent importing of data into the gaussian database or exporting data from the gaussian database. The gaussian database connection configuration information is preset and is set according to actual needs, and the embodiment of the invention is not limited.
For example, the gaussian database connection configuration information includes information such as an access address, a user name, a password, and a driver of the gaussian database. For example, the access address of the gaussian database is: jdbc: postgresql: //122.42.253.142, 8000/eida, user name: batcheida, password: batcheida @123, driven as: org.
And S103, importing external data into the Gaussian database or exporting data from the Gaussian database to an external file according to the data processing configuration information.
Specifically, when the data processing configuration file is a data import configuration file, the server may import the external data into the gaussian database for storage based on the data processing configuration information obtained from the data import configuration file according to the data processing configuration information obtained from the data import configuration file parsing. When the data processing configuration file is a data export configuration file, the server may parse the data processing configuration file to obtain data processing configuration information from the data export configuration file, and export data from the gaussian database to an external file based on the data processing configuration information obtained from the data export configuration file.
According to the data processing method based on the Gaussian database, provided by the embodiment of the invention, the data processing configuration file is analyzed to obtain the data processing configuration information, the Gaussian database is connected with the configuration information based on the Gaussian database, and the external data is imported into the Gaussian database or exported from the Gaussian database to the external file according to the data processing configuration information, so that the import and export of the data of the Gaussian database are realized.
On the basis of the foregoing embodiments, further, the data processing configuration file is a data import configuration file; accordingly, the importing external data into the gaussian database according to the data processing configuration information comprises:
loading external data according to the imported configuration information, and importing the external data into a data table of the Gaussian database in a data stream mode; wherein the import configuration information is obtained by analyzing a data import configuration file.
Specifically, the data processing configuration file is a data import configuration file, the server obtains import configuration information from the data import configuration file through analysis, then loads external data according to the import configuration information, converts the external data into a data stream form, and then imports the data stream into a data table of the gaussian database, so as to import the external data into the data table of the gaussian database.
On the basis of the foregoing embodiments, further, the data import configuration file includes an external file to be loaded and an import gaussian database file, where:
the external file to be loaded comprises an encoding format, a file content separation type and segmentation data of the external file to be loaded;
the imported Gaussian database file comprises a data table name, a field name, an external file path and an external file name.
Specifically, the data import configuration file comprises an external file to be loaded and an import gaussian database file. The external file to be loaded comprises an encoding format, a file content separation type and segmentation data of the external file to be loaded. The file content partition type may be a fixed length, in which the file content is divided into fixed lengths, or a partition, in which the file content is divided into partition data. When the file content division type is a fixed length, the division data is a fixed length of the division content, such as 100 bytes. When the file content division type is a delimiter, the division data is a division identifier such as a comma or a semicolon.
The imported Gaussian database file comprises a data table name, a field name, an external file path and an external file name. The data table name is the name of a data table of a Gaussian database storing external data, the field name is the field name included in the data table corresponding to the data table name, the field name can be multiple, and the field name corresponds to the data in the external file. The external file name refers to a name of a file storing external data. The external file path refers to a storage path of a file corresponding to the external file name. And obtaining external data needing to be imported into the Gaussian database through the external file path and the external file name.
Fig. 2 is a schematic flowchart of a data processing method based on a gaussian database according to a second embodiment of the present invention, and as shown in fig. 2, based on the foregoing embodiments, further, the loading external data according to import configuration information, and importing the external data into a data table of the gaussian database in a data stream form includes:
s201, acquiring an import file corresponding to the external file name according to the external file path and the external file name;
specifically, the server accesses a file corresponding to the external file name according to the external file path and the external file name, and obtains an import file.
S202, carrying out fluidization processing on the imported file to obtain a file byte input stream;
specifically, the server performs streaming processing on the import file, and converts data in the import file into a file byte input stream.
S203, generating a data import statement according to the name of the data table, the segmentation data, the coding format of the external file to be loaded and a preset SQL statement template;
specifically, the server replaces the corresponding parameters in a preset SQL statement template with the data table name, the segmentation data and the coding format of the external file to be loaded, and generates a data import statement. Wherein, the SQL statement template is preset.
And S204, executing the data import statement to import the file byte input stream into the data table corresponding to the data table name, and correspondingly storing the data field corresponding to the field name.
Specifically, the server executes the data import statement, imports a file byte input stream into a data table corresponding to the data table name, so that each segment of data divided by a division identifier or a fixed length corresponds to one data field, and stores each segment of data in correspondence with the corresponding data field. The correspondence relationship between each piece of data and the data field is predetermined when the data storage data is imported.
For example, a method copyFromFile () is constructed, and external data is loaded into a data table of a gaussian database by using a CopyManager class packaged in a gaussian jdbc driver. Firstly, an import file for storing external data is fluidized by a FileInputStream class to form a file byte input stream, then a copymanager. CopyIn method is called to splice SQL statements through a preset SQL statement template to serve as data import statements, and the data import statements are executed to import the file byte input stream into a data table of a Gaussian database. Presetting an SQL statement template as follows: copy table _ name from stdin with delimiter encoding filenputstream, wherein table _ name represents a data table name; deleimiter represents the segmentation data; encoding represents the encoding format of the external file to be loaded; filelnputstream represents a file byte input stream. Wherein the Gaussian database supports copying data FROM a file into a database table using a COPY FROM statement.
On the basis of the foregoing embodiments, further, the data processing configuration file is a data export configuration file; accordingly, the exporting of data from within the gaussian database to an external file according to the data processing configuration information comprises:
inquiring data from the Gaussian database according to the export configuration information, and exporting the inquired data to an external file; wherein the export configuration information is parsed from a data export configuration file.
Specifically, the data processing configuration file is a data export configuration file, the server analyzes the data export configuration file to obtain export configuration information, then queries data from the gaussian database according to the export configuration information, and then exports the queried data stream to an external file, so that data in a data table of the gaussian database is exported to the external file.
On the basis of the foregoing embodiments, further, the data export configuration file includes:
file name, file path, file encoding format, file content partition type, partition data, database query statement, and export data field.
Specifically, the data export configuration file comprises a file name, a file path, a file encoding format, a file content partition type, partition data, a database query statement and an export data field. The file name, the file path, the file coding format, the file content separation type, the separation data, the database query statement and the export data field can be obtained by analyzing the data export configuration file. The file name is the file name of an external file storing data derived from the gaussian database. The file path is a directory storing an external file corresponding to a file name. The file encoding format refers to a file encoding format adopted by data of an external file. The file content partition type may be a fixed length, in which the file content is divided into fixed lengths, or a partition, in which the file content is divided into partition data. When the file content division type is a fixed length, the division data is a fixed length of the division content, such as 100 bytes. When the file content division type is a delimiter, the division data is a division identifier such as a comma or a semicolon. The database query statement is an SQL statement used for querying data from the gaussian database, and is set according to actual needs, which is not limited in the embodiments of the present invention. The database query statement includes a data table of derived data. Derived data fields refer to data fields used by a database query statement.
Fig. 3 is a schematic flowchart of a data processing method based on a gaussian database according to a third embodiment of the present invention, and as shown in fig. 3, based on the foregoing embodiments, further, the querying data from the gaussian database according to export configuration information, and exporting the queried data to a file includes:
s301, obtaining derived data from a Gaussian database by querying according to the database query statement and the derived data field; wherein, the data volume of each derived data is preset;
specifically, the server queries, based on the database query statement and the derived data fields, data corresponding to each derived data field from a gaussian database as derived data. If the queried data volume is large, the condition of memory overflow exists, and the data volume of each time of exported data is preset to prevent the memory overflow.
S302, storing the derived data into an external file corresponding to the file name under the file path according to the file coding format and the segmentation data.
Specifically, the server divides data corresponding to each export data field in the export data according to the division data, and stores the divided export data into an external file corresponding to a file name under the file path according to a file coding format. It can be understood that, if there is no external file corresponding to the file name in the file path, an external file corresponding to the file name is created.
For example, for jdbc in a gaussian database, the data size of each derived data is set by the value of fetchsize, which is positive and whose status is TYPE _ FORWARD _ ONLY, and the value of setFetchSize becomes valid.
And executing conn.preparsestatus to create a status object pstmt, and assigning the inquired derived data to the pstmt through a database inquiry Statement and derived data field inquiry data included in the derived configuration information. The derived configuration information includes a split identification.
Establishing a text file write-in type TextFileWriter, obtaining derived data from the pstmt, segmenting the derived data through segmentation marks, splicing the derived data obtained each time by using a TextFileWriter.
In order to adapt to external data loading of the gaussian database, before parsing the data processing configuration file, the method further includes:
establishing a point-to-point GTP transmission task, and transmitting the file to be loaded into the Gaussian database to an application side receiving directory;
the application side realizes the FSS mounting of the database server and the paas container, so that the file under the receiving directory can be obtained when the paas container runs, and the space of the server is not occupied;
adding an external data loading operation in a distributed batch framework of the Gaussian database, and inserting operation definition and grouping dependency information into the distributed batch framework in an sql import mode; wherein the job definition information: job ID, job name, job execution mode (timed execution, manual execution, dependent job), job grouping, and the like.
Fig. 4 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a fourth embodiment of the present invention, and as shown in fig. 4, the data processing apparatus based on a gaussian database according to the embodiment of the present invention includes an analysis module 401, a connection module 402, and a data processing module 403, where:
the analysis module 401 is configured to analyze the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset; the connection module 402 is configured to connect the gaussian database based on the gaussian database connection configuration information; the data processing module 403 is configured to import external data into the gaussian database or export data from the gaussian database to an external file according to the data processing configuration information.
Specifically, the parsing module 401 may parse the data processing configuration file to obtain data processing configuration information, where the data processing configuration information includes information that needs to be used when importing data or exporting data from the gaussian database. Wherein the data processing profile is preset. The data processing configuration file is a data import configuration file or a data export configuration file. In order to import external data into the gaussian database, a data import configuration file needs to be preset; in order to export data in the gaussian database to an external file, a data export configuration file needs to be set in advance.
The connection module 402 may establish a connection with the gaussian database based on the gaussian database connection configuration information to facilitate subsequent importing or exporting data into or from the gaussian database. The gaussian database connection configuration information is preset and is set according to actual needs, and the embodiment of the invention is not limited.
When the data processing configuration file is a data import configuration file, the data processing module 403 may import the external data into the gaussian database for storage based on the data processing configuration information obtained from the data import configuration file according to the data processing configuration information obtained from the data import configuration file parsing. When the data processing configuration file is a data export configuration file, the server may parse the data processing configuration file to obtain data processing configuration information from the data export configuration file, and export data from the gaussian database to an external file based on the data processing configuration information obtained from the data export configuration file.
The data processing device based on the Gaussian database provided by the embodiment of the invention analyzes the data processing configuration file to obtain the data processing configuration information, connects the Gaussian database based on the Gaussian database connection configuration information, and imports external data into the Gaussian database or exports the data from the Gaussian database to the external file according to the data processing configuration information, thereby realizing the import and export of the data of the Gaussian database.
On the basis of the foregoing embodiments, further, the data processing configuration file is a data import configuration file; correspondingly, the data processing module 403 is specifically configured to:
loading external data according to the imported configuration information, and importing the external data into a data table of the Gaussian database in a data stream mode; wherein the import configuration information is obtained by analyzing a data import configuration file.
On the basis of the foregoing embodiments, further, the data import configuration file includes an external file to be loaded and an import gaussian database file, where:
the external file to be loaded comprises an encoding format, a file content separation type and segmentation data of the external file to be loaded;
the imported Gaussian database file comprises a data table name, a field name, an external file path and an external file name.
Fig. 5 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a fifth embodiment of the present invention, and as shown in fig. 5, on the basis of the foregoing embodiments, further, the data processing module 403 includes an obtaining unit 4031, an obtaining unit 4032, a generating unit 4033, and an executing unit 4034, where:
the obtaining unit 4031 is configured to obtain an import file corresponding to the external file name according to the external file path and the external file name; an obtaining unit 4032 performs streaming processing on the import file to obtain a file byte input stream; the generating unit 4033 generates a data import statement according to the data table name, the partition data, the coding format of the external file to be loaded and a preset SQL statement template; the execution unit 4034 executes the data import statement to import the file byte input stream into the data table corresponding to the data table name and store the data field corresponding to the field name correspondingly.
On the basis of the foregoing embodiments, further, the data processing configuration file is a data export configuration file; correspondingly, the data processing module 403 is specifically configured to:
inquiring data from the Gaussian database according to the export configuration information, and exporting the inquired data to an external file; wherein the export configuration information is parsed from a data export configuration file.
On the basis of the foregoing embodiments, further, the data export configuration file includes:
file name, file path, file encoding format, file content partition type, partition data, database query statement, and export data fields.
Fig. 6 is a schematic structural diagram of a data processing apparatus based on a gaussian database according to a sixth embodiment of the present invention, as shown in fig. 6, on the basis of the foregoing embodiments, further, the data processing module 403 includes a query unit 4035 and a derivation unit 4036, where:
the query unit 4035 is configured to query the gaussian database according to the database query statement and the derived data field to obtain derived data; wherein, the data volume of each derived data is preset; export unit 4036 is configured to store the export data in an external file corresponding to the file name in the file path according to the file encoding format and the partition data.
The embodiment of the apparatus provided in the embodiment of the present invention may be specifically configured to execute the processing flows of the above method embodiments, and the functions of the apparatus are not described herein again, and refer to the detailed description of the above method embodiments.
It should be noted that the data processing method and apparatus based on the gaussian database provided in the embodiment of the present invention may be used in the financial field, and may also be used in any technical field other than the financial field.
Fig. 7 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 7, the electronic device may include: a processor (processor) 701, a communication Interface (Communications Interface) 702, a memory (memory) 703 and a communication bus 704, wherein the processor 701, the communication Interface 702 and the memory 703 complete communication with each other through the communication bus 704. The processor 701 may call logic instructions in the memory 703 to perform the following method: analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset; connecting a Gaussian database based on the Gaussian database connection configuration information; and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file.
In addition, the logic instructions in the memory 703 can be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
The present embodiment discloses a computer program product comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example, including: analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset; connecting a Gaussian database based on the Gaussian database connection configuration information; and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file.
The present embodiment provides a computer-readable storage medium, which stores a computer program, where the computer program causes the computer to execute the method provided by the above method embodiments, for example, the method includes: analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset; connecting a Gaussian database based on the Gaussian database connection configuration information; and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (11)
1. A data processing method based on a Gaussian database is characterized by comprising the following steps:
analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset;
connecting a Gaussian database based on the Gaussian database connection configuration information;
and according to the data processing configuration information, importing external data into a Gaussian database or exporting data from the Gaussian database to an external file.
2. The data processing method of claim 1, wherein the data processing configuration file is a data import configuration file; accordingly, the importing external data into the gaussian database according to the data processing configuration information comprises:
loading external data according to the imported configuration information, and importing the external data into a data table of the Gaussian database in a data stream mode; wherein the import configuration information is obtained by analyzing a data import configuration file.
3. The data processing method according to claim 2, wherein the data import configuration file comprises an external file to be loaded and an import gaussian database file, wherein:
the external file to be loaded comprises an encoding format, a file content separation type and segmentation data of the external file to be loaded;
the imported Gaussian database file comprises a data table name, a field name, an external file path and an external file name.
4. The data processing method according to claim 3, wherein the loading external data according to the import configuration information and importing the external data into the data table of the Gaussian database in a data stream form comprises:
acquiring an import file corresponding to the external file name according to the external file path and the external file name;
carrying out fluidization processing on the imported file to obtain a file byte input stream;
generating a data import statement according to the data table name, the segmentation data, the coding format of the external file to be loaded and a preset SQL statement template;
and executing the data import statement to import the file byte input stream into the data table corresponding to the data table name, and correspondingly storing the data fields corresponding to the field names.
5. The data processing method of any one of claims 1 to 4, wherein the data processing profile is a data export profile; accordingly, the exporting of data from within the gaussian database to an external file according to the data processing configuration information comprises:
inquiring data from the Gaussian database according to the export configuration information, and exporting the inquired data to an external file; wherein the export configuration information is parsed from a data export configuration file.
6. The data processing method of claim 5, wherein the data export profile comprises:
file name, file path, file encoding format, file content partition type, partition data, database query statement, and export data field.
7. The data processing method of claim 6, wherein the querying data from the Gaussian database according to the export configuration information and exporting the queried data to a file comprises:
inquiring from a Gaussian database to obtain derived data according to the database query statement and the derived data field; wherein, the data volume of each derived data is preset;
and storing the derived data into an external file corresponding to the file name under the file path according to the file coding format and the segmentation data.
8. A gaussian database-based data processing apparatus, comprising:
the analysis module is used for analyzing the data processing configuration file to obtain data processing configuration information; wherein the data processing configuration file is preset;
the connection module is used for connecting the Gaussian database based on the Gaussian database connection configuration information;
and the data processing module is used for importing external data into the Gaussian database or exporting data from the Gaussian database to an external file according to the data processing configuration information.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.
11. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210977097.6A CN115238010A (en) | 2022-08-15 | 2022-08-15 | Data processing method and device based on Gaussian database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210977097.6A CN115238010A (en) | 2022-08-15 | 2022-08-15 | Data processing method and device based on Gaussian database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115238010A true CN115238010A (en) | 2022-10-25 |
Family
ID=83679345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210977097.6A Pending CN115238010A (en) | 2022-08-15 | 2022-08-15 | Data processing method and device based on Gaussian database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115238010A (en) |
-
2022
- 2022-08-15 CN CN202210977097.6A patent/CN115238010A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209004B (en) | Code conversion method and device | |
| US8997070B2 (en) | Extension mechanism for scripting language compiler | |
| CN110555030A (en) | SQL statement processing method and device | |
| CN110888842A (en) | File storage method, file query method, file storage device, file query device and file query equipment | |
| CN109062906B (en) | Translation method and device for program language resources | |
| CN112130830B (en) | Interface generation method and device and electronic equipment | |
| CN111858760A (en) | Data processing method and device for heterogeneous database | |
| CN110673839A (en) | Distributed tool configuration construction generation method and system | |
| CN112860730A (en) | SQL statement processing method and device, electronic equipment and readable storage medium | |
| CN112364052A (en) | Heterogeneous data management method, device, equipment and computer readable storage medium | |
| CN111443901A (en) | Business expansion method and device based on Java reflection | |
| CN112052011A (en) | Method and device for combining small programs, electronic equipment and medium | |
| CN114900570A (en) | Standardized data acquisition and transmission method and system | |
| CN111124382A (en) | Attribute assignment method and device in Java and server | |
| CN114218266A (en) | Data query method and device, electronic equipment and storage medium | |
| CN108959294B (en) | Method and device for accessing search engine | |
| CN112000690B (en) | Method and device for analyzing structured operation statement | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN112597105A (en) | Processing method of file associated object, server side equipment and storage medium | |
| CN110555178B (en) | Data proxy method and device | |
| CN115238010A (en) | Data processing method and device based on Gaussian database | |
| CN112541001A (en) | Data query method, device, storage medium and equipment | |
| CN107463618B (en) | Index creating method and device | |
| CN110688355A (en) | Method and device for changing container state | |
| US10114864B1 (en) | List element query support and processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |