CN107992498B

CN107992498B - Method and system for importing data into data warehouse

Info

Publication number: CN107992498B
Application number: CN201610954546.XA
Authority: CN
Inventors: 阮华; 何瑞; 曾凡; 万志颖; 李家昌; 史晓茸
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2016-10-27
Filing date: 2016-10-27
Publication date: 2021-10-22
Anticipated expiration: 2036-10-27
Also published as: CN107992498A

Abstract

The invention provides a method and a system for importing data into a data warehouse. By encapsulating the data import process and providing the webpage supporting the data import function for the user, the user can realize data import without knowing the data import details, so that the data import becomes simple and easy, and the user experience is enhanced. In the data import process, the website server receives and stores the data packet uploaded by the user, the interface machine and the data warehouse are communicated with the data processing server, the technical effect of importing the data in the interface machine into the data warehouse is achieved under the scheduling of the data processing server, and the website server, the data processing server, the interface machine and the data warehouse respectively play their roles, so that the coupling degree of the system is reduced, the stability is improved, and once the data import failure occurs, the reason of the failure can be quickly positioned.

Description

Method and system for importing data into data warehouse

Technical Field

The invention relates to the field of data processing, in particular to a method and a system for importing data into a data warehouse.

Background

Compared with the traditional data warehouse, the distributed data warehouse has stronger data processing and storage and processing capacity, and can support the storage and calculation of hundreds PB level data, thereby providing massive, efficient and stable professional level big data platform support and decision support for various services.

However, because of the high expertise and rigor of data warehouses, it is difficult to import data into a data warehouse. Generally, the method for importing data into the data warehouse is as follows: data are collected through an interactive client capable of directly communicating with a data warehouse, and the data are transmitted to the data warehouse in a command line or packet sending mode.

It can be seen that the existing method for importing data into a data warehouse is not convenient enough, and has high requirements for personnel performing data import, and it is difficult for personnel unfamiliar with relevant logics interacting with the data warehouse to import data into the data warehouse.

Disclosure of Invention

In order to solve the technical problem, the invention provides a method and a system for importing data into a data warehouse. The invention is realized by the following technical scheme:

a method of importing data into a data warehouse, the method comprising:

receiving a data packet forwarded by a website server; the website server responds to a data uploading request of a user, receives and forwards a data packet uploaded by the user, wherein the data packet comprises basic data information and a data file for storing data; the basic data information comprises a database name of a database for storing data and a table name of a data table for storing data;

constructing a data table for storing data in a data warehouse;

obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path;

and importing the data stored in the data file under the target path in the interface machine into the data table.

Further, the website server further performs the following steps:

storing the data packet;

generating a data forwarding request according to the basic information of the user and the basic data information;

sending the data forwarding request and waiting for a response;

judging whether to allow the data packet to be forwarded according to the response:

if yes, forwarding the data packet;

otherwise, the user is informed of the data import failure.

Further, the constructing a data table for storing data in the data warehouse comprises:

generating an instruction for creating a data table according to the data basic information and a structured query language;

executing the instructions for creating the data table in a data warehouse.

Further, still include:

obtaining an execution result of the instruction for creating the data table;

judging whether the execution is successful:

if not, the user is informed that the data import fails.

Further, the obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path further includes:

obtaining a storage format of the data file;

judging whether the storage format is legal or not;

if not, then:

the data in the data file is stored again in a legal storage format to obtain a new data file;

deleting the illegal data file;

and transferring the new data file to the target path.

judging whether the data file is transferred to the target path successfully:

if not, the user is informed that the data import fails.

Further, the importing the data stored in the data file under the target path in the interface machine into the data table includes:

obtaining an import data interface of the data warehouse, wherein the import data interface is used for importing the data stored in the interface machine into the data table;

assigning a form parameter of the imported data interface according to the basic data information and the target path;

and calling the import data interface.

Further, the importing the data stored in the data file under the target path in the interface machine into the data table further includes:

obtaining a calling result of the import data interface:

judging whether the calling fails:

if so, notifying the user that the data import fails;

if not, the user is informed that the data import is successful.

A system for importing data into a data warehouse comprises a website server, a data processing server, an interface machine and the data warehouse; the data warehouse and the interface machine are in communication connection with the data processing server; the website server is in communication connection with the data processing server;

the website server includes:

the monitoring module is used for monitoring the data uploading request;

the request processing module is used for receiving a data packet uploaded by a user and forwarding the data packet to a data processing server;

the data processing server includes:

the receiving module is used for receiving the data packet forwarded by the website server;

the building module is used for building a data table for storing data in the data warehouse;

the unloading module is used for obtaining a target path in the interface machine according to the basic information of the user and the basic information of the data and unloading the data file to the target path;

and the import module is used for importing the data in the data file stored in the interface machine under the target path into the data table.

Further, the request processing module includes:

the storage unit is used for storing the data packet;

a data forwarding request generating unit, configured to generate a data forwarding request according to the basic information of the user and the basic data information;

the data forwarding unit is used for sending the data forwarding request and waiting for a response;

a response result first judging unit, configured to judge whether the data processing server allows forwarding the data packet;

and the response result first processing unit is used for processing according to the output result of the response result first judging unit.

Further, the building module comprises:

the instruction generating unit is used for generating an instruction for creating a data table according to the data basic information and the structured query language;

an execution unit to execute the instruction to create a data table in a data store.

Further, the building module further comprises:

a response result second judgment unit for obtaining an execution result of the instruction for creating the data table;

and the response result second processing unit is used for processing according to the output result of the response result second judging unit.

Further, the unloading module comprises:

the target path acquisition unit is used for acquiring a target path in the interface machine according to the basic information and the basic data information of the user;

and the unloading unit is used for unloading the data packet to the target path.

Further, the unloading module further comprises:

a storage format obtaining unit for obtaining a storage format of the data file;

the legality judging unit is used for judging whether the storage format of the data file is legal or not;

the legality processing unit is used for restoring the data in the data file in a legal storage format to obtain a new data file;

and the deleting unit is used for deleting the illegal data files.

Further, the unloading module further comprises:

a third judging unit of response result, configured to judge whether forwarding the data packet to the target path is successful;

and the response result third processing unit is used for processing according to the output result of the response result third judging unit.

Further, the import module includes:

the acquisition unit is used for acquiring an import data interface of the data warehouse;

the assignment unit is used for assigning the form parameters of the imported data interface according to the basic data information and the target path;

and the calling unit is used for calling the import data interface.

Further, the import module further includes:

the result acquisition unit is used for acquiring the calling result of the imported data interface;

a response result fourth judgment unit for judging whether the call of the import data interface fails;

and the response result fourth processing unit is used for processing according to the output result of the response result fourth judging unit.

The invention provides a method and a system for importing data into a data warehouse, which have the following beneficial effects:

(1) by encapsulating the data import process and providing a webpage supporting the data import function for a user, the user can realize data import without knowing data import details, so that the data import becomes simple and easy to implement, and the user experience is enhanced;

(2) in the data import process, a website server receives and stores data packets uploaded by a user, the interface machine and the data warehouse are communicated with the data processing server, the technical effect of importing the data in the interface machine into the data warehouse is achieved under the scheduling of the data processing server, and the website server, the data processing server, the interface machine and the data warehouse respectively play their roles, so that the coupling degree of the system is reduced, the stability is improved, and once the data import failure occurs, the reason of the failure can be quickly positioned.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flowchart of a method for importing data into a data warehouse according to embodiment 1 of the present invention;

fig. 2 is a schematic diagram of a web page filled with basic data information provided in embodiment 2 of the present invention;

fig. 3 is a flowchart of a method for forwarding a data packet by a web server according to embodiment 2 of the present invention;

fig. 4 is a flowchart of a data forwarding response generation method according to embodiment 2 of the present invention;

fig. 5 is a flowchart of a data file unloading method according to embodiment 2 of the present invention;

FIG. 6 is a flowchart of a method for creating a data table according to embodiment 3 of the present invention;

fig. 7 is a flowchart of a data importing method according to embodiment 3 of the present invention;

FIG. 8 is a block diagram of a system for importing data into a data warehouse provided by embodiment 4 of the present invention;

fig. 9 is a block diagram of a web server according to embodiment 4 of the present invention;

fig. 10 is a block diagram of a data processing server provided in embodiment 4 of the present invention;

FIG. 11 is a block diagram of a building block provided in embodiment 4 of the present invention;

FIG. 12 is a block diagram of a dump module provided in embodiment 4 of the present invention;

fig. 13 is a block diagram of an import module provided in embodiment 4 of the present invention;

fig. 14 is a block diagram of a terminal provided in embodiment 6 of the present invention;

fig. 15 is a block diagram of a server according to embodiment 7 of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1:

an embodiment of the present invention provides a method for importing data into a data warehouse, as shown in fig. 1, where the method includes:

and S101, receiving a data packet forwarded by the website server.

The website server responds to a data uploading request of a user, receives and forwards a data packet uploaded by the user, wherein the data packet comprises basic data information and a data file for storing data.

The website server is a server residing on the internet, and can provide documents to website clients such as browsers and the like, and can also place website files for the website clients to browse or download, and at present, three mainstream website servers are Apache, Nginx and IIS.

The website server displays a webpage supporting the data import function to a user, the user can open the webpage supporting the data import function by accessing a webpage address provided by the website server, and basic data information and an address of an input data file are filled in the webpage, so that the basic data information and a local data file of the user can be uploaded to the website server in a data packet mode.

The data basic information includes a library name of a database for storing data and a table name of a data table for storing data. The library name and the table name are core fields for realizing data import and are not required to be null. The fields necessary to construct a data table in a data warehouse are used to indicate to which data table in a particular database the data is imported.

And S102, constructing a data table for storing the data packet in a data warehouse.

Both ordinary data warehouses and distributed data warehouses typically store data based on a database. Taking a distributed data warehouse as an example, the distributed data warehouse may be customized based on open source software, including but not limited to Hadoop, Hive, PostgreSQL.

Hadoop is a distributed system infrastructure developed by the Apache Foundation, allowing users to develop distributed programs without knowing the details of the distributed bottom layer. The power of the cluster is fully utilized to carry out high-speed operation and storage. Hadoop realizes a Distributed File System (Hadoop Distributed File System), which is called HDFS for short. HDFS is characterized by high fault tolerance and is designed for deployment on inexpensive (low-cost) hardware; and it provides high throughput (high throughput) to access application data, suitable for applications with very large data sets (large data sets). The HDFS can access data in a (streaming access) file system in the form of a stream. The Hadoop distributed file system HDFS can provide high-throughput data access and is very suitable for application on large-scale data sets.

Hive provides the query and management of large datasets stored in a distributed fashion, which itself builds on top of Hadoop. PostgreSQL is an excellent object relational database management system that participates in the composition of a data warehouse and manages data in the data warehouse in the form of data tables. The data warehouse relies on a database for data management, which stores data in the form of data tables, and therefore, in order to import the data in S101 into the data warehouse, it is necessary to construct a data table for storing the data in the data warehouse.

And S103, obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path.

The interface machine is a node of a distributed file system HDFS in a data warehouse for storing files and data, and can be used for importing the data into the data table in the data warehouse and exporting the calculation result obtained from the data warehouse.

And S104, importing the data in the data file stored in the interface machine under the target path into the data table.

And (4) storing the data stored in the interface machine in the S103 into the data table created in the data warehouse in the S102, so that the data can be imported. In this embodiment, the processes in S102 to S104 are all encapsulated, and only the web pages supporting the data import function provided in S101 are exposed to the user, and the user does not need to pay attention to how the data is imported into the data warehouse, and only needs to provide the data to be imported and the basic information of the data.

The embodiment provides a method for importing data into a data warehouse, a user can import data without knowing the relevant logic of the data import data warehouse, the requirement on the user is obviously reduced, and the convenience and the usability of importing data into the data warehouse are obviously improved.

Example 2:

the embodiment of the invention provides a method for importing data into a data warehouse, which comprises the following steps:

s201, receiving a data packet forwarded by the website server.

The website server responds to a data uploading request of a user, receives and forwards a data packet uploaded by the user, wherein the data packet comprises basic data information and a data file for storing data. The data basic information includes a library name of a database for storing data and a table name of a data table for storing data. The library name and the table name are core fields for realizing data import and are not required to be null. The fields necessary to construct a data table in a data warehouse are used to indicate to which data table in a particular database the data is imported.

The website server displays a webpage supporting the data import function to a user, the user can open the webpage supporting the data import function by accessing a webpage address provided by the website server, the webpage filled with basic data information is shown in fig. 2, besides necessary contents, the webpage of the embodiment also provides other contents filled by the user, such as a text name, a business belonging to the webpage, a maximum storage day, a responsible person, a data description and the like in a table, and the contents are helpful for managing data uploaded by the user in a data warehouse.

Further, as shown in fig. 3, the website server further performs the following steps:

A1. and storing the data packet.

A2. And generating a data forwarding request according to the basic information of the user and the basic data information.

A3. Sending the data forwarding request and waiting for a response.

Specifically, the recipient of the data packet determines whether to allow the data packet to be received. As shown in fig. 4, the determination method includes:

B1. checking whether the user has the authority of uploading the data packet;

B2. if not, the data packet is not allowed to be forwarded, and a request failure response is returned;

B3. otherwise, checking whether the table name exists in the data warehouse;

B4. if yes, the data packet is not allowed to be forwarded, and a request failure response is returned;

B5. otherwise, checking whether a legal position for placing the data packet exists;

B6. if not, the data packet is not allowed to be forwarded, and a request failure response is returned;

B7. otherwise, the data packet is allowed to be forwarded, and a request success response is returned.

The request failure responses all carry parameters used for identifying the reason of the request failure, and the website server displays the failure reason to the user according to the parameters.

A4. And judging whether to allow the data packet to be forwarded or not according to the response.

A5. And if so, forwarding the data packet.

A6. Otherwise, the user is informed of the data import failure, and the reason of the import failure is displayed.

S202, constructing a data table for storing the data packet in a data warehouse.

S203, obtaining a target path in the interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path.

And S204, importing the data in the data file stored in the interface machine under the target path into the data table.

In this embodiment, three types of data packets are specifically supported to be imported into the data warehouse, which are a number packet, an IP address packet, and a custom data packet.

(1) Number bag

In some application scenarios, for example, it is necessary to analyze whether some numbers are illegal numbers, and to acquire or change information related to some numbers, it is necessary to query the data of the data repository for the coverage of some numbers, that is, it is necessary to import numbers into the data repository, where the numbers include, but are not limited to, ID numbers for logging in various systems, and phone numbers and fax numbers for communication.

If the imported data packet is a number packet, the constructed database table also comprises a number field besides two necessary fields of the library name and the table name, and the field type can be defined as a character string type. The data file may be a text file, each number being separated by a line break.

(2) IP address packet

In some application scenarios, for example, whether some IP addresses are illegal IP addresses needs to be analyzed, and information related to some IP addresses needs to be acquired or changed, the coverage condition of data in the data warehouse on some IP addresses needs to be queried, that is, the IP addresses need to be imported into the data warehouse.

If the imported data packet is an IP address packet, the constructed database table also comprises IP address fields except two necessary fields of the library name and the table name, and the field type can be defined as a character string type. The data file may be a text file, each number being separated by a line break.

(3) Custom data packet

In some application scenarios, for example, other information with small data volume such as countries, cities, enterprises and the like needs to be analyzed, custom data packages need to be imported into the data warehouse.

If the imported data package is a custom data package, the data file of this embodiment may be in a CSV file form and a plain text file form. CSV files are composed of any number of records, and the records are separated by a certain linefeed character; each record is made up of fields, with separators between fields being other characters or strings, most commonly comma or tab plain text files encoded using UTF-8(8-bit Unicode Transformation Format is a variable length character code for Unicode, also known as ten thousand codes). In this embodiment, the data files in the custom data packet may store three types of data, namely, a string type, an integer type, and a floating point type.

The data files in the custom data packet need to be organized according to a preset standard, such as using line break interval data or using tab interval data. The purpose is to ensure that the data stored in the data file can be correctly parsed and thus correctly imported into the data table.

Further, in S203, as shown in fig. 5, the method further includes:

s2031, obtaining a storage format of the data file;

s2032, judging whether the storage format is legal or not;

s2033. if not, then:

s2034, storing the data in the data file again in a legal storage format to obtain a new data file;

s2035, deleting the illegal data file;

s2036, the new data file is transferred to the target path.

The format of the data file that the interface machine can receive may be different from the format of the data file uploaded by the user, and only the file that is successfully transferred in the interface machine can be imported into the data warehouse in step S204. Therefore, in this embodiment, a step of checking the validity of the data file is provided before the data file is transferred, and illegal data files are converted in time, so that the success rate of importing the data files into the data warehouse is improved.

The embodiment provides another method for importing data into a data warehouse, and in the embodiment, various data can be uploaded to the data warehouse, and validity check is performed on data files, so that the compatibility and robustness of the technical scheme of importing data into the data warehouse are improved.

Example 3:

s301, receiving the data packet forwarded by the website server.

The website server responds to a data uploading request of a user, receives and forwards a data packet uploaded by the user, wherein the data packet comprises basic data information and a data file for storing data. The data basic information includes a library name of a database for storing data and a table name of a data table for storing data.

S302, constructing a data table for storing the data packet in a data warehouse.

Specifically, in this embodiment, the data table is created by interacting with the data warehouse, as shown in fig. 6, the method includes the following steps:

s3021, generating an instruction for creating a data table according to the data basic information and the structured query language.

The basic data information includes a library name and a table name, and may also include several fields, and an SQL statement for creating a data table is constructed according to these contents and the syntax of a Structured Query Language (SQL).

S3022, the instruction for creating the data table is executed in the data warehouse through interaction with the data warehouse.

In this embodiment, an interactive client may be used to interact with the data warehouse in the form of an input command, and of course, other ways may also be used to interact with the data warehouse, such as CGI request. CGI is an interface standard between external applications (CGI programs) and Web servers, a tool for the server to interact with programs on other machines.

S3023, acquiring the execution condition of the instruction for creating the data table;

s2034, judging whether the execution is successful or not, and feeding back an execution result.

Specifically, it is determined whether the execution was successful: if not, the website server is informed of the data import failure, and the website server is informed of the user data import failure and the reason of the data import failure.

And S303, obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path.

In the embodiment, an account is opened on an interface machine for users with data uploading authority, each user corresponds to a root directory, and the root directories can be obtained according to the basic information of the users; the specific position in the root directory can be obtained according to the basic data information, and the target path can be spliced by the root directory and the specific position.

For example, the root directory is hdfs:// ip: port/xxxxx/xxxx, and the library name in the basic data information is: test _ db, table name is also: test _ tb, the target path is hdfs:// ip port/xxxxx/xxxx/test _ db/test _ tb

Specifically, the data file uploaded by the user is uploaded to a target path in the interface machine through a Hadoop command. The Hadoop command is an interface provided by Hadoop and used for operating files in Hadoop.

Specifically, the method further includes determining whether the data file is successfully transferred to the target path:

if not, the website server is informed of the data import failure, and the website server is informed of the user data import failure and the reason of the data import failure.

S304, importing the data in the data file stored in the interface machine under the target path into the data table.

Specifically, as shown in fig. 7, the method includes:

s3041, obtaining an import data interface of the data warehouse, wherein the import data interface is used for importing data stored in an interface machine into the data table;

s3042, assigning a form parameter of the import data interface according to the basic data information and the target path;

s3043, calling the import data interface.

S3044, obtaining a calling result of the import data interface:

s3045, judging whether the calling fails:

s3046, if yes, informing the website server of data import failure, and informing the user of the data import failure and the reason of the data import failure by the website server;

s3047, if not, informing the website server that the data is successfully imported, and informing the user that the data is successfully imported by the website server.

The embodiment provides a method for importing data into a data warehouse, which realizes the rapid import of small data in the data warehouse and provides a function of importing data by one key for a user.

Example 4:

the embodiment provides a system for importing data into a data warehouse, as shown in fig. 8, including a web server 401, a data processing server 402, an interface machine 403, and a data warehouse 404; the data warehouse 404 and the interface machine 403 are both in communication connection with the data processing server 402; the website server 401 is in communication connection with the data processing server 402.

As shown in fig. 9, the website server 401 includes:

the monitoring module 4011 is configured to monitor the request for uploading data;

the request processing module 4012 is configured to receive a data packet uploaded by a user and forward the data packet to the data processing server.

The website server 401 is configured to receive a user request from the monitoring module 4011 and process the user request through the request processing module 4012.

Further, the request processing module 4012 includes:

the storage unit 40121 is configured to store the data packet;

a data forwarding request generating unit 40122, configured to generate a data forwarding request according to the basic information of the user and the basic information of the data;

a data forwarding unit 40123, configured to send the data forwarding request and wait for a response;

a response result first judgment unit 40124 configured to judge whether the data processing server 402 allows forwarding of the data packet;

a response result first processing unit 40125 configured to perform processing according to an output result of the response result first judging unit 40124.

Specifically, the request processing module 4012 is configured to forward the data packet to the data processing server 402, and perform data processing according to an interaction result with the data processing server 402.

As shown in fig. 10, the data processing server 402 includes:

a receiving module 4021, configured to receive the data packet forwarded by the website server.

A building module 4022, configured to build a data table for storing data in the data warehouse.

As shown in fig. 11, the building module 4022 includes:

an instruction generating unit 40221, configured to generate an instruction for creating a data table according to the data basic information and a structured query language;

an executing unit 40222, configured to execute the instruction for creating the data table in the data warehouse.

A response result second judging unit 40223 for obtaining an execution result of the instruction for creating the data table;

a response result second processing unit 40224, configured to perform processing according to an output result of the response result second judging unit 40223.

A dump module 4023, configured to obtain a target path in the interface machine according to the basic information of the user and the basic information of the data, and dump the data file to the target path.

As shown in fig. 12, the unloading module 4023 includes:

a target path obtaining unit 40231, configured to obtain a target path in the interface machine according to the basic information of the user and the basic data information;

a dump unit 40232, configured to dump the data packet to the destination path.

A storage format acquisition unit 40233, configured to obtain a storage format of the data file;

a validity judging unit 40234, configured to judge whether a storage format of the data file is valid;

the legality processing unit 40235 is configured to restore data in the data file in a legal storage format to obtain a new data file;

a deleting unit 40236 is configured to delete an illegal data file.

A third response result determining unit 40237, configured to determine whether forwarding the packet to the target path is successful;

a response result third processing unit 40238 configured to perform processing according to an output result of the response result third judging unit 40237.

Specifically, the storage format obtaining unit 40233, the legitimacy determining unit 40234, the legitimacy processing unit 40235, and the deleting unit 40236 are configured to ensure that a legitimate data file is transmitted to the interface machine 403, thereby ensuring a success rate of data import.

An importing module 4024, configured to import data stored in the data file in the target path in the interface machine into the data table.

Further, as shown in fig. 13, the import module 4024 includes:

an obtaining unit 40241, configured to obtain an import data interface of the data warehouse;

an assignment unit 40242, configured to assign a form parameter of the import data interface according to the basic data information and the target path;

a calling unit 40243, configured to call the import data interface.

A result obtaining unit 40244, configured to obtain a call result of the import data interface;

a response result fourth judging unit 40245, configured to judge whether the call of the import data interface fails;

a response result fourth processing unit 40246 for performing processing according to the output result of the response result fourth judgment unit 40245.

The embodiment and the method embodiment are based on the same inventive concept, and provide a system for importing data into a data warehouse. The embodiment and the method embodiment both provide a data warehouse-based data rapid import function, and actually, a specific data import process is encapsulated through a technical means, so that a user-oriented one-click data import function is provided.

Example 5:

the embodiment of the invention also provides a storage medium. Optionally, in this embodiment, the storage medium may be configured to store program codes executed by the method for importing data into a data warehouse according to the above embodiment.

Optionally, in this embodiment, the storage medium may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

firstly, receiving a data packet forwarded by a website server; the website server responds to a data uploading request of a user, receives and forwards a data packet uploaded by the user, wherein the data packet comprises basic data information and a data file for storing data; the basic data information comprises a database name of a database for storing data and a table name of a data table for storing data;

secondly, constructing a data table for storing data in a data warehouse;

and thirdly, obtaining a target path in the interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path.

And fourthly, importing the data in the data file stored in the interface machine under the target path into the data table.

Optionally, the storage medium is further arranged to store program code for performing the steps of:

a first step of storing the data packet;

secondly, generating a data forwarding request according to the basic information of the user and the basic data information;

thirdly, sending the data forwarding request and waiting for a response;

fourthly, judging whether to allow the data packet to be forwarded according to the response:

fifthly, if yes, forwarding the data packet;

and sixthly, if not, informing the user that the data import fails.

the constructing a data table for storing data in the data warehouse comprises:

firstly, generating an instruction for creating a data table according to the basic data information and a structured query language;

and secondly, executing the instruction for creating the data table in the data warehouse.

thirdly, obtaining an execution result of the instruction for creating the data table;

fourthly, judging whether the execution is successful:

and fifthly, if not, informing the user that the data import fails.

the obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path further comprises:

the first step, obtaining the storage format of the data file;

secondly, judging whether the storage format is legal or not;

step three, if not, the method comprises the following steps:

fourthly, the data in the data file is stored again in a legal storage format to obtain a new data file;

fifthly, deleting the illegal data files;

and sixthly, transferring the new data file to the target path.

seventhly, judging whether the data file is transferred to the target path successfully:

and step eight, if not, notifying the user that the data import fails.

the importing the data stored in the data file under the target path in the interface machine into the data table comprises:

the method comprises the steps that firstly, an import data interface of a data warehouse is obtained, and the import data interface is used for importing data stored in an interface machine into a data table;

secondly, assigning a form parameter of the import data interface according to the basic data information and the target path;

and step three, calling the import data interface.

the importing the data stored in the data file under the target path in the interface machine into the data table further comprises:

fourthly, obtaining the calling result of the import data interface:

and step five, judging whether the calling fails:

step six, if yes, informing the user that the data import fails;

and seventhly, if not, informing the user that the data import is successful.

Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

Example 6:

referring to fig. 14, an embodiment of the present invention provides a terminal, which may be used to implement the method for importing data into a data warehouse provided in the foregoing embodiment. Specifically, the method comprises the following steps:

the terminal may include RF (Radio Frequency) circuitry 110, memory 120 including one or more computer-readable storage media, input unit 130, display unit 140, sensor 150, audio circuitry 160, WiFi (wireless fidelity) module 170, processor 180 including one or more processing cores, and power supply 190. Those skilled in the art will appreciate that the terminal structure shown in fig. 14 is not intended to be limiting and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:

the RF circuit 110 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receives downlink information from a base station and then sends the received downlink information to the one or more processors 180 for processing; in addition, data relating to uplink is transmitted to the base station. In general, the RF circuitry 110 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (low noise amplifier), a duplexer, and the like. In addition, the RF circuitry 110 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to GSM (Global System for mobile communications), GPRS (General Packet Radio Service), CDMA (Code Division multiple access), WCDMA (Wideband Code Division multiple access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

The memory 120 may be used to store software programs and modules, and the processor 180 executes various functional applications and data processing by operating the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs required for functions, and the like; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 120 may further include a memory controller to provide the processor 180 and the input unit 130 with access to the memory 120.

The input unit 130 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. In particular, the input unit 130 may include a touch-sensitive surface 131 as well as other input devices 132. The touch-sensitive surface 131, also referred to as a touch display screen or a touch pad, may collect touch operations by a user on or near the touch-sensitive surface 131 (e.g., operations by a user on or near the touch-sensitive surface 131 using a finger, a stylus, or any other suitable object or attachment), and drive the corresponding connection device according to a predetermined program. Alternatively, the touch sensitive surface 131 may comprise two parts, a touch detection means and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 180, and can receive and execute commands sent by the processor 180. Additionally, the touch-sensitive surface 131 may be implemented using various types of resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch-sensitive surface 131, the input unit 130 may also include other input devices 132. In particular, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 140 may be used to display information input by or provided to a user and various graphic user interfaces of the terminal, which may be configured by graphics, text, icons, video, and any combination thereof. The Display unit 140 may include a Display panel 141, and optionally, the Display panel 141 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Further, the touch-sensitive surface 131 may cover the display panel 141, and when a touch operation is detected on or near the touch-sensitive surface 131, the touch operation is transmitted to the processor 180 to determine the type of the touch event, and then the processor 180 provides a corresponding visual output on the display panel 141 according to the type of the touch event. Although in FIG. 14, touch-sensitive surface 131 and display panel 141 are shown as two separate components to implement input and output functions, in some embodiments, touch-sensitive surface 131 may be integrated with display panel 141 to implement input and output functions.

The terminal may also include at least one sensor 150, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel 141 according to the brightness of ambient light, and a proximity sensor that turns off the display panel 141 and/or a backlight when the terminal is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when the terminal is stationary, and can be used for applications of recognizing terminal gestures (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured in the terminal, detailed description is omitted here.

Audio circuitry 160, speaker 161, microphone 162 may provide an audio interface between a user and the terminal. The audio circuit 160 may transmit the electrical signal converted from the received audio data to the speaker 161, and convert the electrical signal into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electric signal, converts the electric signal into audio data after being received by the audio circuit 160, and then outputs the audio data to the processor 180 for processing, and then to the RF circuit 110 to be transmitted to, for example, another terminal, or outputs the audio data to the memory 120 for further processing. The audio circuit 160 may also include an earbud jack to provide communication of peripheral headphones with the terminal.

WiFi belongs to a short-distance wireless transmission technology, and the terminal can help a user to send and receive e-mails, browse webpages, access streaming media and the like through the WiFi module 170, and provides wireless broadband internet access for the user. Although fig. 14 shows the WiFi module 170, it is understood that it does not belong to the essential constitution of the terminal, and may be omitted entirely as needed within the scope not changing the essence of the invention.

The processor 180 is a control center of the terminal, connects various parts of the entire terminal using various interfaces and lines, performs various functions of the terminal and processes data by operating or executing software programs and/or modules stored in the memory 120 and calling data stored in the memory 120, thereby performing overall monitoring of the terminal. Optionally, processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 180.

The terminal also includes a power supply 190 (e.g., a battery) for powering the various components, which may preferably be logically coupled to the processor 180 via a power management system to manage charging, discharging, and power consumption management functions via the power management system. The power supply 190 may also include any component including one or more of a dc or ac power source, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

Although not shown, the terminal may further include a camera, a bluetooth module, and the like, which are not described herein again. Specifically, in this embodiment, the display unit of the terminal is a touch screen display, the terminal further includes a memory, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for:

secondly, constructing a data table for storing data in a data warehouse;

Further, the memory of the terminal further contains instructions for:

a first step of storing the data packet;

thirdly, sending the data forwarding request and waiting for a response;

fifthly, if yes, forwarding the data packet;

and sixthly, if not, informing the user that the data import fails.

Further, the memory of the terminal further contains instructions for:

the constructing a data table for storing data in the data warehouse comprises:

Further, the memory of the terminal further contains instructions for:

fourthly, judging whether the execution is successful:

and fifthly, if not, informing the user that the data import fails.

Further, the memory of the terminal further contains instructions for:

the first step, obtaining the storage format of the data file;

secondly, judging whether the storage format is legal or not;

step three, if not, the method comprises the following steps:

fifthly, deleting the illegal data files;

and sixthly, transferring the new data file to the target path.

Further, the memory of the terminal further contains instructions for:

and step eight, if not, notifying the user that the data import fails.

Further, the memory of the terminal further contains instructions for:

and step three, calling the import data interface.

Further, the memory of the terminal further contains instructions for:

fourthly, obtaining the calling result of the import data interface:

and step five, judging whether the calling fails:

step six, if yes, informing the user that the data import fails;

and seventhly, if not, informing the user that the data import is successful.

In summary, the terminal provided by the embodiment of the present invention provides a data warehouse-based data rapid import function, and actually encapsulates a specific data import process by a technical means, thereby providing a user-oriented function of importing data in one key.

Example 7:

referring to fig. 15, an embodiment of the present invention provides a server. The server 700 may vary significantly depending on configuration or performance, and may include one or more Central Processing Units (CPUs) 722 (e.g., one or more processors) and memory 732, one or more storage media 730 (e.g., one or more mass storage devices) storing applications 742 or data 744. Memory 732 and storage medium 730 may be, among other things, transient storage or persistent storage. The program stored in the storage medium 730 may include one or more modules (not shown), each of which may include a series of instruction operations for the server. Further, the central processor 722 may be configured to communicate with the storage medium 730, and execute a series of instruction operations in the storage medium 730 on the server 700. The server 700 may also include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input-output interfaces 758, and/or one or more operating systems 741, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so forth. The steps performed by the above-described method embodiment may be based on the server architecture shown in fig. 15.

It should be noted that: the above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A method of importing data into a data repository for application to a data processing server, the method comprising:

constructing a data table for storing data in a data warehouse;

determining a target root directory corresponding to the basic information of the user in an interface machine, wherein the interface machine is a node of a distributed file system in the data warehouse for storing files and data, and each user in the interface machine corresponds to one root directory;

determining a specific position in the target root directory according to the data basic information;

splicing a target path according to the target root directory and the specific position, and transferring the data file to the target path;

and importing the data in the data file under the target path in the interface machine into the data table.

2. The method of claim 1, wherein the website server further performs the steps of:

storing the data packet;

sending the data forwarding request and waiting for a response;

if yes, forwarding the data packet;

otherwise, the user is informed of the data import failure.

3. The method of claim 1, wherein constructing a data table in the data warehouse for storing data comprises:

executing the instructions for creating the data table in a data warehouse.

4. The method of claim 3, further comprising:

obtaining an execution result of the instruction for creating the data table;

judging whether the execution is successful:

if not, the user is informed that the data import fails.

5. The method according to claim 1, wherein the obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and the transferring the data file to the target path further comprises:

obtaining a storage format of the data file;

judging whether the storage format is legal or not;

if not, then:

deleting the illegal data file;

and transferring the new data file to the target path.

6. The method of claim 5, wherein obtaining a target path in an interface machine according to the basic information of the user and the basic information of the data, and transferring the data file to the target path further comprises:

judging whether the data file is transferred to the target path successfully:

if not, the user is informed that the data import fails.

7. The method of claim 1, wherein importing the data stored in the data file in the interface machine under the target path into the data table comprises:

and calling the import data interface.

8. The method of claim 7, wherein importing the data stored in the data file in the interface machine under the target path into the data table further comprises:

obtaining a calling result of the import data interface:

judging whether the calling fails:

if so, notifying the user that the data import fails;

if not, the user is informed that the data import is successful.

9. A system for importing data into a data warehouse is characterized by comprising a website server, a data processing server, an interface machine and the data warehouse; the data warehouse and the interface machine are in communication connection with the data processing server; the website server is in communication connection with the data processing server;

the website server includes:

the monitoring module is used for monitoring the data uploading request;

the data processing server includes:

the unloading module is used for determining a target root directory corresponding to the basic information of the user in an interface machine, the interface machine is a node of a distributed file system in the data warehouse for storing files and data, and each user in the interface machine corresponds to one root directory; determining a specific position in the target root directory according to the data basic information; splicing a target path according to the target root directory and the specific position, and transferring a data file to the target path;

and the importing module is used for importing the data in the data file under the target path in the interface machine into the data table.

10. The system of claim 9, wherein the request processing module comprises:

the storage unit is used for storing the data packet;

11. The system of claim 9, wherein the building module comprises:

12. The system of claim 11, wherein the building module further comprises:

13. The system of claim 9, wherein the dump module comprises:

14. The system of claim 13, wherein the dump module further comprises:

and the deleting unit is used for deleting the illegal data files.

15. The system of claim 14, wherein the dump module further comprises:

16. The system of claim 9, wherein the import module comprises:

and the calling unit is used for calling the import data interface.

17. The system of claim 16, wherein the import module further comprises:

18. An electronic device comprising a processor and a memory, wherein instructions stored in the memory are loaded and executed by the processor to implement a method of importing data into a data repository as claimed in any of claims 1 to 8.

19. A computer storage medium having stored thereon instructions for loading by a processor to perform a method of importing data into a data warehouse according to any one of claims 1 to 8.