CN114297274A - Big data extraction method and device, computer equipment and storage medium - Google Patents
Big data extraction method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114297274A CN114297274A CN202111470058.9A CN202111470058A CN114297274A CN 114297274 A CN114297274 A CN 114297274A CN 202111470058 A CN202111470058 A CN 202111470058A CN 114297274 A CN114297274 A CN 114297274A
- Authority
- CN
- China
- Prior art keywords
- service system
- target service
- database
- target
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a big data extraction method, a big data extraction device, a computer device, a storage medium and a computer program product. The method comprises the following steps: acquiring system category and table information of a target service system; determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system; determining the field type of a target database of the target service system according to the database type of the target service system; extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system; and reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system. By adopting the method, the data extraction efficiency can be improved.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method and an apparatus for extracting big data, a computer device, a storage medium, and a computer program product.
Background
With the development of data storage technology, a big data extraction technology has appeared in order to realize the transfer storage of mass data, for example, the storage of business data of each business system on a big data platform.
In the conventional technology, when a big data platform is accessed to a target service system, usually during a demand investigation, an operation and maintenance manufacturer investigates and determines a database type of the target service system, writes a corresponding data acquisition script according to the database type of the target service system, and when the big data platform executes the corresponding data acquisition script, extracts corresponding data from the database of the target service system.
However, in the conventional method, when a large data platform accesses a target service system of a different database type, data acquisition scripts corresponding to the database types need to be written respectively, so that the large data platform can extract data in the databases of the target service systems of different types, and thus, the efficiency of data extraction is reduced to a certain extent.
Disclosure of Invention
In view of the above, it is necessary to provide a big data extraction method, apparatus, computer device, computer readable storage medium and computer program product capable of improving data extraction efficiency.
In a first aspect, the present application provides a big data extraction method. The method comprises the following steps:
acquiring system category and table information of a target service system;
determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
determining the field type of a target database of the target service system according to the database type of the target service system;
extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system.
In one embodiment, determining the target database field type of the target business system according to the database type of the target business system includes:
and determining the field type of the target database of the target service system according to the preset mapping relation between the field type of the database and the field type of the target service system.
In one embodiment, reading data in the target service system through a preset data acquisition script and a data dictionary of the target service system includes:
acquiring a connection account password and a connection address of the target service system;
decrypting the connection account password of the target service system to obtain a decrypted connection account password;
determining the drive configuration of the target service system according to a second corresponding relation between a preset database type and the drive configuration and the database type of the target service system;
and driving the data acquisition script to execute through the driving configuration of the target service system according to the data dictionary of the target service system and the decrypted connection account password, and connecting to the connection address to read data in the target service system.
In one embodiment, the obtaining of the connection account password of the target service system includes:
and determining the connection account password of the target service system according to the system type of the target service system and a third corresponding relation between the preset system type and the connection account password.
In one embodiment, decrypting the connection account password of the target service system to obtain a decrypted connection account password includes:
acquiring a ciphertext obtained by encrypting the connection account password by the target service system through a secret key;
and decrypting the ciphertext through the secret key to obtain the decrypted connection account password.
In one embodiment, the method further comprises:
acquiring field types of service systems of different database types;
and respectively generating field types mapped to the target database according to the field types of the tables in each service system to obtain a mapping relation between the preset database field type and the target database field type.
In a second aspect, the application further provides a big data extraction device. The device comprises:
the service system information acquisition module is used for acquiring the system type and the table information of the target service system;
the database type acquisition module is used for determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
the field type determining module is used for determining the field type of a target database of the target service system according to the database type of the target service system;
the data dictionary determining module is used for extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and the data reading module is used for reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
acquiring system category and table information of a target service system;
determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
determining the field type of a target database of the target service system according to the database type of the target service system;
extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system.
In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring system category and table information of a target service system;
determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
determining the field type of a target database of the target service system according to the database type of the target service system;
extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring system category and table information of a target service system;
determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
determining the field type of a target database of the target service system according to the database type of the target service system;
extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and reading the data in the target service system through a preset data acquisition script and the data dictionary of the target service system.
The big data extraction method, the device, the computer equipment, the storage medium and the computer program product are used for obtaining the system type and the table information of the target business system, determining the database type of the target business system according to the system type and the first corresponding relation of the target business system, and determining the field type of the target database of the target business system according to the database type of the target business system, so that the data dictionary of the target business system is generated by extracting the data dictionary script according to the field type of the target database of the target business system and the table information of the target business system, and the data in the target business system can be read according to the data dictionary of the target business system and the preset data acquisition script, and the data extraction of the target business system by the big data platform is completed. For any target service system, even if the types of the databases are different, the data in the databases of the target service systems of different types can be extracted by the big data platform without respectively compiling data acquisition scripts corresponding to the types of the databases, so that the efficiency of extracting the data from the service systems by the big data platform is improved to a certain extent.
Drawings
FIG. 1 is a diagram of an application environment of a big data extraction method in one embodiment;
FIG. 2 is a flow diagram illustrating a big data extraction method according to an embodiment;
FIG. 3 is a flow chart illustrating a big data extraction method according to another embodiment;
FIG. 4 is a block diagram of a big data extraction apparatus according to an embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The big data extraction method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. The data processing system comprises a big data platform 10 and a plurality of business systems 20, wherein the big data platform 10 is used as an enterprise server, the big data platform 10 can be a Hadoop distributed file system, and each business system 20 is a database for storing data of different business scenes and comprises data under a specific business scene. Such as sales business system 20, generate a large amount of sales data. The target business system 20 is the business system 20 which needs to extract data when the big data platform 10 performs data analysis. When responding to a data analysis instruction operated by a user, the big data platform 10 determines the target service system 20 according to the data analysis instruction, and then determines the corresponding database type and the target database field type according to the system type of the target service system 20. And extracting through a data dictionary extraction script according to the field type of the target database and the table information of the target service system 20, thereby generating a data dictionary of the target service system 20. And finally, reading the data in the target service system 20 through a preset data acquisition script and a data dictionary of the target service system 20.
In one embodiment, as shown in fig. 2, a big data extraction method is provided, which is described by taking the example that the method is applied to the big data platform in fig. 1, and includes the following steps:
step 202, system type and table information of the target service system are obtained.
The service system is a database for storing relevant service operations, and the data generated by the service system is different according to different service scenes. The system class of the service system is a service scene class of the service system, and the service scene is divided according to the specifically stored service data. The same service scene can have a plurality of service systems, the corresponding service systems have the same system type, and the database structures of the service systems of the same system type are consistent. The target business system is a business system of the big data platform which needs to extract data for business analysis, and comprises a business system which is newly accessed to the big data platform and a business system with updated data. The data sources are generated by corresponding operations of a large number of users under different service scenes. Taking an electric power enterprise as an example, the business system of the electric power enterprise comprises an electric power sale business system and a power distribution business system, and the electric power sale business system comprises electric power sale data of residents. Among them, the electricity sales data of the residents is sales data generated when the residents purchase electricity. The distribution service system comprises the resident electricity meter number and the electricity meter installation address. When the electric power enterprise has a service analysis requirement, the big data platform responds to the service request by inputting the service request to the big data platform, and determines a target service system from all the service systems and the system type and table information of the target service system. In one embodiment, a service request input by a user is obtained, the service request comprises keywords of area A and resident electricity consumption, the big data platform responds to the service request to determine that a target service system comprises a power distribution service system and a power sale service system, and system category and table information of the power distribution service system and the power sale service system is obtained.
On a big data platform, a service system for transmitting data with the big data platform may be compiled by a plurality of database languages, and each database language may be Mysql, Oracle, SqlServer, and the like.
For various database languages, data is stored in tabular form. The table information of the service system refers to the basic information of a table corresponding to a database language used by the current service system. Optionally, the table information comprises a table name. Moreover, each service system usually includes a large number of table structures, and usually, before the data of the service system is extracted by the big data platform, the table name of the target service system needs to be determined.
Step 204, determining the database type of the target service system according to a first corresponding relationship between a preset system type and the database type and the system type of the target service system.
The system type and the database type have a first corresponding relation, and the big data platform stores the first corresponding relation in advance before extracting data. When extracting the data of the target business system, determining the database type of the target business system according to the first corresponding relation and the system type of the target business system. The first corresponding relation is pre-stored in the big data platform, and the big data platform can determine the type of the database of the target business system when determining the system type of the target business system when extracting data each time in the following. The database type of the service system refers to the type of the database language used by the current service system obtained by compiling.
Step 206, determining the field type of the target database of the target service system according to the database type of the target service system.
The field type of the service system is the field type corresponding to the database used by the current service system obtained by compiling. An external table is created on the big data platform, and the field type of the external table is called a target database field type. In order to store data in a target business system on a big data platform, the field type of the target business system needs to be converted into a field type of a target database. For example, when a big data platform creates an external table in hive, the field type of the target business system needs to be converted into the hive database field type during extraction.
The field types are data types in the database, including Binary data types such as Binary, Varbinary, Image, etc., character data types including Char, Varchar, and Text, and Unicode data types including Nchar, Nvarchar, and Ntext, etc.
In one embodiment, the target database field type of the target business system is determined based on the database type of the target business system. Namely, the field type of the target service system is determined according to the database type of the target service system, and the field type of the target database is determined according to the field type of the target service system.
And 208, extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system.
The data dictionary extraction script can generate the data dictionary of the target service system according to the field type of the target database of the target service system and the table information of the target service system. The data dictionary of the target business system defines and describes data items, data structures, data streams, data stores, processing logic and the like of data in the target business system. The data dictionary of the target business system includes descriptions and definitions of data items, data structures, data storage and processing logic, etc. in the target business system. Optionally, the data structure includes a target database field type.
In one embodiment, the table information includes a home mode and a table name, the home mode, the table name and a target database field type of the target business system extracted by the data dictionary extraction script, thereby generating a data dictionary of the target business system, the data dictionary of the target business system including the home mode, the table name and the target database field type.
The attribution mode and the table name are basic information of the target service system, and the large data platform needs to obtain the information in real time through the data dictionary according to the target service system each time. The preset mapping relation between the database field type and the target database field type is stored on the big data platform, the big data platform does not need to acquire the data in real time according to the target service system every time, but queries according to basic information of the target service system when the data needs to be extracted every time, and therefore the data dictionary of the target service system is obtained.
And step 210, reading data in the target service system through a preset data acquisition script and a data dictionary of the target service system.
The data acquisition script is an extraction script used for extracting data in each database, and comprises an sqoop data extraction script. The data acquisition script is preset on the big data platform, and when the big data platform needs to extract data in the target service system database, the data acquisition script extracts the data in the target service system database through data information displayed by the data dictionary of the target service system. When data are extracted, the big data platform executes the data acquisition script, and the data acquisition script automatically extracts corresponding data according to the input data dictionary of the target business system in the execution process.
In the big data extraction method, the big data platform acquires the system type and the table information of the target business system, determines the database type of the target business system according to the system type and the first corresponding relation of the target business system, and determines the field type of the target database of the target business system according to the database type of the target business system, so that the data dictionary of the target business system is generated by extracting the data dictionary script according to the field type of the target database of the target business system and the table information of the target business system, and the data in the target business system can be read according to the data dictionary of the target business system and the preset data acquisition script, thereby completing the data extraction of the target business system by the big data platform. For any target service system, even if the types of the databases are different, the data in the databases of the target service systems of different types can be extracted by the big data platform without respectively compiling data acquisition scripts corresponding to the types of the databases, so that the efficiency of extracting the data from the service systems by the big data platform is improved to a certain extent.
In one embodiment, determining the target database field type of the target business system according to the database type of the target business system includes: and determining the field type of the target database of the target service system according to the preset mapping relation between the field type of the database and the field type of the target service system.
The attribution mode is a mode of a table in a database, and one database can have a plurality of attribution modes; there may be 0 or more tables for one home mode. For example, the distribution service system includes the electricity meter number of a resident and an industrial electricity meter number, both of which are different in that one attribution mode is a resident and the other attribution mode is an industry. The table names refer to the names of all tables in the database, and the names of all tables should be kept different, so that table lookup can be conveniently carried out according to the table names.
The mapping relation between the database field type and the target database field type is prestored in the big data platform, and the mapping relation refers to the corresponding conversion relation between the database field type of the business system and the target database field type. When determining the database type of the target service system, the big data platform searches the field type of the target database corresponding to the field type of the target service system in the mapping relation between the preset database field type and the field type of the target database. For example, when the big data platform determines that the field type of the database in the target service system is integer data, and finds that the field type of the database in the target service system is byte type in the mapping relationship, the data in the target service system with the field type of the database being integer is converted into byte type data corresponding to the target database.
In this embodiment, the field type of the target database is determined according to a mapping relationship between a preset field type of the database and the field type of the target service system.
In one embodiment, reading data in the target business system through a preset data acquisition script and a data dictionary of the target business system includes: acquiring a connection account password and a connection address of the target service system; decrypting the connection account password of the target service system to obtain a decrypted connection account password; determining the drive configuration of the target service system according to a second corresponding relation between a preset database type and the drive configuration and the database type of the target service system; and driving the data acquisition script to execute through the driving configuration of the target service system according to the data dictionary of the target service system and the decrypted connection account password, and connecting to the connection address to read data in the target service system.
Each service system is provided with a connection account password, and the connection account password is a precondition for acquiring a large amount of actual data in the service system. Each connection account password is usually in an encrypted state, and only the decrypted connection account password can normally acquire data in the service system. When the big data platform extracts data in the target business system, the connection account password and the connection address of the target business system are obtained firstly, and the connection account password is in an encrypted state, so that decryption is needed firstly to obtain the decrypted connection account password.
The big data platform also presets a second corresponding relation between the database types and the drive configuration, and the second corresponding relation comprises the database types, the drive configuration corresponding to the database types and the matching/corresponding relation between the database types and the drive configuration. The driving configuration is an engine for driving data extraction, and is used for driving the data acquisition script to extract data from the target business system. When the big data platform extracts the connection account password of the target service system, the drive configuration corresponding to the target service system needs to be searched in the second corresponding relation according to the database type of the target service system.
The preset data acquisition script can determine the data architecture of the target service system according to the data dictionary of the target service system.
And then, the big data platform connects the data acquisition script to the target service system according to the decrypted connection account password, the data dictionary of the target service system, the connection address and the corresponding drive configuration, and drives the data acquisition script to execute through the drive configuration, wherein the data in the target service system is continuously read in the execution process of the data acquisition script.
In this embodiment, the big data platform decrypts the acquired connection account password of the target service system, connects the address, and configures the driver, so as to drive the data acquisition script to extract the data in the target service system, and extract the data in the target service system into the big data platform.
In one embodiment, the obtaining of the connection account password of the target service system includes: and determining the connection account password of the target service system according to the system type of the target service system and a third corresponding relation between the preset system type and the connection account password.
And before decryption, the connection account password stored in the third corresponding relation is in an encrypted state. And before the big data platform executes the data acquisition script, searching a connection account password corresponding to the target service system from the third corresponding relation.
In this embodiment, by obtaining the connection account password corresponding to the target business system from the third corresponding relationship, the obtained connection account password is encrypted, so that a password plaintext is prevented from being configured on the data acquisition script during data extraction, which is beneficial to improving the security of data extraction.
In one embodiment, decrypting the connection account password of the target service system to obtain a decrypted connection account password includes: acquiring a ciphertext obtained by encrypting the connection account password by the target service system through a secret key; and decrypting the ciphertext through the secret key to obtain the decrypted connection account password.
The target business system encrypts in a symmetric encryption mode, and a connection account password of the target business system is encrypted through a secret key to obtain an encrypted ciphertext. When the big data platform obtains the connection account password of the target service system in an encrypted state through the data acquisition script, the connection account password is decrypted by using the same secret key, and the decrypted connection account password is obtained. And the data acquisition script can extract actual data in the target service system according to the decrypted connection account password.
In this embodiment, the cipher text of the connection account password is decrypted by the key, so that the decrypted connection account password is obtained, and the big data platform is convenient to extract data in the target service system according to the data acquisition script.
In one embodiment, a big data extraction method includes: acquiring field types in service systems of different database types; and respectively generating field types mapped to the target database according to the field types in the service systems to obtain a mapping relation between the preset database field type and the target database field type.
Before the mapping relation between the preset database field type and the target database field type is obtained, the field types of the tables in the service systems of different database types are obtained, the field types mapped to the tables in the target database are respectively generated according to the field types of the tables in the service systems, and the target database field type of each service system is obtained. Generating the field type mapped to the table in the target database refers to analyzing according to the field type of the table in the service system and the field type of the table in the target database, determining the data type which needs to be converted and is stored in the target database by the field type of the table in the service system, obtaining the mapping relation between the preset database field type and the field type of the target database, and pre-storing the mapping relation in a big data platform.
In the embodiment, the field types of the service systems of different database types are respectively generated and mapped to the field type of the target database to obtain the mapping relation between the preset database field type and the field type of the target database, so that the large data platform can conveniently and subsequently generate the data dictionary of the target service system.
In one embodiment, as shown in FIG. 3, the big data extraction method includes steps S10-S40, wherein:
s10: and acquiring basic configuration information, wherein the basic configuration information comprises the system type and the table information of the target service system, and the table information comprises an attribution mode and a table name.
S20: and dynamically generating a data dictionary of the target service system according to the solidified configuration content, the configuration basic information and the data dictionary extraction script.
The solidified configuration content comprises a first corresponding relation between a preset system type and a database type, a mapping relation between a preset database field type and a target database field type, a second corresponding relation between the preset database type and the drive configuration and a third corresponding relation between the preset system type and the connection account password. When the data dictionary of the target business system is generated, only the first corresponding relation and the mapping relation in the solidified configuration content are read.
S30: searching in the second corresponding relation and the third corresponding relation according to the system type of the target service system, determining the drive configuration and the connection account password of the corresponding target service system, and decrypting the connection account password of the target service system to obtain a decrypted connection account password; and driving the data acquisition script to execute through the driving configuration of the target service system according to the decrypted connection account password and the data dictionary of the target service system, and connecting to the connection address to read data in the target service system.
S40: and when the data acquisition script is executed, extracting the data of the target service system to the big data platform.
When the data of the target business system is extracted to the big data platform, the data is grounded, and the data extraction work is completed.
In this embodiment, the big data platform obtains the system type and the table information of the target service system, determines the database type of the target service system according to the system type and the first corresponding relationship of the target service system, and determines the target database field type of the target service system according to the database type of the target service system, so as to extract through the data dictionary script according to the target database field type of the target service system and the table information of the target service system, and further generate the data dictionary of the target service system, and according to the data dictionary of the target service system and the preset data acquisition script, data in the target service system can be read, and further, data extraction of the target service system by the big data platform is completed. For any target service system, even if the types of the databases are different, the data in the databases of the target service systems of different types can be extracted by the big data platform without respectively compiling data acquisition scripts corresponding to the types of the databases, so that the efficiency of extracting the data from the service systems by the big data platform is improved to a certain extent.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a big data extraction device for implementing the big data extraction method mentioned above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the big data extraction device provided below can refer to the limitations on the big data extraction method in the above, and details are not described here.
In one embodiment, as shown in fig. 4, there is provided a big data extracting apparatus including: a service system information obtaining module 301, a database type obtaining module 302, a field type determining module 303, a data dictionary determining module 304 and a data reading module 305, wherein:
a service system information obtaining module 301, configured to obtain a system type and table information of a target service system;
a database type obtaining module 302, configured to determine a database type of the target service system according to a first corresponding relationship between a preset system type and the database type and the system type of the target service system;
a field type determining module 303, configured to determine a target database field type of the target service system according to the database type of the target service system;
a data dictionary determining module 304, configured to extract, according to a field type of a target database of the target service system and table information of the target service system, through a data dictionary extraction script to generate a data dictionary of the target service system;
and a data reading module 305, configured to read data in the target service system through a preset data acquisition script and a data dictionary of the target service system.
In an embodiment, the field type determining module is further configured to determine the field type of the target database of the target service system according to a mapping relationship between a preset field type of the database and the field type of the target database, and the field type of the target service system.
In one embodiment, the data reading module includes an account address obtaining module, an account decryption module, a driver configuration determining module, and a data connection module, wherein:
the account address acquisition module is used for acquiring a connection account password and a connection address of the target service system;
the account number decryption module is used for decrypting the connection account number password of the target business system to obtain a decrypted connection account number password;
the driving configuration determining module is used for determining the driving configuration of the target service system according to a second corresponding relation between a preset database type and the driving configuration and the database type of the target service system;
and the data connection module is used for driving the data acquisition script to execute through the driving configuration of the target service system according to the data dictionary of the target service system and the decrypted connection account password, and connecting to the connection address to read data in the target service system.
In an embodiment, the account address obtaining module is further configured to determine the connection account password of the target service system according to the system type of the target service system and a third corresponding relationship between a preset system type and the connection account password.
In one embodiment, the account decryption module includes a ciphertext acquisition module and a decrypted account acquisition module, where:
the ciphertext acquisition module is used for acquiring a ciphertext obtained by encrypting the connection account password by the target service system through a key;
and the decryption account number acquisition module is used for decrypting the ciphertext through the secret key to obtain the decrypted connection account number password.
In one embodiment, the big data extraction apparatus further includes a field type obtaining module and a mapping relation obtaining module, where:
the field type acquisition module is used for acquiring the field types of the service systems of different database types;
and the mapping relation acquisition module is used for respectively generating the field types mapped to the target database according to the field types of the service systems to obtain the mapping relation between the preset database field type and the target database field type.
The modules in the big data extraction device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a big data extraction method.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A big data extraction method is characterized by comprising the following steps:
acquiring system category and table information of a target service system;
determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type;
determining the field type of a target database of the target service system according to the database type of the target service system;
extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and reading data in a database of the target service system through a preset data acquisition script and a data dictionary of the target service system.
2. The method of claim 1, wherein determining the target database field type of the target business system according to the database type of the target business system comprises:
and determining the field type of the target database of the target service system according to the preset mapping relation between the field type of the database and the field type of the target service system.
3. The method of claim 1, wherein reading data in the database of the target business system through a preset data acquisition script and a data dictionary of the target business system comprises:
acquiring a connection account password and a connection address of the target service system;
decrypting the connection account password of the target service system to obtain a decrypted connection account password;
determining the drive configuration of the target service system according to a second corresponding relation between a preset database type and the drive configuration and the database type of the target service system;
and driving the data acquisition script to execute through the driving configuration of the target service system according to the data dictionary of the target service system and the decrypted connection account password, and connecting to the connection address to read data in a database of the target service system.
4. The method of claim 3, wherein obtaining the connection account password of the target business system comprises:
and determining the connection account password of the target service system according to the system type of the target service system and a third corresponding relation between the preset system type and the connection account password.
5. The method according to claim 3, wherein decrypting the connection account password of the target service system to obtain the decrypted connection account password comprises:
acquiring a ciphertext obtained by encrypting the connection account password by the target service system through a secret key;
and decrypting the ciphertext through the secret key to obtain the decrypted connection account password.
6. The method of claim 2, further comprising:
acquiring field types of service systems of different database types;
and respectively generating field types mapped to the target database according to the field types of the tables in each service system to obtain a mapping relation between the preset database field type and the target database field type.
7. A big data extraction apparatus, the apparatus comprising:
the service system information acquisition module is used for acquiring the system type and the table information of the target service system;
the database type acquisition module is used for determining the database type of the target service system according to a first corresponding relation between a preset system type and the database type and the system type of the target service system;
the field type determining module is used for determining the field type of a target database of the target service system according to the database type of the target service system;
the data dictionary determining module is used for extracting through a data dictionary extraction script according to the field type of the target database of the target service system and the table information of the target service system to generate a data dictionary of the target service system;
and the data reading module is used for reading data in a database of the target service system through a preset data acquisition script and the data dictionary of the target service system.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111470058.9A CN114297274A (en) | 2021-12-03 | 2021-12-03 | Big data extraction method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111470058.9A CN114297274A (en) | 2021-12-03 | 2021-12-03 | Big data extraction method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114297274A true CN114297274A (en) | 2022-04-08 |
Family
ID=80964857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111470058.9A Pending CN114297274A (en) | 2021-12-03 | 2021-12-03 | Big data extraction method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114297274A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116909688A (en) * | 2023-09-14 | 2023-10-20 | 中移(苏州)软件技术有限公司 | Database calling method and device, storage medium and electronic equipment |
-
2021
- 2021-12-03 CN CN202111470058.9A patent/CN114297274A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116909688A (en) * | 2023-09-14 | 2023-10-20 | 中移(苏州)软件技术有限公司 | Database calling method and device, storage medium and electronic equipment |
| CN116909688B (en) * | 2023-09-14 | 2024-01-26 | 中移(苏州)软件技术有限公司 | Database calling method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7672967B2 (en) | Method and system for obfuscating data structures by deterministic natural data substitution | |
| CN101587479B (en) | Database management system kernel oriented data encryption/decryption system and method thereof | |
| CN106971121B (en) | Data processing method, device, server and storage medium | |
| US9021259B2 (en) | Encrypted database system, client terminal, encrypted database server, natural joining method, and program | |
| JP5776696B2 (en) | Encrypted database system, client terminal, encrypted database server, natural join method and program | |
| CN108701198A (en) | Conjunction SQL statement can search for encrypting | |
| CN109815719A (en) | A kind of database security encryption system that can search for | |
| US7930560B2 (en) | Personal information management system, personal information management program, and personal information protecting method | |
| JP5348337B2 (en) | Encrypted database management system, client and server, natural join method and program | |
| US10284535B2 (en) | Secure database | |
| CN104992124A (en) | Document safety access method for cloud storage environment | |
| US11080280B2 (en) | Enable merge join on encrypted data | |
| CN114297274A (en) | Big data extraction method and device, computer equipment and storage medium | |
| US20170200020A1 (en) | Data management system, program recording medium, communication terminal, and data management server | |
| JP3921892B2 (en) | Database management apparatus, database system, and recording medium | |
| US20230252166A1 (en) | Searching encrypted data | |
| US20240152630A1 (en) | Security system and method for real-time encryption or decryption of data using key management server | |
| JP6493402B2 (en) | Addition device, deletion device, addition request device, data search system, data search method, and computer program | |
| WO2024130721A1 (en) | Data storage server and client devices for securely storing and retrieving data | |
| US20130036474A1 (en) | Method and Apparatus for Secure Data Representation Allowing Efficient Collection, Search and Retrieval | |
| CN113918997A (en) | Data management system, and control method and device for data encryption and decryption | |
| CN117910055A (en) | Encryption transmission method and device for chip data, chip and storage medium | |
| CN114676164A (en) | Data caching method and device, computer equipment and storage medium | |
| CN116595546A (en) | Data encryption method, device, computer equipment and storage medium | |
| CN118092972A (en) | Thermal updating method, thermal updating device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |