CN112395367A - Database data processing method and device - Google Patents
Database data processing method and device Download PDFInfo
- Publication number
- CN112395367A CN112395367A CN202011248332.3A CN202011248332A CN112395367A CN 112395367 A CN112395367 A CN 112395367A CN 202011248332 A CN202011248332 A CN 202011248332A CN 112395367 A CN112395367 A CN 112395367A
- Authority
- CN
- China
- Prior art keywords
- data table
- data
- table set
- processed
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000007781 pre-processing Methods 0.000 claims abstract description 17
- 230000010354 integration Effects 0.000 claims abstract description 16
- 230000008676 import Effects 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Abstract
One or more embodiments of the present specification provide a database data processing method and apparatus, including: preprocessing a data table set to be processed to obtain a preprocessed data table set; the data table set to be processed is a data table set which is obtained from different insurance systems and is composed of at least one data table; transmitting the preprocessed data table set to a data warehouse module so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set; receiving the integrated data table set transmitted by the data warehouse module; and carrying out inverse processing on the data table set subjected to the integration processing to obtain a data table set subjected to inverse processing. According to the method, the external data warehouse module is used for integrating the data table sets to be processed, so that the data processing efficiency of the system can be improved, and the performance of the system is guaranteed.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of database technologies, and in particular, to a method and an apparatus for processing database data.
Background
Currently, traditional relational databases, such as Oracle, SQLServer, MySQL, etc., are widely used inside most enterprises. The database is simple and convenient to use, easy to understand and maintain, but has low efficiency of relevant reading and writing of mass data based on the limitation of I/O. With the increase of data volume, repeated correlation calculation of mass data consumes more resources and has long processing time, so that the traditional relational database is difficult to support massive data service application.
Disclosure of Invention
In view of the above, one or more embodiments of the present disclosure are directed to a database data processing method and apparatus, so as to solve the problem of low data processing efficiency.
In view of the above, one or more embodiments of the present specification provide a database data processing method, including:
preprocessing a data table set to be processed to obtain a preprocessed data table set; the data table set to be processed is a data table set which is obtained from different insurance systems and is composed of at least one data table;
transmitting the preprocessed data table set to a data warehouse module so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set;
receiving the integrated data table set transmitted by the data warehouse module;
and carrying out inverse processing on the data table set subjected to the integration processing to obtain a data table set subjected to inverse processing.
Optionally, the preprocessing the to-be-processed data table set to obtain a preprocessed data table set includes:
encrypting the sensitive parameters in each data table in the data table set to be processed to obtain ciphertext sensitive parameters, and forming an encrypted data table set to be processed by each data table containing the ciphertext sensitive parameters;
constructing an additional data table, and adding the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed;
and converting each data table in the increment data table set to be processed to obtain each converted data table, and forming the preprocessed data table set by each converted data table.
Optionally, inverse processing is performed on the data table set after the integration processing to obtain a data table set after the inverse processing, including:
screening the additional data table from the data table set subjected to the integration processing, and deleting the additional data table to obtain a filtered data table set;
and decrypting the ciphertext sensitive parameters in each data table in the filtered data table set to generate plaintext sensitive parameters, and obtaining a decrypted data table set consisting of each data table containing the plaintext sensitive parameters.
Optionally, a data item of a data table in the to-be-processed data table set is insurance information, where the insurance information includes customer information;
the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set, and the method comprises the following steps:
standardizing insurance information in each data table in the preprocessed data table set to obtain standardized insurance information; and merging the standardized insurance information to obtain merged insurance information, wherein the merged insurance information is the insurance information with unique client information and uniform data format.
Optionally, the data warehouse module is a hive data warehouse tool;
the module for transmitting the preprocessed data table set to a data warehouse is as follows: importing the preprocessed data table set into the hive data warehouse tool by using a data import and export tool;
receiving the integrated processed data table set transmitted by the data warehouse module as follows: and exporting the integrated processed data table set from the hive data warehouse tool by utilizing the data import and export tool.
An embodiment of the present specification further provides a database data processing apparatus, including:
the preprocessing module is used for preprocessing the data table set to be processed to obtain a preprocessed data table set;
the data sending module is used for transmitting the preprocessed data table set to the data warehouse module so that the data warehouse module can integrate the preprocessed data table set to obtain an integrated data table set;
the data receiving module is used for receiving the integrated data table set transmitted by the data warehouse module;
and the inverse processing module is used for carrying out inverse processing on the data table set subjected to the integration processing to obtain the data table set subjected to the inverse processing.
Optionally, the preprocessing module is configured to encrypt the sensitive parameters in each data table in the set of data tables to be processed to obtain ciphertext sensitive parameters, and each data table containing the ciphertext sensitive parameters forms an encrypted set of data tables to be processed; constructing an additional data table, and adding the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed; and converting each data table in the increment data table set to be processed to obtain each converted data table, and forming the preprocessed data table set by each converted data table.
Optionally, the inverse processing module is configured to screen the additional data table from the integrated data table set, and delete the additional data table to obtain a filtered data table set; and decrypting the ciphertext sensitive parameters in each data table in the filtered data table set to generate plaintext sensitive parameters, and obtaining a decrypted data table set consisting of each data table containing the plaintext sensitive parameters.
Optionally, a data item of a data table in the to-be-processed data table set is insurance information, where the insurance information includes customer information;
the data warehouse module is used for standardizing insurance information in each data table in the preprocessed data table set to obtain standardized insurance information; and merging the standardized insurance information to obtain merged insurance information, wherein the merged insurance information is the insurance information with unique client information and uniform data format.
Optionally, the data warehouse module is a hive data warehouse tool;
the data sending module is used for importing the preprocessed data table set into the hive data warehouse tool by using a data import and export tool;
the data receiving module is used for exporting the integrated data table set from the hive data warehouse tool by using the data import and export tool.
As can be seen from the above, in the database data processing method and apparatus provided in one or more embodiments of the present disclosure, a to-be-processed data table set is preprocessed to obtain a preprocessed data table set, the preprocessed data table set is transmitted to a data warehouse module, so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set, the integrated data table set transmitted by the data warehouse module is received, and the integrated data table set is subjected to inverse processing to obtain an inverse processed data table set. According to the method, the external data warehouse module is used for integrating the data table sets to be processed, so that the data processing efficiency of the system can be improved, and the performance of the system is guaranteed.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.
FIG. 1 is a schematic flow chart of a method according to one or more embodiments of the present disclosure;
FIG. 2 is a schematic diagram of an apparatus according to one or more embodiments of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to one or more embodiments of the present disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in one or more embodiments of the specification is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
In an application scenario, a relational database supports table association processing, data flow between systems can be subjected to data extraction, conversion and integration by configuring tools such as database link, SSIS, DATAX and the like, a reasonable index is created according to actual association SQL, and the integration processing process of a plurality of data tables is completed. For example, when the oracle database is applied, each data table is extracted from the database of each insurance system through the database link and is stored in a partition mode; then creating a partition index and a global index according to application requirements; formulating a data processing rule, and packaging data in a storage area; and automatically scheduling by using an execution plan of the database, and cutting the mass data into cakes and processing the blocks. In the actual processing process, for massive data (such as billions of data tables), it generally takes several hours to create an index, the association operation of the two tables needs 4-5 hours, the association time consumption is multiplied with the increase of the data tables, the use of archiving, REDO and TMP spaces in the calculation process is also increased linearly, the longer the data processing time of the stock is, the more the incremental data backtrack is, and the timeliness of the data processing is low.
In view of this, embodiments of the present disclosure provide a database data processing method, which integrates and processes database data by using an external data module, so as to improve data processing efficiency.
As shown in fig. 1, one or more embodiments of the present specification provide a database data processing method, including:
s101: preprocessing a data table set to be processed to obtain a preprocessed data table set;
in this embodiment, before the data processing is performed by using the external data warehouse module, the set of data tables to be processed needs to be preprocessed.
In some embodiments, the set of pending data tables is a set of data tables comprising at least one data table obtained from a different insurance system. The different insurance systems are, for example, a life insurance system, a financial insurance system, etc. divided according to the types of insurance, or a long insurance system, a short insurance system, a red insurance system, etc. divided according to the types of services, each insurance system respectively creates and maintains its own insurance information, the insurance information includes customer information, policy information, attendant information, etc., and the insurance information can be stored in different data tables according to the information type or other service rules.
S102: transmitting the preprocessed data table set to a data warehouse module so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set;
in this embodiment, the preprocessed data table set is transmitted to the data warehouse module, and the received data table set is integrated by using the external data warehouse module, so as to obtain the integrated data table set. In some approaches, the data warehouse module may use a hive data warehouse tool.
S103: receiving an integrated data table set transmitted by a data warehouse module;
s104: and carrying out inverse processing on the data after the integration processing to obtain a data table set after the inverse processing.
In this embodiment, after receiving the integrated data table set transmitted by the data warehouse module, the data table set needs to be subjected to inverse processing to obtain an inverse processed data table set.
In the database data processing method provided by this embodiment, a to-be-processed data table set is preprocessed to obtain a preprocessed data table set, the preprocessed data table set is transmitted to a data warehouse module, so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set, the integrated data table set transmitted by the data warehouse module is received, and the integrated data table set is subjected to inverse processing to obtain an inverse processed data table set. According to the method, the external data warehouse module is used for integrating the data table sets to be processed, so that the data processing efficiency of the system can be improved, and the performance of the system is guaranteed.
In some embodiments, in step S101, preprocessing the to-be-processed data table set to obtain a preprocessed data table set, including:
encrypting the sensitive parameters in each data table in the data table set to be processed to obtain ciphertext sensitive parameters, and forming an encrypted data table set to be processed by each data table containing the ciphertext sensitive parameters;
constructing an additional data table, and adding the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed;
and converting each data table in the incremental data table set to be processed to obtain each converted data table, and forming a preprocessed data table set by each converted data table.
In this embodiment, since the set of data tables to be processed needs to be transmitted to the external data warehouse module for processing, security of data transmission to the outside is ensured first. On one hand, the sensitive parameters in each data table in the data table to be processed are encrypted according to a preset encryption algorithm to generate ciphertext sensitive parameters, and an encrypted data table set to be processed is obtained; on the other hand, a certain amount of additional data tables are constructed, the data format of the additional data tables is the same as that of the data tables in the data table set to be processed, the additional data tables are added into the encrypted data table set to be processed to obtain an increment data table set to be processed, and the data security is further improved through data confusion; and then, according to the data specification of the data warehouse module, converting each data table in the incremental data table set to be processed into a data table form suitable for processing by the data warehouse module, so that the data warehouse module can process conveniently. By preprocessing the data table set to be processed, a data set which is suitable for processing of the data warehouse module and has high data security can be obtained.
In an application scenario, a to-be-processed data table set is composed of data tables with a plurality of data items as insurance information, the insurance information comprises sensitive parameters such as client names, certificate types, certificate numbers, sexes, contact telephones and the like, before the insurance information is transmitted to an external data warehouse module, the sensitive parameters are encrypted to generate ciphertext sensitive parameters, and the encrypted to-be-processed data table set is obtained; then, according to the data format of the data table in the data table set to be processed, constructing a certain amount of additional data table, mixing the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed, for example, constructing an additional data table containing data items such as client names, identification numbers, contact numbers and the like, adding the additional data table as a new data table into an insurance data table, and hiding the real data table; and then, according to the data specification requirements of the hive data warehouse tool, each data table in the incremental data tables to be processed is converted into data suitable for being processed by the hive data warehouse tool, so that the hive data warehouse tool can process the received data conveniently. In some modes, the preprocessed data table set can be imported into the hive data warehouse tool by using a data import and export tool (such as DataX), and after the hive data warehouse tool integrates the data tables in the data table set, the integrated data table set is exported to the relational database by using the data import and export tool, so that the complex data table correlation operation is performed by using an external module, the relational database is not required to perform operation, and the data processing efficiency of the system can be improved.
In some embodiments, since different insurance systems respectively create and maintain respective insurance information, the data format of the insurance information in each insurance system does not conform to the unified data specification, and there is repeated customer information, and it is necessary to uniformly integrate the insurance information of each insurance system, so as to obtain the insurance information with unique customer information and unified data format specification. And the data warehouse module receives the preprocessed data table set, standardizes insurance information in each data table in the data table set to obtain standardized insurance information, and then combines the standardized insurance information to obtain combined insurance information.
In some aspects, normalizing the insurance information includes: according to a preset rule, at least one data item (such as a client name, a certificate number and the like) in the data table is checked, insurance information which is checked by the rule is screened out, the insurance information which is checked by the rule is converted into a preset format, and the insurance information with a uniform format is obtained.
In some embodiments, the merging the normalized insurance information includes: according to the certificate type, dividing the standardized insurance information into a first class client, a second class client and a third class client;
for one class of clients, screening insurance information according to the client name and the certificate number, merging at least one piece of insurance information with the client name and the certificate number consistent, and distributing a unique merged client number to obtain merged insurance information;
for the second class of clients, screening insurance information according to the client name, the certificate type and the certificate number, merging at least one piece of insurance information with the same client name, certificate type and certificate number, and distributing a unique merged client number to obtain merged insurance information;
for three types of clients, insurance information is screened according to the client name, gender, date of birth, certificate type and certificate number, at least one piece of insurance information with the client name, gender, date of birth, certificate type and certificate number consistent is merged, a unique merged client number is distributed, and merged insurance information is obtained.
In this embodiment, for data table sets which store insurance information and are obtained from different insurance systems, the data warehouse module is used to perform standardization processing and merging processing on each data table in the data table sets, and finally, a data table set which is unique in customer information and uniform in data standard is obtained, and a data table set after integration processing is obtained.
In some embodiments, in step S104, performing inverse processing on the integrated data to obtain an inverse processed data table set, including:
screening an additional data table from the data table set subjected to integration processing, and deleting the additional data table to obtain a filtered data table set;
and decrypting the ciphertext sensitive parameters in each data table in the filtered data table set to generate plaintext sensitive parameters, and obtaining a decrypted data table set consisting of each data table containing the plaintext sensitive parameters.
In this embodiment, after receiving the integrated data table set transmitted by the data warehouse module, corresponding to the preprocessing process, the additional data table is first screened out, the additional data table is deleted, and after the additional data table is filtered, the ciphertext sensitive parameters in each data table are decrypted to obtain plaintext sensitive parameters, so as to obtain a decrypted data table set.
In some embodiments, the preprocessed data table set may be transmitted to the data warehouse module according to a trigger condition or at a regular time, so that the data warehouse module integrates the preprocessed data table set to obtain the integrated data table set. The integrated data table set obtained by the data warehouse module may be configured to be generated, i.e., returned, or acquired by a request acquisition manner, and the specific manner is not particularly limited.
According to the database data processing method, for the data table sets obtained from the insurance systems, the data integration process is completed by using the external data warehouse module, on the basis of ensuring the data security, the data processing time of the relational database is reduced, the data processing efficiency is improved, the data backtracking range is greatly reduced, dynamic data are easy to dock, and the system performance is improved. In an application scenario, when the method of the embodiment is used for integrating and processing data volumes more than billions, online data import and export are realized by using a DataX tool, and data integration processing is completed by using a hive data warehouse tool.
It should be noted that the method of one or more embodiments of the present disclosure may be performed by a single device, such as a computer or server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the devices may perform only one or more steps of the method of one or more embodiments of the present disclosure, and the devices may interact with each other to complete the method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
As shown in fig. 2, the present embodiment further provides a database data processing apparatus, including:
the preprocessing module is used for preprocessing the data table set to be processed to obtain a preprocessed data table set;
the data sending module is used for transmitting the preprocessed data table set to the data warehouse module so that the data warehouse module can integrate the preprocessed data table set to obtain an integrated data table set;
the data receiving module is used for receiving the integrated data table set transmitted by the data warehouse module;
and the inverse processing module is used for carrying out inverse processing on the data after the integrated processing to obtain a data table set after the inverse processing.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
The apparatus of the foregoing embodiment is used to implement the corresponding method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Fig. 3 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (10)
1. A database data processing method, comprising:
preprocessing a data table set to be processed to obtain a preprocessed data table set; the data table set to be processed is a data table set which is obtained from different insurance systems and is composed of at least one data table;
transmitting the preprocessed data table set to a data warehouse module so that the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set;
receiving the integrated data table set transmitted by the data warehouse module;
and carrying out inverse processing on the data table set subjected to the integration processing to obtain a data table set subjected to inverse processing.
2. The method of claim 1, wherein the preprocessing the set of data tables to be processed to obtain a preprocessed set of data tables comprises:
encrypting the sensitive parameters in each data table in the data table set to be processed to obtain ciphertext sensitive parameters, and forming an encrypted data table set to be processed by each data table containing the ciphertext sensitive parameters;
constructing an additional data table, and adding the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed;
and converting each data table in the increment data table set to be processed to obtain each converted data table, and forming the preprocessed data table set by each converted data table.
3. The method of claim 2, wherein performing inverse processing on the integrated processed data table set to obtain an inverse processed data table set comprises:
screening the additional data table from the data table set subjected to the integration processing, and deleting the additional data table to obtain a filtered data table set;
and decrypting the ciphertext sensitive parameters in each data table in the filtered data table set to generate plaintext sensitive parameters, and obtaining a decrypted data table set consisting of each data table containing the plaintext sensitive parameters.
4. The method according to claim 1, wherein the data items of the data tables in the set of data tables to be processed are insurance information, and the insurance information includes customer information;
the data warehouse module integrates the preprocessed data table set to obtain an integrated data table set, and the method comprises the following steps:
standardizing insurance information in each data table in the preprocessed data table set to obtain standardized insurance information; and merging the standardized insurance information to obtain merged insurance information, wherein the merged insurance information is the insurance information with unique client information and uniform data format.
5. The method of claim 1, wherein the data warehouse module is a hive data warehouse tool;
the module for transmitting the preprocessed data table set to a data warehouse is as follows: importing the preprocessed data table set into the hive data warehouse tool by using a data import and export tool;
receiving the integrated processed data table set transmitted by the data warehouse module as follows: and exporting the integrated processed data table set from the hive data warehouse tool by utilizing the data import and export tool.
6. A database data processing apparatus, comprising:
the preprocessing module is used for preprocessing the data table set to be processed to obtain a preprocessed data table set;
the data sending module is used for transmitting the preprocessed data table set to the data warehouse module so that the data warehouse module can integrate the preprocessed data table set to obtain an integrated data table set;
the data receiving module is used for receiving the integrated data table set transmitted by the data warehouse module;
and the inverse processing module is used for carrying out inverse processing on the data table set subjected to the integration processing to obtain the data table set subjected to the inverse processing.
7. The apparatus of claim 6,
the preprocessing module is used for encrypting the sensitive parameters in the data tables in the data table set to be processed to obtain ciphertext sensitive parameters, and the data tables containing the ciphertext sensitive parameters form the encrypted data table set to be processed; constructing an additional data table, and adding the additional data table into the encrypted data table set to be processed to obtain an incremental data table set to be processed; and converting each data table in the increment data table set to be processed to obtain each converted data table, and forming the preprocessed data table set by each converted data table.
8. The apparatus of claim 7,
the inverse processing module is used for screening the additional data table from the data table set subjected to the integration processing, deleting the additional data table and obtaining a filtered data table set; and decrypting the ciphertext sensitive parameters in each data table in the filtered data table set to generate plaintext sensitive parameters, and obtaining a decrypted data table set consisting of each data table containing the plaintext sensitive parameters.
9. The apparatus according to claim 6, wherein the data items of the data tables in the set of data tables to be processed are insurance information, and the insurance information includes customer information;
the data warehouse module is used for standardizing insurance information in each data table in the preprocessed data table set to obtain standardized insurance information; and merging the standardized insurance information to obtain merged insurance information, wherein the merged insurance information is the insurance information with unique client information and uniform data format.
10. The apparatus of claim 6, wherein the data warehouse module is a hive data warehouse tool;
the data sending module is used for importing the preprocessed data table set into the hive data warehouse tool by using a data import and export tool;
the data receiving module is used for exporting the integrated data table set from the hive data warehouse tool by using the data import and export tool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011248332.3A CN112395367A (en) | 2020-11-10 | 2020-11-10 | Database data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011248332.3A CN112395367A (en) | 2020-11-10 | 2020-11-10 | Database data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112395367A true CN112395367A (en) | 2021-02-23 |
Family
ID=74599713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011248332.3A Pending CN112395367A (en) | 2020-11-10 | 2020-11-10 | Database data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112395367A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1924915A (en) * | 2006-09-20 | 2007-03-07 | 中山大学 | Database technique based library intelligent management system |
| CN105045904A (en) * | 2015-08-07 | 2015-11-11 | 北京京东尚科信息技术有限公司 | User data integration method and system based on data warehouse |
| CN105069033A (en) * | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| US20150347705A1 (en) * | 2014-05-28 | 2015-12-03 | Arcadia Solutions, LLC | Systems and methods for electronic health records |
| CN108959309A (en) * | 2017-05-23 | 2018-12-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of data analysis |
| CN110751485A (en) * | 2019-10-28 | 2020-02-04 | 腾讯科技（深圳）有限公司 | Data processing method and equipment |
| CN111737335A (en) * | 2020-07-29 | 2020-10-02 | 太平金融科技服务（上海）有限公司 | Product information integration processing method and device, computer equipment and storage medium |
| CN111882445A (en) * | 2020-07-24 | 2020-11-03 | 前海人寿保险股份有限公司 | Cross-system insurance user information management method, device, equipment and readable medium |
-
2020
- 2020-11-10 CN CN202011248332.3A patent/CN112395367A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1924915A (en) * | 2006-09-20 | 2007-03-07 | 中山大学 | Database technique based library intelligent management system |
| US20150347705A1 (en) * | 2014-05-28 | 2015-12-03 | Arcadia Solutions, LLC | Systems and methods for electronic health records |
| CN105069033A (en) * | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| CN105045904A (en) * | 2015-08-07 | 2015-11-11 | 北京京东尚科信息技术有限公司 | User data integration method and system based on data warehouse |
| CN108959309A (en) * | 2017-05-23 | 2018-12-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of data analysis |
| CN110751485A (en) * | 2019-10-28 | 2020-02-04 | 腾讯科技（深圳）有限公司 | Data processing method and equipment |
| CN111882445A (en) * | 2020-07-24 | 2020-11-03 | 前海人寿保险股份有限公司 | Cross-system insurance user information management method, device, equipment and readable medium |
| CN111737335A (en) * | 2020-07-29 | 2020-10-02 | 太平金融科技服务（上海）有限公司 | Product information integration processing method and device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 王文香 等: "保险业决策支持系统的数据仓库的设计与实现", 计算机系统应用, no. 7, pages 9 - 12 * |
| 赵青: "医疗保险信息系统的数据仓库与数据挖掘初探", 人力资源管理, no. 3, pages 9 - 10 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6709574B2 (en) | Terminal rule engine device and terminal rule operating method | |
| EP3719668A1 (en) | Block chain-based data processing method and device | |
| CN107229619B (en) | Method and device for counting and displaying calling condition of internet service link | |
| CN107798038B (en) | Data response method and data response equipment | |
| EP2930629A1 (en) | Accessing non-relational data stores using structured query language queries | |
| CN110633309A (en) | Block chain transaction processing method and device | |
| CN110457929B (en) | Method and system for sharing heterogeneous HIS (high-speed multimedia subsystem) big data real-time encryption and decryption compressed uplink | |
| CN109471893B (en) | Network data query method, equipment and computer readable storage medium | |
| CN112182004B (en) | Method, device, computer equipment and storage medium for checking data in real time | |
| WO2023236756A1 (en) | Multi-tenant data database allocation method, and program product and electronic device | |
| CN112508719A (en) | Report processing method and device | |
| CN112508720A (en) | Insurance client identity attribute screening method and screening device and electronic equipment | |
| CN107566499A (en) | The methods, devices and systems of data syn-chronization | |
| Kumar et al. | Agent based security model for cloud big data | |
| CN112395367A (en) | Database data processing method and device | |
| CN107391541A (en) | A kind of real time data merging method and device | |
| US9971801B2 (en) | Grid cell data requests | |
| CN112463785B (en) | Data quality monitoring method and device, electronic equipment and storage medium | |
| CN109582476B (en) | Data processing method, device and system | |
| CN109035040B (en) | Policy generation method and device and electronic equipment | |
| CN112465653A (en) | Insurance business processing method, device, equipment and storage medium | |
| CN214756405U (en) | Data processing system | |
| CN110648218A (en) | Credit wind control system and method based on privacy protection and computer device | |
| CN114697121A (en) | Data processing method and device of power monitoring system and computer equipment | |
| CN108959608A (en) | Historical transactional information querying method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |