CN113971295A - Data desensitization method and equipment based on field filtering - Google Patents
Data desensitization method and equipment based on field filtering Download PDFInfo
- Publication number
- CN113971295A CN113971295A CN202111216801.8A CN202111216801A CN113971295A CN 113971295 A CN113971295 A CN 113971295A CN 202111216801 A CN202111216801 A CN 202111216801A CN 113971295 A CN113971295 A CN 113971295A
- Authority
- CN
- China
- Prior art keywords
- data
- large object
- object field
- desensitization
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
Abstract
It is an object of the present application to provide a data desensitization scheme based on field filtering. Specifically, when target data is read from an original table, filtering data of a large object field in the original table; processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field; and extracting desensitization data from the target data and writing the desensitization data into a target table. Further, if a desensitization rule is configured for the data of the large object field, desensitization data corresponding to the data of the large object field is generated according to the desensitization rule, desensitization data corresponding to the data of the large object field is extracted, and the desensitization data is written into the target table. According to the method and the device, in the process of extracting and desensitizing the data, the data of the large object field are filtered, so that the problems of performance reduction and memory overflow caused by reading the data of the large object field are solved.
Description
Technical Field
The application relates to the technical field of information, in particular to a data desensitization technology based on field filtering.
Background
In recent years, with the rapid development of information technology, data security is emphasized by people, and data desensitization technology is widely applied. In the face of complex data desensitization requirements, the traditional extraction mode of the database table cannot meet part of requirements. For example, a user needs to extract data of a table, but the table contains large object fields such as BLOB, LONG, and the like, and if the data of these large object fields are directly subjected to extraction desensitization in a conventional manner, since the values of the large object fields are extremely large (for example, several hundred M or even several G), directly performing extraction desensitization operation may cause performance degradation and excessive memory consumption, and increase the risk of memory overflow.
Disclosure of Invention
An object of the present application is to provide a data desensitization method and apparatus based on field filtering.
According to one aspect of the application, a data desensitization method based on field filtering is provided, wherein the method comprises:
when target data are read from an original table, filtering data of a large object field in the original table;
processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field;
and extracting desensitization data from the target data and writing the desensitization data into a target table.
Further, when reading the target data from the original table, filtering the data of the large object field in the original table, including: when target data is read from an original table, filtering data of a large object field in the original table through an SQL query statement.
Further, the target data does not contain data for the large object field in the original table.
Further, processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field, including: and if desensitization rules are configured for the data of the large object field, generating desensitization data corresponding to the data of the large object field according to the desensitization rules.
Further, processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field, further comprising: and extracting desensitization data corresponding to the data of the large object field, and writing the desensitization data into the target table.
Further, processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field, including: and if no desensitization rule is configured for the data of the large object field, not extracting the data of the large object field and writing the data into the target table.
Further, processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field, further comprising: and setting a large object field in the target table to be null.
According to another aspect of the present application, there is also provided a data desensitization apparatus based on field filtering, wherein the apparatus includes:
the device comprises a first module, a second module and a third module, wherein the first module is used for filtering data of a large object field in an original table when target data are read from the original table;
the second module is used for processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field;
and the third module is used for extracting desensitization data from the target data and writing the desensitization data into the target table.
According to yet another aspect of the present application, there is also provided a computing device, wherein the device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein the computer program instructions, when executed by the processor, trigger the device to perform the field-filtering based data desensitization method.
According to yet another aspect of the present application, there is also provided a computer readable medium having stored thereon computer program instructions executable by a processor to implement the field-filtering based data desensitization method.
According to the scheme provided by the application, when target data are read from an original table, the data of a large object field in the original table are filtered; processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field; and extracting desensitization data from the target data and writing the desensitization data into a target table. Further, if a desensitization rule is configured for the data of the large object field, desensitization data corresponding to the data of the large object field is generated according to the desensitization rule, desensitization data corresponding to the data of the large object field is extracted, and the desensitization data is written into the target table. And if the desensitization rule is not configured for the data of the large object field, not extracting the data of the large object field and writing the data into the target table, and setting the large object field in the target table to be null. According to the method and the device, in the process of extracting and desensitizing the data, the data of the large object field are filtered, so that the problems of performance reduction and memory overflow caused by reading the data of the large object field are solved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a flow diagram of a method for field-filtering-based data desensitization according to an embodiment of the present application;
FIG. 2 is a flow diagram of data desensitization based on field filtering according to an embodiment of the present application;
fig. 3 is a schematic diagram of a data desensitization apparatus based on field filtering according to an embodiment of the present application.
The same or similar reference numbers in the drawings identify the same or similar elements.
Detailed Description
The present application is described in further detail below with reference to the attached figures.
In a typical configuration of the present application, the terminal, the device serving the network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, which include both non-transitory and non-transitory, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, program means, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
The embodiment of the application provides a data desensitization method based on field filtering, and in the process of extracting and desensitizing data, the data of a large object field is filtered, so that the problems of performance reduction and memory overflow caused by reading the data of the large object field are solved.
In a practical scenario, the device implementing the method may be a user equipment, a network device, or a device formed by integrating the user equipment and the network device through a network. The user equipment includes, but is not limited to, a terminal device such as a smartphone, a tablet computer, a Personal Computer (PC), and the like, and the network device includes, but is not limited to, a network host, a single network server, multiple network server sets, or a cloud computing-based computer set. Here, the Cloud is made up of a large number of hosts or web servers based on Cloud Computing (Cloud Computing), which is a type of distributed Computing, one virtual computer consisting of a collection of loosely coupled computers.
Fig. 1 is a flowchart of a data desensitization method based on field filtering according to an embodiment of the present application, and the method includes step S101, step S102, and step S103.
Step S101, when reading the target data from the original table, filtering the data of the large object field in the original table.
For example, the original table contains data to be subjected to data desensitization, and the data to be subjected to data desensitization includes data of large object fields of types such as BLOB, CLOB, LONG, and the like.
In some embodiments, the target data does not include data for a large object field in the original table.
For example, when data is read from the original table, the data of the large object field is filtered, i.e., the data of the large object field in the original table is not read, and then the target data does not contain the data of the large object field in the original table.
In some embodiments, the step S101 includes: when target data is read from an original table, filtering data of a large object field in the original table through an SQL query statement.
For example, when a read operation is performed on the data in the original table, the data of the large object field in the original table is filtered through the generated SQL query statement, that is, the data of the large object field in the original table is not read.
And step S102, processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field.
For example, after the target data in the original table is read into the memory, whether data desensitization operation is performed on the data of the large object field in the original table is judged according to whether a desensitization rule is configured on the data of the large object field. If data desensitization needs to be carried out on the data of the large object field, desensitization data corresponding to the data of the large object field are generated; and if data desensitization on the data of the large object field is not required, setting the large object field to be null in the target table. For the target data in the original table, decimation desensitization is performed in a conventional manner.
In some embodiments, the step S102 includes: if desensitization rules are configured for the data of the large object field, desensitization data corresponding to the data of the large object field are generated according to the desensitization rules; and extracting desensitization data corresponding to the data of the large object field, and writing the desensitization data into the target table. In this case, not only the desensitization data extracted from the target data is written into the target table, but also desensitization data corresponding to the data of the large object field is written into the target table, thereby achieving extraction desensitization of the data of the large object field in the original table.
In some embodiments, the step S102 includes: if the desensitization rule is not configured for the data of the large object field, the data of the large object field is not extracted and written into the target table; and setting a large object field in the target table to be null. In this case, only desensitization data is extracted from the target data and written into the target table, and the large object field is set to null in the target table.
And step S103, extracting desensitization data from the target data and writing the desensitization data into a target table.
For example, when data is written to the target table, the fields in the table are written according to the result of desensitization of the data. Namely, the target data is written into the target table in a conventional manner; for the data of the large object field, according to the processing result in the step S102, writing desensitization data corresponding to the data of the large object field into the target table, or setting the large object field to be null in the target table.
In some embodiments, as shown in fig. 2, filtering the data of the large object field in the original table through an SQL query statement, reading target data from the original table, performing a data desensitization operation on the data of the large object field in the original table, then extracting desensitization data from the target data and writing the desensitization data corresponding to the data of the large object field in the target table, thereby implementing extraction desensitization on the data of the large object field in the original table.
Fig. 3 is a schematic diagram of a data desensitization apparatus based on field filtering according to an embodiment of the present application, which includes a first module 301, a second module 302, and a third module 303.
The first module 301 filters data of a large object field in an original table when target data is read from the original table.
For example, the original table contains data to be subjected to data desensitization, and the data to be subjected to data desensitization includes data of large object fields of types such as BLOB, CLOB, LONG, and the like.
In some embodiments, the target data does not include data for a large object field in the original table.
For example, when data is read from the original table, the data of the large object field is filtered, i.e., the data of the large object field in the original table is not read, and then the target data does not contain the data of the large object field in the original table.
In some embodiments, the first module 301 is configured to: when target data is read from an original table, filtering data of a large object field in the original table through an SQL query statement.
For example, when a read operation is performed on the data in the original table, the data of the large object field in the original table is filtered through the generated SQL query statement, that is, the data of the large object field in the original table is not read.
A second module 302, for processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field.
For example, after the target data in the original table is read into the memory, whether data desensitization operation is performed on the data of the large object field in the original table is judged according to whether a desensitization rule is configured on the data of the large object field. If data desensitization needs to be carried out on the data of the large object field, desensitization data corresponding to the data of the large object field are generated; and if data desensitization on the data of the large object field is not required, setting the large object field to be null in the target table. For the target data in the original table, decimation desensitization is performed in a conventional manner.
In some embodiments, the second module 302 is configured to: if desensitization rules are configured for the data of the large object field, desensitization data corresponding to the data of the large object field are generated according to the desensitization rules; and extracting desensitization data corresponding to the data of the large object field, and writing the desensitization data into the target table. In this case, not only the desensitization data extracted from the target data is written into the target table, but also desensitization data corresponding to the data of the large object field is written into the target table, thereby achieving extraction desensitization of the data of the large object field in the original table.
In some embodiments, the second module 302 is configured to: if the desensitization rule is not configured for the data of the large object field, the data of the large object field is not extracted and written into the target table; and setting a large object field in the target table to be null. In this case, only desensitization data is extracted from the target data and written into the target table, and the large object field is set to null in the target table.
A third module 303, for extracting desensitization data from the target data and writing the desensitization data into the target table.
For example, when data is written to the target table, the fields in the table are written according to the result of desensitization of the data. Namely, the target data is written into the target table in a conventional manner; for the data of the large object field, according to the processing result of the second module 302, writing desensitization data corresponding to the data of the large object field into the target table, or setting the large object field to be null in the target table.
In summary, in the process of performing extraction desensitization on data, the embodiment of the present application filters data in a large object field, thereby avoiding the problems of performance degradation and memory overflow caused by reading the data in the large object field.
In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Program instructions which invoke the methods of the present application may be stored on a fixed or removable recording medium and/or transmitted via a data stream on a broadcast or other signal-bearing medium and/or stored within a working memory of a computer device operating in accordance with the program instructions. Herein, some embodiments of the present application provide a computing device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein the computer program instructions, when executed by the processor, trigger the device to perform the methods and/or aspects of the embodiments of the present application as described above.
Furthermore, some embodiments of the present application also provide a computer readable medium, on which computer program instructions are stored, the computer readable instructions being executable by a processor to implement the methods and/or aspects of the foregoing embodiments of the present application.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In some embodiments, the software programs of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including associated data structures) of the present application may be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Claims (10)
1. A method of data desensitization based on field filtering, wherein the method comprises:
when target data are read from an original table, filtering data of a large object field in the original table;
processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field;
and extracting desensitization data from the target data and writing the desensitization data into a target table.
2. The method of claim 2, wherein filtering data of a large object field in an original table when reading target data from the original table comprises:
when target data is read from an original table, filtering data of a large object field in the original table through an SQL query statement.
3. The method of claim 1 or 2, wherein the target data does not contain data of a large object field in the original table.
4. The method of claim 1, wherein processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field comprises:
and if desensitization rules are configured for the data of the large object field, generating desensitization data corresponding to the data of the large object field according to the desensitization rules.
5. The method of claim 4, wherein processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field further comprises:
and extracting desensitization data corresponding to the data of the large object field, and writing the desensitization data into the target table.
6. The method of claim 1, wherein processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field comprises:
and if no desensitization rule is configured for the data of the large object field, not extracting the data of the large object field and writing the data into the target table.
7. The method of claim 6, wherein processing the data of the large object field in the original table according to whether a desensitization rule is configured for the data of the large object field further comprises:
and setting a large object field in the target table to be null.
8. A field filtering based data desensitization device, wherein the device comprises:
the device comprises a first module, a second module and a third module, wherein the first module is used for filtering data of a large object field in an original table when target data are read from the original table;
the second module is used for processing the data of the large object field in the original table according to whether a desensitization rule is configured on the data of the large object field;
and the third module is used for extracting desensitization data from the target data and writing the desensitization data into the target table.
9. A computing device, wherein the device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein the computer program instructions, when executed by the processor, trigger the device to perform the method of any of claims 1 to 7.
10. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216801.8A CN113971295A (en) | 2021-10-19 | 2021-10-19 | Data desensitization method and equipment based on field filtering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216801.8A CN113971295A (en) | 2021-10-19 | 2021-10-19 | Data desensitization method and equipment based on field filtering |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113971295A true CN113971295A (en) | 2022-01-25 |
Family
ID=79587635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111216801.8A Pending CN113971295A (en) | 2021-10-19 | 2021-10-19 | Data desensitization method and equipment based on field filtering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113971295A (en) |
-
2021
- 2021-10-19 CN CN202111216801.8A patent/CN113971295A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10255108B2 (en) | Parallel execution of blockchain transactions | |
| CN111898139B (en) | Data reading and writing method and device and electronic equipment | |
| US11294973B2 (en) | Codeless information service for abstract retrieval of disparate data | |
| CN112069536A (en) | Method and equipment for realizing desensitization access of database data | |
| WO2016145993A1 (en) | Method and system for user device identification | |
| WO2018097846A1 (en) | Edge store designs for graph databases | |
| CN109918678B (en) | Method and device for identifying field meaning | |
| CN112181902B (en) | Database storage method and device and electronic equipment | |
| US20160246705A1 (en) | Data fabrication based on test requirements | |
| WO2023124217A1 (en) | Method and device for acquiring comprehensively sorted data of multi-column data | |
| CN112765248A (en) | SQL-based data extraction method and equipment | |
| CN114626092A (en) | Desensitization method, system, device and computer storage medium for multi-field data with incidence relation | |
| EP3108400B1 (en) | Virus signature matching method and apparatus | |
| CN113971295A (en) | Data desensitization method and equipment based on field filtering | |
| US9286349B2 (en) | Dynamic search system | |
| CN111159117A (en) | Low-overhead file operation log acquisition method | |
| CN112799951B (en) | Method and equipment for generating automatic test case for charging system | |
| US11580251B1 (en) | Query-based database redaction | |
| CN115470489A (en) | Detection model training method, detection method, device and computer readable medium | |
| US20180113920A1 (en) | Recursive extractor framework for forensics and electronic discovery | |
| CN113590623A (en) | Method, device and equipment for data deep paging query | |
| CN112632211A (en) | Semantic information processing method and equipment for mobile robot | |
| CN110019507B (en) | Data synchronization method and device | |
| CN107766196B (en) | Method and device for starting check of computing device | |
| CN113971184A (en) | Method and equipment for managing operation and maintenance operation based on database proxy server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |