CN115080955A - Target data filtering method and device, electronic equipment and storage medium - Google Patents
Target data filtering method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115080955A CN115080955A CN202210593886.XA CN202210593886A CN115080955A CN 115080955 A CN115080955 A CN 115080955A CN 202210593886 A CN202210593886 A CN 202210593886A CN 115080955 A CN115080955 A CN 115080955A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- target data
- filtering
- comparison result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/51—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Storage Device Security (AREA)
Abstract
The application provides a target data filtering method and device, electronic equipment and a storage medium. The method comprises the following steps: loading a preset data filtering rule based on the form of the agent; receiving a target data interception request; comparing the target data interception request with the data filtering rule to determine a comparison result; and filtering the database according to the comparison result to obtain a target data set. The data filtering rules are loaded in an agent mode, zero code invasion is realized on a service program through a pluggable design mode, the stability of the service and the performance of source data access are better ensured, and the safety of system data is enhanced. And by defining a data filtering rule, metadata to be filtered is customized according to the service data, redundant metadata is prevented from entering the memory of the application program, the memory consumption and the memory recovery pressure of the application program are reduced, and the system stability of the application program is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for filtering target data, an electronic device, and a storage medium.
Background
In the related art, a common metadata loading scheme is based on a framework self-contained or an intrusive modification of the underlying business logic, the modification is irreversible, and the metadata loading process is full-load. Therefore, there are problems of coarse data access granularity, large amount of computing resources occupation and low business security due to invasion of business logic.
Disclosure of Invention
In view of the above, an object of the present application is to provide a method and an apparatus for filtering target data, an electronic device, and a storage medium.
In view of the above, in a first aspect, the present application provides a target data filtering method, including:
loading a preset data filtering rule based on the form of the agent;
receiving a target data interception request;
comparing the target data interception request with the data filtering rule to determine a comparison result;
and filtering the database according to the comparison result to obtain a target data set.
In a possible implementation manner, the loading of the preset data filtering rule based on the proxy form further includes:
accessing object code of the metadata according to the compute engine;
intercepting the bytecode of the target code according to an agent technology;
modifying the bytecode to load the data filtering rule to the metadata.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a black list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a blacklist set; wherein the set of blacklists includes a plurality of first data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
and determining the comparison result in response to the existence of at least one target data tag corresponding to any one of the first data tags.
In a possible implementation manner, the filtering the database according to the comparison result to obtain the target data set further includes:
and refusing to load data corresponding to all target data labels according to the comparison result, and determining the empty set as the target data set.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a white list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a white list set; wherein the white list set comprises a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine a first target number of second data tags corresponding to the target data tags;
and determining the comparison result according to the first target quantity.
In a possible implementation manner, the filtering the database according to the comparison result to obtain the target data set further includes:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a black list set and a white list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a blacklist set and a white list set; wherein the blacklist set includes a plurality of first data tags and the whitelist set includes a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
in response to the absence of at least one target data tag corresponding to any one of the first data tags, determining a first target number of second data tags corresponding to the target data tag;
and determining the comparison result according to the first target quantity.
In a possible implementation manner, the filtering the database according to the comparison result to obtain the target data set further includes:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
In a possible implementation manner, the data filtering rule further includes: a set of default values;
the method further comprises the following steps:
analyzing the acquired service data to determine a default value set; wherein the set of default values includes at least one third data tag;
determining a second target number of third data tags corresponding to at least one target data tag in response to the target data tag not belonging to either a white list set or a black list set;
and determining the comparison result according to the second target number.
In a possible implementation manner, the filtering the database according to the comparison result to obtain the target data set further includes:
and filtering the database according to the comparison result, and acquiring data corresponding to the third data tags in a second target quantity to obtain the target data set.
In one possible implementation, the method further includes:
and adding access limitation to the data corresponding to the second data tag so as to obtain a target data set within a preset time period according to the data corresponding to the second data tag.
In one possible implementation, the method further includes:
responding to a modification instruction of a data filtering rule monitored, and updating the data filtering rule according to the modification instruction;
hot loading the updated data filtering rules based on a form of the agent.
In one possible implementation, the method further includes:
determining the security level of the service data according to the service data; wherein the security level comprises a secret level;
and marking the service data with a first data label in response to the security level of the service data being secret.
In a second aspect, the present application provides a target data filtering apparatus comprising:
the loading module is configured to load preset data filtering rules based on the form of the proxy;
a receiving module configured to receive a target data interception request;
a determination module configured to compare the target data interception request with the data filtering rule to determine a comparison result;
and the filtering module is configured to filter the database according to the comparison result to obtain a target data set.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the target data filtering method according to the first aspect when executing the program.
In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the target data filtering method of the first aspect.
As can be seen from the foregoing, according to the target data filtering method and apparatus, the electronic device and the storage medium provided by the present application, the preset data filtering rule is loaded based on the proxy form, after the interception request of the target data is received, the target data interception request and the data filtering rule can be compared, so as to determine the comparison result, and the target data set is filtered from the database according to different comparison results, thereby implementing the filtering of the target data. The data filtering rules are loaded in an agent mode, zero code invasion is realized on a service program through a pluggable design mode, the stability of the service and the performance of source data access are better ensured, and the safety of system data is enhanced. And by defining a data filtering rule, metadata to be filtered is customized according to the service data, redundant metadata is prevented from entering the memory of the application program, the memory consumption and the memory recovery pressure of the application program are reduced, and the system stability of the application program is improved.
Drawings
In order to more clearly illustrate the technical solutions in the present application or the related art, the drawings needed to be used in the description of the embodiments or the related art will be briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 illustrates an exemplary flowchart of a target data filtering method provided in an embodiment of the present application.
FIG. 2 illustrates an exemplary flow diagram of data filtering rule loading in an embodiment in accordance with the present application.
FIG. 3 illustrates an exemplary flow diagram of a method for target data filtering in conjunction with a set of blacklists, a set of whitelists and a set of default values in accordance with an embodiment of the present application.
FIG. 4 illustrates an exemplary flow diagram of a data filtering rule hot-loading process according to an embodiment of the application.
Fig. 5 shows an exemplary structural diagram of a target data filtering apparatus provided in an embodiment of the present application.
Fig. 6 shows an exemplary structural schematic diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to the accompanying drawings in combination with specific embodiments.
It should be noted that technical terms or scientific terms used in the embodiments of the present application should have a general meaning as understood by those having ordinary skill in the art to which the present application belongs, unless otherwise defined. The use of "first," "second," and similar terms in the embodiments of the present application do not denote any order, quantity, or importance, but rather the terms are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
As described in the background section, nowadays, the data acquisition cost is lower and easier, but the data size is larger and larger, and in mass data, it is desirable to analyze data quickly and load metadata, and at this time, loading of all attributes of certain metadata may be triggered, and because the metadata loading process is full-load, it is impossible to perform fine management according to the service characteristics. When the metadata is accessed, most of the related technologies are based on direct connection to a persistent metadata database, or a local cache policy is added to accelerate the metadata access, but for the case that the access amount is large, such as the access of a computing engine, a higher QPS and a consistency guarantee are needed, the access granularity is rough, and the metadata control with finer granularity cannot be achieved.
The applicant finds through research that the existing metadata loading scheme is basically based on the business logic of the modification bottom layer with self-contained framework or needing to be immersed, and the modification is irreversible, and the business form is various, so that the metadata can not be loaded according to the characteristics of the business in a previously defined way. With the increasing of big data services, the description of data, namely metadata, also begins to expand, which easily causes slow performance in the process of using and searching; how to speed up metadata access. The related technical solution is to add a caching mechanism, but some metadata in the caching mechanism is not needed by the service and the user. Therefore, the related art has the problems of coarse data access granularity, large computing resource occupation and low business safety caused by invasion of business logic.
Therefore, the target data filtering method, the target data filtering device, the electronic equipment and the storage medium provided by the application load the preset data filtering rule based on the proxy form, after the interception request of the target data is received, the target data interception request and the data filtering rule can be compared, the comparison result is further determined, the target data set is filtered from the database according to different comparison results, and therefore the target data are filtered. The data filtering rules are loaded in an agent mode, zero code invasion is realized on a service program through a pluggable design mode, the stability of the service and the performance of source data access are better ensured, and the safety of system data is enhanced. And by defining a data filtering rule, metadata to be filtered is customized according to the service data, redundant metadata is prevented from entering the memory of the application program, the memory consumption and the memory recovery pressure of the application program are reduced, and the system stability of the application program is improved.
The following describes the target data filtering method provided in the embodiments of the present application with specific embodiments.
Fig. 1 illustrates an exemplary flowchart of a target data filtering method provided in an embodiment of the present application.
Referring to fig. 1, a target data filtering method provided in the embodiment of the present application specifically includes the following steps:
s102: and loading preset data filtering rules based on the form of the proxy.
S104: a target data interception request is received.
S106: and comparing the target data interception request with the data filtering rule to determine a comparison result.
S108: and filtering the database according to the comparison result to obtain a target data set.
FIG. 2 illustrates an exemplary flow diagram of data filtering rule loading in an embodiment in accordance with the present application.
Referring to fig. 2, with respect to step S102, in order to ensure that the business code is not invaded when filtering the target data, the data filtering rules may be loaded to the business side using a proxy technique. Specifically, the object code of the metadata may be accessed according to a specific computing engine, where the computing engine may be, for example, impala, hive, or spark, and during the loading of the source data access program code of the computing engine, the bytecode of the object code may be intercepted according to an agent technology (e.g., java agent), the bytecode of the object code may be further dynamically modified through a bytecode modification technology, such as asm, java lost, or cglib technology, and the predefined data filtering rules and rule validation policies are loaded to the metadata of the service end.
In some embodiments, the data filtering rules and the rule validation policy may be set in an object code interceptor, the object code interceptor performs a thermal change of the code during the operation of the object code, and there is no intrusion to the object code service, and when the object code interceptor is not needed, the object code interceptor loaded by using the proxy technology may be directly removed, and compared with a source data access technology in the related art, the access code of the metadata of the computing engine does not need to be modified in an intrusive manner, thereby reducing operations of compiling, packaging, and replacing.
For step S104, a target data interception request sent by the user may be received, where the target data interception request may include at least one target data tag, for example, a name of a list where the target data is located or a service type to which the target data belongs, that is, the user may set the target data tag to indicate a requirement of data required by the user.
In some embodiments, the preset data filtering rule may include a set of blacklists, wherein the set of blacklists may include a plurality of first data tags, each first data tag corresponding to a filtering rule. The first data tag is marked to reject the loaded data, or may be the name of the list where the data is located or the service type of the target data, and in general, one data tag may be a regular expression, for example, a filtering rule of a blacklist set may be represented as inpala. The maximum number of partition days for a user query representing this definition base table is 365 days at the maximum.
It should be noted that, the determining of the blacklist set may determine, according to the obtained information of the service data, which data tags corresponding to the data are divided into the first data tags, that is, the data are divided into the blacklist. Specifically, the security level of the service data may be determined according to the service data, for example, if some data relates to confidential data, the security level of the service data is secret, and if some data is only daily log data, the security level of the service data is regular. Secret level data, as the name implies, because of involving the secret of core technology, when a user wants to obtain secret level target data, the target data needs to be filtered, and cannot be loaded for the user to access, and the secret level data will be marked with a first data tag. It will be appreciated that business personnel can customize which data needs to be protected as desired, thereby tagging the data with the first data tag.
In some embodiments, after analyzing the obtained service data to determine the blacklist set, the target data interception request and the data filtering rule may be compared to determine whether at least one target data tag corresponds to any one first data tag. Specifically, for example, there are two target data tags a and B in the target data interception request, and there are three first data tags A, C and D in the blacklist set in the data filtering rule, so that the target data tag a in the target data interception request corresponds to the first data tag a in the blacklist set, and then a comparison result is determined, where the comparison result indicates that the target data interception request falls into the blacklist set.
Further, since the target data interception request falls into the blacklist set, loading of data corresponding to all target data tags is rejected according to the comparison result, and the empty set is determined as the target data set, that is, no data is loaded for the user. Therefore, data security is guaranteed, leakage of confidential data is avoided, and since at least one target data label of the user relates to confidential data, the user is determined as a risk user in order to guarantee data security, and no data is loaded for the target data interception request of the user.
Still further, if no target data tag corresponds to any first data tag, data corresponding to the target data tag may be loaded according to an indication of the target data tag, thereby determining the target data set.
In some embodiments, the preset data filtering rule may include a white list set, wherein the white list set may include a plurality of second data tags, each second data tag corresponding to a filtering rule. The second data tag is marked to reject the loaded data, or may be the name of the list where the data is located or the service type to which the target data belongs, and in general, a data tag may be a regular expression, for example, a filtering rule of a white list set may be represented as impa.table.whitelist.pm02 _ wizard.dwd _ pm02_ player _, where the meaning of the impa.table.wh _ word is impa computational engine white list (no number of days defined to indicate no limitation to the number of days of the partition), and the meaning pm02_ wizard.dwd _ pm _ 02_ player _ \.
It should be noted that for the regular level data, it can be marked as a second data tag, indicating that the data can be loaded for the user to access. After the obtained service data are analyzed to determine the white list set, the target data interception request and the data filtering rule can be compared, and then the first target number of the second data tags corresponding to the target data tags is determined. For example, if two target data tags are provided, which are a and B, respectively, and three second data tags are provided, which are A, B and C, respectively, the first target number is determined to be 2, and a comparison result is determined according to the first target number, where the comparison result indicates that two target data tags of a data interception request sent by a user are marked as a white list.
Further, the database may be filtered according to the comparison result, and the data corresponding to the second data tags in the first target quantity, that is, all the data corresponding to the two target data tags are obtained, and the data are determined as the target data set.
In some embodiments, if two target data tags are provided, which are a and B, respectively, and three second data tags are provided, which are A, C and D, respectively, it is determined that the first target number is 1, and a comparison result determined according to the first target number indicates that one target data tag of a data interception request sent by a user is marked as a white list, and then the database may be filtered according to the comparison result, data corresponding to the same target data tag a is obtained, and the data are determined as a target data set.
It should be noted that if the first target data is 0, no data may be loaded, that is, the target data set is an empty set.
In some embodiments, a blacklist set and a whitelist set may be set at the same time, and a determination priority is set, for example, to ensure data security, the blacklist set determination priority is set to be the highest, then the target data interception request and the data filtering rule may be compared, and it is determined whether at least one target data tag corresponds to any one first data tag, if yes, no data is loaded, and an empty set is determined as the target data set.
If at least one target data tag does not correspond to any first data tag, the first target number of second data tags corresponding to the target data tag can be determined, that is, as long as no target data tag belongs to the blacklist set, the number of the target data tags corresponding to the second data tags in the whitelist set is determined, so that all data corresponding to the same second data tags as the target data tags are loaded to the user, and the target data set is obtained.
In some embodiments, the preset data filtering rules may include a default set of values, wherein the default set of values may include a plurality of third data tags, each third data tag corresponding to a filtering rule. The third data tag is marked to reject the loaded data, or may be the name of the list where the data tag is located or the service type of the target data, and in general, one data tag may be a regular expression, for example, a filtering rule of a default value set may be represented as impala. Representing the default query maximum partition that does not match a library table in the black and white list.
FIG. 3 illustrates an exemplary flow diagram of a method for target data filtering in conjunction with a set of blacklists, a set of whitelists and a set of default values in accordance with an embodiment of the present application.
It should be noted that, referring to fig. 3, the white list set, the black list set and the default value set may be set simultaneously, so as to maximally guarantee the comprehensiveness of data loading. And if no target data label belongs to the blacklist set and at least one target data label does not belong to the white list set, determining the data which do not belong to the blacklist set and the white list set at the same time as the data corresponding to a third data label in the default value set. And then determining a second target number of third data tags corresponding to the target data tags, for example, the target data tags include two data tags, namely a and B, the blacklist set includes two first data tags, namely C and D, and the whitelist set includes two second data tags, namely a and E, so that when the data corresponding to the target data tag a is loaded to the user, it is determined that the target data tag B belongs to the default value set, and the data corresponding to the target data tag B is also loaded to the user.
Still further, access restrictions may be added to the data corresponding to the second data tag in the white list set, for example, the data corresponding to the second data tag may be loaded to the user, but is limited to the data within a preset time period, for example, the data within one year, and not all the historical data corresponding to the second data tag may be loaded to the user, so as to further improve security monitoring on the data.
It should be noted that, when the data corresponding to the third data tag in the default value set is loaded, no access restriction may be added, and it is ensured that all the historical data corresponding to the third data tag may be loaded for the user to access.
FIG. 4 illustrates an exemplary flow diagram of a data filtering rule hot-loading process according to an embodiment of the application.
Referring to fig. 4, in some embodiments, by using the characteristics of the linux system, in response to monitoring a modification instruction of a data filtering rule, the data filtering rule may be updated in a hot manner according to the modification instruction, and further, the updated data filtering rule is loaded in a hot manner based on a form of proxy, so that the modification of the data filtering rule is realized on the basis that the operation of a service is not hindered, and different data filtering requirements are met.
As can be seen from the foregoing, according to the target data filtering method and apparatus, the electronic device and the storage medium provided by the present application, the preset data filtering rule is loaded based on the proxy form, after the interception request of the target data is received, the target data interception request and the data filtering rule can be compared, so as to determine the comparison result, and the target data set is filtered from the database according to different comparison results, thereby implementing the filtering of the target data. The data filtering rules are loaded in an agent mode, zero code invasion is realized on a service program through a pluggable design mode, the stability of the service and the performance of source data access are better ensured, and the safety of system data is enhanced. And by defining a data filtering rule, metadata to be filtered is customized according to the service data, redundant metadata is prevented from entering the memory of the application program, the memory consumption and the memory recovery pressure of the application program are reduced, and the system stability of the application program is improved.
It should be noted that the method of the embodiment of the present application may be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment, and the multiple devices interact with each other to complete the method.
It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Fig. 5 shows an exemplary structural diagram of a target data filtering apparatus provided in an embodiment of the present application.
Based on the same inventive concept, corresponding to the method of any embodiment, the application also provides a target data filtering device.
Referring to fig. 5, the object data filtering apparatus includes: the device comprises a loading module, a receiving module, a determining module and a filtering module; wherein the content of the first and second substances,
the loading module is configured to load preset data filtering rules based on the form of the proxy;
a receiving module configured to receive a target data interception request;
a determination module configured to compare the target data interception request with the data filtering rule to determine a comparison result;
and the filtering module is configured to filter the database according to the comparison result to obtain a target data set.
In one possible implementation, the loading module is further configured to:
accessing object code of the metadata according to the compute engine;
intercepting the bytecode of the target code according to an agent technology;
modifying the bytecode to load the data filtering rule to the metadata.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a black list set;
the determination module is further configured to:
analyzing the obtained service data to determine a blacklist set; wherein the set of blacklists includes a plurality of first data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
and determining the comparison result in response to the existence of at least one target data tag corresponding to any one of the first data tags.
In one possible implementation, the filtering module is further configured to:
and refusing to load data corresponding to all target data labels according to the comparison result, and determining the empty set as the target data set.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a white list set;
the determination module is further configured to:
analyzing the obtained service data to determine a white list set; wherein the white list set comprises a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine a first target number of second data tags corresponding to the target data tags;
and determining the comparison result according to the first target quantity.
In one possible implementation, the filtering module is further configured to:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
In one possible implementation, the target data intercepting request includes: at least one target data tag; the data filtering rule comprises: a black list set and a white list set;
the determination module is further configured to:
analyzing the obtained service data to determine a blacklist set and a white list set; wherein the blacklist set includes a plurality of first data tags and the whitelist set includes a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
in response to the absence of at least one target data tag corresponding to any one of the first data tags, determining a first target number of second data tags corresponding to the target data tag;
and determining the comparison result according to the first target quantity.
In one possible implementation, the filtering module is further configured to:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
In a possible implementation manner, the data filtering rule further includes: a set of default values;
the determination module is further configured to:
analyzing the acquired service data to determine a default value set; wherein the set of default values includes at least one third data tag;
determining a second target number of third data tags corresponding to at least one target data tag in response to the target data tag not belonging to either a white list set or a black list set;
and determining the comparison result according to the second target number.
In one possible implementation, the filtering module is further configured to:
and filtering the database according to the comparison result, and acquiring data corresponding to the third data tags in a second target quantity to obtain the target data set.
In one possible implementation manner, the apparatus further includes: adding a module;
the add module is configured to:
and adding access limitation to the data corresponding to the second data tag so as to obtain a target data set within a preset time period according to the data corresponding to the second data tag.
In one possible implementation manner, the apparatus further includes: a hot loading module;
the hot load module is configured to:
responding to a modification instruction of a data filtering rule monitored, and updating the data filtering rule according to the modification instruction;
hot loading the updated data filtering rules based on a form of the agent.
In one possible implementation manner, the apparatus further includes: a marking module;
the tagging module is configured to:
determining the security level of the service data according to the service data; wherein the security level comprises a secret level;
and marking the service data with a first data label in response to the security level of the service data being secret.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the various modules may be implemented in the same one or more software and/or hardware implementations as the present application.
The apparatus in the foregoing embodiment is used to implement the corresponding target data filtering method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Fig. 6 shows an exemplary structural schematic diagram of an electronic device provided in an embodiment of the present application.
Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the target data filtering method described in any of the above embodiments is implemented. Fig. 6 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the device may include: a processor 610, a memory 620, an input/output interface 630, a communication interface 640, and a bus 650. Wherein the processor 610, memory 620, input/output interface 630, and communication interface 640 are communicatively coupled to each other within the device via a bus 650.
The processor 610 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present specification.
The Memory 620 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 620 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 620 and called by the processor 610 to be executed.
The input/output interface 630 is used for connecting an input/output module to realize information input and output. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output devices may include a display, speaker, vibrator, indicator light, etc.
The communication interface 640 is used for connecting a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned devices only show the processor 610, the memory 620, the input/output interface 630, the communication interface 640 and the bus 650, in a specific implementation, the devices may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The electronic device of the foregoing embodiment is used to implement the corresponding target data filtering method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above-described embodiment methods, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the target data filtering method according to any of the above embodiments.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
The computer instructions stored in the storage medium of the foregoing embodiment are used to enable the computer to execute the target data filtering method according to any one of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiment, which are not described herein again.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the context of the present application, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures for simplicity of illustration and discussion, and so as not to obscure the embodiments of the application. Furthermore, devices may be shown in block diagram form in order to avoid obscuring embodiments of the application, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the embodiments of the application are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the application, it should be apparent to one skilled in the art that the embodiments of the application can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
The present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present application are intended to be included within the scope of the present application.
Claims (16)
1. A method of filtering target data, comprising:
loading a preset data filtering rule based on the form of the agent;
receiving a target data interception request;
comparing the target data interception request with the data filtering rule to determine a comparison result;
and filtering the database according to the comparison result to obtain a target data set.
2. The method of claim 1, wherein the agent-based form loads preset data filtering rules, further comprising:
accessing object code of the metadata according to the compute engine;
intercepting the bytecode of the target code according to an agent technology;
modifying the bytecode to load the data filtering rule to the metadata.
3. The method of claim 1, wherein the target data intercepting request comprises: at least one target data tag; the data filtering rule comprises: a black list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a blacklist set; wherein the set of blacklists includes a plurality of first data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
and determining the comparison result in response to the existence of at least one target data tag corresponding to any one of the first data tags.
4. The method of claim 3, wherein filtering the database to obtain the target data set according to the comparison result further comprises:
and refusing to load data corresponding to all target data labels according to the comparison result, and determining the empty set as the target data set.
5. The method of claim 1, wherein the target data intercepting request comprises: at least one target data tag; the data filtering rule comprises: a white list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a white list set; wherein the white list set comprises a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine a first target number of second data tags corresponding to the target data tags;
and determining the comparison result according to the first target quantity.
6. The method of claim 5, wherein filtering the database to obtain the target data set according to the comparison result further comprises:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
7. The method of claim 1, wherein the target data intercepting request comprises: at least one target data tag; the data filtering rule comprises: a black list set and a white list set;
the comparing the target data interception request with the data filtering rule to determine a comparison result further comprises:
analyzing the obtained service data to determine a blacklist set and a white list set; wherein the blacklist set includes a plurality of first data tags and the whitelist set includes a plurality of second data tags;
comparing the target data interception request with the data filtering rule to determine whether at least one target data tag corresponds to any one first data tag;
in response to the absence of at least one target data tag corresponding to any one of the first data tags, determining a first target number of second data tags corresponding to the target data tag;
and determining the comparison result according to the first target quantity.
8. The method of claim 7, wherein filtering the database to obtain the target data set according to the comparison result further comprises:
and filtering the database according to the comparison result, and acquiring data corresponding to the second data tags in a first target quantity to obtain the target data set.
9. The method of claim 7, wherein the data filtering rules further comprise: a set of default values;
the method further comprises the following steps:
analyzing the acquired service data to determine a default value set; wherein the set of default values includes at least one third data tag;
determining a second target number of third data tags corresponding to at least one target data tag in response to the target data tag not belonging to either a white list set or a black list set;
and determining the comparison result according to the second target number.
10. The method of claim 9, wherein filtering the database to obtain the target dataset according to the comparison further comprises:
and filtering the database according to the comparison result, and acquiring data corresponding to the third data tags in a second target quantity to obtain the target data set.
11. The method of claim 6 or 8, further comprising:
and adding access limitation to the data corresponding to the second data tag so as to obtain a target data set within a preset time period according to the data corresponding to the second data tag.
12. The method of claim 1, further comprising:
responding to a modification instruction of a data filtering rule monitored, and updating the data filtering rule according to the modification instruction;
hot loading the updated data filtering rules based on a form of the proxy.
13. The method of claim 3 or 5, further comprising:
determining the security level of the service data according to the service data; wherein the security level comprises a secret level;
and marking the service data with a first data label in response to the security level of the service data being secret.
14. A target data filtering apparatus, comprising:
the loading module is configured to load preset data filtering rules based on the form of the proxy;
a receiving module configured to receive a target data interception request;
a determination module configured to compare the target data interception request with the data filtering rule to determine a comparison result;
and the filtering module is configured to filter the database according to the comparison result to obtain a target data set.
15. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 13 when executing the program.
16. A non-transitory computer readable storage medium storing computer instructions for causing a computer to implement the method of any one of claims 1 to 13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210593886.XA CN115080955A (en) | 2022-05-27 | 2022-05-27 | Target data filtering method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210593886.XA CN115080955A (en) | 2022-05-27 | 2022-05-27 | Target data filtering method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115080955A true CN115080955A (en) | 2022-09-20 |
Family
ID=83248911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210593886.XA Pending CN115080955A (en) | 2022-05-27 | 2022-05-27 | Target data filtering method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115080955A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860483A (en) * | 2023-07-20 | 2023-10-10 | 合芯科技有限公司 | Data pruning method and device, computer equipment and storage medium |
-
2022
- 2022-05-27 CN CN202210593886.XA patent/CN115080955A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860483A (en) * | 2023-07-20 | 2023-10-10 | 合芯科技有限公司 | Data pruning method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10560465B2 (en) | Real time anomaly detection for data streams | |
| CN109344153B (en) | Service data processing method and terminal equipment | |
| US10523580B2 (en) | Automatic cloud provisioning based on related internet news and social network trends | |
| CN111291374B (en) | Application program detection method, device and equipment | |
| EP3361417A1 (en) | Smart card read/write methods and devices | |
| CN104321780A (en) | Devices, systems, and methods for monitoring and asserting trust level using persistent trust log | |
| CN109918678B (en) | Method and device for identifying field meaning | |
| CN111488594A (en) | Authority checking method and device based on cloud server, storage medium and terminal | |
| CN113010265A (en) | Pod scheduling method, scheduler, memory plug-in and system | |
| CN115080955A (en) | Target data filtering method and device, electronic equipment and storage medium | |
| US9646157B1 (en) | Systems and methods for identifying repackaged files | |
| CN112988062B (en) | Metadata reading limiting method and device, electronic equipment and medium | |
| US11243756B1 (en) | Extensible resource compliance management | |
| US9430530B1 (en) | Reusing database statistics for user aggregate queries | |
| CN117118698A (en) | Access flow limiting method, device and equipment of metadata server | |
| CN115190010B (en) | Distributed recommendation method and device based on software service dependency relationship | |
| CN116071123A (en) | Commodity data processing method, commodity data processing system, electronic equipment and storage medium | |
| US10795575B2 (en) | Dynamically reacting to events within a data storage system | |
| CN113282541B (en) | File calling method and device and electronic equipment | |
| CN114936368A (en) | Java memory Trojan detection method, terminal device and storage medium | |
| US11093636B2 (en) | Maintaining data protection compliance and data inference from data degradation in cross-boundary data transmission using containers | |
| CN112417324A (en) | Chrome-based URL (Uniform resource locator) interception method and device and computer equipment | |
| CN109446166B (en) | Method for detecting file directory, computer readable storage medium and terminal device | |
| US11782971B2 (en) | Static and dynamic NBMP function image retrieval and scale ranking | |
| US20230409628A1 (en) | Static and dynamic nbmp function image retrieval and scale ranking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |