CN115481299A - Method, system and equipment for detecting product exposure abnormity and computer storage medium - Google Patents
Method, system and equipment for detecting product exposure abnormity and computer storage medium Download PDFInfo
- Publication number
- CN115481299A CN115481299A CN202211119300.2A CN202211119300A CN115481299A CN 115481299 A CN115481299 A CN 115481299A CN 202211119300 A CN202211119300 A CN 202211119300A CN 115481299 A CN115481299 A CN 115481299A
- Authority
- CN
- China
- Prior art keywords
- exposure
- detected
- content
- log
- detection result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000001514 detection method Methods 0.000 claims abstract description 178
- 230000004044 response Effects 0.000 claims abstract description 99
- 230000002159 abnormal effect Effects 0.000 claims abstract description 71
- 238000004364 calculation method Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 114
- 230000005856 abnormality Effects 0.000 claims description 22
- 238000001914 filtration Methods 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 8
- 230000008439 repair process Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 230000004931 aggregating effect Effects 0.000 claims description 5
- 230000015556 catabolic process Effects 0.000 claims description 5
- 238000006731 degradation reaction Methods 0.000 claims description 5
- 238000012916 structural analysis Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 abstract description 4
- 238000004880 explosion Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 34
- 230000008569 process Effects 0.000 description 19
- 238000010586 diagram Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000011897 real-time detection Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000011895 specific detection Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9038—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a method, a system, equipment and a computer storage medium for detecting exposure abnormity of products, which are characterized in that response logs corresponding to exposure contents of products are obtained, a message queue is input, the response logs are extracted from the message queue, the related contents of the response logs are analyzed, the response logs are classified according to the function to be detected for flow calculation, and a detection result is obtained and stored; polling the stored detection result, and triggering and executing the processing flow when the detection result indicates abnormal exposure. And detecting the response logs corresponding to the classified products in real time by adopting a streaming calculation mode according to the function to be detected, and triggering and executing a corresponding processing flow at the first time under the condition that the detection result indicates abnormal exposure. The mode is not only low in time delay, and can find whether the explosion product is abnormal or not at the first time, but also is universal in each product, and the development cost is reduced.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a method, a system and equipment for detecting product exposure abnormity and a computer storage medium.
Background
Most internet products such as e-commerce, streaming media, take-out, content community, news, etc. guide users to browse products mainly through two functions of searching and recommending.
However, with existing searching or recommendation approaches, some unexpected content may be presented to the end user in the event of software system instability or technician error. For example, unqualified goods and other contents can be exposed when the filtering fails, or a large amount of single promotion video contents can be exposed to users when the breaking algorithm fails by the streaming media software, so that the regulation of 'over promotion guidance' of the network trust is violated.
At present, in the prior art, whether the content of exposure is abnormal is generally monitored through image detection, but the prior art has the problems of high development cost and high monitoring delay, and whether the exposure product is abnormal cannot be found at the first time.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a device and a computer storage medium for detecting exposure abnormality of a product, so as to solve the problems that the development cost of detecting exposure abnormality of the product is high, the monitoring delay is high, and it is impossible to find whether an exposure product is abnormal at the first time in the conventional picture detection.
In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:
the first aspect of the embodiment of the invention discloses a method for detecting exposure abnormality of a product, which comprises the following steps:
acquiring a response log corresponding to the exposure content of each product, and inputting the response log into a message queue, wherein the response log comprises an exposure content ID, an exposure position and related content;
extracting each response log from the message queue, analyzing the related content of each response log, classifying each response log by a function to be detected for flow calculation, and obtaining and storing a detection result;
polling the stored detection result, and triggering and executing a processing flow when the detection result indicates abnormal exposure.
Preferably, the extracting each response log from the message queue, analyzing the related content of each response log, performing stream computation on each response log according to the classification of the function to be detected, obtaining and storing the detection result, includes:
extracting each response log from the message queue by using a stream computation engine;
decoding and structural analysis are respectively carried out on each response log to obtain respective corresponding log content;
filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as to-be-detected logs;
and classifying the logs to be detected according to the functions to be detected, and performing corresponding classified function detection on the classified logs to be detected to obtain detection results indicating exposure abnormality or non-exposure abnormality and storing the detection results.
Preferably, the classifying the logs to be detected according to the functions to be detected, and performing the corresponding classified function detection on the classified logs to be detected to obtain and store the detection result indicating abnormal exposure or abnormal non-exposure, includes:
classifying the logs to be detected according to the to-be-detected function of sensitive content exposure detection to obtain a first log to be detected;
acquiring an exposure content ID in the first log to be detected;
retrieving in real time in a blacklist based on the exposure content ID;
if the sensitive content ID matched with the exposure content ID is searched in the blacklist, obtaining a detection result indicating abnormal exposure, and writing the detection result indicating abnormal exposure into a middle stage;
and if the sensitive content ID matched with the exposure content ID is not searched in the blacklist, obtaining a detection result indicating non-exposure abnormity, and writing the detection result indicating non-exposure abnormity into a middle stage.
Preferably, the classifying the logs to be detected according to the functions to be detected, and performing the corresponding classified function detection on the classified logs to be detected to obtain and store the detection result indicating abnormal exposure or abnormal non-exposure, includes:
classifying the logs to be detected according to the function to be detected of the scattering and de-weighting abnormity detection to obtain a second log to be detected;
aiming at each second log to be detected, acquiring an exposure content ID and a product ID in the second log to be detected;
determining a label database associated with the product ID, and inquiring and associating the corresponding label in the label database based on the exposure content ID;
grouping and aggregating according to users to which products indicated by the product IDs belong to obtain a plurality of groups to be detected divided by the users, wherein each group to be detected comprises labels associated with exposure content IDs corresponding to the product IDs of the same user;
counting the total number of the same labels in a pre-configured window aiming at each group to be detected, taking the total number of the same labels as the exposure times of the exposure content indicated by the corresponding exposure content ID, and indicating the same exposure content by the same exposure content ID;
judging whether the total number of the same labels is larger than a preset value or not;
if so, determining that the product indicated by the same label is abnormal in exposure, and writing a detection result indicating abnormal exposure into a middle table;
if not, determining that the product indicated by the same label is abnormal in non-exposure, and writing a detection result indicating the abnormal non-exposure into a middle table.
Preferably, when the detection result indicates that the exposure is abnormal, the triggering and executing process includes:
and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and triggering degradation operation.
Preferably, when the detection result indicates that the exposure is abnormal, the triggering and executing process includes:
and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and a repair prompt.
The second aspect of the embodiment of the present invention discloses a system for detecting exposure abnormality of a product, which includes:
the service module is used for acquiring a response log corresponding to the exposure content of each product and inputting the response log into a message queue, wherein the response log comprises an exposure content ID, an exposure position and related content;
the stream type calculation module is used for extracting each response log from the message queue, analyzing the related content of each response log, performing stream type calculation on each response log by classification of functions to be detected, obtaining a detection result and storing the detection result to the middle station;
and the middle platform is used for polling the stored detection result, and triggering and executing a processing flow when the detection result indicates that the exposure is abnormal.
Preferably, the streaming computation module includes:
an analysis unit for extracting respective response logs from the message queue using a stream calculation engine; decoding and structure analysis are respectively carried out on each response log to obtain respective corresponding log content; filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as to-be-detected logs;
the shunting unit is used for classifying the logs to be detected according to the functions to be detected and inputting the classified logs to be detected to the corresponding function detection unit;
and the function detection unit is used for carrying out corresponding classified function detection on the input classified logs to be detected according to the function classification, and writing the detection result indicating abnormal exposure or abnormal non-exposure into the middle table.
A third aspect of an embodiment of the present invention discloses an electronic device, including a memory and a processor;
the memory for storing a computer program;
the processor is configured to implement the method for detecting exposure anomaly of a product as disclosed in the first aspect of the embodiment of the present invention when the processor calls and executes the computer program stored in the memory.
The fourth aspect of the embodiments of the present invention discloses a computer storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are loaded and executed by a processor, the method for detecting exposure anomaly of a product disclosed in the first aspect of the embodiments of the present invention is implemented.
Based on the method, the system, the equipment and the computer storage medium for detecting the exposure abnormity of the product, which are provided by the embodiment of the invention. The method comprises the steps of inputting a message queue by acquiring a response log corresponding to exposure content of each product, wherein the response log comprises an exposure content ID, an exposure position and related content; extracting each response log from the message queue, analyzing the related content of each response log, classifying each response log by a function to be detected for flow calculation, and obtaining and storing a detection result; polling the stored detection result, and triggering and executing a processing flow when the detection result indicates that the exposure is abnormal. In the embodiment of the invention, the response logs corresponding to the classified products are detected in real time by adopting a streaming computing mode according to the function to be detected, and the corresponding processing flow is triggered and executed at the first time under the condition that the detection result indicates abnormal exposure. The mode is not only low in time delay, and can find whether the explosion product is abnormal or not at the first time, but also is universal in each product, and the development cost is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic diagram of a system for detecting exposure anomaly of a product according to an embodiment of the present invention;
FIG. 2 is a schematic flowchart illustrating a method for detecting abnormal exposure of a product according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a home typing interface of an existing Internet product according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart illustrating a sensitive content detection according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart illustrating detection of a loose duplicate removal anomaly according to an embodiment of the present invention;
FIG. 6 is an exemplary diagram of an exemplary implementation of a scatter-and-repeat anomaly detection disclosed in an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a system for detecting exposure abnormality of a product according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Fig. 1 is a schematic diagram of a product exposure anomaly detection system according to an embodiment of the present invention. The system architecture for detecting the exposure abnormality of the product is composed of a service module 1, a streaming calculation module 2 and a middle platform 3 from left to right in the direction of log data flow.
The service module 1 comprises a plurality of modules, and each module correspondingly processes one product.
Four modules are shown in fig. 1, module 10, module 12, module 13 and module 14.
And response logs of related services of various product exposures are uniformly input into the message queue through the corresponding modules. The message queue can be a Kafka message queue or other similar message queues. The response log is stored in json format.
The response log typically includes the exposure content ID and the exposure position. However, other contents such as the score of the ranking algorithm, the source, etc. may also be included according to different business forms.
The streaming computation module 2 is implemented by a Flink, and the streaming computation module 2 mainly includes a shunting node 20 and a function monitoring node, and executes two steps of shunting and function monitoring.
The function monitoring is realized by function monitoring nodes which respectively execute different function monitoring, and the monitoring of each function monitoring node is processed by a separate task. The detection logic of each individual task is realized in a universal extensible mode, so that modules in the same field can be rapidly connected to each function detection, and each function detection and subsequent newly-added function detection can be adapted to all modules, so that the universality of product exposure abnormity detection is realized.
Four classes of function monitoring nodes, function 21 monitoring node, function 22 monitoring node, function 23 monitoring node and function 24 monitoring node, are shown in fig. 1.
The specific implementation process of the streaming computation module 2 is as follows:
step one, extracting, analyzing and filtering the response logs of the related services of various product exposures from the Kafka message queue. And the response logs are shunted through the functions required to be detected, and each response log is shunted to the functional module of the function required to be detected for monitoring, namely, real-time detection is carried out, so that whether the product exposure corresponding to the response log of the function is abnormal or not can be found in the first time.
Wherein, the extraction mode mainly comprises the following steps: response logs are extracted from the kafka message queue or other similar message queues by a Flink calculation engine or other similar stream calculation engine (such as spark) according to the 'module' field of the response logs, namely, the response logs from the modules are identified.
The main analysis method comprises the following steps: and decoding and structure resolving are carried out on the response log.
The main filtration modes are as follows: and filtering the response log according to the extracted field content and the monitoring requirement, namely filtering logs of other services which possibly exist in the message queue and do not meet the monitoring requirement, and reserving the response log meeting the monitoring requirement.
The main shunting mode is as follows: and after the extraction, the analysis and the filtration, determining response logs meeting monitoring requirements of all the modules, and shunting the functions to be detected according to the response logs to the corresponding function monitoring modules for detection.
For example, the function to be detected of the response log in the module 11 is the function 21 and the function 22, and the module 11 needs to access the function 21 monitoring node and the function 22 monitoring node, and the response log output in the module 11 is pushed to the function 21 monitoring node and the function 22 monitoring node downstream through the shunting node 20. The function to be detected of the response log in the module 12 is the function 23, the module 12 needs to access the function 23 monitoring node, and the response log in the module 12 is pushed to the downstream function 23 monitoring node through the shunting node 20.
And step two, each function monitoring node parallelly detects the abnormity of the response log of the shunt input through the flow calculation with low delay, and writes the obtained detection result into the database of the middle station 3 in real time.
The middle station 3 acquires the detection result by polling the database, gives an alarm when the detection result indicates that the exposure content of the product is abnormal, and triggers degradation or notifies technicians to repair the product.
Based on the product exposure anomaly detection system architecture disclosed by the embodiment of the invention, corresponding detection, alarm and processing flows are executed, and in the detection process, the real-time detection realized by the Flink stream type calculation engine can realize the second-level delay and reduce the delay of product exposure anomaly detection. Different modules are accessed to the corresponding function monitoring nodes by dividing different function monitoring nodes, under the condition that the form of each service display product of the internet is fully considered, the function detection is realized by using universal logic, so that not only can a new module be quickly accessed to the existing function detection, but also the newly realized function detection can be adapted to all modules, and the universality of the detection of the exposure abnormity of the product is realized. Meanwhile, when the function monitoring nodes are used for detection, the calculation tasks of the function monitoring nodes are independently carried out and are isolated from each other, so that mutual influence is avoided, and the function monitoring nodes are deployed in a high-availability cloud cluster and can be timely recovered when the calculation tasks are wrong.
Based on the product exposure anomaly detection system architecture disclosed in fig. 1, the embodiment of the present invention further discloses a product exposure anomaly detection method, as shown in fig. 2, the method includes the following steps:
s201: and acquiring a response log corresponding to the exposure content of each product, and inputting the response log into a message queue.
In S201, the response log includes an exposure content ID, an exposure position, and related content.
The relevant contents correspond to the field contents of the module shown in fig. 1.
The message queue can be a Kafka message queue or other similar message queues.
In the specific process of executing S201, response logs corresponding to the exposed relevant service content of each product are input into the message queue in a uniform format through different modules (such as the modules in fig. 1).
The specific implementation process of S201 can also refer to the implementation process of the service module 1 shown in fig. 1.
S202: and extracting each response log from the message queue, analyzing the related content of each response log, classifying each response log by a function to be detected to perform stream calculation, and obtaining and storing a detection result.
In the specific execution of S202:
first, a stream computation engine is used to extract individual response logs from the message queue.
The stream calculation engine may be a Flink calculation engine or other similar stream calculation engines.
And secondly, decoding and structure analysis are respectively carried out on each response log to obtain respective corresponding log content.
And secondly, filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as the logs to be detected.
And finally, classifying the logs to be detected according to the functions to be detected, and performing corresponding classified function detection on the classified logs to be detected to obtain and store detection results indicating abnormal exposure or abnormal non-exposure.
And executing the function detection of each classification independently and executing the function detection of each classification in parallel.
The specific implementation process of S202 can also refer to the implementation process of the streaming computing module 2 shown in fig. 1.
S203: polling the stored detection result, and triggering and executing a processing flow when the detection result indicates abnormal exposure.
In the specific execution of S203, when the detection result indicates that the exposure is abnormal, the trigger and execution processing flow includes, but is not limited to, the following two.
The first method comprises the following steps: and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and triggering degradation operation.
And the second method comprises the following steps: and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and a repair prompt. And a technician can repair the product service with abnormal exposure according to the repair prompt.
In the embodiment of the invention, the response logs corresponding to the classified products are detected in real time by adopting a stream type calculation mode according to the function to be detected, so that the time delay of the second level can be realized, and the time delay of the detection of the exposure abnormity of the products is reduced. Meanwhile, by dividing different function monitoring, under the condition of fully considering the forms of products displayed by various services of the Internet, the function detection is realized by using universal logic, so that not only can different products be quickly accessed to the existing function detection, but also the new function detection can be adapted to all products, and the universality of the exposure abnormality detection of the products is realized. Meanwhile, all the function detection is executed in parallel, and the detection of different functions is isolated, so that mutual influence is avoided, and the detection efficiency and accuracy are improved. Furthermore, if the system is deployed in a high-availability cloud cluster, the system can be matched with service integration to perform various different function detections, and can also recover in time when a calculation task goes wrong.
In the method for detecting exposure abnormality of a product disclosed in the embodiment of the present invention, the logs to be detected are classified according to functions to be detected, and the classified logs to be detected are subjected to function detection of corresponding classification, so as to obtain a detection result indicating exposure abnormality or non-exposure abnormality and store the detection result.
As shown in fig. 3, the home typing interface of most internet products is a home typing interface for searching and recommending two modules to guide the user. There is generally a column of search bar 31 at the top of the screen, and the display part below the search bar is used to display the recommended products or the search results after the user searches. Shown in FIG. 3 is a single column of the display interface 32 on which the first recommended product, the second recommended product, the third recommended product, and the fourth recommended product are displayed. In any display mode, the detection of the exposure abnormality of the product is the same.
Two kinds of function monitoring (function detection) which are most commonly required by internet products at present are taken as an example for description.
The first method comprises the following steps: and detecting sensitive content.
Sensitive content is a necessary function for each product of the internet, and because content can be uploaded by a user at a B end or a C end on the internet at present, all internet products need to filter the sensitive content of exposed content, which is generally realized by accessing a blacklist.
Based on this, as shown in fig. 4, the specific detection process of the sensitive content detection is as follows:
s401: and classifying the logs to be detected according to the to-be-detected function of sensitive content exposure detection to obtain a first log to be detected.
S402: and acquiring the exposure content ID in the first log to be detected.
S403: retrieving in real time in a blacklist based on the exposure content ID, and if a sensitive content ID matched with the exposure content ID is retrieved in the blacklist, executing S404; if the sensitive content ID matching the exposure content ID is not retrieved from the blacklist, S405 is performed.
Since there are many blacklists, the form of the blacklist for different services is different, and there will be multiple blacklists.
For example, companies may filter exposed content through blacklists provided by the national internet information office.
For example, in some internet products, a blackout function is provided for the user, such as in a streaming media product, where the user specifies to blackout all animation-related content, the recommendation algorithm performs content filtering according to the user's personal blacklist.
Therefore, in performing S403, multiple filtering, i.e., real-time detection, is required.
In an embodiment of the present invention, real-time search may be performed according to a blacklist set by a technician.
In an embodiment of the present invention, a real-time search may be performed according to a blacklist related to a service type of a currently detected product.
Before S403 is executed in detail, the blacklist (table) may be configured to the flow, that is, to the streaming calculation of S202 shown in fig. 2, in a configuration manner.
S404: and obtaining a detection result indicating abnormal exposure, and writing the detection result indicating abnormal exposure into the middle stage.
S405: and obtaining a detection result indicating non-exposure abnormity, and writing the detection result indicating non-exposure abnormity into the middle table.
When S404 and S405 are specifically executed, the respective detection results are written into the database of the middle station.
And the second method comprises the following steps: and (5) scattering and removing weight abnormality detection.
The effect of the ripping and deduplication is to avoid exposing a single content or a single type of content to the user, thereby recommending a rich variety of content to the user, improving the quality of the overall exposed content, and avoiding violating regulations such as web-letter "over-publicity guidance" (i.e., exposing a large amount of single-star content to the user).
Based on this, as shown in fig. 5, the specific detection process of the detection of the loose duplicate removal anomaly is as follows:
s501: and classifying the logs to be detected according to the function to be detected of the duplicate removal anomaly detection needing to be scattered to obtain a second log to be detected.
S502: and acquiring the exposure content ID and the product ID in each second log to be detected.
S503: and determining a label database associated with the product ID, and inquiring and associating the corresponding label in the label database based on the exposure content ID.
S504: and grouping and aggregating according to the users to which the products indicated by the product IDs belong to obtain a plurality of groups to be detected, which are divided by the users.
In S504, each pending detection group includes tags associated with exposure content IDs corresponding to a plurality of product IDs of the same user.
Different colors may be used to distinguish different users in a particular application.
S505: and counting the total number of the same labels in a pre-configured window aiming at each group to be detected, and taking the total number of the same labels as the exposure times of the exposure content indicated by the corresponding exposure content ID.
In S505, the same exposure content ID indicates the same exposure content.
The window is preconfigured, i.e. the windowing of each pending group is preconfigured. The unit of the pre-configured window may be the number of exposures n, or may be the time t. n is a positive number greater than 1. t is a positive number and the time unit is seconds, minutes or hours.
S506: judging whether the total number of the same labels is greater than a preset value or not, and if so, executing S507; if not, go to S508.
In the specific process of S506, if the unit of the preconfigured window is the exposure number n, the preset value is the preset exposure number. Judging whether the total number of the same labels is larger than a preset value or not: and judging whether the exposure times of the exposure contents indicated by the exposure content ID are larger than the preset exposure times.
If the pre-configured window is time t, the preset value is the exposure times in each time unit t. Judging whether the total number of the same labels is larger than a preset value or not: it is determined whether the number of exposures of the exposure content indicated by the exposure content ID is greater than the number of exposures per t time unit.
S507: and determining the exposure abnormality of the product indicated by the same label, and writing a detection result indicating the exposure abnormality into the middle table.
S508: and determining that the product indicated by the same label is abnormal in non-exposure, and writing a detection result indicating the abnormal in non-exposure into a middle table.
When S507 and S508 are specifically executed, the respective detection results are written into the database of the middle station.
The above process of scatter-and-repeat anomaly detection is illustrated here in fig. 6.
Stage one is the supplementary label dimension:
associating the original data stream with a tag database in real time, supplementing tag dimensions, taking each square as a one-time exposure record, and distinguishing different users by colors, wherein a user A and a user B are shown in FIG. 6; the numbers in the squares represent the complemented labels, with labels S1 and S2 shown in FIG. 6.
Stage two windowing of the user groups:
and after the stage II is entered, grouping and aggregating each user, and windowing to extract n exposures or exposures within each t minute.
Stage three is grouping by label:
and after entering the third stage, counting each label.
Assume that the threshold for this detection is that the number of exposures per tag per window cannot be greater than 1, where:
user a, tag S1, exposure number: 1;
user a, tag S2, exposure number: 2;
user B, tag S1, exposure number: 1;
user B, tag S2, exposure number: 1;
the product of tag 2 of user a in fig. 6, with more than 1 exposure per windowing, triggers an exception.
Based on the method for detecting abnormal exposure of a product disclosed in the embodiment of the present invention, the embodiment of the present invention further discloses a system for detecting abnormal exposure of a product, as shown in fig. 7, the system includes: a service module 71, a streaming calculation module 72 and a central station 73.
The service module 71 is configured to obtain a response log corresponding to the exposure content of each product, and input the response log into a message queue, where the response log includes an exposure content ID, an exposure position, and related content.
The specific implementation process of the service module 71 can be referred to the service module 1 shown in fig. 1.
And the stream type calculation module 72 is configured to extract each response log from the message queue, analyze the related content of each response log, perform stream type calculation on each response log by classification of the function to be detected, obtain a detection result, and store the detection result in the middle station 73.
The specific implementation of the streaming computation module 72 can be referred to the streaming computation module 2 shown in fig. 1.
Specifically, the streaming calculation module 72 includes:
an analysis unit for extracting respective response logs from the message queue using a stream calculation engine; decoding and structure analysis are respectively carried out on each response log to obtain respective corresponding log content; and filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as the logs to be detected.
And the shunting unit is used for classifying the logs to be detected according to the functions to be detected and inputting the classified logs to be detected to the corresponding function detection unit.
And the function detection unit is used for carrying out corresponding classified function detection on the input classified logs to be detected according to the function classification, and writing the detection result indicating abnormal exposure or abnormal non-exposure into the middle stage.
If the function to be detected is sensitive content exposure detection, the function detection unit is specifically configured to:
classifying the logs to be detected according to the to-be-detected function of sensitive content exposure detection to obtain a first log to be detected; acquiring an exposure content ID in the first log to be detected; retrieving in real time in a blacklist based on the exposure content ID; if the sensitive content ID matched with the exposure content ID is searched in the blacklist, obtaining a detection result indicating abnormal exposure, and writing the detection result indicating abnormal exposure into a middle stage; and if the sensitive content ID matched with the exposure content ID is not searched in the blacklist, obtaining a detection result indicating non-exposure abnormity, and writing the detection result indicating non-exposure abnormity into a middle stage.
If the function of waiting to examine is for scattering heavy anomaly detection, this function detecting element specifically is used for:
classifying the logs to be detected according to the function to be detected of the scattering and de-weighting abnormity detection to obtain a second log to be detected; aiming at each second log to be detected, acquiring an exposure content ID and a product ID in the second log to be detected; determining a label database associated with the product ID, and inquiring and associating the corresponding label in the label database based on the exposure content ID; grouping and aggregating according to users to which products indicated by the product IDs belong to obtain a plurality of groups to be detected divided by the users, wherein each group to be detected comprises labels associated with exposure content IDs corresponding to the product IDs of the same user; counting the total number of the same labels in a pre-configuration window for each group to be detected, taking the total number of the same labels as the exposure times of exposure content indicated by corresponding exposure content ID, and indicating the same exposure content by the same exposure content ID; judging whether the total number of the same labels is larger than a preset value or not; if so, determining that the product indicated by the same label is abnormal in exposure, and writing a detection result indicating abnormal exposure into the middle stage; if not, determining that the product indicated by the same label is not in abnormal non-exposure, and writing a detection result indicating the abnormal non-exposure into the middle stage.
And the middle stage 73 is used for polling the stored detection result, and triggering and executing a processing flow when the detection result indicates that the exposure is abnormal.
The specific implementation of the middle stage 73 can be seen in the middle stage 3 shown in fig. 1.
In an embodiment of the present invention, the middle stage 73 is specifically configured to poll the stored detection result, and when the detection result indicates that the exposure is abnormal, output an alarm prompt and trigger a degradation operation.
In an embodiment of the present invention, the middle stage 73 is specifically configured to poll the stored detection result, and output an alarm prompt and a repair prompt when the detection result indicates that the exposure is abnormal.
In the embodiment of the invention, in the detection process, the streaming computation module can realize the time delay of a second level through the real-time detection realized by the Flink streaming computation engine, and the time delay of the detection of the exposure abnormity of the product is reduced. By dividing different function detection units, different products are accessed to the corresponding function detection units, under the condition that the forms of products are shown by all services of the Internet are fully considered, the function detection is realized by using universal logic, so that not only is the new module quickly accessed to the existing function detection, but also the newly realized function detection can be adapted to all products, and the universality of the detection of the exposure abnormity of the products is realized. Meanwhile, during detection, the calculation tasks of the function detection units are independently performed and isolated from each other, so that mutual influence is avoided, and the function detection units are deployed in a high-availability cloud cluster and can be timely recovered when the calculation tasks are wrong.
The embodiment of the invention also provides electronic equipment which comprises a memory and a processor.
The memory is used for storing a computer program.
The processor is configured to implement the method for detecting exposure anomaly of a product disclosed in the embodiment of the present invention when the processor calls and executes the computer program stored in the memory.
The embodiment of the present invention further provides a computer storage medium, where one or more computer-executable instructions are stored in the computer storage medium, and when the computer-executable instructions are loaded and executed by a processor, the method for detecting exposure anomaly of a product disclosed in the embodiment of the present invention is implemented.
All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, the system or system embodiments, which are substantially similar to the method embodiments, are described in a relatively simple manner, and reference may be made to some descriptions of the method embodiments for relevant points. The above-described system and system embodiments are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement without inventive effort.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the components and steps of the various examples have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for detecting exposure abnormality of a product, the method comprising:
acquiring a response log corresponding to the exposure content of each product, and inputting the response log into a message queue, wherein the response log comprises an exposure content ID, an exposure position and related content;
extracting each response log from the message queue, analyzing the related content of each response log, classifying each response log by a function to be detected for flow calculation, and obtaining and storing a detection result;
polling the stored detection result, and triggering and executing a processing flow when the detection result indicates abnormal exposure.
2. The method of claim 1, wherein the extracting each response log from the message queue, analyzing the related content of each response log, performing streaming computation on each response log in a classification of functions to be examined, and obtaining and storing a detection result comprises:
extracting each response log from the message queue by using a stream computation engine;
decoding and structural analysis are respectively carried out on each response log to obtain respective corresponding log content;
filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as to-be-detected logs;
and classifying the logs to be detected according to functions to be detected, and performing corresponding classified function detection on the classified logs to be detected to obtain and store detection results indicating abnormal exposure or abnormal non-exposure.
3. The method according to claim 2, wherein the classifying the logs to be detected according to the functions to be detected, performing the corresponding classified function detection on the classified logs to be detected, and obtaining and storing the detection result indicating exposure abnormality or non-exposure abnormality comprises:
classifying the logs to be detected according to the to-be-detected function of sensitive content exposure detection to obtain a first log to be detected;
acquiring an exposure content ID in the first log to be detected;
retrieving in real time in a blacklist based on the exposure content ID;
if the sensitive content ID matched with the exposure content ID is searched in the blacklist, obtaining a detection result indicating abnormal exposure, and writing the detection result indicating abnormal exposure into a middle stage;
and if the sensitive content ID matched with the exposure content ID is not searched in the blacklist, obtaining a detection result indicating non-exposure abnormity, and writing the detection result indicating non-exposure abnormity into a middle stage.
4. The method according to claim 2, wherein the classifying the log to be detected according to the function to be detected, and performing the function detection of the classified log to be detected in corresponding classification to obtain and store the detection result indicating the exposure abnormality or the non-exposure abnormality comprises:
classifying the logs to be detected according to the function to be detected of the scattering and de-weighting abnormal detection to obtain a second log to be detected;
aiming at each second log to be detected, acquiring an exposure content ID and a product ID in the second log to be detected;
determining a label database associated with the product ID, and inquiring and associating the corresponding label in the label database based on the exposure content ID;
grouping and aggregating according to users to which products indicated by the product IDs belong to obtain a plurality of groups to be detected divided by the users, wherein each group to be detected comprises labels associated with exposure content IDs corresponding to the product IDs of the same user;
counting the total number of the same labels in a pre-configuration window for each group to be detected, taking the total number of the same labels as the exposure times of exposure content indicated by corresponding exposure content ID, and indicating the same exposure content by the same exposure content ID;
judging whether the total number of the same labels is larger than a preset value or not;
if so, determining that the product indicated by the same label is abnormal in exposure, and writing a detection result indicating abnormal exposure into a middle table;
if not, determining that the product indicated by the same label is not in abnormal non-exposure, and writing a detection result indicating the abnormal non-exposure into the middle stage.
5. The method according to any one of claims 1 to 4, wherein when the detection result indicates that the exposure is abnormal, triggering and executing the processing flow comprises:
and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and triggering degradation operation.
6. The method according to any one of claims 1 to 4, wherein when the detection result indicates that the exposure is abnormal, triggering and executing the processing flow comprises:
and when the detection result indicates that the exposure is abnormal, outputting an alarm prompt and a repair prompt.
7. A product exposure anomaly detection system, said system comprising:
the service module is used for acquiring a response log corresponding to the exposure content of each product and inputting the response log into a message queue, wherein the response log comprises an exposure content ID, an exposure position and related content;
the stream type calculation module is used for extracting each response log from the message queue, analyzing the related content of each response log, performing stream type calculation on each response log by classification of functions to be detected, obtaining a detection result and storing the detection result to the middle station;
and the middle platform is used for polling the stored detection result, and triggering and executing a processing flow when the detection result indicates that the exposure is abnormal.
8. The system of claim 7, wherein the streaming module comprises:
an analysis unit for extracting respective response logs from the message queue using a stream calculation engine; decoding and structural analysis are respectively carried out on each response log to obtain respective corresponding log content; filtering based on the log content corresponding to each response log, and reserving the response logs with the log contents meeting the detection requirements as to-be-detected logs;
the shunting unit is used for classifying the logs to be detected according to the functions to be detected and inputting the classified logs to be detected to the corresponding function detection units;
and the function detection unit is used for carrying out corresponding classified function detection on the input classified logs to be detected according to the function classification, and writing the detection result indicating abnormal exposure or abnormal non-exposure into the middle stage.
9. An electronic device, wherein the electronic device comprises a memory and a processor;
the memory for storing a computer program;
the processor, when calling and executing the computer program stored in the memory, is used for implementing the product exposure abnormality detection method according to any one of claims 1 to 6.
10. A computer storage medium having stored thereon computer-executable instructions which, when loaded and executed by a processor, implement the method of detecting exposure anomalies in a product according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211119300.2A CN115481299A (en) | 2022-09-14 | 2022-09-14 | Method, system and equipment for detecting product exposure abnormity and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211119300.2A CN115481299A (en) | 2022-09-14 | 2022-09-14 | Method, system and equipment for detecting product exposure abnormity and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115481299A true CN115481299A (en) | 2022-12-16 |
Family
ID=84423996
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211119300.2A Pending CN115481299A (en) | 2022-09-14 | 2022-09-14 | Method, system and equipment for detecting product exposure abnormity and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115481299A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115629299A (en) * | 2022-12-19 | 2023-01-20 | 柯泰光芯(常州)测试技术有限公司 | Semiconductor chip testing method for realizing isolation Kelvin test |
-
2022
- 2022-09-14 CN CN202211119300.2A patent/CN115481299A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115629299A (en) * | 2022-12-19 | 2023-01-20 | 柯泰光芯(常州)测试技术有限公司 | Semiconductor chip testing method for realizing isolation Kelvin test |
| CN115629299B (en) * | 2022-12-19 | 2023-03-17 | 柯泰光芯(常州)测试技术有限公司 | Semiconductor chip testing method for realizing isolation Kelvin test |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Landauer et al. | System log clustering approaches for cyber security applications: A survey | |
| JP6919569B2 (en) | Log analysis systems, methods, and recording media | |
| CN113489713B (en) | Network attack detection method, device, equipment and storage medium | |
| CN108600172B (en) | Method, device and equipment for detecting database collision attack and computer readable storage medium | |
| CN113254255B (en) | Cloud platform log analysis method, system, device and medium | |
| CN111913860B (en) | Operation behavior analysis method and device | |
| CN113271322B (en) | Abnormal flow detection method and device, electronic equipment and storage medium | |
| US11477223B2 (en) | Systems and methods for analyzing cybersecurity events | |
| CN112671767B (en) | Security event early warning method and device based on alarm data analysis | |
| CN115481299A (en) | Method, system and equipment for detecting product exposure abnormity and computer storage medium | |
| CN113010637A (en) | Text auditing method and device | |
| CN113515434A (en) | Abnormity classification method, abnormity classification device, abnormity classification equipment and storage medium | |
| CN115033876A (en) | Log processing method, log processing device, computer device and storage medium | |
| CN107609020B (en) | Log classification method and device based on labels | |
| WO2017179778A1 (en) | Search method and apparatus using big data | |
| CN116881962A (en) | Security monitoring system, method, device and storage medium | |
| CN111475405A (en) | Regression testing method and device, computer equipment and storage medium | |
| CN115186001A (en) | Patch processing method and device | |
| EP3361405A1 (en) | Enhancement of intrusion detection systems | |
| CN114996080A (en) | Data processing method, device, equipment and storage medium | |
| JP2009289016A (en) | Method for analyzing text data in communication service application, text data analyzing device, and program for the same | |
| CN114189585A (en) | Crank call abnormity detection method and device and computing equipment | |
| Pina | Automatic detection of anomalous user access patterns to sensitive data | |
| CN116991680B (en) | Log noise reduction method and electronic equipment | |
| CN116015873B (en) | Network security alarm processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |