CN113570201A - Data processing method, device, equipment, storage medium and program product - Google Patents

Data processing method, device, equipment, storage medium and program product Download PDF

Info

Publication number
CN113570201A
CN113570201A CN202110745572.2A CN202110745572A CN113570201A CN 113570201 A CN113570201 A CN 113570201A CN 202110745572 A CN202110745572 A CN 202110745572A CN 113570201 A CN113570201 A CN 113570201A
Authority
CN
China
Prior art keywords
feature
original
wind control
features
binding relationship
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110745572.2A
Other languages
Chinese (zh)
Inventor
黎明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202110745572.2A priority Critical patent/CN113570201A/en
Publication of CN113570201A publication Critical patent/CN113570201A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0635Risk analysis of enterprise or organisation activities
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/211Selection of the most significant subset of features

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • General Physics & Mathematics (AREA)
  • Strategic Management (AREA)
  • Data Mining & Analysis (AREA)
  • Economics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Game Theory and Decision Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Development Economics (AREA)
  • Evolutionary Computation (AREA)
  • Educational Administration (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present disclosure relates to a data processing method, apparatus, electronic device, storage medium, and program product, wherein the method comprises: analyzing a wind control event of the wind control historical data to obtain an original characteristic; establishing a binding relationship between the wind control event and the original characteristics, and storing the original characteristics and the binding relationship to a characteristic database; and searching the original characteristics and the binding relationship from the characteristic database according to the binding relationship reading request of the terminal, and returning the original characteristics and the binding relationship to the terminal. The method and the device avoid manual selection and binding of the original characteristics for the wind control events in the RCP, realize automatic binding of the wind control events and the original characteristics of the wind control events, and improve the efficiency of binding operation. The terminal is responsible for sending the binding relation and reading the request and showing the function of the original characteristic, and does not need to transmit data with the server in order to bind the wind control event and the original characteristic, so that the operating pressure of the terminal is reduced.

Description

Data processing method, device, equipment, storage medium and program product
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.
Background
Risk control (wind control for short) means that a risk manager takes various measures and methods to eliminate or reduce various possibilities of occurrence of a risk event, or a risk controller reduces loss caused when a risk event occurs.
One operation performed by a user on the terminal may be understood as one event in the wind control system, i.e. a wind control event. The operation information carried in the operation, such as a user name, a terminal identifier, an operation object, and the like, can be understood as the wind control characteristics in the wind control event. After the wind control event is bound with the wind control characteristics, the wind control event can be analyzed according to the wind control characteristics so as to control the wind control event. In the related art, the wind control events and the wind control features are bound manually on a Rich Client Platform (RCP), and the binding operation is inefficient.
Disclosure of Invention
The present disclosure provides a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, to at least solve the problem in the related art that a binding operation between a wind control event and a wind control feature is manually performed by a human, and the binding operation is inefficient. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a data processing method, the method including: acquiring wind control historical data to be processed; analyzing the wind control events of the wind control historical data to obtain original characteristics; establishing a binding relationship between the wind control event and the original characteristic, and storing the original characteristic and the binding relationship to a characteristic database; receiving a binding relationship reading request sent by a terminal; and searching the original characteristics and the binding relationship from the characteristic database according to the binding relationship reading request, and returning the original characteristics and the binding relationship to the terminal.
Optionally, the analyzing the wind control event of the wind control historical data to obtain an original feature includes: and analyzing the original features from the wind control events of the wind control historical data according to field identifications corresponding to fields where the original features are located.
Optionally, the binding relationship reading request includes an event identifier of the wind control event; the searching the original feature and the binding relationship from the feature database according to the binding relationship reading request comprises: searching the wind control event from the characteristic database according to the event identification; and searching the original characteristics and the binding relation from the characteristic database according to the wind control event.
Optionally, after the returning the original feature and the binding relationship to the terminal, the method further includes: receiving a feature display request sent by the terminal; searching an auxiliary feature related to the original feature from the feature database according to the feature display request; extracting candidate features from the auxiliary features according to preset strategy information; and returning the candidate characteristics to the terminal.
Optionally, the feature display request includes a feature identifier of the original feature; the searching the auxiliary features related to the original features from the feature database according to the feature showing request comprises the following steps: searching the feature database for at least one of the following auxiliary features related to the original feature according to the feature identification: derived features, aggregated features, and service features.
Optionally, after the storing the raw features to a feature database, the method further comprises: searching historical original features having a binding relation with the wind control events from the feature database; and deleting the features which are different from the original features in the historical original features.
Optionally, the method further comprises: acquiring characteristic indexes of sample characteristics according to a preset period; comparing the characteristic index acquired in the current period with the characteristic index acquired in the previous period to obtain a period comparison result; and storing the period comparison result to the feature database, and displaying the feature index and the period comparison result according to the form of a chart.
Optionally, the method further comprises: acquiring a characteristic index of a target characteristic according to a preset period; and generating alarm information according to the characteristic indexes and corresponding index thresholds, and sending the alarm information to the terminal.
Optionally, the generating alarm information according to the feature index and the corresponding index threshold includes: comparing the characteristic indicator to the corresponding indicator threshold; and when the characteristic index is larger than the corresponding index threshold value, generating the alarm information.
Optionally, the generating alarm information according to the feature index and the corresponding index threshold includes: comparing the period comparison result with the corresponding index threshold; and when the periodic comparison result is greater than the corresponding index threshold value, generating the alarm information.
Optionally, the characteristic indicator comprises at least one of: null rate, number of unique values, top distribution ratio and feature quantile value; the graph includes at least one of: a single-axis line graph corresponding to the null rate or the number of unique values, a column packing graph corresponding to the top distribution proportion, and a multi-axis line graph corresponding to the feature quantile value.
According to a second aspect of embodiments of the present disclosure, there is provided a data processing apparatus, the apparatus comprising: the acquisition unit is configured to acquire wind control historical data to be processed; the analysis unit is configured to analyze the wind control events of the wind control historical data to obtain original characteristics; the binding unit is configured to establish a binding relationship between the wind control event and the original characteristic, and store the original characteristic and the binding relationship into a characteristic database; a receiving unit configured to execute a binding relationship reading request transmitted by a receiving terminal; a searching unit configured to perform searching for the original feature and the binding relationship from the feature database according to the binding relationship reading request, and return the original feature and the binding relationship to the terminal.
Optionally, the parsing unit is configured to perform parsing to obtain the original feature from the wind control event of the wind control history data according to a field identifier corresponding to a field where the original feature is located.
Optionally, the binding relationship reading request includes an event identifier of the wind control event; the search unit includes: an event searching unit configured to perform a search for the wind control event from the feature database according to the event identification; a feature and relationship search unit configured to perform a search for the original features and the binding relationships from the feature database according to the wind control events.
Optionally, the receiving unit is further configured to perform receiving, after the searching unit returns the original feature and the binding relationship to the terminal, a feature display request sent by the terminal; the searching unit is further configured to search the feature database for auxiliary features related to the original features according to the feature showing request; the device further comprises: and the extraction unit is configured to extract candidate features from the auxiliary features according to preset strategy information and return the candidate features to the terminal.
Optionally, the feature display request includes a feature identifier of the original feature; the searching unit is configured to search the feature database for at least one of the following auxiliary features related to the original feature according to the feature identification: derived features, aggregated features, and service features.
Optionally, the searching unit is further configured to perform, after the binding unit stores the original features into a feature database, searching a historical original feature having a binding relationship with the wind control event from the feature database; the device further comprises: a deleting unit configured to perform deletion of a feature different from the original feature in the history original feature.
Optionally, the obtaining unit is further configured to perform obtaining a feature index of the sample feature according to a preset period; the device further comprises: the comparison unit is configured to compare the characteristic index acquired in the current period with the characteristic index acquired in the previous period to obtain a period comparison result; and the display unit is configured to store the period comparison result into the characteristic database and display the characteristic index and the period comparison result according to the form of a chart.
Optionally, the obtaining unit is further configured to perform obtaining a feature index of the target feature according to a preset period; the device further comprises: and the alarm unit is configured to generate alarm information according to the characteristic indexes and corresponding index thresholds and send the alarm information to the terminal.
Optionally, the alarm unit includes: an index comparison unit configured to perform comparison of the feature index with the corresponding index threshold; an alarm generation unit configured to perform generation of the alarm information when the characteristic index is larger than the corresponding index threshold.
Optionally, the index comparing unit is further configured to perform comparison between the period comparison result and the corresponding index threshold; the alarm generation unit is further configured to execute generating the alarm information when the period comparison result is greater than the corresponding index threshold.
Optionally, the characteristic indicator comprises at least one of: null rate, number of unique values, top distribution ratio and feature quantile value; the graph includes at least one of: a single-axis line graph corresponding to the null rate or the number of unique values, a column packing graph corresponding to the top distribution proportion, and a multi-axis line graph corresponding to the feature quantile value.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data processing method of the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method according to the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the data processing method of the first aspect described above.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
according to the data processing scheme provided by the embodiment of the disclosure, the original features of the wind control events are analyzed from the historical wind control data to be processed, and then the binding relationship between the wind control events and the original features is established, so that the wind control events are prevented from being manually selected and bound by RCP, the wind control events and the original features of the wind control events are automatically bound, and the efficiency of binding operation is improved. Furthermore, the binding relationships and the original features are stored to a feature database. After receiving a binding relationship reading request sent by a terminal (such as RCP), the original features and the binding relationship can be searched from the feature database, and the original features and the binding relationship are returned to the terminal, so that the terminal can display the original features having the binding relationship with the wind control event. The terminal is responsible for sending the binding relation and reading the request and showing the function of the original characteristic, and does not need to transmit data with the server in order to bind the wind control event and the original characteristic, so that the operating pressure of the terminal is reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a flow diagram illustrating a data processing method according to an exemplary embodiment.
FIG. 2 is a graphical illustration of several characteristic indicators and cycle comparison results shown in accordance with an exemplary embodiment.
FIG. 3 is a flowchart illustrating a process for aggregating feature indicators of a wind control feature at a timed rate according to an example embodiment.
FIG. 4 is a flow diagram illustrating a process for timing hints for available ones of the wind-controlled features in accordance with an exemplary embodiment.
FIG. 5 is a block diagram illustrating a data processing apparatus according to an example embodiment.
FIG. 6 is a block diagram illustrating a data processing electronic device in accordance with an exemplary embodiment.
FIG. 7 is a block diagram illustrating an electronic device for processing data in accordance with an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Fig. 1 is a flowchart illustrating a data processing method according to an exemplary embodiment, and the data processing method may be applied to a server, which may be a wind control system server or provide technical services for a wind control system, as shown in fig. 1.
The data processing method shown in fig. 1 may specifically include the following steps.
In step S11, the pending wind control history data is acquired.
In an embodiment of the present disclosure, the wind control history data may be composed of one or more wind control events, and each wind control event may include one or more original features therein. In practical application, the to-be-processed wind control historical data can be read from the database. The database may be a ClickHouse (a listing store database). The clickwouse may store the wind control events transmitted by the respective clients.
In a practical example, the wind-controlled event may be a wind-controlled event in a short video application, such as a comment operation, a share operation, and the like. The original characteristic of the wind control event can be the account of the executor commenting the operation and the terminal information such as the terminal IP where the account of the executor logs in. The embodiments of the present disclosure do not specifically limit the event type, the event content, and the feature type, the feature content, and the like of the original features of the wind-controlled event.
In step S12, the wind control events of the wind control history data are analyzed to obtain the original features.
In the embodiment of the disclosure, one or more wind control events can be extracted from the wind control historical data, and the original features of the wind control events are obtained by analyzing the one or more wind control events. For example, the to-be-processed wind control history data includes wind control events S1 and S2, and an original feature T1 is resolved for the wind control event S1 and an original feature T2 is resolved for the wind control event S2.
In step S13, a binding relationship between the wind control event and the original feature is established, and the original feature and the binding relationship are stored in the feature database.
In the embodiment of the disclosure, after the original features of the wind control events are analyzed, the binding relationship between the wind control events and the respective original features is established. In the above example, a binding relationship B1 between the wind control event S1 and the original feature T1 is established, and a binding relationship B2 between the wind control event S2 and the original feature T2 is established. The original signature T1 and the binding relationship B1 are then stored to a signature database, and the original signature T2 and the binding relationship B2 are stored to a signature database.
In step S14, the binding relationship read request sent by the terminal is received.
In an embodiment of the present disclosure, the terminal may be an RCP, and a binding relationship reading request is generated on the terminal, where the binding relationship reading request is used to obtain, from the server, the original features bound to the wind control event.
In step S15, the original feature and the binding relationship are searched from the feature database according to the binding relationship read request, and the original feature and the binding relationship are returned to the terminal.
In the embodiment of the present disclosure, the binding relationship and the original feature corresponding to the binding relationship read request are searched from the feature database, and then the searched binding relationship and original feature are transmitted to the terminal. The binding relationship and the original characteristics can be shown on the terminal. For example, the terminal sends a binding relationship reading request for the wind control event S1 to the server, and the server returns the original feature T1 and the binding relationship B1 of the wind control event S1 to the terminal, then the terminal may expose the original feature T1 bound with the wind control event S1.
According to the data processing scheme provided by the embodiment of the disclosure, the original features of the wind control events are analyzed from the historical wind control data to be processed, and then the binding relationship between the wind control events and the original features is established, so that the wind control events are prevented from being manually selected and bound by RCP, the wind control events and the original features of the wind control events are automatically bound, and the efficiency of binding operation is improved. Furthermore, the binding relationships and the original features are stored to a feature database. After receiving a binding relationship reading request sent by a terminal (such as RCP), the original features and the binding relationship can be searched from the feature database, and the original features and the binding relationship are returned to the terminal, so that the terminal can display the original features having the binding relationship with the wind control event. The terminal is responsible for sending the binding relation and reading the request and showing the function of the original characteristic, and does not need to transmit data with the server in order to bind the wind control event and the original characteristic, so that the operating pressure of the terminal is reduced.
In an exemplary embodiment of the present disclosure, when the original feature is obtained by analyzing the wind control event of the wind control historical data, the original feature may be analyzed from the wind control event of the wind control historical data according to the field identifier corresponding to the field where the original feature is located. Wherein, the field identifications corresponding to the field where each original feature is located are different from each other. For example, a field corresponding to a field in which the account of the executor in the original feature is located is identified as "user", and the content of the field corresponding to the field identified as "user" in the wind control history data is used as the account of the executor in the original feature. And a field corresponding to a field where the terminal IP is located in the original characteristic is marked as 'IP', and the field content corresponding to the field marked as 'IP' in the wind control historical data is used as the original characteristic terminal IP. According to the embodiment of the method and the device, the original features can be stored in the wind control historical data in a field form, the field identification of the field is used as the keyword of the original features in the wind control historical data, the original features can be analyzed from the wind control historical data through the field identification, the original features are prevented from being manually selected from the wind control historical data, and the original feature identification efficiency is improved.
In an exemplary embodiment of the disclosure, the binding relationship reading request sent by the terminal to the server includes an event identifier of the wind control event. For example, the binding relationship read request is used to obtain the original feature bound to the wind control event S1 from the server, and the binding relationship read request includes the event identifier Sb1 of the wind control event S1. When the original features and the binding relations are searched from the feature database according to the binding relation reading request, the wind control events can be searched from the feature database according to the event identifications, and then the original features and the binding relations are searched from the feature database according to the wind control events. In the above example, the wind control event S1 is searched from the feature database according to the event identifier Sb1, and then the original feature T1 and the binding relationship B1 of the wind control event S1 are searched from the feature database according to the wind control event S1. Embodiments of the present disclosure store the original features and binding relationships of the wind events to a feature database, which may be MySQL (a relational database management system). The wind control events, the original features of the wind control events and the binding relationship between the wind control events and the original features may be stored in a feature database. And the characteristic database is different from the ClickHouse for storing the historical data of the wind control, namely, the wind control event, the original characteristics of the wind control event and the binding relationship between the wind control event and the original characteristics are stored in the independent characteristic database, and the characteristic database searches the binding relationship and the original characteristics in the characteristic database aiming at the binding relationship reading request of the terminal, so that the searching operation in the ClickHouse is avoided, and the reading pressure of the ClickHouse is reduced.
In an exemplary embodiment of the disclosure, after the server returns the original features and the binding relationship to the terminal, a feature exposure request sent by the terminal may be received, where the feature exposure request is used to obtain candidate features related to the original features of the wind control event. And the server searches auxiliary features related to the original features from the feature database according to the feature display request, extracts candidate features from the auxiliary features according to preset strategy information, and then returns the candidate features to the terminal. The feature presentation request may include a feature identifier of the original feature, for example, the feature presentation request includes a feature identifier Tb1 of the original feature T1. It should be noted that, in addition to the original features of the wind control events, the feature database may also store the auxiliary features of the wind control events, and the original features and the auxiliary features of the same wind control events have a corresponding relationship. In practical applications, the assist feature related to the original feature may be searched according to the feature identifier of the original feature and the corresponding relationship between the original feature and the assist feature, and the assist feature may include at least one of: derived features, aggregated features, and service features. After the auxiliary features are obtained through searching, further judgment needs to be made as to which auxiliary features can be used as candidate features and which auxiliary features cannot be used as candidate features according to preset strategy information for the wind control event. Candidate features aiming at the wind control event can be set in the preset strategy information, if the derived features are specified in the preset strategy information to be used as the candidate features in the auxiliary features, the derived features are extracted from the auxiliary features, and the derived features are used as the candidate features of the wind control event; and if the preset strategy information specifies that the service features are used as candidate features in the auxiliary features, extracting the service features from the auxiliary features, and using the service features as the candidate features of the wind control events. Embodiments of the present disclosure may return a candidate feature of a wind control event to the terminal, the candidate feature being a feature unbound to the wind control event. The terminal can display the candidate features, so that a user can select one or more candidate features from the candidate features, and then the selected candidate features are bound with the wind control event, thereby providing a display function of features which are not bound with the wind control event, and allowing the terminal user to bind the unbound features according to actual needs.
In an exemplary embodiment of the disclosure, after storing the original features in the feature database, the feature database may be further searched for historical original features having a binding relationship with the wind control event, and features different from the original features may be deleted from the historical original features. For example, after the raw feature T1 of the wind control event S1 is stored in the feature database, the historical raw feature LT bound by the wind control event S1 is searched in the feature database. If the original signature T1 contains the signatures T01 and T02, and the historical original signature LT contains the signatures T01, T02 and T03, the signature T03 is considered as a useless original signature, and the signature T03 can be deleted from the signature database. The embodiment of the disclosure can compare the original features newly stored in the feature database with the historical original features, delete useless original features and reduce the data occupation amount of the feature database.
In an exemplary embodiment of the present disclosure, the characteristic index of the sample characteristic may also be obtained according to a preset period, the characteristic index obtained in the current period is compared with the characteristic index obtained in the previous period to obtain a period comparison result, the period comparison result is stored in the characteristic database, and the characteristic index and the period comparison result are displayed according to a form of a chart. In practical applications, the preset period may be minutes, days, weeks, etc., and generally, the preset period may be set to 30 minutes or one day. The sample feature may be any one of the primary and auxiliary features described above. The characteristic indicator may comprise one of: null rate, number of unique (unique) values, top (top) distribution fraction, feature quantile value. The graph may contain at least one of: a single-axis line graph corresponding to the null rate or the number of unique values, a column packing graph corresponding to the top distribution proportion, and a multi-axis line graph corresponding to the feature quantile value. For example, the null rate k01 of the derived signature T3 is obtained in the first period, the null rate k02 of the derived signature T3 is obtained in the second period, and the null rate k01 is compared with the null rate k02 to obtain the period comparison result bk. As shown in fig. 2, fig. 2 shows a graphical representation of several characteristic indicators and the results of the period comparison.
The feature indicators in fig. 2 may include null rate, unique value number, top distribution ratio, and feature quantile value indicators. The graph type corresponding to the null rate is a line graph, the graph type corresponding to the unique value number is a line graph, the graph type corresponding to the top distribution fraction is a bar packing graph, and the graph type corresponding to the feature quantile value index is a multiaxial line graph (each quantile is represented by one line). The embodiment of the disclosure can acquire the characteristic indexes of the sample characteristics according to the preset period, further analyze the characteristic indexes of the preset periods, and display the analysis result in the form of the chart, thereby avoiding the manual sampling analysis in the RCP, solving the problem of low analysis efficiency caused by the manual analysis of the characteristic indexes of the sample characteristics in a large batch, and simultaneously displaying the characteristic indexes and the analysis result in the form of the chart more intuitively.
In an exemplary embodiment of the disclosure, a feature index of a target feature may also be obtained according to a preset period, alarm information is generated according to the feature index and a corresponding index threshold, and the alarm information is sent to a terminal. In practical application, alarm information is generated according to the characteristic index and the corresponding index threshold, and the alarm information can be executed according to the following two ways: comparing the characteristic index with a corresponding index threshold value, and generating alarm information when the characteristic index is greater than the corresponding index threshold value; and when the characteristic index is less than or equal to the corresponding index threshold value, not generating alarm information. For example, a top distribution ratio z4 of a target feature (such as a service feature T4) acquired in a first period is compared with a top distribution ratio threshold zy, and when the top distribution ratio z4 is greater than the top distribution ratio threshold zy, alarm information is generated. The alarm information is used to prompt the service feature T4 that the top distribution fraction z4 exceeds the distribution fraction threshold zy during the first period. Comparing the period comparison result with a corresponding index threshold value, and generating alarm information when the period comparison result is greater than the corresponding index threshold value; and when the period comparison result is less than or equal to the corresponding index threshold value, not generating alarm information. For example, a top distribution ratio z4 of a target feature (for example, service feature T4) acquired in a first period is compared with a top distribution ratio z5 of service feature T4 acquired in a second period to obtain a period comparison result z45, a period comparison result z45 is compared with a top distribution ratio variation threshold zby, and when the period comparison result z45 is greater than the top distribution ratio variation threshold zby, an alarm message is generated to indicate that the variation of the top distribution ratio of service feature T4 between the first period and the second period exceeds the top distribution ratio variation threshold. The embodiment of the disclosure can compare the characteristic indexes of the original characteristic and the auxiliary characteristic of the wind control event with the corresponding index threshold values, and generate alarm information for the characteristic indexes with abnormality, thereby avoiding the manual abnormality detection of the characteristic indexes in the RCP, and solving the problem of low abnormality detection efficiency caused by the manual abnormality detection of the characteristic indexes of mass sample characteristics. In addition, in the process of anomaly detection, not only can the characteristic indexes acquired in a single period be detected, but also the variation of the characteristic indexes in a plurality of periods can be detected, so that comprehensive anomaly detection of the characteristic indexes is realized, and the accuracy of anomaly detection of the characteristic indexes is improved.
Based on the above description about an embodiment of a data processing method, a wind control characteristic analysis scheme of a wind control system is described below. The wind control feature analysis scheme can be applied to Kafka (distributed log system), and the wind control feature analysis scheme can contain two parts: and the first part is used for carrying out polymerization treatment on the characteristic indexes of the wind control characteristics at regular time. And a second part, namely performing prompt processing on available features in the wind control features at regular time.
As shown in fig. 3, fig. 3 is a flowchart illustrating a process of aggregating the feature indicators of the wind control features at regular time.
Topic at Kafka (one Topic may be considered a class of tasks) specifies which class of feature indicators are to be calculated on a half-hour or daily basis. And pulling wind control historical data from the ClickHouse for a Consumer (a subscribing user, which is used for subscribing to Topic and acquiring the progress of tasks, wherein each task is executed by a corresponding Consumer) subscribed to Topic, acquiring characteristic indexes of wind control characteristics in the wind control historical data, and storing the characteristic indexes into a characteristic value trend table (main _ args _ true) of MySQL. If a plurality of characteristic indexes are stored in the characteristic value trend table, the aggregation processing of the plurality of characteristic indexes is realized. For example, if the number of the characteristic indicators calculated by pulling the wind control history data from the clickwouse every 30 minutes is 34469, the data amount of the characteristic value trend table per day is 48 × 34469 — 1654512. And after receiving the feature index probability request of the RCP, returning the feature indexes in the feature value trend table to the RCP.
And moreover, abnormality detection can be carried out on the characteristic indexes in the characteristic value trend table at regular time, if the abnormal characteristic indexes exist, alarm information is generated and sent to the RCP or the terminal of the responsible person.
As shown in FIG. 4, FIG. 4 illustrates a flow chart for timing the hinting of available ones of the wind-controlled features.
Topic at Kafka specifies which types of wind control events to obtain the available wind control features, on a half-hour or daily basis. The Consumer subscribed to the Topic pulls the wind control historical data from the ClickHouse, acquires the wind control features in the wind control historical data, and stores the wind control features in an available feature table (main _ event _ type _ available _ feature) of MySQL. For example, the wind control history data is pulled once per day from clickwouse, the wind control event included in the wind control history data pulled once is 928, and the average wind control characteristic of each wind control event is 52, so that the data amount of the available characteristic table per day is 52 × 928 to 48256. After receiving the available feature request of the RCP, the wind-controlled features in the available feature table are returned to the RCP.
FIG. 5 is a block diagram illustrating a data processing apparatus according to an example embodiment. The apparatus may be applied to a server, and may specifically include the following elements.
An acquisition unit 51 configured to perform acquisition of wind control history data to be processed;
the analyzing unit 52 is configured to analyze the wind control events of the wind control historical data to obtain original features;
a binding unit 53 configured to perform establishing a binding relationship between the wind control event and the original feature, and store the original feature and the binding relationship to a feature database;
a receiving unit 54 configured to execute a binding relationship reading request transmitted by the receiving terminal;
a searching unit 55 configured to perform a search for the original feature and the binding relationship from the feature database according to the binding relationship reading request, and return the original feature and the binding relationship to the terminal.
In an exemplary embodiment of the disclosure, the parsing unit 52 is configured to perform parsing out the original features from the wind control events of the wind control history data according to field identifications corresponding to fields where the original features are located.
In an exemplary embodiment of the present disclosure, the binding relationship reading request includes an event identifier of the wind control event;
the search unit 55 includes: an event searching unit configured to perform a search for the wind control event from the feature database according to the event identification; a feature and relationship search unit configured to perform a search for the original features and the binding relationships from the feature database according to the wind control events.
In an exemplary embodiment of the present disclosure, the receiving unit 54 is further configured to perform receiving, after the searching unit 55 returns the original feature and the binding relationship to the terminal, a feature showing request sent by the terminal;
the searching unit 55 is further configured to perform a search for an auxiliary feature related to the original feature from the feature database according to the feature exhibition request;
the device further comprises:
and the extraction unit is configured to extract candidate features from the auxiliary features according to preset strategy information and return the candidate features to the terminal.
In an exemplary embodiment of the present disclosure, the feature exhibition request includes a feature identifier of the original feature;
the searching unit 55 is configured to perform a search of the feature database for at least one of the following auxiliary features related to the original feature according to the feature identification: derived features, aggregated features, and service features.
In an exemplary embodiment of the present disclosure, the searching unit 55 is further configured to perform, after the binding unit 53 stores the original features in a feature database, searching the feature database for historical original features having a binding relationship with the wind control event;
the device further comprises: a deleting unit configured to perform deletion of a feature different from the original feature in the history original feature.
In an exemplary embodiment of the present disclosure, the obtaining unit 51 is further configured to perform obtaining a feature index of a sample feature according to a preset period;
the device further comprises: the comparison unit is configured to compare the characteristic index acquired in the current period with the characteristic index acquired in the previous period to obtain a period comparison result;
and the display unit is configured to store the period comparison result into the characteristic database and display the characteristic index and the period comparison result according to the form of a chart.
In an exemplary embodiment of the present disclosure, the obtaining unit 51 is further configured to perform obtaining a feature index of a target feature according to a preset period;
the device further comprises:
and the alarm unit is configured to generate alarm information according to the characteristic indexes and corresponding index thresholds and send the alarm information to the terminal.
In an exemplary embodiment of the present disclosure, the alarm unit includes:
an index comparison unit configured to perform comparison of the feature index with the corresponding index threshold;
an alarm generation unit configured to perform generation of the alarm information when the characteristic index is larger than the corresponding index threshold.
In an exemplary embodiment of the disclosure, the index comparing unit is further configured to perform comparing the period comparison result with the corresponding index threshold;
the alarm generation unit is further configured to execute generating the alarm information when the period comparison result is greater than the corresponding index threshold.
In an exemplary embodiment of the disclosure, the characteristic indicator includes at least one of: null rate, number of unique values, top distribution ratio and feature quantile value; the graph includes at least one of: a single-axis line graph corresponding to the null rate or the number of unique values, a column packing graph corresponding to the top distribution proportion, and a multi-axis line graph corresponding to the feature quantile value.
With regard to the apparatus in the above-described embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
FIG. 6 is a block diagram illustrating a data processing electronic device 600, according to an example embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 6, electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an interface to input/output (I/O) 612, a sensor component 614, and a communication component 616.
The processing component 602 generally controls overall operation of the electronic device 600, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or a portion of the steps of the data processing method described above. Further, the processing component 602 can include one or more modules that facilitate interaction between the processing component 602 and other components. For example, the processing component 602 can include a multimedia module to facilitate interaction between the multimedia component 608 and the processing component 602.
The memory 604 is configured to perform operations to store various types of data to support operations at the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phonebook data, messages, images, videos, and so forth. The memory 604 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Power supply component 606 provides power to the various components of electronic device 600. The power components 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 600.
The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 600 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 610 is configured to perform outputting and/or inputting audio signals. For example, the audio component 610 may include a Microphone (MIC) configured to perform receiving external audio signals when the electronic device 600 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 604 or transmitted via the communication component 616. In some embodiments, audio component 610 further includes a speaker for outputting audio signals.
The I/O interface 612 provides an interface between the processing component 602 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor component 614 includes one or more sensors for providing status assessment of various aspects of the electronic device 600. For example, the sensor component 614 may detect an open/closed state of the electronic device 600, the relative positioning of components, such as a display and keypad of the electronic device 600, the sensor component 614 may also detect a change in the position of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, orientation or acceleration/deceleration of the electronic device 600, and a change in the temperature of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 616 is configured to perform communications that facilitate wired or wireless manners between the electronic device 600 and other devices. The electronic device 600 may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 600 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described data processing methods.
In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as the memory 604 comprising instructions, executable by the processor 620 of the electronic device 600 to perform the data processing method described above is also provided. Alternatively, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by the processor 620 of the electronic device 600 to perform the above-described data processing method. Alternatively, the computer program may be stored in a computer readable storage medium of the electronic device 600, which may be a non-transitory computer readable storage medium, for example, ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
FIG. 7 is a block diagram illustrating an electronic device 700 for processing data according to an example embodiment. For example, the electronic device 700 may be provided as a server. Referring to fig. 7, electronic device 700 includes a processing component 722 that further includes one or more processors, and memory resources, represented by memory 732, for storing instructions, such as applications, that are executable by processing component 722. The application programs stored in memory 732 may include one or more modules that each correspond to a set of instructions. Further, the processing component 722 is configured to execute instructions to perform the data processing methods described above.
The electronic device 700 may also include a power component 726 that is configured to perform power management of the electronic device 700, a wired or wireless network interface 750 that is configured to perform connecting the electronic device 700 to a network, and an input output (I/O) interface 758. The electronic device 700 may operate based on an operating system stored in memory 732, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A method of data processing, the method comprising:
acquiring wind control historical data to be processed;
analyzing the wind control events of the wind control historical data to obtain original characteristics;
establishing a binding relationship between the wind control event and the original characteristic, and storing the original characteristic and the binding relationship to a characteristic database;
receiving a binding relationship reading request sent by a terminal;
and searching the original characteristics and the binding relationship from the characteristic database according to the binding relationship reading request, and returning the original characteristics and the binding relationship to the terminal.
2. The method of claim 1, wherein the parsing the wind control events of the wind control historical data into raw features comprises:
and analyzing the original features from the wind control events of the wind control historical data according to field identifications corresponding to fields where the original features are located.
3. The method of claim 1, wherein the binding relationship read request includes an event identifier of the wind control event;
the searching the original feature and the binding relationship from the feature database according to the binding relationship reading request comprises:
searching the wind control event from the characteristic database according to the event identification;
and searching the original characteristics and the binding relation from the characteristic database according to the wind control event.
4. The method of claim 1, wherein after said returning said original feature and said binding to said terminal, said method further comprises:
receiving a feature display request sent by the terminal;
searching an auxiliary feature related to the original feature from the feature database according to the feature display request;
extracting candidate features from the auxiliary features according to preset strategy information;
and returning the candidate characteristics to the terminal.
5. The method of claim 4, wherein the feature exposure request includes a feature identifier of the original feature;
the searching the auxiliary features related to the original features from the feature database according to the feature showing request comprises the following steps:
searching the feature database for at least one of the following auxiliary features related to the original feature according to the feature identification: derived features, aggregated features, and service features.
6. The method of claim 1, wherein after said storing said raw features to a feature database, said method further comprises:
searching historical original features having a binding relation with the wind control events from the feature database;
and deleting the features which are different from the original features in the historical original features.
7. A data processing apparatus, characterized in that the apparatus comprises:
the acquisition unit is configured to acquire wind control historical data to be processed;
the analysis unit is configured to analyze the wind control events of the wind control historical data to obtain original characteristics;
the binding unit is configured to establish a binding relationship between the wind control event and the original characteristic, and store the original characteristic and the binding relationship into a characteristic database;
a receiving unit configured to execute a binding relationship reading request transmitted by a receiving terminal;
a searching unit configured to perform searching for the original feature and the binding relationship from the feature database according to the binding relationship reading request, and return the original feature and the binding relationship to the terminal.
8. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 6.
9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the data processing method according to any one of claims 1 to 6 when executed by a processor.
CN202110745572.2A 2021-06-30 2021-06-30 Data processing method, device, equipment, storage medium and program product Pending CN113570201A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110745572.2A CN113570201A (en) 2021-06-30 2021-06-30 Data processing method, device, equipment, storage medium and program product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110745572.2A CN113570201A (en) 2021-06-30 2021-06-30 Data processing method, device, equipment, storage medium and program product

Publications (1)

Publication Number Publication Date
CN113570201A true CN113570201A (en) 2021-10-29

Family

ID=78163344

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110745572.2A Pending CN113570201A (en) 2021-06-30 2021-06-30 Data processing method, device, equipment, storage medium and program product

Country Status (1)

Country Link
CN (1) CN113570201A (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130144894A1 (en) * 2003-11-26 2013-06-06 Jason Michael DeStefano Method and Apparatus For Retrieving and Combining Summarized Log Data In a Distributed Log Data Processing System
CN104598631A (en) * 2015-02-05 2015-05-06 北京航空航天大学 Distributed data processing platform
CN108304308A (en) * 2018-02-07 2018-07-20 平安普惠企业管理有限公司 User behavior monitoring method, device, computer equipment and storage medium
CN110674022A (en) * 2019-09-27 2020-01-10 北京三快在线科技有限公司 Behavior data acquisition method and device and storage medium
CN110912943A (en) * 2019-12-30 2020-03-24 北京明朝万达科技股份有限公司 Cross-network traffic analysis system
CN111708740A (en) * 2020-06-16 2020-09-25 荆门汇易佳信息科技有限公司 Mass search query log calculation analysis system based on cloud platform
CN112559913A (en) * 2020-12-11 2021-03-26 车智互联(北京)科技有限公司 Data processing method and device, computing equipment and readable storage medium
CN113034000A (en) * 2021-03-24 2021-06-25 数贸科技(北京)有限公司 Wind control processing method and device, computing equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130144894A1 (en) * 2003-11-26 2013-06-06 Jason Michael DeStefano Method and Apparatus For Retrieving and Combining Summarized Log Data In a Distributed Log Data Processing System
CN104598631A (en) * 2015-02-05 2015-05-06 北京航空航天大学 Distributed data processing platform
CN108304308A (en) * 2018-02-07 2018-07-20 平安普惠企业管理有限公司 User behavior monitoring method, device, computer equipment and storage medium
CN110674022A (en) * 2019-09-27 2020-01-10 北京三快在线科技有限公司 Behavior data acquisition method and device and storage medium
CN110912943A (en) * 2019-12-30 2020-03-24 北京明朝万达科技股份有限公司 Cross-network traffic analysis system
CN111708740A (en) * 2020-06-16 2020-09-25 荆门汇易佳信息科技有限公司 Mass search query log calculation analysis system based on cloud platform
CN112559913A (en) * 2020-12-11 2021-03-26 车智互联(北京)科技有限公司 Data processing method and device, computing equipment and readable storage medium
CN113034000A (en) * 2021-03-24 2021-06-25 数贸科技(北京)有限公司 Wind control processing method and device, computing equipment and storage medium

Similar Documents

Publication Publication Date Title
CN111431727B (en) Group display method, device, terminal, server and system
CN104301528A (en) Information displaying method and device
CN111405302A (en) Live broadcast room attention reminding method and device, electronic equipment and storage medium
CN113259226B (en) Information synchronization method and device, electronic equipment and storage medium
CN105183631A (en) Equipment test methods and devices
CN111428150A (en) Information display method and device, electronic equipment, server and storage medium
CN107316207B (en) Method and device for acquiring display effect information
CN109274825B (en) Message reminding method and device
CN111897474A (en) File processing method and electronic equipment
CN112948704A (en) Model training method and device for information recommendation, electronic equipment and medium
CN112134987B (en) Information processing method and device and electronic equipment
CN110213062B (en) Method and device for processing message
CN111651627A (en) Data processing method and device, electronic equipment and storage medium
CN113570201A (en) Data processing method, device, equipment, storage medium and program product
CN114666643B (en) Information display method and device, electronic equipment and storage medium
CN112291581B (en) Server, terminal equipment, information processing method and device
CN115794853A (en) Method and device for updating government affair data resource catalog, electronic equipment and medium
CN114189719A (en) Video information extraction method and device, electronic equipment and storage medium
CN114239531A (en) Template recommendation method and device, electronic equipment and storage medium
CN114896165A (en) Testing method and device of conversation robot system, electronic equipment and storage medium
CN113709572A (en) Content promotion method and device, electronic equipment and storage medium
CN113760946A (en) Pre-verification processing method, device, equipment and medium applied to data source migration
CN114900709B (en) Object information display method and device, electronic equipment and storage medium
CN115484471B (en) Method and device for recommending anchor
CN113239701B (en) Data analysis processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination