CN118132543A - Data processing method, device, terminal equipment and medium - Google Patents
Data processing method, device, terminal equipment and medium Download PDFInfo
- Publication number
- CN118132543A CN118132543A CN202211527302.5A CN202211527302A CN118132543A CN 118132543 A CN118132543 A CN 118132543A CN 202211527302 A CN202211527302 A CN 202211527302A CN 118132543 A CN118132543 A CN 118132543A
- Authority
- CN
- China
- Prior art keywords
- preset
- information
- data
- query
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 36
- 230000002159 abnormal effect Effects 0.000 claims abstract description 44
- 238000012545 processing Methods 0.000 claims description 33
- 238000012549 training Methods 0.000 claims description 23
- 238000000034 method Methods 0.000 claims description 21
- 230000002787 reinforcement Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure relates to a data processing method, a device, a terminal device and a medium, wherein the data processing method comprises the following steps: determining preset data information; inquiring the preset data information based on a preset inquiry rule to obtain inquiry result information; and inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not. By adopting the technical scheme, the preset data information is queried based on the preset query rule, the query result information is input into the pre-trained data judgment model, the judgment result representing whether the preset data information is abnormal or not is obtained, the efficiency and the rationality of the configuration of the query rule are improved, and the accuracy of the abnormal judgment of the data information is improved.
Description
Technical Field
The disclosure relates to the field of data quality monitoring, and in particular relates to a data processing method, a data processing device, terminal equipment and a medium.
Background
With the rapid development of electronic technology, computers are widely used in various industries, and various application programs and business systems can generate a large amount of data in the use process. Data quality is that data meets the expected purpose, when the data accurately shows real world reality, the data can be regarded as high quality, the high quality data is beneficial to the profit of enterprises, and the lack of the high quality data can seriously obstruct the development of the enterprises. Therefore, in the process of analyzing and managing data, it is necessary to effectively detect and check abnormal data so as to ensure the quality of the data in the data warehouse.
However, in the current data analysis and management process, the detection and the investigation of abnormal data have the problems of low efficiency and poor accuracy.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides a data processing method, apparatus, terminal device, and medium.
According to a first aspect of embodiments of the present disclosure, there is provided a data processing method, including:
Determining preset data information;
inquiring the preset data information based on a preset inquiry rule to obtain inquiry result information;
And inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not.
In some exemplary embodiments of the present disclosure, the preset data information corresponds to a plurality of preset query rules, and the querying the preset data information based on the preset query rules to obtain query result information includes:
determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition;
when the trigger condition is met, determining each preset query rule to be executed based on the preset configuration information and the trigger condition;
And respectively inquiring the preset data information based on the determined preset inquiry rules to obtain the inquiry result information.
In some exemplary embodiments of the present disclosure, the preset trigger condition includes at least one of the following conditions:
Reaching a preset execution time;
reaching a preset execution period;
The preset data information is changed.
In some exemplary embodiments of the present disclosure, the determining, based on the preset configuration information and the trigger condition, each preset query rule to be executed includes:
Determining time information in the trigger condition;
and inquiring each preset inquiry rule, and determining the preset inquiry rule matched with the time information as each preset inquiry rule to be executed.
In some exemplary embodiments of the disclosure, the inputting the query result information into a pre-trained data judgment model to obtain a judgment result includes:
the data judging model performs feature extraction on the query result information, and the extracted feature information comprises time feature information, historical data information and/or historical data fluctuation information;
and determining the judging result based on the characteristic information.
In some exemplary embodiments of the present disclosure, the data processing method further includes:
processing the query result information to obtain a training sample;
and performing reinforcement training on the data judgment model based on the training sample.
In some exemplary embodiments of the present disclosure, the data processing method further includes:
and outputting early warning information when the judgment result is that the preset data information is abnormal.
According to a second aspect of the embodiments of the present disclosure, there is provided a data processing apparatus, characterized in that the data processing apparatus includes:
the determining module is used for determining preset data information;
The query module is used for querying the preset data information based on a preset query rule to obtain query result information;
the processing module is used for inputting the query result information into a pre-trained data judgment model to obtain a judgment result, and the judgment result is used for representing whether the preset data information is abnormal or not.
In some exemplary embodiments of the present disclosure, the preset data information corresponds to a plurality of preset query rules, and the query module is further configured to:
determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition;
when the trigger condition is met, determining each preset query rule to be executed based on the preset configuration information and the trigger condition;
And respectively inquiring the preset data information based on the determined preset inquiry rules to obtain the inquiry result information.
In some exemplary embodiments of the present disclosure, the preset trigger condition includes at least one of the following conditions:
Reaching a preset execution time;
reaching a preset execution period;
The preset data information is changed.
According to a third aspect of embodiments of the present disclosure, there is provided a terminal device comprising:
a display screen;
a processor;
A memory for storing processor-executable instructions;
Wherein the processor is configured to execute executable instructions in the memory to implement the data processing method of the display screen provided in the first aspect of the present disclosure.
According to a fourth aspect of embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, implement the data processing method provided by the first aspect of the present disclosure.
The method has the following beneficial effects: and inquiring the preset data information based on the preset inquiry rule, inputting the inquiry result information into a pre-trained data judgment model to obtain a judgment result representing whether the preset data information is abnormal, thereby improving the efficiency and rationality of the configuration of the inquiry rule and improving the accuracy of the abnormality judgment of the data information.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow chart illustrating a data processing method according to an exemplary embodiment.
FIG. 2 is a flow chart illustrating a data processing method according to an exemplary embodiment.
FIG. 3 is a flowchart illustrating a data processing method according to an exemplary embodiment.
FIG. 4 is a flowchart illustrating a data processing method according to an exemplary embodiment.
FIG. 5 is a flowchart illustrating a data processing method according to an exemplary embodiment.
Fig. 6 is a block diagram of a data processing apparatus according to an exemplary embodiment.
Fig. 7 is a block diagram of a data processing apparatus according to an exemplary embodiment.
Fig. 8 is a block diagram of a terminal device according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the accompanying claims.
Before explaining the data processing method provided in the present disclosure, description is first made of data warehouse information related in the present disclosure.
With the rapid development of electronic technology, computers are widely used in various industries, and various application programs and business systems can generate a large amount of data in the use process. Data quality is that data meets the expected purpose, when the data accurately shows real world reality, the data can be regarded as high quality, the high quality data is beneficial to the profit of enterprises, and the lack of the high quality data can seriously obstruct the development of the enterprises. Therefore, in the process of analyzing and managing data, it is necessary to effectively detect and check abnormal data so as to ensure the quality of the data in the data warehouse.
Structured query language SQL (Structured Query Language) is a computer language used to store, retrieve, and modify data stored in a relational data store.
PB (petabyte) is a unit of data storage capacity, which is equal to 50 bytes to 2, or approximately 1000 TB in value, is a higher level unit of storage.
APACHE HIVE is a distributed fault tolerant data warehouse system that enables large scale analysis with stored information that allows users to analyze such information to make data driven decisions, e.g., users can read, write and manage PB level data using SQL through the system. Wherein the Hive table is a data storage table format implemented based on APACHE HIVE data stores.
Apache Iceberg is an open table format developed by Netflix for analyzing large datasets. Iceberg tables are tables that use a high performance format that works in a manner similar to SQL tables added to prest and Spark.
However, in the current data analysis and management process, when abnormal data is detected and examined, the abnormal data is examined according to a fixed judgment logic, so that on one hand, the application limitation is high, and the accuracy is poor, and on the other hand, a complex operation process is needed, and the examination efficiency is low.
In order to solve the above problems, the present disclosure provides a data processing method, which queries preset data information based on a preset query rule, inputs query result information into a pre-trained data judgment model to obtain a judgment result representing whether the preset data information is abnormal, improves the efficiency and rationality of query rule configuration, and improves the accuracy of data information abnormality judgment through intelligent early warning.
An exemplary embodiment of the present disclosure provides a data processing method, as shown in fig. 1, including:
S101, determining preset data information;
s102, inquiring preset data information based on a preset inquiry rule to obtain inquiry result information;
s103, inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not.
In step S101, the data warehouse has a plurality of data tables, including offline data tables, such as Hive tables, iceberg tables, etc., and the user selects the data table to be monitored from the data warehouse, and extracts the information of table names, fields, records, etc. in the selected data table as the preset data information.
In step S102, the data information may be analyzed from six dimensions of integrity, validity, timeliness, consistency, accuracy, uniqueness, and single column, cross-row, and cross-table. In order to meet the query analysis requirements of each dimension, different query rules are set, such as a data integrity check rule, a data timeliness check rule, a field uniqueness check rule, a record number consistency check rule and the like, wherein the data integrity check rule is used for checking whether data in a data table has missing or empty data, the data timeliness check rule is used for checking the time interval from generation to checking of the data, and other query rule contents are not repeated here. In addition, the user can set the query rule according to the data query requirement. And according to the query requirement, utilizing keyword matching, selecting a corresponding query rule for preset data information in the data quality monitoring system as preset query information, and automatically generating a standard SQL query statement based on the preset data information and the preset query rule. And querying preset data information by utilizing the SQL query statement, analyzing the preset data information into two-dimensional result data, and extracting data information related to a preset query rule from a plurality of two-dimensional result data to obtain query result information.
In step S103, the data determination model is used to determine whether the data information is abnormal. For example, if it is to be determined from the dimension of the integrity whether the data is abnormal, the data determination model obtains the query result information that is queried based on the data integrity check rule, and if it is determined that there is missing or null data (null) in the query result information by analysis, it is determined that the data information is abnormal; if the time interval from the generation to the checking of the data in the query result information is larger than a preset value, judging that the data information is abnormal. The data judgment model can be obtained by training a preset initial model, and by way of example, sample data can be input into the preset initial model, and training is performed by taking a label corresponding to the sample data as expected output to obtain the data judgment model. In addition, the data judgment model can also utilize the query result information and the judgment result to carry out learning training, optimize the judgment standard of various query rules and enable the judgment result to be more close to the application requirement of the data information.
In some embodiments, the judgment result may be an identifier, for example, the data judgment model judges the data integrity of the query result information, if the analysis finds that the query result information has a missing or null data (null), the judgment data information is abnormal, and an identifier is output to represent the abnormality; otherwise, if no missing or null data (null) exists in the query result information, judging that the data information is normal, and outputting an identification representation to be normal.
In other embodiments, the determination result may be a percentage of abnormal data, for example, the data determination model determines the data timeliness of the query result information, where the query result information includes 100 data, and if the analysis finds that the time interval from the generation to the viewing of 30 data in the query result information is greater than a preset value, the 30 data information is determined to be abnormal, and the output determination result is 30% of the abnormal data.
In the method, the preset data information is queried based on the preset query rule, query result information is input into the pre-trained data judgment model, and the judgment result representing whether the preset data information is abnormal is obtained, so that the efficiency and the rationality of query rule configuration are improved, and the accuracy of data information abnormality judgment is improved.
According to an exemplary embodiment, as shown in fig. 2, the data processing method in this embodiment includes:
s201, determining preset data information;
S202, determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition;
S203, when the trigger condition is met, determining each preset query rule to be executed based on preset configuration information and the trigger condition;
S204, respectively inquiring the preset data information based on the determined preset inquiry rules to obtain inquiry result information;
s205, inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not.
Steps S201, S204, and S205 are the same as the implementation in the above embodiment, and are not described here again.
In step S202, the preset configuration information includes a correspondence between preset query rules and preset trigger conditions, where each preset query rule may be set with a corresponding preset trigger condition, and the preset trigger condition is determined according to query content and application scenario of the preset query rule, in addition to direct use selected by the user. Wherein the preset trigger condition includes at least one of the following conditions:
reaching a preset execution time; reaching a preset execution period; the preset data information is changed.
The query rule may set the trigger condition singly or may set a plurality of trigger conditions simultaneously. For example, only nine am every day is set as the execution time of the query rule, and the data quality monitoring system triggers the preset query rule at nine am every day to process the preset data information. For another example, the triggering condition may be set to reach a preset execution period and change preset data information, where the execution period is set to be three hours, the preset query rule is triggered again every time three hours elapses, the preset data information is queried again, and when the preset data information in the data table is changed, the preset query rule is triggered, and the data information changing action is used to notify the data quality monitoring system to execute the preset query rule, so as to implement the real-time query effect on the preset data information.
When the preset data information corresponds to a plurality of preset query rules, each preset query rule can adopt the same trigger condition, or can adopt different trigger conditions, for example, one part of trigger conditions corresponding to the preset query rules reach preset execution time, and the other part of trigger conditions corresponding to the preset query rules are changed for the preset data information.
In step S203, the preset data information may correspond to a plurality of preset query rules, where different preset query rules may have the same or different trigger conditions, and the corresponding preset query rules may be executed only when the trigger conditions meet the trigger conditions of the preset query rules. For example, the triggering condition of the preset query rules a and B is that the application scenario of the preset data information is manager information management, the triggering condition of the preset query rules C and D is that the preset data information includes data of a income field, and if the current preset data information is manager information and does not include data of the income field, the triggering condition of the preset query rules a and B is only satisfied, and the preset query rules a and B are executed; if the current preset data information is the client information and comprises data of income fields, the triggering conditions of the preset query rules C and D are only met, and the preset query rules C and D are executed.
In some embodiments, in step S203, when the trigger condition is satisfied, determining, based on the preset configuration information and the trigger condition, each preset query rule to be executed includes:
s1, determining time information in a trigger condition;
S2, inquiring each preset inquiry rule, and determining the preset inquiry rule matched with the time information as each preset inquiry rule to be executed.
Because the data table in the data warehouse generally generates data according to the time partition, and the preset trigger condition of the preset query rule also includes time information, in step S1, the time information in the trigger condition needs to be determined to find the preset query rule matched with the trigger condition. In step S2, each preset query rule is used for querying different dimensions of the data information, different preset trigger conditions are used, and different time information is included, and the corresponding preset query rule can be executed to query the preset data information only when the time information included in the preset query rule is consistent with the time information in the trigger conditions and the preset trigger conditions are met. For example, the preset trigger condition of the preset query rule a is nine points in the morning, and the time information is 20220713; the preset triggering condition of the preset query rule B is nine points in the morning, and the time information is 20220720; the preset triggering condition of the preset query rule C is a period of three hours, and the time information is 20220720; the preset trigger condition of the preset query rule D is that the preset data information is changed, and the time information is 20220713, if the time information in the trigger condition is date= 20220713, the preset query rule a is executed at nine points in the morning, and if the preset data information is changed, the preset query rule D is executed, and since the time information of the preset query rules B and C is different from the time information in the trigger condition, the preset query rules B and C are not executed even if the preset trigger condition can be reached.
In the method, each preset query rule required to be executed for processing the preset data information is determined by utilizing the preset trigger condition and the time information to obtain the query result information, so that the complexity of configuring the preset query rule can be greatly reduced, and the efficiency and the rationality of configuring the query rule are improved.
According to an exemplary embodiment, as shown in fig. 3, the data processing method in this embodiment includes:
S301, determining preset data information;
s302, inquiring preset data information based on preset inquiry rules to obtain inquiry result information;
S303, extracting features of query result information by the data judging model, wherein the extracted feature information comprises time feature information, historical data information and/or historical data fluctuation information;
s304, determining a judging result based on the characteristic information.
Steps S301 and S302 are the same as the implementation in the above embodiment, and are not described here again.
In step S303, the obtained query result information includes various feature information, including time feature information, history data information, and history data fluctuation information. The time feature information refers to a time type to which the time information included in the query result information is classified, for example, the time information in the query result information is 20220501, the time feature information is a national legal festival holiday, the time information in the query result information is 20220618, the time feature information is a special activity time, the time information in the query result information is 20220720, and the time feature information is a common time. The historical data information refers to that in a fixed time period, preset data information is queried by using a current preset query rule, all obtained query result information can be set to 30 days, 15 days or 5 days in the fixed time period, and specific numerical values can be set according to the content of the preset data information and the application requirement of the preset query rule. The history data fluctuation information refers to that, with a certain query result information as reference information, all query result information different from the reference information in a fixed period of time is taken as history data fluctuation information.
In step S304, the data judgment model can obtain the fluctuation characteristics of the preset data information in different time types within the fixed time period from the historical data fluctuation information, and map the fluctuation characteristics to different data judgment models, such as a statistical model, a neural network model, and the like, so that different judgment standards exist when judging whether each preset data information is abnormal, the fixed judgment standards are not relied on any more, the judgment standards based on the characteristic information are closer to the application scene of the preset data information, and the accuracy of judging the abnormality of the preset data information can be improved.
According to an exemplary embodiment, as shown in fig. 4, the data processing method in this embodiment includes:
S401, determining preset data information;
S402, inquiring preset data information based on preset inquiry rules to obtain inquiry result information;
s403, inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not;
s404, processing the query result information to obtain a training sample;
s405, performing reinforcement training on the data judgment model based on the training sample.
Steps S401, S402, and S403 are the same as the implementation in the above embodiment, and are not described here again.
In step S404, if the obtained determination result indicates that the preset data information has an abnormality, it indicates that the preset data information has abnormal data. In order to make the data judgment model have a better training effect, abnormal data in the query result information needs to be cleared, and a training sample is obtained. In step S405, the training sample without abnormal data is used as an input of the data judgment model, so that the content and standard of the normal data can be intuitively known, the data judgment model is enabled to strengthen training, and the standard of the data judgment model for judging that the query result information is abnormal is continuously corrected and optimized.
In the method, the training sample is obtained by processing the query result information, and the training sample is used for carrying out reinforcement training on the data judgment model, so that the utilization rate of the query result information can be improved, the data judgment model is continuously optimized, and the rationality and accuracy of the judgment of the data judgment model are improved.
According to an exemplary embodiment, as shown in fig. 5, the data processing method in this embodiment includes:
S501, determining preset data information;
s502, inquiring preset data information based on a preset inquiry rule to obtain inquiry result information;
s503, inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not;
s504, outputting early warning information when the judgment result is that the preset data information is abnormal.
Steps S501, S502, and S503 are the same as the implementation in the above embodiment, and are not described here again.
In step S504, if the determined result is that the preset data information is abnormal, it indicates that abnormal data exists in the preset data information, and the abnormal data may reduce the data quality and directly affect the result of data analysis. Therefore, when the judgment result is that the preset data information is abnormal, the early warning information is output to remind the user. The early warning information may include the source, content, time, location, etc. of the abnormal data. The early warning information can remind the user in a telephone mode, the user can be notified in a message card mode, and the specific output mode can be set according to the use habit of the user.
In the method, after judging that the preset data information is abnormal, the device outputs the early warning information, can remind a user to timely process the data information, effectively avoid the diffusion of the abnormal information, improve the data quality,
Exemplary embodiments of the present disclosure provide a data processing apparatus. As shown in fig. 6, a block diagram of a data processing apparatus is shown in the present disclosure.
The block diagram comprises: a determining module 61, a querying module 62, and a processing module 63. A determining module 61, configured to determine preset data information; the query module 62 is configured to query the preset data information based on a preset query rule to obtain query result information; the processing module 63 is configured to input the query result information into a pre-trained data judgment model, and obtain a judgment result, where the judgment result is used to characterize whether the preset data information is abnormal.
In an exemplary embodiment of the present disclosure, the preset data information corresponds to a plurality of preset query rules, and the query module 62 is further configured to: determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition; when the trigger condition is met, determining each preset query rule to be executed based on the preset configuration information and the trigger condition; and respectively inquiring the preset data information based on the determined preset inquiry rules to obtain inquiry result information.
In an exemplary embodiment of the present disclosure, the preset trigger condition includes at least one of the following conditions: reaching a preset execution time; reaching a preset execution period; the preset data information is changed.
In an exemplary embodiment of the present disclosure, the query module 62 is further configured to:
determining time information in the trigger condition; and inquiring each preset inquiry rule, and determining the preset inquiry rule matched with the time information as each preset inquiry rule to be executed.
In an exemplary embodiment of the present disclosure, the processing module 63 is further configured to:
The data judging model performs feature extraction on the query result information, and the extracted feature information comprises time feature information, historical data information and/or historical data fluctuation information; and determining a judging result based on the characteristic information.
According to an exemplary embodiment, as shown in fig. 7, the data processing apparatus further includes:
The training module 71 is configured to process the query result information to obtain a training sample; and performing reinforcement training on the data judgment model based on the training sample.
In an exemplary embodiment of the present disclosure, the data processing apparatus further includes:
and an output module 72, configured to output early warning information when the judgment result is that the preset data information is abnormal.
The specific manner in which the respective modules perform the operations in relation to the terminal apparatus in the above-described embodiments has been described in detail in relation to the embodiments of the method, and will not be described in detail here.
Fig. 8 is a block diagram illustrating a terminal device 800 according to an exemplary embodiment. For example, the terminal device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, or the like.
Referring to fig. 8, a terminal device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.
The processing component 802 generally controls overall operation of the terminal device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interactions between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the terminal device 800. Examples of such data include instructions for any application or method operating on terminal device 800, contact data, phonebook data, messages, pictures, video, and the like. The memory 804 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 806 provides power to the various components of the terminal device 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal device 800.
The multimedia component 808 includes a screen between the terminal device 800 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. The front camera and/or the rear camera may receive external multimedia data when the terminal device 800 is in an operation mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the terminal device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 further includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of the terminal device 800. For example, the sensor assembly 814 may detect an on/off state of the terminal device 800, a relative positioning of the assemblies, such as a display and keypad of the terminal device 800, the sensor assembly 814 may also detect a change in position of the terminal device 800 or a component of the terminal device 800, the presence or absence of a user's contact with the terminal device 800, an orientation or acceleration/deceleration of the terminal device 800, and a change in temperature of the terminal device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate communication between the terminal device 800 and other devices, either wired or wireless. The terminal device 800 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the terminal device 800 can be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 804, including instructions executable by processor 820 of terminal device 800 to perform the data processing methods described above. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
A non-transitory computer readable storage medium, which when executed by a processor of a terminal device, causes a processing apparatus of the terminal device to perform the data processing method provided by the exemplary embodiments of the present disclosure.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (12)
1. A data processing method, characterized in that the data processing method comprises:
Determining preset data information;
inquiring the preset data information based on a preset inquiry rule to obtain inquiry result information;
And inputting the query result information into a pre-trained data judgment model to obtain a judgment result, wherein the judgment result is used for representing whether the preset data information is abnormal or not.
2. The data processing method according to claim 1, wherein the preset data information corresponds to a plurality of preset query rules, the querying the preset data information based on the preset query rules to obtain query result information includes:
determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition;
when the trigger condition is met, determining each preset query rule to be executed based on the preset configuration information and the trigger condition;
And respectively inquiring the preset data information based on the determined preset inquiry rules to obtain the inquiry result information.
3. The data processing method according to claim 2, wherein the preset trigger condition includes at least one of the following conditions:
Reaching a preset execution time;
reaching a preset execution period;
The preset data information is changed.
4. The data processing method according to claim 2, wherein the determining each preset query rule to be executed based on the preset configuration information and the trigger condition includes:
Determining time information in the trigger condition;
and inquiring each preset inquiry rule, and determining the preset inquiry rule matched with the time information as each preset inquiry rule to be executed.
5. The method according to any one of claims 1 to 4, wherein inputting the query result information into a pre-trained data judgment model to obtain a judgment result includes:
the data judging model performs feature extraction on the query result information, and the extracted feature information comprises time feature information, historical data information and/or historical data fluctuation information;
and determining the judging result based on the characteristic information.
6. The data processing method according to any one of claims 1 to 4, characterized in that the data processing method further comprises:
processing the query result information to obtain a training sample;
and performing reinforcement training on the data judgment model based on the training sample.
7. The data processing method according to any one of claims 1 to 4, characterized in that the data processing method further comprises:
and outputting early warning information when the judgment result is that the preset data information is abnormal.
8. A data processing apparatus, characterized in that the data processing apparatus comprises:
the determining module is used for determining preset data information;
The query module is used for querying the preset data information based on a preset query rule to obtain query result information;
the processing module is used for inputting the query result information into a pre-trained data judgment model to obtain a judgment result, and the judgment result is used for representing whether the preset data information is abnormal or not.
9. The data processing apparatus according to claim 8, wherein the preset data information corresponds to a plurality of the preset query rules, and the query module is further configured to:
determining preset configuration information, wherein the preset configuration information is used for representing the corresponding relation between a preset query rule and a preset trigger condition;
when the trigger condition is met, determining each preset query rule to be executed based on the preset configuration information and the trigger condition;
And respectively inquiring the preset data information based on the determined preset inquiry rules to obtain the inquiry result information.
10. The data processing apparatus according to claim 9, wherein the preset trigger condition includes at least one of:
Reaching a preset execution time;
reaching a preset execution period;
The preset data information is changed.
11. A terminal device, characterized in that the terminal device comprises:
a display screen;
a processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to execute executable instructions in the memory to implement the data processing method of any of claims 1 to 7.
12. A non-transitory computer readable storage medium having stored thereon executable instructions which when executed by a processor implement the data processing method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211527302.5A CN118132543A (en) | 2022-12-01 | 2022-12-01 | Data processing method, device, terminal equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211527302.5A CN118132543A (en) | 2022-12-01 | 2022-12-01 | Data processing method, device, terminal equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118132543A true CN118132543A (en) | 2024-06-04 |
Family
ID=91229014
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211527302.5A Pending CN118132543A (en) | 2022-12-01 | 2022-12-01 | Data processing method, device, terminal equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118132543A (en) |
-
2022
- 2022-12-01 CN CN202211527302.5A patent/CN118132543A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10061762B2 (en) | Method and device for identifying information, and computer-readable storage medium | |
| CN111539443A (en) | Image recognition model training method and device and storage medium | |
| CN111813932B (en) | Text data processing method, text data classifying device and readable storage medium | |
| CN115118695A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN110738267B (en) | Image classification method, device, electronic equipment and storage medium | |
| CN112486770B (en) | Client dotting reporting method and device, electronic equipment and storage medium | |
| CN107977127B (en) | Method, device and terminal for updating page | |
| CN113779257A (en) | Method, device, equipment, medium and product for analyzing text classification model | |
| CN111797746B (en) | Face recognition method, device and computer readable storage medium | |
| CN110795314B (en) | Method and device for detecting slow node and computer readable storage medium | |
| CN111125388B (en) | Method, device and equipment for detecting multimedia resources and storage medium | |
| CN112269730A (en) | Abnormal log detection method, abnormal log detection device, and storage medium | |
| CN113590605B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN118132543A (en) | Data processing method, device, terminal equipment and medium | |
| CN112988822B (en) | Data query method, device, equipment, readable storage medium and product | |
| CN112333233B (en) | Event information reporting method and device, electronic equipment and storage medium | |
| CN116136855A (en) | Account grouping determination method and device, electronic equipment and storage medium | |
| CN114710401B (en) | Abnormality positioning method and device | |
| CN113206772B (en) | Method, device, equipment, medium and product for judging correctness of response message | |
| CN116303101B (en) | Test case generation method, device and equipment | |
| CN115225702B (en) | Information pushing method and device, electronic equipment and storage medium | |
| CN114238728B (en) | Vehicle data processing method, device and equipment | |
| CN110209775B (en) | Text processing method and device | |
| CN114528318A (en) | Data processing method and device, electronic equipment, storage medium and product | |
| CN116781974A (en) | Data processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |