CN114048104A - Monitoring method, device, equipment and storage medium - Google Patents
Monitoring method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114048104A CN114048104A CN202111405118.9A CN202111405118A CN114048104A CN 114048104 A CN114048104 A CN 114048104A CN 202111405118 A CN202111405118 A CN 202111405118A CN 114048104 A CN114048104 A CN 114048104A
- Authority
- CN
- China
- Prior art keywords
- model
- user
- monitoring
- sample set
- nl2sql
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012549 training Methods 0.000 claims abstract description 36
- 238000012806 monitoring device Methods 0.000 claims abstract description 7
- 230000006399 behavior Effects 0.000 claims description 53
- 230000006870 function Effects 0.000 claims description 36
- 238000011156 evaluation Methods 0.000 claims description 32
- 238000003062 neural network model Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 11
- 230000003287 optical effect Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000014759 maintenance of location Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000010992 reflux Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
- G06F11/3423—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time where the assessed time is active or idle time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3447—Performance evaluation by modeling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/865—Monitoring of software
Abstract
The invention discloses a monitoring method, a monitoring device, monitoring equipment and a storage medium. The method comprises the following steps: acquiring historical behavior information of a user; the method comprises the steps of inputting historical user behavior information into a monitoring model to obtain a monitoring result of a first NL2SQL model, wherein the monitoring model is obtained through a first target sample set iterative training deep recommendation model.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a monitoring method, a monitoring device, monitoring equipment and a storage medium.
Background
The application field of artificial intelligence is wider and wider, NL2SQL converts natural language into SQL statements which can serve as an intelligent interface of a database, and users who are unfamiliar with the database can quickly find data wanted by the users. Thus, the NL2SQL model is increasingly applied in structured data queries. With the development of technology over time, how, when and how often the NL2SQL model is updated is a common problem in practical applications.
The artificial intelligence model deployed on-line needs to be updated and iterated as the external environment is transformed over time. The current artificial intelligence model updating method mainly comprises the following steps: online updates and manual updates.
And updating on line, pushing a model with a version number larger than the existing version number on line, and replacing the new model by the on-line program unconditionally. The method has the main defects that scenes which cannot be accessed by a network are not considered, the model can only be updated mechanically, the model cannot be updated intelligently, and the intelligent decision that the model needs not to be updated, when to be updated and how often to be updated is not made.
The manual updating consumes a large amount of labor cost, the model updating efficiency is low, the real-time performance cannot be achieved, the model cannot be updated intelligently, and the intelligent decision that the model needs not to be updated, when to be updated and how often to be updated cannot be made intelligently.
Disclosure of Invention
Embodiments of the present invention provide a monitoring method, an apparatus, a device, and a storage medium, which can solve the problems that online updating does not take into account a scene that a network cannot access, a model can only be updated mechanically, and the model cannot be updated intelligently, as well as the problems that manual updating consumes a large amount of labor cost, model updating efficiency is low, real-time performance is not achieved, and the model cannot be updated intelligently, and can determine whether the model needs to be updated according to user historical behavior information.
In a first aspect, an embodiment of the present invention provides a monitoring method, including:
acquiring historical behavior information of a user;
inputting the historical user behavior information into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
Further, the user historical behavior information includes: the user evaluates the feedback information and the usage frequency information.
Further, before the inputting the historical user behavior information into the monitoring model to obtain the monitoring result of the first NL2SQL model, the method further includes:
and iteratively training the first neural network model through the second target sample set to obtain a first NL2SQL model.
Further, iteratively training the first neural network model through the second target sample set to obtain a first NL2SQL model, including:
establishing a first neural network model;
inputting a first historical query statement sample in the second target sample set into the first neural network model to obtain a predicted SQL statement;
training parameters of the first neural network model according to a first target function formed by the SQL statements corresponding to the prediction SQL statement sample and the first historical query statement sample;
and returning to execute the operation of inputting the historical query statement samples in the second target sample set into the first neural network model to obtain the predicted SQL statement until a first NL2SQL model is obtained.
Further, after the inputting the historical user behavior information into the monitoring model to obtain the monitoring result of the first NL2SQL model, the method further includes:
if the monitoring result is updated, generating a positive sample set and a negative sample set according to the first target sample set and the user evaluation feedback information;
and iteratively training the first NL2SQL model through the positive sample set and the negative sample set to obtain a target NL2SQL model.
Further, generating a positive sample set and a negative sample set according to the first target sample set and the user evaluation feedback information, including:
if the user evaluation feedback information is positive feedback, adding a historical query statement and an SQL statement corresponding to the user evaluation feedback information to a positive sample set;
and if the user evaluation feedback information is negative feedback, adding the historical query statement and the SQL statement corresponding to the user evaluation feedback information to a negative sample set.
Further, the usage frequency information includes: at least one of the number of user questions and answers, the question-answer duration, the user retention rate, the total number of first NL2SQL model calls in the preset time, the average number of user calls and the call frequency.
In a second aspect, an embodiment of the present invention further provides a monitoring apparatus, where the apparatus includes:
the acquisition module is used for acquiring historical behavior information of a user;
and the monitoring module is used for inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the monitoring method according to any one of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the monitoring method according to any one of the embodiments of the present invention.
The embodiment of the invention obtains the historical behavior information of the user; inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of a first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set, the problems that a scene which cannot be accessed by a network is not considered in online updating, the model can only be updated mechanically and the model cannot be updated intelligently can be solved, a large amount of labor cost is consumed in manual updating, the model updating efficiency is low, real-time performance cannot be achieved, and the model cannot be updated intelligently can be solved, and whether the model needs to be updated or not can be judged according to the historical behavior information of the user.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a flow chart of a monitoring method in an embodiment of the invention;
FIG. 2 is a schematic diagram of a monitoring device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer-readable storage medium containing a computer program in an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
The term "include" and variations thereof as used herein are intended to be open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment".
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Fig. 1 is a flowchart of a monitoring method according to an embodiment of the present invention, where this embodiment is applicable to a situation of monitoring a first NL2SQL model, and the method may be executed by a monitoring device according to an embodiment of the present invention, where the monitoring device may be implemented in a software and/or hardware manner, as shown in fig. 1, the method specifically includes the following steps:
and S110, acquiring historical behavior information of the user.
Wherein the user historical behavior information may include: the user evaluates the feedback information and the usage frequency information. The user evaluation feedback information may include: the user can directly evaluate the question and answer effect, such as 'good answer', 'wrong answer' and 'no answer', and the question and answer precision can be indirectly calculated according to the user evaluation feedback information; the usage frequency information mainly includes: the system comprises at least one of the number of user questions and answers, the question and answer duration, the new/old user use condition, the user retention rate, the total number of users, the user number increase speed, the total online time of the first NL2SQL model, the application time of the first NL2SQL model, the total number of times of calling of the first NL2SQL model, the average user calling number, the recent calling frequency (near three days, near one week and near one month), the negative evaluation number and the negative evaluation increase frequency.
Specifically, the manner of obtaining the user historical behavior information may be: and directly obtaining the direct evaluation of the user on the question answering effect. The method for acquiring the historical behavior information of the user can also be as follows: and counting the use frequency information of the user. For example, the number of user questions and answers within a preset time may be obtained, or the total number of first NL2SQL model calls within the preset time may be obtained.
And S120, inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
The main implementation flow of the first NL2SQL model is as follows: the method comprises the steps of performing syntactic analysis on a query statement input by a user by using word segmentation, part-of-speech tagging, entity identification, dependency syntax and the like, filling a semantic slot by using various methods such as rules, word vectors, language models, deep learning and the like, specifically comprising a query field, an aggregation function, a screening condition, a packet field and the like, and generating a corresponding SQL query statement based on the filled information.
Wherein the training process of the first NL2SQL model comprises: establishing a first neural network model; inputting a first historical query statement sample in the first target sample set into the first neural network model to obtain a predicted SQL statement; training parameters of the first neural network model according to a first target function formed by the SQL statements corresponding to the prediction SQL statement sample and the first historical query statement sample; and returning to execute the operation of inputting the historical query statement samples in the first target sample set into the first neural network model to obtain a prediction SQL statement until a first NL2SQL model is obtained.
The main implementation process of the monitoring model is as follows: and (3) taking the historical behavior information of the user as a model input feature, and constructing a monitoring model based on a deep recommendation model of deep FM. Simple linear models, lacking the ability to learn high dimensional features, are difficult to learn from training samples to important features that never or rarely occur. The deep FM model is an end-to-end model which can be extracted from original features to various complexity features, and comprises FM and DNN, wherein the FM model can be used for extracting low-dimensional features, and the DNN can be used for extracting high-dimensional features. The model does not need artificial characteristic engineering; the input is the original characteristic, the FM and DNN share the input vector characteristic, and the deep FM model training speed is high.
Specifically, the method for obtaining the monitoring model through the iterative training of the deep recommendation model by the first target sample set may be as follows: establishing a depth recommendation model; inputting the user historical behavior information samples in the first target sample set into a depth recommendation model to obtain a prediction monitoring result; training parameters of the deep recommendation model according to a second objective function formed by the predicted monitoring result and the monitoring result corresponding to the user historical behavior information sample; and returning to execute the operation of inputting the user historical behavior information samples in the first target sample set into a deep recommendation model to obtain a prediction monitoring result until a monitoring model is obtained.
Wherein, the monitoring result may be: at least one of update, no update, and how often.
Specifically, the historical user behavior information is input into the monitoring model to obtain the monitoring result of the first NL2SQL model, for example, the historical user behavior information is obtained, and the historical user behavior information is input into the monitoring model to obtain the monitoring result of the first NL2SQL model, which is updated immediately.
Optionally, the user historical behavior information includes: the user evaluates the feedback information and the usage frequency information.
Wherein the user evaluation feedback information may include: the user may directly evaluate the question-answering effect, for example, by any one of "good answer", "wrong answer", and "unanswered answer".
Wherein the usage frequency information may include: at least one of the number of user questions and answers, the question-answer duration, the user retention rate, the total number of first NL2SQL model calls in the preset time, the average number of user calls and the call frequency. For example, the usage frequency information may include: the system comprises at least one of the number of user questions and answers, the question and answer duration, the new/old user use condition, the user retention rate, the total number of users, the user number increase speed, the total online time of the first NL2SQL model, the application time of the first NL2SQL model, the total number of times of calling of the first NL2SQL model, the average user calling number, the recent calling frequency (near three days, near one week and near one month), the negative evaluation number and the negative evaluation increase frequency.
Optionally, before the inputting the user historical behavior information into the monitoring model to obtain the monitoring result of the first NL2SQL model, the method further includes:
and iteratively training the first neural network model through the second target sample set to obtain a first NL2SQL model.
Wherein the second set of target samples comprises: the historical query statement samples and SQL statements corresponding to the historical query statement samples.
Optionally, iteratively training the first neural network model through the second target sample set to obtain the first NL2SQL model, including:
establishing a first neural network model;
inputting a first historical query statement sample in the second target sample set into the first neural network model to obtain a predicted SQL statement;
training parameters of the first neural network model according to a first target function formed by the SQL statements corresponding to the prediction SQL statement sample and the first historical query statement sample;
and returning to execute the operation of inputting the historical query statement samples in the second target sample set into the first neural network model to obtain the predicted SQL statement until a first NL2SQL model is obtained.
Optionally, after the inputting the user historical behavior information into the monitoring model to obtain the monitoring result of the first NL2SQL model, the method further includes:
if the monitoring result is updated, generating a positive sample set and a negative sample set according to the first target sample set and the user evaluation feedback information;
and iteratively training the first NL2SQL model through the positive sample set and the negative sample set to obtain a target NL2SQL model.
The manner of generating the positive sample set and the negative sample set according to the first target sample set and the user evaluation feedback information may be: if the user evaluation feedback information is positive feedback, adding a historical query statement and an SQL statement corresponding to the user evaluation feedback information to a positive sample set; and if the user evaluation feedback information is negative feedback, adding the historical query statement and the SQL statement corresponding to the user evaluation feedback information to a negative sample set.
In one example, if the monitoring result is an update, a model update mechanism is triggered. And analyzing, classifying and correcting the negative samples (error samples), updating the training set by combining the positive samples, and retraining the first NL2SQL model. And after the first NL2SQL model is trained, comparing the question and answer precision of the new model and the old model, and determining to continue training or update the models according to the comparison result. After the model is updated, the user behavior characteristics, the relevant statistical information and the like of the new model are recorded and collected, and a new model monitoring round is started.
Optionally, generating a positive sample set and a negative sample set according to the first target sample set and the user evaluation feedback information includes:
if the user evaluation feedback information is positive feedback, adding a historical query statement and an SQL statement corresponding to the user evaluation feedback information to a positive sample set;
and if the user evaluation feedback information is negative feedback, adding the historical query statement and the SQL statement corresponding to the user evaluation feedback information to a negative sample set.
The positive feedback may be a positive evaluation of the user on the question-answering effect, and may be, for example, "answer well". The negative feedback may be a negative evaluation of the user on the question-answering effect, and may be, for example, "wrong answer".
Optionally, the using frequency information includes: at least one of the number of user questions and answers, the question-answer duration, the user retention rate, the total number of first NL2SQL model calls in the preset time, the average number of user calls and the call frequency.
The preset time may be set by a user or a system, which is not limited in the embodiment of the present invention.
Optionally, the iteratively training the depth recommendation model through the first target sample set includes:
establishing a depth recommendation model;
inputting the user historical behavior information samples in the first target sample set into the deep recommendation model to obtain a prediction monitoring result;
training parameters of the deep recommendation model according to an objective function formed by the predicted monitoring result and the monitoring result corresponding to the user historical behavior information sample, wherein the objective function comprises: a focus loss function, a KL divergence loss function, and a cross entropy loss function;
and returning to execute the operation of inputting the user historical behavior information samples in the first target sample set into the deep recommendation model to obtain a predicted monitoring result until a monitoring model is obtained.
Wherein the focus loss function is an improvement on the basis of a cross entropy function. Mainly for solving the problem of serious imbalance of the proportion of positive and negative samples.
Wherein the KL divergence loss function is a measure of the degree of match between two distributions (e.g., two lines).
The cross entropy loss function is an important concept in Shannon information theory, and is mainly used for measuring difference information between two probability distributions.
Specifically, the weights of the Focal local Loss function, the KL divergence Loss function, and the cross entropy Loss function may be set in advance, and for example, the weight of the Focal local Loss function may be set to 0.5, the weight of the KL divergence Loss function may be set to 0.3, and the weight of the cross entropy Loss function may be set to 0.2.
The embodiment of the invention provides an NL2SQL model online monitoring and updating technology and method, which are used for intelligently and automatically monitoring and updating an NL2SQL model online. The monitoring model can autonomously learn user behaviors, model calling statistical information, historical information and the like, and intelligently and automatically optimize the NL2SQL model by combining a sample formed by using data feedback reflux by a front-end user, so that the NL2SQL model can be intelligently decided to be updated in no need of updating, how to update and how often.
According to the technical scheme of the embodiment, historical behavior information of a user is acquired; inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of a first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set, the problems that a scene which cannot be accessed by a network is not considered in online updating, the model can only be updated mechanically and the model cannot be updated intelligently can be solved, a large amount of labor cost is consumed in manual updating, the model updating efficiency is low, real-time performance cannot be achieved, and the model cannot be updated intelligently can be solved, and whether the model needs to be updated or not can be judged according to the historical behavior information of the user.
Fig. 2 is a schematic structural diagram of a monitoring device according to an embodiment of the present invention. This embodiment is applicable to the monitoring of the first NL2SQL model, the monitoring apparatus may be implemented in software and/or hardware, and the monitoring apparatus may be integrated in any device providing a monitoring function, as shown in fig. 2, and the monitoring apparatus specifically includes: an acquisition module 210 and a monitoring module 220.
The acquisition module is used for acquiring historical behavior information of a user;
and the monitoring module is used for inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
According to the technical scheme of the embodiment, historical behavior information of a user is acquired; inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of a first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set, the problems that a scene which cannot be accessed by a network is not considered in online updating, the model can only be updated mechanically and the model cannot be updated intelligently can be solved, a large amount of labor cost is consumed in manual updating, the model updating efficiency is low, real-time performance cannot be achieved, and the model cannot be updated intelligently can be solved, and whether the model needs to be updated or not can be judged according to the historical behavior information of the user.
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. FIG. 3 illustrates a block diagram of an electronic device 312 suitable for use in implementing embodiments of the present invention. The electronic device 312 shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of the use of the embodiment of the present invention. Device 312 is a computing device for typical trajectory fitting functions.
As shown in fig. 3, electronic device 312 is in the form of a general purpose computing device. The components of the electronic device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 that couples the various system components including the storage device 328 and the processors 316.
The processor 316 executes various functional applications and data processing by executing programs stored in the storage device 328, for example, implementing the monitoring method provided by the above-described embodiment of the present invention:
acquiring historical behavior information of a user;
inputting the historical user behavior information into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
Fig. 4 is a schematic structural diagram of a computer-readable storage medium containing a computer program according to an embodiment of the present invention. Embodiments of the present invention provide a computer-readable storage medium 61, on which a computer program 610 is stored, which when executed by one or more processors implements the monitoring method as provided by all inventive embodiments of the present application:
acquiring historical behavior information of a user;
inputting the historical user behavior information into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (Hyper Text Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of monitoring, comprising:
acquiring historical behavior information of a user;
inputting the historical user behavior information into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
2. The method of claim 1, wherein the user historical behavior information comprises: the user evaluates the feedback information and the usage frequency information.
3. The method of claim 2, before inputting the historical behavior information of the user into the monitoring model to obtain the monitoring result of the first NL2SQL model, further comprising:
and iteratively training the first neural network model through the second target sample set to obtain a first NL2SQL model.
4. The method of claim 3, wherein iteratively training the first neural network model through the second set of target samples yields a first NL2SQL model comprising:
establishing a first neural network model;
inputting a first historical query statement sample in the second target sample set into the first neural network model to obtain a predicted SQL statement;
training parameters of the first neural network model according to a first target function formed by the SQL statements corresponding to the prediction SQL statement sample and the first historical query statement sample;
and returning to execute the operation of inputting the historical query statement samples in the second target sample set into the first neural network model to obtain the predicted SQL statement until a first NL2SQL model is obtained.
5. The method of claim 4, after inputting the historical behavior information of the user into the monitoring model to obtain the monitoring result of the first NL2SQL model, further comprising:
if the monitoring result is updated, generating a positive sample set and a negative sample set according to the first target sample set and the user evaluation feedback information;
and iteratively training the first NL2SQL model through the positive sample set and the negative sample set to obtain a target NL2SQL model.
6. The method of claim 5, wherein generating a positive sample set and a negative sample set from the first target sample set and the user rating feedback information comprises:
if the user evaluation feedback information is positive feedback, adding a historical query statement and an SQL statement corresponding to the user evaluation feedback information to a positive sample set;
and if the user evaluation feedback information is negative feedback, adding the historical query statement and the SQL statement corresponding to the user evaluation feedback information to a negative sample set.
7. The method of claim 1, wherein iteratively training the depth recommendation model through the first set of target samples comprises:
establishing a depth recommendation model;
inputting the user historical behavior information samples in the first target sample set into the deep recommendation model to obtain a prediction monitoring result;
training parameters of the deep recommendation model according to an objective function formed by the predicted monitoring result and the monitoring result corresponding to the user historical behavior information sample, wherein the objective function comprises: a focus loss function, a KL divergence loss function, and a cross entropy loss function;
and returning to execute the operation of inputting the user historical behavior information samples in the first target sample set into the deep recommendation model to obtain a predicted monitoring result until a monitoring model is obtained.
8. A monitoring device, comprising:
the acquisition module is used for acquiring historical behavior information of a user;
and the monitoring module is used for inputting the historical behavior information of the user into a monitoring model to obtain a monitoring result of the first NL2SQL model, wherein the monitoring model is obtained by iteratively training a deep recommendation model through a first target sample set.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the processors to implement the method of any of claims 1-7.
10. A computer-readable storage medium containing a computer program, on which the computer program is stored, characterized in that the program, when executed by one or more processors, implements the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111405118.9A CN114048104A (en) | 2021-11-24 | 2021-11-24 | Monitoring method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111405118.9A CN114048104A (en) | 2021-11-24 | 2021-11-24 | Monitoring method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114048104A true CN114048104A (en) | 2022-02-15 |
Family
ID=80210850
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111405118.9A Pending CN114048104A (en) | 2021-11-24 | 2021-11-24 | Monitoring method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114048104A (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572998A (en) * | 2015-01-07 | 2015-04-29 | 北京云知声信息技术有限公司 | Updating method and device of question answer sequencing model for automatic question answer system |
| US20160371589A1 (en) * | 2015-06-17 | 2016-12-22 | Yahoo! Inc. | Systems and methods for online content recommendation |
| CN107729542A (en) * | 2017-10-31 | 2018-02-23 | 咪咕音乐有限公司 | A kind of information methods of marking and device and storage medium |
| CN108427992A (en) * | 2018-03-16 | 2018-08-21 | 济南飞象信息科技有限公司 | A kind of machine learning training system and method based on edge cloud computing |
| CN108921603A (en) * | 2018-06-21 | 2018-11-30 | 中国农业银行股份有限公司 | User's evaluation method and relevant apparatus |
| CN110276446A (en) * | 2019-06-26 | 2019-09-24 | 北京百度网讯科技有限公司 | The method and apparatus of model training and selection recommendation information |
| WO2020119030A1 (en) * | 2018-12-11 | 2020-06-18 | 平安科技（深圳）有限公司 | Model training method, device and equipment used for answering questions and storage medium |
| CN111538910A (en) * | 2020-06-23 | 2020-08-14 | 上海摩莱信息科技有限公司 | Intelligent recommendation method and device and computer storage medium |
| CN112307214A (en) * | 2019-07-26 | 2021-02-02 | 株式会社理光 | Deep reinforcement learning-based recommendation method and recommendation device |
| CN112748941A (en) * | 2020-08-06 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Feedback information-based target application program updating method and device |
| CN112783921A (en) * | 2021-01-26 | 2021-05-11 | 中国银联股份有限公司 | Database operation method and device |
| CN112990294A (en) * | 2021-03-10 | 2021-06-18 | 挂号网（杭州）科技有限公司 | Training method and device of behavior discrimination model, electronic equipment and storage medium |
| CN113360512A (en) * | 2021-06-21 | 2021-09-07 | 特赞（上海）信息科技有限公司 | Model updating method and device based on user feedback and storage medium |
-
2021
- 2021-11-24 CN CN202111405118.9A patent/CN114048104A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572998A (en) * | 2015-01-07 | 2015-04-29 | 北京云知声信息技术有限公司 | Updating method and device of question answer sequencing model for automatic question answer system |
| US20160371589A1 (en) * | 2015-06-17 | 2016-12-22 | Yahoo! Inc. | Systems and methods for online content recommendation |
| CN107729542A (en) * | 2017-10-31 | 2018-02-23 | 咪咕音乐有限公司 | A kind of information methods of marking and device and storage medium |
| CN108427992A (en) * | 2018-03-16 | 2018-08-21 | 济南飞象信息科技有限公司 | A kind of machine learning training system and method based on edge cloud computing |
| CN108921603A (en) * | 2018-06-21 | 2018-11-30 | 中国农业银行股份有限公司 | User's evaluation method and relevant apparatus |
| WO2020119030A1 (en) * | 2018-12-11 | 2020-06-18 | 平安科技（深圳）有限公司 | Model training method, device and equipment used for answering questions and storage medium |
| CN110276446A (en) * | 2019-06-26 | 2019-09-24 | 北京百度网讯科技有限公司 | The method and apparatus of model training and selection recommendation information |
| CN112307214A (en) * | 2019-07-26 | 2021-02-02 | 株式会社理光 | Deep reinforcement learning-based recommendation method and recommendation device |
| CN111538910A (en) * | 2020-06-23 | 2020-08-14 | 上海摩莱信息科技有限公司 | Intelligent recommendation method and device and computer storage medium |
| CN112748941A (en) * | 2020-08-06 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Feedback information-based target application program updating method and device |
| CN112783921A (en) * | 2021-01-26 | 2021-05-11 | 中国银联股份有限公司 | Database operation method and device |
| CN112990294A (en) * | 2021-03-10 | 2021-06-18 | 挂号网（杭州）科技有限公司 | Training method and device of behavior discrimination model, electronic equipment and storage medium |
| CN113360512A (en) * | 2021-06-21 | 2021-09-07 | 特赞（上海）信息科技有限公司 | Model updating method and device based on user feedback and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110366734B (en) | Optimizing neural network architecture | |
| JP2021534493A (en) | Techniques for building knowledge graphs within a limited knowledge domain | |
| CN110019616B (en) | POI (Point of interest) situation acquisition method and equipment, storage medium and server thereof | |
| CN110765247B (en) | Input prompting method and device for question-answering robot | |
| CN112712069B (en) | Question judging method and device, electronic equipment and storage medium | |
| US20190114711A1 (en) | Financial analysis system and method for unstructured text data | |
| CN111708876A (en) | Method and device for generating information | |
| CN117033667B (en) | Knowledge graph construction method and device, storage medium and electronic equipment | |
| CN110019849B (en) | Attention mechanism-based video attention moment retrieval method and device | |
| CN111340220A (en) | Method and apparatus for training a predictive model | |
| US11501655B2 (en) | Automated skill tagging, knowledge graph, and customized assessment and exercise generation | |
| CN113826113A (en) | Counting rare training data for artificial intelligence | |
| CN111444677A (en) | Reading model optimization method, device, equipment and medium based on big data | |
| CN111738010A (en) | Method and apparatus for generating semantic matching model | |
| CN113127060A (en) | Software function point identification method based on natural language pre-training model (BERT) | |
| CN117112744A (en) | Assessment method and device for large language model and electronic equipment | |
| CN115186738B (en) | Model training method, device and storage medium | |
| CN114048104A (en) | Monitoring method, device, equipment and storage medium | |
| CN114117239A (en) | House resource pushing method, device and equipment | |
| CN115526177A (en) | Training of object association models | |
| CN111459917B (en) | Knowledge base management method, device and processing equipment | |
| CN117494068B (en) | Network public opinion analysis method and device combining deep learning and causal inference | |
| EP4328805A1 (en) | Method and apparatus for generating target deep learning model | |
| CN116755683B (en) | Data processing method and related device | |
| CN116737964B (en) | Artificial intelligence brain system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |