CN108664605A - A kind of model evaluation method and system - Google Patents
A kind of model evaluation method and system Download PDFInfo
- Publication number
- CN108664605A CN108664605A CN201810439522.XA CN201810439522A CN108664605A CN 108664605 A CN108664605 A CN 108664605A CN 201810439522 A CN201810439522 A CN 201810439522A CN 108664605 A CN108664605 A CN 108664605A
- Authority
- CN
- China
- Prior art keywords
- value
- sample
- fragment
- metadata
- calculate node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Educational Administration (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Game Theory and Decision Science (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of model evaluation method and system, the method includes：Data distributing node obtains history access record to build model evaluation sample set in real time；Various kinds is originally distributed to multiple first order calculate nodes, first order calculate node extracts the metadata for calculating and presetting needed for evaluation index from sample, and metadata is distributed to multiple second level calculate nodes, second level calculate node carries out fragment polymerization according to metadata, fragment polymerizing value is sent to center calculation node, center calculation node, which carries out the fragment polymerizing value of each second level calculate node to summarize polymerization, obtains assessed value.Solving offline evaluation in the prior art can not effect on the line of monitoring model in time, it is higher for the positioning complexity of problem on line, it can not real-time assessment models, assess the relatively low problem of accuracy, model can be assessed according to real time data, assessment accuracy is improved, the positioning complexity of problem on line is reduced.
Description
Technical field
The present embodiments relate to network technique field more particularly to a kind of model evaluation method and system.
Background technology
In searching for commending system, the order models of machine learning etc., by elements such as businessman, commodity or contents
It carries out after giving a mark online, businessman, commodity or content is ranked up according to marking, finally according to ranking results by businessman, quotient
Product or commending contents are to user.To increasingly prominent to the importance of the Performance Evaluation of order models.
In the prior art, the major programme of offline evaluation includes：First, it as unit of the time cycles such as day, week, obtains in batches
Take the history usage record of business；Then, by off-line model to the offline sample set that is generated according to history usage record into
Row prediction marking, and AUC (Area Under the Curve of Receiver Operating are calculated according to marking
Characteristic, the area below recipient's operating characteristic curve), MAP (Mean Average Precision, it is average
Accuracy rate mean value), NDCG (Normalized Discounted Cumulative Gain, normalization lose storage gain) etc.
Evaluation index, as Performance Evaluation value.
However, offline evaluation can not in time monitoring model line on effect, it is higher for the positioning complexity of problem on line,
Can not real-time assessment models, assessment accuracy it is relatively low.
Invention content
The present invention provides a kind of model evaluation method and system, to solve the above problem of prior art model evaluation.
According to the first aspect of the invention, a kind of model evaluation method is provided, model evaluation system, the system are applied to
System includes：Data distributing node, and the associated at least one first order calculate node of the data distributing node, with each first order
The associated at least one second level calculate node of calculate node, with each associated center calculation section of second level calculate node
Point, the method includes：
The data distributing node obtains history access record in real time, and builds model from the history access record and comment
Estimate sample set, the sample set includes at least one sample；
Various kinds is originally distributed at least one first order calculate node, the first order meter by the data distributing node
Operator node divides the metadata of the sample for the metadata needed for the default evaluation index of extraction calculating from the sample
At least one second level calculate node is issued, the second level calculate node according to the metadata of each sample for being divided
Piece polymerize, and the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent to the center
Calculate node, the center calculation node are used to carry out the fragment polymerizing value that each second level calculate node obtains to summarize polymerization,
Assessed value is calculated.
According to the second aspect of the invention, a kind of model evaluation system is provided, the system comprises：Data distribution section
Point, it is associated extremely with each first order calculate node with the associated at least one first order calculate node of the data distributing node
A few second level calculate node, with each associated center calculation node of second level calculate node, the data distribution section
Point builds model evaluation sample set, the sample for obtaining history access record in real time from the history access record
Concentration includes at least one sample；Various kinds is originally distributed to at least one first order calculate node, the first order calculates
Node distributes the metadata of the sample for the metadata needed for the default evaluation index of extraction calculating from the sample
To at least one second level calculate node, the second level calculate node is used to carry out fragment according to the metadata of each sample
Polymerization, is calculated the corresponding fragment polymerizing value of the evaluation index, and the fragment polymerizing value is sent to the middle scheming
Operator node, the center calculation node are counted for carrying out the fragment polymerizing value that each second level calculate node obtains to summarize polymerization
Calculation obtains assessed value.
According to the third aspect of the invention we, a kind of electronic equipment is provided, including：
Processor, memory and it is stored in the computer journey that can be run on the memory and on the processor
Sequence, the processor realize the aforementioned model evaluation method when executing described program.
According to the fourth aspect of the invention, provide a kind of readable storage medium storing program for executing, when the instruction in the storage medium by
When the processor of electronic equipment executes so that electronic equipment is able to carry out the aforementioned model evaluation method.
An embodiment of the present invention provides a kind of model evaluation method and system, the method includes：The data distribution section
Point obtains history access record in real time, and model evaluation sample set is built from the history access record, in the sample set
Including at least one sample；Various kinds is originally distributed at least one first order calculate node, institute by the data distributing node
State first order calculate node for from the sample extraction calculate the metadata preset needed for evaluation index, and by the sample
Metadata be distributed at least one second level calculate node, the second level calculate node is used for the member according to each sample
Data carry out fragment polymerization, the corresponding fragment polymerizing value of the evaluation index are calculated, and the fragment polymerizing value is sent
To the center calculation node, the fragment polymerizing value that the center calculation node is used to obtain each second level calculate node carries out
Summarize polymerization, assessed value is calculated.Solve offline evaluation in the prior art can not effect on the line of monitoring model in time, it is right
It is higher in the positioning complexity of problem on line, can not real-time assessment models, the relatively low problem of assessment accuracy can be according to real-time
Data assess model, improve assessment accuracy, reduce the positioning complexity of problem on line.
Description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the present invention
Example, for those of ordinary skill in the art, without having to pay creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is a kind of specific steps flow chart for model evaluation method that the embodiment of the present invention one provides；
Fig. 2 is a kind of specific steps flow chart of model evaluation method provided by Embodiment 2 of the present invention；
Fig. 2A is the schematic diagram that the present invention assesses sample set by sliding target window structure；
Fig. 2 B are the schematic diagrames that the present invention carries out predicted value discretization；
Fig. 3 is a kind of structure chart for model evaluation system that the embodiment of the present invention three provides.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, the every other implementation that those of ordinary skill in the art are obtained without creative efforts
Example, shall fall within the protection scope of the present invention.
Embodiment one
Referring to Fig.1, a kind of specific steps flow chart of the model evaluation method provided it illustrates the embodiment of the present invention one.
Step 101, the data distributing node obtains history access record, and the structure from the history access record in real time
Established model assesses sample set, and the sample set includes at least one sample.
Wherein, history access record is the platform access record of model to be assessed application, including foreground and background server
Access record.For example, for taking out platform, user's platform when ordering take-away on platform can record the browsed quotient of user
The information such as family, vegetable, certainly, the embodiment of the present invention are concerned with for calculating the information needed for evaluation index value.
The embodiment of the present invention can access record by obtaining in real time, to ensure that the model evaluation sample set of structure is line
Upper state-of-the-art record improves the timeliness of assessment.Specifically, it can be obtained according to period certain time and access record.It is appreciated that
Time cycle can determine according to the update cycle of daily record.
Specifically, model evaluation sample set is built from the history access record, first, is led to from history access record
Cross the determination of the means such as format check records whether corresponding data are normal, for example, for 5 data of setting, if in record only
There are three, then judge the data exception, abandons the data；Then, related data is associated, wherein related data includes
But it is not limited to：Same user is with the paged data once accessed, and same user is with the foreground client data once accessed with after
The data etc. of platform server obtain assessment sample set.
The embodiment of the present invention is applied to model evaluation system, and model evaluation system includes：Data distributing node, with the number
According to the associated at least one first order calculate node of distribution node, with the associated at least one second level of each first order calculate node
Calculate node, with each associated center calculation node of second level calculate node.
Wherein, data distributing node is for collecting history access record, and tectonic model is assessed from history access record
Sample set.In practical applications, in order to improve efficiency, multiple data distributing nodes can be set.
First order calculate node is the downstream node of data distributing node.
Second level calculate node is the downstream node of first order calculate node.It is appreciated that each first order calculate node
At least correspond to a second level calculate node.
Center calculation node is the downstream node of the second calculate node.It should be noted that since center calculation node is used
It is calculated in summarizing, to there are one center calculation nodes.
It is appreciated that each calculate node can be CPU (Central Processing Unit, a central processing list
Member), or a server.The embodiment of the present invention does not limit it.
In model evaluation system as shown in Figure 3, including a data distributing node, two first order calculate nodes, five
A second level calculate node, a center calculation node.As can be seen that the model evaluation system of the embodiment of the present invention is streaming system
System.
Step 102, various kinds is originally distributed at least one first order calculate node by the data distributing node, described
First order calculate node for from the sample extraction calculate the metadata preset needed for evaluation index, and by the sample
Metadata is distributed at least one second level calculate node, and the second level calculate node is used for first number according to each sample
According to fragment polymerization is carried out, the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent to
The center calculation node, the center calculation node is for converging the fragment polymerizing value that each second level calculate node obtains
Assessed value is calculated in total polymerization.
Wherein, it is to treat assessment models to carry out assessing used computing object to preset evaluation index, can often be passed through
Functional expression indicates.In practical applications, evaluation index includes very much, such as：AUC(Area Under the Curve of
Receiver Operating Characteristic, the area below recipient's operating characteristic curve), MAP (Mean
Average Precision, Average Accuracy mean value), NDCG (Normalized Discounted Cumulative Gain,
Storage gain is lost in normalization), clicking rate, trade company's exposure clicking rate etc..
It calculates the metadata preset needed for evaluation index and frequently includes model predication value, sample value, sample labeling etc..When
So, for particular service, it is also necessary to some special datas.It in practical applications, can be according to identification information or preset
Format obtains corresponding metadata.
The embodiment of the present invention is handled data distribution to multiple nodes to improve calculating speed respectively.Multiple
Level-one calculate node can carry out the extraction of metadata parallel, and to improve extraction rate, multiple second calculate nodes can be parallel
Preliminary polymerization operation is carried out, to improve the arithmetic speed of center calculation node.
In conclusion an embodiment of the present invention provides a kind of model evaluation method, the method includes：The data distribution
Node obtains history access record in real time, and model evaluation sample set, the sample set are built from the history access record
It include at least one sample；Various kinds is originally distributed at least one first order calculate node by the data distributing node,
The first order calculate node for from the sample extraction calculate the metadata preset needed for evaluation index, and by the sample
This metadata is distributed at least one second level calculate node, and the second level calculate node is used for according to each sample
Metadata carries out fragment polymerization, the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent out
Send to the center calculation node, the fragment polymerizing value that the center calculation node is used to obtain each second level calculate node into
Row summarizes polymerization, and assessed value is calculated.Solve offline evaluation in the prior art can not effect on the line of monitoring model in time,
It is higher for the positioning complexity of problem on line, can not real-time assessment models, the relatively low problem of assessment accuracy can be according to reality
When data model is assessed, improve assessment accuracy, reduce the positioning complexity of problem on line.
Embodiment two
With reference to Fig. 2, it illustrates a kind of specific steps flow charts of model evaluation method provided by Embodiment 2 of the present invention.
Step 201, the data distributing node obtains history access record in real time.
The step is referred to the detailed description of step 101, and details are not described herein.
Step 202, according to preset single sliding length, preset target window is inserted into the history access record
The new position sliding for accessing record.
Wherein, single sliding length is too small, and newest history access record can not be got by being easy to cause；Single sliding length
It spends big, is easy to cause and slides stylish history access record and do not generate.Single sliding length can use integer history to access
Number is recorded to indicate.It is appreciated that single sliding length needs are set according to application scenarios, it can also be with other unit come table
Show, the embodiment of the present invention does not limit it.
As shown in Figure 2 A, B is history access record, and the left side is the history access record being newly inserted into, and the right is old history
Record is accessed, B1 is former target window, and B2 is the target window after primary sliding, and B3 is single sliding length.
Step 203, according to the history access record between the initial position and end position of the sliding window, mould is built
Type assesses sample set, and the sample set includes at least one sample.
Specifically, first, initial position and the end position of sliding window are determined；Then, it is read from history access record
It takes the access between initial position and end position to record and is used as model evaluation sample set.
Wherein it is determined that initial position and the end position of sliding window, it can be by former initial position plus single sliding length
Degree, obtains current initial position, end position is obtained current end position plus single sliding length.It is appreciated that first
When beginningization, initial position 0, end position is the extension position of target window.
Step 204, various kinds is originally distributed at least one first order calculate node by the data distributing node, described
First order calculate node for from the sample extraction calculate the metadata preset needed for evaluation index, and by the sample
Metadata is distributed at least one second level calculate node, and the second level calculate node is used for first number according to each sample
According to fragment polymerization is carried out, the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent to
The center calculation node, the center calculation node is for converging the fragment polymerizing value that each second level calculate node obtains
Assessed value is calculated in total polymerization.
The step is referred to the detailed description of step 102, and details are not described herein.
Optionally, in another embodiment of the invention, the above-mentioned extraction from the sample, which calculates, presets evaluation index
The step of required metadata, including sub-step 20401 to 20402：
Sub-step 20401, for default first kind evaluation index, if preset model to be assessed has been reached the standard grade, from described
Original predictive value and sample authentic signature are extracted in sample as metadata.
Wherein, first kind evaluation index includes but not limited to：AUC indexs.
Sample authentic signature corresponds to whether sample is positive sample.For example, positive sample can be indicated with 1, negative sample can use 0
It indicates.It is appreciated that can also indicate that positive negative sample, the embodiment of the present invention do not limit it with other identifier.
It is appreciated that for the model reached the standard grade, model can calculate original predictive value according to characteristic value when being run on line, from
And original predictive value can be recorded in history access record.It can directly be obtained from history access record in model evaluation
Get original predictive value.
Sub-step 20402, for default first kind evaluation index, if preset model to be assessed is not reached the standard grade, from described
Characteristic value and sample authentic signature are extracted in sample, and original predictive is calculated according to the characteristic value by the model to be assessed
Value, using the original predictive value and sample authentic signature as metadata.
It is appreciated that for the model that do not reach the standard grade, original predictive value can not be calculated when being run on line, so as to by feature
Value is recorded in history access record.Original predictive value is calculated according to characteristic value first in model evaluation.
Optionally, in another embodiment of the invention, the above-mentioned extraction from the sample, which calculates, presets evaluation index
The step of required metadata, including sub-step 20403 to 20404：
Sub-step 20403, for default second class evaluation index, if preset model to be assessed has been reached the standard grade, from described
Original predictive value, sample authentic signature and inquiry mark are extracted in sample is used as metadata.
Wherein, the second class evaluation index includes but not limited to：MAP, NDCG etc..Relative to first kind evaluation index, second
Class evaluation index is generally required according to inquiry mark to being assessed with a query result.
The the second class evaluation index reached the standard grade also is needed other than original predictive value to be extracted, sample authentic signature
Extract inquiry mark.
Sub-step 20404, for default second class evaluation index, if preset model to be assessed is not reached the standard grade, from described
Characteristic value, sample authentic signature and inquiry mark are extracted in sample, and by the model to be assessed according to the characteristic value meter
Original predictive value is calculated, regard the original predictive value, sample authentic signature and inquiry mark as metadata.
For the second class evaluation index that do not reach the standard grade, other than characteristic value to be extracted, sample authentic signature, it is also necessary to carry
Inquiry is taken to identify.
Optionally, in another embodiment of the invention, the above-mentioned metadata by the sample is distributed at least one
The step of second level calculate node, including：
Sub-step 20405 presses the original predictive value in the metadata of the sample default first kind evaluation index
It is blocked according to presetting digit capacity, as the predicted value in key value and the metadata.
Wherein, presetting digit capacity is bigger, and accuracy is better, but computation complexity is higher；Presetting digit capacity is smaller, and accuracy is got over
Difference, but computation complexity is lower.It is appreciated that presetting digit capacity can be set according to practical application scene, the embodiment of the present invention pair
It is not limited.In practical applications, presetting digit capacity can get after decimal point six to eight.
Specifically, block often to round up after decimal point and block, for example, after decimal point two block,
Then 95.678 block after not 95.68.
In practical applications, it can be identified according to predicted value or preset rules are accurately positioned to original predictive value.For example, working as
There are character string " #forecast Value：" when, then it obtains the character string and specifies the numerical value of digit as original predictive later
Value.
Sub-step 20406, for presetting the second class evaluation index, the extraction inquiry mark from the metadata of the sample,
And it regard inquiry mark as key value.
For the second class evaluation index, the result of same inquiry can be distributed to same calculate node by the embodiment of the present invention
On, so as to count the evaluation index of same query result.
The metadata of the sample is distributed at least one second level by sub-step 20407 according to the key value
Calculate node.
Sample is distributed to multiple second level calculate nodes by the embodiment of the present invention according to key value, to ensure each the as possible
The load difference of two level calculate node is smaller.
Optionally, in another embodiment of the invention, above-mentioned default evaluation index corresponds to first kind evaluation index, on
It states the corresponding fragment polymerizing value of default evaluation index and includes the first fragment polymerizing value, the second fragment polymerizing value, it is above-mentioned according to various kinds
This metadata carries out fragment polymerization, the step of evaluation index corresponding fragment polymerizing value is calculated, including sub-step
20408 to 20409：
Sub-step 20408 determines at least one sequencing numbers according to the presetting digit capacity.
Specifically, when presetting digit capacity is bigger, sequencing numbers number is more, and assessment is more accurate；When presetting digit capacity is smaller, sequence
Number number is fewer, and assessment is more inaccurate.
As shown in Figure 2 B, when presetting digit capacity is two, the number of sequencing numbers rank is 100, two neighboring sequence
The value difference 0.01 of number.
Sub-step 20409 counts the predicted value in the metadata according to the sample authentic signature and belongs to each row respectively
The positive sample number and negative sample number of sequence number obtain the corresponding first fragment polymerizing value of each sequencing numbers and the polymerization of the second fragment
Value.
As shown in Figure 2 B, after being blocked by sub-step 20405, when predicted value is 0.96, then this blocks predicted value and belongs to
Sequencing numbers 96.And so on, all predicted values of blocking are divided in each sequencing numbers, and count each sequencing numbers and correspond to
Positive sample number and negative sample number.In 2B, P is corresponding positive sample number in each sequencing numbers, and N is corresponding negative in each sequence
Sample number.
Based on the definition for authenticity sample labeling in sub-step 20401, it is 1 that positive sample, which corresponds to sample authentic signature,
Sample, negative sample correspond to the sample that sample authentic signature is 0.
Optionally, in another embodiment of the invention, the above-mentioned fragment polymerization for obtaining each second level calculate node
Value carries out the step of summarizing polymerization, assessed value is calculated, including sub-step 20410 to 20412：
Sub-step 20410 calculates the sum of the first fragment polymerizing value that each second level calculate node obtains, obtains the first polymerization
Value.
Sub-step 20410 to 20412 summarizes the fragment polymerizing value that sub-step 20408 to 20409 obtains, and obtains
A kind of evaluation index value.
The embodiment of the present invention also needs to summarize the first fragment polymerizing value that each second calculate node obtains.Specifically
The calculation formula on ground, the first polymerizing value P is as follows：
Wherein, PiFor the corresponding first fragment polymerizing value of sequencing numbers i, M is sequencing numbers sum.
Sub-step 20411 calculates the sum of the second fragment polymerizing value that each second level calculate node obtains, obtains the second polymerization
Value.
The embodiment of the present invention also needs to summarize the second fragment polymerizing value that each second calculate node obtains.Specifically
The calculation formula on ground, the second polymerizing value N is as follows：
Wherein, NiFor the corresponding second fragment polymerizing value of sequencing numbers i.
Sub-step 20412, according to the corresponding first fragment polymerizing value of each sequencing numbers and the second fragment polymerizing value, institute
The first polymerizing value and the second polymerizing value are stated, calculates first kind evaluation index as assessed value.
Specifically, first kind evaluation index PV1Calculation formula it is as follows：
Wherein, PiIt is the corresponding first fragment polymerizing value of i, P for sequencing numberskFor the sequencing numbers k before sequencing numbers i
Corresponding first fragment polymerizing value, NkFor the corresponding negative sample sums of sequencing numbers k before sequencing numbers i.
Optionally, in another embodiment of the invention, above-mentioned default evaluation index corresponds to the second class evaluation index, on
It includes third fragment polymerizing value to state the corresponding fragment polymerizing value of default evaluation index, and the above-mentioned metadata according to each sample is divided
Piece polymerize, the step of evaluation index corresponding fragment polymerizing value is calculated, including：
The metadata of each sample is sorted out according to inquiry mark, is obtained at least once by sub-step 20413
Inquire corresponding metadata sequence.
The embodiment of the present invention corresponding metadata will be divided in a sequence according to inquiry mark with one query, to
Each inquiry can be assessed, such as average accuracy, the accuracy of each inquiry can be calculated first, then,
The average value for the accuracy repeatedly inquired is counted again.
Sub-step 20414 inquires corresponding metadata sequence, according to the prediction in each metadata at least once for described
Value carries out descending sort to the metadata sequence.
Specifically, insertion sort, quicksort, selected and sorted etc. may be used in sort method.The embodiment of the present invention is to tool
Body sort method do not limit.
Sub-step 20415, for the metadata sequence after sequence, according to the sample authentic signature in each metadata, meter
The second class evaluation index is calculated, as third fragment polymerizing value.
For example, for the metadata sequence i on the calculate node l of the second level, the accuracy of query result on j-th of position
Calculation formula is as follows：
Wherein, Rl,i,kSample for the metadata of upper k-th of the position metadata sequence i on the calculate node l of the second level is true
Real label, authentic specimen correspond to 1, and non-genuine sample corresponds to 0.
To which the average accuracy of the metadata sequence i on the calculate node l of the second level is as follows：
Wherein, rell,i,jFor the sample of the metadata of upper j-th of the position metadata sequence i on the calculate node l of the second level
Authentic signature, J are the length of metadata sequence i.
Optionally, in another embodiment of the invention, the above-mentioned fragment polymerization for obtaining each second level calculate node
Value carries out the step of summarizing polymerization, assessed value is calculated, including sub-step 20418：
Sub-step 20416 is averaged to the third fragment polymerizing value that each second level calculate node obtains, is assessed
Value.
Sub-step 20416 summarizes the fragment polymerizing value that sub-step 20413 to 20414 obtains, and obtains the second class and comments
Estimate index value.
In the embodiment of the present invention, Centroid needs the third point for the multiple queries for obtaining multiple second level calculate nodes
Piece polymerizing value, is averaged, and assessed value is obtained.Specifically, assessed value PV2Calculation formula it is as follows：
In practical applications, all inquiries of the second calculate node can also be calculated first in the calculate node of the second level
Then the average value of third fragment polymerizing value calculates the average value of each second level calculate node, as assessment on Centroid
Value.
Optionally, in another embodiment of the invention, gather in the fragment for obtaining each second level calculate node
After conjunction value carries out the step of summarizing polymerization, assessed value is calculated, further include：
Sub-step 20417 preserves the assessed value into default assessment database.
Specifically, assessed value is preserved according to the time into assessment database, so as to be become according to the dynamic of assessed value
Change, position and solves the problems, such as on line.
Wherein, assessment database can be the database of application platform, or individual database.The present invention is implemented
Example does not limit it.
In conclusion an embodiment of the present invention provides a kind of model evaluation method, the method includes：The data distribution
Node obtains history access record in real time, and model evaluation sample set, the sample set are built from the history access record
It include at least one sample；Various kinds is originally distributed at least one first order calculate node by the data distributing node,
The first order calculate node for from the sample extraction calculate the metadata preset needed for evaluation index, and by the sample
This metadata is distributed at least one second level calculate node, and the second level calculate node is used for according to each sample
Metadata carries out fragment polymerization, the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent out
Send to the center calculation node, the fragment polymerizing value that the center calculation node is used to obtain each second level calculate node into
Row summarizes polymerization, and assessed value is calculated.Solve offline evaluation in the prior art can not effect on the line of monitoring model in time,
It is higher for the positioning complexity of problem on line, can not real-time assessment models, the relatively low problem of assessment accuracy can be according to reality
When data model is assessed, improve assessment accuracy, reduce the positioning complexity of problem on line.Further, it is also possible to
Different metadata is extracted according to different evaluation indexes and distributes the key value of data, to realize flexible evaluation.
Embodiment three
With reference to Fig. 3, it illustrates a kind of structure charts for model evaluation system that the embodiment of the present invention three provides, specifically such as
Under.
Data distributing node 301, and the associated at least one first order calculate node 302 of the data distributing node 301,
With the associated at least one second level calculate node 303 of each first order calculate node 302, with each second level calculate node
303 associated center calculation nodes 304, the data distributing node 301 for obtaining history access record in real time, and from described
Model evaluation sample set is built in history access record, the sample set includes at least one sample；Various kinds is originally distributed to
At least one first order calculate node 302, the first order calculate node 302 are used for the extraction from the sample and calculate in advance
If the metadata needed for evaluation index, and the metadata of the sample is distributed at least one second level calculate node
303, the second level calculate node 303 is used to carry out fragment polymerization according to the metadata of each sample, and the assessment is calculated
The corresponding fragment polymerizing value of index, and the fragment polymerizing value is sent to the center calculation node 304, the center calculation
Assessed value is calculated for carrying out the fragment polymerizing value that each second level calculate node 303 obtains to summarize polymerization in node 304.
Optionally, in another embodiment of the invention, above-mentioned first order calculate node 302 is used for：For default the
A kind of evaluation index extracts original predictive value from the sample and sample is true if preset model to be assessed has been reached the standard grade
Label is used as metadata；For default first kind evaluation index, if preset model to be assessed is not reached the standard grade, from the sample
Characteristic value and sample authentic signature are extracted, and original predictive value is calculated according to the characteristic value by the model to be assessed, it will
The original predictive value and sample authentic signature are as metadata.
Optionally, in another embodiment of the invention, above-mentioned first order calculate node 302 is used for：For default the
Two class evaluation indexes extract original predictive value, sample is really marked if preset model to be assessed has been reached the standard grade from the sample
Note and inquiry mark are used as metadata；For default second class evaluation index, if preset model to be assessed is not reached the standard grade, from institute
It states and extracts characteristic value, sample authentic signature and inquiry mark in sample, and by the model to be assessed according to the characteristic value
Original predictive value is calculated, regard the original predictive value, sample authentic signature and inquiry mark as metadata.
Optionally, in another embodiment of the invention, above-mentioned first order calculate node 302 is used for：For default the
A kind of evaluation index blocks the original predictive value in the metadata of the sample according to presetting digit capacity, as key value
With the predicted value in the metadata；For presetting the second class evaluation index, the extraction inquiry mark from the metadata of the sample
Know, and regard inquiry mark as key value；The metadata of the sample is distributed to according to the key value at least one
The second level calculate node 303.
Optionally, in another embodiment of the invention, above-mentioned default evaluation index corresponds to first kind evaluation index, on
It includes the first fragment polymerizing value, the second fragment polymerizing value, above-mentioned second level meter to state the corresponding fragment polymerizing value of default evaluation index
Operator node 303 is used for：At least one sequencing numbers are determined according to the presetting digit capacity；It is united respectively according to the sample authentic signature
Positive sample number and negative sample number that the predicted value in the metadata belongs to each sequencing numbers are counted, it is corresponding to obtain each sequencing numbers
First fragment polymerizing value and the second fragment polymerizing value.
Optionally, in another embodiment of the invention, scheming operator node 304 is used among the above：Calculate each second level
The sum of the first fragment polymerizing value that calculate node 303 obtains, obtains the first polymerizing value；Each second level calculate node 303 is calculated to obtain
The sum of second fragment polymerizing value arrived, obtains the second polymerizing value；According to the corresponding first fragment polymerizing value of each sequencing numbers
And the second fragment polymerizing value, first polymerizing value and the second polymerizing value, first kind evaluation index is calculated as assessed value.
Optionally, in another embodiment of the invention, above-mentioned default evaluation index corresponds to the second class evaluation index, institute
It includes third fragment polymerizing value to state the corresponding fragment polymerizing value of default evaluation index, and the second level calculate node 303 is used for：It will
The metadata of each sample is sorted out according to inquiry mark, obtains being directed to and inquires corresponding metadata sequence at least once；
Inquire corresponding metadata sequence at least once for described, according to the predicted value in each metadata to the metadata sequence into
Row descending sort；Is calculated by the second class and is commented according to the sample authentic signature in each metadata for metadata sequence after sequence
Index is estimated, as third fragment polymerizing value.
Optionally, in another embodiment of the invention, scheming operator node 304 is used among the above：Each second level is counted
The third fragment polymerizing value that operator node 303 obtains is averaged, and assessed value is obtained.
Optionally, in another embodiment of the invention, above-mentioned data distributing node 301 is used for：According to preset list
Preset target window is inserted into the new position for accessing record to the history access record and slided by secondary sliding length；According to institute
The history access record between the initial position of sliding window and end position is stated, model evaluation sample set is built.
Optionally, in another embodiment of the invention, scheming operator node 304 is used among the above：By the assessed value
It preserves into default assessment database.
In conclusion an embodiment of the present invention provides a kind of model evaluation system, the system comprises：Data distribution section
Point, it is associated extremely with each first order calculate node with the associated at least one first order calculate node of the data distributing node
A few second level calculate node, with each associated center calculation node of second level calculate node, the data distribution section
Point builds model evaluation sample set, the sample for obtaining history access record in real time from the history access record
Concentration includes at least one sample；Various kinds is originally distributed to at least one first order calculate node, the first order calculates
Node distributes the metadata of the sample for the metadata needed for the default evaluation index of extraction calculating from the sample
To at least one second level calculate node, the second level calculate node is used to carry out fragment according to the metadata of each sample
Polymerization, is calculated the corresponding fragment polymerizing value of the evaluation index, and the fragment polymerizing value is sent to the middle scheming
Operator node, the center calculation node are counted for carrying out the fragment polymerizing value that each second level calculate node obtains to summarize polymerization
Calculation obtains assessed value.Solve offline evaluation in the prior art can not effect on the line of monitoring model in time, for problem on line
Positioning complexity it is higher, can not real-time assessment models, the relatively low problem of assessment accuracy can be according to real time data to model
It is assessed, improves assessment accuracy, reduce the positioning complexity of problem on line.Further, it is also possible to be commented according to different
Estimate the different metadata of index extraction and distribute the key value of data, to realize flexible evaluation.
The embodiment of the present invention additionally provides a kind of electronic equipment, including：Processor, memory and it is stored in the storage
On device and the computer program that can run on the processor, realize that foregoing model is commented when the processor executes described program
Estimate method.
The embodiment of the present invention additionally provides a kind of readable storage medium storing program for executing, when the instruction in the storage medium is by electronic equipment
Processor execute when so that electronic equipment is able to carry out foregoing model appraisal procedure.
For system embodiments, since it is basically similar to the method embodiment, so fairly simple, the correlation of description
Place illustrates referring to the part of embodiment of the method.
Algorithm and display be not inherently related to any certain computer, virtual system or miscellaneous equipment provided herein.
Various general-purpose systems can also be used together with teaching based on this.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the present invention is not also directed to any certain programmed language.It should be understood that can utilize various
Programming language realizes the content of invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that the implementation of the present invention
Example can be put into practice without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this description.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of each inventive aspect,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：It is i.e. required to protect
Shield the present invention claims the more features of feature than being expressly recited in each claim.More precisely, as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific implementation mode are expressly incorporated in the specific implementation mode, wherein each claim itself
All as a separate embodiment of the present invention.
Those skilled in the art, which are appreciated that, to carry out adaptively the module in the equipment in embodiment
Change and they are arranged in the one or more equipment different from the embodiment.It can be the module or list in embodiment
Member or component be combined into a module or unit or component, and can be divided into addition multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it may be used any
Combination is disclosed to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so to appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification (including adjoint power
Profit requires, abstract and attached drawing) disclosed in each feature can be by providing the alternative features of identical, equivalent or similar purpose come generation
It replaces.
The all parts embodiment of the present invention can be with hardware realization, or to run on one or more processors
Software module realize, or realized with combination thereof.It will be understood by those of skill in the art that can use in practice
Microprocessor or digital signal processor (DSP) come realize in model evaluation system according to the ... of the embodiment of the present invention some or
The some or all functions of person's whole component.The present invention is also implemented as one for executing method as described herein
Divide either whole equipment or program of device.It is such to realize that the program of the present invention be stored in computer-readable medium
On, or can be with the form of one or more signal.Such signal can be downloaded from internet website and be obtained, or
Person provides on carrier signal, or provides in any other forms.
It should be noted that the present invention will be described rather than limits the invention for above-described embodiment, and ability
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference mark between bracket should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" before element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be by the same hardware branch
To embody.The use of word first, second, and third does not indicate that any sequence.These words can be explained and be run after fame
Claim.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all essences in the present invention
All any modification, equivalent and improvement etc., should all be included in the protection scope of the present invention made by within refreshing and principle.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all contain
Lid is within protection scope of the present invention.Therefore, protection scope of the present invention should be subject to the protection scope in claims.
Claims (13)
1. a kind of model evaluation method, which is characterized in that it is applied to model evaluation system, the system comprises：Data distribution section
Point, it is associated extremely with each first order calculate node with the associated at least one first order calculate node of the data distributing node
A few second level calculate node, and each associated center calculation node of second level calculate node, the method includes：
The data distributing node obtains history access record in real time, and model evaluation sample is built from the history access record
This collection, the sample set include at least one sample；
Various kinds is originally distributed at least one first order calculate node by the data distributing node, and the first order calculates section
The metadata of the sample is distributed to by point for the metadata needed for the default evaluation index of extraction calculating from the sample
At least one second level calculate node, the second level calculate node are used to carry out fragment according to the metadata of each sample poly-
It closes, the corresponding fragment polymerizing value of the evaluation index is calculated, and the fragment polymerizing value is sent to the center calculation
Node, the center calculation node are calculated for carrying out the fragment polymerizing value that each second level calculate node obtains to summarize polymerization
Obtain assessed value.
2. according to the method described in claim 1, it is characterized in that, the extraction from the sample calculates and presets evaluation index
The step of required metadata, including：
For default first kind evaluation index, if preset model to be assessed has been reached the standard grade, extracted from the sample original pre-
Measured value and sample authentic signature are as metadata；
For the default first kind evaluation index characteristic value is extracted from the sample if preset model to be assessed is not reached the standard grade
With sample authentic signature, and by the model to be assessed according to the characteristic value calculate original predictive value, will be described original pre-
Measured value and sample authentic signature are as metadata.
3. according to the method described in claim 1, it is characterized in that, the extraction from the sample calculates and presets evaluation index
The step of required metadata, including：
It is extracted from the sample original pre- for default second class evaluation index if preset model to be assessed has been reached the standard grade
Measured value, sample authentic signature and inquiry mark are used as metadata；
For default second class evaluation index, if preset model to be assessed is not reached the standard grade, from the sample extract characteristic value,
Sample authentic signature and inquiry mark, and original predictive value is calculated according to the characteristic value by the model to be assessed, by institute
It states original predictive value, sample authentic signature and inquiry mark and is used as metadata.
4. according to the method described in claim 1, it is characterized in that, the metadata by the sample be distributed to it is at least one
The step of second level calculate node, including：
For default first kind evaluation index, the original predictive value in the metadata of the sample is cut according to presetting digit capacity
It is disconnected, as the predicted value in key value and the metadata；
For presetting the second class evaluation index, the extraction inquiry mark from the metadata of the sample, and the inquiry is identified
As key value；
The metadata of the sample is distributed at least one second level calculate node according to the key value.
5. according to the method described in claim 2, it is characterized in that, the corresponding fragment polymerizing value of the default evaluation index includes
First fragment polymerizing value, the second fragment polymerizing value, it is described that fragment polymerization is carried out according to the metadata of each sample, it is calculated described
The step of evaluation index corresponding fragment polymerizing value, including：
At least one sequencing numbers are determined according to the presetting digit capacity；
The positive sample number that the predicted value in the metadata belongs to each sequencing numbers is counted respectively according to the sample authentic signature
With negative sample number, the corresponding first fragment polymerizing value of each sequencing numbers and the second fragment polymerizing value are obtained.
6. according to the method described in claim 5, it is characterized in that, the fragment polymerization that each second level calculate node is obtained
Value carries out the step of summarizing polymerization, assessed value is calculated, including：
The sum of the first fragment polymerizing value that each second level calculate node obtains is calculated, the first polymerizing value is obtained；
The sum of the second fragment polymerizing value that each second level calculate node obtains is calculated, the second polymerizing value is obtained；
According to the corresponding first fragment polymerizing value of each sequencing numbers and the second fragment polymerizing value, first polymerizing value and
Dimerization value calculates first kind evaluation index as assessed value.
7. according to the method described in claim 3, it is characterized in that, the corresponding fragment polymerizing value of the default evaluation index includes
Third fragment polymerizing value, it is described that fragment polymerization is carried out according to the metadata of each sample, it is corresponding that the evaluation index is calculated
The step of fragment polymerizing value, including：
The metadata of each sample is sorted out according to inquiry mark, obtains being directed to and inquires corresponding metadata at least once
Sequence；
Corresponding metadata sequence is inquired at least once for described, according to the predicted value in each metadata to the metadata sequence
Row carry out descending sort；
The second class evaluation index is calculated according to the sample authentic signature in each metadata for the metadata sequence after sequence,
As third fragment polymerizing value.
8. the method according to the description of claim 7 is characterized in that the fragment polymerization that each second level calculate node is obtained
Value carries out the step of summarizing polymerization, assessed value is calculated, including：
It averages to the third fragment polymerizing value that each second level calculate node obtains, obtains assessed value.
9. according to the method described in claim 1, it is characterized in that, described build model evaluation from the history access record
The step of sample set, including：
According to preset single sliding length, preset target window is inserted into new access record to the history access record
It slides position；
According to the history access record between the initial position and end position of the sliding window, model evaluation sample is built
Collection.
10. according to the method described in claim 1, it is characterized in that, in the fragment for obtaining each second level calculate node
After polymerizing value carries out the step of summarizing polymerization, assessed value is calculated, further include：
The assessed value is preserved into default assessment database.
11. a kind of model evaluation system, which is characterized in that the system comprises：Data distributing node, with the data distribution section
The associated at least one first order calculate node of point is calculated with the associated at least one second level of each first order calculate node and is saved
Point, with each associated center calculation node of second level calculate node, the data distributing node for obtaining history in real time
Record is accessed, and builds model evaluation sample set from the history access record, the sample set includes at least one sample
This；Various kinds is originally distributed to at least one first order calculate node, the first order calculate node is used for from the sample
Middle extraction calculates the metadata preset needed for evaluation index, and the metadata of the sample is distributed at least one described second
Grade calculate node, the second level calculate node are used to carry out fragment polymerization according to the metadata of each sample, are calculated described
The corresponding fragment polymerizing value of evaluation index, and the fragment polymerizing value is sent to the center calculation node, the middle scheming
Assessed value is calculated for carrying out the fragment polymerizing value that each second level calculate node obtains to summarize polymerization in operator node.
12. a kind of electronic equipment, which is characterized in that including：
Processor, memory and it is stored in the computer program that can be run on the memory and on the processor,
It is characterized in that, the processor realizes the model evaluation as described in one or more in claim 1-10 when executing described program
Method.
13. a kind of readable storage medium storing program for executing, which is characterized in that when the instruction in the storage medium is held by the processor of electronic equipment
When row so that electronic equipment is able to carry out the model evaluation method as described in one or more in claim to a method 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810439522.XA CN108664605B (en) | 2018-05-09 | 2018-05-09 | Model evaluation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810439522.XA CN108664605B (en) | 2018-05-09 | 2018-05-09 | Model evaluation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108664605A true CN108664605A (en) | 2018-10-16 |
| CN108664605B CN108664605B (en) | 2021-03-09 |
Family
ID=63778896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810439522.XA Active CN108664605B (en) | 2018-05-09 | 2018-05-09 | Model evaluation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108664605B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109784494A (en) * | 2018-11-28 | 2019-05-21 | 同盾控股有限公司 | A kind of machine learning method and device based on pyspark |
| CN111125071A (en) * | 2019-11-25 | 2020-05-08 | 珠海格力电器股份有限公司 | Method and device for evaluating real-time data reliability and storage medium |
| CN111950738A (en) * | 2020-08-10 | 2020-11-17 | 中国平安人寿保险股份有限公司 | Machine learning model optimization effect evaluation method and device, terminal and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170046422A1 (en) * | 2014-05-30 | 2017-02-16 | Huawei Technologies Co., Ltd. | Data Mining Method and Apparatus |
| CN106875206A (en) * | 2016-07-18 | 2017-06-20 | 阿里巴巴集团控股有限公司 | Acquisition of information, assessment, questionnaire method, device and server |
| CN106934493A (en) * | 2017-02-28 | 2017-07-07 | 北京科技大学 | A kind of construction method of power customer appraisal Model |
| CN107038167A (en) * | 2016-02-03 | 2017-08-11 | 普华诚信信息技术有限公司 | Big data excavating analysis system and its analysis method based on model evaluation |
-
2018
- 2018-05-09 CN CN201810439522.XA patent/CN108664605B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170046422A1 (en) * | 2014-05-30 | 2017-02-16 | Huawei Technologies Co., Ltd. | Data Mining Method and Apparatus |
| CN107038167A (en) * | 2016-02-03 | 2017-08-11 | 普华诚信信息技术有限公司 | Big data excavating analysis system and its analysis method based on model evaluation |
| CN106875206A (en) * | 2016-07-18 | 2017-06-20 | 阿里巴巴集团控股有限公司 | Acquisition of information, assessment, questionnaire method, device and server |
| CN106934493A (en) * | 2017-02-28 | 2017-07-07 | 北京科技大学 | A kind of construction method of power customer appraisal Model |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109784494A (en) * | 2018-11-28 | 2019-05-21 | 同盾控股有限公司 | A kind of machine learning method and device based on pyspark |
| CN111125071A (en) * | 2019-11-25 | 2020-05-08 | 珠海格力电器股份有限公司 | Method and device for evaluating real-time data reliability and storage medium |
| CN111950738A (en) * | 2020-08-10 | 2020-11-17 | 中国平安人寿保险股份有限公司 | Machine learning model optimization effect evaluation method and device, terminal and storage medium |
| CN111950738B (en) * | 2020-08-10 | 2023-09-15 | 中国平安人寿保险股份有限公司 | Machine learning model optimization effect evaluation method, device, terminal and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108664605B (en) | 2021-03-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105868373B (en) | Method and device for processing key data of power business information system | |
| CN107688591B (en) | A kind of actuarial treating method and apparatus | |
| CN104756106A (en) | Characterizing data sources in a data storage system | |
| CN107146089A (en) | The single recognition methods of one kind brush and device, electronic equipment | |
| CN106874335B (en) | Behavior data processing method and device and server | |
| CN108664605A (en) | A kind of model evaluation method and system | |
| CN107248052A (en) | A kind of commodity stocks information determines method, apparatus and system | |
| CN107408114A (en) | Based on transactions access pattern-recognition connection relation | |
| CN109002900A (en) | Predict method, apparatus, computer equipment and the storage medium of insurance risk probability | |
| CN107168977A (en) | A kind of optimization method and device of data query | |
| CN107291745B (en) | Data index management method and device | |
| CN109241068A (en) | The method, apparatus and terminal device that foreground and background data compares | |
| CN114037219A (en) | Data evaluation method and device and electronic equipment | |
| CN105740434B (en) | Network information methods of marking and device | |
| CN110689211A (en) | Method and device for evaluating website service capability | |
| CN104199945A (en) | Data storing method and device | |
| CN109634964A (en) | Cost Index exponent data base management system | |
| CN107688595B (en) | Information retrieval Accuracy Evaluation, device and computer readable storage medium | |
| CN113792084A (en) | Data heat analysis method, device, equipment and storage medium | |
| CN106021511A (en) | Processing method and apparatus of network accounts | |
| CN104462556A (en) | Method and device for recommending question and answer page related questions | |
| CN109101562A (en) | Find method, apparatus, computer equipment and the storage medium of target group | |
| CN101882259A (en) | Method and equipment for filtering entity relationship instance | |
| CN110111131A (en) | The determination method and device of false visitor's standing breath | |
| CN115840771B (en) | Data processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |