CN107590065A

CN107590065A - Algorithm model detection method, device, equipment and system

Info

Publication number: CN107590065A
Application number: CN201610539668.2A
Authority: CN
Inventors: 倪静
Original assignee: Alibaba Group Holding Ltd
Current assignee: Zhejiang Tmall Technology Co Ltd
Priority date: 2016-07-08
Filing date: 2016-07-08
Publication date: 2018-01-16
Anticipated expiration: 2036-07-08
Also published as: CN107590065B

Abstract

The application provides a kind of algorithm model detection method, device, equipment and system, and the system includes：Evaluation and test module, for being evaluated and tested by algorithm model to information to be evaluated, the algorithm model includes algorithm model to be measured and history algorithm model, and the history algorithm model is a preceding iterative model for the algorithm model to be measured；Algorithm model detection module, for determining the accuracy of the algorithm model to be measured；Algorithm model training module, for training the algorithm model according to training sample set.The accuracy of algorithm model treatment user's evaluation information to be measured is detected by the result of history algorithm model processing user's evaluation information, the test accuracy of algorithm model is improved, is not required to manual testing's algorithm model, improve the test speed of algorithm model.

Description

Algorithm model detection method, device, equipment and system

Technical field

The application is related to Internet technology, more particularly to a kind of algorithm model detection method and device.

Background technology

In the prior art, algorithm model is as the Cognitive Mode of the mankind is, it is necessary to constantly be corrected, be constantly trained to, specifically Ground, algorithm model constantly change the parameter of its own during sample data is handled so that the continuous quilt of algorithm model Iteration, still, if being allowed to continuous iteration, the algorithm model after iteration is not tested, will be unable to after determining iteration Whether algorithm model can handle sample data according to default accuracy.

Prior art has following two methods to the method for testing of algorithm model：One kind is manual testing, for example, algorithm mould Type has carried out processing to 100 sample datas and has obtained 100 results, has the correct result of processing in 100 results, also there is place The result of mistake is managed, for the accuracy of verification algorithm model, how many correct result in artificial judgment 100 results, How many individual error results, with the accuracy and/or error rate of assessment algorithm model.But when algorithm model needs to handle largely Sample data, and algorithm model iteration cycle it is shorter when, manual testing's is less efficient.Another kind is to pre-establish two Data set, the data that a data are concentrated are really correct, and the data in another data set are really wrong, algorithm mould Type handles the data in the two data sets successively, produces result corresponding to each data, according to corresponding to each data Result and the data correctness of itself, determine the accuracy and/or error rate of the algorithm model.But algorithm mould The data of type actual treatment are not limited to the data in the two data sets, if algorithm model is handled into the correct of preset data The accuracy and/or error rate of rate and/or error rate as the non-default data of algorithm model processing, equally can not accurately be detected Go out the accuracy and/or error rate of algorithm model.

Therefore, prior art lacks a kind of method of testing of accurate and quick algorithm model.

The content of the invention

The application provides a kind of algorithm model detection method, device, equipment and system, with quick accurate detection algorithm mould Type.

On one side, the application provides a kind of information evaluating system, including：

Evaluation and test module, for being evaluated and tested by algorithm model to information to be evaluated, the algorithm model includes to be measured Algorithm model and history algorithm model, the history algorithm model are a preceding iterative models for the algorithm model to be measured；

Algorithm model detection module, for obtaining first sample collection corresponding to algorithm model to be measured and history algorithm model pair Identical target sample in the second sample set answered；Obtain and obtained after the algorithm model to be measured is handled the target sample The first results set arrived, and the history algorithm model target sample is handled after obtained the second result set Close；According to first results set and second results set, first set is determined, the first set is described The set that the target sample of same treatment result is formed is corresponding with one results set and second results set；According to described The number of target sample in the number of target sample in first set, and first results set, it is determined that described treat The accuracy of method of determining and calculating model；

Algorithm model training module, for training the algorithm model according to training sample set.

On the other hand, the application provides a kind of user's evaluation information detection method, including：

Obtain multiple user's evaluation informations；

The multiple user's evaluation information is differentiated respectively by algorithm model to be measured to obtain the first results set, institute State the first differentiation that the first results set includes obtaining after the algorithm model to be measured differentiates each user's evaluation information As a result；

The multiple user's evaluation information is differentiated respectively by history algorithm model to obtain the second results set, institute State the second differentiation that the second results set includes obtaining after the history algorithm model differentiates each user's evaluation information As a result；

According to first results set and second results set, determine first set, the first set be The collection that user's evaluation information of identical differentiation result is formed is corresponding with first results set and second results set Close；

The number of user's evaluation information in the first set, and of the multiple user's evaluation information Number, determine the accuracy of the algorithm model to be measured；

According to the accuracy of the algorithm model to be measured, determine that the multiple user's evaluation information is corresponding respectively and differentiate knot Fruit；

Wherein, the history algorithm model is a preceding iterative model for the algorithm model to be measured.

On the other hand, the application provides a kind of algorithm model detection method, including：

Obtain identical in the second sample set corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model Target sample, the history algorithm model is a preceding iterative model for the algorithm model to be measured；

Obtain the first results set obtained after the algorithm model to be measured is handled the target sample, Yi Jisuo State the second results set obtained after history algorithm model is handled the target sample；

According to first results set and second results set, determine first set, the first set be The set that the target sample of same treatment result is formed is corresponding with first results set and second results set；

Target sample in the number of target sample in the first set, and first results set Number, determine the accuracy of the algorithm model to be measured.

Another further aspect, the application provide a kind of algorithm model detection means, including：

First acquisition module, for obtaining corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model Identical target sample in second sample set, the history algorithm model are the preceding an iteration moulds of the algorithm model to be measured Type；

Second acquisition module, for obtaining obtained after the algorithm model to be measured is handled the target sample One results set, and the history algorithm model target sample is handled after obtained the second results set；

First determining module, for according to first results set and second results set, determining first set, The first set is that the target of same treatment result is corresponding with first results set and second results set The set that sample is formed；

Second determining module, for the number of the target sample in the first set, and first result The number of target sample in set, determine the accuracy of the algorithm model to be measured.

Further aspect, the application provide a kind of detection device, including：Memory and processor；

The memory and processor coupling, the memory are used to store the first sample corresponding to algorithm model to be measured Second sample set corresponding to this collection and history algorithm model, the history algorithm model be the algorithm model to be measured it is preceding once Iterative model；And the storage algorithm model to be measured the target sample is handled after obtained the first results set and The second results set that the history algorithm model obtains after handling the target sample；

The processor is used to determine identical target sample in the first sample collection and second sample set；According to First results set and second results set, determine first set, and the first set is in first result The set that the target sample of same treatment result is formed is corresponding with set and second results set；According to the described first collection The number of target sample in the number of target sample in conjunction, and first results set, determine the algorithm to be measured The accuracy of model.

In this application, the first results set obtained after being handled by algorithm model to be measured target sample, with And history algorithm model same target sample is handled after obtained the second results set, determine algorithm model to be measured and History algorithm model result identical target sample, is determined as same according to algorithm model to be measured and history algorithm model The number of the target sample of individual result, and algorithm model to be measured are determined as the number of the target sample of the result, it may be determined that go out The accuracy of algorithm model to be measured, algorithm model to be measured is detected by the result of history algorithm model processing user's evaluation information The accuracy of user's evaluation information is handled, the test accuracy of algorithm model is improved, in addition, being not required to manual testing's algorithm mould Type, improve the test speed of algorithm model.

Brief description of the drawings

In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are this hairs Some bright embodiments, for those of ordinary skill in the art, without having to pay creative labor, can be with Other accompanying drawings are obtained according to these accompanying drawings.

Fig. 1 is the structural representation for the information evaluating system that the embodiment of the present application one provides；

Fig. 2 is the structural representation for the information evaluating system that the embodiment of the present application two provides；

Fig. 3 is the flow chart for user's evaluation information detection method that the embodiment of the application one provides；

Fig. 3 A are the schematic diagram for the algorithm model testing result that the embodiment of the present application one provides；

Fig. 3 B are the schematic diagram for the algorithm model testing result that the embodiment of the present application one provides；

Fig. 3 C are the schematic diagram for the algorithm model testing result that the embodiment of the present application one provides；

Fig. 3 D are the schematic diagram for the algorithm model testing result that the embodiment of the present application one provides；

Fig. 4 is the flow chart for the algorithm model detection method that the embodiment of the present application one provides；

Fig. 4 A are the schematic diagram for the algorithm model testing result that the embodiment of the present application two provides；

Fig. 4 B are the schematic diagram for the algorithm model testing result that the embodiment of the present application two provides；

Fig. 4 C are the schematic diagram for the algorithm model testing result that the embodiment of the present application two provides；

Fig. 5 is the flow chart for the algorithm model detection method that the embodiment of the present application two provides；

Fig. 6 is the flow chart for the algorithm model detection method that the embodiment of the present application three provides；

Fig. 7 is the flow chart for the algorithm model detection method that the embodiment of the present application four provides；

Fig. 7 A are the schematic diagram for the algorithm model testing result that the embodiment of the present application five provides；

Fig. 8 is the flow chart for the algorithm model detection method that the embodiment of the present application five provides；

Fig. 9 is the flow chart for the algorithm model detection method that the embodiment of the present application six provides；

Figure 10 is the flow chart for the algorithm model detection method that the embodiment of the present application seven provides；

Figure 11 is the structural representation for the algorithm model detection means that the embodiment of the present application one provides；

Figure 12 is the structural representation for the algorithm model detection means that the embodiment of the present application two provides；

Figure 13 is the structural representation for the algorithm model detection means that the embodiment of the present application three provides；

Figure 14 is the structural representation for the algorithm model detection means that the embodiment of the present application four provides；

Figure 15 is the structural representation for the detection device that the embodiment of the application one provides；

Figure 16 is the structural representation for the detection device that another embodiment of the application provides.

Embodiment

Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represent same or analogous key element.Following exemplary embodiment Described in embodiment do not represent and the consistent all embodiments of the present invention.On the contrary, they be only with it is such as appended The example of the consistent apparatus and method of some aspects being described in detail in claims, of the invention.

With the development of Internet technology, webpage not only turns into user and obtains the source of information, while also becomes user Upload information and the approach of shopping online, and the advertisement in webpage is also more and more, such as user have purchased on shopping webpage One commodity, user can input evaluation information after receiving the commodity in shopping webpage, to evaluate the commodity, and portion User is divided to take this opportunity to input advertisement in value column.For another example, forum Web pages are for sharing information between user, still Certain customers can also issue some junk information by forum Web pages.In addition, the form of advertisement is also constantly changing, for example, net There are multiple links in page, advertisement is pointed in which part link, and user is difficult to recognize which link is directed to advertisement.Except this it Outside, user is also possible to the URL of the input error in the address field of browser, and the wrong unified resource is determined It is advertisement page or junk information corresponding to position symbol is possible.The application is suitable for passing through algorithm model to such as user webpage Input information, URL, pictorial information etc. are evaluated and tested, and to detect the advertisement in webpage or junk information, are Algorithm model is improved to the accuracy of identification of advertisement or junk information, it is necessary to be trained by sample data to algorithm model, Algorithm model is constantly iterated in the training process, still, if being allowed to continuous iteration, not to the algorithm model after iteration Tested, will be unable to determine the algorithm model after iteration whether can it is anticipated that accuracy detect advertisement in webpage Or junk information, therefore, it is necessary to detected to the algorithm model of continuous iteration, such as detect its accuracy and/or error rate.

Fig. 1 is the structural representation for the information evaluating system that the embodiment of the present application one provides, as shown in figure 1, the information is commented Examining system includes：Evaluation and test module 31, algorithm model detection module 32 and algorithm model training module 33.

Evaluation and test module 31, for being evaluated and tested by algorithm model to information to be evaluated, the algorithm model includes treating Method of determining and calculating model and history algorithm model, the history algorithm model are a preceding iterative models for the algorithm model to be measured.

Algorithm model detection module 32, for obtaining first sample collection corresponding to algorithm model to be measured and history algorithm model Identical target sample in corresponding second sample set；Obtain after the algorithm model to be measured handled the target sample The first obtained results set, and the history algorithm model target sample is handled after obtained the second result Set；According to first results set and second results set, first set is determined, the first set is described The set that the target sample of same treatment result is formed is corresponding with first results set and second results set；According to institute The number of the target sample in the number of the target sample in first set, and first results set is stated, it is determined that described The accuracy of algorithm model to be measured.

Algorithm model training module 33, for training the algorithm model according to training sample set.

As shown in figure 1, algorithm model training module 33 is from the acquisition algorithm model of evaluation and test module 31；Algorithm model training module 33 pairs of algorithm models are trained, and generate new algorithm model；New algorithm model is sent to by algorithm model training module 33 Evaluation and test module 31；Evaluation and test module 31 is evaluated and tested according to new algorithm model to information to be evaluated, generates evaluation result, example Such as, the comment information of user is evaluated and tested, whether the comment information for testing user is advertisement, and evaluation result includes：It is advertisement, non- Advertisement, optionally, 1 identified ad of the present embodiment, with the 0 non-advertisement of mark；Evaluation result is sent to algorithm by evaluation and test module 31 Model checking module 32；Algorithm model detection module 32 is according to evaluation result, it is determined that the accuracy of new algorithm model；Algorithm mould The accuracy of new algorithm model is sent to algorithm model training module 33 by type detection module 32；Algorithm model training module 33 According to the accuracy of new algorithm model, it is determined that whether new algorithm model needs to continue iteration.

Wherein, evaluation and test module 31 is specifically used for the evaluation information of user's input, user in shopping webpage in forum Web pages On the information shared etc. evaluated and tested, judge whether it is advertisement or junk information, or to the link in webpage, unified resource Finger URL is evaluated and tested, and judges whether it points to advertisement or junk information, and the method for evaluation and test is defeated to user by algorithm model The evaluation information entered is evaluated and tested.Evaluation and test module 31 needs algorithm model training module 33 to pass through training sample set pair evaluation and test module Algorithm model used in 31 is trained, and iteration goes out new algorithm model, in order to avoid new algorithm model do not reach it is expected Detection results detect the algorithm, it is necessary to algorithm model detection module 32 detects to the algorithm model used in evaluation and test module 31 Model iterates to the accuracy and/or error rate during a certain version.

In the present embodiment, the first results set for being obtained after being handled by algorithm model to be measured target sample, with And history algorithm model same target sample is handled after obtained the second results set, determine algorithm model to be measured and History algorithm model result identical target sample, is determined as same according to algorithm model to be measured and history algorithm model The number of the target sample of individual result, and algorithm model to be measured are determined as the number of the target sample of the result, it may be determined that go out The accuracy of algorithm model to be measured, algorithm model to be measured is detected by the result of history algorithm model processing user's evaluation information The accuracy of user's evaluation information is handled, the test accuracy of algorithm model is improved, in addition, being not required to manual testing's algorithm mould Type, improve the test speed of algorithm model.

Fig. 2 is the structural representation for the information evaluating system that the embodiment of the present application two provides, as shown in Fig. 2 shown in Fig. 1 On the basis of embodiment, information evaluating system also includes：Display module 34, memory 35, data processing platform (DPP) 36, management and control module 37 and algorithm model iteration control module 38.

Display module 34, for showing User Interface.

Memory 35, for storing the information to be evaluated in the User Interface, the packet to be evaluated Include following at least one information：The information of user's input, URL, pictorial information.

Data processing platform (DPP) 36, for obtaining the information to be evaluated from memory 35, for the evaluation and test module 31 pairs of information to be evaluated are evaluated and tested.

Management and control module 37, for judging the information to be evaluated when the evaluation and test module 31 to need limited information When, control the display module 34 to shield the information for needing to be limited.In addition, management and control module 37 is additionally operable to issuing the need The user account for the information to be limited carries out punishment management, such as deducts the integration of the user account, limits its login etc..

Algorithm model iteration control module 38, for the algorithm to be measured detected according to the algorithm model detection module 32 The accuracy and/or error rate of model, control whether the algorithm model training module 33 needs to stop algorithm mould described in iteration Type.

In addition, the training sample that the training sample used in algorithm model training module 33 is concentrated includes following at least one： It is described to need limited information, report information, calling information, information identified in advance.As shown in Fig. 2 algorithm model is trained The training sample source of module 33 can be that seller complains system 41, malice buyer prosecution system 42, limited information prosecution system 44th, the report information and/or calling information that system 45 provides are complained in artificial customer service, can also be what advertisement prosecution system 43 provided Identified information in advance, it can also be that evaluated module 31 is judged as needing limited information.In addition, seller complains system 41st, malice buyer prosecution system 42, limited information prosecution system 44, artificial customer service complain system 45 provide report information and/ Or calling information, and the information identified in advance that advertisement prosecution system 43 provides can also be synchronized to data processing platform (DPP) 36。

Fig. 3 is the flow chart for user's evaluation information detection method that the embodiment of the application one provides, as shown in figure 3, the party Method can also comprise the following steps：

Step S801, multiple user's evaluation informations are obtained.

The present embodiment is by taking user's evaluation information in shopping webpage as an example, for example, obtaining 10 from a certain shopping webpage User's evaluation information, as shown in Figure 3A, 10 user's evaluation informations are identified with digital 1-10 respectively.

Step S802, the multiple user's evaluation information is differentiated to obtain the first knot respectively by algorithm model to be measured Fruit set, first results set include what is obtained after the algorithm model to be measured differentiates to each user's evaluation information First differentiates result.

Algorithm model to be measured to 10 user's evaluation informations judges whether determine each user's evaluation information respectively By restricted information, in the present embodiment, to include advertising message by restricted information, in addition, being believed in other embodiments by limitation Breath can also be gambling advertisement information, forbid picture for being spread etc..

Specifically, if algorithm model to be measured judges that user's evaluation information is advertising message, to user's evaluation information mark 1 is designated as, if algorithm model to be measured judges that user's evaluation information is non-advertising message, 0 is labeled as to user's evaluation information, is treated Method of determining and calculating model judges after terminating that each user's evaluation information is corresponding with a result of determination 1 to 10 user's evaluation informations Or 0, the result of determination obtained after algorithm model to be measured is respectively processed to 10 user's evaluation informations forms the first knot Fruit set.Optionally, algorithm model to be measured judges that preceding 3 user's evaluation informations are non-advertisements in 10 user's evaluation informations, 7 user's evaluation informations are advertisements afterwards, i.e., algorithm model to be measured judges that user's evaluation information 1-3 result of determination is 0, Yong Huping Valency information 4-10 result of determination is 1.Specifically, as shown in Figure 3 B, the first results set includes each first results set The identification information of user's evaluation information, and algorithm model to be measured is to the result of determination of user's evaluation information.

Step S803, the multiple user's evaluation information is differentiated to obtain the second knot respectively by history algorithm model Fruit set, second results set include what is obtained after the history algorithm model differentiates to each user's evaluation information Second differentiates result.

10 user's evaluation informations are judged respectively by history algorithm model, determine each user's evaluation information Whether it is by restricted information, in the present embodiment, advertising message is included by restricted information, in addition, being limited in other embodiments Information processed can also be gambling advertisement information, forbid picture for being spread etc..

For example, history algorithm model judges that preceding 5 user's evaluation informations are non-advertisements in 10 user's evaluation informations, after 5 user's evaluation informations are advertisements, i.e., history algorithm model judges that user's evaluation information 1-5 result of determination is 0, user's evaluation Information 6-10 result of determination is 1.Similarly, history algorithm model can be obtained to be respectively processed 10 user's evaluation informations The second results set obtained afterwards, as shown in Figure 3 C, the second results set includes each user and evaluates letter second results set The identification information of breath, and history algorithm model is to the result of determination of user's evaluation information.

In the present embodiment, the history algorithm model is a preceding iterative model for the algorithm model to be measured.Assuming that The iteration cycle of algorithm model is one day, for example, history algorithm model is the iterative model on May 8, algorithm model to be measured is 5 The iterative model of months 9 days, then algorithm model to be measured formed by history algorithm model iteration.

Step S804, according to first results set and second results set, first set is determined, described first Set is that user's evaluation information of identical differentiation result is corresponding with first results set and second results set The set of composition.

In the present embodiment, the first set is algorithm model to be measured described in first results set and described gone through History algorithm model is determined as the set that user's evaluation information of advertising message is formed；Or the first set is described first Algorithm model to be measured described in results set and the history algorithm model are determined as user's evaluation information of non-advertising message The set of composition.

Such as shown in Fig. 3 D, the judgement knot of history algorithm model and algorithm model to be measured to user's evaluation information 1-3,6-10 Fruit is identical, and the result of determination to user's evaluation information 4,5 is different, and the present embodiment is by history algorithm model and to be measured Method model is determined as that user's evaluation information of advertisement forms first set 50, or by history algorithm model and algorithm mould to be measured Type is determined as that user's evaluation information of non-advertisement forms first set 50.

Step S805, the number of user's evaluation information in the first set, and the multiple user evaluation The number of information, determine the accuracy of the algorithm model to be measured.

Step S805 is consistent with step S104, and specific method, here is omitted.

Step S806, according to the accuracy of the algorithm model to be measured, determine that the multiple user's evaluation information is right respectively The differentiation result answered.

In the present embodiment, can be by algorithm model to be measured point if the accuracy of the algorithm model to be measured is more than threshold value The other differentiation result to multiple user's evaluation informations differentiates result as final corresponding to multiple user's evaluation informations difference.

In the present embodiment, the first result for being obtained after being handled by algorithm model to be measured multiple user's evaluation informations Set, and history algorithm model same multiple user's evaluation informations are handled after obtained the second results set, really Fixed algorithm model to be measured and history algorithm model result identical user's evaluation information, according to algorithm model to be measured and history Algorithm model is determined as that the number of user's evaluation information of same result, and algorithm model to be measured are determined as the result The number of user's evaluation information, it may be determined that go out the accuracy of algorithm model to be measured, handling user by history algorithm model evaluates The result of information detects the accuracy of algorithm model treatment user's evaluation information to be measured, improve user's evaluation information whether be The differentiation degree of accuracy of advertising message.

Fig. 4 is the flow chart for the algorithm model detection method that the embodiment of the present application one provides, as shown in figure 4, this method bag Include following steps：

Step S101, the second sample corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model is obtained Identical target sample is concentrated, the history algorithm model is a preceding iterative model for the algorithm model to be measured；

The present embodiment is calculated by taking user's evaluation information in shopping webpage as an example using user's evaluation information as sample, detection The accuracy of advertisement, the version of initial algorithm model are denoted as V in method Model Identification user's evaluation information₀, when initial algorithm After the accuracy of model is more than certain threshold value, start to be iterated initial algorithm model, the version of the algorithm model after iteration Originally it is denoted as V_n(n >=1), the present embodiment specifically uses V_n-1The result of the algorithm model processing user evaluation information of version detects V_nVersion The accuracy of this algorithm model processing user's evaluation information, if algorithm model is periodic iterations, with the previous cycle The result of algorithm model processing user's evaluation information handles the correct of user's evaluation information to detect the algorithm model of current period Rate, i.e., before the algorithm model in previous cycle as history algorithm model is the algorithm model algorithm model i.e. to be measured of current period An iteration model.User's evaluation information of algorithm model treatment to be measured forms first sample collection, the processing of history algorithm model User's evaluation information forms the second sample set, the sample that first sample collection and the second sample set overlap.

For example, the iteration cycle of algorithm model is one day, history algorithm model is the iterative model on May 8, algorithm to be measured Model is the iterative model on May 9, and algorithm model to be measured is formed by history algorithm model iteration, it is assumed that history algorithm Model handled 100 user's evaluation informations May 8, and 100 user's evaluation informations form the second sample set, to be measured Method model handled 200 user's evaluation informations May 9, and 200 user's evaluation informations form first sample collection, detection 5 Identical user's evaluation information in month 100 user's evaluation informations of 8 days and 200 user's evaluation informations on May 9, and will Identical user evaluation information is as identical target sample in first sample collection and the second sample set, it is assumed that first sample collection 10 identical user's evaluation informations are concentrated with the second sample, 10 identical user evaluation informations are respectively with digital 1-10 It is identified, as shown in Figure 3A.

Step S102, the first result set obtained after the algorithm model to be measured is handled the target sample is obtained Close, and the history algorithm model target sample is handled after obtained the second results set；

Algorithm model to be measured to 10 user's evaluation informations judges whether determine each user's evaluation information respectively For advertisement, specifically, if algorithm model to be measured judges that user's evaluation information is advertisement, 1 is labeled as to user's evaluation information, If algorithm model to be measured judges that user's evaluation information is non-advertisement, 0 is labeled as to user's evaluation information, algorithm model to be measured Each user's evaluation information, which is corresponding with a result of determination 1 or 0, will be to be measured is judged after terminating to 10 user's evaluation informations The result of determination that algorithm model obtains after being respectively processed to 10 user's evaluation informations forms the first results set, optional , algorithm model to be measured judges that preceding 3 user's evaluation informations are non-advertisements in 10 user's evaluation informations, and rear 7 users comment Valency information is advertisement, i.e., algorithm model to be measured judges that user's evaluation information 1-3 result of determination is 0, user's evaluation information 4-10 Result of determination be 1.Specifically, as shown in Figure 3 B, the first results set includes each user and evaluates letter first results set The identification information of breath, and algorithm model to be measured is to the result of determination of user's evaluation information.

Optionally, history algorithm model judges that preceding 5 user's evaluation informations are non-advertisements in 10 user's evaluation informations, 5 user's evaluation informations are advertisements afterwards, i.e., history algorithm model judges that user's evaluation information 1-5 result of determination is 0, Yong Huping Valency information 6-10 result of determination is 1.Similarly, history algorithm model can be obtained to locate 10 user's evaluation informations respectively The second results set obtained after reason, second results set as shown in Figure 3 C, including each user evaluate by the second results set The identification information of information, and history algorithm model is to the result of determination of user's evaluation information.

Step S103, according to first results set and second results set, first set is determined, described first Set is that the target sample that same treatment result is corresponding with first results set and second results set is formed Set；

Because history algorithm model and algorithm model to be measured judge that the accuracy of user's evaluation information is different, then history algorithm Model and algorithm model to be measured may be identical to the result of determination of same user's evaluation information, may be different.Such as Fig. 3 D institutes Show, history algorithm model and algorithm model to be measured are identicals to user's evaluation information 1-3,6-10 result of determination, to user The result of determination of evaluation information 4,5 is different, and history algorithm model and algorithm model to be measured are determined as extensively by the present embodiment User's evaluation information of announcement forms first set 50, or history algorithm model and algorithm model to be measured are determined as into non-advertisement User's evaluation information form first set 50.

Step S104, in the number of the target sample in the first set, and first results set The number of target sample, determine the accuracy of the algorithm model to be measured.

It can be seen from step S103, each target sample in first set is history algorithm model and algorithm model to be measured It is determined as user's evaluation information of advertisement or non-advertisement, then user's evaluation information in first set is actual for advertisement or non-wide The possibility of announcement is larger.

If first set includes history algorithm model and algorithm model to be measured is determined as user's evaluation information of advertisement, The number of user's evaluation information in first set, and algorithm model to be measured is determined as advertisement in the first results set The number of user's evaluation information, it may be determined that go out the accuracy of algorithm model to be measured.

If first set includes history algorithm model and algorithm model to be measured is determined as user's evaluation information of non-advertisement, The then number of user's evaluation information in first set, and in the first results set algorithm model to be measured be determined as it is non-wide The number of user's evaluation information of announcement, it may be determined that go out the accuracy of algorithm model to be measured.

Fig. 5 is the flow chart for the algorithm model detection method that the embodiment of the present application two provides, as shown in figure 5, shown in Fig. 4 On the basis of embodiment, step S103 method may include steps of：

Step S201, according to first results set, determine corresponding to the first result in first results set Target sample；

It can be seen from step S103, algorithm model to be measured judges that preceding 3 users evaluate letter in 10 user's evaluation informations Breath is non-advertisement, and rear 7 user's evaluation informations are advertisements, then in the first results set user's evaluation information 1-3 result of determination It is 0, user's evaluation information 4-10 result of determination is 1, it is preferable that the present embodiment is using advertisement as the first result, by non-advertisement As the second result, selected from the first results set and be determined as that user's evaluation information of advertisement is used by algorithm model to be measured Family evaluation information 4-10, dotted line frame as shown in Figure 4 A.

Step S202, according to second results set, determine corresponding to the first result in second results set Target sample；

It can be seen from step S103, history algorithm model judges that preceding 5 users evaluate letter in 10 user's evaluation informations Breath is non-advertisement, and rear 5 user's evaluation informations are advertisements, then in the second results set user's evaluation information 1-5 result of determination It is 0, user's evaluation information 6-10 result of determination is 1, it is preferable that the present embodiment is using advertisement as the first result, by non-advertisement As the second result, selected from the second results set and be determined as that user's evaluation information of advertisement is used by history algorithm model Family evaluation information 6-10, dotted line frame as shown in Figure 4 B.

Step S203, target sample corresponding to the first result in first results set and second result are determined The common factor of target sample corresponding to the first result in set is the first set.

The user's evaluation information 4-10 and the second result of advertisement will be determined as in first results set by algorithm model to be measured It is determined as that user's evaluation information 6-10 of the advertisement i.e. user's evaluation information 6-10 of common factor is formed in set by history algorithm model First set, dotted line frame as shown in Figure 4 C.

In addition, the present embodiment also can be using advertisement as the second result, using non-advertisement as the first result, now, the first collection The determination method of conjunction with advertisement as the first result, non-advertisement as the second result when, the determination method of first set is consistent.

In the present embodiment, by target sample and the second results set corresponding to the first result in the first results set The first result corresponding to target sample common factor as first set, there is provided determine the simple and effective side of first set Method.

Fig. 6 is the flow chart for the algorithm model detection method that the embodiment of the present application three provides, as shown in fig. 6, shown in Fig. 5 On the basis of embodiment, step S104 method may include steps of：

Step S301, the number of target sample and first knot corresponding to the first result in the first set are calculated First ratio of the number of target sample corresponding to the first result in fruit set；

The present embodiment is using advertisement as the first result, using non-advertisement as the second result, it can be seen from step S203, first Set includes user's evaluation information 6-10 i.e. history algorithm model and algorithm model to be measured is determined as that the user of advertisement evaluates letter Breath.That be determined as advertisement by algorithm model to be measured in the first results set is user's evaluation information 4-10.

The user for being determined as advertisement by history algorithm model and algorithm model to be measured in first set evaluates letter The number T of the user's evaluation information for being determined as advertisement by algorithm model to be measured in the number T and the first results set of breath_n's Ratio obtains the first ratioHerein, T is equal to 5, T_nEqual to 7.

Step S302, the ratio for determining first ratio and attenuation coefficient is the accuracy of the algorithm model to be measured, The attenuation coefficient is equal in the number of target sample and the 4th results set corresponding to the first result in the 3rd results set The first result corresponding to target sample number ratio, the 3rd results set is the history algorithm model to described The results set that the second sample in second sample set obtains after being handled, the 4th results set are the algorithms to be measured The results set that the first sample that model is concentrated to the first sample obtains after handling.

Accuracy of the ratio of first ratio and attenuation coefficient as algorithm model to be measured, it is preferable that attenuation coefficient according to The 3rd results set and institute that the history algorithm model obtains after handling the second sample in second sample set State the 4th results set obtained after the first sample that algorithm model to be measured concentrates the first sample is handled to determine, tool Body, the second sample set corresponding to history algorithm model is if the history algorithm model being described in detail in step S101 is in processing on May 8 100 user's evaluation informations, history algorithm model sentences to each user's evaluation information in 100 user's evaluation informations The 3rd results set is obtained after fixed, the 3rd results set includes each user's evaluation information in 100 articles of user's evaluation informations Identification information, and history algorithm model is to the result of determination of each user's evaluation information；First sample corresponding to algorithm model to be measured This collection is the 200 user's evaluation informations handled such as the algorithm model to be measured being described in detail in step S101 May 9, algorithm to be measured Model obtains the 4th results set after judging each user's evaluation information in 200 articles of user's evaluation informations, and the 4th Results set includes the identification information of each user's evaluation information in 200 user's evaluation informations, and algorithm model pair to be measured The result of determination of each user's evaluation information.It is assumed that it is determined as that the user of advertisement comments by history algorithm model in the 3rd results set The number of valency information is F_n-1, it is determined as the number of user's evaluation information of advertisement in the 4th results set by algorithm model to be measured For F_n, then attenuation coefficientAbove-mentioned first ratioWith attenuation coefficientRatio be described to be measured The accuracy of method model

In the present embodiment, on the premise of the higher accuracy of history algorithm model is ensured, handled by history algorithm model The result of user's evaluation information detects the accuracy of algorithm model treatment user's evaluation information to be measured, improves algorithm model Test accuracy.

Fig. 7 is the flow chart for the algorithm model detection method that the embodiment of the present application four provides, as shown in fig. 7, shown in Fig. 6 On the basis of embodiment, after step S104, this method can also comprise the following steps：

Step S401, according to the accuracy of the algorithm model to be measured and the accuracy of the history algorithm model, it is determined that Whether the algorithm model to be measured needs to stop iteration.

Specifically, if the accuracy of the algorithm model to be measured is more than or equal to the accuracy of the history algorithm model, Then determine that the algorithm model to be measured need not stop iteration；If the accuracy of the algorithm model to be measured is calculated less than the history The accuracy of method model, it is determined that the algorithm model to be measured needs to stop iteration.

Because history algorithm model is a preceding iterative model for algorithm model to be measured, then detection history algorithm model is correct The method of rate is consistent with the method for detecting algorithm model accuracy to be measured, and the present embodiment is by comparing the correct of history algorithm model Rate and the accuracy of algorithm model to be measured, determine whether algorithm model to be measured makes progress compared to history algorithm model, specifically, If the accuracy of the algorithm model to be measured is more than or equal to the accuracy of the history algorithm model, illustrate algorithm mould to be measured Type makes progress compared to history algorithm model, and algorithm model to be measured can further iterate to the iterative model in next cycle；If institute The accuracy for stating algorithm model to be measured is less than the accuracy of the history algorithm model, then illustrates algorithm model to be measured compared to going through History algorithm model not, it is necessary to treat method of determining and calculating model corrected by progress, i.e., algorithm model to be measured needs to stop iteration, otherwise The accuracy of the iterative model in next cycle can not be ensured.

In addition, shown in the step of step S401 can also be after embodiment illustrated in fig. 5 step S104 or Fig. 6 The step of after embodiment step S302.

In the present embodiment, by comparing the accuracy of history algorithm model and the accuracy of algorithm model to be measured, it is determined that treating Whether method of determining and calculating model is needed to stop iteration, and the iteration in next cycle is ensure that while algorithm model accuracy to be measured is improved The accuracy of model.

Fig. 8 is the flow chart for the algorithm model detection method that the embodiment of the present application five provides, as shown in figure 8, shown in Fig. 4 On the basis of embodiment, after step S104, this method can also comprise the following steps：

Step S501, according to first results set and second results set, second set is determined, described second Set is that the target sample that different disposal result is corresponding with first results set and second results set is formed Set；

It can be seen from step S103, because history algorithm model and algorithm model to be measured judge the correct of user's evaluation information Rate is different, then history algorithm model and algorithm model to be measured may be identical to the result of determination of same user's evaluation information, can Can be different.For example, 10 user's evaluation informations are identified with digital 1-10 respectively, history algorithm model judges that 10 users comment Preceding 5 user's evaluation informations are non-advertisements in valency information, and rear 5 user's evaluation informations are advertisements, i.e., history algorithm model judges User's evaluation information 1-5 result of determination is 0, and user's evaluation information 6-10 result of determination is 1；Algorithm model to be measured judges should Preceding 3 user's evaluation informations are non-advertisements in 10 user's evaluation informations, and rear 7 user's evaluation informations are advertisements, i.e., to be measured Method model judges that user's evaluation information 1-3 result of determination is 0, and user's evaluation information 4-10 result of determination is 1.It can be seen that go through History algorithm model and algorithm model to be measured are identicals to user's evaluation information 1-3,6-10 result of determination, and user is evaluated and believed The result of determination of breath 4,5 is different, and history algorithm model is determined as that non-advertisement, algorithm model to be measured are determined as by the present embodiment User's evaluation information of advertisement forms second set, or history algorithm model is determined as into advertisement, algorithm model to be measured judge Second set is formed for user's evaluation information of non-advertisement.

Specifically, according to first results set and second results set, second set, second collection are determined Conjunction is that the target sample that different disposal result is corresponding with first results set and second results set is formed Set, including：According to first results set, target sample corresponding to the first result in first results set is determined This；According to second results set, target sample corresponding to the second result in second results set is determined；Determine institute State corresponding to the second result in target sample and second results set corresponding to the first result in the first results set The common factor of target sample is the second set.

Preferably, the present embodiment is using advertisement as the first result, using non-advertisement as the second result, from the first results set In select the user's evaluation information i.e. user's evaluation information 4-10 for being determined as advertisement by algorithm model to be measured, from the second result set The user's evaluation information i.e. user's evaluation information 1-5 for being determined as non-advertisement by history algorithm model is selected in conjunction, by the first knot It is determined as in fruit set by algorithm model to be measured in the user's evaluation information 4-10 and the second results set of advertisement by history algorithm Model is determined as that user's evaluation information 1-5 of non-advertisement is occured simultaneously, and the common factor is user's evaluation information 4 and 5, by user Evaluation information 4 and 5 forms second set, the dotted line frame 51 of second set as shown in Figure 7 A, then by algorithm to be measured in second set The number of user's evaluation information that model is determined as advertisement, is determined as non-advertisement by history algorithm model is 2.

Step S502, in the number of the target sample in the second set, and second results set The number of target sample, determine the error rate of the algorithm model to be measured.

Specifically, the mesh in the number of the target sample in the second set, and second results set The number of standard specimen sheet, the error rate of the algorithm model to be measured is determined, including：Determine the target sample in the second set The ratio of the number of target sample corresponding to the second result in number and second results set is the algorithm mould to be measured The error rate of type.

In the second results set, history algorithm model judges that user's evaluation information 1-5 is non-advertisement, then the second result set The number for being determined as user's evaluation information of non-advertisement in conjunction by history algorithm model is 5, according to be measured in second set Method model is determined as advertisement, be determined as by history algorithm model non-advertisement user's evaluation information number M and the second result set It is determined as the number M of user's evaluation information of non-advertisement in conjunction by history algorithm model_n-1Ratio obtain algorithm model to be measured Error rateHerein, M is equal to 2, M_n-1Equal to 5.

In addition, the step of step S501 and S502 can also be after embodiment illustrated in fig. 2 step S104 or figure The step of after 3 illustrated embodiment step S302, the step of can also be after embodiment illustrated in fig. 4 step S401.

In the present embodiment, detected by the result of history algorithm model processing user's evaluation information at algorithm model to be measured The error rate of user's evaluation information is managed, improves the test accuracy of algorithm model.

Fig. 9 is the flow chart for the algorithm model detection method that the embodiment of the present application six provides, as shown in figure 9, shown in Fig. 8 On the basis of embodiment, after step S502, this method can also comprise the following steps：

Step S601, according to the error rate of the algorithm model to be measured and the error rate of the history algorithm model, it is determined that Whether the algorithm model to be measured needs to stop iteration.

Specifically, if the error rate of the algorithm model to be measured is less than or equal to the error rate of the history algorithm model, Then determine that the algorithm model to be measured need not stop iteration；If the error rate of the algorithm model to be measured is calculated more than the history The error rate of method model, it is determined that the algorithm model to be measured needs to stop iteration.

Because history algorithm model is a preceding iterative model for algorithm model to be measured, then detection history algorithm model mistake The method of rate is consistent with the method for detecting algorithm model errors rate to be measured, and the present embodiment is by comparing the mistake of history algorithm model Rate and the error rate of algorithm model to be measured, determine whether algorithm model to be measured makes progress compared to history algorithm model, specifically, If the error rate of the algorithm model to be measured is less than the error rate of the history algorithm model, illustrate that algorithm model to be measured is compared Made progress in history algorithm model, algorithm model to be measured can further iterate to the iterative model in next cycle；It is if described to be measured The error rate of algorithm model is more than or equal to the error rate of the history algorithm model, then illustrates algorithm model to be measured compared to going through History algorithm model not, it is necessary to treat method of determining and calculating model corrected by progress, i.e., algorithm model to be measured needs to stop iteration, otherwise The accuracy of the iterative model in next cycle can not be ensured.

In the present embodiment, by comparing the error rate of history algorithm model and the error rate of algorithm model to be measured, it is determined that treating Whether method of determining and calculating model is needed to stop iteration, and the iteration in next cycle is ensure that while algorithm model errors rate to be measured is reduced The accuracy of model.

Figure 10 is the flow chart for the algorithm model detection method that the embodiment of the present application seven provides, as shown in Figure 10, in Fig. 9 institutes On the basis of showing embodiment, if after step S601 determines that the algorithm model to be measured need not stop iteration, this method may be used also To comprise the following steps：

Step S701, acquisition and target information to be evaluated is stored；

Because the quantity of shopping webpage is a lot, each shopping webpage may be browsed by different users, same user Evaluation information, therefore, it is necessary to the evaluation information inputted to user in shopping webpage may also be inputted in different shopping webpages Obtained and stored, is especially stored in database.

Step S702, evaluation and test in real time or offline evaluation and test are carried out to the target information by the algorithm model to be measured；

Algorithm model to be measured can be evaluated and tested in real time to the evaluation information in database, i.e., increasing one newly in database comments Valency information is evaluated and tested to the evaluation information at once, judges whether it is advertisement, can also be evaluated and tested offline, i.e., is increased newly in database To the plurality of evaluation information batch processing after multiple evaluation informations, or the evaluation letter of multiple history to being stored in database Cease batch processing.

If step S703, described target information to be evaluated is to need limited information, the target information is shielded；

If the evaluation information that algorithm model inspection to be measured goes out user's input is advertisement, advertisement belongs to the limited information of needs, The advertisement is then shielded on shopping webpage, it is necessary to which explanation, after user inputs evaluation information in shopping webpage, the evaluation is believed Breath is not to be immediately displayed in shopping webpage, but needs algorithm model to be measured to judge the evaluation information, determines it It is not advertisement, when being normal evaluation content, the evaluation information is included on shopping webpage, if algorithm model to be measured judges to be somebody's turn to do Evaluation information is advertisement, then it is shielded, and makes not showing the advertisement on shopping webpage.

Step S704, according to the target information shielded, training sample set is established；

The present embodiment builds training sample set with the advertisement shielded, trains algorithm model to be measured with training sample set, i.e., Algorithm model evaluating to be measured is fed back into algorithm model to be measured for the result of advertisement, method of determining and calculating model is treated and is trained so that The detection process of algorithm model to be measured forms a closed loop procedure.

In addition, the training sample set also includes the first data source and the second data source, first data source includes selling The report information of family report buyer and/or the calling information of Buyer Complaint seller, second data source include identified in advance Information.System 41, malice buyer prosecution system 42, limited letter are complained for example, first data source is seller as shown in Figure 2 Cease prosecution system 44, the report information and/or calling information that artificial customer service complaint system 45 provides, the second data source is that advertisement is lifted The information identified in advance that reporting system 43 provides, the forming process of identified information is as follows in advance：By a large amount of text informations, Image information is distributed to reviewer, and text information, image information are manually recognized by the reviewer, judge the word Whether information, image information are advertisement, and the reviewer can be specifically user, students etc..

Step S705, the algorithm model to be measured is trained by the training sample set.

It is information identified in advance to concentrate the training sample that includes due to training sample, such as advertising message, non-wide Information, report information, calling information are accused, algorithm model to be measured is trained by the training sample set, algorithm model to be measured can be made such as Cognitive Mode with the mankind equally identifies which type of information is advertisement, and which type of information is non-advertisement, therefore, trains sample The training sample of this concentration is more complete, more accurate, and the recognition capability of the algorithm model to be measured trained is more accurate.

In addition, step S701- steps S705 can also be on the basis of embodiment illustrated in fig. 6, if step S401 is determined The step of algorithm model to be measured need not stop after iteration.

In the present embodiment, by the way that algorithm model evaluating to be measured is fed back into algorithm model to be measured for the result of advertisement, treat Method of determining and calculating model is trained so that the detection process of algorithm model to be measured forms a closed loop procedure, improves to be measured The detection precision of method model, in addition, being sold with advertisement, the report information of seller report buyer and/or the Buyer Complaint shielded The calling information of family, the artificial information identified in advance build training sample set, ensure that the training sample that training sample is concentrated More complete, more accurate, the recognition capability of the algorithm model to be measured trained by training sample set is more accurate.

The algorithm model detection means of one or more embodiments described in detail below according to the application.These algorithms Model inspection device can be implemented in the architecture of the vehicles or mobile terminal, can also be implemented in server and In the interactive system of client.It will be understood by those skilled in the art that these algorithm model detection means can be used it is commercially available The step of nextport hardware component NextPort is instructed by this programme is configured to form.For example, processor module (or processing module, processing Unit) can use from Texas Instruments, Intel company, ARM companies, etc. enterprise single-chip microcomputer, microcontroller, Wei Chu Manage the components such as device.

Figure 11 is the structural representation for the algorithm model detection means that the embodiment of the present application one provides, as shown in figure 11, should Device includes：First acquisition module 11, the second acquisition module 12, the first determining module 13, the second determining module 14.

First acquisition module 11, it is corresponding for obtaining first sample collection corresponding to algorithm model to be measured and history algorithm model The second sample set in identical target sample, the history algorithm model is the preceding an iteration mould of the algorithm model to be measured Type；

Second acquisition module 12, for obtaining what is obtained after the algorithm model to be measured is handled the target sample First results set, and the history algorithm model target sample is handled after obtained the second results set；

First determining module 13, for according to first results set and second results set, determining the first collection Close, the first set is that the mesh of same treatment result is corresponding with first results set and second results set The set of this composition of standard specimen；

Second determining module 14, for the number of the target sample in the first set, and first knot The number of target sample in fruit set, determine the accuracy of the algorithm model to be measured.

Figure 11 shown devices can perform the algorithm model detection method described in embodiment illustrated in fig. 3, its realization principle and Technique effect repeats no more.

Figure 12 is the structural representation for the algorithm model detection means that the embodiment of the present application two provides, as shown in figure 12, On the basis of embodiment illustrated in fig. 11, the first determining module 13 is specifically used for according to first results set, determines described first Target sample corresponding to the first result in results set；According to second results set, second results set is determined In the first result corresponding to target sample；Determine target sample and institute corresponding to the first result in first results set The common factor for stating target sample corresponding to the first result in the second results set is the first set.

Second determining module 14 includes computing unit 141, determining unit 142.

Computing unit 141, for calculating the number of target sample and institute corresponding to the first result in the first set State the first ratio of the number of target sample corresponding to the first result in the first results set；

Determining unit 142, for determining the ratio of first ratio and attenuation coefficient as the algorithm model to be measured Accuracy, the attenuation coefficient is equal to the number of target sample corresponding to the first result in the 3rd results set and the 4th result The ratio of the number of target sample corresponding to the first result in set, the 3rd results set are the history algorithm models The results set obtained after handling the second sample in second sample set, the 4th results set is described treats The results set that the first sample that method of determining and calculating model is concentrated to the first sample obtains after handling.

Figure 12 shown devices can perform Fig. 4, the algorithm model detection method described in 5 illustrated embodiments, its realization principle Repeated no more with technique effect.

Figure 13 is the structural representation for the algorithm model detection means that the embodiment of the present application three provides, as shown in figure 13, On the basis of embodiment illustrated in fig. 12, algorithm model detection means also includes the 3rd determining module 15, and the 3rd determining module 15 is used for According to the accuracy of the algorithm model to be measured and the accuracy of the history algorithm model, determine that the algorithm model to be measured is It is no to need to stop iteration.

Specifically, the 3rd determining module 15 is used for when the accuracy of the algorithm model to be measured is more than or equal to the history During the accuracy of algorithm model, determine that the algorithm model to be measured need not stop iteration；When the algorithm model to be measured just When true rate is less than the accuracy of the history algorithm model, determine that the algorithm model to be measured needs to stop iteration.

In addition, the first determining module 13 is additionally operable to according to first results set and second results set, it is determined that Second set, the second set are to be corresponding with different disposal knot in first results set and second results set The set that the target sample of fruit is formed；Second determining module 14 is additionally operable to of the target sample in the second set Number, and the number of the target sample in second results set, determine the error rate of the algorithm model to be measured.

Alternatively, the first determining module 13 is specifically used for according to first results set, determines first result set Target sample corresponding to the first result in conjunction；According to second results set, in second results set is determined Target sample corresponding to two results；Determine target sample and described second corresponding to the first result in first results set The common factor of target sample corresponding to the second result in results set is the second set.

Alternatively, the second determining module 14 is specifically used for the number for determining the target sample in the second set and described The ratio of the number of target sample corresponding to the second result in second results set is the error rate of the algorithm model to be measured.

In addition, the 3rd determining module 15 is additionally operable to the error rate according to the algorithm model to be measured and the history algorithm mould The error rate of type, determines whether the algorithm model to be measured needs to stop iteration.

Alternatively, the 3rd determining module 15 is specifically used for the error rate when the algorithm model to be measured less than or equal to described During the error rate of history algorithm model, determine that the algorithm model to be measured need not stop iteration；When the algorithm model to be measured Error rate be more than the history algorithm model error rate when, determine the algorithm model to be measured need stop iteration.

Figure 13 shown devices can perform the algorithm model detection method described in the illustrated embodiment of Fig. 6,7,8, and it realizes former Reason and technique effect repeat no more.

Figure 14 is the structural representation for the algorithm model detection means that the embodiment of the present application four provides, as shown in figure 14, On the basis of embodiment illustrated in fig. 13, algorithm model detection means also include memory module 16, evaluation and test module 17, shroud module 18, Training sample set establishes module 19, training module 20.

Memory module 16, for storing the target information to be evaluated.

Evaluation and test module 17, for carrying out evaluating and testing or commenting offline in real time to the target information by the algorithm model to be measured Survey.

Shroud module 18, for when the target information to be evaluated is to need limited information, shielding the target Information.

Training sample set establishes module 19, for according to the target information shielded, establishing training sample set.

Training module 20, for training the algorithm model to be measured by the training sample set.

In addition, the training sample set also includes the first data source and the second data source, first data source includes selling The report information of family report buyer and/or the calling information of Buyer Complaint seller, second data source include identified in advance Information.

Figure 14 shown devices can perform the algorithm model detection method described in embodiment illustrated in fig. 9, its realization principle and Technique effect repeats no more.

On the algorithm model detection means in above-described embodiment, wherein modules, unit performs the specific side of operation Formula is described in detail in the embodiment about this method, and explanation will be not set forth in detail herein.

The built-in function and structure of algorithm model detection means are the foregoing described, as shown in figure 15, in practice, the algorithm mould Type detection means can be realized as detection device, and the detection device can be terminal device or server, the detection device Including：Memory and processor.

Figure 16 is the structural representation for the terminal device that another embodiment of the application provides, as shown in figure 16.Reference picture 16, Terminal device 2000 can include following one or more assemblies：Processing component 2002, memory 2004, power supply module 2006, Multimedia groupware 2008, audio-frequency assembly 2010, input/output (I/O) interface 2012, sensor cluster 2014, and communication set Part 2016.

Processing component 2002 generally controls the integrated operation of terminal device 2000, such as leads to display, call, data The operation that letter, camera operation and record operation are associated.Processing component 2002 can include one or more processors 2020 Execute instruction, to complete above-mentioned method.In addition, processing component 2002 can include one or more modules, it is easy to treatment group Interaction between part 2002 and other assemblies.For example, processing component 2002 can include multi-media module, to facilitate multimedia group Interaction between part 2008 and processing component 2002.

Memory 2004 is configured as storing various types of data to support the operation in terminal device 2000.These numbers According to example include being used for the instruction of any application program or method operated on terminal device 2000, contact data, electricity Talk about book data, message, picture, video etc..Memory 2004 can be by any kind of volatibility or non-volatile memory device Or combinations thereof is realized, such as static RAM (SRAM), Electrically Erasable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory EPROM (EPROM), programmable read only memory (PROM), read-only storage (ROM), magnetic memory, flash memory, disk or CD.

Power supply module 2006 provides electric power for the various assemblies of terminal device 2000.Power supply module 2006 can include power supply Management system, one or more power supplys, and other groups associated with generating, managing and distributing electric power for terminal device 2000 Part.

Multimedia groupware 2008 is included in the screen of one output interface of offer between terminal device 2000 and user. In some embodiments, screen can include liquid crystal display (LCD) and touch panel (TP).If screen includes touch panel, Screen may be implemented as touch-screen, to receive the input signal from user.Touch panel includes one or more touch and passed Sensor is with the gesture on sensing touch, slip and touch panel.The touch sensor can not only sensing touch or slip be dynamic The border of work, but also detect the duration and pressure related to the touch or slide.In certain embodiments, it is more Media component 2008 includes a front camera and/or rear camera.When terminal device 2000 is in operator scheme, such as clap When taking the photograph pattern or video mode, front camera and/or rear camera can receive outside multi-medium data.It is each preposition Camera and rear camera can be a fixed optical lens system or have focusing and optical zoom capabilities.

Audio-frequency assembly 2010 is configured as output and/or input audio signal.For example, audio-frequency assembly 2010 includes a wheat Gram wind (MIC), when terminal device 2000 is in operator scheme, during such as call model, logging mode and speech recognition mode, Mike Wind is configured as receiving external audio signal.The audio signal received can be further stored in memory 2004 or via Communication component 2016 is sent.In certain embodiments, audio-frequency assembly 2010 also includes a loudspeaker, for exporting audio letter Number.

Input/output interface 2012 provides interface between processing component 2002 and peripheral interface module, and above-mentioned periphery connects Mouth mold block can be keyboard, click wheel, button etc..These buttons may include but be not limited to：Home button, volume button, startup Button and locking press button.

Sensor cluster 2014 includes one or more sensors, for providing the shape of various aspects for terminal device 2000 State is assessed.For example, sensor cluster 2014 can detect opening/closed mode of terminal device 2000, component it is relatively fixed Position, such as the display and keypad that the component is terminal device 2000, sensor cluster 2014 can be set with detection terminal Position for 2000 or 2,000 1 components of terminal device changes, the existence or non-existence that user contacts with terminal device 2000, The temperature change of the orientation of terminal device 2000 or acceleration/deceleration and terminal device 2000.Sensor cluster 2014 can include connecing Nearly sensor, it is configured to detect the presence of object nearby in no any physical contact.Sensor cluster 2014 is also Optical sensor can be included, such as CMOS or ccd image sensor, for being used in imaging applications.In certain embodiments, should Sensor cluster 2014 can also include acceleration transducer, and gyro sensor, Magnetic Sensor, pressure sensor or temperature pass Sensor.

Communication component 2016 is configured to facilitate the logical of wired or wireless way between terminal device 2000 and other equipment Letter.Terminal device 2000 can access it is wireless airborne based on communication standard, such as WiFi, 2G or 3G, or combinations thereof.One In individual exemplary embodiment, communication component 2016 via broadcast channel receive broadcast singal from external broadcasting management system or Broadcast related information.In one exemplary embodiment, the communication component 2016 also includes near-field communication (NFC) module, with Promote junction service.For example, radio frequency identification (RFID) technology can be based in NFC module, Infrared Data Association (IrDA) technology, surpass Broadband (UWB) technology, bluetooth (BT) technology and other technologies are realized.

In the exemplary embodiment, terminal device 2000 can by one or more application specific integrated circuits (ASIC), Digital signal processor (DSP), digital signal processing appts (DSPD), PLD (PLD), field-programmable gate array Arrange (FPGA), controller, microcontroller, microprocessor or other electronic components to realize, for performing the above method.

In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instructing, example are additionally provided Such as include the memory 2004 of instruction, above-mentioned instruction can be performed by the processor 2020 of terminal device 2000 to complete above-mentioned side Method.For example, the non-transitorycomputer readable storage medium can be by any kind of volatibility or non-volatile memories Equipment or combinations thereof are realized, such as static RAM (SRAM), Electrically Erasable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory EPROM (EPROM), programmable read only memory (PROM), read-only storage (ROM), magnetic memory, flash memory, disk or CD.

A kind of non-transitorycomputer readable storage medium, when the instruction in the storage medium is by the processing of terminal device When device performs so that terminal device is able to carry out a kind of algorithm model detection method, and methods described includes：

Finally it should be noted that：Various embodiments above is only to illustrate the technical scheme of the application, rather than its limitations；To the greatest extent The application is described in detail with reference to foregoing embodiments for pipe, it will be understood by those within the art that：Its according to The technical scheme described in foregoing embodiments can so be modified, either which part or all technical characteristic are entered Row equivalent substitution；And these modifications or replacement, the essence of appropriate technical solution is departed from each embodiment technology of the application The scope of scheme.

Claims

A kind of 1. information evaluating system, it is characterised in that including：

Evaluation and test module, for being evaluated and tested by algorithm model to information to be evaluated, the algorithm model includes algorithm to be measured Model and history algorithm model, the history algorithm model are a preceding iterative models for the algorithm model to be measured；

Algorithm model detection module, for obtaining corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model Identical target sample in second sample set；Obtain what is obtained after the algorithm model to be measured is handled the target sample First results set, and the history algorithm model target sample is handled after obtained the second results set； According to first results set and second results set, first set is determined, the first set is described first The set that the target sample of same treatment result is formed is corresponding with results set and second results set；According to described The number of target sample in one set, and the number of the target sample in first results set, are determined described to be measured The accuracy of algorithm model；

Algorithm model training module, for training the algorithm model according to training sample set.
2. system according to claim 1, it is characterised in that the system also includes：

Display module, for showing User Interface；

Memory, for storing the information to be evaluated in the User Interface, the information to be evaluated includes as follows At least one information：The information of user's input, URL, pictorial information.
3. system according to claim 2, it is characterised in that the system also includes：

Data processing platform (DPP), for obtaining the information to be evaluated from the memory, so that the evaluation and test module is to institute Information to be evaluated is stated to be evaluated and tested.
4. system according to claim 3, it is characterised in that the system also includes：

Management and control module, for when the evaluation and test module judges the information to be evaluated to need limited information, controlling The display module shielding is described to need limited information.
5. system according to claim 4, it is characterised in that the system also includes：

Algorithm model iteration control module, for the algorithm model to be measured that is detected according to the algorithm model detection module just True rate and/or error rate, control whether the algorithm model training module needs to stop algorithm model described in iteration.
6. according to the system described in claim any one of 1-5, it is characterised in that the training sample bag that the training sample is concentrated Include following at least one：It is described to need limited information, seller to report report information, the letter of complaint of Buyer Complaint seller of buyer Breath, information identified in advance.
7. according to the system described in claim any one of 1-5, it is characterised in that the information to be evaluated is evaluated including user Information；

The evaluation and test module is specifically used for evaluating and testing user's evaluation information by algorithm model, determines that the user comments Whether valency information is advertising message.
A kind of 8. user's evaluation information detection method, it is characterised in that including：

Obtain multiple user's evaluation informations；

Differentiated to obtain the first results set respectively to the multiple user's evaluation information by algorithm model to be measured, described One results set includes the first differentiation result obtained after the algorithm model to be measured differentiates to each user's evaluation information；

Differentiated to obtain the second results set respectively to the multiple user's evaluation information by history algorithm model, described Two results sets include the second differentiation result obtained after the history algorithm model differentiates to each user's evaluation information；

According to first results set and second results set, first set is determined, the first set is described The set that user's evaluation information of identical differentiation result is formed is corresponding with first results set and second results set；

The number of user's evaluation information in the first set, and the number of the multiple user's evaluation information, really The accuracy of the fixed algorithm model to be measured；

According to the accuracy of the algorithm model to be measured, determine that the multiple user's evaluation information is corresponding respectively and differentiate result；

Wherein, the history algorithm model is a preceding iterative model for the algorithm model to be measured.
9. according to the method for claim 8, it is characterised in that described that the multiple user is commented by algorithm model to be measured Valency information differentiated respectively, including：

Whether it is by restricted information by each user's evaluation information of algorithm Model checking to be measured；

It is described that the multiple user's evaluation information is differentiated respectively by history algorithm model, including：

Differentiate whether each user's evaluation information is by restricted information by the history algorithm model；

Wherein, it is described that advertising message is included by restricted information.
10. according to the method for claim 9, it is characterised in that the first set is institute in first results set State algorithm model to be measured and the history algorithm model is determined as the set that user's evaluation information of advertising message is formed；Or

The first set is that algorithm model to be measured described in first results set and the history algorithm model differentiate The set formed for user's evaluation information of non-advertising message.
A kind of 11. algorithm model detection method, it is characterised in that including：

Obtain identical mesh in the second sample set corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model Standard specimen sheet, the history algorithm model are a preceding iterative models for the algorithm model to be measured；

The first results set obtained after the algorithm model to be measured is handled the target sample is obtained, and described is gone through The second results set that history algorithm model obtains after handling the target sample；

According to first results set and second results set, first set is determined, the first set is described The set that the target sample of same treatment result is formed is corresponding with first results set and second results set；

Of target sample in the number of target sample in the first set, and first results set Number, determine the accuracy of the algorithm model to be measured.
12. according to the method for claim 11, it is characterised in that described according to first results set and described second Results set, first set is determined, including：

According to first results set, target sample corresponding to the first result in first results set is determined；

According to second results set, target sample corresponding to the first result in second results set is determined；

Determine first in target sample and second results set corresponding to the first result in first results set As a result the common factor of corresponding target sample is the first set.
13. according to the method for claim 12, it is characterised in that the target sample in the first set The number of target sample in number, and first results set, the accuracy of the algorithm model to be measured is determined, wrapped Include：

Calculate the in the number of target sample corresponding to the first result in the first set and first results set First ratio of the number of target sample corresponding to one result；

The ratio for determining first ratio and attenuation coefficient is the accuracy, the attenuation coefficient etc. of the algorithm model to be measured The number of target sample is corresponding with the first result in the 4th results set corresponding to the first result in the 3rd results set Target sample number ratio, the 3rd results set is the history algorithm model in second sample set The results set that second sample obtains after being handled, the 4th results set are the algorithm models to be measured to described first The results set that first sample in sample set obtains after being handled.
14. according to the method described in claim any one of 11-13, it is characterised in that described to determine the algorithm model to be measured Accuracy after, in addition to：

According to the accuracy of the algorithm model to be measured and the accuracy of the history algorithm model, the algorithm mould to be measured is determined Whether type needs to stop iteration.
15. according to the method for claim 14, it is characterised in that the accuracy according to the algorithm model to be measured and The accuracy of the history algorithm model, determines whether the algorithm model to be measured needs to stop iteration, including：

If the accuracy of the algorithm model to be measured is more than or equal to the accuracy of the history algorithm model, it is determined that described to treat Method of determining and calculating model need not stop iteration；

If the accuracy of the algorithm model to be measured is less than the accuracy of the history algorithm model, it is determined that the algorithm to be measured Model needs to stop iteration.
16. according to the method described in claim any one of 11-13, it is characterised in that also include：

According to first results set and second results set, second set is determined, the second set is described The set that the target sample of different disposal result is formed is corresponding with first results set and second results set；

Of target sample in the number of target sample in the second set, and second results set Number, determine the error rate of the algorithm model to be measured.
17. according to the method for claim 16, it is characterised in that described according to first results set and described second Results set, determines second set, and the second set is right in first results set and second results set There should be the set that the target sample of different disposal result is formed, including：

According to first results set, target sample corresponding to the first result in first results set is determined；

According to second results set, target sample corresponding to the second result in second results set is determined；

Determine second in target sample and second results set corresponding to the first result in first results set As a result the common factor of corresponding target sample is the second set.
18. according to the method for claim 17, it is characterised in that the target sample in the second set The number of target sample in number, and second results set, the error rate of the algorithm model to be measured is determined, wrapped Include：

Determine mesh corresponding to the second result in the number and second results set of the target sample in the second set The ratio of the number of standard specimen sheet is the error rate of the algorithm model to be measured.
19. according to the method for claim 18, it is characterised in that the error rate for determining the algorithm model to be measured it Afterwards, in addition to：

According to the error rate of the algorithm model to be measured and the error rate of the history algorithm model, the algorithm mould to be measured is determined Whether type needs to stop iteration.
20. according to the method for claim 19, it is characterised in that the error rate according to the algorithm model to be measured and The error rate of the history algorithm model, determines whether the algorithm model to be measured needs to stop iteration, including：

If the error rate of the algorithm model to be measured is less than or equal to the error rate of the history algorithm model, it is determined that described to treat Method of determining and calculating model need not stop iteration；

If the error rate of the algorithm model to be measured is more than the error rate of the history algorithm model, it is determined that the algorithm to be measured Model needs to stop iteration.
21. the method according to claim 15 or 20, it is characterised in that described to determine that the algorithm model to be measured After stopping iteration, in addition to：

Obtain and store target information to be evaluated；

Evaluation and test in real time or offline evaluation and test are carried out to the target information by the algorithm model to be measured.
22. according to the method for claim 21, it is characterised in that it is described by the algorithm model to be measured to the target After information carries out evaluation and test in real time or offline evaluation and test, in addition to：

If the target information to be evaluated is to need limited information, the target information is shielded.
23. according to the method for claim 22, it is characterised in that after the shielding target information, in addition to：

According to the target information shielded, training sample set is established；

The algorithm model to be measured is trained by the training sample set.
24. according to the method for claim 23, it is characterised in that the target sample includes user's evaluation information, described First result includes advertising message, and second result includes non-advertising message；

The training sample set also includes the first data source and the second data source, and first data source includes seller and reports buyer Report information and/or Buyer Complaint seller calling information, second data source includes information identified in advance.
A kind of 25. algorithm model detection means, it is characterised in that including：

First acquisition module, for obtaining second corresponding to first sample collection corresponding to algorithm model to be measured and history algorithm model Identical target sample in sample set, the history algorithm model are a preceding iterative models for the algorithm model to be measured；

Second acquisition module, for obtaining the first knot obtained after the algorithm model to be measured is handled the target sample Fruit set, and the history algorithm model target sample is handled after obtained the second results set；

First determining module, it is described for according to first results set and second results set, determining first set First set is that the target sample of same treatment result is corresponding with first results set and second results set The set of composition；

Second determining module, for the number of the target sample in the first set, and first results set In target sample number, determine the accuracy of the algorithm model to be measured.
26. device according to claim 25, it is characterised in that first determining module is specifically used for according to described the One results set, determine target sample corresponding to the first result in first results set；

According to second results set, target sample corresponding to the first result in second results set is determined；

Determine first in target sample and second results set corresponding to the first result in first results set As a result the common factor of corresponding target sample is the first set.
27. device according to claim 26, it is characterised in that second determining module includes：

Computing unit, for calculating the number of target sample corresponding to the first result in the first set and first knot First ratio of the number of target sample corresponding to the first result in fruit set；

Determining unit, the ratio for determining first ratio and attenuation coefficient are the accuracy of the algorithm model to be measured, The attenuation coefficient is equal in the number of target sample and the 4th results set corresponding to the first result in the 3rd results set The first result corresponding to target sample number ratio, the 3rd results set is the history algorithm model to described The results set that the second sample in second sample set obtains after being handled, the 4th results set are the algorithms to be measured The results set that the first sample that model is concentrated to the first sample obtains after handling.
28. according to the device described in claim any one of 25-27, it is characterised in that also include：

3rd determining module, for the accuracy according to the algorithm model to be measured and the accuracy of the history algorithm model, Determine whether the algorithm model to be measured needs to stop iteration.
29. device according to claim 28, it is characterised in that first acquisition module is additionally operable to obtain to be evaluated Target information；

Described device also includes：

Memory module, for storing the target information to be evaluated；

Evaluation and test module, for carrying out evaluation and test in real time or offline evaluation and test to the target information by the algorithm model to be measured.
30. device according to claim 29, it is characterised in that also include：

Shroud module, for when the target information to be evaluated is to need limited information, shielding the target information.
31. device according to claim 30, it is characterised in that also include：

Training sample set establishes module, for according to the target information shielded, establishing training sample set；

Training module, for training the algorithm model to be measured by the training sample set.
A kind of 32. detection device, it is characterised in that including：Memory and processor；

The memory and processor coupling, the memory are used to store first sample collection corresponding to algorithm model to be measured With history algorithm model corresponding to the second sample set, the history algorithm model is the preceding an iteration of the algorithm model to be measured Model；And the storage algorithm model to be measured the target sample is handled after obtained the first results set and described The second results set that history algorithm model obtains after handling the target sample；

The processor is used to determine identical target sample in the first sample collection and second sample set；According to described First results set and second results set, determine first set, and the first set is in first results set The set formed with the target sample that same treatment result is corresponding with second results set；According in the first set Target sample number, and the number of the target sample in first results set determines the algorithm model to be measured Accuracy.