CN111754126A - Method and system for evaluating applications - Google Patents

Method and system for evaluating applications Download PDF

Info

Publication number
CN111754126A
CN111754126A CN202010603359.3A CN202010603359A CN111754126A CN 111754126 A CN111754126 A CN 111754126A CN 202010603359 A CN202010603359 A CN 202010603359A CN 111754126 A CN111754126 A CN 111754126A
Authority
CN
China
Prior art keywords
evaluation
data
sample
indexes
score
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010603359.3A
Other languages
Chinese (zh)
Inventor
雷徽
张多坤
杨进
戚立才
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
AlipayCom Co ltd
Original Assignee
Alipay Hangzhou Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alipay Hangzhou Information Technology Co Ltd filed Critical Alipay Hangzhou Information Technology Co Ltd
Priority to CN202010603359.3A priority Critical patent/CN111754126A/en
Publication of CN111754126A publication Critical patent/CN111754126A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Strategic Management (AREA)
  • Data Mining & Analysis (AREA)
  • Economics (AREA)
  • Educational Administration (AREA)
  • Physics & Mathematics (AREA)
  • Development Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Artificial Intelligence (AREA)
  • Game Theory and Decision Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The method and the system for evaluating the application provided by the specification evaluate the target application through an evaluation model. The evaluation model can fuse the subjective score of the user with an objective score obtained according to the click rate to obtain the target score applied by the sample, and calculate and select L evaluation indexes meeting a preset relation according to the correlation relation between the candidate evaluation indexes and the target score; and training an evaluation model by taking the feature data corresponding to the L evaluation indexes applied to the sample as training data and the target indexes applied to the sample as training targets to obtain weights corresponding to the L evaluation indexes. The method and the system can fit the subjective score value and the objective score value to obtain the weight of each evaluation index, so that the evaluation model can evaluate the target application by combining subjective feeling and objective score, and the evaluation result is more comprehensive and objective.

Description

Method and system for evaluating applications
Technical Field
The present disclosure relates to the field of internet technologies, and in particular, to a method and a system for evaluating an application.
Background
With the rapid development of information technology and internet technology, more and more users choose to transact business online. As online transactions scale up, more and more services are provided by individual APPs. For example, a merchant or an APP platform may provide various applets, life numbers, service numbers, etc. in an APP to provide a variety of services for a user. In order to improve the experience of the user, various services provided in the APP need to be scored, excellent services are supported, and poor services are supervised, so that the overall quality of the services is improved, and the adhesion degree of the user is improved. When a service is scored, multiple scoring indexes are generally required to be established for evaluation from multiple dimensions. In the scoring system in the prior art, when a scoring model is constructed, an expert usually gives an evaluation index and assigns the weight of each evaluation index, so that the subjectivity is strong, and a scoring result may have a certain deviation.
Therefore, a method and system for evaluating applications more comprehensively and objectively is needed.
Disclosure of Invention
The present specification provides a method and system that can more comprehensively and objectively evaluate an application.
In a first aspect, the present specification provides a method of evaluating an application, comprising: obtaining an evaluation model configured to evaluate a target application based on L evaluation indicators, comprising: receiving evaluation data of the target application, wherein the evaluation data comprises feature data corresponding to the L evaluation indexes, and L is an integer greater than 1; performing data processing on the evaluation data to obtain an evaluation result of the target application; and outputting the evaluation result, wherein the evaluation model is obtained based on evaluation data of a sample set and target score training, the sample set comprises a plurality of sample applications, the target score comprises a combination of a first score value and a second score value, the first score value represents a subjective score value of the sample application by a user, and the second score value represents an objective score value of the sample application; acquiring evaluation data of the target application; and inputting the evaluation data of the target application into the evaluation model to obtain an evaluation result of the target application.
In some embodiments, the first score value comprises an expert score value of the sample application, and the second score value comprises a click rate score value of the sample application.
In some embodiments, each of the L evaluation indexes includes at least one sub-evaluation index, and the L evaluation indexes includes M sub-evaluation indexes, where M is an integer equal to or greater than L.
In some embodiments, the performing data processing on the evaluation data to obtain an evaluation result of the target application includes: performing the data processing on the evaluation data of at least one sub-evaluation index in each evaluation index to obtain a combined evaluation value corresponding to each evaluation index; and performing the data processing on the L combined evaluation values corresponding to the L evaluation indexes to generate the evaluation result of the target application.
In some embodiments, the training of the assessment model comprises: obtaining a target score applied by each sample in the sample set; determining the L evaluation indicators of the evaluation model; and training the evaluation model by using the evaluation data applied to each sample in the sample set as training data and using the target score applied to each sample in the sample set as a training target.
In some embodiments, the obtaining the target score for each sample application in the sample set comprises: obtaining the first scoring value and the second scoring value applied by each sample in the sample set; and performing feature fusion calculation on the first scoring value and the second scoring value to obtain a target score applied to each sample in the sample set.
In some embodiments, the feature fusion calculation comprises a weighted sum calculation.
In some embodiments, the determining the L evaluation indicators of the evaluation model comprises: determining P candidate evaluation indexes, wherein P is an integer greater than or equal to L; and selecting the L evaluation indexes from the P candidate evaluation indexes according to feature screening conditions based on the correlation between the P candidate evaluation indexes and the target score.
In some embodiments, selecting the L evaluation indicators from the P candidate evaluation indicators includes: acquiring feature data of the P candidate evaluation indexes applied to each sample in the sample set, and constructing P candidate evaluation data sets corresponding to the P candidate evaluation indexes; calculating a correlation of each of the P candidate evaluation metrics to the target score; and selecting the L evaluation indexes with the correlation meeting a preset threshold value from the P candidate evaluation indexes.
In some embodiments, the selecting, from the P candidate evaluation indicators, the L evaluation indicators having a correlation satisfying a preset threshold includes: selecting Q candidate evaluation indexes with correlation meeting a preset threshold value from the P candidate evaluation indexes, wherein Q is an integer which is less than or equal to P and is greater than or equal to M; calculating the linear relation between every two of the Q candidate evaluation indexes; combining or deleting one of the candidate characteristic evaluation indexes with the linear relation larger than a preset linear threshold value to obtain the M sub-evaluation indexes; and dividing the M sub-evaluation indexes into the L evaluation indexes.
In some embodiments, training the evaluation model by using the evaluation data applied to each sample in the sample set as training data and the target score applied to each sample in the sample set as training targets comprises: obtaining evaluation data applied to all samples in the sample set to generate a training data set; performing feature processing on the training data set by using feature engineering, wherein the feature engineering comprises binning; based on the result of the feature processing, taking the training data set as training data, taking a target score applied by each sample in the sample set as a training target, and training the evaluation model; and obtaining the weights corresponding to the L evaluation indexes.
In some embodiments, the weights corresponding to the L evaluation indicators include: a weight corresponding to each of the L evaluation indexes; and the weight corresponding to each sub-evaluation index in the M sub-evaluation indexes.
In some embodiments, the feature processing the training data set using feature engineering comprises: dividing evaluation data in the training data set into M evaluation data sets corresponding to the M sub-evaluation indexes; performing binning processing on each evaluation data set in the M evaluation data sets to obtain N bins, wherein N is an integer greater than 1; and determining an evaluation value corresponding to each bin.
In a second aspect, the present specification provides a system for evaluating an application, comprising at least one storage medium and at least one processor, the at least one storage medium comprising at least one set of instructions for evaluating an application; and the at least one processor, communicatively coupled to the at least one storage medium, wherein when the system is operating, the at least one processor reads the at least one instruction set and performs the method of application evaluation described in the first aspect of the specification as indicated by the at least one instruction set.
According to the technical scheme, the method and the system for evaluating the application, which are provided by the specification, evaluate the target application through the evaluation model. The evaluation model can fuse the subjective score of the user with an objective score obtained according to the click rate to obtain a target score applied by the sample, and calculate and select L evaluation indexes meeting a preset relation according to the correlation relation between the candidate evaluation indexes and the target score; and training the evaluation model by taking the characteristic data corresponding to the L evaluation indexes applied to the sample as training data and the target score applied to the sample as a training target to obtain the weights corresponding to the L evaluation indexes. The method and the system can fit the subjective score value and the objective score value to obtain the weight of each evaluation index, so that the evaluation model can evaluate the target application by combining subjective feeling and objective score, and the evaluation result is more comprehensive and objective.
Additional features of the methods and systems for evaluating applications provided herein will be set forth in part in the description which follows. The following numerical and exemplary descriptions will be readily apparent to those of ordinary skill in the art in view of the description. The inventive aspects of the method, system, and storage medium for evaluating applications provided in this specification can be fully explained by the practice or use of the methods, apparatus, and combinations described in the detailed examples below.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram illustrating an application scenario of a system for evaluating an application provided in accordance with an embodiment of the present description;
FIG. 2 illustrates a schematic diagram of an apparatus for evaluating an application provided in accordance with an embodiment of the present description;
FIG. 3 illustrates a flow diagram of a method of evaluating an application provided in accordance with an embodiment of the present description;
FIG. 4 illustrates a schematic structural diagram of an evaluation model provided in accordance with an embodiment of the present description; and
fig. 5 is a flowchart illustrating a training method of an evaluation model provided in accordance with an embodiment of the present disclosure.
Detailed Description
The following description is presented to enable any person skilled in the art to make and use the present description, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present description. Thus, the present description is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.
The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. For example, as used herein, the singular forms "a", "an" and "the" may include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," and/or "including," when used in this specification, are intended to specify the presence of stated integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
These and other features of the present specification, as well as the operation and function of the elements of the structure related thereto, and the combination of parts and economies of manufacture, may be particularly improved upon in view of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the specification. It should also be understood that the drawings are not drawn to scale.
The flow diagrams used in this specification illustrate the operation of system implementations according to some embodiments of the specification. It should be clearly understood that the operations of the flow diagrams may be performed out of order. Rather, the operations may be performed in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.
With the increasing application services provided by each APP, in order to improve the experience of the user in using the APP, a more comprehensive, objective and scientific evaluation system is needed to evaluate various applications. On one hand, the existing application can be hierarchically optimized through the evaluation result of the evaluation system, so that higher flow is given to excellent application, and application with poor evaluation result is supervised; on the other hand, the evaluation of the user on the application can be obtained through the evaluation system, the interaction between the user and a merchant providing the application service is promoted, the merchant is supervised to promote the overall level of the application service, and the user experience and the user adhesion are promoted.
The method and the system for evaluating the application provided by the specification fuse the subjective score of the user with the objective score obtained according to the click rate to obtain the target score of the application, use the feature data corresponding to a plurality of evaluation indexes of the application as the training data, use the target score of the application as the training target, train the evaluation model, and obtain the weight corresponding to each evaluation index. The evaluation model provided by the method and the system provided by the specification can be used for evaluating the application service by combining subjective feeling and objective scoring, so that the evaluation result is more comprehensive, objective and scientific.
Fig. 1 shows a schematic diagram of an application scenario of a system 100 for evaluating an application. System 100 may include server 200, client 300, network 120, and database 150.
Server 200 may store data or instructions for performing the methods of evaluating an application described herein and may execute or be used to execute the data and/or instructions. The server 200 may be a device that provides a corresponding service for the APP of the client 300. The APP may include, but is not limited to: chat-type APP program, shopping-type APP program, video-type APP program, financing-type APP program, etc., such as Payment treasureTMTaobao medicineTMJingdongTMAnd the like. The server 200 provides a service corresponding to the APP. The server 200 may be the same device, may be different devices, may be a single device, or may be a system composed of a plurality of devices. Server 200 may provide a variety of services to user 110. Each service may be provided by a different applet. For example, Payment treasureTMThe APP may provide a variety of services to the user 110, such as "hungry" applet may provide take-away services, "drip travel" may provide taxi taking services, and so on.
User 110 is a user 110 of client 300. The client 300 may be a device that interacts with the server 200. For example, the client 300 may be a smart device loaded with a target APP. The client 300 may be communicatively coupled to the server 200. In some embodiments, the client 300 may have one or more Applications (APPs) installed. The APP can provide the user 110 with the ability to interact with the outside world and an interface over the network 120. In some embodiments, the client 300 may include a mobile device 300-1, a tablet computer 300-2, a laptop computer 300-3, an in-built device of a motor vehicle 300-4, or the like, or any combination thereof. In some embodiments, mobile device 300-1 may include a smart home device, a smart mobile device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart television, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smartphone, personal digital assistant, or the like, or any combination thereof. In some embodiments, the client 300 may be a device with location technology for locating the location of the client 300.
Network 120 may facilitate the exchange of information and/or data. As shown in fig. 1, client 300, server 200, database 150 may be connected to network 120 and communicate information and/or data with each other via network 120. For example, client 300 may obtain services from server 200 via network 120. In some embodiments, the network 120 may be any type of wired or wireless network, as well as combinations thereof. For example, network 120 may include a cable network, a wireline network, a fiber optic network, a telecommunications network, an intranet, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), the Public Switched Telephone Network (PSTN), a Bluetooth network, a ZigBee network, a Near Field Communication (NFC) network, or the like. In some embodiments, network 120 may include one or more network access points. For example, network 120 may include wired or wireless network access points, such as base stations and/or internet exchange points 120-1, 120-2, through which one or more components of client 300, server 200, database 150 may connect to network 120 to exchange data and/or information.
Database 150 may store data and/or instructions. In some embodiments, database 150 may store data obtained from clients 300. In some embodiments, database 150 may store data and/or instructions for performing the methods of evaluating applications described in this specification. In some embodiments, database 150 may store state data for various applications provided by server 200. In some embodiments, the database 150 may store the client 300's evaluation of the APP. Server 200 and client 300 may have access to database 150, and server 200 and client 300 may access data or instructions stored in database 150 via network 120. In some embodiments, database 150 may be directly connected to server 200 and client 300. In some embodiments, database 150 may be part of server 200. In some embodiments, database 150 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or the like, or any combination thereof. Exemplary mass storage may include magnetic disks, optical disks, solid state drives, and non-transitory storage media. Example removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tape, and the like. Typical volatile read and write memory may include Random Access Memory (RAM). Example RAM may include Dynamic RAM (DRAM), double-date rate synchronous dynamic RAM (DDR SDRAM), Static RAM (SRAM), thyristor RAM (T-RAM), zero-capacitance RAM (Z-RAM), and the like. Exemplary ROM can include Masked ROM (MROM), Programmable ROM (PROM), virtually programmable ROM (PEROM), electrically programmable ROM (EEPROM), compact disk (CD-ROM), digital versatile disk ROM, and the like. In some embodiments, database 150 may be implemented on a cloud platform. For example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, or the like, or any combination thereof.
As shown in fig. 1, a user 110 uses services provided by various applications provided by the APP on a client 300; server 200 may collect status data generated during reuse of the application and execute instructions of a method for evaluating the application stored in a memory built in server 200 and/or database 150 to evaluate the application to obtain an evaluation result.
Fig. 2 shows a schematic diagram of an apparatus for evaluating an application. The device may be an application 200.
The server 200 may perform the method of evaluating an application described herein. The method of evaluating an application is described elsewhere in this specification. The method P200 of evaluating an application is introduced, for example, in the description of fig. 3 and 5.
As shown in fig. 2, server 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, server 200 may also include a communication port 250 and an internal communication bus 210. Meanwhile, the server 200 may also include an I/O component 260.
Internal communication bus 210 may connect various system components including storage medium 230 and processor 220.
I/O components 260 support input/output between server 200 and other components.
Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a transitory storage medium. For example, the data storage device may include one or more of a magnetic disk 232, a read only memory medium (ROM)234, or a random access memory medium (RAM) 236. The storage medium 230 further includes at least one set of instructions stored in the data storage device. The instructions are computer program code that may include programs, routines, objects, components, data structures, procedures, modules, and the like that perform the methods of evaluating applications provided herein.
The communication port 250 is used for data communication between the server 200 and the outside. For example, server 200 may be connected to network 120 through communication port 250.
The at least one processor 220 is communicatively coupled to at least one storage medium 230 via an internal communication bus 210. The at least one processor 220 is configured to execute the at least one instruction set. When the system 100 is running, the at least one processor 220 reads the at least one instruction set and performs the method for evaluating an application P200 provided herein according to the instructions of the at least one instruction set. The processor 220 may perform all the steps involved in the method P200 of evaluating an application. Processor 220 may be in the form of one or more processors, and in some embodiments, processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, Reduced Instruction Set Computers (RISC), Application Specific Integrated Circuits (ASICs), application specific instruction set processors (ASIPs), Central Processing Units (CPUs), Graphics Processing Units (GPUs), Physical Processing Units (PPUs), microcontroller units, Digital Signal Processors (DSPs), Field Programmable Gate Arrays (FPGAs), Advanced RISC Machines (ARM), Programmable Logic Devices (PLDs), any circuit or processor capable of executing one or more functions, or the like, or any combination thereof. For illustrative purposes only, only one processor 220 is depicted in server 200 in this description. However, it should be noted that server 200 may also include multiple processors, and thus, the operations and/or method steps disclosed in this specification may be performed by one processor as described herein, or may be performed by a combination of multiple processors. For example, if in this description processor 220 of server 200 performs steps a and B, it should be understood that steps a and B may also be performed jointly or separately by two different processors 220 (e.g., a first processor performing step a, a second processor performing step B, or both a first and second processor performing steps a and B).
Fig. 3 shows a flow chart of a method P200 of evaluating an application. As previously described, the server 200 may execute the method P200 for evaluating an application provided herein. Specifically, the processor 220 in the server 200 may read a set of instructions stored in its local storage medium and/or the database 150, and then execute the method P200 for evaluating an application provided herein according to the specification of the set of instructions. The method P200 may include performing, by at least one processor 220, the steps of:
s220: an evaluation model 600 is obtained.
Fig. 4 shows a schematic structural diagram of an evaluation model 600 provided according to an embodiment of the present specification. As shown in fig. 4, the evaluation model 600 may be configured to evaluate the target application based on L evaluation indexes, resulting in an evaluation result of the target application. Wherein L is an integer greater than 1. The evaluation model 600 may be constructed based on a neural network model (NN). The target application may be any applet, application public number, application service number, etc. provided on the APP on the client 300 corresponding to the server 200. The evaluation model 600 may evaluate any one application on the APP. As shown in FIG. 4, the evaluation model 600 may include an input layer 620, a hidden layer 640, and an output layer 660.
The input layer 620 may receive evaluation data of the target application. The evaluation data may include feature data corresponding to the L evaluation indexes. The L evaluation indexes may be influence factors that can influence the evaluation result of the target application. The evaluation data may be feature data corresponding to the L evaluation indexes. The characteristic data corresponding to the same evaluation index is different, and the score values corresponding to the evaluation indexes may also be different. The L evaluation indicators may be a basic experience, a service richness, an operation conversion, an operation liveness, an operation quality, and the like. The basic experience may be a real experience of the user 110 on the target application when the user 110 uses the target application on the client 300, and the real experience may be quantified by data, for example, the feature data corresponding to the basic experience service may be a score of the user 110 on the target application, the feature data corresponding to the basic experience service may also be a complaint of a problem that occurs in a use process of the target application by the user 110, and the server 200 may generate the feature data of the basic experience service of the target application according to a frequency of the complaint and a solution of the complaint, and so on. The base experience can reflect the advantages and disadvantages of the target application in programming, problem solving and the like. The service richness may be a degree of diversification of services provided by the target application. The feature data corresponding to the service richness may be service category data, coverage data, and the like provided by the target application. The business transformation may be that the target application transforms the user 110 who clicked on the target application into a consuming user or a user who actually operates a transaction through the target application, and so on. The feature data corresponding to the business conversion may be ratio data of data paid by the user and data clicked by the user in the target application, or ratio data of data registered by the user and data clicked by the user, and the like. The business activity may be the frequency with which the target application publishes new dynamics or launches new services. The characteristic data corresponding to the business activity level may be frequency data of issuing new dynamic or launching new services by the target application. Such as the frequency of posting tweets, the frequency of interactions with the user 110, and so forth. The business quality may be feedback information of the user 110 to the target application. The characteristic data corresponding to the business quality may be comment or reply rate data of the user 110 on the tweet of the target application, or may be response data of the user 110 on the target application, such as a response rate of the user 110 on a preferential event launched by the target application, or may be forwarding rate data of the user 110 on the target application, and so on. The business activity and the business quality may reflect the ability of the target application to promote activity and pull new users. The L evaluation indicators may include evaluation indicators of the target application by the user 110, such as the base experience, and may also include business indicators of the target application, such as the business activity, the business quality, and so on. In some embodiments, each of the L evaluation indexes may include at least one sub-evaluation index. The L evaluation indexes may include M sub-evaluation indexes, where M is an integer greater than or equal to L. The sub-evaluation indexes included in each evaluation index may be the same or different. For example, the base experience may include user scores, plug-in usage counts, failure rates, flash back rates, and so forth. The quality of business may include a forwarding rate, a reply rate, and the like. The L evaluation indicators can evaluate the target application from a plurality of different angles, so that the evaluation result of the evaluation model 600 on the target application is more comprehensive, objective and scientific. Different types of APP provide applications for which the assessment metrics may differ. The L evaluation indexes can be obtained during the training process of the evaluation model 600. The training method of the evaluation model 600 will be described in detail later in the description.
The hidden layer 640 may be configured to perform data processing on the evaluation data to obtain an evaluation result of the target application. The hidden layer 640 may convert the evaluation data into an evaluation value corresponding to the value range according to the value range in which the evaluation data is located. The evaluation value may be a score, for example, an arbitrary score between 0 and 10, or an arbitrary score between 0 and 5, or an arbitrary score between 0 and 1, or the like. The value interval division rule of the evaluation data and the evaluation value corresponding to the value interval may be obtained in the training process of the evaluation model 600. The hidden layer 640 may perform data processing on the evaluation values corresponding to the L evaluation indexes. The hidden layer 640 may be a logistic regression layer, a fully connected layer, a convolutional layer, an average pooling layer, or the like. The data processing may be a logistic regression algorithm, a full join algorithm, a convolution algorithm, an average pooling algorithm, and the like. For convenience of illustration, the logistic regression algorithm is described herein as an example. The evaluation values may be in one-to-one correspondence with the L evaluation indexes. When each of the L evaluation indexes includes at least one sub-evaluation index, the evaluation values correspond to the M sub-evaluation indexes one-to-one. When each of the L evaluation indexes includes at least one sub-evaluation index, the hidden layer 640 may perform data processing on the evaluation data of the at least one sub-evaluation index of each evaluation index to obtain a combined evaluation value corresponding to each evaluation index, where the L evaluation indexes correspond to L combined evaluation values; then, the hidden layer 640 may perform data processing on the L combined evaluation values, generating the evaluation result of the target application. For convenience of description, we may label the combined evaluation value corresponding to each of the L evaluation indexes as S1,S2,…,Si,…,SLWherein i is 1,2, …, L. Let us mark the weight corresponding to the ith evaluation index in the L evaluation indexes as wi. Let us label the evaluation value corresponding to the sub-evaluation index included in the ith evaluation index as Si1,Si2,…,Sij…, wherein j is 1,2, …. Let us mark the weight corresponding to the jth sub-evaluation index contained in the ith evaluation index as wij. We label the final composite evaluation value of the target application as S0。
The comprehensive evaluation value S of the target application is used0Can be expressed as the following equation:
Figure BDA0002559949290000151
a combined evaluation value S corresponding to the ith evaluation index of the L evaluation indexesiCan be expressed as the following equation:
Si＝wi1Si1+wi2Si2+…+wijSij+ … formula (2)
As can be seen, the evaluation result is a comprehensive expression of L combined evaluation values corresponding to the L evaluation indexes. The combined evaluation value of each evaluation index is a comprehensive expression of the at least one sub-evaluation index. Therefore, the evaluation model 600 can not only keep monotonicity of each evaluation index, but also increase nonlinear expression of each evaluation index, so that the evaluation result of the evaluation model 600 is more accurate and comprehensive. Of course, the hidden layer 640 may also directly perform the data processing on the M evaluation values corresponding to the M sub-evaluation indexes.
The output layer 660 may be configured to output the evaluation result of the target application. The evaluation result may be a comprehensive evaluation value applied to the target. The evaluation result may be a score, such as any score between 0 and 10, or any score between 0 and 5, or any score between 0 and 1, etc. The evaluation result may also be an evaluation level divided based on the composite evaluation value, such as S level, a level, B level, C level, D level, and the like. Each level may also be divided into at least one sub-level, e.g., S level may be divided into S +, S, and S-, etc. The evaluation level may be graded according to the score calculated by the hidden layer 640.
The assessment model 600 may be trained based on the assessment data and the target scores of the sample set. The sample set includes T sample applications, where T is a positive integer greater than 1. The sample application may be an application randomly drawn from a plurality of applications in the APP. The target score may comprise a combination of the first score value and the second score value. The first score value may represent a subjective score value applied by the user 110 to the sample. The first score value may be an expert score value of the sample application, i.e., a score value of the sample application by an expert in the field to which the service provided by the sample application belongs. For example, a score between 0 and 10, or a score between 0 and 5, or a score between 0 and 1, and even a score between 0 and 100, etc. The first score value may also be an average pooling of the score values of the sample application by the users 110 using the sample application. The second score value may represent an objective score value of the sample application. The second score value may be a click rate score value of the sample application, a conversion rate score value of the sample application, and so on. The click rate score value may be a ratio of the number of clicks and the number of exposures applied by the sample. The conversion score value may be a ratio of the number of successful conversions and the number of clicks of the sample application. The number of successful conversions may be the number of transactions transacted or requests resolved using the application. For example, in a "how hungry" applet, the conversion score value may be the ratio of the number of orders placed in the applet to the total number of clicks into the applet.
As can be seen, the objective scores include both the subjective perception scores of the user 110 for the sample application and the objective click scores of the sample application. Therefore, the evaluation model 600 obtained by using the target score as a training target has a more comprehensive and scientific evaluation result for target application. Training the evaluation model 600 based on the evaluation data and the target score of the sample set can obtain the weights corresponding to the L evaluation indexes in the evaluation model 600. Wherein, the weight corresponding to the L evaluation indexes may include a weight w corresponding to each evaluation indexi. The L evaluationsThe weight corresponding to the index may also be a weight w corresponding to each of the M sub-evaluation indexesij. The combined evaluation value corresponding to each evaluation index may be obtained by averaging and pooling evaluation values corresponding to at least one corresponding sub-evaluation index in each evaluation index, or may be obtained by weighted summation of evaluation values corresponding to at least one corresponding sub-evaluation index in each evaluation index.
As shown in fig. 3, the method P200 may further include:
s240: and acquiring evaluation data of the target application.
The server 200 may obtain the evaluation data of the target application from the client 300 or the database 150.
S260: and inputting the evaluation data of the target application into the evaluation model to obtain an evaluation result of the target application.
The server 200 may periodically evaluate the applications on the APP to obtain evaluation results of all the applications, and may rank all the applications based on the evaluation results of all the applications. The server 200 may allocate traffic resources and an exposure rate according to the ranking results of all the applications, and the like, so as to improve the application service quality and improve the user experience and user adhesion.
Fig. 5 shows a flowchart of a training method P500 of an evaluation model 600 provided according to an embodiment of the present description. The training process of the assessment model 600 may be performed on the server 200, or on other devices. For convenience of illustration, we will describe below with an example in which the training process of the evaluation model 60 is performed on the server 200. As shown in fig. 5, the training step of the assessment model 600 may include performing, by the at least one processor 220 of the server 200:
s520: obtaining a target score for each sample application in the sample set.
As previously mentioned, the target score may comprise a combination of the first score value and the second score value. Specifically, step S520 may include: the server 200 respectively obtains the first scoring value and the second scoring value of each sample application in the sample set; and performing feature fusion calculation on the first scoring value and the second scoring value to obtain a target score applied to each sample in the sample set. Wherein the feature fusion calculation may be a weighted sum calculation. The sum of the weights of the first and second score values is 1. The weights of the first scoring value and the second scoring value can be 0.5, and the weights of the first scoring value and the second scoring value can also be obtained through machine learning. For example, the weights of the first score value and the second score value are obtained by machine learning with the objective that the loss function of the first score value and the loss function of the second score value are minimum. It should be noted that, between the feature fusion calculation of the first score value and the second score value, normalization processing needs to be performed on the first score value and the second score value, so that the floating intervals of the first score value and the second score value are the same. For example, the first score value may be a value between 0 and 10, the second score value may be a value between 0 and 1, and the normalization process may be a value between 0 and 10, a value between 0 and 1, or other value ranges.
S540: the L evaluation indexes of the evaluation model 600 are determined.
As described above, the L evaluation indexes may include M sub evaluation indexes. The L evaluation indexes of the determination evaluation model 600 may be the M sub-evaluation indexes. The L evaluation metrics may be used to evaluate the sample application. Thus, the L evaluation indicators may be indicators that have an effect on the target score applied to the sample. That is, when the evaluation data corresponding to each of the L evaluation indexes in the sample application changes, the target score of the sample application also changes accordingly. Specifically, step S540 may include performing, by the at least one processor 220 of the server 200:
s542: p candidate evaluation indexes are determined.
And P is an integer which is greater than or equal to L. The P candidate evaluation indexes can be mined manually or by an artificial intelligence algorithm. The P candidate evaluation indices may be indices that have an effect on the target score.
S544: and selecting the L evaluation indexes from the P candidate evaluation indexes according to feature screening conditions based on the correlation between the P candidate evaluation indexes and the target score.
After the P candidate evaluation indexes are mined, feature engineering processing needs to be performed on the P candidate evaluation indexes, and a plurality of candidate evaluation indexes meeting screening conditions are selected as the L evaluation indexes. The feature engineering may be to screen out a plurality of candidate evaluation indexes, of which the correlation satisfies a screening condition, from the P candidate evaluation indexes according to the correlation between each candidate evaluation index of the P candidate evaluation indexes and the target score. Specifically, in step S544, feature engineering processing may be performed on the feature data corresponding to the P candidate evaluation indexes. The feature engineering may be to select an independent variable affecting the dependent variable according to a feature selection condition by using the target score of the sample set as the dependent variable and using the feature data corresponding to each candidate evaluation index of the P candidate evaluation indexes as an independent variable.
Specifically, step S544 may include:
s544-2: feature data corresponding to the P candidate evaluation indexes applied to each sample in the sample set are obtained, and P candidate evaluation data sets corresponding to the P candidate evaluation indexes are constructed.
The feature engineering may include preprocessing a plurality of feature data corresponding to each candidate evaluation index, for example, performing non-dimensionalization conversion on the feature data to make feature data of different dimensions have the same dimension, for example, performing normalization processing on the feature data by using a maximum value and a minimum value, for example, performing normalization processing on the feature data by using a z-score algorithm; as another example, missing values are filled in using a minimum, average, or mode according to the feature data type; as another example, the outliers are smoothed, such as with a z-score algorithm, or the like. The plurality of feature data corresponding to each candidate evaluation index may include feature data corresponding to a current candidate evaluation index in the T sample applications.
S544-4: calculating a correlation of each of the P candidate evaluation metrics to the target score.
Step S544-2 may obtain a candidate evaluation data set corresponding to each of the P candidate evaluation indexes. The feature data in each candidate evaluation data set can be used as discrete data of the independent variable, the target scores applied by the T samples can be used as discrete data of the dependent variable, and the discrete data of the dependent variable and the discrete data of the independent variable are in one-to-one correspondence. In step S544-4, a correlation coefficient of the independent variable and the dependent variable may be calculated using a correlation coefficient algorithm. The correlation coefficient algorithm may be a pearson correlation coefficient algorithm, a spearman correlation coefficient algorithm, or the like. In step S544-4, a statistical algorithm may be used to calculate a relationship curve between the independent variable and the dependent variable, a linear relationship diagram between the independent variable and the dependent variable is fitted through the relationship curve, and a slope value of the linear relationship may be used as a correlation coefficient between the independent variable and the dependent variable.
S544-6: and selecting the L evaluation indexes with the correlation meeting a preset threshold value from the P candidate evaluation indexes.
The preset threshold may include a first threshold and a second threshold. The first threshold may be greater than the second threshold. The correlation satisfying the preset threshold may be a correlation coefficient having the correlation coefficient between the first threshold and the second threshold, including the first threshold and the second threshold. When the correlation coefficient of the independent variable and the dependent variable is lower than the second threshold, it is proved that the correlation of the independent variable and the dependent variable is low, and the influence of the independent variable on the dependent variable is low or insignificant, and therefore, the candidate evaluation index corresponding to the independent variable is not suitable as the evaluation index of the evaluation model 600. When the correlation coefficient of the independent variable and the dependent variable is higher than the first threshold, it is proved that the correlation between the independent variable and the dependent variable is very high, the influence of the independent variable on the dependent variable is very large, and the influence degree of the independent variable on the dependent variable is far higher than the influence degree of other independent variables on the dependent variable, so that the influence of other independent variables on the dependent variable is hardly reflected by the independent variable, the comprehensiveness of the evaluation index of the evaluation model 600 is reduced, and the accuracy of the evaluation result is reduced. Therefore, the correlation coefficient of the L evaluation indexes and the target score should be between the first threshold and the second threshold. As described above, the L evaluation indexes may include M sub evaluation indexes. Therefore, the correlation coefficient of the M sub-evaluation indicators with the target score should be between the first threshold and the second threshold. It should be noted that there may be a case where there is collinearity between the M sub-evaluation indexes. The co-linearity refers to a strong linear relationship between the independent variables. Different sub-evaluation indexes with linear relationship may have negative influence on the evaluation result, so that the evaluation model 600 lacks stability. Therefore, in determining the L evaluation indexes, the collinear relationship between the L evaluation indexes should be eliminated.
Specifically, step S544-6 may include: selecting Q candidate evaluation indexes with correlation meeting a preset threshold value from the P candidate evaluation indexes, wherein Q is an integer which is less than or equal to P and is greater than or equal to M; calculating the linear relation between every two of the Q candidate evaluation indexes; combining or deleting one of the candidate characteristic evaluation indexes with the linear relation larger than a preset linear threshold value to obtain the M sub-evaluation indexes; and dividing the M sub-evaluation indexes into the L evaluation indexes.
The server 200 may determine whether a linear relationship exists between each two of the Q candidate evaluation indexes by using a correlation coefficient algorithm. The correlation coefficient algorithm may be a pearson correlation coefficient algorithm, a spearman correlation coefficient algorithm, or the like. The server 200 may also determine, for each of the Q candidate evaluation indexes, a correlation factor between the current candidate evaluation index and another candidate evaluation index, with the current candidate evaluation index as a dependent variable and another candidate evaluation index except the current candidate evaluation index among the Q candidate evaluation indexes as an independent variable, and determine whether there is a linear relationship between the current candidate evaluation index and the another candidate evaluation index by using the correlation factor. There may be a correlation between the Q candidate evaluation indexes, and this correlation is mostly embodied as some function mapping relationship, such as the number of clicks of an application and the number of conversions of the application, and the number of conversions may increase as the number of clicks increases. In practical applications, in order to measure the correlation between the candidate evaluation indexes more conveniently, the correlation is usually simplified into a linear relationship, and the corresponding correlation factor may be a variance expansion factor (VIF). Multicollinearity (Multicollinearity) refers to that the dependent variables in a linear regression model are distorted or difficult to estimate accurately due to the existence of an exact correlation or a high correlation. VIF may be used to evaluate the multiple collinearity between the Q candidate evaluation indices. The larger the value of VIF, the more severe the multicollinearity. The variance expansion factor of each of the Q candidate evaluation indexes is determined by using a regression analysis in a manner that, for each of the Q candidate evaluation indexes, a current candidate evaluation index is used as a dependent variable, and other candidate evaluation indexes of the Q candidate evaluation indexes are used as independent variables. In practical application, the variance expansion factor of the current candidate evaluation index can be determined through logistic regression, and the logistic regression independent variable is generally in a linear line relation, so that the operation is more convenient.
And respectively comparing the correlation factor of each candidate evaluation index in the Q candidate evaluation indexes with a preset linear threshold, and screening the Q candidate evaluation indexes according to the comparison result. The linear threshold value generally represents a limit that the evaluation model 600 can tolerate for the correlation between the Q candidate evaluation indexes. And comparing the correlation factor with the linear threshold, and screening the M sub-evaluation indexes according to the comparison result. When the correlation factor appears as a variance inflation factor, in general: when 0< VIF <10, there is no multicollinearity, that is, there is no linear relationship between the current candidate evaluation index and other candidate evaluation indexes; when VIF is more than or equal to 10 and less than 100, stronger multiple collinearity exists; when VIF ≧ 100, there is severe multicollinearity, that is, the candidate evaluation index can be linearly expressed by other candidate evaluation indices. The linear threshold may be set according to specific requirements of the evaluation model 600, and if the evaluation model 600 requires that each selected sub-evaluation index has strong interpretability and each parameter has strong independence, the requirement on each sub-evaluation index is strict, and at this time, each sub-evaluation index VIF value is generally required to be smaller, for example, smaller than 2, or 2.5, or 3, or even smaller, and so on. The server 200 may combine the candidate evaluation indexes having multiple collinearity, for example, may combine the number of clicks and the number of conversions into the conversion rate, or may delete one of the candidate evaluation indexes having multiple collinearity. The server 200 determines the correlation factor of each candidate evaluation index, and compares each correlation factor with the linear threshold, so as to screen the Q candidate evaluation indexes, thereby improving the screening efficiency.
As described above, the L evaluation indices may include the M sub-evaluation indices. The server 200 may divide the M sub-evaluation indexes into the L evaluation indexes according to the attribute characteristics of the M sub-evaluation indexes. For example, the flicker rate and the white screen rate may be divided into an evaluation index.
After the server 200 determines the L evaluation indexes, the evaluation model 600 may be trained to obtain weights w corresponding to the L evaluation indexesiAnd obtaining the weight w corresponding to the M sub-evaluation indexesij. As shown in fig. 5, the training method of the assessment model 600 may further include performing, by the at least one processor 220 of the server 200:
s560: the evaluation model 600 is trained using the evaluation data applied to each sample in the sample set as training data and the target score applied to each sample in the sample set as a training target.
Specifically, step S560 may include:
and acquiring evaluation data applied to all samples in the sample set to generate a training data set. The evaluation data applied to all samples may include evaluation data corresponding to the L evaluation indexes, that is, evaluation data corresponding to the M sub-evaluation indexes.
Performing feature processing on the training data set using feature engineering, the feature engineering including binning. As mentioned above, the feature engineering may include preprocessing the data in the training data set, for example, performing non-dimensionalization transformation on the evaluation data to make the evaluation data with different dimensions have the same dimension, for example, normalizing the evaluation data by using the maximum and minimum values, for example, normalizing the evaluation data by using a z-score algorithm; as another example, missing values are filled in using a minimum, average, or mode according to the evaluation data type; as another example, the outliers are smoothed, such as with a z-score algorithm, or the like. The feature engineering may also include binning. The feature processing of the training data set by using feature engineering may include performing binning on the training data set, that is, performing discrete binning or segmentation on the evaluation data corresponding to each sub-evaluation index in the training data set according to the influence of the evaluation data on the target score, dividing the evaluation data into a plurality of bins, where each bin represents a data value interval, and each bin may include a plurality of evaluation data. The binning method may include multiple methods, such as a supervised binning method and an unsupervised binning method. The supervised binning method may further include a minimum description length binning method, a minimum entropy binning method, and the like. The unsupervised binning method may include an equidistant binning method, an equal frequency sharing method, and the like. In this specification, a supervised binning method is taken as an example to perform binning processing on the evaluation data in the training data set. Specifically, the feature processing the training data set using feature engineering may include:
and dividing the evaluation data in the training data set into M evaluation data sets corresponding to the M sub-evaluation indexes. The server 200 needs to perform binning processing on each of the M sub-evaluation indexes. The server 200 may divide the evaluation data in the training data set into the M evaluation data sets corresponding to the M sub-evaluation indexes. Each evaluation data set comprises evaluation data corresponding to the sub-evaluation indexes corresponding to the evaluation data set in the T sample applications.
And performing box separation processing on each evaluation data set in the M evaluation data sets to obtain N boxes. Server 200 may perform a binning process on each evaluation data set to divide the data in each evaluation data set into N bins, e.g., bin 1, bin 2, bin 3, … …, bin N. Wherein N is an integer greater than 1. The number of bins for different sets of assessment data may be the same or different. The N bins of each evaluation dataset represent the value intervals of the N data. The server 200 may use the target score as a dependent variable, use data corresponding to the target score as dependent variable data, use the current sub-evaluation index as an independent variable, and use data in the evaluation data set corresponding to the current sub-evaluation index as the independent variable data. The independent variable data and the dependent variable data are in one-to-one correspondence. The server 200 may perform supervised binning on the data of the independent variable according to the value of the dependent variable. In the binning of each evaluation data set, the woe (weight of occurrence) value and IV (information value) value of each bin may be calculated according to the proportion of high and low scores of the target score corresponding to the evaluation data in each bin so that the IV value is optimal to guarantee the binning effect. The calculation method of the boxed WOE value and IV value is a conventional calculation method, and details are not repeated in the specification.
An evaluation value corresponding to each bin is determined. In order to quantify the evaluation data, the server 200 needs to give a corresponding evaluation value to each bin. The server 200 may calculate the evaluation value corresponding to each bin by using the target score as a dependent variable and using N bins corresponding to the current sub-evaluation index as independent variables. The evaluation value corresponding to each bin may also be manually labeled.
Step S560 may further include: based on the result of the feature processing, the training data set is used as training data, the target score applied to each sample in the sample set is used as a training target, and the evaluation model 600 is trained to obtain weights corresponding to the L evaluation indexes. Wherein the weight corresponding to the L evaluation indexes comprises the weight w corresponding to each evaluation indexiAnd the weight w corresponding to each sub-evaluation index in the M sub-evaluation indexesij。
In summary, the method P200 and the system 100 for evaluating an application provided in the present specification evaluate a target application through the evaluation model 600. The evaluation model 600 may fuse the subjective score of the user with an objective score obtained according to the click rate to obtain the target score applied by the sample, and calculate and select L evaluation indexes satisfying a preset relationship according to a correlation between the candidate evaluation index and the target score; and training the evaluation model 600 by taking the feature data corresponding to the L evaluation indexes applied to the sample as training data and the target scores applied to the sample as training targets to obtain the weights corresponding to the L evaluation indexes. The method and the system can fit the subjective score value and the objective score value to obtain the weight of each evaluation index, so that the evaluation model 600 can evaluate the target application by combining subjective feeling and objective score, and the evaluation result is more comprehensive and objective.
And fusing the subjective score of the user and the objective score obtained according to the click rate to obtain a target score applied by the sample, training the evaluation model by taking the characteristic data corresponding to the plurality of evaluation indexes applied by the sample as training data and the applied target score as a training target to obtain the weight corresponding to each evaluation index, and evaluating the application by using the trained evaluation model. The method and the system can fit the subjective score value and the objective score value to obtain the weight of each evaluation index. The evaluation model provided by the method and the system can evaluate the application service by combining subjective feeling and objective scoring, so that the evaluation result is more comprehensive and objective.
Another aspect of the present description provides a non-transitory storage medium storing at least one set of executable instructions for evaluating an application, which when executed by a processor, direct the processor to perform the steps of the method P200 for evaluating an application described herein. In some possible implementations, various aspects of the description may also be implemented in the form of a program product including program code. The program code is adapted to cause the server 200 to perform the steps of evaluating an application as described in the present specification when the program product is run on the server 200. A program product for implementing the above method may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on the server 200. However, the program product of the present specification is not so limited, and in this specification, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system (e.g., the processor 220). The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Program code for carrying out operations for this specification may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on server 200, partly on server 200, as a stand-alone software package, partly on server 200 and partly on a remote computing device, or entirely on a remote computing device or server (server 200). In the case of a remote computing device, the remote computing device may be connected to the server 200 through the network 120, or may be connected to an external computing device.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
In conclusion, upon reading the present detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure can be presented by way of example only, and not limitation. Those skilled in the art will appreciate that the present specification contemplates various reasonable variations, enhancements and modifications to the embodiments, even though not explicitly described herein. Such alterations, improvements, and modifications are intended to be suggested by this specification, and are within the spirit and scope of the exemplary embodiments of this specification.
Furthermore, certain terminology has been used in this specification to describe embodiments of the specification. For example, "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various portions of this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined as suitable in one or more embodiments of the specification.
It should be appreciated that in the foregoing description of embodiments of the specification, various features are grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the specification, for the purpose of aiding in the understanding of one feature. This is not to be taken as an admission that any of the features are required in combination, and it is fully possible for one skilled in the art to extract some of the features as separate embodiments when reading this specification. That is, embodiments in this specification may also be understood as an integration of a plurality of sub-embodiments. And each sub-embodiment described herein is equally applicable to less than all features of a single foregoing disclosed embodiment.
Each patent, patent application, publication of a patent application, and other material, such as articles, books, descriptions, publications, documents, articles, and the like, cited herein is hereby incorporated by reference. All matters hithertofore set forth herein except as related to any prosecution history, may be inconsistent or conflicting with this document or any prosecution history which may have a limiting effect on the broadest scope of the claims. Now or later associated with this document. For example, if there is any inconsistency or conflict in the description, definition, and/or use of terms associated with any of the included materials with respect to the terms, descriptions, definitions, and/or uses associated with this document, the terms in this document are used.
Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of the present specification. Other modified embodiments are also within the scope of this description. Accordingly, the disclosed embodiments are to be considered in all respects as illustrative and not restrictive. Those skilled in the art may implement the applications in this specification in alternative configurations according to the embodiments in this specification. Therefore, the embodiments of the present description are not limited to the embodiments described precisely in the application.

Claims (14)

1. A method of evaluating an application, comprising:
obtaining an evaluation model configured to evaluate a target application based on L evaluation indicators, comprising:
receiving evaluation data of the target application, wherein the evaluation data comprises feature data corresponding to the L evaluation indexes, and L is an integer greater than 1;
performing data processing on the evaluation data to obtain an evaluation result of the target application; and
the result of the evaluation is output and,
the evaluation model is obtained by training based on evaluation data of a sample set and target scores, the sample set comprises a plurality of sample applications, the target scores comprise a combination of a first score value and a second score value, the first score value represents a subjective score value of a user for the sample applications, and the second score value represents an objective score value of the sample applications;
acquiring evaluation data of the target application; and
and inputting the evaluation data of the target application into the evaluation model to obtain an evaluation result of the target application.
2. The method of claim 1, wherein the first score value comprises an expert score value for the sample application and the second score value comprises a click-through rate score value for the sample application.
3. The method of claim 1, wherein each of the L evaluation indexes comprises at least one sub-evaluation index, and the L evaluation indexes comprises M sub-evaluation indexes, where M is an integer greater than or equal to L.
4. The method of claim 3, wherein the data processing the evaluation data to obtain the evaluation result of the target application comprises:
performing the data processing on the evaluation data of at least one sub-evaluation index in each evaluation index to obtain a combined evaluation value corresponding to each evaluation index; and
and performing the data processing on the L combined evaluation values corresponding to the L evaluation indexes to generate the evaluation result of the target application.
5. The method of any of claims 3, wherein the training of the assessment model comprises:
obtaining a target score applied by each sample in the sample set;
determining the L evaluation indicators of the evaluation model; and
and taking the evaluation data applied to each sample in the sample set as training data, taking the target score applied to each sample in the sample set as a training target, and training the evaluation model.
6. The method of claim 5, wherein said obtaining a target score applied to each sample in the sample set comprises:
obtaining the first scoring value and the second scoring value applied by each sample in the sample set; and
and performing feature fusion calculation on the first scoring value and the second scoring value to obtain a target score applied to each sample in the sample set.
7. The method of claim 6, wherein the feature fusion calculation comprises a weighted sum calculation.
8. The method of claim 5, wherein said determining said L assessment indicators of said assessment model comprises:
determining P candidate evaluation indexes, wherein P is an integer greater than or equal to L; and
and selecting the L evaluation indexes from the P candidate evaluation indexes according to feature screening conditions based on the correlation between the P candidate evaluation indexes and the target score.
9. The method of claim 8, wherein selecting the L evaluation indicators from the P candidate evaluation indicators comprises:
acquiring feature data of the P candidate evaluation indexes applied to each sample in the sample set, and constructing P candidate evaluation data sets corresponding to the P candidate evaluation indexes;
calculating a correlation of each of the P candidate evaluation metrics to the target score; and
and selecting the L evaluation indexes with the correlation meeting a preset threshold value from the P candidate evaluation indexes.
10. The method of claim 9, wherein said selecting the L evaluation indexes with the correlation satisfying a preset threshold from the P candidate evaluation indexes comprises:
selecting Q candidate evaluation indexes with correlation meeting a preset threshold value from the P candidate evaluation indexes, wherein Q is an integer which is less than or equal to P and is greater than or equal to M;
calculating the linear relation between every two of the Q candidate evaluation indexes;
combining or deleting one of the candidate characteristic evaluation indexes with the linear relation larger than a preset linear threshold value to obtain the M sub-evaluation indexes; and
and dividing the M sub-evaluation indexes into the L evaluation indexes.
11. The method of claim 5, wherein training the assessment model using the assessment data applied to each sample in the sample set as training data and the goal score applied to each sample in the sample set as training goals comprises:
obtaining evaluation data applied to all samples in the sample set to generate a training data set;
performing feature processing on the training data set by using feature engineering, wherein the feature engineering comprises binning;
based on the result of the feature processing, taking the training data set as training data, taking a target score applied by each sample in the sample set as a training target, and training the evaluation model; and
and obtaining the weights corresponding to the L evaluation indexes.
12. The method of claim 11, wherein the weights corresponding to the L evaluation indicators comprise:
a weight corresponding to each of the L evaluation indexes; and
and the weight corresponding to each sub-evaluation index in the M sub-evaluation indexes.
13. The method of claim 12, wherein the feature processing the training dataset using feature engineering comprises:
dividing evaluation data in the training data set into M evaluation data sets corresponding to the M sub-evaluation indexes;
performing binning processing on each evaluation data set in the M evaluation data sets to obtain N bins, wherein N is an integer greater than 1; and
an evaluation value corresponding to each bin is determined.
14. A system for evaluating an application, comprising:
at least one storage medium comprising at least one set of instructions for applying an evaluation; and
at least one processor communicatively coupled to the at least one storage medium,
wherein when the system is running, the at least one processor reads the at least one instruction set and performs the method of application evaluation of any of claims 1-13 in accordance with the instructions of the at least one instruction set.
CN202010603359.3A 2020-06-29 2020-06-29 Method and system for evaluating applications Pending CN111754126A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010603359.3A CN111754126A (en) 2020-06-29 2020-06-29 Method and system for evaluating applications

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010603359.3A CN111754126A (en) 2020-06-29 2020-06-29 Method and system for evaluating applications

Publications (1)

Publication Number Publication Date
CN111754126A true CN111754126A (en) 2020-10-09

Family

ID=72677880

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010603359.3A Pending CN111754126A (en) 2020-06-29 2020-06-29 Method and system for evaluating applications

Country Status (1)

Country Link
CN (1) CN111754126A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112528159A (en) * 2020-12-24 2021-03-19 北京百度网讯科技有限公司 Characteristic quality evaluation method and device, electronic equipment and storage medium
CN112596992A (en) * 2020-11-25 2021-04-02 新华三大数据技术有限公司 Application activity calculation method and device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102055613A (en) * 2010-12-13 2011-05-11 宁波大学 Network quality evaluation method
CN104516897A (en) * 2013-09-29 2015-04-15 国际商业机器公司 Method and device for sorting application objects
CN105095411A (en) * 2015-07-09 2015-11-25 中山大学 Method and system for predicting APP ranking based on App quality
CN109102164A (en) * 2018-07-20 2018-12-28 广东省科技基础条件平台中心 Platform evaluation method, apparatus, computer equipment and storage medium
CN109960650A (en) * 2018-09-04 2019-07-02 中国平安人寿保险股份有限公司 Application assessment method, apparatus, medium and electronic equipment based on big data
CN110782349A (en) * 2019-10-25 2020-02-11 支付宝(杭州)信息技术有限公司 Model training method and system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102055613A (en) * 2010-12-13 2011-05-11 宁波大学 Network quality evaluation method
CN104516897A (en) * 2013-09-29 2015-04-15 国际商业机器公司 Method and device for sorting application objects
CN105095411A (en) * 2015-07-09 2015-11-25 中山大学 Method and system for predicting APP ranking based on App quality
CN109102164A (en) * 2018-07-20 2018-12-28 广东省科技基础条件平台中心 Platform evaluation method, apparatus, computer equipment and storage medium
CN109960650A (en) * 2018-09-04 2019-07-02 中国平安人寿保险股份有限公司 Application assessment method, apparatus, medium and electronic equipment based on big data
CN110782349A (en) * 2019-10-25 2020-02-11 支付宝(杭州)信息技术有限公司 Model training method and system

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112596992A (en) * 2020-11-25 2021-04-02 新华三大数据技术有限公司 Application activity calculation method and device
CN112528159A (en) * 2020-12-24 2021-03-19 北京百度网讯科技有限公司 Characteristic quality evaluation method and device, electronic equipment and storage medium
CN112528159B (en) * 2020-12-24 2024-03-26 北京百度网讯科技有限公司 Feature quality assessment method and device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
US11562012B2 (en) System and method for providing technology assisted data review with optimizing features
Ngo et al. Factor-based big data and predictive analytics capability assessment tool for the construction industry
CN107908740B (en) Information output method and device
Moula et al. Credit default prediction modeling: an application of support vector machine
US11107166B2 (en) Multi-step day sales outstanding forecasting
KR102322845B1 (en) Method, device and system for deriving brand marketing strategy based on artificial intelligence
Sinha et al. FDI inflow, ICT expansion and economic growth: An empirical study on Asia-pacific developing countries
US11036811B2 (en) Categorical data transformation and clustering for machine learning using data repository systems
US20230132448A1 (en) Heterogeneous graph embedding
US20200234218A1 (en) Systems and methods for entity performance and risk scoring
WO2019072128A1 (en) Object identification method and system therefor
US20220036385A1 (en) Segment Valuation in a Digital Medium Environment
WO2020257782A1 (en) Factory risk estimation using historical inspection data
US20230013086A1 (en) Systems and Methods for Using Machine Learning Models to Automatically Identify and Compensate for Recurring Charges
US20180075765A1 (en) System and method for transmission of market-ready education curricula
US10678821B2 (en) Evaluating theses using tree structures
Kim et al. Optimal urban population size: national vs local economic efficiency
US20230297862A1 (en) Performing predictive inferences using multiple predictive models
CN111754126A (en) Method and system for evaluating applications
US11556897B2 (en) Job-post budget recommendation based on performance
US11900400B2 (en) Enhanced survey information synthesis
US20230153662A1 (en) Bayesian modeling for risk assessment based on integrating information from dynamic data sources
CN114519376A (en) Data segmentation using neural networks
US20210357699A1 (en) Data quality assessment for data analytics
Almomani et al. Selecting a good stochastic system for the large number of alternatives

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20230116

Address after: 200120 Floor 15, No. 447, Nanquan North Road, Free Trade Pilot Zone, Pudong New Area, Shanghai

Applicant after: Alipay.com Co.,Ltd.

Address before: 310000 801-11 section B, 8th floor, 556 Xixi Road, Xihu District, Hangzhou City, Zhejiang Province

Applicant before: Alipay (Hangzhou) Information Technology Co.,Ltd.

TA01 Transfer of patent application right
RJ01 Rejection of invention patent application after publication

Application publication date: 20201009

RJ01 Rejection of invention patent application after publication