WO2022015404A1

WO2022015404A1 - Sample assessment

Info

Publication number: WO2022015404A1
Application number: PCT/US2021/032133
Authority: WO
Inventors: Shaoguang MAO; Frank Kao-Ping K SOONG; Yan Xia; Jonathan Y. Tien
Original assignee: Microsoft Technology Licensing LLC
Current assignee: Microsoft Technology Licensing LLC
Priority date: 2020-07-15
Filing date: 2021-05-13
Publication date: 2022-01-20
Anticipated expiration: 2023-01-15
Also published as: CN113962275A; CN113962275B

Abstract

According to implementations of the present disclosure, there is provided a solution for sample assessment. In this solution, a set of comparison results is generated based on an input sample and a reference sample set. Each comparison result in the set of comparison results indicates a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set. Then, at least one reference sample is determined from the reference sample set based on the set of comparison results. The at least one reference sample matches the input sample in terms of the target attribute. A score of the input sample with respect to the target attribute is determined based on a respective score of the at least one reference sample with respect to the target attribute.

Description

SAMPLE ASSESSMENT

BACKGROUND

[0001] In various types of data processing, it is often necessary to assess a certain attribute of a sample, and a result of the assessment is usually presented in the form of numerical values or levels. Therefore, samples need to be scored. For example, there might be a need to assess a speech sample from a user in Computer Assisted Language Learning. The speech sample may need to be scored in terms of oral fluency, pronunciation accuracy and the like. In addition, there are also similar needs for sample assessment in many application scenarios such as text assessment, age estimation, credit assessment and so on.

SUMMARY

[0002] According to implementations of the present disclosure, there is provided a solution for sample assessment. In this solution, a set of comparison results is generated based on an input sample and a reference sample set. Each comparison result in the set of comparison results indicates a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set. Then, at least one reference sample is determined from the reference sample set based on the set of comparison results. The at least one reference sample matches the input sample in terms of the target attribute. The target attribute of the input sample is close or similar to that of the determined at least one reference sample. A score of the input sample with respect to the target attribute is determined based on a respective score of the at least one reference sample with respect to the target attribute. In the solution, the score prediction is converted to a series of simpler binary comparisons or binary classifications, i.e., comparison between the input sample and the reference sample. A series of binary comparisons help to accurately find the reference sample that is close to the input sample in terms of the target attribute. Therefore, the accuracy of the score of the input sample determined based on the score of the reference sample may be guaranteed. In this way, more efficient, accurate and robust sample assessment can be achieved.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Fig. 1 illustrates a block diagram of a computing device which can implement a plurality of implementations of the present disclosure;

[0005] Fig. 2 illustrates an architecture diagram of a system for sample assessment according to implementations of the present disclosure;

[0006] Fig. 3 illustrates a block program of a training stage according to some implementations of the present disclosure;

[0007] Fig. 4 illustrates a block diagram of an inference stage according to some implementations of the present disclosure;

[0008] Fig. 5 illustrates a flowchart of a method for sample assessment according to some implementations of the present disclosure.

[0009] Throughout the drawings, the same or similar reference signs refer to the same or similar elements.

DETAILED DESCRIPTION

[0010] The present disclosure will now be discussed with reference to several example implementations. It is to be understood these implementations are discussed only for the purpose of enabling persons skilled in the art to better understand and thus implement the present disclosure, rather than suggesting any limitations on the scope of the subject matter. [0011] As used herein, the term “includes” and its variants are to be read as open terms that mean “includes, but is not limited to.” The term “based on” is to be read as “based at least in part on.” The term “one implementation” and “an implementation” are to be read as “at least one implementation.” The term “another implementation” is to be read as “at least one other implementation.” The terms “first,” “second,” and the like may refer to different or same objects. Other definitions, explicit and implicit, may be included below. [0012] As used herein, the term “neural network” can handle inputs and provide corresponding outputs and it generally includes an input layer, an output layer and one or more hidden layers between the input and output layers. The neural network used in the deep learning applications generally includes a plurality of hidden layers to extend the depth of the network. Individual layers of the neural network are connected in sequence, such that an output of a preceding layer is provided as an input for a following layer, where the input layer receives the input of the neural network while the output of the output layer acts as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes the input from the preceding layer. In the text, the terms “neural network”, “network” and “neural network model” may be used interchangeably.

[0013] As used herein, an “attribute” of a sample refers to a property or characteristic of the sample in one or more specific aspects. A “target attribute” of the sample may be one or more attributes or a comprehensive attribute which are selected for the purpose of the sample assessment. Therefore, the “target attribute” of the sample may differ depending on a purpose of the sample assessment. For example, in an assessment of a language learning sample, the “target attribute” may include, but is not limited to, oral fluency, pronunciation accuracy, language proficiency, etc. In age estimation, the “target attribute” may refer to an age of a user corresponding to a sample. In credit assessment, the “target attribute” may refer to a credit level of a user corresponding to a sample.

[0014] As briefly described above, there is a need for sample assessment in many application scenarios, i.e., to score one or more attributes of a sample. Take an application scenario of language learning for example. A language learning sample of a user may need to be assessed. For example, the speech sample (e.g., a recording) from the user may need to be scored with respect to oral fluency. A common scoring scheme for oral fluency is 5- point mean opinion score (MOS), where a score of 1 stands for “bad,” a score of 2 stands for “poor,” a score of 3 stands for “fair,” a score of 4 stands for “good,” and a score of 5 stands for “excellent.” In another example, a text sample (e.g., an essay) from a user may need to be scored. Scoring of samples in language learning application scenarios usually involve subjective assessment.

[0015] Results of subjective assessment are limited by assessors, e.g., experts. Different assessors might make different or even significantly different assessments. The same assessor also might make different assessments of the same sample at different times. Such subjective assessment usually calls for several assessors, thereby resulting in a lot of time and labor costs. In a scenario of online speech learning, such subjective assessment fails to provide an assessment result to a user in real time.

[0016] For automatic assessment, a quantitative and objectively measurable scoring method is needed. Different scoring models have been proposed, such as multi-class classifier, Gaussian model, etc. In these existing schemes, different scores of samples with respect to the target attribute (e.g., oral fluency) are independent, and the intrinsic ordinal property between these different scores is ignored. Also, regression approaches are proposed where scores are regarded as numerical values. However, in such approaches, training non-stationary kernels for regression is challenging and is prone to overfitting in training. [0017] Ordinal regression aims at classifying or predicting numerical values from labelled samples where the labels of the samples exhibit a natural ordering. Training an ordinal regression model requires labelled samples, while the labels of the samples, i.e., scores of samples with respect to the target attribute, are obtained from subjective assessment of assessors. Therefore, the number of such labelled samples is usually small, which does not help to obtain a robust and accurate ordinal regression model.

[0018] In view of the above, according to implementations of the present disclosure, a solution for sample assessment is provided so as to solve one or more of the above problems and other potential problems. In the solution, a set of comparison results are generated based on an input sample and a reference sample set. Each comparison result indicates a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set. Then, at least one reference sample is determined from the reference sample set based on the set of comparison results. The at least one reference sample matches the input sample in terms of the target attribute. The target attribute of the input sample is close or similar to the target attribute of the determined at least one reference sample. Next, a score of the input sample with respect to the target attribute is determined based on a respective score of the at least one reference sample with respect to the target attribute.

[0019] The process of scoring the input sample otherwise involves complex multi-class classification or ordinal regression. The solution for sample assessment proposed herein may be regarded as ordinal regression based on the reference samples, which converts originally complex multi-class classification or ordinal regression to a series of simpler binary comparisons or binary classifications, i.e., comparison between the input sample and the reference samples. A series of binary comparisons help to accurately find the reference sample that is close to the input sample in terms of the target attribute. Therefore, the accuracy of the score of the input sample which is determined based on the score of the reference sample may be guaranteed. In this way, more efficient, accurate and robust sample assessment can be achieved.

[0020] Various example implementations of the present disclosure are described in detail below in conjunction with the drawings.

Example Environment

[0021] Fig. 1 illustrates a block diagram of a computing device 100 that can implement a plurality of implementations of the present disclosure. It should be understood that the computing device 100 shown in Fig. 1 is only exemplary and should not constitute any limitation on the functions and scopes of the implementations described by the present disclosure. As shown in Fig. 1, the computing device 100 includes a computing device 100 in the form of a general purpose computing device. Components of the computing device 100 may include, but is not limited to, one or more processors or processing units 110, a memory 120, a storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0022] In some implementations, the computing device 100 may be implemented as various user terminals or service terminals with computing capability. The service terminals may be servers, large-scale computing devices, and the like provided by a variety of service providers. The user terminal, for example, is a mobile terminal, a fixed terminal or a portable terminal of any type, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, Internet nodes, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio/video player, a digital camera/video, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any other combination thereof, including accessories and peripherals of these devices or any other combination thereof. It may also be predicted that the computing device 100 can support any type of user-specific interface (such as a “wearable” circuit, and the like). [0023] The processing unit 110 may be a physical or virtual processor and may execute various processing based on the programs stored in the memory 120. In a multi-processor system, a plurality of processing units executes computer-executable instructions in parallel to enhance parallel processing capability of the computing device 100. The processing unit 110 can also be known as a central processing unit (CPU), microprocessor, controller and microcontroller.

[0024] The computing device 100 usually includes a plurality of computer storage mediums. Such mediums may be any attainable medium accessible by the computing device 100, including but not limited to, a volatile and non-volatile medium, a removable and non-removable medium. The memory 120 may be a volatile memory (e.g., a register, a cache, a Random Access Memory (RAM)), a non-volatile memory (such as, a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), flash), or any combination thereof. The memory 120 may include sample processing modules 122, which are configured to perform various functions described herein. The sample processing module 122 may be accessed and operated by the processing unit 110 to realize corresponding functions.

[0025] The storage device 130 may be a removable or non-removable medium, and may include a machine-readable medium (e.g., a memory, a flash drive, a magnetic disk) or any other medium, which may be used for storing information and/or data and be accessed within the computing device 100 The computing device 100 may further include additional removable/non-removable, volatile/non-volatile storage mediums. Although not shown in Fig. 1, there may be provided a disk drive for reading from or writing into a removable and non-volatile disk and an optical disc drive for reading from or writing into a removable and non-volatile optical disc. In such cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0026] The communication unit 140 implements communication with another computing device via a communication medium. Additionally, functions of components of the computing device 100 may be realized by a single computing cluster or a plurality of computing machines, and these computing machines may communicate through communication connections. Therefore, the computing device 100 may operate in a networked environment using a logic connection to one or more other servers, a Personal Computer (PC) or a further general network node.

[0027] The input device 150 may be one or more various input devices, such as a mouse, a keyboard, a trackball, a voice-input device, and the like. The output device 160 may be one or more output devices, e.g., a display, a loudspeaker, a printer, and so on. The computing device 100 may also communicate through the communication unit 140 with one or more external devices (not shown) as required, where the external device, e.g., a storage device, a display device, and so on, communicates with one or more devices that enable users to interact with the computing device 100, or with any device (such as a network card, a modem, and the like) that enable the computing device 100 to communicate with one or more other computing devices. Such communication may be executed via an Input/Output (I/O) interface (not shown).

[0028] In some implementations, apart from being integrated on an individual device, some or all of the respective components of the computing devices 100 may also be set in the form of a cloud computing architecture. In the cloud computing architecture, these components may be remotely arranged and may cooperate to implement the functions described by the present disclosure. In some implementations, the cloud computing provides computation, software, data access and storage services without informing a terminal user of physical locations or configurations of systems or hardware providing such services. In various implementations, the cloud computing provides services via a Wide Area Network (such as Internet) using a suitable protocol. For example, the cloud computing provider provides, via the Wide Area Network, the applications, which can be accessed through a web browser or any other computing component. Software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote location. The computing resources in the cloud computing environment may be merged or spread at a remote datacenter. The cloud computing infrastructure may provide, via a shared datacenter, the services even though they are shown as a single access point for the user. Therefore, components and functions described herein can be provided using the cloud computing architecture from a service provider at a remote location. Alternatively, components and functions may also be provided from a conventional server, or they may be mounted on a client device directly or in other ways. [0029] The computing device 100 may be used for implementing sample assessment in various implementations of the present disclosure. As shown in Fig. 1, the computing device 100 may receive an input sample 170 through the input device 150. The input sample 170 is usually associated with a user. For example, in an application scenario of age estimation, the input sample 170 may comprise an image of the user’s face; in an application scenario of language learning, the input sample 170 may comprise a piece of speech spoken by the user, or a piece of text written by the user.

[0030] In some implementations, the input sample 170 may be raw data, such as raw speech data and text data. In such implementations, the computing device 100 may pre- process the input raw data so as to extract features of the input sample 170 for the target attribute. In some implementations, the input sample 170 may be pre-processed data. The input sample 170 may comprise one or more features extracted for the target attribute based on the raw data. For example, if an article or a passage needs to be comprehensively assessed, the extracted features may include, but are not limited to, the average sentence length, the shortest sentence length, the longest sentence length, the average paragraph length, the shortest paragraph length, the longest paragraph length, the number of words, the number of sentences, the number of paragraphs, a grammatical error rate, a spelling error rate, a unary /binary/ternary grammatical probability, etc.

[0031] A reference sample set 190 may be stored in the computing device 100. Alternatively, or in addition, the computing device 100 may receive the reference sample set 190 from the input device 150 or via the communication unit 140. The reference sample set 190 comprises a plurality of reference samples of the same type as the input sample 170. Each reference sample may have a score with respect to the target attribute, which is also referred to as “a predetermined score” herein. The predetermined score of the reference sample may be given by an assessor or determined based on a score from an assessor. Therefore, the reference sample may be regarded as a labelled sample. Similar to the input sample, the reference sample may be raw data, or may comprise features extracted for the target attribute.

[0032] The computing device 100 may implement the solution for sample assessment described herein to determine an assessment result 180 for the input sample 170. The assessment result 180 may be presented in the form of a score of the input sample 170 with respect to the target attribute. For example, in the above example of oral fluency, the assessment result 180 may comprise a score of the input sample 170 with respect to oral fluency, e.g., one of the five scores “1”, “2”, “3”, “4”, and “5”. Alternatively, or in addition, the assessment result may also be presented in the form of a level of the input sample 170 with respect to the target attribute. For example, the assessment result 180 may comprise one of the levels “bad”, “poor”, “fair”, “good”, and “excellent” corresponding to the five scores “1”, “2”, “3”, “4”, and “5”.

[0033] Although Fig. 1 shows only one input sample 170, it should be understood that the computing device 100 may process a plurality of such input samples. In addition, although in the example of Fig. 1, the computing device 100 receives the input sample 170 from the input device 150, and provides the assessment result 180 via the output device 160. This is merely illustrative without any limitation. The computing device 100 may receive the input sample 170 from other devices (not shown) via the communication unit 140, and/or provide the assessment result 180 to externally via the communication unit 140.

System Architecture

[0034] Fig. 2 shows an architecture diagram of a system 200 for sample assessment according to implementations of the present disclosure. The system 200 may be implemented in the computing device 100 of Fig. 1. For example, in some implementations, the system 200 may be implemented as at least one portion of the sample processing module 122 of the computing device 100 of Fig. 1, i.e., implemented as a computer program module. As shown in Fig. 2, the system 200 may generally comprise a training subsystem 210 and an inference subsystem 220. It should be understood that the structure and functionality of the system 200 are described only for the purpose of illustration rather than suggesting any limitation on the scope of the present disclosure. Implementations of the present disclosure may also be implemented in different structures and/or functionality.

[0035] In general, the sample assessment solution according to implementations of the present disclosure may be divided into two stages: a training stage and an inference stage. In the training stage, the training subsystem 210 may receive a training sample set 230, and train a classifier 250 for sample assessment based on the training sample set 230.

[0036] The training sample set 230 may comprise a plurality of training samples of the same type as the input sample 170. Each training sample may have a score with respect to the target attribute. The score of the training sample may be given in advance by an assessor or determined based on a score from an assessor. Therefore, the training sample may be regarded as a labelled sample.

[0037] In some implementations, each training sample may comprise raw data, e.g., speech data and text data. In such implementations, the training subsystem 210 may comprise a feature extracting module for extracting for the target attribute one or more features of the training sample. In some implementations, each training sample may be processed data, which comprises one or more features extracted for the target attribute based on the raw data.

[0038] The trained classifier 250 may predict a relative level of two samples in terms of the target attribute. In other words, the trained classifier 250 may determine a probability that a target attribute of one sample in the two samples is superior to the target attribute of the other sample. For example, the trained classifier 250 may provide a probability that one speech sample is superior to another speech sample in terms of oral fluency.

[0039] The classifier 250 may be implemented using any appropriate model. For example, the classifier 250 may be a binary classifier based on a neural network. The neural network may have any appropriate network structure. In another example, the classifier 250 may be a classifier based on a support vector machine. The scope of the present disclosure is not limited in this regard.

[0040] In the inference stage, the inference subsystem 220 may receive the trained classifier 250, the reference sample set 190 and the input sample 170. The inference subsystem 220 may determine a score of the input sample 170 with respect to the target attribute based on the reference sample set 190.

[0041] The reference sample set 190 may be determined based on the training sample set 230. In such implementations, the labelled samples can be fully utilized. In some implementations, the reference sample set 190 may comprise all training samples in the training sample set 230. In such implementations, the distribution of reference samples is the distribution of training samples, which is closer to the true score distribution. Therefore, in the inference stage, the input sample 170 is compared with respective reference samples with a distribution closer to the true score distribution. This helps to locate a reference sample that is closer to the input sample 170 in terms of the target attribute, thereby more accurately determining the score of the input sample. In other implementations, the reference sample set 190 may comprise a portion of training samples in the training sample set 230. How to determine the reference sample from the training sample set 230 will be described in detail below.

[0042] Alternatively, the reference sample set 190 may be independent of the training sample set 230. Although the reference sample is of the same type as the training sample, the reference sample may be obtained independently from the training sample. For example, the reference sample and the training sample may be associated with different users.

[0043] Example processes of the training stage and the inference stage will be described with reference to Figs. 3 and 4, respectively.

Training Stage

[0044] The overall architecture of the system 200 for sample assessment is described above. Some implementations of the training stage implemented by the training subsystem 210 will be described below. Fig. 3 shows a block diagram of the training stage according to some implementations of the present disclosure. The training of the classifier 250 will be described below by taking a neural network as an example, however, this is merely exemplary and not intended to limit the scope of the present disclosure.

[0045] In the training stage, the training subsystem 210 generates a plurality of training sample pairs 311, 312, ..., 31N based on the training sample set 230, where N is a natural number. The plurality of training sample pairs 311, 312, ..., 3 IN may form a set of training sample pairs, which is abbreviated as “a training pair set 310.” Each training sample pair comprises two training samples. As shown in Fig. 3, the training sample pair 311 comprises training samples 311-1 and 311-2, the training sample pair 312 comprises training samples 312-1 and 312-2, and the training sample pair 3 IN comprises training samples 31N- 1 and 31N-2.

[0046] The training samples 311-1, 312-1, ..., 31N-1 may be collectively referred to as first training samples or individually referred to as a first training sample, and the training samples 311-2, 312-2, ..., 31N-2 may be collectively referred to as second training samples or individually referred to as a second training sample. The first training sample may have a first score with respect to the target attribute, and the second training sample may have a second score with respect to the target attribute. It should be understood that the first score and the second score may be the same or different.

[0047] Then, the training subsystem 210 may train the classifier 250 by using the plurality of training sample pairs 311, 312, ..., 31N. A comparison result generated by the trained classifier 250 based on the first training sample and the second training sample conforms to the relative level of the first score and the second score. In other words, the judgment made by the trained classifier 250 as to whether the target attribute of the first training sample is superior to the target attribute of the second training sample conforms to the relative level of the first score and the second score.

[0048] By combining training samples into training sample pairs, the training sample set for training the classifier 250 is actually expanded. In addition, the score of a training sample is usually the subjective assessment made by an assessor. As mentioned above, the subjective assessment is limited to the assessor. Therefore, scores of training samples might be unevenly distributed, which brings about an adverse effect on the modeling of sample assessment. The combined training sample pairs according to the present disclosure may reduce such adverse effect.

[0049] An example process of training the classifier 250 by using the training sample set 230 is described in detail below Assume that the feature of the z-th training sample in the training sample set 230 is represented by a vector x_t e X, and the score of the z-th training sample is represented by y_t e Y = {ri,r₂, —,r_k} , where *£z ’ ^anc^ the symbol ^>-

indicates the relative level among different scores r_k. The training sample set 230 with Nx training samples is represented by

[0050] In some implementations, the training subsystem 210 may combine any two training samples in the training sample set 230 into a training sample pair in the training pair set 310. In such implementations, the training sample set 310 may be represented as

P = {(X_j, Xj), ^z _j\_{i-1 -1} , where both the first training sample (¾, y,) and the second training sample (L7, y,j belong to the training sa ple set . The label Z_j for training the classifier 250 indicates whether y_i is larger than y,·, and may be represented as an equation below: _{z =} if (y; ³ yj)

⁷ lo, otherwise (1)^.

[0051] Next, the training subsystem 210 may use each training sample pair in the training pair set P =

to train the classifier 250, so as to compare which training sample in the training sample pair is superior with respect to the target attribute.

[0052] In other implementations, the training subsystem 210 may first select a plurality of anchored samples from the training sample set 230 to form an anchored sample set. For example, the training sample set 230 may be divided into a plurality of groups of training samples (e.g., buckets) according to scores of training samples in the training sample set 230 with respect to the target attribute. Scores of each group of training samples with respect to the target attribute are the same. Then, a certain number of training samples may be selected as anchored samples from each group of training samples. For example, in the example of oral fluency, the training sample set 230 may be divided into five groups or buckets corresponding to scores “1”, “2”, “3”, “4”, and “5”, respectively. Then, a certain number of training samples are selected (e.g., randomly selected) as anchored samples from each bucket.

[0053] Next, the training subsystem 210 may combine an anchored sample in the anchored sample set and a training sample in the training sample set 230 into a training sample pair. In this way, the training pair set 310 for training the classifier 250 may be generated.

[0054] Given a training sample set X ~

for each score r_A, (k — 1 , . . , K), the training subsystem 210 may randomly select Ni training samples (xi,yl) with J ^“ k as anchored samples to form the anchored sample set A. The training subsystem 210 may determine a training sample subset D with ND training samples. The training sample subset D and the anchored sample set A may have no intersection set. For example, the training sample subset D may comprise the remaining training samples in the training sample set 230 other than anchored samples. Alternatively, the training sample subset D and the anchored sample set A may have an intersection set. For example, the training sample subset D may comprise all training samples in the training sample set 230.

[0055] In this case, the training sample subset D with ND training samples may be represented as D = {x_j>y_j}_j^₁, and the anchored sample set A may be represented as A =

where NA is the number of anchored samples for each score, K is the total number of scores. Thereby, a training pair set

P be generated based on a Cartesian product between the

training sample subset D and the anchored sample set A . In the training pair set P, the first training sample ( X_j,y_j ) belongs to the training sample subset D, and the second training sample (A«_> N) belongs to the anchored sample set A. The label zf for training the classifier 250 represents whether yj is larger than n, and may be represented by the equation below: zf = O' ^if{y> ^{³ ¾)} (2).

¹ 10, otherwise

[0056] Next, the training subsystem 210 may use each training sample pair in the training pair set P = the classifier 250, so as to compare which

training sample in the training sample pair is superior with respect to the target attribute.

Inference Stage

[0057] Some implementations of the training stage have been described above. The trained classifier 250 may be used by the inference subsystem 220 to determine a score of the input sample 170. Fig. 4 shows a block diagram 400 of the inference stage according to some implementations of the present disclosure.

[0058] In the inference stage, the inference subsystem 220 generates a set of comparison results based on the input sample 170 and the reference sample set 190 which is represented as B below. As mentioned with reference to Fig. 2, in some implementations, the reference sample set 190 may comprise all training samples in the training sample set 230. That is, in such implementations, B=X.

[0059] In other implementations, the reference sample set 190 may comprise a portion of the training samples in the training sample set 230. That is, in such implementations, B is a subset of X. As an example, in the online assessment scenario, the assessment result 180 may need to be provided in real time and rapidly. If the training sample set 230 has a large number of training samples and the computing device 100 has limited computing power, then a portion of the training sample set 230 may be selected as the reference sample set 190. For example, the reference sample set 190 may be the anchored sample set A as described above. It should be understood that even if the anchored sample set A is not used in the training stage, the reference sample set 190 may be generated from the training sample set 230 by using the above approach for generating the anchored sample set A.

[0060] In the example of Fig. 4, the reference sample set 190 may comprise a plurality of reference samples 410-1, 410-2, ..., 410-M, which may be collectively referred to as reference samples 410 or individually referred to as a reference sample 410, where M is a natural number. The inference subsystem 220 may combine the input sample 170 and each reference sample 410 into a respective input-reference sample pair. Fig. 4 shows input- reference sample pairs 411, 412, ..., 41M.

[0061] The inference subsystem 220 inputs the input-reference sample pairs 411, 412, ..., 41M to the trained classifier 250, respectively. The classifier 250 may generate respective comparison results 420-1, 420-2, ..., 420-M, which may be collectively referred to as comparison results 420 or individually referred to as a comparison result 420.

[0062] Each comparison result 420 indicates a probability that a target attribute of the input sample 170 is superior to the target attribute of a respective reference sample 410 in the reference sample set 190. The comparison result 420 may indicate the probability that the target attribute of the input sample 170 is superior to the target attribute of the respective reference sample 410 in any appropriate manner. For example, the comparison result 420 may comprise the probability that the target attribute of the input sample 170 is superior to the target attribute of the respective reference sample 410. In another example, the comparison result 420 may comprise a probability that the target attribute of the input sample 170 is not superior (or inferior) to the target attribute of the respective reference sample 410.

[0063] In some implementations, the probability indicated by the comparison result 420 may be a hard decision with a binary value outputted by the classifier 250. For example, in such implementations, the probability with a value of 1 may represent that the target attribute of the input sample 170 is superior to the target attribute of the respective reference sample 410; while the probability with a value of 0 may represent that the target attribute of the input sample 170 is not superior to the target attribute of the respective reference sample 410.

[0064] In some implementations, the probability indicated by the comparison result 420 may be a soft decision outputted by the classifier 250, e.g., posteriors estimated by the classifier 250. In such implementations, the magnitude of the probability indicated by the comparison result 420 may represent the confidence of whether the target attribute of the input sample 170 is superior to the target attribute of the respective reference sample 410. In implementations where the hard decision is used as the comparison result 420, cumulative quantization errors might occur, while cumulative quantization errors might degrade the accuracy of a final assessment result. Such cumulative quantization errors may be effectively reduced by using posteriors estimated by the classifier 250, which helps to achieve accurate sample assessment.

[0065] The feature of the input sample 170 is represented by x', a predicted score of the input sample 170 with respect to the target attribute is represented by y', the feature of the i- th reference sample in the reference sample set 190 is represented by oq and a score of the i- th reference sample with respect to the target attribute is represented by s_a.. Then, the i- th comparison result may be generated by using the trained classifier 250 based on the feature x' of the input sample and the feature oq of the i- th reference sample.

F{x', oq) = P(y' ³ s_ai\x', oq) (3) where a_t B . Therefore, in such implementations, F(x', a ) may be defined as the softmax function in the last layer of the neural network for implementing the classifier 250. [0066] Still with reference to Fig. 4, the inference subsystem 220 determines, from the reference sample set 190 based on the set of comparison results 420-1, 420-2, ..., 420-M, at least one reference sample matching the input sample 170 in terms of the target attribute. The determined at least one reference sample is also referred to as “target reference sample” herein, e.g., a target reference sample 430 as shown in Fig. 4. The target reference sample 430 is close or similar to the input sample in the target attribute space. For example, the target reference sample 430 may be a reference sample that is closest to the input sample in the target attribute space. For example, in the application scenario of language learning, the target reference sample 430 may be closest to the input sample 170 in terms of oral fluency as compared to other reference samples in the reference sample set 190. Next, the inference subsystem 220 may determine a score of the input sample 170 with respect to the target attribute based on a score of the target reference sample 430 with respect to the target attribute, thereby providing the assessment result 180.

[0067] Various ways may be used to determine the target reference sample 430 based on the comparison result 420. In some implementations, the target reference sample 430 may be determined from the reference sample set 190 based on a value of the probability indicated by the comparison result 420. For example, in the implementations where the comparison result 420 comprises a posterior probability (e.g., Equation (3)) outputted by the classifier 250, a comparison result with the posterior probability being equal or close to a predetermined threshold may be determined, and then the reference sample 410 corresponding to such a comparison result may be determined as the target reference sample 430. The predetermined threshold, e.g., 0.5, may be determined depending on the specific implementation of the classifier 250. If the posterior probability included in the comparison result 420-2 is equal to the predetermined threshold or its difference with the predetermined threshold falls within a certain range, then the reference sample 410-2 may be determined as the target reference sample 430. In such implementations, if there is only one target reference sample 430, then the inference subsystem 220 may determine the score of the target reference sample 430 with respect to the target attribute as the score of the input sample 170 with respect to the target attribute. If there is a plurality of target reference samples 430, then the inference subsystem 220 may determine a weighted average of scores of the plurality of target reference samples 430 with respect to the target attribute as the score of the input sample 170 with respect to the target attribute.

[0068] In some implementations, the target reference sample 430 may be determined from the reference sample set 190 by a ranking approach. For example, the inference subsystem 220 may rank the input sample 170 and the reference sample 410 in the reference sample set 190 in terms of the target attribute based on the comparison result 420. Then, the inference subsystem 220 may determine the target reference sample 430 based on a result of the ranking.

[0069] Since the reference sample 410 in the reference sample set 190 has a predetermined score with respect to the target attribute, e.g., a score given by an assessor, the reference sample 410 in the reference sample set 190 may have a predetermined rank according to the predetermined score. In such implementations, ranking the input sample 170 and the reference sample 410 may be implemented as predicting a ranking position of the input sample 170 relative to the ordered reference samples 410 in terms of the target attribute. For example, probabilities indicated by the comparison results 420-1, 420-2, ..., 420-M may be summed up. A result of the summation may correspond to the predicted rank of the input sample 170. A reference sample in the reference sample set 190 for which the predetermined rank corresponds to the predicted rank may be determined as the target reference sample 430.

[0070] Continuing with the example of Equation (3), the predicted rank r( ') of the input sample with the feature represented as x' relative to the ordered reference samples 410 may be expressed as below:

[0071] In some implementations, the predicted rank r(x') may be rounded. A reference sample (also referred to as “a first reference sample”) in the reference sample set 190 for which the predetermined rank corresponds to the rounded predicted rank r(x') may be determined as the target reference sample 430. Accordingly, the score of the target reference sample 430 with respect to the target attribute may be determined as the score of the input sample 170 with respect to the target attribute. That is, the score s(x') of the input sample with the feature represented as x' with respect to the target attribute may be expressed as below:

(5)·

[0072] In some implementations, a reference sample (also referred to as “a second reference sample) in the reference sample set 190 for which the predetermined rank is higher than the predicted rank r(x') and a reference sample (also referred to as “a third reference sample”) in the reference sample set 190 for which the predetermined rank is lower than the predicted rank r(x') may be determined as the target reference samples 430. Then, the score of the input sample 170 with respect to the target attribute may be determined by weighting respective scores of the second reference sample and the third reference sample with respect to the target attribute. For example, a weighting coefficient may be determined based on a value of the predicted rank r(x'). In such implementations, a score with a finer-granularity of the input sample 170 may be obtained by weighting scores of a plurality of reference samples.

[0073] The inference process according to some implementations of the present disclosure has been described above. In these implementations, reference samples in the reference sample set which are close or similar to the input sample in terms of the target attribute may be determined through a series of comparison results between the input sample and reference samples. Then, the score of the input sample may be determined based on scores of such reference samples. Apparently, reference samples in the reference sample set actually act as anchor points, which are used to locate the input sample in terms of the target attribute. As a result, the determined score of the input sample with respect to the target attribute is accurate and robust. In this way, the solution of the present disclosure can achieve a smaller mean absolute error and higher correlation that is close to human labelers, with less training samples.

[0074] After determining the score of the input sample 170 with respect to the target attribute, further processing may be performed in different application scenarios based on the score. For example, in the language learning scenario, courses matching the score may be recommended to a user based on the score. In the credit rating scenario, services matching the score may be recommended to a user based on the score.

[0075] In addition, a plurality of scores of the input sample 170 with respect to different attributes (which may be regarded as different dimensions) may be determined using the sample assessment solution of the present disclosure. A plurality of scores of a plurality of input samples with respect to target attribute(s) may be further determined, where the plurality of input samples are associated with different users. Then, the plurality of input samples may be clustered based on the plurality of scores. In this way, it may be considered that users associated with a plurality of input samples in the same cluster are closest in terms of target attribute(s). For example, such users may be divided into the same group or class in the language learning or other teaching scenario.

[0076] In some cases, the input sample 170 and the score of the input sample 170 determined by the inference subsystem 220 may be added to the training sample set 230 so as to update the classifier 250. For example, if the score of the input sample 170 determined by the inference subsystem 220 is considered as accurate through manual assessment, then the input sample 170 and its score may be used to further train the classifier 250. Alternatively, or in addition, the input sample 170 and its score may further be added to the reference sample set 190.

Example Method and Example Implementation

[0077] Fig. 5 shows a flowchart of a method 500 for sample assessment according to some implementations of the present disclosure. The method 500 may be implemented by the computing device 100, e.g., may be implemented at the sample processing module 122 in the memory 120 of the computing device 100.

[0078] As shown in Fig. 5, at block 510, the computing device 100 generates a set of comparison results based on an input sample and a reference sample set. Each comparison result in the set of comparison results indicates a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set. At block 520, the computing device 100 determines at least one reference sample from the reference sample set based on the set of comparison results. The at least one reference sample matches the input sample in terms of the target attribute. The target attribute of the input sample is close or similar to that of the determined at least one reference sample. At block 530, the computing device 100 determines a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.

[0079] In some implementations, determining the at least one reference sample from the reference sample set comprises: ranking the input sample and reference samples in the reference sample set in terms of the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

[0080] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining from the reference sample set, a first reference sample with the same rank as the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute based on a score of the first reference sample with respect to the target attribute.

[0081] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a second reference sample with a higher rank than the input sample and a third reference sample with a lower rank than the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute by weighting respective scores of the second and the third reference samples with respect to the target attribute.

[0082] In some implementations, generating the set of comparison results comprises: generating a comparison result for a fourth reference sample in the reference sample set by using a trained classifier based on the input sample and the fourth reference sample. The comparison result indicates a probability that the target attribute of the input sample is superior to the target attribute of the fourth reference sample.

[0083] In some implementations, the method 500 further comprises: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs comprising a first training sample with a first score with respect to the target attribute and a second training sample with a second score with respect to the target attribute; and training the classifier based on the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first and second training samples matches a relative level of the first score to the second score.

[0084] In some implementations, the method 500 further comprises: determining the reference sample set based on the training sample set.

[0085] In some implementations, determining the reference sample set based on the training sample set comprises: dividing the training sample set into a plurality of sets of training samples according to scores of training samples in the training sample set with respect to the target attribute; and selecting reference samples in the reference sample set from each set of training samples in the plurality of sets of training samples.

[0086] As seen from the foregoing description, the sample assessment solution according to implementations of the present disclosure converts the score prediction to a series of simpler binary comparisons or binary classifications, i.e., comparison between the input sample and the reference samples. A series of binary comparisons help to accurately find out the reference sample that is close to the input sample in terms of the target attribute. Therefore, the accuracy of the score of the input sample determined based on the score of the reference sample may be guaranteed. In this way, more efficient, accurate and robust sample assessment can be achieved.

[0087] Some example implementations of the present disclosure are listed below.

[0088] In an aspect, the present disclosure provides a computer-implemented method. The method comprises: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set; determining at least one reference sample from the reference sample set based on the set of comparison results, the at least one reference sample matching the input sample in terms of the target attribute; and determining a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.

[0089] In some implementations, determining the at least one reference sample from the reference sample set comprises: ranking the input sample and reference samples in the reference sample set in terms of the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

[0090] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining from the reference sample set, a first reference sample with the same rank as the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute based on a score of the first reference sample with respect to the target attribute.

[0091] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a second reference sample with a higher rank than the input sample and a third reference sample with a lower rank than the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute by weighting respective scores of the second and third reference samples with respect to the target attribute.

[0092] In some implementations, generating the set of comparison results comprises: generating a comparison result for a fourth reference sample in the reference sample set by using a trained classifier based on the input sample and the fourth reference sample, the comparison result indicating a probability that the target attribute of the input sample is superior to the target attribute of the fourth reference sample.

[0093] In some implementations, the method further comprises: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs comprising a first training sample with a first score with respect to the target attribute and a second training sample with a second score with respect to the target attribute; and training the classifier based on the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first and second training samples matches a relative level of the first score to the second score.

[0094] In some implementations, the method further comprises: determining the reference sample set based on the training sample set.

[0095] In some implementations, determining the reference sample set based on the training sample set comprises: dividing the training sample set into a plurality of sets of training samples according to scores of training samples in the training sample set with respect to the target attribute; and selecting reference samples in the reference sample set from each set of training samples in the plurality of sets of training samples.

[0096] In another aspect, the present disclosure provides an electronic device. The electronic device comprises: a processing unit; and a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set; determining at least one reference sample from the reference sample set based on the set of comparison results, the at least one reference sample matching the input sample in terms of the target attribute; and determining a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.

[0097] In some implementations, determining the at least one reference sample from the reference sample set comprises: ranking the input sample and reference samples in the reference sample set in terms of the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

[0098] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a first reference sample with the same rank as the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute based on a score of the first reference sample with respect to the target attribute.

[0099] In some implementations, determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a second reference sample with a higher rank than the input sample and a third reference sample with a lower rank than the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute by weighting respective scores of the second and third reference samples with respect to the target attribute.

[00100] In some implementations, generating the set of comparison results comprises: generating a comparison result for a fourth reference sample in the reference sample set by using a trained classifier based on the input sample and the fourth reference sample, the comparison result indicating a probability that the target attribute of the input sample is superior to the target attribute of the fourth reference sample.

[00101] In some implementations, the method further comprises: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs comprising a first training sample with a first score with respect to the target attribute and a second training sample with a second score with respect to the target attribute; and training the classifier based on the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first and second training samples matches a relative level of the first score to the second score.

[00102] In some implementations, the method further comprises: determining the reference sample set based on the training sample set.

[00103] In some implementations, determining the reference sample set based on the training sample set comprises: dividing the training sample set into a plurality of sets of training samples according to scores of training samples in the training sample set with respect to the target attribute; and selecting reference samples in the reference sample set from each set of training samples in the plurality of sets of training samples.

[00104] In a further aspect, the present disclosure provides a computer program product being tangibly stored in a computer storage medium and comprising machine-executable instructions which, when executed by a device, causing the device to perform the method of the above aspect.

[00105] In a further aspect, the present disclosure provides a computer-readable medium having machine-executable instructions stored thereon which, when executed by a device, cause the device to perform the method of the above aspect.

[00106] The functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[00107] Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or a server. [00108] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine- readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. [00109] Further, although operations are depicted in a particular order, it should be understood that the operations are required to be executed in the particular order shown or in a sequential order, or all operations shown are required to be executed to achieve the expected results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure. Certain features that are described in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable sub combination.

[00110] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method, comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set; determining at least one reference sample from the reference sample set based on the set of comparison results, the at least one reference sample matching the input sample in terms of the target attribute; and determining a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.

2. The method of claim 1, wherein determining the at least one reference sample from the reference sample set comprises: ranking the input sample and reference samples in the reference sample set in terms of the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

3. The method of claim 2, wherein determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a first reference sample with the same rank as the input sample as the at least one reference sample, and wherein determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute based on a score of the first reference sample with respect to the target attribute.

4. The method of claim 2, wherein determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a second reference sample with a higher rank than the input sample and a third reference sample with a lower rank than the input sample as the at least one reference sample, and wherein determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute by weighting respective scores of the second and third reference samples with respect to the target attribute.

5. The method of claim 1, wherein generating the set of comparison results comprises: generating a comparison result for a fourth reference sample in the reference sample set by using a trained classifier based on the input sample and the fourth reference sample, the comparison result indicating a probability that the target attribute of the input sample is superior to the target attribute of the fourth reference sample.

6. The method of claim 5, further comprising: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs comprising a first training sample with a first score with respect to the target attribute and a second training sample with a second score with respect to the target attribute; and training the classifier based on the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first and second training samples matches a relative level of the first score to the second score.

7. The method of claim 6, further comprising: determining the reference sample set based on the training sample set.

8. The method of claim 7, wherein determining the reference sample set based on the training sample set comprises: dividing the training sample set into a plurality of sets of training samples according to scores of training samples in the training sample set with respect to the target attribute; and selecting reference samples in the reference sample set from each set of training samples in the plurality of sets of training samples.

9. An electronic device, comprising: a processing unit; and a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set; determining at least one reference sample from the reference sample set based on the set of comparison results, the at least one reference sample matching the input sample in terms of the target attribute; and determining a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.

10. The device of claim 9, wherein determining the at least one reference sample from the reference sample set comprises: ranking the input sample and reference samples in the reference sample set in terms of the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

11. The device of claim 10, wherein determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a first reference sample with the same rank as the input sample as the at least one reference sample, and wherein determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute based on a score of the first reference sample with respect to the target attribute.

12. The device of claim 10, wherein determining the at least one reference sample based on the result of the ranking comprises: determining, from the reference sample set, a second reference sample with a higher rank than the input sample and a third reference sample with a lower rank than the input sample as the at least one reference sample, and wherein determining the score of the input sample with respect to the target attribute comprises: determining the score of the input sample with respect to the target attribute by weighting respective scores of the second and third reference samples with respect to the target attribute.

13. The device of claim 9, wherein generating the set of comparison results comprises: generating a comparison result for a fourth reference sample in the reference sample set by using a trained classifier based on the input sample and the fourth reference sample, the comparison result indicating a probability that the target attribute of the input sample is superior to the target attribute of the fourth reference sample.

14. The device of claim 13, wherein the acts further comprises: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs comprising a first training sample with a first score with respect to the target attribute and a second training sample with a second score with respect to the target attribute; and training the classifier based on the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first and second training samples matches a relative level of the first score to the second score.

15. A computer program product being tangibly stored in a computer storage medium and comprising machine-executable instructions which, when executed by a device, cause the device to perform acts comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is superior to the target attribute of a respective reference sample in the reference sample set; determining at least one reference sample from the reference sample set based on the set of comparison results, the at least one reference sample matching the input sample in terms of the target attribute; and determining a score of the input sample with respect to the target attribute based on a respective score of the at least one reference sample with respect to the target attribute.