WO2016122532A1 - Net promoter score determination - Google Patents

Net promoter score determination Download PDF

Info

Publication number
WO2016122532A1
WO2016122532A1 PCT/US2015/013540 US2015013540W WO2016122532A1 WO 2016122532 A1 WO2016122532 A1 WO 2016122532A1 US 2015013540 W US2015013540 W US 2015013540W WO 2016122532 A1 WO2016122532 A1 WO 2016122532A1
Authority
WO
WIPO (PCT)
Prior art keywords
comments
data
nps
sentiment
score
Prior art date
Application number
PCT/US2015/013540
Other languages
French (fr)
Inventor
Bindu NARAYAN
Ramakanth KANAGOVI
Siddharth ARUN
Original Assignee
Hewlett Packard Enterprise Development Lp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Enterprise Development Lp filed Critical Hewlett Packard Enterprise Development Lp
Priority to PCT/US2015/013540 priority Critical patent/WO2016122532A1/en
Publication of WO2016122532A1 publication Critical patent/WO2016122532A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions

Definitions

  • Customer loyalty to a company, product, and/or service is determined by a variety of factors. For example, customer loyalty may be determined by surveys that are conducted at point of purchase. Companies may utilize a variety of resources to increase the loyalty of customers with the expectation that company objectives will be met or surpassed.
  • Figure 1 illustrates an architecture of a net promoter score (NPS) determination apparatus, according to an example of the present disclosure
  • Figure 2 illustrates training set generation, according to an example of the present disclosure
  • Figure 3 illustrates data classification as promoters, passives, and detractors, according to an example of the present disclosure
  • Figure 4 illustrates a hierarchical framework of words, according to an example of the present disclosure
  • Figure 5 illustrates sentiment determination, according to an example of the present disclosure
  • Figure 6 illustrates semantic similarity between words as a function of distance between the words in a tree structure, according to an example of the present disclosure
  • Figure 7 illustrates semantic similarity determination, according to an example of the present disclosure
  • FIG. 8 illustrates review recommend score (RRS) determination, according to an example of the present disclosure
  • Figure 9 illustrates average review sentiment (ARS) determination, according to an example of the present disclosure
  • FIG. 10 illustrates aggregated review sentiment score (ARSS) determination, according to an example of the present disclosure
  • Figure 11 illustrates a method for NPS determination, according to an example of the present disclosure
  • Figure 12 illustrates further details of the method for NPS determination, according to an example of the present disclosure
  • Figure 13 illustrates further details of the method for NPS determination, according to an example of the present disclosure.
  • Figure 14 illustrates a computer system, according to an example of the present disclosure.
  • the terms “a” and “an” are intended to denote at least one of a particular element.
  • the term “includes” means includes but not limited to, the term “including” means including but not limited to.
  • the term “based on” means based at least in part on.
  • Net promoter score is metric that is derived from a question, such as, how likely is it that a respondent would recommend a company, product, and/or service to a friend or colleague?
  • Respondents may be categorized as promoters, passives, or detractors. Promoters may include those respondents who respond with a score of 9 or 10 (e.g., on a scale of 1 -10) and are considered loyal enthusiasts.
  • Detractors may include those respondents who respond with a score of 0 to 6 (e.g., dissatisfied respondents).
  • Passives may include those respondents who respond with a score of 7 or 8 (e.g., respondents who do not directly affect the NPS).
  • NPS may be determined by subtracting the percentage of respondents who are detractors from the percentage of respondents who are promoters.
  • the information that is used to determine NPS may be ascertained by using surveys at predetermined time intervals (e.g., annually, semi-annually, etc.). Such surveys may attempt to cover all customer segments and geographies where a company has a presence. The surveys may be performed by undertaking a representative sampling methodology, which can be time consuming and expensive. [0021] Once a NPS based survey is completed and processed, a company may determine whether customer sentiments have changed from previous surveys. However, customer sentiments may change drastically at any given time (e.g., overnight). With a wide variety of choices that are available to customers, customers may switch to a competitor at any given time. Thus a company may need to virtually constantly engage and track its customers' changing needs. A company may also need to initiate remedial actions virtually instantaneously to drive improvements in customer loyalty, and to enable profitable growth. A company may also need to deliver real-time information to employees, so that the employees may act on customer feedback and achieve results.
  • predetermined time intervals e.g., annually, semi-annually, etc
  • NPS may be determined in real-time.
  • the data that is used to determine the NPS may be received from perpetually available data sources that receive unsolicited feedback in the on-line space.
  • unsolicited feedback include feedback from internal data sources such as a company's websites, feedback forums, call centers, or other company related e-commerce websites.
  • unsolicited feedback may also include feedback from external data sources that include online stores such as AMAZON, E-BAY, etc., social media sources such as TWITTER,
  • NPS question i.e., how likely is it that a respondent would recommend a company, product, and/or service to a friend or colleague?.
  • An analyst may guess from what is being told, if an author (e.g., reviewer of a company, product, and/or service) of a comment (e.g., a review comment) is willing to recommend a company, product, and/or service.
  • the apparatus and method disclosed herein may receive data from the internal and external data sources, and perform a sentiment and theme analysis.
  • the received data may be analyzed by using a dictionary to identify adverbs and adjectives.
  • the semantic similarity of adverbs and adjectives from the data may be analyzed, for example, with respect to the word
  • a lexical database of English may be used to analyze the semantic similarity of adverbs and adjectives from the data to the word "recommend”.
  • a NPS score may be determined as a sum product of similarity scores and sentiments. Based on the NPS score and/or other metrics as disclosed herein, the comment and/or the author of the comment may be classified as a promoter, a detractor, or passive. Thus, the apparatus and method disclosed herein may process the sentiments derived from the data, and estimate a specific metric designated as the NPS score.
  • the real-time NPS determination may provide for a company to operate its growth engine to operate at peak efficiency.
  • the real-time NPS determination may help employees clarify and simplify the job of satisfying customers.
  • the real-time NPS determination may facilitate identification and engagement with customers throughout their shopping endeavor. Further, the real-time NPS determination may also facilitate analysis of a company's performance on a frequent basis.
  • the real-time NPS determined may be used to generate a complete operational model to drive growth for a company.
  • the real-time NPS may be used for visibility of company sentiment and opinions.
  • the real-time NPS may be used to generate a complete feedback loop with respect to operations of a company.
  • the real-time NPS may also provide a cost benefit in that instead of investment in intermittent surveys, the data that is used may be sourced from customer feedback and opinions available on social media and other web sources in digital unstructured format.
  • a company's performance and the NPS metric may be more readily equated based on the real-time NPS determination.
  • Figure 1 illustrates an architecture of a NPS determination apparatus (hereinafter also referred to as "apparatus 100"), according to an example of the present disclosure.
  • the apparatus 100 is depicted as including a classification module 102 to implement a classification technique, a semantic similarity module 104 to implement a semantic similarity technique, an average sentiment module 106 to implement an average sentiment technique, and a sentiment aggregation module 108 to implement a sentiment aggregation technique, where each of these techniques may be used to determine a NPS 110.
  • a NPS determination module 112 may determine the NPS 110.
  • the apparatus 100 may receive data 114 from an internal data source 116 and/or an external data source 118, where the data 114 may be used by the classification module 102, the semantic similarity module 104, the average sentiment module 106, and/or the sentiment aggregation module 108 to determine the NPS 110.
  • the apparatus 100 may combine sentiment analysis and word associations to analyze text (e.g., in the form of documents, comments, reviews, etc.) from the data 114, and determine if the author of the text is willing to recommend or not.
  • the apparatus 100 may determine whether the author related to the text is a promoter, a detractor, or passive.
  • the data 114 may include unsolicited feedback available in the on-line space.
  • the data 114 may include data from internal data sources (e.g., the internal data source 116) such as a company's websites, feedback forums, call centers, or other company related e- commerce websites.
  • the data 114 may also include data from external data sources (e.g., the external data source 118) that include online stores such as AMAZON, E-BAY, etc., social media sources such as TWITTER, FACEBOOK, BLOGS, YOUTUBE, etc., and commercial sources such as LINKEDLN or communications with account managers.
  • external data sources e.g., the external data source 118
  • online stores such as AMAZON, E-BAY, etc.
  • social media sources such as TWITTER, FACEBOOK, BLOGS, YOUTUBE, etc.
  • commercial sources such as LINKEDLN or communications with account managers.
  • the apparatus 100 may utilize a classification technique implemented by the classification module 102, a semantic similarity technique implemented by the semantic similarity module 104, an average sentiment technique implemented by the average sentiment module 106, and/or a sentiment aggregation technique implemented by the sentiment aggregation module 108.
  • the classification module 102 may implement machine learning to classify comments from the data 114 as promoter, passive, and detractor.
  • Machine learning is a scientific discipline that explores the construction and study of machine readable instructions to learn from data.
  • the classification module 102 may utilize, for example, support vector machine (SVM) to recognize patterns in the data 114.
  • SVM support vector machine
  • the classification module 102 may generate a training data set.
  • the training data set may represent the data on which an analytical model is generated by the classification module 102.
  • the classification module 102 may implement a non-supervised learning technique on the training data set. That is, the classification module 102 may utilize a training data set that is not manually determined to classify an initial set of the data 114 as promoters, detractors, or passives.
  • the classification module 102 may determine the sentiment orientation of every document (i.e., including a plurality of comments), text (i.e., data that forms a comment), and/or comment from the data 114 using the lexicon approach (note, references to a comment from the data 114 may similarly apply to a document and/or text from the data 114).
  • the lexicon approach may be based on opinion words in a comment. Opinion words are words that are generally used to express positive or negative sentiments, e.g., "good” and "bad”.
  • the classification module 102 may utilize a dictionary of positive, negative, and neutral opinion words.
  • the classification module 102 may query the dictionary to determine the sentiment orientation of the document, text, and/or comment from the data 114.
  • the overall sentiment of a document, text, and/or comment from the data 114 may be determined by a number of positive, negative, and neutral words in the document, text, and/or comments, respectively, from the data 114, and scored accordingly.
  • the comment, "love our new printer, and now we can print anywhere in our house with this great device” will be expected to receive a high positive sentiment score, (e.g., +5) because of the many positive words such as “love”, “anywhere", and “great”, whereas the comment "hate this product, and I have lost so much productivity over the print cartridges which are very expensive” would be expected to receive a high negative sentiment score (e.g., -6) because of the many negative words such as "hate", “lost", and "expensive”.
  • the comments (and/or documents, text, etc.) from the data 114 that are above the 80 th percentile for sentiment scores may be used to form the promoter training set, and the comments that are below the 20 th percentile for sentiment scores may be used to form the detractor training set.
  • Figure 2 illustrates training set generation, according to an example of the present disclosure.
  • the sentiment scores range from -6 to +6, 20 th percentile will correspond to -4.4, and 80 th percentile will correspond to +3.4.
  • comments with a sentiment score > 3.4 may be classified as promoters, and comments with a sentiment score ⁇ -4.4 may be classified as detractors. This ensures that the comments forming the promoter training set are actually positive, and the comments forming the detractor training set are actually negative.
  • a corresponding to an exact zero (or approximate zero) sentiment score may be considered. From the comments corresponding to an exact zero sentiment score, comments which have a maximum number of neutral words may be selected. A sufficient quantity of such comments that have a maximum number of neutral words may be selected for the training data. For example, assuming a number of promoter training comments are 50 and a number of detractor training comments are 100, a number of the passive training comments may be selected as a number between the number of promoter training comments and the number of detractor training comments (e.g., 87). Thus the training sets for the different sentiment classes, i.e., promoters, detractors, and passives may be formed without the user providing any manual input. For the classification module 102, the combined approach of using lexicon and machine learning may be referred to as a pseudo- unsupervised technique.
  • the classification module 102 may generate two hyper planes so as to achieve maximum separation between the three categories (e.g., promoters, passives, and detractors) in the training data.
  • Figure 3 illustrates data classification as promoters, passives, and detractors, according to an example of the present disclosure.
  • the hyper planes at 300 and 302 may be converted into equations.
  • the equations may be determined as follows:
  • C1, C2, C50 are combinations of some of the relevant words.
  • the remaining data 114 i.e., data other than the training set
  • the NPS 110 may be determined by the NPS determination module 112 as follows:
  • the semantic similarity module 104 may determine semantic similarities by using natural language processing (NLP).
  • NLP is a field of computer science and linguistics pertaining to interactions between computers and human (natural) languages.
  • the semantic similarity module 104 may utilize a hierarchical framework of words to map to the human cognitive view of
  • Figure 4 illustrates a hierarchical framework of words of the semantic similarity module 104, according to an example of the present disclosure.
  • the semantic similarity module 104 may include a lexical taxonomy (i.e., a lexical database) that includes the hierarchical relationships between words and concepts.
  • the semantic similarity module 104 may include a lexical taxonomy for the English (or another) language.
  • the lexical taxonomy may function as a dictionary designed specifically for NLP.
  • the lexical taxonomy may be structured in such a way that the most related words with respect to a sense of the words are grouped closer to each other.
  • the semantic similarity between any two words may be represented as a function of distance between the words in a tree structure. This network of words may be used to estimate the NPS.
  • sentiment analysis may refer to the use of NLP to determine the attitude of an author of a comment with respect to a topic, or the overall contextual polarity of a document (e.g., data that includes a plurality of comments).
  • the semantic similarity module 104 may capture themes (e.g., nouns) that are discussed in a comment. For example, for the comments "I would suggest to take a look at this laptop. I am happy with the product as it is working great. It does not have a very good battery, though", the semantic similarity module 104 may determine "laptop", “product”, and “battery” as themes. Further, positive and negative words and expressions may describe each of these themes.
  • the semantic similarity module 104 may generate sentiment scores.
  • a comment with a positive tone may receive a positive score
  • a comment with a negative tone may receive a negative score
  • a neutral comment may receive a zero score.
  • the magnitude of the sentiment score may reflect the intensity of negativity or positivity of the context.
  • Figure 5 illustrates sentiment determination for the comments "I would suggest to take a look at this laptop. I am happy with the product as it is working great. It does not have a very good battery, though," according to an example of the present disclosure. For example, referring to Figure 5, the second comment “I am happy with the product as it is working great” is more positive than the first comment "I would suggest to take a look at this laptop," and the third comment "It does not have a very good battery, though" reflects a negative sentiment.
  • the semantic similarity module 104 may determine semantic similarities between the word "recommend” (or another word as needed) and all the adjectives and/or adverbs associated with the themes.
  • the semantic similarity module 104 may use the base word as
  • Figure 6 illustrates semantic similarity between words as a function of distance between the words in a tree structure, according to an example of the present disclosure.
  • the semantic similarity between any two words may be represented as a function of distance between the words in the tree structure.
  • the semantic similarity between the word "recommend” and “propose” is 0.88, while the semantic similarity between the words
  • Figure 7 illustrates semantic similarity determination, according to an example of the present disclosure.
  • the semantic similarities between the word "recommend” and all the adjectives and/or adverbs associated with the themes of comments 1-3 may be determined as shown.
  • the semantic similarity module 104 may determine a review recommend score (RRS) for each comment (or set of comments as shown in Figure 8).
  • the RRS may represent the sum product of the similarity scores with the theme level sentiment. For the example of Figure 7, the RRS may be determined as follows:
  • the RRS value of 2.3 may be linearly recalibrated on a scale, for example, of 0 to 10. As shown in Figure 8, the recalibrated value for the example of RRS OBS27 is 8.2. If the recalibrated RRS value is less than 6, then the author of the comment may be classified as detractor, if the RRS value is in between 6 and 9, then the author of the comment may be classified as passive, and if the RRS value is 9 or greater than 9, then the author of the comment may be classified as promoter. In the example of RRS OBS27 , the author of the comment may be classified as passive as the score, 8.2 is between 6 and 9.
  • the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1).
  • the average sentiment module 106 may determine the average sentiment of reviews to classify the reviews (or the author of the reviews) as promoters, detractors, or passives.
  • the average sentiment module 106 may utilize the sentiment analysis as described herein with reference to the semantic similarity module 104.
  • the average sentiment module 106 may determine average sentiment for each sentence (e.g., comment), or a plurality of sentences in a review.
  • the average sentiment module 106 may first identify themes in a sentence.
  • the average sentiment module 106 may then determine sentiment scores for all themes in a sentence. If a sentence has one theme, then the theme sentiment score itself may be considered as the Average Sentence Sentiment (ASS). If there is more than one theme, each theme may have a separate sentiment score. These sentiment scores may be averaged out to determine an average value, ASS. For example, considering a sentence in a review, "The printer speed is good, though the print quality is poor", the average sentiment module 106 may determine "printer speed" and "print quality" as two themes.
  • Sentiment score for each of these themes may be determined as +2 and -1 respectively.
  • the ASS for all sentences may be averaged at a review level to determine an average review sentiment (ARS).
  • ARS average review sentiment
  • Figure 9 illustrates ARS determination, according to an example of the present disclosure.
  • the average sentiment module 106 may use thresholds that generate the highest accuracy to classify an author of a review as a promoter, a detractor, or passive. If the ARS is greater than 0.3, the author may be classified as a promoter, if the ARS is negative, then the author may be classified as a detractor, and if the ARS is in between 0 and 0.3, then the author may be classified as passive. For the example of Figure 9 where the ARS value is 0.66, the author of the review may be classified as a "promoter".
  • the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1).
  • the sentiment aggregation module 108 may aggregate theme level sentiments of a review based on the frequency of theme occurrences across all reviews.
  • the sentiment aggregation module 108 may determine a weighted sum of theme level sentiment to determine an aggregated review sentiment score (ARSS).
  • the sentiment aggregation module 108 may use the ARSS to classify the reviews as promoters, detractors, or passives.
  • the sentiment aggregation module 108 may utilize the sentiment analysis as described herein with reference to the semantic similarity module 104.
  • themes across all reviews may be determined, and the frequency of occurrence of each unique theme may be plotted. For example, if there are 1000 reviews, the theme "laptop” may have appeared 800 times, the theme “battery” may have appeared 400 times, the theme “product” may have appeared 240 times, etc., for the rest of all exhaustive themes.
  • the sentiment aggregation module 108 may determine weights corresponding to each theme by taking the ratio of frequency of occurrence of a particular theme to the sum of frequencies of occurrence of all exhaustive themes. The theme occurring a maximum number of times may be assigned the highest weight.
  • the sentiment aggregation module 108 may determine a sum product of theme sentiments and their corresponding weights to generate the ARSS for that review.
  • the sentiment aggregation module 108 may use thresholds that generate the highest accuracy to classify an author of a review as a promoter, a detractor, or passive. If the ARSS is greater than 0.3, the author may be classified as a promoter, if the ARSS is negative, then the author may be classified as a detractor, and if the ARSS is in between 0 and 0.3, then the author may be classified as passive. For the example of Figure 10 where the ARSS value is 0.29, the author of the review may be classified as passive.
  • the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1 ).
  • the modules and other elements of the apparatus 100 may be machine readable instructions stored on a non-transitory computer readable medium.
  • the apparatus 100 may include or be a non-transitory computer readable medium.
  • the modules and other elements of the apparatus 100 may be hardware or a combination of machine readable instructions and hardware.
  • Figures 11 -13 respectively illustrate flowcharts of methods 1100, 1200, and 1300 for NPS determination, corresponding to the example of the NPS determination apparatus 100 whose construction is described in detail above.
  • the methods 1100, 1200, and 1300 may be implemented on the NPS determination apparatus 100 with reference to Figures 1 -10 by way of example and not limitation.
  • the methods 1100, 1200, and 1300 may be practiced in other apparatus.
  • the method may include receiving data from an internal data source and/or an external data source.
  • the classification module 102 may receive the data 114 from the internal data source 116 and/or the external data source 118.
  • the method may include determining a sentiment orientation of comments in a set of the data.
  • the classification module 102 may determine a sentiment orientation of comments in a set of the data 114.
  • the method may include scoring the sentiment orientation of the comments in the set of the data.
  • the classification module 102 may score the sentiment orientation of the comments in the set of the data 114.
  • the method may include classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set.
  • the classification module 102 may classify the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set.
  • the method may include utilizing hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors.
  • hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors.
  • classification module 102 may utilize hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors.
  • the hyper planes may be generated so as to achieve maximum separation between promoters, passives, and detractors.
  • the method may include utilizing the classified comments from the training data set and the another set of the data to determine the NPS.
  • the NPS determination module 112 may utilize the classified comments from the training data set and the another set of the data to determine the NPS 110.
  • the equations for the hyper planes may be used to classify the another set of the data into promoters, passives, and detractors.
  • the internal data source for a company may include the company's website, the company's feedback forum, and/or the company's call center
  • the external data source may include a third party online store, a third party social media source, and/or a third party
  • scoring the sentiment orientation of the comments in the set of the data may further include identifying a number of positive, negative, and neutral opinion words in each comment of the comments in the set of the data.
  • classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for the training data set may further include classifying the comments that are above an 80 th percentile and below a 20 tt percentile based on the scoring respectively as promoters and detractors for a training data set.
  • the method 1100 may further include classifying the comments in the set of the data that include an approximately zero sentiment score as passive comments.
  • the another set of the data may include all of the comments of the data excluding the comments in the set of the data.
  • the method may include receiving data from an internal data source and/or an external data source, where the data includes comments.
  • the semantic similarity module 104 may receive the data 114 from the internal data source 116 and/or the external data source 118, where the data includes comments.
  • the method may include determining a theme for each of the comments. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may determine a theme for each of the comments. If a comment includes a plurality of themes, all themes for the comment may be determined.
  • the method may include determining a sentiment score for each of the comments.
  • the semantic similarity module 104 may determine a sentiment score for each of the comments. If a comment includes a plurality of themes, sentiment scores for each of the themes in the comment may be determined.
  • the method may include determining a semantic similarity score between a predetermined word and an adjective and/or an adverb associated with the theme.
  • the semantic similarity module 104 may determine a semantic similarity score between a predetermined word and an adjective and/or an adverb associated with the theme.
  • the method may include determining a RRS for the comments based on a sum product of the semantic similarity score and the sentiment score for each of the comments.
  • the semantic similarity module 104 may determine a RRS for the comments based on a sum product of the semantic similarity score and the sentiment score for each of the comments.
  • the method may include utilizing the RRS to classify the comments to generate a NPS.
  • the NPS determination module 112 may utilize the RRS to classify the comments to generate the NPS 110.
  • a magnitude of the sentiment score may be indicative of an intensity of negativity or positivity of a comment of the comments.
  • the predetermined word may be recommend.
  • utilizing the RRS to classify the comments to generate the NPS may further include determining whether the RRS falls between predetermined ranges to classify each of the comments as detractor, passive, or promoter.
  • utilizing the RRS to classify the comments to generate the NPS may further include recalibrating the RRS to a predetermined scale to determine whether the RRS falls between predetermined ranges to classify the comments as detractor, passive, or promoter.
  • the theme may include a noun in a comment.
  • the method may include receiving data from an internal data source and/or an external data source, where the data includes sentences that form a review.
  • the average sentiment module 106 may receive the data 114 from the internal data source 116 and/or the external data source 118, where the data 114 includes sentences that form a review.
  • the method may include determining themes in each of the sentences of the review.
  • the average sentiment module 106 may determine themes in each of the sentences of the review.
  • the method may include determining sentiment scores for each of the determined themes.
  • the average sentiment module 106 may determine sentiment scores for each of the determined themes.
  • the method may include identifying sentences from the sentences of the review that have a theme.
  • the average sentiment module 106 may identify sentences from the sentences of the review that have a theme.
  • the method may include utilizing the sentiment score of a corresponding theme to determine an ASS.
  • the average sentiment module 106 may utilize the sentiment score of a corresponding theme to determine an ASS.
  • the method may include determining an ARS for the review based on the ASS of all of the identified sentences.
  • the average sentiment module 106 may determine an ARS for the review based on the ASS of all of the identified sentences.
  • the method may include utilizing the ARS to classify the sentences to generate a NPS.
  • the average sentiment module 106 may utilize the ARS to classify the sentences to generate a NPS.
  • the machine readable instructions to utilize the ARS to classify the comments to generate the NPS may further include determining whether the ARS falls between predetermined ranges to classify each of the comments as detractor, passive, or promoter.
  • Figure 14 shows a computer system 1400 that may be used with the examples described herein.
  • the computer system 1400 may represent a generic platform that includes components that may be in a server or another computer system.
  • the computer system 1400 may be used as a platform for the apparatus 100.
  • the computer system 1400 may execute, by a processor (e.g., a single or multiple processors) or other hardware processing circuit, the methods, functions and other processes described herein.
  • a computer readable medium which may be non-transitory, such as hardware storage devices (e.g., RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), hard drives, and flash memory).
  • RAM random access memory
  • ROM read only memory
  • EPROM erasable, programmable ROM
  • EEPROM electrically erasable, programmable ROM
  • hard drives e.g., hard drives, and flash memory
  • the computer system 1400 may include a processor 1402 that may implement or execute machine readable instructions performing some or all of the methods, functions and other processes described herein. Commands and data from the processor 1402 may be communicated over a communication bus 1404.
  • the computer system may also include a main memory 1406, such as a random access memory (RAM), where the machine readable instructions and data for the processor 1402 may reside during runtime, and a secondary data storage 1408, which may be non-volatile and stores machine readable instructions and data.
  • the memory and data storage are examples of computer readable mediums.
  • the memory 1406 may include a NPS determination module 1420 including machine readable instructions residing in the memory 1406 during runtime and executed by the processor 1402.
  • the NPS determination module 1420 may include the modules of the apparatus 100 shown in Figure 1 .
  • the computer system 1400 may include an I/O device 1410, such as a keyboard, a mouse, a display, etc.
  • the computer system may include a network interface 1412 for connecting to a network.
  • Other known electronic components may be added or substituted in the computer system.

Abstract

According to an example, NPS determination may include: receiving data from an internal data source and/or an external data source; determining a sentiment orientation of comments; scoring the sentiment orientation of the comments; classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set; utilizing hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors; and utilizing the classified comments from the training data set and the another set of the data to determine the NPS.

Description

NET PROMOTER SCORE DETERMINATION
BACKGROUND
[0001] Customer loyalty to a company, product, and/or service is determined by a variety of factors. For example, customer loyalty may be determined by surveys that are conducted at point of purchase. Companies may utilize a variety of resources to increase the loyalty of customers with the expectation that company objectives will be met or surpassed.
BRIEF DESCRIPTION OF DRAWINGS
[0002] Features of the present disclosure are illustrated by way of example and not limited in the following figure(s), in which like numerals indicate like elements, in which:
[0003] Figure 1 illustrates an architecture of a net promoter score (NPS) determination apparatus, according to an example of the present disclosure;
[0004] Figure 2 illustrates training set generation, according to an example of the present disclosure;
[0005] Figure 3 illustrates data classification as promoters, passives, and detractors, according to an example of the present disclosure;
[0006] Figure 4 illustrates a hierarchical framework of words, according to an example of the present disclosure;
[0007] Figure 5 illustrates sentiment determination, according to an example of the present disclosure;
[0008] Figure 6 illustrates semantic similarity between words as a function of distance between the words in a tree structure, according to an example of the present disclosure;
[0009] Figure 7 illustrates semantic similarity determination, according to an example of the present disclosure;
[0010] Figure 8 illustrates review recommend score (RRS) determination, according to an example of the present disclosure;
[0011] Figure 9 illustrates average review sentiment (ARS) determination, according to an example of the present disclosure;
[0012] Figure 10 illustrates aggregated review sentiment score (ARSS) determination, according to an example of the present disclosure;
[0013] Figure 11 illustrates a method for NPS determination, according to an example of the present disclosure;
[0014] Figure 12 illustrates further details of the method for NPS determination, according to an example of the present disclosure;
[0015] Figure 13 illustrates further details of the method for NPS determination, according to an example of the present disclosure; and
[0016] Figure 14 illustrates a computer system, according to an example of the present disclosure.
DETAILED DESCRIPTION
[0017] For simplicity and illustrative purposes, the present disclosure is described by referring mainly to examples. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be readily apparent however, that the present disclosure may be practiced without limitation to these specific details. In other instances, some methods and structures have not been described in detail so as not to unnecessarily obscure the present disclosure.
[0018] Throughout the present disclosure, the terms "a" and "an" are intended to denote at least one of a particular element. As used herein, the term "includes" means includes but not limited to, the term "including" means including but not limited to. The term "based on" means based at least in part on.
[0019] Net promoter score (NPS) is metric that is derived from a question, such as, how likely is it that a respondent would recommend a company, product, and/or service to a friend or colleague? Respondents may be categorized as promoters, passives, or detractors. Promoters may include those respondents who respond with a score of 9 or 10 (e.g., on a scale of 1 -10) and are considered loyal enthusiasts. Detractors may include those respondents who respond with a score of 0 to 6 (e.g., dissatisfied respondents). Passives may include those respondents who respond with a score of 7 or 8 (e.g., respondents who do not directly affect the NPS). NPS may be determined by subtracting the percentage of respondents who are detractors from the percentage of respondents who are promoters.
[0020] The information that is used to determine NPS may be ascertained by using surveys at predetermined time intervals (e.g., annually, semi-annually, etc.). Such surveys may attempt to cover all customer segments and geographies where a company has a presence. The surveys may be performed by undertaking a representative sampling methodology, which can be time consuming and expensive. [0021] Once a NPS based survey is completed and processed, a company may determine whether customer sentiments have changed from previous surveys. However, customer sentiments may change drastically at any given time (e.g., overnight). With a wide variety of choices that are available to customers, customers may switch to a competitor at any given time. Thus a company may need to virtually constantly engage and track its customers' changing needs. A company may also need to initiate remedial actions virtually instantaneously to drive improvements in customer loyalty, and to enable profitable growth. A company may also need to deliver real-time information to employees, so that the employees may act on customer feedback and achieve results.
[0022] In order to address the aforementioned aspects related to NPS
determination, according to examples, a NPS determination apparatus and a method for NPS determination are disclosed herein. For the apparatus and method disclosed herein, NPS may be determined in real-time. For the apparatus and method disclosed herein, the data that is used to determine the NPS may be received from perpetually available data sources that receive unsolicited feedback in the on-line space. Examples of unsolicited feedback include feedback from internal data sources such as a company's websites, feedback forums, call centers, or other company related e-commerce websites. Examples of unsolicited feedback may also include feedback from external data sources that include online stores such as AMAZON, E-BAY, etc., social media sources such as TWITTER,
FACEBOOK, BLOGS, YOUTUBE, etc., and commercial sources such as
LINKEDIN or communications with account managers.
[0023] With respect to the internal and external data sources that may be used to determine the NPS as disclosed herein, generally, such sources do not address the NPS question (i.e., how likely is it that a respondent would recommend a company, product, and/or service to a friend or colleague?). An analyst may guess from what is being told, if an author (e.g., reviewer of a company, product, and/or service) of a comment (e.g., a review comment) is willing to recommend a company, product, and/or service.
[0024] According to examples, the apparatus and method disclosed herein may receive data from the internal and external data sources, and perform a sentiment and theme analysis. The received data may be analyzed by using a dictionary to identify adverbs and adjectives. The semantic similarity of adverbs and adjectives from the data may be analyzed, for example, with respect to the word
"recommend". For example, a lexical database of English (or another language as needed) may be used to analyze the semantic similarity of adverbs and adjectives from the data to the word "recommend". A NPS score may be determined as a sum product of similarity scores and sentiments. Based on the NPS score and/or other metrics as disclosed herein, the comment and/or the author of the comment may be classified as a promoter, a detractor, or passive. Thus, the apparatus and method disclosed herein may process the sentiments derived from the data, and estimate a specific metric designated as the NPS score.
[0025] For the apparatus and method disclosed herein, the real-time NPS determination may provide for a company to operate its growth engine to operate at peak efficiency. The real-time NPS determination may help employees clarify and simplify the job of satisfying customers. The real-time NPS determination may facilitate identification and engagement with customers throughout their shopping endeavor. Further, the real-time NPS determination may also facilitate analysis of a company's performance on a frequent basis.
[0026] For the apparatus and method disclosed herein, the real-time NPS determined may be used to generate a complete operational model to drive growth for a company. The real-time NPS may be used for visibility of company sentiment and opinions. The real-time NPS may be used to generate a complete feedback loop with respect to operations of a company. Further, the real-time NPS may also provide a cost benefit in that instead of investment in intermittent surveys, the data that is used may be sourced from customer feedback and opinions available on social media and other web sources in digital unstructured format. Moreover, a company's performance and the NPS metric may be more readily equated based on the real-time NPS determination.
[0027] Figure 1 illustrates an architecture of a NPS determination apparatus (hereinafter also referred to as "apparatus 100"), according to an example of the present disclosure. Referring to Figure 1 , the apparatus 100 is depicted as including a classification module 102 to implement a classification technique, a semantic similarity module 104 to implement a semantic similarity technique, an average sentiment module 106 to implement an average sentiment technique, and a sentiment aggregation module 108 to implement a sentiment aggregation technique, where each of these techniques may be used to determine a NPS 110. For example, based on the implementation of the classification technique, the semantic similarity technique, the average sentiment technique, and/or the sentiment aggregation technique, a NPS determination module 112 may determine the NPS 110. The apparatus 100 may receive data 114 from an internal data source 116 and/or an external data source 118, where the data 114 may be used by the classification module 102, the semantic similarity module 104, the average sentiment module 106, and/or the sentiment aggregation module 108 to determine the NPS 110.
[0028] Generally, the apparatus 100 may combine sentiment analysis and word associations to analyze text (e.g., in the form of documents, comments, reviews, etc.) from the data 114, and determine if the author of the text is willing to recommend or not. The apparatus 100 may determine whether the author related to the text is a promoter, a detractor, or passive. The data 114 may include unsolicited feedback available in the on-line space. For example, the data 114 may include data from internal data sources (e.g., the internal data source 116) such as a company's websites, feedback forums, call centers, or other company related e- commerce websites. The data 114 may also include data from external data sources (e.g., the external data source 118) that include online stores such as AMAZON, E-BAY, etc., social media sources such as TWITTER, FACEBOOK, BLOGS, YOUTUBE, etc., and commercial sources such as LINKEDLN or communications with account managers.
[0029] The apparatus 100 may utilize a classification technique implemented by the classification module 102, a semantic similarity technique implemented by the semantic similarity module 104, an average sentiment technique implemented by the average sentiment module 106, and/or a sentiment aggregation technique implemented by the sentiment aggregation module 108.
[0030] With respect to the classification technique implemented by the classification module 102, the classification module 102 may implement machine learning to classify comments from the data 114 as promoter, passive, and detractor. Machine learning is a scientific discipline that explores the construction and study of machine readable instructions to learn from data. The classification module 102 may utilize, for example, support vector machine (SVM) to recognize patterns in the data 114.
[0031] In order to classify comments from the data 114, the classification module 102 may generate a training data set. The training data set may represent the data on which an analytical model is generated by the classification module 102. The classification module 102 may implement a non-supervised learning technique on the training data set. That is, the classification module 102 may utilize a training data set that is not manually determined to classify an initial set of the data 114 as promoters, detractors, or passives.
[0032] With respect to determination of the training data set, the classification module 102 may determine the sentiment orientation of every document (i.e., including a plurality of comments), text (i.e., data that forms a comment), and/or comment from the data 114 using the lexicon approach (note, references to a comment from the data 114 may similarly apply to a document and/or text from the data 114). The lexicon approach may be based on opinion words in a comment. Opinion words are words that are generally used to express positive or negative sentiments, e.g., "good" and "bad". The classification module 102 may utilize a dictionary of positive, negative, and neutral opinion words. The classification module 102 may query the dictionary to determine the sentiment orientation of the document, text, and/or comment from the data 114.
[0033] The overall sentiment of a document, text, and/or comment from the data 114 may be determined by a number of positive, negative, and neutral words in the document, text, and/or comments, respectively, from the data 114, and scored accordingly. For example, the comment, "love our new printer, and now we can print anywhere in our house with this great device" will be expected to receive a high positive sentiment score, (e.g., +5) because of the many positive words such as "love", "anywhere", and "great", whereas the comment "hate this product, and I have lost so much productivity over the print cartridges which are very expensive" would be expected to receive a high negative sentiment score (e.g., -6) because of the many negative words such as "hate", "lost", and "expensive".
[0034] Based on this sentiment scoring, the comments (and/or documents, text, etc.) from the data 114 that are above the 80th percentile for sentiment scores may be used to form the promoter training set, and the comments that are below the 20th percentile for sentiment scores may be used to form the detractor training set.
[0035] For example, Figure 2 illustrates training set generation, according to an example of the present disclosure. Referring to Figure 2, assuming that the sentiment scores range from -6 to +6, 20th percentile will correspond to -4.4, and 80th percentile will correspond to +3.4. For the example of Figure 2, comments with a sentiment score > 3.4 may be classified as promoters, and comments with a sentiment score < -4.4 may be classified as detractors. This ensures that the comments forming the promoter training set are actually positive, and the comments forming the detractor training set are actually negative.
[0036] Comments that are near the median with respect to sentiment score may not be taken into consideration as their scoring may not represent their inherent sentiment. To form the training-set for the passives, comments
corresponding to an exact zero (or approximate zero) sentiment score may be considered. From the comments corresponding to an exact zero sentiment score, comments which have a maximum number of neutral words may be selected. A sufficient quantity of such comments that have a maximum number of neutral words may be selected for the training data. For example, assuming a number of promoter training comments are 50 and a number of detractor training comments are 100, a number of the passive training comments may be selected as a number between the number of promoter training comments and the number of detractor training comments (e.g., 87). Thus the training sets for the different sentiment classes, i.e., promoters, detractors, and passives may be formed without the user providing any manual input. For the classification module 102, the combined approach of using lexicon and machine learning may be referred to as a pseudo- unsupervised technique.
[0037] The classification module 102 may generate two hyper planes so as to achieve maximum separation between the three categories (e.g., promoters, passives, and detractors) in the training data. For example, Figure 3 illustrates data classification as promoters, passives, and detractors, according to an example of the present disclosure. Referring to Figure 3, the hyper planes at 300 and 302 may be converted into equations. For example, the equations may be determined as follows:
Promoter <- 0.9558974*C1 + 0.0174228*C2 - 0.01331341*C3 +...+ 0.08439072X50≥1.2363929
Passive <- 1.2363929 > 0.9558974*01 + 0.0174228*02 - 0.01331341*C3 +...+ 0.08439072*050 > 1.0518031
Detractor <- 0.9558974*01 + 0.0174228*02 - 0.01331341*03 +...+ 0.08439072*050 < 1.0518031 For the hyper plane equations, C1, C2, C50 are combinations of some of the relevant words. Using these equations, the remaining data 114 (i.e., data other than the training set) may be classified as promoters, passives, or detractors.
[0038] In order to determine the NPS based on the classification technique implemented by the classification module 102, based on the respective number of promoters, detractors, and passives for a given time frame, the NPS 110 may be determined by the NPS determination module 112 as follows:
pjpS— (Number °f promoters-Number of Detractors) ^ Equation (1)
Total Number of comments
[0039] With respect to the semantic similarity technique implemented by the semantic similarity module 104, the semantic similarity module 104 may determine semantic similarities by using natural language processing (NLP). NLP is a field of computer science and linguistics pertaining to interactions between computers and human (natural) languages. The semantic similarity module 104 may utilize a hierarchical framework of words to map to the human cognitive view of
classification (i.e., taxonomy). Figure 4 illustrates a hierarchical framework of words of the semantic similarity module 104, according to an example of the present disclosure.
[0040] The semantic similarity module 104 may include a lexical taxonomy (i.e., a lexical database) that includes the hierarchical relationships between words and concepts. For example, the semantic similarity module 104 may include a lexical taxonomy for the English (or another) language. The lexical taxonomy may function as a dictionary designed specifically for NLP. The lexical taxonomy may be structured in such a way that the most related words with respect to a sense of the words are grouped closer to each other. The semantic similarity between any two words may be represented as a function of distance between the words in a tree structure. This network of words may be used to estimate the NPS.
[0041] With respect to the semantic similarity technique implemented by the semantic similarity module 104, sentiment analysis may refer to the use of NLP to determine the attitude of an author of a comment with respect to a topic, or the overall contextual polarity of a document (e.g., data that includes a plurality of comments). The semantic similarity module 104 may capture themes (e.g., nouns) that are discussed in a comment. For example, for the comments "I would suggest to take a look at this laptop. I am happy with the product as it is working great. It does not have a very good battery, though", the semantic similarity module 104 may determine "laptop", "product", and "battery" as themes. Further, positive and negative words and expressions may describe each of these themes. For example, "suggest" is a positive word describing "laptop", and "does not have" is a negative expression describing "battery". Based on these positive and negative words and expressions, the semantic similarity module 104 may generate sentiment scores. A comment with a positive tone may receive a positive score, a comment with a negative tone may receive a negative score, and a neutral comment may receive a zero score. The magnitude of the sentiment score may reflect the intensity of negativity or positivity of the context.
[0042] Figure 5 illustrates sentiment determination for the comments "I would suggest to take a look at this laptop. I am happy with the product as it is working great. It does not have a very good battery, though," according to an example of the present disclosure. For example, referring to Figure 5, the second comment "I am happy with the product as it is working great" is more positive than the first comment "I would suggest to take a look at this laptop," and the third comment "It does not have a very good battery, though" reflects a negative sentiment.
[0043] In order to determine semantic similarity, the semantic similarity module 104 may determine semantic similarities between the word "recommend" (or another word as needed) and all the adjectives and/or adverbs associated with the themes. The semantic similarity module 104 may use the base word as
"recommend", based on the assumption that the concept of NPS revolves around determining the degree of "likelihood to recommend" any company, product, and/or brand.
[0044] Figure 6 illustrates semantic similarity between words as a function of distance between the words in a tree structure, according to an example of the present disclosure. Referring to Figure 6, the semantic similarity between any two words may be represented as a function of distance between the words in the tree structure. For example, the semantic similarity between the word "recommend" and "propose" is 0.88, while the semantic similarity between the words
"recommend" and "exclaim" is 0.28. This is because the words closest with respect to the contextual sense they make may be grouped closer together than the ones which are less similar.
[0045] Figure 7 illustrates semantic similarity determination, according to an example of the present disclosure. Referring to Figure 7, the semantic similarities between the word "recommend" and all the adjectives and/or adverbs associated with the themes of comments 1-3 (for the example of Figure 5) may be determined as shown.
[0046] The semantic similarity module 104 may determine a review recommend score (RRS) for each comment (or set of comments as shown in Figure 8). The RRS may represent the sum product of the similarity scores with the theme level sentiment. For the example of Figure 7, the RRS may be determined as follows:
RRSOBS27 = 1 * 0.65 + 2 * 0.5 + 2 * 0.6 - 1 * 0.55 = 2.3
The RRS value of 2.3 may be linearly recalibrated on a scale, for example, of 0 to 10. As shown in Figure 8, the recalibrated value for the example of RRSOBS27 is 8.2. If the recalibrated RRS value is less than 6, then the author of the comment may be classified as detractor, if the RRS value is in between 6 and 9, then the author of the comment may be classified as passive, and if the RRS value is 9 or greater than 9, then the author of the comment may be classified as promoter. In the example of RRSOBS27, the author of the comment may be classified as passive as the score, 8.2 is between 6 and 9. [0047] In order to determine NPS based on the semantic similarity technique implemented by the semantic similarity module 104, based on the count of promoters, detractors, and passives, the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1).
[0048] With respect to the average sentiment technique implemented by the average sentiment module 106, the average sentiment module 106 may determine the average sentiment of reviews to classify the reviews (or the author of the reviews) as promoters, detractors, or passives. The average sentiment module 106 may utilize the sentiment analysis as described herein with reference to the semantic similarity module 104.
[0049] The average sentiment module 106 may determine average sentiment for each sentence (e.g., comment), or a plurality of sentences in a review. The average sentiment module 106 may first identify themes in a sentence. The average sentiment module 106 may then determine sentiment scores for all themes in a sentence. If a sentence has one theme, then the theme sentiment score itself may be considered as the Average Sentence Sentiment (ASS). If there is more than one theme, each theme may have a separate sentiment score. These sentiment scores may be averaged out to determine an average value, ASS. For example, considering a sentence in a review, "The printer speed is good, though the print quality is poor", the average sentiment module 106 may determine "printer speed" and "print quality" as two themes. Sentiment score for each of these themes may be determined as +2 and -1 respectively. The ASS for this sentence may be determined as (Sum of sentiments)/(Number of themes). In this example, the ASS may be determined to be (2-1 )/2 = 0.5. The ASS for all sentences may be averaged at a review level to determine an average review sentiment (ARS).
Figure 9 illustrates ARS determination, according to an example of the present disclosure. For the example of Figure 9, there are three sentences, each with an ASS of 1 , 2, and -1 respectively. The average sentiment module 106 may determine the ARS as (Sum of all ASSs)/(Number of sentences). In this example, the ARS may be determined to be 0.66 (i.e., (1 + 2 - 1)/3 = 0.66).
[0050] With respect to classification based on ARS, based on the distribution of ARS across reviews, the average sentiment module 106 may use thresholds that generate the highest accuracy to classify an author of a review as a promoter, a detractor, or passive. If the ARS is greater than 0.3, the author may be classified as a promoter, if the ARS is negative, then the author may be classified as a detractor, and if the ARS is in between 0 and 0.3, then the author may be classified as passive. For the example of Figure 9 where the ARS value is 0.66, the author of the review may be classified as a "promoter".
[0051] In order to determine NPS based on the average sentiment technique implemented by the average sentiment module 106, based on the count of promoters, detractors, and passives, the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1).
[0052] With respect to the sentiment aggregation technique implemented by the sentiment aggregation module 108, the sentiment aggregation module 108 may aggregate theme level sentiments of a review based on the frequency of theme occurrences across all reviews. The sentiment aggregation module 108 may determine a weighted sum of theme level sentiment to determine an aggregated review sentiment score (ARSS). The sentiment aggregation module 108 may use the ARSS to classify the reviews as promoters, detractors, or passives.
[0053] The sentiment aggregation module 108 may utilize the sentiment analysis as described herein with reference to the semantic similarity module 104.
[0054] With respect to determination of an aggregated review sentiment score, themes across all reviews may be determined, and the frequency of occurrence of each unique theme may be plotted. For example, if there are 1000 reviews, the theme "laptop" may have appeared 800 times, the theme "battery" may have appeared 400 times, the theme "product" may have appeared 240 times, etc., for the rest of all exhaustive themes. The sentiment aggregation module 108 may determine weights corresponding to each theme by taking the ratio of frequency of occurrence of a particular theme to the sum of frequencies of occurrence of all exhaustive themes. The theme occurring a maximum number of times may be assigned the highest weight.
[0055] For the example related to the 1000 reviews, assuming that the sum of frequencies of occurrence of all themes is 3000, then the sentiment aggregation module 108 may determine the weight associated with "laptop" as 800/3000=0.26. Similarly the sentiment aggregation module 108 may determine the weight associated with "battery" as 400/3000=0.13, and the weight associated with "product" is 240/3000=0.08.
[0056] Based on the determination of theme weights, for each review (e.g., including multiple comments), the sentiment aggregation module 108 may determine a sum product of theme sentiments and their corresponding weights to generate the ARSS for that review. Figure 10 illustrates ARSS determination, according to an example of the present disclosure. Referring to Figure 10, the sentiment aggregation module 108 may determine the ARSS score as follows: 1 * 0.26 + 2 * 0.08 - 1 * 013 = 0.29.
[0057] With respect to classification based on ARSS, based on the distribution of ARSS across all reviews, the sentiment aggregation module 108 may use thresholds that generate the highest accuracy to classify an author of a review as a promoter, a detractor, or passive. If the ARSS is greater than 0.3, the author may be classified as a promoter, if the ARSS is negative, then the author may be classified as a detractor, and if the ARSS is in between 0 and 0.3, then the author may be classified as passive. For the example of Figure 10 where the ARSS value is 0.29, the author of the review may be classified as passive.
[0058] In order to determine NPS based on the sentiment aggregation technique implemented by the sentiment aggregation module 108, based on the count of promoters, detractors, and passives, the NPS determination module 112 may determine the NPS 110 for a given time frame by using Equation (1 ).
[0059] The modules and other elements of the apparatus 100 may be machine readable instructions stored on a non-transitory computer readable medium. In this regard, the apparatus 100 may include or be a non-transitory computer readable medium. In addition, or alternatively, the modules and other elements of the apparatus 100 may be hardware or a combination of machine readable instructions and hardware.
[0060] Figures 11 -13 respectively illustrate flowcharts of methods 1100, 1200, and 1300 for NPS determination, corresponding to the example of the NPS determination apparatus 100 whose construction is described in detail above. The methods 1100, 1200, and 1300 may be implemented on the NPS determination apparatus 100 with reference to Figures 1 -10 by way of example and not limitation. The methods 1100, 1200, and 1300 may be practiced in other apparatus.
[0061] Referring to Figure 11 , for the method 1100, at block 1102, the method may include receiving data from an internal data source and/or an external data source. For example, referring to Figure 1 , the classification module 102 may receive the data 114 from the internal data source 116 and/or the external data source 118.
[0062] At block 1104, the method may include determining a sentiment orientation of comments in a set of the data. For example, referring to Figure 1 , the classification module 102 may determine a sentiment orientation of comments in a set of the data 114.
[0063] At block 1106, the method may include scoring the sentiment orientation of the comments in the set of the data. For example, referring to Figure 1 , the classification module 102 may score the sentiment orientation of the comments in the set of the data 114.
[0064] At block 1108, the method may include classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set. For example, referring to Figure 1 , the classification module 102 may classify the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set.
[0065] At block 1110, the method may include utilizing hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors. For example, referring to Figures 1 and 3, the
classification module 102 may utilize hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors. For example, the hyper planes may be generated so as to achieve maximum separation between promoters, passives, and detractors.
[0066] At block 1112, the method may include utilizing the classified comments from the training data set and the another set of the data to determine the NPS. For example, referring to Figure 1 , the NPS determination module 112 may utilize the classified comments from the training data set and the another set of the data to determine the NPS 110. For example, the equations for the hyper planes may be used to classify the another set of the data into promoters, passives, and detractors.
[0067] According to an example, for the method 1100, the internal data source for a company may include the company's website, the company's feedback forum, and/or the company's call center, and the external data source may include a third party online store, a third party social media source, and/or a third party
commercial resource.
[0068] According to an example, for the method 1100, scoring the sentiment orientation of the comments in the set of the data may further include identifying a number of positive, negative, and neutral opinion words in each comment of the comments in the set of the data.
[0069] According to an example, for the method 1100, classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for the training data set may further include classifying the comments that are above an 80th percentile and below a 20tt percentile based on the scoring respectively as promoters and detractors for a training data set.
[0070] According to an example, the method 1100 may further include classifying the comments in the set of the data that include an approximately zero sentiment score as passive comments.
[0071] According to an example, for the method 1100, the another set of the data may include all of the comments of the data excluding the comments in the set of the data.
[0072] Referring to Figure 12, for the method 1200, at block 1202, the method may include receiving data from an internal data source and/or an external data source, where the data includes comments. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may receive the data 114 from the internal data source 116 and/or the external data source 118, where the data includes comments.
[0073] At block 1204, the method may include determining a theme for each of the comments. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may determine a theme for each of the comments. If a comment includes a plurality of themes, all themes for the comment may be determined.
[0074] At block 1206, the method may include determining a sentiment score for each of the comments. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may determine a sentiment score for each of the comments. If a comment includes a plurality of themes, sentiment scores for each of the themes in the comment may be determined.
[0075] At block 1208, for each of the comments, the method may include determining a semantic similarity score between a predetermined word and an adjective and/or an adverb associated with the theme. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may determine a semantic similarity score between a predetermined word and an adjective and/or an adverb associated with the theme.
[0076] At block 1210, the method may include determining a RRS for the comments based on a sum product of the semantic similarity score and the sentiment score for each of the comments. For example, referring to Figures 1 and 4-8, the semantic similarity module 104 may determine a RRS for the comments based on a sum product of the semantic similarity score and the sentiment score for each of the comments.
[0077] At block 1212, the method may include utilizing the RRS to classify the comments to generate a NPS. For example, referring to Figures 1 and 4-8, the NPS determination module 112 may utilize the RRS to classify the comments to generate the NPS 110.
[0078] According to an example, for the method 1200, a magnitude of the sentiment score may be indicative of an intensity of negativity or positivity of a comment of the comments.
[0079] According to an example, for the method 1200, the predetermined word may be recommend.
[0080] According to an example, for the method 1200, utilizing the RRS to classify the comments to generate the NPS may further include determining whether the RRS falls between predetermined ranges to classify each of the comments as detractor, passive, or promoter.
[0081] According to an example, for the method 1200, utilizing the RRS to classify the comments to generate the NPS may further include recalibrating the RRS to a predetermined scale to determine whether the RRS falls between predetermined ranges to classify the comments as detractor, passive, or promoter. [0082] According to an example, for the method 1200, the theme may include a noun in a comment.
[0083] Referring to Figure 13, for the method 1300, at block 1302, the method may include receiving data from an internal data source and/or an external data source, where the data includes sentences that form a review. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may receive the data 114 from the internal data source 116 and/or the external data source 118, where the data 114 includes sentences that form a review.
[0084] At block 1304, the method may include determining themes in each of the sentences of the review. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may determine themes in each of the sentences of the review.
[0085] At block 1306, the method may include determining sentiment scores for each of the determined themes. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may determine sentiment scores for each of the determined themes.
[0086] At block 1308, the method may include identifying sentences from the sentences of the review that have a theme. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may identify sentences from the sentences of the review that have a theme.
[0087] At block 1310, for each of the identified sentences, the method may include utilizing the sentiment score of a corresponding theme to determine an ASS. For example, referring to Figures 1 and 4-9, for each of the identified sentences, the average sentiment module 106 may utilize the sentiment score of a corresponding theme to determine an ASS.
[0088] At block 1312, the method may include determining an ARS for the review based on the ASS of all of the identified sentences. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may determine an ARS for the review based on the ASS of all of the identified sentences.
[0089] At block 1314, the method may include utilizing the ARS to classify the sentences to generate a NPS. For example, referring to Figures 1 and 4-9, the average sentiment module 106 may utilize the ARS to classify the sentences to generate a NPS.
[0090] According to an example, for the method 1300, the machine readable instructions to utilize the ARS to classify the comments to generate the NPS may further include determining whether the ARS falls between predetermined ranges to classify each of the comments as detractor, passive, or promoter.
[0091] Figure 14 shows a computer system 1400 that may be used with the examples described herein. The computer system 1400 may represent a generic platform that includes components that may be in a server or another computer system. The computer system 1400 may be used as a platform for the apparatus 100. The computer system 1400 may execute, by a processor (e.g., a single or multiple processors) or other hardware processing circuit, the methods, functions and other processes described herein. These methods, functions and other processes may be embodied as machine readable instructions stored on a computer readable medium, which may be non-transitory, such as hardware storage devices (e.g., RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), hard drives, and flash memory).
[0092] The computer system 1400 may include a processor 1402 that may implement or execute machine readable instructions performing some or all of the methods, functions and other processes described herein. Commands and data from the processor 1402 may be communicated over a communication bus 1404. The computer system may also include a main memory 1406, such as a random access memory (RAM), where the machine readable instructions and data for the processor 1402 may reside during runtime, and a secondary data storage 1408, which may be non-volatile and stores machine readable instructions and data. The memory and data storage are examples of computer readable mediums. The memory 1406 may include a NPS determination module 1420 including machine readable instructions residing in the memory 1406 during runtime and executed by the processor 1402. The NPS determination module 1420 may include the modules of the apparatus 100 shown in Figure 1 .
[0093] The computer system 1400 may include an I/O device 1410, such as a keyboard, a mouse, a display, etc. The computer system may include a network interface 1412 for connecting to a network. Other known electronic components may be added or substituted in the computer system.
[0094] What has been described and illustrated herein is an example along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Many variations are possible within the spirit and scope of the subject matter, which is intended to be defined by the following claims - and their equivalents -- in which all terms are meant in their broadest reasonable sense unless otherwise indicated.

Claims

What is claimed is:
1 . A method for net promoter score (NPS) determination, the method
comprising: receiving data from at least one of an internal data source and an external data source; determining a sentiment orientation of comments in a set of the data; scoring the sentiment orientation of the comments in the set of the data; classifying, by a processor, the comments that are above and below
predetermined thresholds based on the scoring respectively as promoters and detractors for a training data set; utilizing hyper planes generated from the training data set to classify comments in another set of the data as promoters and detractors; and utilizing the classified comments from the training data set and the another set of the data to determine the NPS.
2. The method of claim 1 , wherein the internal data source for a company includes at least one of the company's website, the company's feedback forum, and the company's call center, and the external data source includes a third party online store, a third party social media source, and a third party commercial resource.
3. The method of claim 1 , wherein scoring the sentiment orientation of the comments in the set of the data further comprises: identifying a number of positive, negative, and neutral opinion words in each comment of the comments in the set of the data.
4. The method of claim 1 , wherein classifying the comments that are above and below predetermined thresholds based on the scoring respectively as promoters and detractors for the training data set further comprises: classifying the comments that are above an 80th percentile and below a 20th percentile based on the scoring respectively as promoters and detractors for a training data set.
5. The method of claim 1 , further comprising: classifying the comments in the set of the data that include an approximately zero sentiment score as passive comments.
6. The method of claim 1 , wherein the another set of the data includes all of the comments of the data excluding the comments in the set of the data.
7. A net promoter score (NPS) determination apparatus comprising: a processor; and a memory storing machine readable instructions that when executed by the processor cause the processor to: receive data from at least one of an internal data source and an external data source, wherein the data includes comments; determine a theme for each of the comments; determine a sentiment score for each of the comments; for each of the comments, determine a semantic similarity score between a predetermined word and at least one of an adjective and an adverb associated with the theme; determine a review recommend score (RRS) for the comments based on a sum product of the semantic similarity score and the sentiment score for each of the comments; and utilize the RRS to classify the comments to generate a NPS.
8. The NPS determination apparatus according to claim 7, wherein a magnitude of the sentiment score is indicative of an intensity of negativity or positivity of a comment of the comments.
9. The NPS determination apparatus according to claim 7, wherein the
predetermined word is recommend.
10. The NPS determination apparatus according to claim 7, wherein the machine readable instructions to utilize the RRS to classify the comments to generate the NPS further comprise instructions to: determine whether the RRS falls between predetermined ranges to classify each of the comments as detractor, passive, or promoter.
11 . The NPS determination apparatus according to claim 7, wherein the machine readable instructions to utilize the RRS to classify the comments to generate the NPS further comprise instructions to: recalibrate the RRS to a predetermined scale to determine whether the RRS falls between predetermined ranges to classify the comments as detractor, passive, or promoter.
12. The NPS determination apparatus according to claim 7, wherein the theme includes a noun in a comment.
13. A non-transitory computer readable medium having stored thereon machine readable instructions to provide net promoter score (NPS) determination, the machine readable instructions, when executed, cause a processor to: receive data from at least one of an internal data source and an external data source, wherein the data includes sentences that form a review; determine themes in each of the sentences of the review; determine sentiment scores for each of the determined themes; identify sentences from the sentences of the review that have a theme; for each of the identified sentences, utilize the sentiment score of a
corresponding theme to determine an average sentence sentiment (ASS); determine an average review sentiment (ARS) for the review based on the ASS of all of the identified sentences; and utilize the ARS to classify the sentences to generate a NPS.
14. The non-transitory computer readable medium according to claim 13, wherein the machine readable instructions to utilize the ARS to classify the sentences to generate the NPS further comprise instructions to: determine whether the ARS falls between predetermined ranges to classify each of the sentences as detractor, passive, or promoter.
15. The non-transitory computer readable medium according to claim 13, wherein the internal data source for a company includes at least one of the company's website, the company's feedback forum, and the company's call center, and the external data source includes a third party online store, a third party social media source, and a third party commercial resource.
PCT/US2015/013540 2015-01-29 2015-01-29 Net promoter score determination WO2016122532A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2015/013540 WO2016122532A1 (en) 2015-01-29 2015-01-29 Net promoter score determination

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2015/013540 WO2016122532A1 (en) 2015-01-29 2015-01-29 Net promoter score determination

Publications (1)

Publication Number Publication Date
WO2016122532A1 true WO2016122532A1 (en) 2016-08-04

Family

ID=56543955

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/013540 WO2016122532A1 (en) 2015-01-29 2015-01-29 Net promoter score determination

Country Status (1)

Country Link
WO (1) WO2016122532A1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107358346A (en) * 2017-07-03 2017-11-17 中国联合网络通信集团有限公司 It is directed to the evaluation information treating method and apparatus of communication quality
CN109299463A (en) * 2018-09-26 2019-02-01 武汉斗鱼网络科技有限公司 A kind of calculation method and relevant device of emotion score
CN109685560A (en) * 2018-12-17 2019-04-26 泰康保险集团股份有限公司 Big data processing method, device, medium and electronic equipment
WO2020006206A1 (en) * 2018-06-28 2020-01-02 LANGUAGE LOGIC, LLC d.b.a. ASCRIBE Systems and methods for automatically detecting sentiments and assigning and analyzing quantitative values to the sentiments expressed in text
WO2020055487A1 (en) * 2018-09-11 2020-03-19 Microsoft Technology Licensing, Llc Sentiment analysis of net promoter score (nps) verbatims
US10878196B2 (en) 2018-10-02 2020-12-29 At&T Intellectual Property I, L.P. Sentiment analysis tuning
US10977670B2 (en) 2018-01-23 2021-04-13 Mass Minority Inc. Method and system for determining and monitoring brand performance based on paid expenditures
CN113674031A (en) * 2021-08-30 2021-11-19 广州快决测信息科技有限公司 System and method for analyzing net recommended value questionnaire data
US11657414B1 (en) 2018-06-21 2023-05-23 Optum, Inc. Systems and methods for quantitatively predicting changes to employee net promoter scores

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080133488A1 (en) * 2006-11-22 2008-06-05 Nagaraju Bandaru Method and system for analyzing user-generated content
US20090265332A1 (en) * 2008-04-18 2009-10-22 Biz360 Inc. System and Methods for Evaluating Feature Opinions for Products, Services, and Entities
US20130066873A1 (en) * 2011-09-14 2013-03-14 Microsoft Corporation Automatic generation of digital composite product reviews

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080133488A1 (en) * 2006-11-22 2008-06-05 Nagaraju Bandaru Method and system for analyzing user-generated content
US20090265332A1 (en) * 2008-04-18 2009-10-22 Biz360 Inc. System and Methods for Evaluating Feature Opinions for Products, Services, and Entities
US20130066873A1 (en) * 2011-09-14 2013-03-14 Microsoft Corporation Automatic generation of digital composite product reviews

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ALAA HAMOUDA ET AL.: "Reviews Classification Using SentiWordNet Lexicon", THE ONLINE JOURNAL ON COMPUTER SCIENCE AND INFORMATION TECHNOLOGY(OJCSIT, vol. 2, no. 1, 2011, pages 120 - 123 *
SVETLANA KIRITCHENKO ET AL.: "Sentiment Analysis of Short Informal Texts", JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, vol. 50, August 2014 (2014-08-01), pages 723 - 762 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107358346A (en) * 2017-07-03 2017-11-17 中国联合网络通信集团有限公司 It is directed to the evaluation information treating method and apparatus of communication quality
CN107358346B (en) * 2017-07-03 2020-09-08 中国联合网络通信集团有限公司 Evaluation information processing method and device for communication quality
US10977670B2 (en) 2018-01-23 2021-04-13 Mass Minority Inc. Method and system for determining and monitoring brand performance based on paid expenditures
US11657414B1 (en) 2018-06-21 2023-05-23 Optum, Inc. Systems and methods for quantitatively predicting changes to employee net promoter scores
WO2020006206A1 (en) * 2018-06-28 2020-01-02 LANGUAGE LOGIC, LLC d.b.a. ASCRIBE Systems and methods for automatically detecting sentiments and assigning and analyzing quantitative values to the sentiments expressed in text
US11062094B2 (en) 2018-06-28 2021-07-13 Language Logic, Llc Systems and methods for automatically detecting sentiments and assigning and analyzing quantitate values to the sentiments expressed in text
WO2020055487A1 (en) * 2018-09-11 2020-03-19 Microsoft Technology Licensing, Llc Sentiment analysis of net promoter score (nps) verbatims
CN109299463A (en) * 2018-09-26 2019-02-01 武汉斗鱼网络科技有限公司 A kind of calculation method and relevant device of emotion score
US10878196B2 (en) 2018-10-02 2020-12-29 At&T Intellectual Property I, L.P. Sentiment analysis tuning
CN109685560A (en) * 2018-12-17 2019-04-26 泰康保险集团股份有限公司 Big data processing method, device, medium and electronic equipment
CN113674031A (en) * 2021-08-30 2021-11-19 广州快决测信息科技有限公司 System and method for analyzing net recommended value questionnaire data

Similar Documents

Publication Publication Date Title
Kumar et al. Opinion mining and sentiment analysis on online customer review
US11301632B2 (en) Systems and methods for natural language processing and classification
Khan et al. Sentiment analysis and the complex natural language
WO2016122532A1 (en) Net promoter score determination
US11080304B2 (en) Feature vector profile generation for interviews
US10282737B2 (en) Analyzing sentiment in product reviews
US11574026B2 (en) Analytics-driven recommendation engine
Mukherjee et al. Effect of negation in sentences on sentiment analysis and polarity detection
Cataldi et al. Good location, terrible food: detecting feature sentiment in user-generated reviews
Ashok et al. A personalized recommender system using Machine Learning based Sentiment Analysis over social data
US11030533B2 (en) Method and system for generating a transitory sentiment community
Tuarob et al. A product feature inference model for mining implicit customer preferences within large scale social media networks
Fachrina et al. Aspect-sentiment classification in opinion mining using the combination of rule-based and machine learning
Irawaty et al. Vectorizer comparison for sentiment analysis on social media youtube: A case study
US20210056982A1 (en) Sentimental and emotional analysis of electronic documents
Suryana et al. Dynamic convolutional neural network for eliminating item sparse data on recommender system.
Shamim et al. Critical product features’ identification using an opinion analyzer
Nithya et al. Correlation of feature score to overall sentiment score for identifying the promising features
Fiarni et al. Implementing rule-based and naive bayes algorithm on incremental sentiment analysis system for Indonesian online transportation services review
Li Short or long review?-text analytics and machine learning approaches to online reputation
Al-Bnd et al. Sentiment analysis and opinion mining via microblogging in social media like: twitter
US20200184521A1 (en) Method and system for initiating an interface concurrent with generation of a transitory sentiment community
US11113306B1 (en) Methods and apparatus for person-centric multichannel opinion mining in data lakes
Rodrigues et al. Aspect Based Sentiment Analysis on Product Reviews
Aishwarya et al. Summarization and Prioritization of Amazon Reviews based on multi-level credibility attributes

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15880428

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15880428

Country of ref document: EP

Kind code of ref document: A1