CN111859985B - AI customer service model test method and device, electronic equipment and storage medium - Google Patents

AI customer service model test method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN111859985B
CN111859985B CN202010719768.XA CN202010719768A CN111859985B CN 111859985 B CN111859985 B CN 111859985B CN 202010719768 A CN202010719768 A CN 202010719768A CN 111859985 B CN111859985 B CN 111859985B
Authority
CN
China
Prior art keywords
corpus
customer service
service model
data
test
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010719768.XA
Other languages
Chinese (zh)
Other versions
CN111859985A (en
Inventor
宫雪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Huaqi Information Technology Co ltd
Shenzhen Lian Intellectual Property Service Center
Original Assignee
Shanghai Huaqi Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Huaqi Information Technology Co ltd filed Critical Shanghai Huaqi Information Technology Co ltd
Priority to CN202010719768.XA priority Critical patent/CN111859985B/en
Publication of CN111859985A publication Critical patent/CN111859985A/en
Application granted granted Critical
Publication of CN111859985B publication Critical patent/CN111859985B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G06F16/355—Class or cluster creation or modification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/01—Customer relationship services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0282—Rating or review of business operators or products
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Economics (AREA)
  • Health & Medical Sciences (AREA)
  • Marketing (AREA)
  • Computational Linguistics (AREA)
  • General Business, Economics & Management (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Game Theory and Decision Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to artificial intelligence, and provides an AI customer service model test method, an apparatus, an electronic device and a storage medium, which can obtain pre-configured standard corpus comprising input standard corpus and output standard corpus from a corpus database, analyze the similarity of the standard corpus based on a semantic similarity algorithm to obtain classified corpus, enable subsequent tests to be more targeted, expand the classified corpus based on a preset word stock, generate test samples comprising input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus, enable coverage of the test samples to be more comprehensive, effectively solve the problem of insufficient test data, input the input data into an AI customer service model to be tested to obtain output data, call the output data and the expected data with a configuration script, output test results of the AI customer service model to be tested, and further realize rapid automatic test of the AI customer service model. The application also relates to a block chain technology, and test results can be stored in the block chain.

Description

AI customer service model test method and device, electronic equipment and storage medium
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to an AI customer service model test method and device, electronic equipment and a storage medium.
Background
In the prior art, the accuracy of the output result of the AI (Artificial Intelligence ) customer service model is generally evaluated by adopting a manual marking mode, so that the time consumption is long, and hysteresis can be generated due to uncontrollability of manual operation.
Disclosure of Invention
In view of the above, it is necessary to provide a method, a device, an electronic device and a storage medium for testing an AI customer service model, which can test the AI customer service model purposefully based on a semantic similarity algorithm, and the coverage of a test sample is more comprehensive, so that the problem of insufficient test data is effectively solved, and the rapid automatic test of the AI customer service model is realized based on an artificial intelligence means.
An AI customer service model test method, comprising:
when a test instruction for an AI customer service model to be tested is received, obtaining a pre-configured standard corpus from a corpus, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
performing similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
Expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
inputting the input data into the AI customer service model to be tested to obtain output data;
and calling the output data and the expected data by using a configuration script, and outputting a test result of the AI customer service model to be tested.
According to a preferred embodiment of the present invention, the performing similarity analysis on the standard corpus based on the semantic similarity algorithm, to obtain the classified corpus includes:
converting the standard corpus into semantic vectors based on natural language processing;
calculating cosine distances among word vectors in the semantic vectors by adopting a cosine similarity algorithm;
and classifying each word vector according to the cosine distance to obtain the classified corpus.
According to a preferred embodiment of the present invention, the expanding the classified corpus based on a preset word stock includes:
for each target word vector in the classified corpus, calculating the similarity between the target word vector and the word vector in the preset word stock;
Acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector;
and adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample.
According to a preferred embodiment of the present invention, the invoking the output data and the expected data with a configuration script, and outputting the test result of the AI customer service model to be tested includes:
writing the output data and the expected data into Excel by adopting POI to generate an Excel file;
determining the file name of the excel file and specifying interface parameters of an interface;
modifying the configuration script with the file name and the interface parameter;
and calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
According to a preferred embodiment of the present invention, the test result includes an accuracy rate, and the AI customer service model test method further includes:
acquiring a first number of data in the output data, wherein the similarity between the data and the expected data is greater than or equal to a first similarity;
determining a second amount of the output data;
Calculating a quotient of the first number and the second number as the accuracy.
According to a preferred embodiment of the present invention, the test result further includes a recall rate, and the AI customer service model test method further includes:
for first output data under any category, determining first expected data corresponding to the first output data;
acquiring a third number of data in the first output data, wherein the similarity between the data and the first expected data is greater than or equal to a second similarity;
determining a fourth amount of the first output data;
and calculating the quotient of the third quantity and the fourth quantity as the recall rate under the arbitrary category.
According to a preferred embodiment of the present invention, the AI customer service model test method further includes:
and when the AI customer service model to be detected is detected to be updated, the detection of the AI customer service model to be detected is re-executed.
An AI customer service model test apparatus, the AI customer service model test apparatus comprising:
the acquisition unit is used for acquiring a pre-configured standard corpus from a corpus when a test instruction of an AI customer service model to be tested is received, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
The analysis unit is used for carrying out similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
the expansion unit is used for expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
the input unit is used for inputting the input data into the AI customer service model to be tested to obtain output data;
and the test unit is used for calling the output data and the expected data by using a configuration script and outputting a test result of the AI customer service model to be tested.
According to a preferred embodiment of the invention, the analysis unit is specifically adapted to:
converting the standard corpus into semantic vectors based on natural language processing;
calculating cosine distances among word vectors in the semantic vectors by adopting a cosine similarity algorithm;
and classifying each word vector according to the cosine distance to obtain the classified corpus.
According to a preferred embodiment of the invention, the expansion unit is specifically configured to:
for each target word vector in the classified corpus, calculating the similarity between the target word vector and the word vector in the preset word stock;
Acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector;
and adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample.
According to a preferred embodiment of the invention, the test unit is specifically adapted to:
writing the output data and the expected data into Excel by adopting POI to generate an Excel file;
determining the file name of the excel file and specifying interface parameters of an interface;
modifying the configuration script with the file name and the interface parameter;
and calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
According to a preferred embodiment of the present invention, the test result includes an accuracy rate, and the obtaining unit is further configured to obtain a first number of data having a similarity with the expected data greater than or equal to a first similarity in the output data;
the AI customer service model test device further comprises:
a determining unit configured to determine a second number of the output data;
and the calculating unit is used for calculating the quotient of the first quantity and the second quantity as the accuracy rate.
According to a preferred embodiment of the present invention, the test result further includes a recall rate, and the determining unit is configured to determine, for first output data under any category, first expected data corresponding to the first output data;
the acquiring unit is further configured to acquire a third number of data in the first output data, where the similarity with the first expected data is greater than or equal to a second similarity;
the determining unit is further configured to determine a fourth amount of the first output data;
the calculating unit is further configured to calculate a quotient of the third number and the fourth number as a recall rate under the arbitrary category.
According to a preferred embodiment of the present invention, the test unit is further configured to re-execute the detection of the AI customer service model to be tested when it is detected that the AI customer service model to be tested is updated.
An electronic device, the electronic device comprising:
a memory storing at least one instruction; and
And the processor executes the instructions stored in the memory to realize the AI customer service model test method.
A computer-readable storage medium having stored therein at least one instruction that is executed by a processor in an electronic device to implement the AI customer service model test method.
According to the technical scheme, when a test instruction for the AI customer service model to be tested is received, the pre-configured standard corpus is obtained from the corpus, the standard corpus comprises the input standard corpus and the output standard corpus, similarity analysis is carried out on the standard corpus based on a semantic similarity algorithm to obtain the classified corpus, subsequent tests are more targeted, the classified corpus is further expanded based on a preset word stock to generate a test sample, the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus, the coverage of the generated test sample is more comprehensive, the problem of insufficient test data can be effectively solved, the input data is input into the AI customer service model to be tested, the output data and the expected data are called by a configuration script, the test result of the AI customer service model to be tested is output, and further the AI customer service model to be tested is rapidly and automatically tested.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of the AI customer service model test method of the invention.
FIG. 2 is a functional block diagram of a preferred embodiment of the AI customer service model test apparatus of the invention.
Fig. 3 is a schematic structural diagram of an electronic device according to a preferred embodiment of the present invention for implementing the AI customer service model test method.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a flow chart of a preferred embodiment of the AI customer service model test method of the invention. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs.
The AI customer service model test method is applied to one or more electronic devices, wherein the electronic devices are devices capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware comprises, but is not limited to, a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device and the like.
The electronic device may be any electronic product that can interact with a user in a human-computer manner, such as a personal computer, tablet computer, smart phone, personal digital assistant (Personal Digital Assistant, PDA), game console, interactive internet protocol television (Internet Protocol Television, IPTV), smart wearable device, etc.
The electronic device may also include a network device and/or a user device. Wherein the network device includes, but is not limited to, a single network server, a server group composed of a plurality of network servers, or a Cloud based Cloud Computing (Cloud Computing) composed of a large number of hosts or network servers.
The network in which the electronic device is located includes, but is not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN), and the like.
S10, when a test instruction of the AI customer service model to be tested is received, a pre-configured standard corpus is obtained from a corpus.
In at least one embodiment of the present invention, the test instruction of the AI customer service model to be tested may be triggered by related staff, or may be automatically triggered according to a preset test time according to actual use.
In at least one embodiment of the present invention, the corpus may be a preconfigured database having a plurality of corpora.
In at least one embodiment of the present invention, the standard corpus includes an input standard corpus and an output standard corpus.
The AI (Artificial Intelligence ) customer service model to be tested is used for automatically solving questions of customers.
In practical applications, questions of users are various, and taking a loan scenario as an example, the users may ask questions of various links involved in loan, for example: interest rate problems, pay-out problems, repayment problems, etc., and may even include various tune-up statements, etc.
For various scenes classified in this way, firstly, during initial modeling, classifying each scene, collecting real questions of a user, and using standard answers of the model to ensure that the answers meet the company's unified telephone operation, have no legal risk and the like, and enter a testing stage after modeling is completed.
Specifically, the AI customer service model can give automatic solutions to the user questions under different scenes, so that the accuracy of the AI customer service model solutions is particularly important, and the AI customer service model needs to be tested.
In at least one embodiment of the present invention, the input standard corpus refers to a collected standardized corpus input by a user.
In at least one embodiment of the present invention, the output standard corpus refers to a standard answer corpus to the input standard corpus.
S11, carrying out similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus.
Specifically, the similarity analysis is performed on the standard corpus based on the semantic similarity algorithm, and obtaining the classified corpus includes:
converting the standard corpus into semantic vectors based on natural language processing;
further adopting a cosine similarity algorithm to calculate cosine distances among word vectors in the semantic vectors;
and the electronic equipment classifies each word vector according to the cosine distance to obtain the classified corpus.
Through the implementation mode, the standard corpus can be classified, so that the subsequent test is more targeted.
S12, expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus.
Specifically, the electronic device expands the classified corpus based on a preset word stock, and generating a test sample includes:
for each target word vector in the classified corpus, the electronic equipment calculates the similarity between the target word vector and the word vector in the preset word stock;
acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector;
And adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample.
For example: for the corpus "how much your company interest," the corpus includes word vectors corresponding to the following words: "your", "company", "interest" and "how much", wherein the word vector "company" and the word vector "interest" are keywords, the electronic device can select the "unit" of the close meaning word of "company" and the like, and the "like" of interest "and" interest rate ", and the like, through similarity calculation, and then the corpus" how much you company interest "can include the expansion corpus" how much you unit interest rate ", and the like.
Through the implementation mode, the electronic equipment can enable the coverage of the generated test sample to be more comprehensive by expanding the classification corpus, and further can realize more accurate test of the AI customer service model to be tested.
In addition, the existing corpus is mainly based on production data, the data size is limited, and the problem of insufficient test data can be effectively solved by the embodiment.
S13, inputting the input data into the AI customer service model to be tested to obtain output data.
In this embodiment, the AI customer service model to be tested can respond to the input data to output voice.
The output data is an output result obtained by the AI customer service model to be tested responding to the input data, and belongs to automatic solutions of the models.
For example: when the input data is 'how much the income of the loan is', the output data 'how much the income of the loan is 50 ten thousand' is obtained through the automatic processing such as the analysis and the matching of the AI customer service model to be tested.
S14, calling the output data and the expected data by using a configuration script, and outputting a test result of the AI customer service model to be tested.
Specifically, the invoking the output data and the expected data with the configuration script, and outputting the test result of the AI customer service model to be tested includes:
the electronic equipment writes the output data and the expected data into Excel by adopting POI to generate an Excel file;
further, the electronic device determines the file name of the excel file and interface parameters of a designated interface;
further, the electronic device modifies the configuration script with the file name and the interface parameter;
And calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
Wherein the interface parameters may include, but are not limited to: server IP (Internet Protocol ), port, test URI (Uniform Resource Identifier ), etc.
POIs are tools for reading and writing documents of office, and components (such as HSSF and XSS) in POIs can read and write excel.
The configuration script can be preconfigured so as to be convenient for directly calling after modifying related parameters, thereby improving the operation efficiency and simplifying the development flow.
Through the embodiment, the excel file is generated by combining with the POI technology, and the generated excel file is further called by the modified configuration script, so that the test result of the AI customer service model to be tested is output, and the rapid automatic test can be realized.
In at least one embodiment of the present invention, the test results include an accuracy rate, the method further comprising:
acquiring a first number of data in the output data, wherein the similarity between the data and the expected data is greater than or equal to a first similarity;
determining a second amount of the output data;
Further, the electronic device calculates a quotient of the first number and the second number as the accuracy.
For example: there are 10000 test samples for a test, 5000 correct output data, and accuracy rate=5000/10000=50%.
In at least one embodiment of the invention, the test results further comprise a recall, the method further comprising:
for first output data under any category, determining first expected data corresponding to the first output data;
acquiring a third number of data in the first output data, wherein the similarity between the data and the first expected data is greater than or equal to a second similarity;
determining a fourth amount of the first output data;
and further calculating the quotient of the third quantity and the fourth quantity as the recall rate under the arbitrary category.
Through the implementation mode, the accuracy and recall rate of the AI customer service model to be detected can be calculated, so that the AI customer service model to be detected can be effectively detected.
In at least one embodiment of the present invention, the AI customer service model test method further includes:
and when the AI customer service model to be detected is detected to be updated, the detection of the AI customer service model to be detected is re-executed.
Because the adjustment of the threshold value can affect the conditions of each classification in the optimization process of the model, such as the improvement of the accuracy rate of interest problems, the accuracy rate of user operation types can be reduced, but the accuracy rate of high-frequency problems of users needs to be in an ascending trend, the AI customer service model to be detected needs to be re-detected, so that the AI customer service model to be detected can be guaranteed to respond optimally to the questions of the users.
It should be noted that, in order to improve the security and privacy of data, the test results may be stored in the blockchain.
According to the technical scheme, when a test instruction for the AI customer service model to be tested is received, the pre-configured standard corpus is obtained from the corpus, the standard corpus comprises the input standard corpus and the output standard corpus, similarity analysis is carried out on the standard corpus based on a semantic similarity algorithm to obtain the classified corpus, subsequent tests are more targeted, the classified corpus is further expanded based on a preset word stock to generate a test sample, the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus, the coverage of the generated test sample is more comprehensive, the problem of insufficient test data can be effectively solved, the input data is input into the AI customer service model to be tested, the output data and the expected data are called by a configuration script, the test result of the AI customer service model to be tested is output, and further the AI customer service model to be tested is rapidly and automatically tested.
FIG. 2 is a functional block diagram of the AI customer service model test apparatus of a preferred embodiment of the invention. The AI customer service model test device 11 includes an acquisition unit 110, an analysis unit 111, an expansion unit 112, an input unit 113, a test unit 114, a determination unit 115, and a calculation unit 116. The module/unit referred to in the present invention refers to a series of computer program segments capable of being executed by the processor 13 and of performing a fixed function, which are stored in the memory 12. In the present embodiment, the functions of the respective modules/units will be described in detail in the following embodiments.
When receiving a test instruction for the AI customer service model to be tested, the obtaining unit 110 obtains a pre-configured standard corpus from the corpus.
In at least one embodiment of the present invention, the test instruction of the AI customer service model to be tested may be triggered by related staff, or may be automatically triggered according to a preset test time according to actual use.
In at least one embodiment of the present invention, the corpus may be a preconfigured database having a plurality of corpora.
In at least one embodiment of the present invention, the standard corpus includes an input standard corpus and an output standard corpus.
The AI (Artificial Intelligence ) customer service model to be tested is used for automatically solving questions of customers.
In practical applications, questions of users are various, and taking a loan scenario as an example, the users may ask questions of various links involved in loan, for example: interest rate problems, pay-out problems, repayment problems, etc., and may even include various tune-up statements, etc.
For various scenes classified in this way, firstly, during initial modeling, classifying each scene, collecting real questions of a user, and using standard answers of the model to ensure that the answers meet the company's unified telephone operation, have no legal risk and the like, and enter a testing stage after modeling is completed.
Specifically, the AI customer service model can give automatic solutions to the user questions under different scenes, so that the accuracy of the AI customer service model solutions is particularly important, and the AI customer service model needs to be tested.
In at least one embodiment of the present invention, the input standard corpus refers to a collected standardized corpus input by a user.
In at least one embodiment of the present invention, the output standard corpus refers to a standard answer corpus to the input standard corpus.
The analysis unit 111 performs similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus.
Specifically, the analyzing unit 111 performs similarity analysis on the standard corpus based on a semantic similarity algorithm, and the obtaining the classified corpus includes:
converting the standard corpus into semantic vectors based on natural language processing;
further adopting a cosine similarity algorithm to calculate cosine distances among word vectors in the semantic vectors;
the analysis unit 111 classifies each word vector according to the cosine distance to obtain the classified corpus.
Through the implementation mode, the standard corpus can be classified, so that the subsequent test is more targeted.
Further, the expansion unit 112 expands the classified corpus based on a preset word stock to generate a test sample.
The test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus.
Specifically, the expanding unit 112 expands the classified corpus based on a preset word stock, and generating the test sample includes:
for each target word vector in the classified corpus, the expansion unit 112 calculates the similarity between the target word vector and the word vector in the preset word stock;
Acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector;
and adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample.
For example: for the corpus "how much your company interest," the corpus includes word vectors corresponding to the following words: "your", "company", "interest" and "how much", wherein the word vector "company" and the word vector "interest" are keywords, the expansion unit 112 may select the "unit" of the close meaning word of "company" and so on, and the "income", "interest rate" of the similar word of "interest" and so on through similarity calculation, and then the corpus "how much" of interest of your company may include the expansion corpus "how much" of your unit interest, and so on.
Through the above embodiment, the expansion unit 112 may enable the coverage of the generated test sample to be more comprehensive by expanding the classification corpus, so as to implement more accurate test of the AI customer service model to be tested.
In addition, the existing corpus is mainly based on production data, the data size is limited, and the problem of insufficient test data can be effectively solved by the embodiment.
The input unit 113 inputs the input data into the AI customer service model to be tested, and obtains output data.
In this embodiment, the AI customer service model to be tested can respond to the input data to output voice.
The output data is an output result obtained by the AI customer service model to be tested responding to the input data, and belongs to automatic solutions of the models.
For example: when the input data is 'how much the income of the loan is', the output data 'how much the income of the loan is 50 ten thousand' is obtained through the automatic processing such as the analysis and the matching of the AI customer service model to be tested.
Specifically, the test unit 114 invokes the output data and the expected data with a configuration script, and outputs a test result of the AI customer service model to be tested.
The test unit 114 invokes the output data and the expected data with a configuration script, and outputs a test result of the AI customer service model to be tested, including:
the test unit 114 writes the output data and the expected data into Excel by using POI to generate an Excel file;
further, the test unit 114 determines the file name of the excel file and the interface parameter of the designated interface;
Still further, the test unit 114 modifies the configuration script with the file name and the interface parameters;
and calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
Wherein the interface parameters may include, but are not limited to: server IP (Internet Protocol ), port, test URI (Uniform Resource Identifier ), etc.
POIs are tools for reading and writing documents of office, and components (such as HSSF and XSS) in POIs can read and write excel.
The configuration script can be preconfigured so as to be convenient for directly calling after modifying related parameters, thereby improving the operation efficiency and simplifying the development flow.
Through the embodiment, the excel file is generated by combining with the POI technology, and the generated excel file is further called by the modified configuration script, so that the test result of the AI customer service model to be tested is output, and the rapid automatic test can be realized.
In at least one embodiment of the present invention, the test result includes an accuracy rate, and the obtaining unit 110 obtains a first number of data having a similarity with the expected data greater than or equal to a first similarity in the output data;
The determining unit 115 determines a second amount of the output data;
further, the calculation unit 116 calculates a quotient of the first number and the second number as the accuracy.
For example: there are 10000 test samples for a test, 5000 correct output data, and accuracy rate=5000/10000=50%.
In at least one embodiment of the present invention, the test result further includes a recall rate, and for the first output data under any category, the determining unit 115 determines first expected data corresponding to the first output data;
the acquiring unit 110 acquires a third number of data having a similarity with the first desired data greater than or equal to a second similarity in the first output data;
the determining unit 115 determines a fourth number of the first output data;
the calculation unit 116 further calculates the quotient of the third number and the fourth number as the recall under the arbitrary category.
Through the implementation mode, the accuracy and recall rate of the AI customer service model to be detected can be calculated, so that the AI customer service model to be detected can be effectively detected.
In at least one embodiment of the present invention, the test unit 114 re-performs the detection of the AI customer service model under test when it detects that there is an update to the AI customer service model under test.
Because the adjustment of the threshold value can affect the conditions of each classification in the optimization process of the model, such as the improvement of the accuracy rate of interest problems, the accuracy rate of user operation types can be reduced, but the accuracy rate of high-frequency problems of users needs to be in an ascending trend, the AI customer service model to be detected needs to be re-detected, so that the AI customer service model to be detected can be guaranteed to respond optimally to the questions of the users.
It should be noted that, in order to improve the security and privacy of data, the test results may be stored in the blockchain.
According to the technical scheme, when a test instruction for the AI customer service model to be tested is received, the pre-configured standard corpus is obtained from the corpus, the standard corpus comprises the input standard corpus and the output standard corpus, similarity analysis is carried out on the standard corpus based on a semantic similarity algorithm to obtain the classified corpus, subsequent tests are more targeted, the classified corpus is further expanded based on a preset word stock to generate a test sample, the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus, the coverage of the generated test sample is more comprehensive, the problem of insufficient test data can be effectively solved, the input data is input into the AI customer service model to be tested, the output data and the expected data are called by a configuration script, the test result of the AI customer service model to be tested is output, and further the AI customer service model to be tested is rapidly and automatically tested.
Fig. 3 is a schematic structural diagram of an electronic device according to a preferred embodiment of the present invention for implementing the AI customer service model test method.
The electronic device 1 may comprise a memory 12, a processor 13 and a bus, and may further comprise a computer program, such as an AI customer service model test program, stored in the memory 12 and executable on the processor 13.
It will be appreciated by those skilled in the art that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation of the electronic device 1, the electronic device 1 may be a bus type structure, a star type structure, the electronic device 1 may further comprise more or less other hardware or software than illustrated, or a different arrangement of components, for example, the electronic device 1 may further comprise an input-output device, a network access device, etc.
It should be noted that the electronic device 1 is only used as an example, and other electronic products that may be present in the present invention or may be present in the future are also included in the scope of the present invention by way of reference.
The memory 12 includes at least one type of readable storage medium including flash memory, a removable hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 12 may in some embodiments be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. The memory 12 may in other embodiments also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the electronic device 1. Further, the memory 12 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 12 may be used not only for storing application software installed in the electronic device 1 and various types of data, such as codes of AI customer service model test programs, but also for temporarily storing data that has been output or is to be output.
The processor 13 may be comprised of integrated circuits in some embodiments, for example, a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors, a combination of various control chips, and the like. The processor 13 is a Control Unit (Control Unit) of the electronic device 1, connects the respective components of the entire electronic device 1 using various interfaces and lines, executes or executes programs or modules (for example, executes AI customer service model test programs or the like) stored in the memory 12, and invokes data stored in the memory 12 to perform various functions of the electronic device 1 and process data.
The processor 13 executes the operating system of the electronic device 1 and various types of applications installed. The processor 13 executes the application program to implement the steps in the above-described embodiments of the AI customer service model test method, for example, the steps shown in fig. 1: s10, S11, S12, S13, S14.
Alternatively, the processor 13 may implement the functions of the modules/units in the above-described device embodiments when executing the computer program, for example:
When a test instruction for an AI customer service model to be tested is received, obtaining a pre-configured standard corpus from a corpus, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
performing similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
inputting the input data into the AI customer service model to be tested to obtain output data;
and calling the output data and the expected data by using a configuration script, and outputting a test result of the AI customer service model to be tested.
Illustratively, the computer program may be partitioned into one or more modules/units that are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules/units may be a series of instruction segments of a computer program capable of performing a specific function for describing the execution of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition unit 110, an analysis unit 111, an expansion unit 112, an input unit 113, a test unit 114, a determination unit 115, a calculation unit 116.
The integrated units implemented in the form of software functional modules described above may be stored in a computer readable storage medium. The software functional module is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (processor) to execute the AI customer service model test method according to the embodiments of the present invention.
The integrated modules/units of the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on this understanding, the present invention may also be implemented by a computer program for instructing a relevant hardware device to implement all or part of the procedures of the above-mentioned embodiment method, where the computer program may be stored in a computer readable storage medium and the computer program may be executed by a processor to implement the steps of each of the above-mentioned method embodiments.
Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
Further, the computer-usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created from the use of blockchain nodes, and the like.
The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
The bus may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, only one arrow is shown in FIG. 3, but only one bus or one type of bus is not shown. The bus is arranged to enable a connection communication between the memory 12 and at least one processor 13 or the like.
Although not shown, the electronic device 1 may further comprise a power source (such as a battery) for powering the various components, which may preferably be logically connected to the at least one processor 13 via a power management means, so as to perform functions such as charge management, discharge management, and power consumption management via the power management means. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device 1 may further include various sensors, bluetooth modules, wi-Fi modules, etc., which will not be described herein.
Further, the electronic device 1 may also comprise a network interface, optionally the network interface may comprise a wired interface and/or a wireless interface (e.g. WI-FI interface, bluetooth interface, etc.), typically used for establishing a communication connection between the electronic device 1 and other electronic devices.
The electronic device 1 may optionally further comprise a user interface, which may be a Display, an input unit, such as a Keyboard (Keyboard), or a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (organic light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device 1 and for displaying a visual user interface.
It should be understood that the embodiments described are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
Fig. 3 shows only an electronic device 1 with components 12-13, it being understood by a person skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or may combine certain components, or a different arrangement of components.
In connection with fig. 1, the memory 12 in the electronic device 1 stores a plurality of instructions to implement an AI customer service model test method, which are executable by the processor 13 to implement:
when a test instruction for an AI customer service model to be tested is received, obtaining a pre-configured standard corpus from a corpus, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
performing similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
inputting the input data into the AI customer service model to be tested to obtain output data;
And calling the output data and the expected data by using a configuration script, and outputting a test result of the AI customer service model to be tested.
Specifically, the specific implementation method of the above instructions by the processor 13 may refer to the description of the relevant steps in the corresponding embodiment of fig. 1, which is not repeated herein.
In the several embodiments provided in the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division when actually implemented.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the system claims can also be implemented by means of software or hardware by means of one unit or means. The terms second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.

Claims (8)

1. The AI customer service model test method is characterized by comprising the following steps:
when a test instruction for an AI customer service model to be tested is received, obtaining a pre-configured standard corpus from a corpus, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
performing similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
expanding the classified corpus based on a preset word stock to generate a test sample, wherein the test sample comprises the steps of calculating the similarity of each target word vector in the classified corpus and the word vector in the preset word stock; acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector; adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
inputting the input data into the AI customer service model to be tested to obtain output data;
Calling the output data and the expected data by using a configuration script, and outputting a test result of the AI customer service model to be tested, wherein the test result comprises writing the output data and the expected data into Excel by using POIs to generate an Excel file; determining the file name of the excel file and specifying interface parameters of an interface; modifying the configuration script with the file name and the interface parameter; and calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
2. The AI customer service model test method of claim 1, wherein the performing similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus comprises:
converting the standard corpus into semantic vectors based on natural language processing;
calculating cosine distances among word vectors in the semantic vectors by adopting a cosine similarity algorithm;
and classifying each word vector according to the cosine distance to obtain the classified corpus.
3. The AI customer service model test method of claim 1, wherein the test results include accuracy, the AI customer service model test method further comprising:
Acquiring a first number of data in the output data, wherein the similarity between the data and the expected data is greater than or equal to a first similarity;
determining a second amount of the output data;
calculating a quotient of the first number and the second number as the accuracy.
4. The AI customer service model test method of claim 1, wherein the test results further comprise a recall, the AI customer service model test method further comprising:
for first output data under any category, determining first expected data corresponding to the first output data;
acquiring a third number of data in the first output data, wherein the similarity between the data and the first expected data is greater than or equal to a second similarity;
determining a fourth amount of the first output data;
and calculating the quotient of the third quantity and the fourth quantity as the recall rate under the arbitrary category.
5. The AI customer service model test method of claim 1, further comprising:
and when the AI customer service model to be detected is detected to be updated, the detection of the AI customer service model to be detected is re-executed.
6. An AI customer service model test device, characterized in that the AI customer service model test device comprises:
The acquisition unit is used for acquiring a pre-configured standard corpus from a corpus when a test instruction of an AI customer service model to be tested is received, wherein the standard corpus comprises an input standard corpus and an output standard corpus;
the analysis unit is used for carrying out similarity analysis on the standard corpus based on a semantic similarity algorithm to obtain a classified corpus;
the expansion unit is used for expanding the classified corpus based on a preset word stock to generate a test sample, and calculating the similarity between each target word vector in the classified corpus and the word vector in the preset word stock; acquiring a word vector with the similarity larger than or equal to the preset similarity with the target word vector from the preset word stock as an expanded word vector of the target word vector; adding the expanded word vector of each target word vector to the classified corpus to obtain the test sample, wherein the test sample comprises input data corresponding to the input standard corpus and expected data corresponding to the output standard corpus;
the input unit is used for inputting the input data into the AI customer service model to be tested to obtain output data;
The test unit is used for calling the output data and the expected data through a configuration script, outputting a test result of the AI customer service model to be tested, and writing the output data and the expected data into Excel by adopting POIs to generate an Excel file; determining the file name of the excel file and specifying interface parameters of an interface; modifying the configuration script with the file name and the interface parameter; and calling the excel file by using the modified configuration script, and outputting a test result of the AI customer service model to be tested.
7. An electronic device, the electronic device comprising:
a memory storing at least one instruction; and
A processor executing instructions stored in the memory to implement the AI customer service model test method of any of claims 1-5.
8. A computer-readable storage medium, characterized by: the computer-readable storage medium having stored therein at least one instruction for execution by a processor in an electronic device to implement the AI customer service model test method of any of claims 1-5.
CN202010719768.XA 2020-07-23 2020-07-23 AI customer service model test method and device, electronic equipment and storage medium Active CN111859985B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010719768.XA CN111859985B (en) 2020-07-23 2020-07-23 AI customer service model test method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010719768.XA CN111859985B (en) 2020-07-23 2020-07-23 AI customer service model test method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111859985A CN111859985A (en) 2020-10-30
CN111859985B true CN111859985B (en) 2023-09-12

Family

ID=72950961

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010719768.XA Active CN111859985B (en) 2020-07-23 2020-07-23 AI customer service model test method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111859985B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112307771B (en) * 2020-10-29 2021-05-28 平安科技（深圳）有限公司 Course analysis method, device, equipment and medium based on emotion analysis

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106599317A (en) * 2016-12-30 2017-04-26 上海智臻智能网络科技股份有限公司 Test data processing method and device for question-answering system and terminal
CN107861951A (en) * 2017-11-17 2018-03-30 康成投资（中国）有限公司 Session subject identifying method in intelligent customer service
CN108804567A (en) * 2018-05-22 2018-11-13 平安科技（深圳）有限公司 Method, equipment, storage medium and device for improving intelligent customer service response rate
CN110674292A (en) * 2019-08-27 2020-01-10 腾讯科技（深圳）有限公司 Man-machine interaction method, device, equipment and medium
CN110909539A (en) * 2019-10-15 2020-03-24 平安科技（深圳）有限公司 Word generation method, system, computer device and storage medium of corpus
CN111177351A (en) * 2019-12-20 2020-05-19 北京淇瑀信息科技有限公司 Method, device and system for acquiring natural language expression intention based on rule
CN111177307A (en) * 2019-11-22 2020-05-19 深圳壹账通智能科技有限公司 Test scheme and system based on semantic understanding similarity threshold configuration

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104573028B (en) * 2015-01-14 2019-01-25 百度在线网络技术（北京）有限公司 Realize the method and system of intelligent answer
US20200081939A1 (en) * 2018-09-11 2020-03-12 Hcl Technologies Limited System for optimizing detection of intent[s] by automated conversational bot[s] for providing human like responses

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106599317A (en) * 2016-12-30 2017-04-26 上海智臻智能网络科技股份有限公司 Test data processing method and device for question-answering system and terminal
CN107861951A (en) * 2017-11-17 2018-03-30 康成投资（中国）有限公司 Session subject identifying method in intelligent customer service
CN108804567A (en) * 2018-05-22 2018-11-13 平安科技（深圳）有限公司 Method, equipment, storage medium and device for improving intelligent customer service response rate
CN110674292A (en) * 2019-08-27 2020-01-10 腾讯科技（深圳）有限公司 Man-machine interaction method, device, equipment and medium
CN110909539A (en) * 2019-10-15 2020-03-24 平安科技（深圳）有限公司 Word generation method, system, computer device and storage medium of corpus
CN111177307A (en) * 2019-11-22 2020-05-19 深圳壹账通智能科技有限公司 Test scheme and system based on semantic understanding similarity threshold configuration
CN111177351A (en) * 2019-12-20 2020-05-19 北京淇瑀信息科技有限公司 Method, device and system for acquiring natural language expression intention based on rule

Also Published As

Publication number Publication date
CN111859985A (en) 2020-10-30

Similar Documents

Publication Publication Date Title
US11822568B2 (en) Data processing method, electronic equipment and storage medium
CN111652279B (en) Behavior evaluation method and device based on time sequence data and readable storage medium
CN108228567B (en) Method and device for extracting short names of organizations
CN108121699B (en) Method and apparatus for outputting information
CN112181835B (en) Automatic test method, device, computer equipment and storage medium
CN109542757A (en) Interface testing environment determines method, apparatus, electronic equipment and storage medium
CN111949708A (en) Multi-task prediction method, device, equipment and medium based on time sequence feature extraction
US10762089B2 (en) Open ended question identification for investigations
CN111831708A (en) Missing data-based sample analysis method and device, electronic equipment and medium
CN115525750A (en) Robot phonetics detection visualization method and device, electronic equipment and storage medium
CN115510188A (en) Text keyword association method, device, equipment and storage medium
CN116402166A (en) Training method and device of prediction model, electronic equipment and storage medium
CN114862140A (en) Behavior analysis-based potential evaluation method, device, equipment and storage medium
CN111859985B (en) AI customer service model test method and device, electronic equipment and storage medium
CN111950707B (en) Behavior prediction method, device, equipment and medium based on behavior co-occurrence network
CN113591881A (en) Intention recognition method and device based on model fusion, electronic equipment and medium
CN107071553A (en) Method, device and computer readable storage medium for modifying video and voice
CN112348615A (en) Method and device for auditing information
CN113419951B (en) Artificial intelligent model optimization method and device, electronic equipment and storage medium
CN112528183B (en) Webpage component layout method and device based on big data, electronic equipment and medium
CN114780688A (en) Text quality inspection method, device and equipment based on rule matching and storage medium
CN114548114A (en) Text emotion recognition method, device, equipment and storage medium
CN114781833B (en) Capability assessment method, device and equipment based on business personnel and storage medium
CN113723611B (en) Business factor generation method, device, equipment and medium based on causal inference
CN116364223B (en) Feature processing method, device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20230814

Address after: Room 1528, No. 1500 Century Avenue, China (Shanghai) Pilot Free Trade Zone, Pudong New Area, Shanghai, 200000

Applicant after: Shanghai Huaqi Information Technology Co.,Ltd.

Address before: 518000 Room 202, block B, aerospace micromotor building, No.7, Langshan No.2 Road, Xili street, Nanshan District, Shenzhen City, Guangdong Province

Applicant before: Shenzhen LIAN intellectual property service center

Effective date of registration: 20230814

Address after: 518000 Room 202, block B, aerospace micromotor building, No.7, Langshan No.2 Road, Xili street, Nanshan District, Shenzhen City, Guangdong Province

Applicant after: Shenzhen LIAN intellectual property service center

Address before: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.)

Applicant before: PING AN PUHUI ENTERPRISE MANAGEMENT Co.,Ltd.

GR01 Patent grant
GR01 Patent grant