CN104809062B - A kind of method of testing and system of artificial intelligence response system - Google Patents

A kind of method of testing and system of artificial intelligence response system Download PDF

Info

Publication number
CN104809062B
CN104809062B CN201510193639.0A CN201510193639A CN104809062B CN 104809062 B CN104809062 B CN 104809062B CN 201510193639 A CN201510193639 A CN 201510193639A CN 104809062 B CN104809062 B CN 104809062B
Authority
CN
China
Prior art keywords
session
test
artificial intelligence
current sessions
aggregation
Prior art date
Application number
CN201510193639.0A
Other languages
Chinese (zh)
Other versions
CN104809062A (en
Inventor
王昌
范亚平
孙胜方
Original Assignee
北京京东尚科信息技术有限公司
北京京东世纪贸易有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京京东尚科信息技术有限公司, 北京京东世纪贸易有限公司 filed Critical 北京京东尚科信息技术有限公司
Priority to CN201510193639.0A priority Critical patent/CN104809062B/en
Publication of CN104809062A publication Critical patent/CN104809062A/en
Application granted granted Critical
Publication of CN104809062B publication Critical patent/CN104809062B/en

Links

Abstract

The present invention discloses a kind of method of testing and system of artificial intelligence response system, and method includes:Obtaining includes the session aggregation of multiple session elements, and the session element is session or scene end of identification;Session context generation step, including:Session element is taken out in order from the session aggregation as current sessions element, creates session context;The problem of in session context to included by artificial intelligence response system input current sessions element, capture the actual answer of the artificial intelligence response system response described problem;Next session element is taken out in order from session aggregation as current sessions element, if current sessions element is session, then perform the conversation test step, otherwise session context is terminated, if there is next session element in session aggregation, session context generation step is then performed, is otherwise terminated.The present invention can cover the testing requirement of more scenes, multi-user and complicated premise calls.

Description

A kind of method of testing and system of artificial intelligence response system
Technical field
The present invention relates to artificial intelligence correlative technology field, the method for testing of particularly a kind of artificial intelligence response system and System.
Background technology
Artificial intelligence is the application for natural language processing, the content to be paid close attention to of the test to artificial intelligence and tradition Automatic test it is inconsistent, particularly the web of artificial intelligence response system is applied, in addition to page function, main one Individual test emphasis be corresponding input problem whether can to corresponding result, adjusted for the variation of natural language and model Instability, test need to carry out it is substantial amounts of return checking, waste substantial amounts of manpower and time.
Prior art completes automatic test corresponding to artificial intelligence response system using such a way:
Use the automated test tool of increasing income based on Google:Selenium operates to web page, completes to put question to and returns To answer, the preservation content of input results only includes problem and answer, and the session operation of next is directly performed after the completion of execution, and Result is recorded.As a result record content only includes problem and actual result.
The shortcomings that prior art:
(1) problem result can be because the content of context enters different scenes so as to cause subproblem not enter into just True handling process, as a result accuracy is inadequate.
(2) a kind of usage scenario can only be covered by once performing, and be not easy to switch in scene execution, different scenes is needed The effect that just can guarantee that and reach more scenes is performed a plurality of times.
(3) session data sequential organization is indefinite, lacks execution logic.
(4) implementing result preserves imperfection, only exports corresponding result and preserves, is not easy to carry out data statistics and maintenance.
The content of the invention
Based on this, it is necessary to for the technical problem that prior art test accuracy is inadequate, there is provided a kind of artificial intelligence should Answer the method for testing and system of system.
A kind of method of testing of artificial intelligence response system, including:
Session aggregation obtaining step, including:Obtaining includes the session aggregation of multiple session elements, and the session element is meeting Words or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each described Session includes:Problem, scene identity and expected answer;
Session context generation step, including:Session element is taken out in order from the session aggregation as current sessions Element, if current sessions element is session, create session context;
Conversation test step, including:Wrapped in session context to artificial intelligence response system input current sessions element The problem of including, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current meeting The expection answer matches included by element are talked about, then the test result for recording current sessions element is by otherwise recording current meeting The test result for talking about element is failure;
Session element checking step, including:Next session element is taken out in order from session aggregation as current meeting Element is talked about, if current sessions element is scene end of identification, terminates session context, performs and terminate checking step, if worked as Preceding session element is session, then performs the conversation test step;
Terminate checking step, including:If there is next session element in session aggregation, session context generation is performed Step, otherwise terminate.
A kind of test system of artificial intelligence response system, including:
Session aggregation acquisition module, is used for:Obtaining includes the session aggregation of multiple session elements, and the session element is meeting Words or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each described Session includes:Problem, scene identity and expected answer;
Session context generation module, is used for:Session element is taken out in order from the session aggregation as current sessions Element, if current sessions element is session, create session context;
Conversation test module, is used for:Wrapped in session context to artificial intelligence response system input current sessions element The problem of including, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current meeting The expection answer matches included by element are talked about, then the test result for recording current sessions element is by otherwise recording current meeting The test result for talking about element is failure;
Session element checks module, is used for:Next session element is taken out in order from session aggregation as current meeting Element is talked about, if current sessions element is scene end of identification, terminates session context, performs and terminates to check module, if worked as Preceding session element is session, then performs the conversation test module;
Terminate to check module, be used for:If there is next session element in session aggregation, session context generation is performed Module, otherwise terminate.
The present invention passes through the flow that increase scene end of identification and correlation are concentrated in session so that automatic test can be The situation of multiple scenes is once tackled in complete test, more scenes, multi-user and complicated premise calls can be covered Testing requirement.
Brief description of the drawings
Fig. 1 is a kind of workflow diagram of the method for testing of artificial intelligence response system of the present invention;
Fig. 2 is the workflow diagram of preferred embodiment;
Fig. 3 is a kind of construction module figure of the test system of artificial intelligence response system of the present invention.
Embodiment
The present invention will be further described in detail with specific embodiment below in conjunction with the accompanying drawings.
It is as shown in Figure 1 a kind of workflow diagram of the method for testing of artificial intelligence response system of the present invention, including:
Step S101, including:Obtaining includes the session aggregation of multiple session elements, and the session element is session or field Scape end of identification, at least one session element that the session aggregation includes is scene end of identification, and each session includes: Problem, scene identity and expected answer;
Step S102, including:Session element is taken out in order from the session aggregation as current sessions element, if Current sessions element is session, then creates session context;
Step S103, including:To included by artificial intelligence response system input current sessions element in session context Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member The test result of element is failure;
Step S104, including:Next session element is taken out in order from session aggregation as current sessions element, such as Fruit current sessions element is scene end of identification, then terminates session context, step S105 is performed, if current sessions element is meeting Words, then perform the step S103;
Step S105, including:If there is next session element in session aggregation, step S102 is performed, is otherwise tied Beam.
The present invention obtains session aggregation by step S101, includes session and scene end of identification in session aggregation, Meanwhile session context is created in step s 102, and in step S104, if current sessions element is scene end of identification, Then terminate session context, so as to realize the differentiation to different sessions scene, before more scenes, multi-user, complexity can be covered The testing requirement made demands, response logic is avoided to be influenceed by context.
Wherein, in step s 102, if the session element taken out is not session, can be ignored.
In one of the embodiments, for the establishment session context to log in browser, the end session context is to close Close browser.
The present embodiment ensures current sessions to scenery control mainly in the cookies by refreshing current sessions scene The independence of scene.The present invention, by wanting scene set end of identification, allows program or script, it is realized that working as when being handled Test is when proceed to this, and all sessions for representing this scene have finished, it is necessary to cookies to current sessions It is updated, or old cookies is cleared up, ensures that a new session context can be entered during next session (session).The mode the present embodiment for updating session is realized using browser is closed.
Browser is logged in when performing step S102, and when performing step S104, if current sessions element is scene End of identification, then browser is closed, so that the cookies of session context is updated.
Preferably, when performing session start, mark that a scene can be set to start, such as a variable EndFlage, perform initialize when while starting (or new scene), be set to a value, such as False.When into one During individual scene, session is proceeded by by triggering the action of login browser, will be from team when current session is completed in execution Next session is taken out in row, judges whether this session content includes the scene end of identification of setting, if do not included, By the contents extraction of this session, performed.If it find that comprising scene end of identification, then EndFlage is set to True, Program will close browser according to this EndFlage.So as to complete a complete scene.
Such flow guarantees not perform session in by the environment of context interference at one, has so as to reach distinctive Targetedly test purpose.
In one of the embodiments, the session also includes the account information associated with scene identity, the step In S103, logging in the artificial intelligence using the account information should system.
In general, automatic test includes multiple test cases, and each test case completes one or session.Pass through this Invention so that the test case of automatic test can include multiple test scenes, and by by account information and scene identity It is associated, then different account informations can be used for different session contexts, so as to accomplish test process completely by counting It is driven according to test case.
In one of the embodiments, the session aggregation is carried out test data by the way of form and preserved, its In per data line be a session element.
, it is necessary to carry out test data support, this reality by substantial amounts of language material in the test based on artificial intelligence response system Example is applied in order to be more easily managed to test data, the mode for employing form carries out test data and preserved, and protects Deposit form with behavior unit to be split, data line is a session element.Different column datas represent different session informations.
In one of the embodiments, the step S103, in addition to:Newly-generated result object, by current sessions element Test result write the result object and preserve.
In the present embodiment, after actual answer corresponding to grabbing, expected answer is contrasted with actual answer at once, So as to obtain test result.Due to directly being contrasted and being preserved after completing to capture, so as to ensure the validity of result.Together When, due to saving test result in result object, therefore when follow-up output result is shown can definitely and effectively, Analyticity strengthens.
The workflow diagram of preferred embodiment is illustrated in figure 2, including:
Step S201, read session aggregation.
Session aggregation includes multiple session elements, and session element can be session or scene end of identification.Each session Preserving type it is as shown in table 1.
The preserving type of the session of table 1
Problem Scene identity Test account It is expected that answer
In addition to distinguishing different scenes, the invalid session mark of a tape identification can be added in test session set Remember the end of scene, i.e. scene end of identification, be the data framework of session aggregation as shown in table 2.
The data framework of the session aggregation of table 2
Problem Scene identity Test account It is expected that answer
Problem Scene identity Test account It is expected that answer
Scene end of identification It is empty It is empty It is empty
Problem Scene identity Test account It is expected that answer
Problem Scene identity Test account It is expected that answer
Before perform script or flow, test data can be stored in number often capable in internal memory by the first step in the form of queue An independent object is stored in session list according to this, and the data of corresponding row are sequentially added into session team in a manner of poll Row.Session is set to form the list of session in a certain order.The follow-up execution of aspect.The easy handling guarantee of queuing data The flexibility of digital independent, operation easily can be entered to session data.
In order to simplify maintenance cost, herein in actual treatment the session of Same Scene by first of only each scene Session includes scene identity and account information.Subsequently will no longer it change.Until scene end of identification occurs.
All sessions can be read in a manner of poll to be extracted, and by queue (List) access characteristics, The session of corresponding mark is directly accessed in the case of specific, ensures the operability of data.
Step S202, generate new session context (Session).
Preferably, new session is generated by logging in browser.
Step S203, problem is inputted to artificial intelligence response system.
Step S204, the actual answer that crawl artificial intelligence response system is returned.
Step S205, obtain the comparing result after actual answer is contrasted with expected answer.
After content corresponding to having captured, corresponding expected answer and the actual answer actually obtained are contrasted, pin To including expected answer or the situation completely the same with expected answer in actual answer, then the test result of this session can be given Labeled as Pass (expression passes through), do not include if actual answer or can be surveyed if being not equal to expected answer completely to this session Test result is labeled as Fail (expression does not pass through).
Step S206, preserve test result.
After completion result verification is performed, corresponding test result is write into result document.Then perform again next time Session is read, and after each session execution verifies, the content of session generates a result object, and this result object is led to Cross wiring method and write direct destination file, preserved.
Step S207, if scene end point then performs step S208, otherwise perform step S203.
If the session element read is scene end of identification, it is judged as scene end point, is otherwise judged as not having also To scene end point;
Step S208, close current session.
Current session closing is realized by closing browser.
Step S209, if also including next session in session aggregation, step S202 is performed, otherwise performs step S210。
Step S210, preserve final result.
After performing and completing all session, can finally preserve once result, finally can by reading result document or Carry out the interpretation of result of next step.In order to which easy analysis result object comprises at least content as shown in table 3.
The content of the result object of table 3
Problem It is expected that answer Actual answer Test result
A kind of construction module figure of the test system of artificial intelligence response system of the present invention is illustrated in figure 3, including:
Session aggregation acquisition module 301, is used for:Obtaining includes the session aggregation of multiple session elements, the session element For session or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each The session includes:Problem, scene identity and expected answer;
Session context generation module 302, is used for:Session element is taken out in order from the session aggregation as current meeting Element is talked about, if current sessions element is session, creates session context;
Conversation test module 303, is used for:In session context current sessions element institute is inputted to artificial intelligence response system Including the problem of, capture the actual answer of artificial intelligence response system response described problem, if actual answer with it is current Expection answer matches included by session element, the then test result for recording current sessions element are by the way that otherwise record is current The test result of session element is failure;
Session element checks module 304, is used for:Next session element is taken out in order from session aggregation as current Session element, if current sessions element is scene end of identification, terminate session context, perform and terminate to check module 305, such as Fruit current sessions element is session, then performs the conversation test module 303;
Terminate to check module 305, be used for:If there is next session element in session aggregation, session context is performed Generation module 302, otherwise terminates.
In one of the embodiments, for the establishment session context to log in browser, the end session context is to close Close browser.
In one of the embodiments, the session also includes the account information associated with scene identity, and the session is surveyed In die trial block, logging in the artificial intelligence using the account information should system.
In one of the embodiments, the session aggregation is carried out test data by the way of form and preserved, its In per data line be a session element.
In one of the embodiments, the conversation test module, in addition to:Newly-generated result object, by current sessions The test result of element writes the result object and preserved.
Embodiment described above only expresses the several embodiments of the present invention, and its description is more specific and detailed, but simultaneously Therefore the limitation to the scope of the claims of the present invention can not be interpreted as.It should be pointed out that for one of ordinary skill in the art For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to the guarantor of the present invention Protect scope.Therefore, the protection domain of patent of the present invention should be determined by the appended claims.

Claims (10)

  1. A kind of 1. method of testing of artificial intelligence response system, it is characterised in that including:
    Session aggregation obtaining step, including:Obtaining includes the session aggregations of multiple session elements, the session element for session or Person's scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each session Including:Problem, scene identity and expected answer;
    Session context generation step, including:Session element is taken out in order from the session aggregation as current sessions element, If current sessions element is session, session context is created;
    Conversation test step, including:To included by artificial intelligence response system input current sessions element in session context Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member The test result of element is failure;
    Session element checking step, including:Next session element is taken out in order from session aggregation as current sessions member Element, if current sessions element is scene end of identification, terminate session context, perform and terminate checking step, if current meeting Words element is session, then performs the conversation test step;
    Terminate checking step, including:If there is next session element in session aggregation, session context generation step is performed Suddenly, otherwise terminate.
  2. 2. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the establishment session field For scape to log in browser, the end session context is closing browser.
  3. 3. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the session also includes The account information associated with scene identity, in the conversation test step, the artificial intelligence is logged in using the account information Should system.
  4. 4. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the session aggregation is adopted Test data is carried out with the mode of form to be preserved, each of which row data are a session elements.
  5. 5. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the conversation test step Suddenly, in addition to:Newly-generated result object, the test result of current sessions element is write into the result object and preserved.
  6. A kind of 6. test system of artificial intelligence response system, it is characterised in that including:
    Session aggregation acquisition module, is used for:Obtaining includes the session aggregations of multiple session elements, the session element for session or Person's scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each session Including:Problem, scene identity and expected answer;
    Session context generation module, is used for:Session element is taken out in order from the session aggregation as current sessions element, If current sessions element is session, session context is created;
    Conversation test module, is used for:To included by artificial intelligence response system input current sessions element in session context Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member The test result of element is failure;
    Session element checks module, is used for:Next session element is taken out in order from session aggregation as current sessions member Element, if current sessions element is scene end of identification, terminate session context, perform and terminate to check module, if current meeting Words element is session, then performs the conversation test module;
    Terminate to check module, be used for:If there is next session element in session aggregation, session context generation mould is performed Block, otherwise terminate.
  7. 7. the test system of artificial intelligence response system according to claim 6, it is characterised in that the establishment session field For scape to log in browser, the end session context is closing browser.
  8. 8. the test system of artificial intelligence response system according to claim 6, it is characterised in that the session also includes The account information associated with scene identity, in the conversation test module, the artificial intelligence is logged in using the account information Should system.
  9. 9. the test system of artificial intelligence response system according to claim 6, it is characterised in that the session aggregation is adopted Test data is carried out with the mode of form to be preserved, each of which row data are a session elements.
  10. 10. the test system of artificial intelligence response system according to claim 6, it is characterised in that the conversation test Module, in addition to:Newly-generated result object, the test result of current sessions element is write into the result object and preserved.
CN201510193639.0A 2015-04-22 2015-04-22 A kind of method of testing and system of artificial intelligence response system CN104809062B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510193639.0A CN104809062B (en) 2015-04-22 2015-04-22 A kind of method of testing and system of artificial intelligence response system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510193639.0A CN104809062B (en) 2015-04-22 2015-04-22 A kind of method of testing and system of artificial intelligence response system

Publications (2)

Publication Number Publication Date
CN104809062A CN104809062A (en) 2015-07-29
CN104809062B true CN104809062B (en) 2017-12-01

Family

ID=53693904

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510193639.0A CN104809062B (en) 2015-04-22 2015-04-22 A kind of method of testing and system of artificial intelligence response system

Country Status (1)

Country Link
CN (1) CN104809062B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107402878B (en) * 2016-05-19 2020-09-01 北京京东尚科信息技术有限公司 Test method and device
CN106503242B (en) * 2016-11-08 2019-04-19 上海智臻智能网络科技股份有限公司 A kind of intelligent interactive method, device and server
CN107463640B (en) * 2017-07-19 2020-12-18 一拓通信集团股份有限公司 Artificial intelligence based information data acquisition method
CN110908919A (en) * 2019-12-02 2020-03-24 上海市软件评测中心有限公司 Response test system based on artificial intelligence and application thereof

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1744056A (en) * 2004-09-04 2006-03-08 华为技术有限公司 Automatic test method of application
US7945899B2 (en) * 2003-06-16 2011-05-17 American Megatrends, Inc. Method and system for remote software testing
CN103049367A (en) * 2011-10-11 2013-04-17 神州数码信息系统有限公司 Automatic testing method for software
CN103853842A (en) * 2014-03-20 2014-06-11 百度在线网络技术(北京)有限公司 Automatic question and answer method and system
CN104142959A (en) * 2013-05-10 2014-11-12 中国银联股份有限公司 Automated test device and method based on expert knowledge database

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9336124B2 (en) * 2013-02-28 2016-05-10 Apple Inc. Automatic remote execution of an application

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7945899B2 (en) * 2003-06-16 2011-05-17 American Megatrends, Inc. Method and system for remote software testing
CN1744056A (en) * 2004-09-04 2006-03-08 华为技术有限公司 Automatic test method of application
CN103049367A (en) * 2011-10-11 2013-04-17 神州数码信息系统有限公司 Automatic testing method for software
CN104142959A (en) * 2013-05-10 2014-11-12 中国银联股份有限公司 Automated test device and method based on expert knowledge database
CN103853842A (en) * 2014-03-20 2014-06-11 百度在线网络技术(北京)有限公司 Automatic question and answer method and system

Also Published As

Publication number Publication date
CN104809062A (en) 2015-07-29

Similar Documents

Publication Publication Date Title
US10243679B2 (en) Vulnerability detection
US10929879B2 (en) Method and apparatus for identification of fraudulent click activity
Tambosi et al. A framework to optimize biodiversity restoration efforts based on habitat amount and landscape connectivity
TWI621077B (en) Character recognition method and server for claim documents
CN107358157B (en) Face living body detection method and device and electronic equipment
CA3001839C (en) Call detail record analysis to identify fraudulent activity and fraud detection in interactive voice response systems
CN104268006B (en) The back method and device of key mouse script
US8099443B2 (en) Apparatus for building address database and method thereof
CN104123500B (en) A kind of Android platform malicious application detection method and device based on deep learning
CN104104717B (en) Deliver channel data statistical approach and device
CN107239666B (en) Method and system for desensitizing medical image data
US9015685B2 (en) Code analysis for simulation efficiency improvement
CN106227494B (en) A kind of application operating method and terminal
CN105844283A (en) Method for identifying category of image, image search method and image search device
CN107145799A (en) A kind of data desensitization method and device
US8332509B2 (en) Parameterized computer monitoring system
TWI677846B (en) Method and device for transferring robot customer service to manual customer service
CN104580451B (en) Realize the method and system and electronic equipment of multi-platform account synchronization
CN105989283B (en) A kind of method and device identifying virus mutation
CN107483458A (en) The recognition methods of network attack and device, computer-readable recording medium
CN103927484A (en) Malicious program behavior capture method based on Qemu
CA2994770A1 (en) Model integration tool
CN105184160B (en) A kind of method of the Android phone platform application program malicious act detection based on API object reference relational graphs
CN102930210A (en) System and method for automatically analyzing, detecting and classifying malicious program behavior
CN103607413B (en) Method and device for detecting website backdoor program

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
EXSB Decision made by sipo to initiate substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant