CN104809062B - A kind of method of testing and system of artificial intelligence response system - Google Patents
A kind of method of testing and system of artificial intelligence response system Download PDFInfo
- Publication number
- CN104809062B CN104809062B CN201510193639.0A CN201510193639A CN104809062B CN 104809062 B CN104809062 B CN 104809062B CN 201510193639 A CN201510193639 A CN 201510193639A CN 104809062 B CN104809062 B CN 104809062B
- Authority
- CN
- China
- Prior art keywords
- session
- test
- artificial intelligence
- current sessions
- aggregation
- Prior art date
Links
- 230000004044 response Effects 0.000 title claims abstract description 47
- 238000010998 test method Methods 0.000 title claims abstract description 5
- 230000002776 aggregation Effects 0.000 claims abstract description 50
- 238000004220 aggregation Methods 0.000 claims abstract description 48
- 230000000875 corresponding Effects 0.000 description 10
- 235000014510 cooky Nutrition 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000034 method Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006011 modification reaction Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- BUGBHKTXTAQXES-UHFFFAOYSA-N selenium Chemical compound data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0nMS4wJyBlbmNvZGluZz0naXNvLTg4NTktMSc/Pgo8c3ZnIHZlcnNpb249JzEuMScgYmFzZVByb2ZpbGU9J2Z1bGwnCiAgICAgICAgICAgICAgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJwogICAgICAgICAgICAgICAgICAgICAgeG1sbnM6cmRraXQ9J2h0dHA6Ly93d3cucmRraXQub3JnL3htbCcKICAgICAgICAgICAgICAgICAgICAgIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJwogICAgICAgICAgICAgICAgICB4bWw6c3BhY2U9J3ByZXNlcnZlJwp3aWR0aD0nMzAwcHgnIGhlaWdodD0nMzAwcHgnIHZpZXdCb3g9JzAgMCAzMDAgMzAwJz4KPCEtLSBFTkQgT0YgSEVBREVSIC0tPgo8cmVjdCBzdHlsZT0nb3BhY2l0eToxLjA7ZmlsbDojRkZGRkZGO3N0cm9rZTpub25lJyB3aWR0aD0nMzAwJyBoZWlnaHQ9JzMwMCcgeD0nMCcgeT0nMCc+IDwvcmVjdD4KPHRleHQgeD0nMTEyLjM5NycgeT0nMTcwJyBjbGFzcz0nYXRvbS0wJyBzdHlsZT0nZm9udC1zaXplOjQwcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTpub25lO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWY7dGV4dC1hbmNob3I6c3RhcnQ7ZmlsbDojM0I0MTQzJyA+UzwvdGV4dD4KPHRleHQgeD0nMTM5Ljk5NycgeT0nMTcwJyBjbGFzcz0nYXRvbS0wJyBzdHlsZT0nZm9udC1zaXplOjQwcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTpub25lO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWY7dGV4dC1hbmNob3I6c3RhcnQ7ZmlsbDojM0I0MTQzJyA+ZTwvdGV4dD4KPHBhdGggZD0nTSAxMTIuMzYxLDE0NiBMIDExMi4zNTQsMTQ1LjgyOCBMIDExMi4zMzEsMTQ1LjY1NyBMIDExMi4yOTUsMTQ1LjQ4OSBMIDExMi4yNDQsMTQ1LjMyNSBMIDExMi4xNzksMTQ1LjE2NiBMIDExMi4xLDE0NS4wMTIgTCAxMTIuMDA5LDE0NC44NjcgTCAxMTEuOTA1LDE0NC43MjkgTCAxMTEuNzksMTQ0LjYwMSBMIDExMS42NjUsMTQ0LjQ4MyBMIDExMS41MywxNDQuMzc3IEwgMTExLjM4NiwxNDQuMjgyIEwgMTExLjIzNCwxNDQuMjAxIEwgMTExLjA3NiwxNDQuMTMyIEwgMTEwLjkxMywxNDQuMDc4IEwgMTEwLjc0NiwxNDQuMDM3IEwgMTEwLjU3NiwxNDQuMDEyIEwgMTEwLjQwNCwxNDQgTCAxMTAuMjMyLDE0NC4wMDQgTCAxMTAuMDYxLDE0NC4wMjMgTCAxMDkuODkyLDE0NC4wNTYgTCAxMDkuNzI3LDE0NC4xMDMgTCAxMDkuNTY2LDE0NC4xNjUgTCAxMDkuNDExLDE0NC4yNCBMIDEwOS4yNjMsMTQ0LjMyOCBMIDEwOS4xMjQsMTQ0LjQyOSBMIDEwOC45OTMsMTQ0LjU0MSBMIDEwOC44NzMsMTQ0LjY2NCBMIDEwOC43NjMsMTQ0Ljc5NyBMIDEwOC42NjYsMTQ0LjkzOSBMIDEwOC41ODEsMTQ1LjA4OCBMIDEwOC41MDksMTQ1LjI0NSBMIDEwOC40NTEsMTQ1LjQwNyBMIDEwOC40MDcsMTQ1LjU3MyBMIDEwOC4zNzgsMTQ1Ljc0MyBMIDEwOC4zNjMsMTQ1LjkxNCBMIDEwOC4zNjMsMTQ2LjA4NiBMIDEwOC4zNzgsMTQ2LjI1NyBMIDEwOC40MDcsMTQ2LjQyNyBMIDEwOC40NTEsMTQ2LjU5MyBMIDEwOC41MDksMTQ2Ljc1NSBMIDEwOC41ODEsMTQ2LjkxMiBMIDEwOC42NjYsMTQ3LjA2MSBMIDEwOC43NjMsMTQ3LjIwMyBMIDEwOC44NzMsMTQ3LjMzNiBMIDEwOC45OTMsMTQ3LjQ1OSBMIDEwOS4xMjQsMTQ3LjU3MSBMIDEwOS4yNjMsMTQ3LjY3MiBMIDEwOS40MTEsMTQ3Ljc2IEwgMTA5LjU2NiwxNDcuODM1IEwgMTA5LjcyNywxNDcuODk3IEwgMTA5Ljg5MiwxNDcuOTQ0IEwgMTEwLjA2MSwxNDcuOTc3IEwgMTEwLjIzMiwxNDcuOTk2IEwgMTEwLjQwNCwxNDggTCAxMTAuNTc2LDE0Ny45ODggTCAxMTAuNzQ2LDE0Ny45NjMgTCAxMTAuOTEzLDE0Ny45MjIgTCAxMTEuMDc2LDE0Ny44NjggTCAxMTEuMjM0LDE0Ny43OTkgTCAxMTEuMzg2LDE0Ny43MTggTCAxMTEuNTMsMTQ3LjYyMyBMIDExMS42NjUsMTQ3LjUxNyBMIDExMS43OSwxNDcuMzk5IEwgMTExLjkwNSwxNDcuMjcxIEwgMTEyLjAwOSwxNDcuMTMzIEwgMTEyLjEsMTQ2Ljk4OCBMIDExMi4xNzksMTQ2LjgzNCBMIDExMi4yNDQsMTQ2LjY3NSBMIDExMi4yOTUsMTQ2LjUxMSBMIDExMi4zMzEsMTQ2LjM0MyBMIDExMi4zNTQsMTQ2LjE3MiBMIDExMi4zNjEsMTQ2IEwgMTEwLjM2MSwxNDYgWicgc3R5bGU9J2ZpbGw6IzAwMDAwMDtmaWxsLXJ1bGU6ZXZlbm9kZDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6IzAwMDAwMDtzdHJva2Utd2lkdGg6MHB4O3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZS1vcGFjaXR5OjE7JyAvPgo8cGF0aCBkPSdNIDExMi4zNjEsMTU0IEwgMTEyLjM1NCwxNTMuODI4IEwgMTEyLjMzMSwxNTMuNjU3IEwgMTEyLjI5NSwxNTMuNDg5IEwgMTEyLjI0NCwxNTMuMzI1IEwgMTEyLjE3OSwxNTMuMTY2IEwgMTEyLjEsMTUzLjAxMiBMIDExMi4wMDksMTUyLjg2NyBMIDExMS45MDUsMTUyLjcyOSBMIDExMS43OSwxNTIuNjAxIEwgMTExLjY2NSwxNTIuNDgzIEwgMTExLjUzLDE1Mi4zNzcgTCAxMTEuMzg2LDE1Mi4yODIgTCAxMTEuMjM0LDE1Mi4yMDEgTCAxMTEuMDc2LDE1Mi4xMzIgTCAxMTAuOTEzLDE1Mi4wNzggTCAxMTAuNzQ2LDE1Mi4wMzcgTCAxMTAuNTc2LDE1Mi4wMTIgTCAxMTAuNDA0LDE1MiBMIDExMC4yMzIsMTUyLjAwNCBMIDExMC4wNjEsMTUyLjAyMyBMIDEwOS44OTIsMTUyLjA1NiBMIDEwOS43MjcsMTUyLjEwMyBMIDEwOS41NjYsMTUyLjE2NSBMIDEwOS40MTEsMTUyLjI0IEwgMTA5LjI2MywxNTIuMzI4IEwgMTA5LjEyNCwxNTIuNDI5IEwgMTA4Ljk5MywxNTIuNTQxIEwgMTA4Ljg3MywxNTIuNjY0IEwgMTA4Ljc2MywxNTIuNzk3IEwgMTA4LjY2NiwxNTIuOTM5IEwgMTA4LjU4MSwxNTMuMDg4IEwgMTA4LjUwOSwxNTMuMjQ1IEwgMTA4LjQ1MSwxNTMuNDA3IEwgMTA4LjQwNywxNTMuNTczIEwgMTA4LjM3OCwxNTMuNzQzIEwgMTA4LjM2MywxNTMuOTE0IEwgMTA4LjM2MywxNTQuMDg2IEwgMTA4LjM3OCwxNTQuMjU3IEwgMTA4LjQwNywxNTQuNDI3IEwgMTA4LjQ1MSwxNTQuNTkzIEwgMTA4LjUwOSwxNTQuNzU1IEwgMTA4LjU4MSwxNTQuOTEyIEwgMTA4LjY2NiwxNTUuMDYxIEwgMTA4Ljc2MywxNTUuMjAzIEwgMTA4Ljg3MywxNTUuMzM2IEwgMTA4Ljk5MywxNTUuNDU5IEwgMTA5LjEyNCwxNTUuNTcxIEwgMTA5LjI2MywxNTUuNjcyIEwgMTA5LjQxMSwxNTUuNzYgTCAxMDkuNTY2LDE1NS44MzUgTCAxMDkuNzI3LDE1NS44OTcgTCAxMDkuODkyLDE1NS45NDQgTCAxMTAuMDYxLDE1NS45NzcgTCAxMTAuMjMyLDE1NS45OTYgTCAxMTAuNDA0LDE1NiBMIDExMC41NzYsMTU1Ljk4OCBMIDExMC43NDYsMTU1Ljk2MyBMIDExMC45MTMsMTU1LjkyMiBMIDExMS4wNzYsMTU1Ljg2OCBMIDExMS4yMzQsMTU1Ljc5OSBMIDExMS4zODYsMTU1LjcxOCBMIDExMS41MywxNTUuNjIzIEwgMTExLjY2NSwxNTUuNTE3IEwgMTExLjc5LDE1NS4zOTkgTCAxMTEuOTA1LDE1NS4yNzEgTCAxMTIuMDA5LDE1NS4xMzMgTCAxMTIuMSwxNTQuOTg4IEwgMTEyLjE3OSwxNTQuODM0IEwgMTEyLjI0NCwxNTQuNjc1IEwgMTEyLjI5NSwxNTQuNTExIEwgMTEyLjMzMSwxNTQuMzQzIEwgMTEyLjM1NCwxNTQuMTcyIEwgMTEyLjM2MSwxNTQgTCAxMTAuMzYxLDE1NCBaJyBzdHlsZT0nZmlsbDojMDAwMDAwO2ZpbGwtcnVsZTpldmVub2RkO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowcHg7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlLW9wYWNpdHk6MTsnIC8+Cjwvc3ZnPgo= data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0nMS4wJyBlbmNvZGluZz0naXNvLTg4NTktMSc/Pgo8c3ZnIHZlcnNpb249JzEuMScgYmFzZVByb2ZpbGU9J2Z1bGwnCiAgICAgICAgICAgICAgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJwogICAgICAgICAgICAgICAgICAgICAgeG1sbnM6cmRraXQ9J2h0dHA6Ly93d3cucmRraXQub3JnL3htbCcKICAgICAgICAgICAgICAgICAgICAgIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJwogICAgICAgICAgICAgICAgICB4bWw6c3BhY2U9J3ByZXNlcnZlJwp3aWR0aD0nODVweCcgaGVpZ2h0PSc4NXB4JyB2aWV3Qm94PScwIDAgODUgODUnPgo8IS0tIEVORCBPRiBIRUFERVIgLS0+CjxyZWN0IHN0eWxlPSdvcGFjaXR5OjEuMDtmaWxsOiNGRkZGRkY7c3Ryb2tlOm5vbmUnIHdpZHRoPSc4NScgaGVpZ2h0PSc4NScgeD0nMCcgeT0nMCc+IDwvcmVjdD4KPHRleHQgeD0nMjAuMjA3NCcgeT0nNTMuNTkwOScgY2xhc3M9J2F0b20tMCcgc3R5bGU9J2ZvbnQtc2l6ZToyM3B4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6bm9uZTtmb250LWZhbWlseTpzYW5zLXNlcmlmO3RleHQtYW5jaG9yOnN0YXJ0O2ZpbGw6IzNCNDE0MycgPlM8L3RleHQ+Cjx0ZXh0IHg9JzM2LjIwMjgnIHk9JzUzLjU5MDknIGNsYXNzPSdhdG9tLTAnIHN0eWxlPSdmb250LXNpemU6MjNweDtmb250LXN0eWxlOm5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7ZmlsbC1vcGFjaXR5OjE7c3Ryb2tlOm5vbmU7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZjt0ZXh0LWFuY2hvcjpzdGFydDtmaWxsOiMzQjQxNDMnID5lPC90ZXh0Pgo8cGF0aCBkPSdNIDE3Ljg4OTIsMzkuNjgxOCBMIDE3Ljg4NDksMzkuNTgyMiBMIDE3Ljg3MiwzOS40ODMzIEwgMTcuODUwNywzOS4zODU4IEwgMTcuODIxMiwzOS4yOTA2IEwgMTcuNzgzNSwzOS4xOTgzIEwgMTcuNzM4LDM5LjEwOTUgTCAxNy42ODUxLDM5LjAyNSBMIDE3LjYyNTEsMzguOTQ1MyBMIDE3LjU1ODUsMzguODcxMSBMIDE3LjQ4NTcsMzguODAyOSBMIDE3LjQwNzMsMzguNzQxMiBMIDE3LjMyNCwzOC42ODY0IEwgMTcuMjM2MiwzOC42MzkxIEwgMTcuMTQ0NywzOC41OTk0IEwgMTcuMDUwMSwzOC41Njc4IEwgMTYuOTUzMSwzOC41NDQ0IEwgMTYuODU0NSwzOC41Mjk0IEwgMTYuNzU1LDM4LjUyMyBMIDE2LjY1NTMsMzguNTI1MSBMIDE2LjU1NjIsMzguNTM1OSBMIDE2LjQ1ODMsMzguNTU1IEwgMTYuMzYyNCwzOC41ODI2IEwgMTYuMjY5MywzOC42MTgzIEwgMTYuMTc5NiwzOC42NjE4IEwgMTYuMDkzOSwzOC43MTI5IEwgMTYuMDEzLDM4Ljc3MTIgTCAxNS45Mzc0LDM4LjgzNjIgTCAxNS44Njc2LDM4LjkwNzUgTCAxNS44MDQyLDM4Ljk4NDUgTCAxNS43NDc3LDM5LjA2NjcgTCAxNS42OTg1LDM5LjE1MzQgTCAxNS42NTY4LDM5LjI0NCBMIDE1LjYyMzIsMzkuMzM3OSBMIDE1LjU5NzcsMzkuNDM0MyBMIDE1LjU4MDYsMzkuNTMyNiBMIDE1LjU3MjEsMzkuNjMyIEwgMTUuNTcyMSwzOS43MzE3IEwgMTUuNTgwNiwzOS44MzEgTCAxNS41OTc3LDM5LjkyOTMgTCAxNS42MjMyLDQwLjAyNTcgTCAxNS42NTY4LDQwLjExOTYgTCAxNS42OTg1LDQwLjIxMDMgTCAxNS43NDc3LDQwLjI5NyBMIDE1LjgwNDIsNDAuMzc5MiBMIDE1Ljg2NzYsNDAuNDU2MiBMIDE1LjkzNzQsNDAuNTI3NCBMIDE2LjAxMyw0MC41OTI1IEwgMTYuMDkzOSw0MC42NTA3IEwgMTYuMTc5Niw0MC43MDE4IEwgMTYuMjY5Myw0MC43NDU0IEwgMTYuMzYyNCw0MC43ODExIEwgMTYuNDU4Myw0MC44MDg2IEwgMTYuNTU2Miw0MC44Mjc4IEwgMTYuNjU1Myw0MC44Mzg1IEwgMTYuNzU1LDQwLjg0MDYgTCAxNi44NTQ1LDQwLjgzNDIgTCAxNi45NTMxLDQwLjgxOTIgTCAxNy4wNTAxLDQwLjc5NTkgTCAxNy4xNDQ3LDQwLjc2NDIgTCAxNy4yMzYyLDQwLjcyNDYgTCAxNy4zMjQsNDAuNjc3MiBMIDE3LjQwNzMsNDAuNjIyNSBMIDE3LjQ4NTcsNDAuNTYwOCBMIDE3LjU1ODUsNDAuNDkyNiBMIDE3LjYyNTEsNDAuNDE4MyBMIDE3LjY4NTEsNDAuMzM4NyBMIDE3LjczOCw0MC4yNTQxIEwgMTcuNzgzNSw0MC4xNjU0IEwgMTcuODIxMiw0MC4wNzMgTCAxNy44NTA3LDM5Ljk3NzggTCAxNy44NzIsMzkuODgwNCBMIDE3Ljg4NDksMzkuNzgxNSBMIDE3Ljg4OTIsMzkuNjgxOCBMIDE2LjczMDEsMzkuNjgxOCBaJyBzdHlsZT0nZmlsbDojMDAwMDAwO2ZpbGwtcnVsZTpldmVub2RkO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowcHg7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlLW9wYWNpdHk6MTsnIC8+CjxwYXRoIGQ9J00gMTcuODg5Miw0NC4zMTgyIEwgMTcuODg0OSw0NC4yMTg1IEwgMTcuODcyLDQ0LjExOTYgTCAxNy44NTA3LDQ0LjAyMjIgTCAxNy44MjEyLDQzLjkyNyBMIDE3Ljc4MzUsNDMuODM0NiBMIDE3LjczOCw0My43NDU5IEwgMTcuNjg1MSw0My42NjEzIEwgMTcuNjI1MSw0My41ODE3IEwgMTcuNTU4NSw0My41MDc0IEwgMTcuNDg1Nyw0My40MzkyIEwgMTcuNDA3Myw0My4zNzc1IEwgMTcuMzI0LDQzLjMyMjggTCAxNy4yMzYyLDQzLjI3NTQgTCAxNy4xNDQ3LDQzLjIzNTggTCAxNy4wNTAxLDQzLjIwNDEgTCAxNi45NTMxLDQzLjE4MDggTCAxNi44NTQ1LDQzLjE2NTggTCAxNi43NTUsNDMuMTU5NCBMIDE2LjY1NTMsNDMuMTYxNSBMIDE2LjU1NjIsNDMuMTcyMiBMIDE2LjQ1ODMsNDMuMTkxNCBMIDE2LjM2MjQsNDMuMjE4OSBMIDE2LjI2OTMsNDMuMjU0NiBMIDE2LjE3OTYsNDMuMjk4MiBMIDE2LjA5MzksNDMuMzQ5MyBMIDE2LjAxMyw0My40MDc1IEwgMTUuOTM3NCw0My40NzI2IEwgMTUuODY3Niw0My41NDM4IEwgMTUuODA0Miw0My42MjA4IEwgMTUuNzQ3Nyw0My43MDMgTCAxNS42OTg1LDQzLjc4OTcgTCAxNS42NTY4LDQzLjg4MDQgTCAxNS42MjMyLDQzLjk3NDMgTCAxNS41OTc3LDQ0LjA3MDcgTCAxNS41ODA2LDQ0LjE2OSBMIDE1LjU3MjEsNDQuMjY4MyBMIDE1LjU3MjEsNDQuMzY4IEwgMTUuNTgwNiw0NC40Njc0IEwgMTUuNTk3Nyw0NC41NjU3IEwgMTUuNjIzMiw0NC42NjIxIEwgMTUuNjU2OCw0NC43NTYgTCAxNS42OTg1LDQ0Ljg0NjYgTCAxNS43NDc3LDQ0LjkzMzMgTCAxNS44MDQyLDQ1LjAxNTUgTCAxNS44Njc2LDQ1LjA5MjUgTCAxNS45Mzc0LDQ1LjE2MzggTCAxNi4wMTMsNDUuMjI4OCBMIDE2LjA5MzksNDUuMjg3MSBMIDE2LjE3OTYsNDUuMzM4MiBMIDE2LjI2OTMsNDUuMzgxNyBMIDE2LjM2MjQsNDUuNDE3NCBMIDE2LjQ1ODMsNDUuNDQ1IEwgMTYuNTU2Miw0NS40NjQxIEwgMTYuNjU1Myw0NS40NzQ5IEwgMTYuNzU1LDQ1LjQ3NyBMIDE2Ljg1NDUsNDUuNDcwNiBMIDE2Ljk1MzEsNDUuNDU1NiBMIDE3LjA1MDEsNDUuNDMyMiBMIDE3LjE0NDcsNDUuNDAwNiBMIDE3LjIzNjIsNDUuMzYwOSBMIDE3LjMyNCw0NS4zMTM2IEwgMTcuNDA3Myw0NS4yNTg4IEwgMTcuNDg1Nyw0NS4xOTcxIEwgMTcuNTU4NSw0NS4xMjg5IEwgMTcuNjI1MSw0NS4wNTQ3IEwgMTcuNjg1MSw0NC45NzUgTCAxNy43MzgsNDQuODkwNSBMIDE3Ljc4MzUsNDQuODAxNyBMIDE3LjgyMTIsNDQuNzA5NCBMIDE3Ljg1MDcsNDQuNjE0MiBMIDE3Ljg3Miw0NC41MTY3IEwgMTcuODg0OSw0NC40MTc4IEwgMTcuODg5Miw0NC4zMTgyIEwgMTYuNzMwMSw0NC4zMTgyIFonIHN0eWxlPSdmaWxsOiMwMDAwMDA7ZmlsbC1ydWxlOmV2ZW5vZGQ7ZmlsbC1vcGFjaXR5OjE7c3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjBweDtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2Utb3BhY2l0eToxOycgLz4KPC9zdmc+Cg== [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Abstract
The present invention discloses a kind of method of testing and system of artificial intelligence response system, and method includes：Obtaining includes the session aggregation of multiple session elements, and the session element is session or scene end of identification；Session context generation step, including：Session element is taken out in order from the session aggregation as current sessions element, creates session context；The problem of in session context to included by artificial intelligence response system input current sessions element, capture the actual answer of the artificial intelligence response system response described problem；Next session element is taken out in order from session aggregation as current sessions element, if current sessions element is session, then perform the conversation test step, otherwise session context is terminated, if there is next session element in session aggregation, session context generation step is then performed, is otherwise terminated.The present invention can cover the testing requirement of more scenes, multi-user and complicated premise calls.
Description
Technical field
The present invention relates to artificial intelligence correlative technology field, the method for testing of particularly a kind of artificial intelligence response system and
System.
Background technology
Artificial intelligence is the application for natural language processing, the content to be paid close attention to of the test to artificial intelligence and tradition
Automatic test it is inconsistent, particularly the web of artificial intelligence response system is applied, in addition to page function, main one
Individual test emphasis be corresponding input problem whether can to corresponding result, adjusted for the variation of natural language and model
Instability, test need to carry out it is substantial amounts of return checking, waste substantial amounts of manpower and time.
Prior art completes automatic test corresponding to artificial intelligence response system using such a way：
Use the automated test tool of increasing income based on Google:Selenium operates to web page, completes to put question to and returns
To answer, the preservation content of input results only includes problem and answer, and the session operation of next is directly performed after the completion of execution, and
Result is recorded.As a result record content only includes problem and actual result.
The shortcomings that prior art：
(1) problem result can be because the content of context enters different scenes so as to cause subproblem not enter into just
True handling process, as a result accuracy is inadequate.
(2) a kind of usage scenario can only be covered by once performing, and be not easy to switch in scene execution, different scenes is needed
The effect that just can guarantee that and reach more scenes is performed a plurality of times.
(3) session data sequential organization is indefinite, lacks execution logic.
(4) implementing result preserves imperfection, only exports corresponding result and preserves, is not easy to carry out data statistics and maintenance.
The content of the invention
Based on this, it is necessary to for the technical problem that prior art test accuracy is inadequate, there is provided a kind of artificial intelligence should
Answer the method for testing and system of system.
A kind of method of testing of artificial intelligence response system, including：
Session aggregation obtaining step, including：Obtaining includes the session aggregation of multiple session elements, and the session element is meeting
Words or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each described
Session includes：Problem, scene identity and expected answer；
Session context generation step, including：Session element is taken out in order from the session aggregation as current sessions
Element, if current sessions element is session, create session context；
Conversation test step, including：Wrapped in session context to artificial intelligence response system input current sessions element
The problem of including, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current meeting
The expection answer matches included by element are talked about, then the test result for recording current sessions element is by otherwise recording current meeting
The test result for talking about element is failure；
Session element checking step, including：Next session element is taken out in order from session aggregation as current meeting
Element is talked about, if current sessions element is scene end of identification, terminates session context, performs and terminate checking step, if worked as
Preceding session element is session, then performs the conversation test step；
Terminate checking step, including：If there is next session element in session aggregation, session context generation is performed
Step, otherwise terminate.
A kind of test system of artificial intelligence response system, including：
Session aggregation acquisition module, is used for：Obtaining includes the session aggregation of multiple session elements, and the session element is meeting
Words or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each described
Session includes：Problem, scene identity and expected answer；
Session context generation module, is used for：Session element is taken out in order from the session aggregation as current sessions
Element, if current sessions element is session, create session context；
Conversation test module, is used for：Wrapped in session context to artificial intelligence response system input current sessions element
The problem of including, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current meeting
The expection answer matches included by element are talked about, then the test result for recording current sessions element is by otherwise recording current meeting
The test result for talking about element is failure；
Session element checks module, is used for：Next session element is taken out in order from session aggregation as current meeting
Element is talked about, if current sessions element is scene end of identification, terminates session context, performs and terminates to check module, if worked as
Preceding session element is session, then performs the conversation test module；
Terminate to check module, be used for：If there is next session element in session aggregation, session context generation is performed
Module, otherwise terminate.
The present invention passes through the flow that increase scene end of identification and correlation are concentrated in session so that automatic test can be
The situation of multiple scenes is once tackled in complete test, more scenes, multi-user and complicated premise calls can be covered
Testing requirement.
Brief description of the drawings
Fig. 1 is a kind of workflow diagram of the method for testing of artificial intelligence response system of the present invention；
Fig. 2 is the workflow diagram of preferred embodiment；
Fig. 3 is a kind of construction module figure of the test system of artificial intelligence response system of the present invention.
Embodiment
The present invention will be further described in detail with specific embodiment below in conjunction with the accompanying drawings.
It is as shown in Figure 1 a kind of workflow diagram of the method for testing of artificial intelligence response system of the present invention, including：
Step S101, including：Obtaining includes the session aggregation of multiple session elements, and the session element is session or field
Scape end of identification, at least one session element that the session aggregation includes is scene end of identification, and each session includes：
Problem, scene identity and expected answer；
Step S102, including：Session element is taken out in order from the session aggregation as current sessions element, if
Current sessions element is session, then creates session context；
Step S103, including：To included by artificial intelligence response system input current sessions element in session context
Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member
Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member
The test result of element is failure；
Step S104, including：Next session element is taken out in order from session aggregation as current sessions element, such as
Fruit current sessions element is scene end of identification, then terminates session context, step S105 is performed, if current sessions element is meeting
Words, then perform the step S103；
Step S105, including：If there is next session element in session aggregation, step S102 is performed, is otherwise tied
Beam.
The present invention obtains session aggregation by step S101, includes session and scene end of identification in session aggregation,
Meanwhile session context is created in step s 102, and in step S104, if current sessions element is scene end of identification,
Then terminate session context, so as to realize the differentiation to different sessions scene, before more scenes, multi-user, complexity can be covered
The testing requirement made demands, response logic is avoided to be influenceed by context.
Wherein, in step s 102, if the session element taken out is not session, can be ignored.
In one of the embodiments, for the establishment session context to log in browser, the end session context is to close
Close browser.
The present embodiment ensures current sessions to scenery control mainly in the cookies by refreshing current sessions scene
The independence of scene.The present invention, by wanting scene set end of identification, allows program or script, it is realized that working as when being handled
Test is when proceed to this, and all sessions for representing this scene have finished, it is necessary to cookies to current sessions
It is updated, or old cookies is cleared up, ensures that a new session context can be entered during next session
(session).The mode the present embodiment for updating session is realized using browser is closed.
Browser is logged in when performing step S102, and when performing step S104, if current sessions element is scene
End of identification, then browser is closed, so that the cookies of session context is updated.
Preferably, when performing session start, mark that a scene can be set to start, such as a variable
EndFlage, perform initialize when while starting (or new scene), be set to a value, such as False.When into one
During individual scene, session is proceeded by by triggering the action of login browser, will be from team when current session is completed in execution
Next session is taken out in row, judges whether this session content includes the scene end of identification of setting, if do not included,
By the contents extraction of this session, performed.If it find that comprising scene end of identification, then EndFlage is set to True,
Program will close browser according to this EndFlage.So as to complete a complete scene.
Such flow guarantees not perform session in by the environment of context interference at one, has so as to reach distinctive
Targetedly test purpose.
In one of the embodiments, the session also includes the account information associated with scene identity, the step
In S103, logging in the artificial intelligence using the account information should system.
In general, automatic test includes multiple test cases, and each test case completes one or session.Pass through this
Invention so that the test case of automatic test can include multiple test scenes, and by by account information and scene identity
It is associated, then different account informations can be used for different session contexts, so as to accomplish test process completely by counting
It is driven according to test case.
In one of the embodiments, the session aggregation is carried out test data by the way of form and preserved, its
In per data line be a session element.
, it is necessary to carry out test data support, this reality by substantial amounts of language material in the test based on artificial intelligence response system
Example is applied in order to be more easily managed to test data, the mode for employing form carries out test data and preserved, and protects
Deposit form with behavior unit to be split, data line is a session element.Different column datas represent different session informations.
In one of the embodiments, the step S103, in addition to：Newly-generated result object, by current sessions element
Test result write the result object and preserve.
In the present embodiment, after actual answer corresponding to grabbing, expected answer is contrasted with actual answer at once,
So as to obtain test result.Due to directly being contrasted and being preserved after completing to capture, so as to ensure the validity of result.Together
When, due to saving test result in result object, therefore when follow-up output result is shown can definitely and effectively,
Analyticity strengthens.
The workflow diagram of preferred embodiment is illustrated in figure 2, including：
Step S201, read session aggregation.
Session aggregation includes multiple session elements, and session element can be session or scene end of identification.Each session
Preserving type it is as shown in table 1.
The preserving type of the session of table 1
| Problem | Scene identity | Test account | It is expected that answer |
In addition to distinguishing different scenes, the invalid session mark of a tape identification can be added in test session set
Remember the end of scene, i.e. scene end of identification, be the data framework of session aggregation as shown in table 2.
The data framework of the session aggregation of table 2
| Problem | Scene identity | Test account | It is expected that answer |
| Problem | Scene identity | Test account | It is expected that answer |
| Scene end of identification | It is empty | It is empty | It is empty |
| Problem | Scene identity | Test account | It is expected that answer |
| Problem | Scene identity | Test account | It is expected that answer |
Before perform script or flow, test data can be stored in number often capable in internal memory by the first step in the form of queue
An independent object is stored in session list according to this, and the data of corresponding row are sequentially added into session team in a manner of poll
Row.Session is set to form the list of session in a certain order.The follow-up execution of aspect.The easy handling guarantee of queuing data
The flexibility of digital independent, operation easily can be entered to session data.
In order to simplify maintenance cost, herein in actual treatment the session of Same Scene by first of only each scene
Session includes scene identity and account information.Subsequently will no longer it change.Until scene end of identification occurs.
All sessions can be read in a manner of poll to be extracted, and by queue (List) access characteristics,
The session of corresponding mark is directly accessed in the case of specific, ensures the operability of data.
Step S202, generate new session context (Session).
Preferably, new session is generated by logging in browser.
Step S203, problem is inputted to artificial intelligence response system.
Step S204, the actual answer that crawl artificial intelligence response system is returned.
Step S205, obtain the comparing result after actual answer is contrasted with expected answer.
After content corresponding to having captured, corresponding expected answer and the actual answer actually obtained are contrasted, pin
To including expected answer or the situation completely the same with expected answer in actual answer, then the test result of this session can be given
Labeled as Pass (expression passes through), do not include if actual answer or can be surveyed if being not equal to expected answer completely to this session
Test result is labeled as Fail (expression does not pass through).
Step S206, preserve test result.
After completion result verification is performed, corresponding test result is write into result document.Then perform again next time
Session is read, and after each session execution verifies, the content of session generates a result object, and this result object is led to
Cross wiring method and write direct destination file, preserved.
Step S207, if scene end point then performs step S208, otherwise perform step S203.
If the session element read is scene end of identification, it is judged as scene end point, is otherwise judged as not having also
To scene end point；
Step S208, close current session.
Current session closing is realized by closing browser.
Step S209, if also including next session in session aggregation, step S202 is performed, otherwise performs step
S210。
Step S210, preserve final result.
After performing and completing all session, can finally preserve once result, finally can by reading result document or
Carry out the interpretation of result of next step.In order to which easy analysis result object comprises at least content as shown in table 3.
The content of the result object of table 3
| Problem | It is expected that answer | Actual answer | Test result |
A kind of construction module figure of the test system of artificial intelligence response system of the present invention is illustrated in figure 3, including：
Session aggregation acquisition module 301, is used for：Obtaining includes the session aggregation of multiple session elements, the session element
For session or scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each
The session includes：Problem, scene identity and expected answer；
Session context generation module 302, is used for：Session element is taken out in order from the session aggregation as current meeting
Element is talked about, if current sessions element is session, creates session context；
Conversation test module 303, is used for：In session context current sessions element institute is inputted to artificial intelligence response system
Including the problem of, capture the actual answer of artificial intelligence response system response described problem, if actual answer with it is current
Expection answer matches included by session element, the then test result for recording current sessions element are by the way that otherwise record is current
The test result of session element is failure；
Session element checks module 304, is used for：Next session element is taken out in order from session aggregation as current
Session element, if current sessions element is scene end of identification, terminate session context, perform and terminate to check module 305, such as
Fruit current sessions element is session, then performs the conversation test module 303；
Terminate to check module 305, be used for：If there is next session element in session aggregation, session context is performed
Generation module 302, otherwise terminates.
In one of the embodiments, for the establishment session context to log in browser, the end session context is to close
Close browser.
In one of the embodiments, the session also includes the account information associated with scene identity, and the session is surveyed
In die trial block, logging in the artificial intelligence using the account information should system.
In one of the embodiments, the session aggregation is carried out test data by the way of form and preserved, its
In per data line be a session element.
In one of the embodiments, the conversation test module, in addition to：Newly-generated result object, by current sessions
The test result of element writes the result object and preserved.
Embodiment described above only expresses the several embodiments of the present invention, and its description is more specific and detailed, but simultaneously
Therefore the limitation to the scope of the claims of the present invention can not be interpreted as.It should be pointed out that for one of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to the guarantor of the present invention
Protect scope.Therefore, the protection domain of patent of the present invention should be determined by the appended claims.
Claims (10)
- A kind of 1. method of testing of artificial intelligence response system, it is characterised in that including：Session aggregation obtaining step, including：Obtaining includes the session aggregations of multiple session elements, the session element for session or Person's scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each session Including：Problem, scene identity and expected answer；Session context generation step, including：Session element is taken out in order from the session aggregation as current sessions element, If current sessions element is session, session context is created；Conversation test step, including：To included by artificial intelligence response system input current sessions element in session context Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member The test result of element is failure；Session element checking step, including：Next session element is taken out in order from session aggregation as current sessions member Element, if current sessions element is scene end of identification, terminate session context, perform and terminate checking step, if current meeting Words element is session, then performs the conversation test step；Terminate checking step, including：If there is next session element in session aggregation, session context generation step is performed Suddenly, otherwise terminate.
- 2. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the establishment session field For scape to log in browser, the end session context is closing browser.
- 3. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the session also includes The account information associated with scene identity, in the conversation test step, the artificial intelligence is logged in using the account information Should system.
- 4. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the session aggregation is adopted Test data is carried out with the mode of form to be preserved, each of which row data are a session elements.
- 5. the method for testing of artificial intelligence response system according to claim 1, it is characterised in that the conversation test step Suddenly, in addition to：Newly-generated result object, the test result of current sessions element is write into the result object and preserved.
- A kind of 6. test system of artificial intelligence response system, it is characterised in that including：Session aggregation acquisition module, is used for：Obtaining includes the session aggregations of multiple session elements, the session element for session or Person's scene end of identification, at least one session element that the session aggregation includes is scene end of identification, each session Including：Problem, scene identity and expected answer；Session context generation module, is used for：Session element is taken out in order from the session aggregation as current sessions element, If current sessions element is session, session context is created；Conversation test module, is used for：To included by artificial intelligence response system input current sessions element in session context Problem, the actual answer of the artificial intelligence response system response described problem is captured, if actual answer and current sessions member Expection answer matches included by element, the then test result for recording current sessions element are by otherwise recording current sessions member The test result of element is failure；Session element checks module, is used for：Next session element is taken out in order from session aggregation as current sessions member Element, if current sessions element is scene end of identification, terminate session context, perform and terminate to check module, if current meeting Words element is session, then performs the conversation test module；Terminate to check module, be used for：If there is next session element in session aggregation, session context generation mould is performed Block, otherwise terminate.
- 7. the test system of artificial intelligence response system according to claim 6, it is characterised in that the establishment session field For scape to log in browser, the end session context is closing browser.
- 8. the test system of artificial intelligence response system according to claim 6, it is characterised in that the session also includes The account information associated with scene identity, in the conversation test module, the artificial intelligence is logged in using the account information Should system.
- 9. the test system of artificial intelligence response system according to claim 6, it is characterised in that the session aggregation is adopted Test data is carried out with the mode of form to be preserved, each of which row data are a session elements.
- 10. the test system of artificial intelligence response system according to claim 6, it is characterised in that the conversation test Module, in addition to：Newly-generated result object, the test result of current sessions element is write into the result object and preserved.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510193639.0A CN104809062B (en) | 2015-04-22 | 2015-04-22 | A kind of method of testing and system of artificial intelligence response system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510193639.0A CN104809062B (en) | 2015-04-22 | 2015-04-22 | A kind of method of testing and system of artificial intelligence response system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104809062A CN104809062A (en) | 2015-07-29 |
| CN104809062B true CN104809062B (en) | 2017-12-01 |
Family
ID=53693904
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510193639.0A CN104809062B (en) | 2015-04-22 | 2015-04-22 | A kind of method of testing and system of artificial intelligence response system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104809062B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107402878B (en) * | 2016-05-19 | 2020-09-01 | 北京京东尚科信息技术有限公司 | Test method and device |
| CN106503242B (en) * | 2016-11-08 | 2019-04-19 | 上海智臻智能网络科技股份有限公司 | A kind of intelligent interactive method, device and server |
| CN107463640B (en) * | 2017-07-19 | 2020-12-18 | 一拓通信集团股份有限公司 | Artificial intelligence based information data acquisition method |
| CN110908919A (en) * | 2019-12-02 | 2020-03-24 | 上海市软件评测中心有限公司 | Response test system based on artificial intelligence and application thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1744056A (en) * | 2004-09-04 | 2006-03-08 | 华为技术有限公司 | Automatic test method of application |
| US7945899B2 (en) * | 2003-06-16 | 2011-05-17 | American Megatrends, Inc. | Method and system for remote software testing |
| CN103049367A (en) * | 2011-10-11 | 2013-04-17 | 神州数码信息系统有限公司 | Automatic testing method for software |
| CN103853842A (en) * | 2014-03-20 | 2014-06-11 | 百度在线网络技术（北京）有限公司 | Automatic question and answer method and system |
| CN104142959A (en) * | 2013-05-10 | 2014-11-12 | 中国银联股份有限公司 | Automated test device and method based on expert knowledge database |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9336124B2 (en) * | 2013-02-28 | 2016-05-10 | Apple Inc. | Automatic remote execution of an application |
-
2015
- 2015-04-22 CN CN201510193639.0A patent/CN104809062B/en active IP Right Grant
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7945899B2 (en) * | 2003-06-16 | 2011-05-17 | American Megatrends, Inc. | Method and system for remote software testing |
| CN1744056A (en) * | 2004-09-04 | 2006-03-08 | 华为技术有限公司 | Automatic test method of application |
| CN103049367A (en) * | 2011-10-11 | 2013-04-17 | 神州数码信息系统有限公司 | Automatic testing method for software |
| CN104142959A (en) * | 2013-05-10 | 2014-11-12 | 中国银联股份有限公司 | Automated test device and method based on expert knowledge database |
| CN103853842A (en) * | 2014-03-20 | 2014-06-11 | 百度在线网络技术（北京）有限公司 | Automatic question and answer method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104809062A (en) | 2015-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10243679B2 (en) | Vulnerability detection | |
| US10929879B2 (en) | Method and apparatus for identification of fraudulent click activity | |
| Tambosi et al. | A framework to optimize biodiversity restoration efforts based on habitat amount and landscape connectivity | |
| TWI621077B (en) | Character recognition method and server for claim documents | |
| CN107358157B (en) | Face living body detection method and device and electronic equipment | |
| CA3001839C (en) | Call detail record analysis to identify fraudulent activity and fraud detection in interactive voice response systems | |
| CN104268006B (en) | The back method and device of key mouse script | |
| US8099443B2 (en) | Apparatus for building address database and method thereof | |
| CN104123500B (en) | A kind of Android platform malicious application detection method and device based on deep learning | |
| CN104104717B (en) | Deliver channel data statistical approach and device | |
| CN107239666B (en) | Method and system for desensitizing medical image data | |
| US9015685B2 (en) | Code analysis for simulation efficiency improvement | |
| CN106227494B (en) | A kind of application operating method and terminal | |
| CN105844283A (en) | Method for identifying category of image, image search method and image search device | |
| CN107145799A (en) | A kind of data desensitization method and device | |
| US8332509B2 (en) | Parameterized computer monitoring system | |
| TWI677846B (en) | Method and device for transferring robot customer service to manual customer service | |
| CN104580451B (en) | Realize the method and system and electronic equipment of multi-platform account synchronization | |
| CN105989283B (en) | A kind of method and device identifying virus mutation | |
| CN107483458A (en) | The recognition methods of network attack and device, computer-readable recording medium | |
| CN103927484A (en) | Malicious program behavior capture method based on Qemu | |
| CA2994770A1 (en) | Model integration tool | |
| CN105184160B (en) | A kind of method of the Android phone platform application program malicious act detection based on API object reference relational graphs | |
| CN102930210A (en) | System and method for automatically analyzing, detecting and classifying malicious program behavior | |
| CN103607413B (en) | Method and device for detecting website backdoor program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |