CN111986652A - Vehicle-mounted intelligent voice system testing and evaluating method - Google Patents
Vehicle-mounted intelligent voice system testing and evaluating method Download PDFInfo
- Publication number
- CN111986652A CN111986652A CN202010760388.0A CN202010760388A CN111986652A CN 111986652 A CN111986652 A CN 111986652A CN 202010760388 A CN202010760388 A CN 202010760388A CN 111986652 A CN111986652 A CN 111986652A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- voice
- testing
- environment
- test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 142
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000006870 function Effects 0.000 claims abstract description 32
- 230000004044 response Effects 0.000 claims abstract description 14
- 238000010998 test method Methods 0.000 claims description 28
- 230000003993 interaction Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 4
- 238000012430 stability testing Methods 0.000 claims description 4
- 230000003068 static effect Effects 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 5
- 238000013461 design Methods 0.000 abstract description 3
- 238000011156 evaluation Methods 0.000 description 11
- 238000011161 development Methods 0.000 description 2
- 230000002618 waking effect Effects 0.000 description 2
- 238000004378 air conditioning Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000013112 stability test Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/01—Assessment or evaluation of speech recognition systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a vehicle-mounted intelligent voice system testing and evaluating method, which comprises a voice awakening success rate, a voice false awakening rate, a voice recognition success rate, a sound source positioning accuracy rate, a voiceprint identification rate, an echo eliminating capability, response recognition time, voice resource occupation and system stability, and provides a corresponding testing method. The testing method disclosed by the invention not only covers the most basic functions of the vehicle-mounted intelligent voice system, namely the awakening success rate and the recognition success rate, ensures that the basic functions can meet the design and use requirements, but also provides the test contents of the false awakening rate, the sound source positioning accuracy rate, the echo eliminating effect and the recognition response time, can objectively evaluate the experience of the voice system, and ensures that the voice system reduces the bad experience brought to users. The contents of voice resource occupation and system stability require that the performance of the voice system is stable and reliable, and the overall performance of the vehicle-mounted sound entertainment host is not influenced.
Description
Technical Field
The invention relates to the field of automobiles, in particular to a vehicle-mounted intelligent voice system testing and evaluating method based on a vehicle-mounted sound entertainment system.
Background
At present, with the development of the automobile industry and the internet industry, the intelligent internet automobile is gradually changed, the functions are continuously powerful, and the vehicle-mounted intelligent voice system is one of the products. The vehicle-mounted intelligent voice system is embedded in a vehicle-mounted intelligent sound host machine to be developed, interaction between a person and a vehicle and control of the person on the vehicle can be achieved, and the vehicle-mounted intelligent voice system has a primary AI function. However, because the working environment of the automobile is complex, the awakening success rate and the recognition success rate of the vehicle-mounted voice are often greatly different under different working conditions, and different hardware carried by different vehicle-mounted sound hosts have different resource occupancy rates of the voice to the host system, so that the use of the voice may have a large influence on the performance of the host, which is a problem to be avoided by the voice system.
Common intelligent networking car in the existing market all carries with intelligent voice system, supports the people and realizes people to the control of intelligent stereo set host computer, the control of people to whole car part function through awakening up to pronunciation to and partial amusement experience. The current voice testing method mainly starts from basic functions of ensuring man-machine conversation, and has the core of the success rate of awakening and recognizing voice by people. However, as the user's experience requirements on the intelligent automobile become higher and higher, the voice false wake-up rate, the sound source positioning accuracy rate, the voiceprint identification rate, the echo cancellation capability, the response recognition time, the voice resource occupation, and the voice system stability become one of the metrics of the voice quality, and have a greater influence on the user experience.
The automobile product with the vehicle-mounted intelligent voice system in the market at present has different voice functions due to the difference of a vehicle-mounted sound entertainment host system, some automobile products only support the physical awakening of a hard key, some automobile products support the voice awakening, some automobile products support the awakening without voice, and the interactivity of the voice is greatly different. With the development of the vehicle-mounted audio entertainment host system, the vehicle-mounted intelligent voice system is inevitably developed towards the functional trends of voice awakening, awakening-free, visible and readable, intelligent conversation, one-language-one-arrival, partition voice and the like, and with the higher and higher requirements of users on the experience of automobile products, the experience of voice is one of the important standards for evaluating the quality of the voice system. Therefore, the vehicle-mounted voice system testing and evaluating method only aiming at the voice awakening rate and the recognition success rate is lack of comprehensiveness, and the basic functionality and the experience of the voice system cannot be accurately judged, so that the quality of the voice system cannot be objectively judged.
Disclosure of Invention
The invention mainly aims to provide a vehicle-mounted intelligent voice system testing and evaluating method, provides a special testing method and evaluating indexes aiming at basic use functions, use experience and use stability of vehicle-mounted intelligent voice, and is favorable for objectively evaluating the quality of a voice system.
The technical scheme adopted by the invention is as follows: a vehicle-mounted intelligent voice system testing and evaluating method comprises the following steps:
testing the success rate of voice awakening;
testing the voice false wake-up rate;
testing the success rate of voice recognition;
testing the sound source positioning accuracy;
testing the voiceprint identification rate;
testing the echo cancellation capability;
responding to the recognition time test;
testing voice resource occupation;
and (5) testing the stability of the system.
Further, the voice wake-up success rate test includes:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the test method comprises the following steps:
testing the awakening rate of the awakening words in different scenes, wherein the single scene needs to be tested at least 100 times;
wake-up success rate (number of correct wake-ups/total number of tests) 100%.
Further, the voice false wake-up rate test comprises:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the test method comprises the following steps:
continuously playing a section of recording without awakening words by using a high-fidelity sound box in the vehicle, wherein the testing time is more than 6h, and recording the awakening times;
false wake-up rate is the number of false wake-ups/total duration (h).
Further, the voice recognition success rate test comprises:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the reliable network connection of the vehicle-mounted sound entertainment host is ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps; WiFi may also be connected, but the upload and download speeds are still no lower than 100kbps and 50 kbps.
The test method comprises the following steps:
based on a voice function corpus list supported by a vehicle-mounted voice system, all function corpora are executed under each test environment, each test is carried out for at least 3 times, if the recognition is correct once in the recognition process, the function is considered to be normal, and the times of normal functions are counted after the test is finished;
the recognition success rate is (number of functional normalizations/total number of tests) 100%.
Further, the sound source positioning accuracy test comprises:
and (3) testing environment: the vehicle is in a static quiet environment;
the test method comprises the following steps: firstly, a main driver wakes up voice and then speaks a random instruction, and a secondary driver uses 5 effective instruction sounds to interfere; and (5) repeating the operation after the main and auxiliary drives are exchanged for 2 h.
Still further, the voiceprint discrimination test comprises:
and (3) testing environment: the vehicle is in a static quiet environment;
the test method comprises the following steps: randomly selecting 5 testers, respectively registering voiceprints and naming under an intelligent voice full mode; five persons who have registered voiceprints respectively carry out 20 independent awakening tests, record the awakening success times, and repeatedly operate in a full mode and a sound source positioning mode; and repeating the operation by a sixth test person who does not register the voiceprint, and recording the awakening success times.
Still further, the echo cancellation capability test comprises:
and (3) testing environment: a vehicle standing environment;
the test method comprises the following steps: randomly playing the linguistic data with the voice feedback broadcast, continuously playing for 6 hours, judging according to the result of the voice typewriter, and recording the recognized times of the voice feedback broadcast; randomly playing the multimedia content of the non-melody voice, keeping a voice awakening state, playing for 6h in a circulating way, and recording the frequency of the multimedia voice recorded by voice recognition; and respectively testing the normal decibel environment and the high decibel environment in the vehicle.
Still further, the response identifies a time test;
and (3) testing environment: the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
intercepting a voice log of a vehicle-mounted sound entertainment host, judging a time interval from the end of recording to the acquisition of an identification result through a log time point, and specially setting a log intercepting mode aiming at a tested host;
after the method for intercepting the voice log is selected, the tested vehicle environment is randomly selected based on the corpus functions supported by the voice system, all the corpora are executed, each corpus is executed once, and the average time of recognition response of each corpus is calculated through the voice log.
Further, the voice resource occupation test;
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
the background of the vehicle-mounted sound entertainment host only keeps running applications and services which are started along with the host by default, a relevant test recording tool is connected for voice interaction, the memory occupation value and the CPU occupation value are recorded every 1s, and after the interaction is finished, the memory occupation mean value and the peak value in the process and the CPU occupation mean value and the peak value are counted.
Still further, the system stability test includes:
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
starting the equipment and eliminating the problems of the equipment, continuously testing according to a voice interface calling flow through a batch stability testing tool, generating efficiency data, synchronously monitoring a CPU (central processing unit) and a memory, keeping running for 48h, and checking whether abnormal phenomena such as identification rate reduction, memory leakage, voice collapse and the like occur.
The invention has the advantages that:
the testing method disclosed by the invention not only covers the most basic functions of the vehicle-mounted intelligent voice system, namely the awakening success rate and the recognition success rate, ensures that the basic functions can meet the design and use requirements, but also provides the test contents of the false awakening rate, the sound source positioning accuracy rate, the echo eliminating effect and the recognition response time, can objectively evaluate the experience of the voice system, and ensures that the voice system reduces the bad experience brought to users. The test content of the voiceprint authentication rate is the requirement that the voice system maximize the recognition accuracy of the registered voiceprint. The contents of voice resource occupation and system stability require that the performance of the voice system is stable and reliable, and the overall performance of the vehicle-mounted sound entertainment host is not influenced.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention.
FIG. 1 is a flow chart of a vehicle-mounted intelligent voice system testing and evaluating method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following terms and definitions apply to the present invention:
speech recognition, the system recognizes and understands the speech signal through the recording tool, the noise reduction chip and the built-in software, and converts the speech signal into corresponding text or instruction
Awakening, namely enabling the voice assistant to be activated and in a state of waiting for inputting instructions through a physical hard key or a specific awakening vocabulary
One shot (One word) and a continuous recognition mode of 'awakening word + semantic recognition', realizes seamless docking between awakening and control, has the characteristics of low delay, small interval and strong continuity, and conforms to the habit of user speaking continuity
And when voice is awakened, the position of the speaker is quickly determined through a sound source positioning technology, and only the awakening direction is recorded, so that the interference of other noise around the awakened speaker in the instruction sending process can be effectively avoided, and the identification rate is effectively improved.
The method is characterized in that the method is free of awakening, a dialogue mode capable of awakening once and inputting a plurality of instructions sequentially is extended on the basis of the traditional question-answer interaction mode, and the function of directly speaking the corpus without awakening the support part
The echo cancellation function can suppress echo signals and make the system only collect sounds for waking up people.
Referring to fig. 1, as shown in fig. 1, the present invention provides a testing and evaluating method for an intelligent voice system on a vehicle, which includes a voice wake-up success rate, a voice false wake-up rate, a voice recognition success rate, a sound source positioning accuracy rate, a voiceprint identification rate, an echo cancellation capability, a response recognition time, a voice resource occupation, and a system stability, and provides a corresponding testing method.
The special test method and the evaluation index provided aiming at the basic use function, the use experience and the use stability of the vehicle-mounted intelligent voice are beneficial to objectively evaluating the quality of the voice system.
The testing method disclosed by the invention not only covers the most basic functions of the vehicle-mounted intelligent voice system, namely the awakening success rate and the recognition success rate, ensures that the basic functions can meet the design and use requirements, but also provides the test contents of the false awakening rate, the sound source positioning accuracy rate, the echo eliminating effect and the recognition response time, can objectively evaluate the experience of the voice system, and ensures that the voice system reduces the bad experience brought to users. The test content of the voiceprint authentication rate is the requirement that the voice system maximize the recognition accuracy of the registered voiceprint. The contents of voice resource occupation and system stability require that the performance of the voice system is stable and reliable, and the overall performance of the vehicle-mounted sound entertainment host is not influenced.
Awakening success rate:
and (3) testing environment:
the test environment has a great influence on the test result, so that certain limitation needs to be made on the test environment. The volume of the environment in the vehicle has a normal decibel environment and a high decibel environment. The environment outside the vehicle is divided into a quiet environment and a noisy environment, and is mainly influenced by the driving environment, the driving condition, the window switch and the air speed of an air conditioner.
The environment in the vehicle:
normal decibel (wakening availability: available): the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary in the automobile can normally communicate.
High decibel (wake up availability: basic available): the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; subjective feeling is that people at main and auxiliary driving positions in the automobile have difficulty in communication and the volume in the automobile is large.
Environment outside the vehicle:
a quiet environment: a driving environment-parking lot is arranged in the corresponding scene; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air-conditioning wind speed-off or not exceeding 2 grades
Noisy environment: corresponding scenes comprise driving environments, namely urban areas, suburbs, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; air speed of air conditioner is above-2 grade
The test method comprises the following steps:
and testing the awakening rate of the awakening word in each different scene, wherein the single scene needs to be tested at least 100 times.
Wake-up success rate (number of correct wake-ups/total number of tests) 100%.
Evaluation indexes are as follows:
under the normal decibel environment in the car: the quiet environment outside the vehicle is more than or equal to 90 percent, and the noisy environment outside the vehicle is more than or equal to 80 percent;
in-vehicle high decibel environment: the quiet environment outside the vehicle is more than or equal to 80 percent, and the noisy environment outside the vehicle is more than or equal to 75 percent;
an example of a test report format is shown in table 1.
Table 1 wake-up success rate test report example
False wake-up rate: the test environment is the same as above;
the test method comprises the following steps:
and continuously playing a section of recording without awakening words by using a high-fidelity sound box in the vehicle, wherein the testing time is more than 6h, and recording the awakening times.
False wake-up rate is the number of false wake-up times/total duration (h)
Evaluation indexes are as follows:
the false awakening rate is less than or equal to 0.5 times/h.
And (3) identification success rate:
and (3) testing environment:
on the basis of the environment, the environment in the vehicle is not further distinguished, and the environment outside the vehicle is further extracted to form a high-noise environment on the basis of a noisy environment (the corresponding scene is that a window is opened, and the air speed of an air conditioner is more than 2 grades); the reliable network connection of the vehicle-mounted sound entertainment host is ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps; WiFi may also be connected, but the upload and download speeds are still no lower than 100kbps and 50 kbps.
The test method comprises the following steps:
based on a voice function corpus list supported by a vehicle-mounted voice system, all function corpora are executed under each test environment, each test is carried out for at least 3 times, if the recognition is correct once in the recognition process, the function is considered to be normal, and the times of normal functions are counted after the test is finished.
The recognition success rate is (number of functional normalizations/total number of tests) 100%.
Evaluation indexes are as follows:
the quiet environment is more than or equal to 90 percent;
the noisy environment is more than or equal to 85 percent;
the high noise environment is more than or equal to 80 percent.
An example of a test report format is shown in table 2.
Table 2 identification success rate test report example
Sound source localization accuracy (applicable only to vehicle-mounted intelligent voice systems with sound source distinguishing function):
and (3) testing environment:
the vehicle is in a quiet environment.
The test method comprises the following steps:
firstly, a main driver wakes up voice and then speaks a random instruction, and a secondary driver uses 5 effective instruction sounds to interfere; and (5) repeating the operation after the main and auxiliary drives are exchanged for 2 h.
After the main driving is awakened, except for the auxiliary driving, the back row personnel on the vehicle carry out voice instruction interference.
The increasing method comprises the following steps: the main driving tester uses the awakening words to perform voice awakening, the awakening times are not less than 100 times, whether the positioning sound source after voice awakening is located in the main driving position or not is observed, and in principle, the sound source is positioned at the main driving position, so that when people at other positions input instructions, a voice system does not recognize the instructions. And the main and auxiliary drives are exchanged and then repeatedly operated.
Evaluation indexes are as follows:
the sound source positioning accuracy is more than or equal to 99 percent, namely, the condition that one side wakes up the other side does not occur basically.
Voiceprint discrimination:
and (3) testing environment:
the vehicle is in a quiet environment.
The test method comprises the following steps:
randomly selecting 5 testers, respectively registering voiceprints and naming under an intelligent voice full mode; five persons who have registered voiceprints respectively carry out 20 independent awakening tests, record the awakening success times, and repeatedly operate in a full mode and a sound source positioning mode (if sound source distinguishing is supported); and repeating the operation by a sixth test person who does not register the voiceprint, and recording the awakening success times.
In case of waking up voice, the voiceprint unregistered person A recites the same session repeatedly at least 20 times, and the number of recorded words is recorded.
The discrimination success rate is 1-number of recognized words/total number of recognized words.
Evaluation indexes are as follows:
the voiceprint identification success rate is more than or equal to 85 percent;
the false voiceprint identification rate is less than or equal to 15 percent.
Echo cancellation effect:
and (3) testing environment:
and the vehicle standing environment is the same as 2.2.2 in volume decibel environment.
The test method comprises the following steps:
randomly playing the linguistic data with the voice feedback broadcast, continuously playing for 6 hours, judging according to the result of the voice typewriter, and recording the recognized times of the voice feedback broadcast; randomly playing the multimedia content of the non-melody voice, keeping a voice awakening state, playing for 6h in a circulating way, and recording the frequency of the multimedia voice recorded by voice recognition; and respectively testing the normal decibel environment and the high decibel environment in the vehicle.
Evaluation indexes are as follows:
the times of the sound of the non-speaker which is identified by mistake is less than or equal to 0.5 times/h;
the number of times of misrecognizing the multimedia sound is less than or equal to 0.5 times/h.
Identifying a response time:
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps.
The test method comprises the following steps:
the response time of voice recognition is strictly the time from the end of recording to the acquisition of the recognition result, and because different voice systems have differences in feedback modes after semantic recognition and feedback to different instructions, the voice log of the vehicle-mounted sound entertainment host is intercepted, and the time interval from the end of recording to the acquisition of the recognition result is judged by the log time point, so that the method is relatively accurate. The method for intercepting the voice logs by each host is inconsistent, so that a log intercepting mode needs to be specially set for the tested host.
After the method for intercepting the voice log is selected. And randomly selecting a tested vehicle environment based on the corpus function supported by the voice system, executing all corpora, executing each corpus once, and calculating the average time of recognition response of each corpus through the voice log.
Evaluation indexes are as follows:
the average recognition response time is less than or equal to 3 s.
Voice resource occupation:
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps.
The test method comprises the following steps:
the background of the vehicle-mounted sound entertainment host only keeps running applications and services which are started along with the default of the host, and is connected with related test recording tools for voice interaction, the memory occupation value and the CPU occupation value are recorded every 1s, and after the interaction is finished, the memory occupation mean value and the peak value in the process are counted, and the CPU occupation mean value and the peak value are counted
Evaluation indexes are as follows:
the memory occupation average value is the sum/statistic of the statistic sample;
memory occupation peak value is equal to the maximum value of the statistical sample
Each set of vehicle-mounted voice system needs to set the evaluation index acceptance standard according to the self condition of the test product due to different hardware conditions, system conditions and function quantity.
And (3) system stability:
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps.
The test method comprises the following steps:
starting the equipment and eliminating the problems of the equipment, continuously testing according to a voice interface calling flow through a batch stability testing tool, generating efficiency data, synchronously monitoring a CPU (central processing unit) and a memory, keeping running for 48h, and checking whether abnormal phenomena such as identification rate reduction, memory leakage, voice collapse and the like occur.
Evaluation indexes are as follows:
the voice system runs for 48h continuously, no resource is leaked, and the resource occupation is reasonable.
Awakening success rate, wherein at least 100 test samples are required;
the false wake-up rate is that the played sound without wake-up words needs to be guaranteed to be high-fidelity, and the test duration needs to be at least 6 h;
the success rate of recognition is realized, and all corpora supported by the voice system need to be tested;
voiceprint discrimination rate, voiceprint registration number at least needs 5, unregistered number at least needs 1
The echo cancellation effect needs at least 6h for the test duration;
voice resource occupation, and recording parameters at least need to comprise a memory and a CPU;
the system stability and the equipment running time are at least 48 h.
The method needs the host embedded in the vehicle-mounted sound entertainment system to comprise an intelligent voice module, and the voice module supports voice awakening, voice recognition, corpus setting, sound zone positioning (unnecessary) and echo elimination. The tester tests each content by using the awakening vocabulary, the functional linguistic data, the voice log, the external recording and the parameter monitoring tool of the voice system respectively through the testing method of the steps. The test data can be used to evaluate the quality of the speech system.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A vehicle-mounted intelligent voice system testing and evaluating method is characterized by comprising the following steps:
testing the success rate of voice awakening;
testing the voice false wake-up rate;
testing the success rate of voice recognition;
testing the sound source positioning accuracy;
testing the voiceprint identification rate;
testing the echo cancellation capability;
responding to the recognition time test;
testing voice resource occupation;
and (5) testing the stability of the system.
2. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the voice wake-up success rate test comprises:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the test method comprises the following steps:
testing the awakening rate of the awakening words in different scenes, wherein the single scene needs to be tested at least 100 times;
wake-up success rate (number of correct wake-ups/total number of tests) 100%.
3. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the voice false wake-up rate testing comprises:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the test method comprises the following steps:
continuously playing a section of recording without awakening words by using a high-fidelity sound box in the vehicle, wherein the testing time is more than 6h, and recording the awakening times;
false wake-up rate is the number of false wake-ups/total duration.
4. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the voice recognition success rate test comprises:
testing the environment in the vehicle:
normal decibel: the decibel value of multimedia playing in the car is between 55 dB and 65dB, and the speaker volume is between 50 dB and 70 dB; the subjective feeling is that people at the driving positions of the main and the auxiliary vehicles in the vehicle can normally communicate;
high decibel, the decibel value of multimedia playing in the car is between 65dB and 75dB, and the speaker volume is between 55 dB and 70 dB; the subjective feeling is that people at the main driving position and the auxiliary driving position in the automobile are difficult to communicate, and the volume in the automobile is large;
environment outside the vehicle:
a quiet environment, corresponding to a scene, having a driving environment-a parking lot; running condition-standing or vehicle speed not higher than 40 km/h; window open-close-window closed; air conditioner wind speed-off or no more than 2 gear;
the corresponding scenes comprise driving environments, namely urban areas, suburban areas, express ways and tunnels; driving conditions-vehicle speed higher than 40 km/h; window switch-front window open or skylight open; the air speed of the air conditioner is above-2 grade;
the reliable network connection of the vehicle-mounted sound entertainment host is ensured, the network environment is ensured to be 3G or above, the running speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the reliable network connection of the vehicle-mounted sound entertainment host is ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps; WiFi can also be connected, but the uploading and downloading speeds are still not lower than 100kbps and 50 kbps;
the test method comprises the following steps:
based on a voice function corpus list supported by a vehicle-mounted voice system, all function corpora are executed under each test environment, each test is carried out for at least 3 times, if the recognition is correct once in the recognition process, the function is considered to be normal, and the times of normal functions are counted after the test is finished;
the recognition success rate is (number of functional normalizations/total number of tests) 100%.
5. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the sound source positioning accuracy testing comprises:
and (3) testing environment: the vehicle is in a static quiet environment;
the test method comprises the following steps: firstly, a main driver wakes up voice and then speaks a random instruction, and a secondary driver uses 5 effective instruction sounds to interfere; and (5) repeating the operation after the main and auxiliary drives are exchanged for 2 h.
6. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the voiceprint authentication rate test comprises:
and (3) testing environment: the vehicle is in a static quiet environment;
the test method comprises the following steps: randomly selecting 5 testers, respectively registering voiceprints and naming under an intelligent voice full mode; five persons who have registered voiceprints respectively carry out 20 independent awakening tests, record the awakening success times, and repeatedly operate in a full mode and a sound source positioning mode; and repeating the operation by a sixth test person who does not register the voiceprint, and recording the awakening success times.
7. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the echo cancellation capability test comprises:
and (3) testing environment: a vehicle standing environment;
the test method comprises the following steps: randomly playing the linguistic data with the voice feedback broadcast, continuously playing for 6 hours, judging according to the result of the voice typewriter, and recording the recognized times of the voice feedback broadcast; randomly playing the multimedia content of the non-melody voice, keeping a voice awakening state, playing for 6h in a circulating way, and recording the frequency of the multimedia voice recorded by voice recognition; and respectively testing the normal decibel environment and the high decibel environment in the vehicle.
8. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the response recognition time test;
and (3) testing environment: the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
intercepting a voice log of a vehicle-mounted sound entertainment host, judging a time interval from the end of recording to the acquisition of an identification result through a log time point, and specially setting a log intercepting mode aiming at a tested host;
after the method for intercepting the voice log is selected, the tested vehicle environment is randomly selected based on the corpus functions supported by the voice system, all the corpora are executed, each corpus is executed once, and the average time of recognition response of each corpus is calculated through the voice log.
9. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, characterized in that the voice resource occupation testing;
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
the background of the vehicle-mounted sound entertainment host only keeps running applications and services which are started along with the host by default, a relevant test recording tool is connected for voice interaction, the memory occupation value and the CPU occupation value are recorded every 1s, and after the interaction is finished, the memory occupation mean value and the peak value in the process and the CPU occupation mean value and the peak value are counted.
10. The vehicle-mounted intelligent voice system testing and evaluating method according to claim 1, wherein the system stability testing comprises:
and (3) testing environment:
the vehicle environment has no requirement, but the reliable network connection of the vehicle-mounted audio entertainment host computer is required to be ensured, the network environment is ensured to be 3G or above, the uplink speed is not lower than 100kbps, and the downlink speed is not lower than 50 kbps;
the test method comprises the following steps:
starting the equipment and eliminating the problems of the equipment, continuously testing according to a voice interface calling flow through a batch stability testing tool, generating efficiency data, synchronously monitoring a CPU (central processing unit) and a memory, keeping running for 48h, and checking whether abnormal phenomena such as identification rate reduction, memory leakage, voice collapse and the like occur.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010760388.0A CN111986652B (en) | 2020-07-31 | 2020-07-31 | Vehicle-mounted intelligent voice system testing and evaluating method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010760388.0A CN111986652B (en) | 2020-07-31 | 2020-07-31 | Vehicle-mounted intelligent voice system testing and evaluating method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111986652A true CN111986652A (en) | 2020-11-24 |
| CN111986652B CN111986652B (en) | 2022-12-13 |
Family
ID=73445701
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010760388.0A Active CN111986652B (en) | 2020-07-31 | 2020-07-31 | Vehicle-mounted intelligent voice system testing and evaluating method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111986652B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113391975A (en) * | 2021-06-10 | 2021-09-14 | Oppo广东移动通信有限公司 | Terminal device testing method and device, storage medium and electronic device |
| CN113643704A (en) * | 2021-08-18 | 2021-11-12 | 中国第一汽车股份有限公司 | Test method, upper computer, system and storage medium of vehicle-mounted machine voice system |
| CN113900960A (en) * | 2021-12-02 | 2022-01-07 | 中汽研软件测评(天津)有限公司 | Intelligent vehicle mechanical arm test platform system |
| EP4102499A1 (en) * | 2021-06-08 | 2022-12-14 | Apollo Intelligent Connectivity (Beijing) Technology Co., Ltd. | Wakeup testing method and apparatus, electronic device and readable storage medium |
| WO2023061126A1 (en) * | 2021-10-15 | 2023-04-20 | 青岛海尔电冰箱有限公司 | Test system for refrigerator voice module |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107643967A (en) * | 2017-10-27 | 2018-01-30 | 深圳市传测科技有限公司 | A kind of automatic speech wakes up test device, system and method |
| CN109473094A (en) * | 2018-11-12 | 2019-03-15 | 东风汽车有限公司 | Vehicle-mounted control screen voice recognition rate testing method, electronic equipment and system |
| CN110581915A (en) * | 2019-08-30 | 2019-12-17 | Oppo广东移动通信有限公司 | Stability testing method and device, storage medium and electronic equipment |
| CN110751031A (en) * | 2019-09-16 | 2020-02-04 | 合肥泾渭信息通讯有限公司 | Fatigue detection system suitable for driver |
| CN111142073A (en) * | 2019-12-08 | 2020-05-12 | 中国航空工业集团公司洛阳电光设备研究所 | Method for testing positioning accuracy of airborne 3D audio direction |
| CN111314780A (en) * | 2020-03-27 | 2020-06-19 | 苏州科达科技股份有限公司 | Method and device for testing echo cancellation function and storage medium |
| CN111341313A (en) * | 2020-03-04 | 2020-06-26 | 北京声智科技有限公司 | Vehicle-mounted multi-sound-zone sound source detection method, device and system |
-
2020
- 2020-07-31 CN CN202010760388.0A patent/CN111986652B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107643967A (en) * | 2017-10-27 | 2018-01-30 | 深圳市传测科技有限公司 | A kind of automatic speech wakes up test device, system and method |
| CN109473094A (en) * | 2018-11-12 | 2019-03-15 | 东风汽车有限公司 | Vehicle-mounted control screen voice recognition rate testing method, electronic equipment and system |
| CN110581915A (en) * | 2019-08-30 | 2019-12-17 | Oppo广东移动通信有限公司 | Stability testing method and device, storage medium and electronic equipment |
| CN110751031A (en) * | 2019-09-16 | 2020-02-04 | 合肥泾渭信息通讯有限公司 | Fatigue detection system suitable for driver |
| CN111142073A (en) * | 2019-12-08 | 2020-05-12 | 中国航空工业集团公司洛阳电光设备研究所 | Method for testing positioning accuracy of airborne 3D audio direction |
| CN111341313A (en) * | 2020-03-04 | 2020-06-26 | 北京声智科技有限公司 | Vehicle-mounted multi-sound-zone sound source detection method, device and system |
| CN111314780A (en) * | 2020-03-27 | 2020-06-19 | 苏州科达科技股份有限公司 | Method and device for testing echo cancellation function and storage medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4102499A1 (en) * | 2021-06-08 | 2022-12-14 | Apollo Intelligent Connectivity (Beijing) Technology Co., Ltd. | Wakeup testing method and apparatus, electronic device and readable storage medium |
| JP2022187977A (en) * | 2021-06-08 | 2022-12-20 | アポロ インテリジェント コネクティヴィティ （ベイジン） テクノロジー カンパニー リミテッド | Wake-up test method, device, electronic device and readable storage medium |
| CN113391975A (en) * | 2021-06-10 | 2021-09-14 | Oppo广东移动通信有限公司 | Terminal device testing method and device, storage medium and electronic device |
| CN113643704A (en) * | 2021-08-18 | 2021-11-12 | 中国第一汽车股份有限公司 | Test method, upper computer, system and storage medium of vehicle-mounted machine voice system |
| WO2023061126A1 (en) * | 2021-10-15 | 2023-04-20 | 青岛海尔电冰箱有限公司 | Test system for refrigerator voice module |
| CN113900960A (en) * | 2021-12-02 | 2022-01-07 | 中汽研软件测评(天津)有限公司 | Intelligent vehicle mechanical arm test platform system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111986652B (en) | 2022-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111986652B (en) | Vehicle-mounted intelligent voice system testing and evaluating method | |
| CN109147765B (en) | Audio quality comprehensive evaluation method and system | |
| US9263040B2 (en) | Method and system for using sound related vehicle information to enhance speech recognition | |
| US9530401B2 (en) | Apparatus and method for reporting speech recognition failures | |
| CN1306472C (en) | System and method for transmitting speech activity in a distributed voice recognition system | |
| US8719035B2 (en) | Method and apparatus for recognizing and reacting to user personality in accordance with speech recognition system | |
| US9418674B2 (en) | Method and system for using vehicle sound information to enhance audio prompting | |
| US7729911B2 (en) | Speech recognition method and system | |
| US20130185078A1 (en) | Method and system for using sound related vehicle information to enhance spoken dialogue | |
| CN110475170A (en) | Control method, device, mobile terminal and the storage medium of earphone broadcast state | |
| CN103634472A (en) | Method, system and mobile phone for judging mood and character of user according to call voice | |
| CN107600075A (en) | The control method and device of onboard system | |
| CN108062212A (en) | A kind of voice operating method and device based on scene | |
| EP4033483B1 (en) | Method and apparatus for testing vehicle-mounted voice device, electronic device and storage medium | |
| CN109273006A (en) | Sound control method, onboard system, vehicle and the storage medium of onboard system | |
| CN110008481B (en) | Translated voice generating method, device, computer equipment and storage medium | |
| CN107580155A (en) | Networking telephone quality determination method, device, computer equipment and storage medium | |
| CN101645716A (en) | Vehicle-borne communication system having voice recognition function and recognition method thereof | |
| CN113643704A (en) | Test method, upper computer, system and storage medium of vehicle-mounted machine voice system | |
| JP2003509705A (en) | Voice recognition method and voice recognition device | |
| CN106094673A (en) | Automobile wakes up word system and control method thereof up | |
| KR102407055B1 (en) | Apparatus and method for measuring dialogue quality index through natural language processing after speech recognition | |
| CN110197663A (en) | A kind of control method, device and electronic equipment | |
| US8244538B2 (en) | Measuring double talk performance | |
| CN115440192A (en) | Testing method, device, equipment and storage medium of voice recognition system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |