CN110705651A - Method, device, equipment and medium for testing video identification accuracy - Google Patents
Method, device, equipment and medium for testing video identification accuracy Download PDFInfo
- Publication number
- CN110705651A CN110705651A CN201910988787.XA CN201910988787A CN110705651A CN 110705651 A CN110705651 A CN 110705651A CN 201910988787 A CN201910988787 A CN 201910988787A CN 110705651 A CN110705651 A CN 110705651A
- Authority
- CN
- China
- Prior art keywords
- test
- video
- picture
- recognition
- accuracy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 259
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000002372 labelling Methods 0.000 claims abstract description 27
- 238000006243 chemical reaction Methods 0.000 claims abstract description 9
- 238000004590 computer program Methods 0.000 claims description 12
- 230000003993 interaction Effects 0.000 claims description 9
- 238000013515 script Methods 0.000 abstract description 12
- 238000010276 construction Methods 0.000 abstract description 7
- 230000003068 static effect Effects 0.000 abstract description 5
- 230000008901 benefit Effects 0.000 abstract description 4
- 230000002349 favourable effect Effects 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 41
- 230000002452 interceptive effect Effects 0.000 description 11
- 238000011156 evaluation Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000004088 simulation Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000013100 final test Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013095 identification testing Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The application provides a method, a device, equipment and a medium for testing video identification accuracy, wherein the method for testing the video identification accuracy comprises the following steps: acquiring a test video and converting the test video into a test picture; reading the test picture through a pre-constructed analog video identification module, carrying out image identification on the test picture, and outputting an image identification result; and comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to obtain the recognition accuracy. The method and the device have the advantages that the accuracy test of dynamic video streams which are difficult to quantize is converted into the accuracy test of static pictures which are easy to quantize, automatic scripts such as video-to-picture conversion, result comparison and statistics are used, the problems that a data set is difficult to construct and test are difficult can be solved, the method of using a language script is favorable for carrying out automatic construction on steps in the method, and efficiency is greatly improved.
Description
Technical Field
The present application relates to the field of testing technologies, and in particular, to the field of testing technologies for image recognition, and more particularly, to a method, an apparatus, a device, and a medium for testing accuracy of video recognition.
Background
In the development process of the subject three-examination intelligent evaluation system, besides the traditional function test, the accuracy test needs to be carried out on the identification of the algorithm.
A data set needs to be constructed in the traditional algorithm identification test, the accuracy of the algorithm can be reflected through the output obtained through the input, an accurate reference is provided for development, and whether the optimization is effective or not is verified.
However, the algorithm identification test of the subject three-test intelligent evaluation system has the following difficulties: firstly, a video is dynamically read from a vehicle-mounted camera in real time, and a stable data set cannot be constructed; secondly, a large amount of data sets (with magnitude of ten thousand levels) are needed to reflect the effect of algorithm identification, and a large amount of manpower and material resources are consumed for the construction of the video data sets; and thirdly, a plurality of groups of actions may exist in a section of video, namely, a plurality of output results may exist in one input, whether the results are correct or not needs to be judged and counted respectively, and the test can be performed only in a manual mode, so that a lot of time is consumed.
Content of application
In view of the above-mentioned shortcomings of the prior art, an object of the present application is to provide a method, an apparatus, a device and a medium for testing video recognition accuracy, which are used to solve the technical problems of difficult data set construction and difficult testing in the existing video recognition accuracy testing.
To achieve the above and other related objects, a first aspect of the present application provides a method for testing video recognition accuracy, comprising: acquiring a test video and converting the test video into a test picture; reading the test picture through a pre-constructed analog video identification module, carrying out image identification on the test picture, and outputting an image identification result; and comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to obtain the recognition accuracy.
In some embodiments of the first aspect of the present application, the obtaining a test video and converting the test video into a test picture includes: acquiring test videos of different scene classifications; and calling a computer vision library to read the test video and perform picture interception on the test video to acquire test pictures of different scene classifications.
In some embodiments of the first aspect of the present application, the test pictures are stored separately according to scene classification; the pre-labeling of the test picture comprises a scene classification name, a test picture name and a labeling result; the output image identification result comprises a scene classification name, a test picture name and a labeling result; the identification accuracy rate comprises video identification accuracy rates under different scene classifications.
In some embodiments of the first aspect of the present application, the output image recognition result is output in a table form.
In some embodiments of the first aspect of the present application, the pre-configured analog video recognition module and the actual video recognition module employ the same hardware interaction structure and software recognition algorithm program.
In some embodiments of the first aspect of the present application, the method for testing video identification accuracy further includes generating identification accuracy comparison reports for different scene classifications, identification accuracy comparison reports for different directions and/or angles in the same scene, and identification accuracy comparison reports for different versions of analog video identification modules according to the obtained identification accuracy.
To achieve the above and other related objects, a second aspect of the present application provides an apparatus for testing video recognition accuracy, comprising: the video acquisition module is used for acquiring a test video; the conversion module is used for converting the test video into a test picture; the testing module is used for reading the testing picture through a pre-constructed analog video identification module, carrying out image identification on the testing picture and outputting an image identification result; and the recognition rate acquisition module is used for comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to acquire the recognition accuracy rate.
In some embodiments of the first aspect of the present application, the means for testing video identification accuracy further comprises: and the test report module is used for generating recognition accuracy rate comparison reports of different scene classifications, recognition accuracy rate comparison reports of different directions and/or angles in the same scene, and recognition accuracy rate comparison reports of analog video recognition modules of different versions according to the acquired recognition accuracy rate.
To achieve the above and other related objects, a third aspect of the present application provides an electronic terminal comprising: a processor and a memory; the memory is used for storing a computer program; the processor is configured to execute the computer program stored in the memory to cause the electronic terminal to execute the method for testing video identification accuracy as described above.
To achieve the above and other related objects, a fourth aspect of the present application is a computer-readable storage medium having a computer program stored thereon, characterized in that: which when executed by a processor implements the method for testing video recognition accuracy as described above.
As described above, the method, apparatus, device and medium for testing video identification accuracy of the present application have the following beneficial effects:
the method and the device have the advantages that the accuracy test of dynamic video streams which are difficult to quantize is converted into the accuracy test of static pictures which are easy to quantize, automatic scripts such as video-to-picture conversion, result comparison and statistics are used, the problems that a data set is difficult to construct and test are difficult can be solved, the method of using a language script is favorable for carrying out automatic construction on steps in the method, and efficiency is greatly improved.
Drawings
Fig. 1 is a schematic overall flowchart of a method for testing accuracy of video identification according to an embodiment of the present application.
Fig. 2 is a schematic flowchart illustrating a process of obtaining test pictures of different scene classifications according to an embodiment of the present application.
Fig. 3 is a flowchart illustrating an implementation process of a method for testing accuracy of video recognition according to an embodiment of the present application.
Fig. 4 is a schematic diagram illustrating the comparison between the operation process of the pre-configured analog video recognition module and the operation process of the actual video recognition module according to an embodiment of the present application.
Fig. 5 is a schematic block diagram of an apparatus for testing accuracy of video recognition in an embodiment of the present application.
Fig. 6 is a schematic block diagram of an apparatus for testing accuracy of video recognition according to an embodiment of the present application.
Fig. 7 is a schematic structural diagram of an electronic terminal according to an embodiment of the present application.
Description of the element reference numerals
100 device for testing video identification accuracy
110 video acquisition module 110
120 conversion module
130 test module
140 recognition rate acquisition module
150 test report module
1101 processor
1102 memory
S100 to S300
S110 to S120
Detailed Description
The following description of the embodiments of the present application is provided by way of specific examples, and other advantages and effects of the present application will be readily apparent to those skilled in the art from the disclosure herein. The present application is capable of other and different embodiments and its several details are capable of modifications and/or changes in various respects, all without departing from the spirit of the present application. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises," "comprising," and/or "comprising," when used in this specification, specify the presence of stated features, operations, elements, components, items, species, and/or groups, but do not preclude the presence, or addition of one or more other features, operations, elements, components, items, species, and/or groups thereof. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. Thus, "A, B or C" or "A, B and/or C" means "any of the following: a; b; c; a and B; a and C; b and C; A. b and C ". An exception to this definition will occur only when a combination of elements, functions or operations are inherently mutually exclusive in some way.
The invention provides a method, a device, equipment and a medium for testing video identification accuracy, which are used for improving the accuracy and robustness of table line detection, improving the applicability of table reconstruction and key information extraction and solving the technical problems of difficult data set construction and difficult testing in the conventional video identification accuracy test.
The principles and embodiments of a method, an apparatus, a device and a medium for testing video recognition accuracy according to the present embodiment will be described in detail below, so that those skilled in the art can understand the method, the apparatus, the device and the medium for testing video recognition accuracy without creative labor.
Fig. 1 shows a flow chart of a method for testing accuracy of video recognition according to an embodiment of the present invention.
It should be noted that the method for testing the video identification accuracy can be applied to various types of hardware devices. The hardware device is, for example, a controller, specifically, an arm (advanced RISC machines) controller, an fpga (field Programmable Gate array) controller, a soc (system on chip) controller, a dsp (digital signal processing) controller, or an mcu (micro controller unit) controller, etc. The hardware devices may also be, for example, a computer that includes components such as memory, a memory controller, one or more processing units (CPUs), a peripheral interface, RF circuitry, audio circuitry, speakers, a microphone, an input/output (I/O) subsystem, a display screen, other output or control devices, and external ports; the computer includes, but is not limited to, Personal computers such as desktop computers, notebook computers, tablet computers, smart phones, smart televisions, Personal Digital assistants) Personal Digital Assistants (PDAs). In other embodiments, the hardware device may also be a server, where the server may be arranged on one or more entity servers according to various factors such as functions and loads, or may be formed by a distributed or centralized server cluster, and this embodiment is not limited in this embodiment.
As shown in fig. 1, in the present embodiment, the method for testing the video identification accuracy includes steps S100 to S300.
Step S100, obtaining a test video and converting the test video into a test picture;
step S200, reading the test picture through a pre-constructed analog video identification module, carrying out image identification on the test picture, and outputting an image identification result;
and step S300, comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to obtain the recognition accuracy.
The following describes steps S100 to S300 of the method for testing video recognition accuracy in this embodiment in detail.
Step S100, obtaining a test video and converting the test video into a test picture.
In this embodiment, a test video is directly captured by an image capture device, and the test video is obtained from the image capture device, or obtained from a server in which a desired video is stored.
Specifically, as shown in fig. 2, the obtaining a test video and converting the test video into a test picture includes:
step S110, acquiring test videos of different scene classifications;
and step S120, calling a computer vision library to read the test video and perform picture interception on the test video to acquire test pictures of different scene classifications.
In this embodiment, a scene test concept is introduced, and the test video is selected to cover various different scenes to form the test video. And testing the accuracy of algorithm identification in different scenes to promote development and carry out targeted optimization.
In order to test the influence of different scenes on the algorithm identification accuracy, the recorded test videos are classified, and the test videos of different classes are respectively stored in different folders. For example, the influence of the test light on the identification can be recorded under strong light and low light respectively, and the test video can be stored in two folders of strong light and low light respectively. Other factors including vehicle type, driver dressing, etc. were tested for impact on identification as above. The test videos under different scene categories (i.e., different folders, the same below) are obtained through step S110.
In this embodiment, the called computer vision library is an OpenCv computer vision library, OpenCv provides interfaces of languages such as Python, Ruby, MATLAB, and the like, and can implement many general algorithms in the aspects of image processing and computer vision. A Python script is written to call the OpenCv library, and the function of converting the video into the picture in the embodiment is realized through a video-to-picture algorithm built in the OpenCv library.
In this embodiment, in order to make each test picture have obvious difference (equivalent to making different test cases), two selected test pictures are separated by a plurality of frames, for example, 1 picture is selected as a final test picture every 100 pictures. As an alternative to this step, other methods or tools may be used to convert the video into pictures. Test pictures under different scene classifications are obtained through step S120.
In the embodiment, the accuracy test of the dynamic video stream which is difficult to quantize is converted into the accuracy test of the static picture which is easy to quantize.
And S200, reading the test picture through a pre-constructed analog video identification module, carrying out image identification on the test picture, and outputting an image identification result.
In this embodiment, the test pictures are stored according to the scene classification; the output image recognition result comprises a scene classification name, a test picture name and a labeling result.
In this embodiment, the pre-configured analog video recognition module and the actual video recognition module adopt the same hardware interaction structure and software recognition algorithm program.
That is, the accuracy of the actual video recognition algorithm can be tested only by adopting the actual video recognition algorithm as the algorithm for simulating video recognition. If the actual video is identified to have a hardware interaction platform, the simulation should also have a corresponding hardware configuration. And will not be described in detail herein.
And step S300, comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to obtain the recognition accuracy.
The identification accuracy rate comprises video identification accuracy rates under different scene classifications.
In some embodiments, the output image recognition result is output in a table form.
And uploading the test pictures in different folders, namely under different scene classifications, to a server specified path, and after running a test program, reading the test pictures in the specified path by the program, and identifying and outputting results. The output content includes: scene classification (folder name) + test picture file name + algorithm recognition result. The recognition result can be automatically input into Excel, the effect is shown in table 1 and table 2, and the test recognition result of the test picture under different scene classifications is obtained.
Table 1 test identification results of test pictures under classification of strong light scenes
| Name of folder | Test picture | Annotating results |
| Hard light | 1.jpg | Front left |
| Hard light | 2.jpg | Front left |
| Hard light | 3.jpg | Straight ahead |
| Hard light | 4.jpg | Straight ahead |
| Hard light | 5.jpg | Right front side |
| Hard light | 6.jpg | Right front side |
| Hard light | 7.jpg | Endoscope |
| Hard light | 8.jpg | Right front side |
Table 2 test identification results of test pictures under classification of low light scene
| Name of folder | Test picture | Annotating results |
| Low light type | 1.jpg | Endoscope |
| Low light type | 2.jpg | Endoscope |
| Low light type | 3.jpg | Right front side |
| Low light type | 4.jpg | Right front side |
| Low light type | 5.jpg | Right front side |
| Hard light | 6.jpg | Right side |
| Hard light | 7.jpg | Right side |
| Hard light | 8.jpg | Head-lowering watch block |
In this embodiment, the pre-labeling of the test picture includes a scene classification name, a test picture name, and a labeling result. And outputting the pre-label of the test picture in a table form. For example, the correct result of each test picture is labeled to indicate the face orientation of the driver in the picture, and as an embodiment, the scene classification (folder) of the picture, the picture name and the labeling information are input into the exel document, or the correct result is recorded in other ways, so as to obtain the pre-labeled result, i.e. the expected result, of the test picture under each different scene classification, and the effect is shown in table 3 and table 4.
Table 3 test identification results of test pictures under classification of strong light scenes
| Name of folder | Test picture | Annotating results |
| Hard light | 1.jpg | Front left |
| Hard light | 2.jpg | Front left |
| Hard light | 3.jpg | Straight ahead |
| Hard light | 4.jpg | Front left |
| Hard light | 5.jpg | Right front side |
| Hard light | 6.jpg | Endoscope |
| Hard light | 7.jpg | Endoscope |
| Hard light | 8.jpg | Right front side |
TABLE 4 test identification results of test pictures under classification of low-light scenes
| Name of folder | Test picture | Annotating results |
| Low light type | 1.jpg | Endoscope |
| Low light type | 2.jpg | Endoscope |
| Low light type | 3.jpg | Right front side |
| Low light type | 4.jpg | Right front side |
| Low light type | 5.jpg | Endoscope |
| Hard light | 6.jpg | Right side |
| Hard light | 7.jpg | Right side |
| Hard light | 8.jpg | Head-lowering watch block |
For example, by using the folder name and the test picture name as indexes, the obtained pre-labeling result is compared with the obtained actual image identification result, and the identification accuracy is obtained. Where the statistics can be made using an automated script, as shown in table 5.
TABLE 5 statistical results of recognition accuracy
| Direction of rotation | Correct number of samples | Number of erroneous samples | Algorithm recognition accuracy |
| On the left side | OK_left | NG_left | OK_left/(OK_left+NG_left) |
| Front left | OK_leftfront | NG_leftfront | OK_leftfront/(OK_leftfront+NG_leftfront) |
| Straight ahead | OK_front | NG_front | OK_front/(OK_front+NG_front) |
| Right front side | OK_rightfront | NG_rightfront | OK_rightfront/(OK_rightfront+NG_rightfront) |
| Right side | OK_rightfront | NG_rightfront | OK_rightfront/(OK_rightfront+NG_rightfront) |
| Endoscope | OK_endscopic | NG_endscopic | OK_endscopic/(OK_endscopic+NG_endscopic) |
| Instrument panel | OK_panel | NG_panel | OK_panel/(OK_panel+NG_panel) |
| Head-lowering watch holder | OK_gear | NG_gear | OK_gear/(OK_gear+NG_gear) |
If the expected result of the picture A is right ahead and the actual result is right ahead, the test picture is correctly identified, and the OK _ front variable is added with 1; if the actual result of the test picture is in other directions, the test picture is identified as wrong, and 1 is added to the NG _ front variable. And the other test pictures are similar to each other, so that the correct sample total amount and the error sample total amount in each direction can be finally obtained, and the algorithm identification accuracy in each direction can be further obtained.
In the specific technical scheme of the method, automatic scripts of video-to-picture conversion, result comparison, statistics and the like are used, and the testing efficiency is improved.
In this embodiment, the method for testing the video recognition accuracy further includes generating recognition accuracy comparison reports of different scene classifications, recognition accuracy comparison reports of different directions and/or angles in the same scene, and recognition accuracy comparison reports of analog video recognition modules of different versions according to the obtained recognition accuracy.
That is to say, the method for testing video identification accuracy of the present embodiment may perform generation of test reports in multiple dimensions: the algorithm identification accuracy comparison report of each direction, the algorithm identification accuracy comparison report of each scene, the algorithm identification accuracy comparison report of each version and the like.
As shown in fig. 3, the implementation process of the method for testing video identification accuracy according to the present embodiment is described in detail by taking the video identification accuracy of the test subject three-test intelligent evaluation system as an example.
The driver randomly makes 8-direction actions of left, left front, right front, endoscope, instrument panel and head lowering view file in a real scene by using a matched camera of the subject three-examination intelligent judging system, and images shot by the camera are recorded and placed in a storage device. In order to test the influence of different scenes on the algorithm identification accuracy, the recorded videos are classified, and different classes are stored in different folders respectively. For example, the influence of the light on the identification can be tested, and the video can be recorded under strong light and weak light respectively and stored in two folders of strong light and weak light respectively. Other factors including vehicle type, driver dressing, etc. were tested for impact on identification as above. And obtaining videos under different scene classifications (namely different folders and the same below).
Writing Python script: and calling OpenCv to read the test video and perform picture interception, converting the obtained test video into pictures, and selecting 1 frame as a final test picture every 100 frames in order to make each test picture have obvious difference so as to obtain the test pictures under different scene classifications.
The correct result of each test picture is labeled, and the scene classification (folder to which the test picture belongs), the picture name and the labeling information of the test picture are input into the exel document, so that the labeling result, namely the expected result, of the test picture under different scene classifications is obtained, and the effect is shown in table 3 and table 4.
And simulating the workflow of the subject three-examination intelligent evaluation system, and deploying a test program (a simulation video identification module). Fig. 4 shows a main body composition and workflow pair of the simulation subject three-examination intelligent evaluation system and the test program.
The subject three-examination intelligent evaluation system main body can be divided into a camera, an interaction end and an algorithm library, wherein the camera has the main function of taking images and providing system input, the interaction end is a part for connecting the camera and the algorithm library, the main function is the interaction of the camera and the algorithm library, the algorithm library is the core of the system, and the main function is the identification result.
The workflow of the subject three-examination intelligent evaluation system can be divided into four parts:
1. the interactive end reads pictures from the camera according to frames;
2. the interactive terminal transmits the picture stream to an algorithm library;
3. the algorithm library identifies the picture and returns the result to the interactive terminal;
4. and the interactive end outputs the result received from the algorithm library.
The main body composition of the test program (simulation video identification module) can be divided into two parts of an interactive end and an algorithm library, and the work flow is also divided into four parts:
1. the interactive terminal reads pictures from the local server;
2. the interactive terminal transmits the picture stream to an algorithm library;
3. the algorithm library identifies the picture and returns the result to the interactive terminal;
4. and the interactive end outputs the result received from the algorithm library.
From the above, the main difference between the test program (analog video identification module) and the subject three-examination intelligent evaluation system is the step 1 in the workflow, the system reads pictures from the camera for the interactive terminal, the test program reads pictures from the local server for the interactive terminal, and although the input mode of the test program is different from that of the system, the identification of the pictures by the algorithm library is not affected, so that the test program can obtain correct output through the analog system, and the reliability of the test result is ensured. With each iteration of the algorithm (i.e. updating of the algorithm library), the test program needs to be updated and deployed on the test server.
And uploading the obtained test pictures of different folders, namely under different scene classifications, to a server specified path, and after running a test program, reading the test pictures by the program in the specified path, and identifying and outputting results. The output content includes: scene classification (folder name) + test picture file name + algorithm recognition result. The results can be automatically input into Excel, the effects are shown in tables 1 and 2, and actual results of the test pictures under different scene classifications are obtained.
By using the folder name and the test picture name as indexes, the obtained pre-labeling result is compared with the obtained actual image identification result to obtain the identification accuracy, as shown in table 5. It may then be chosen to perform the generation of test reports for multiple dimensions.
The method for testing the video identification accuracy rate of the embodiment provides a solution for the difficulty in constructing a stable data set when the accuracy test of algorithm identification is carried out on the subject three-test intelligent evaluation system: i.e. the testing of dynamic video streams is converted into the testing of still pictures. The method constructs a data set, can save a large amount of manpower and material resources, and can construct ten thousand-level effective pictures as a test set by using 10min videos; the method uses an automatic script mode to compare and count the test results, the whole process is high in ductility, complete automatic construction can be achieved from deployment, batch running, counting and report issuing, and the test efficiency is improved.
As shown in fig. 5, the present embodiment further provides an apparatus 100 for testing video recognition accuracy, where the apparatus 100 for testing video recognition accuracy includes: a video acquisition module 110, a conversion module 120, a test module 130, and a recognition rate acquisition module 140.
In this embodiment, the video obtaining module 110 is used for obtaining a test video.
In this embodiment, the conversion module 120 is configured to convert the test video into a test picture.
Specifically, the conversion module 120 obtains test videos of different scene classifications, and calls a computer vision library to read the test videos and perform image capturing on the test videos, so as to obtain test images of different scene classifications.
In this embodiment, a scene test concept is introduced, and the test video is selected to cover various different scenes to form the test video. And testing the accuracy of algorithm identification in different scenes to promote development and carry out targeted optimization.
In order to test the influence of different scenes on the algorithm identification accuracy, the recorded test videos are classified, and the test videos of different classes are respectively stored in different folders. For example, the influence of the test light on the identification can be recorded under strong light and low light respectively, and the test video can be stored in two folders of strong light and low light respectively. Other factors including vehicle type, driver dressing, etc. were tested for impact on identification as above.
In this embodiment, in order to make each test picture have obvious difference (equivalent to making different test cases), two selected test pictures are separated by a plurality of frames, for example, 1 picture is selected as a final test picture every 100 pictures.
In the embodiment, the accuracy test of the dynamic video stream which is difficult to quantize is converted into the accuracy test of the static picture which is easy to quantize.
In this embodiment, the test module 130 is configured to read the test picture through a pre-configured analog video recognition module, perform image recognition on the test picture, and output an image recognition result.
In this embodiment, the test pictures are stored according to the scene classification; the output image recognition result comprises a scene classification name, a test picture name and a labeling result.
In this embodiment, the pre-configured analog video recognition module and the actual video recognition module adopt the same hardware interaction structure and software recognition algorithm program.
That is, the accuracy of the actual video recognition algorithm can be tested only by adopting the actual video recognition algorithm as the algorithm for simulating video recognition. If the actual video is identified to have a hardware interaction platform, the simulation should also have a corresponding hardware configuration. And will not be described in detail herein.
In this embodiment, the identification rate obtaining module 140 is configured to compare and analyze the image identification result with a labeling result of a pre-labeled test picture, so as to obtain an identification accuracy rate.
The identification accuracy rate comprises video identification accuracy rates under different scene classifications.
In some embodiments, the output image recognition result is output in a table form.
And uploading the test pictures in different folders, namely under different scene classifications, to a server specified path, and after running a test program, reading the test pictures in the specified path by the program, and identifying and outputting results. The output content includes: scene classification (folder name) + test picture file name + algorithm recognition result. The recognition result can be automatically input into Excel, the effect is shown in fig. 3, and the test recognition results of the test pictures under different scene classifications are obtained.
In this embodiment, the pre-labeling of the test picture includes a scene classification name, a test picture name, and a labeling result. And outputting the pre-label of the test picture in a table form. For example, as an embodiment, the correct result of each test picture is labeled to mark the face orientation of the driver in the picture, and the scene classification (belonging folder), the picture name and the labeling information of the picture are input into the exel document, or the correct result is recorded in other ways, so as to obtain the pre-labeled result, i.e. the expected result, of the test picture under different scene classifications, and the effect is shown in fig. 4.
For example, by using the folder name and the test picture name as indexes, the obtained pre-labeling result is compared with the obtained actual image identification result, and the identification accuracy is obtained. Where the statistics can be made using an automated script, as shown in fig. 5.
As shown in fig. 6, the apparatus 100 for testing video identification accuracy further includes: the test report module 150 is configured to generate recognition accuracy rate comparison reports for different scene classifications, recognition accuracy rate comparison reports for different directions and/or angles in the same scene, and recognition accuracy rate comparison reports for analog video recognition modules of different versions according to the obtained recognition accuracy rates.
As shown in fig. 6, a schematic structural diagram of an electronic terminal in an embodiment of the present application is shown, where the electronic terminal includes, but is not limited to, a Personal computer such as a desktop computer, a notebook computer, a tablet computer, a smart phone, a smart television, and a Personal Digital Assistant (PDA). In other embodiments, the electronic terminal may also be a server, and the server may be arranged on one or more entity servers according to various factors such as functions, loads, and the like, or may be formed by a distributed or centralized server cluster, which is not limited in this embodiment.
As shown in fig. 7, the electronic terminal comprises a processor 1101 and a memory 1102; the memory 1102 is connected to the processor 1101 through a system bus and is used for storing computer programs, and the processor 1101 is used for running the computer programs, so that the electronic terminal executes the method for testing the video identification accuracy. The method for testing the video identification accuracy has been described in detail above, and is not described herein again.
It should be noted that the above mentioned system bus may be Peripheral Component Interconnect (PCI) bus or Extended industry standard Architecture) Extended Industrial Standard Architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices) such as clients, read-write libraries, and read-only libraries. The Memory may comprise Random Access Memory) Random Access Memory, RAM for short), and may also comprise non-volatile Memory), such as at least one disk Memory.
The Processor 1101 may be a general-purpose Processor, including a Central Processing Unit (CPU)), a Network Processor (NP), and so on; but also Digital Signal processor) Digital Signal Processing, DSP), Application specific integrated Circuit), ASIC, Field Programmable Gate Array), Field-Programmable Gate Array, or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components.
Furthermore, the present embodiment also provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the method for testing accuracy of video recognition. The method for testing the video identification accuracy has been described in detail above, and is not described herein again.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
To sum up, this application turns into the accuracy test of static, the picture of easily quantizing with the accuracy test of dynamic, difficult quantization's video stream, has used automatic scripts such as video commentaries on classics picture, result comparison and statistics, not only can solve the difficult, test difficult problem of data set structure, does benefit to the mode that uses the language script moreover and carries out automated construction to step wherein, has promoted efficiency by a wide margin. Therefore, the application effectively overcomes various defects in the prior art and has high industrial utilization value.
The above embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.
Claims (10)
1. A method for testing video recognition accuracy, comprising:
acquiring a test video and converting the test video into a test picture;
reading the test picture through a pre-constructed analog video identification module, carrying out image identification on the test picture, and outputting an image identification result;
and comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to obtain the recognition accuracy.
2. The method for testing video recognition accuracy of claim 1, wherein the obtaining a test video and converting the test video into a test picture comprises:
acquiring test videos of different scene classifications;
and calling a computer vision library to read the test video and perform picture interception on the test video to acquire test pictures of different scene classifications.
3. The method for testing video recognition accuracy according to claim 2, wherein the test pictures are stored separately according to scene classification; the pre-labeling of the test picture comprises a scene classification name, a test picture name and a labeling result; the output image identification result comprises a scene classification name, a test picture name and a labeling result; the identification accuracy rate comprises video identification accuracy rates under different scene classifications.
4. The method for testing video recognition accuracy of claim 1, wherein the outputted image recognition result is outputted in a table form.
5. The method for testing video recognition accuracy of claim 1, wherein the pre-configured analog video recognition module and the actual video recognition module employ the same hardware interaction structure and software recognition algorithm program.
6. The method for testing video recognition accuracy according to claim 1, further comprising generating recognition accuracy comparison reports for different scene classifications, recognition accuracy comparison reports for different directions and/or angles in the same scene, and recognition accuracy comparison reports for different versions of analog video recognition modules according to the obtained recognition accuracy.
7. An apparatus for testing video recognition accuracy, comprising:
the video acquisition module is used for acquiring a test video;
the conversion module is used for converting the test video into a test picture;
the testing module is used for reading the testing picture through a pre-constructed analog video identification module, carrying out image identification on the testing picture and outputting an image identification result;
and the recognition rate acquisition module is used for comparing and analyzing the image recognition result and a labeling result of a pre-labeled test picture to acquire the recognition accuracy rate.
8. The apparatus for testing video identification accuracy of claim 7, wherein said apparatus for testing video identification accuracy further comprises: and the test report module is used for generating recognition accuracy rate comparison reports of different scene classifications, recognition accuracy rate comparison reports of different directions and/or angles in the same scene, and recognition accuracy rate comparison reports of analog video recognition modules of different versions according to the acquired recognition accuracy rate.
9. An electronic terminal, comprising: a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to execute the memory-stored computer program to cause the electronic terminal to perform the method for testing video identification accuracy according to any one of claims 1 to 6.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements the method for testing video identification accuracy of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988787.XA CN110705651A (en) | 2019-10-17 | 2019-10-17 | Method, device, equipment and medium for testing video identification accuracy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988787.XA CN110705651A (en) | 2019-10-17 | 2019-10-17 | Method, device, equipment and medium for testing video identification accuracy |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110705651A true CN110705651A (en) | 2020-01-17 |
Family
ID=69200450
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910988787.XA Pending CN110705651A (en) | 2019-10-17 | 2019-10-17 | Method, device, equipment and medium for testing video identification accuracy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110705651A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931812A (en) * | 2020-07-01 | 2020-11-13 | 广州视源电子科技股份有限公司 | Visual algorithm testing method and device, storage medium and electronic equipment |
| CN112052820A (en) * | 2020-09-15 | 2020-12-08 | 展讯通信（上海）有限公司 | Test method and device for identifying AI scene identification rate, storage medium and terminal |
| CN112395208A (en) * | 2021-01-19 | 2021-02-23 | 珠海亿智电子科技有限公司 | Automatic test method, device, equipment and storage medium for AI recognition device |
| CN112559369A (en) * | 2020-12-23 | 2021-03-26 | 上海眼控科技股份有限公司 | Automatic testing method, automatic testing equipment and storage medium |
| CN113138932A (en) * | 2021-05-13 | 2021-07-20 | 北京字节跳动网络技术有限公司 | Method, device and equipment for verifying gesture recognition result of algorithm library |
| CN113495833A (en) * | 2020-04-03 | 2021-10-12 | 杭州海康威视系统技术有限公司 | Software testing method, device and system based on video event and readable storage medium |
| CN113642443A (en) * | 2021-08-06 | 2021-11-12 | 深圳市宏电技术股份有限公司 | Model testing method and device, electronic equipment and storage medium |
| CN113660482A (en) * | 2021-07-28 | 2021-11-16 | 上海立可芯半导体科技有限公司 | Automatic testing method and device for AI camera equipment or module |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018112783A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳前海达闼云端智能科技有限公司 | Image recognition method and device |
| CN110162462A (en) * | 2019-04-16 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Test method, system and the computer equipment of face identification system based on scene |
-
2019
- 2019-10-17 CN CN201910988787.XA patent/CN110705651A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018112783A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳前海达闼云端智能科技有限公司 | Image recognition method and device |
| CN110162462A (en) * | 2019-04-16 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Test method, system and the computer equipment of face identification system based on scene |
Non-Patent Citations (1)
| Title |
|---|
| 吴立金等: "一种非侵入的GUI自动化测试系统设计", 《计算机测量与控制》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495833A (en) * | 2020-04-03 | 2021-10-12 | 杭州海康威视系统技术有限公司 | Software testing method, device and system based on video event and readable storage medium |
| CN113495833B (en) * | 2020-04-03 | 2024-02-23 | 杭州海康威视系统技术有限公司 | Software testing method, device and system based on video event and storage medium |
| CN111931812A (en) * | 2020-07-01 | 2020-11-13 | 广州视源电子科技股份有限公司 | Visual algorithm testing method and device, storage medium and electronic equipment |
| CN112052820A (en) * | 2020-09-15 | 2020-12-08 | 展讯通信（上海）有限公司 | Test method and device for identifying AI scene identification rate, storage medium and terminal |
| CN112559369A (en) * | 2020-12-23 | 2021-03-26 | 上海眼控科技股份有限公司 | Automatic testing method, automatic testing equipment and storage medium |
| CN112395208A (en) * | 2021-01-19 | 2021-02-23 | 珠海亿智电子科技有限公司 | Automatic test method, device, equipment and storage medium for AI recognition device |
| CN113138932A (en) * | 2021-05-13 | 2021-07-20 | 北京字节跳动网络技术有限公司 | Method, device and equipment for verifying gesture recognition result of algorithm library |
| CN113660482A (en) * | 2021-07-28 | 2021-11-16 | 上海立可芯半导体科技有限公司 | Automatic testing method and device for AI camera equipment or module |
| CN113642443A (en) * | 2021-08-06 | 2021-11-12 | 深圳市宏电技术股份有限公司 | Model testing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110705651A (en) | Method, device, equipment and medium for testing video identification accuracy | |
| JP6868119B2 (en) | Holographic anti-counterfeit code inspection method and equipment | |
| CN112561080B (en) | Sample screening method, sample screening device and terminal equipment | |
| CN111144493B (en) | Method for automatically identifying algorithm index test, storage medium and electronic terminal | |
| CN110826646A (en) | Robot vision testing method and device, storage medium and terminal equipment | |
| CN113255516A (en) | Living body detection method and device and electronic equipment | |
| CN114494863A (en) | Animal cub counting method and device based on Blend Mask algorithm | |
| CN109840212B (en) | Function test method, device and equipment of application program and readable storage medium | |
| CN113138916B (en) | Automatic testing method and system for picture structuring algorithm based on labeling sample | |
| CN107071553B (en) | Method, device and computer readable storage medium for modifying video and voice | |
| CN113158773A (en) | Training method and training device for living body detection model | |
| CN112287923A (en) | Card information identification method, device, equipment and storage medium | |
| CN109766089B (en) | Code generation method and device based on dynamic diagram, electronic equipment and storage medium | |
| CN111815748A (en) | Animation processing method and device, storage medium and electronic equipment | |
| CN111859933A (en) | Training method, recognition method, device and equipment of Malay recognition model | |
| CN115022201B (en) | Data processing function test method, device, equipment and storage medium | |
| CN110827261B (en) | Image quality detection method and device, storage medium and electronic equipment | |
| US20220058530A1 (en) | Method and device for optimizing deep learning model conversion, and storage medium | |
| CN114821272A (en) | Image recognition method, image recognition system, image recognition medium, electronic device, and target detection model | |
| CN115004245A (en) | Target detection method, target detection device, electronic equipment and computer storage medium | |
| CN113139617A (en) | Power transmission line autonomous positioning method and device and terminal equipment | |
| CN109344836B (en) | Character recognition method and equipment | |
| CN114648656A (en) | Image recognition method and device, terminal equipment and readable storage medium | |
| CN116187299B (en) | Scientific and technological project text data verification and evaluation method, system and medium | |
| CN116401151A (en) | Precision testing method and system for picture identification algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20240126 |