Disclosure of Invention
In view of this, the present application provides an image verification method and system for entertainment system testing to improve the recognition accuracy.
According to one aspect of the present application, there is provided an image verification method for entertainment system testing, the method comprising:
acquiring an image sent to display equipment by a central control device of an entertainment system, and acquiring an image on the display equipment shot by a camera;
carrying out image recognition on the acquired image sent by the central control device to obtain a first recognition result; performing image recognition on the acquired image on the display device shot by the camera to obtain a second recognition result;
and synchronizing the first recognition result and the second recognition result, and performing image verification on the first recognition result and the second recognition result.
Preferably, synchronizing the first recognition result and the second recognition result comprises:
determining a time difference between an image sent to the display equipment by a central control device of the entertainment system and an image on the display equipment shot by the camera;
and synchronizing the first recognition result and the second recognition result according to the time difference.
Preferably, the image recognition of the acquired image transmitted by the central control device comprises: acquiring a standard image containing a target object; identifying a target object in an image sent by a central control device by using an image identification algorithm and obtaining the position of the target object in the image; comparing the position of the target object in the image with the position of the target object in the standard image, and taking the comparison result as a first identification result;
the image recognition of the acquired image shot by the camera on the display device comprises: acquiring a standard image containing a target object; identifying a target object in an image shot by the camera on the display equipment by using an image identification algorithm and obtaining the position of the target object in the image; comparing the position of the target object in the image with the position of the target object in the standard image, and taking the comparison result as a second identification result;
acquiring an image on a display device captured by a camera includes: and sampling the image shot by the camera on the display equipment according to a preset sampling frequency to obtain the image for image recognition.
Preferably, the image recognition of the acquired image transmitted by the central control device comprises:
identifying a target object in the image by using an image identification algorithm as a first identification result;
the image recognition of the acquired image shot by the camera on the display device comprises:
and identifying the target object in the image by using an image identification algorithm as a second identification result.
Preferably, the image verification of the first recognition result and the second recognition result includes:
comparing whether the target object in the first recognition result is the same as the target object in the second recognition result.
According to another aspect of the present application, the present application provides an image verification system for entertainment system testing, the entertainment system includes a central control device and a display device, and is characterized in that the system further includes an image capturing device, a camera and an image verification unit;
the image intercepting equipment is arranged between the central control device and the display equipment and is used for acquiring the image sent by the central control device and sending the image to the display equipment;
the camera is used for shooting images on the display equipment;
the image verification unit is used for acquiring an image sent to the display equipment by the central control device of the entertainment system and acquiring an image shot by the camera; carrying out image recognition on the acquired image sent by the central control device to obtain a first recognition result; performing image recognition on the obtained image shot by the camera to obtain a second recognition result; and synchronizing the first recognition result and the second recognition result, and performing image verification on the first recognition result and the second recognition result.
Preferably, the system further comprises a time difference determining module for determining the time difference between the acquisition of the image sent to the display device by the central control device of the entertainment system and the acquisition of the image on the display device shot by the camera; and the image checking unit is also used for synchronizing the first identification result and the second identification result according to the time difference.
Preferably, the image verification unit is further configured to sample the received image sent to the display device by the central control device according to a preset sampling frequency, so as to obtain an image for image recognition; and sampling the image shot by the camera on the display equipment according to a preset sampling frequency to obtain the image for image recognition.
Preferably, the image verification unit is further configured to acquire a standard image containing the target object; identifying a target object in an image sent by a central control device by using an image identification algorithm and obtaining the position of the target object in the image; comparing the position of the target object in the image with the position of the target object in the standard image, and taking the comparison result as a first identification result; identifying a target object in an image shot by the camera on the display equipment by using an image identification algorithm and obtaining the position of the target object in the image; and comparing the position of the target object in the image with the position of the target object in the standard image, and taking the comparison result as a second identification result.
Preferably, the image verification unit is further configured to compare whether the target object in the first recognition result is the same as the target object in the second recognition result.
The technical scheme provided by the application can increase the accuracy of image recognition and improve the recognition speed, not only meets the end-to-end test purpose, but also can improve the recognition efficiency.
Additional features and advantages of the present application will be described in detail in the detailed description which follows.
Detailed Description
In addition, the features of the embodiments and the respective embodiments in the present application may be combined with each other without conflict.
The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
A conventional infotainment system on a vehicle generally transmits a Low Voltage Differential Signaling (LVDS), and a central control device provides an image to a display device, such as a touch screen, in the LVDS, and a user feeds back a behavior to the central control device through the touch screen. In the prior art, the technical defects exist in the process of capturing the image of the display device only by a camera and a black screen.
To solve the problems in the prior art, the present application provides an image verification system for entertainment system testing, as shown in fig. 1. In order to obtain the LVDS signal transmitted by the central control device, an image capturing device, such as a digital auto moving company mgb (modular Frame marker) shown in fig. 1, is added between the central control device and the display device. In the process of testing the infotainment system on the vehicle, images on a human-computer interaction interface are captured simultaneously through two device channels, namely a camera and a Low Voltage Differential Signaling (LVDS) image capture device (MGB), so that the images on the infotainment system are identified and compared through an openCV algorithm, the identification precision and speed are improved, and the end-to-end test purpose of simulating the eyes of a user in the test process is met.
Preferably, the entertainment system is placed on the test stand and the camera and entertainment system are enclosed in a black box to minimize the effect of reflected light on the quality of the image captured by the camera. Preferably, because the MGB device has a network transmission function and the network camera is also selected as the camera, the video stream signals of both devices can be broadcast by the router in the same network segment.
The image verification unit in fig. 1 may be a notebook computer or a desktop computer, or may be other computing units with processing capability, such as an industrial personal computer. It should be noted that, in the present application, data collected by the MGB and data collected by the camera may be transmitted to the image verification unit through a network, or image data may be directly transferred to the image verification unit by using a storage device of the MGB and the camera.
In fig. 1, the MGB is serially connected in the path between the central controller and the display screen, and the LVDS signal is blocked, so that the LVDS is recorded into the MGB and is output to the display screen as it is as the LVDS signal without affecting the display of the display screen. When the network is used for transmitting data, the image is monitored and forwarded to the router, so that other equipment in all network segments can capture the image.
Because the video signals received by the camera and the MGB are streaming H264 coding video signals, an image verification unit, such as an industrial personal computer, is required to convert the video stream signals into image input through sampling at a certain frequency, and then some analysis and operation can be performed through image processing. Preferably, the industrial personal computer implements sampling through software, for example, receives video stream data in a form of a Python code network socket message, then reads each frame data of the video signal url through a VideoCapture module in the openCV, and reads the video into a picture through a sampling frequency of 500 milliseconds (since the video is 30 frames, that is, each frame interval is 33.3 milliseconds), so that on one hand, consumption of image processing can be reduced, and on the other hand, synchronization of image processing of the camera and the MGB device is facilitated. Fig. 2 shows a schematic diagram of image samples, where the interval between two samples is about 500ms, and the frame number of a video is usually 30 frames, that is, the interval between each frame is 33ms, because the interval is not a multiple of 33ms, it is possible to avoid the situation that no picture frame occurs at the time point of sampling, and at the same time, image processing is performed only on the key frame number node, instead of processing each frame, which reduces the processing load of the CPU of the industrial personal computer.
TABLE 1 Camera and MGB Performance index comparison table
It can be found from table 1 that under the same performance condition of the receiving-end industrial personal computer and the same network transmission condition, the MGB is superior to the camera in performance index in the points of picture input delay, recognition accuracy and recognition speed. Based on a network transmission performance of 100Mbps, i.e., a transmission rate of 12.5MB/s, the size of the artwork per frame is about 6MB bytes at 1920 x 1080 resolution. The transmission of each graph therefore requires a time delay of 6/12.5=0.48s =480 ms. The MGB collects original images matched with the infotainment system, the resolution is usually less than 1080P, the size of each frame of original image is about 3-4MB bytes, so the delay time is different, and the average time is about 300 ms.
In the testing process, the MGB cannot meet the end-to-end testing requirement, and a problem may occur, for example, in the touch screen itself, the user cannot see the screen picture, but the MGB can still capture the picture to complete the image recognition. Therefore, in order to avoid the situation and improve the test performance, the MGB and the camera are adopted to perform image recognition and comparison at the same time. However, since the delay times are different, it is necessary to synchronize the received images.
For the image obtained by sampling, the industrial personal computer can perform image recognition meeting the test target, respectively process the image obtained by the MGB and the image shot by the camera, and can directly meet the requirement of finding an object with a specific characteristic value in a specific certain image by utilizing the matchTemplate function in the openCV, for example, finding the position of an air conditioner button on an HMI interface.
As shown in table 1, since the camera image usually lags behind the picture obtained by MGB, in the test, a posterior mechanism is usually adopted, i.e. the camera image comparison result is compared with the experimental result of the image acquisition card. Because the received image resolutions are different, the image sources of the camera and the MGB cannot be completely compared based on the pixel points, and the image source synchronization is not performed based on the picture, but the result synchronization is performed based on the result obtained by image identification.
The picture collected by the MGB is usually 200ms earlier than the picture collected by the camera, and meanwhile, because the picture collected by the MGB is the original picture and the processing speed is averagely 20ms faster, the final result is that the event of the acquisition result processed by the image source of the acquisition card is about 220ms earlier than that of the image source of the camera. Therefore, when data is acquired through a network and real-time online testing is performed, after an image sent by the MGB is sampled within 500ms of sampling time, a result of camera image acquisition is waited (for 220 ms), and then image verification is performed.
When the image is verified, the object identified by the image shot by the camera and the object identified by the image acquired by the MGB corresponding to the image after the image synchronization can be compared. This process may also be implemented using the matchTemplate function. And finishing image verification by judging whether the object shot by the camera is the same as the object on the image acquired by the MGB, thereby realizing the test process.
Accordingly, the present application provides an image verification method for entertainment system testing, the method comprising:
step 101, acquiring an image sent to a display device by a central control device of an entertainment system, and acquiring an image on the display device shot by a camera;
103, carrying out image recognition on the acquired image sent by the central control device to obtain a first recognition result; performing image recognition on the acquired image on the display device shot by the camera to obtain a second recognition result;
and 105, synchronizing the first recognition result and the second recognition result, and performing image verification on the first recognition result and the second recognition result.
In step 101, in order to reduce the difficulty of data processing and facilitate image synchronization, the acquired image sent by the central control device and the acquired image captured by the camera may be sampled, for example, the sampling time interval is 500 ms.
Step 103, when the graph sent by the central control device is identified, a standard image containing the target object can be obtained, the target object in the image sent by the central control device is identified by using an image identification algorithm and the position of the target object in the image is obtained, the position of the target object in the image is compared with the position of the target object in the standard image, and the comparison result is used as a first identification result; when the image on the display device shot by the camera is imaged, a standard image containing a target object can be acquired, the target object in the image on the display device shot by the camera is identified by using an image identification algorithm and the position of the target object in the image is obtained, the position of the target object in the image is compared with the position of the target object in the standard image, and the comparison result is used as a second identification result. In this step, the object in the image can be identified by using the existing image identification technology as the basis for image verification. Generally, an object identified according to an image shot by a camera can be compared with an object in an acquired image sent by a central control device, so that an error can be tested when the display of a display screen is incorrect, but the image sent by the central control device is correct, the reason for the error can be further analyzed, and the existing problems can be solved.
In step 105, the time difference between the image sent by the central control device and the image shot by the camera can be used to determine the waiting time after the image sent by the central control device is sampled, so that the image sent by the central control device and the image shot by the camera can be synchronized. As described above, the object in the image shot by the camera and the object in the image sent by the central control device can be compared, so that the image verification is realized. If the two images are the same, the test is passed, if the two images are different, the images shot by the camera need to be returned as main test results, and the images sent by the central control device are fed back in an attached mode, so that the problems existing in analysis and positioning are facilitated.
It should be noted that the above steps are not executed in strict order, and those skilled in the art may change the order of executing the steps according to the specific test environment, and the implementation scheme does not depart from the purpose of the present application, and is also covered by the technical scheme protected by the present application.
In the testing process, if the processing results corresponding to the image shot by the camera and the image obtained by the MGB are different, the image processing result obtained by the camera is taken as a main result, and an image processing result based on the image acquisition card is given in an attached mode. The test report can be used for positioning the reason of the problem, such as the problem of the screen, the problem of camera position calibration or the problem of an image recognition algorithm, by analyzing the test report when the test results are different. The method and the device can help improve reliability and optimize result analysis efficiency in the aspect of HMI (human machine interface) test of the vehicle machine.
The above description is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the present application, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present application should be included in the scope of the present application.