CN117036831B - Method, system and medium for detecting splash screen based on time sequence feature modeling - Google Patents
Method, system and medium for detecting splash screen based on time sequence feature modeling Download PDFInfo
- Publication number
- CN117036831B CN117036831B CN202311293486.8A CN202311293486A CN117036831B CN 117036831 B CN117036831 B CN 117036831B CN 202311293486 A CN202311293486 A CN 202311293486A CN 117036831 B CN117036831 B CN 117036831B
- Authority
- CN
- China
- Prior art keywords
- screen
- video frame
- test unit
- basic video
- splash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000001514 detection method Methods 0.000 claims abstract description 159
- 238000012360 testing method Methods 0.000 claims abstract description 129
- 238000012549 training Methods 0.000 claims description 57
- 238000013145 classification model Methods 0.000 claims description 16
- 238000010276 construction Methods 0.000 claims description 15
- 238000003860 storage Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 12
- 238000005520 cutting process Methods 0.000 claims description 6
- 238000003708 edge detection Methods 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 238000006073 displacement reaction Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 230000007547 defect Effects 0.000 abstract description 12
- 238000012545 processing Methods 0.000 description 6
- 230000002123 temporal effect Effects 0.000 description 6
- 125000004122 cyclic group Chemical group 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000013142 basic testing Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
Abstract
The application discloses a method, a system and a medium for detecting a splash screen based on time sequence feature modeling, wherein the method comprises the following steps: acquiring continuous multi-frame screen images of equipment to be detected in a preset test time period; dividing the continuous multi-frame screen image into a plurality of basic video frame test units, each basic video frame test unit comprising: a preset frame number and continuous multi-frame screen images; extracting effective detection areas of screen images of each frame in a basic video frame test unit; and inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit. The invention ensures that the detection model has higher defect detection accuracy and robustness under the dynamic complex background.
Description
Technical Field
The application relates to the technical field of screen detection, in particular to a method, a system and a medium for detecting a screen flash based on time sequence feature modeling.
Background
In the research and development process, the factory test and the later use process of users of the electronic equipment, the phenomenon of screen flash often occurs due to the driving adaptation, the software and hardware setting and other reasons. In general, testing of electronic devices is performed in various processes of production, users are simulated in different environments, functions of a screen display of the electronic devices are detected, and a test scene of the screen display cannot be fixed, for example, a video is randomly played, and a game popular at a certain time is played. Therefore, the screen-flicking defect generally occurs in a dynamic scene, and the screen suddenly appears a single flicker and returns to normal in a very short time, or continuously appears a plurality of flicker. In order to improve user experience and reduce labor cost when detecting the problem of screen-flashing of electronic products, it is necessary to develop a screen-flashing detection algorithm adapted to dynamic background.
In the prior art, the technology of screen flash detection utilizes a traditional visual algorithm. For example, when the number of brightness changes of a screen within a preset time is detected to detect a screen flash, a plurality of time points are usually set, the brightness A1 of a current screen picture is detected at a first time point, the brightness A2 of the current screen picture is detected at a second time point, when the difference value between the first brightness value A1 and the second brightness value A2 is larger than a difference value threshold value C, the screen change at the current time point is determined, the number of brightness changes within the preset time period is smaller than or equal to the number threshold value, the equipment is considered to have the screen flash defect, the method for detecting the screen flash by only calculating the brightness difference of the screen is difficult to adapt to a dynamic scene, the brightness difference of different scenes is large, and the fixed brightness threshold value is difficult to be compatible with all background scenes, so that detection omission and false detection occur when facing the dynamic scene. Still other methods use a tone scale to solve the robustness problem of single brightness, and obtain n screen images of a terminal screen in a target period, where n is a positive integer; obtaining the tone scale ratio of each color contained in each screen in the n screens, wherein the tone scale ratio refers to the ratio of the number of pixels of the color to the number of all pixels contained in the screen; selecting the maximum tone scale ratio of the tone scales of the colors contained in each screen picture; according to the maximum tone ratio of each screen picture, whether each screen picture is in a periodic-like relationship is determined, if the maximum tone ratio of most of the n screen pictures is in a periodic-like relationship, the maximum tone ratio of the n screen pictures can be determined to be in a periodic-like relationship, the tone ratio of the white screen is close to 1 because the white screen and the original screen picture are switched according to a certain time interval, and the tone ratio of the original screen picture is low because a plurality of colors exist in the original screen picture, so that the frequent switching of the white screen and the original screen picture is in a periodic-like relationship, and whether the flash screen phenomenon occurs in a target period is judged through the periodic-like relationship of the tone ratio. The method adopts the tone scale characteristic to reduce the interference degree of the detection algorithm by a dynamic complex scene, but the proposed periodic-like change is based on a flashing pure-color screen, such as a flashing black screen, a flashing white screen and the like, and the tone scale of the flashing screen is an extreme value, so that when the tiny flashing phenomenon occurs, the method cannot take the effect, for example, the flashing screen frame only wholly generates a slight white film phenomenon on an original image frame, and various colors in the original background can still be represented through the white film.
In view of the above problems, an effective technical solution is currently needed.
Disclosure of Invention
The invention aims to provide a method, a system and a medium for detecting screen-flashing based on time sequence feature modeling, which can detect screen-flashing defects of an electronic display device screen under a dynamic background.
The application provides a method for detecting a splash screen based on time sequence feature modeling, which comprises the following steps:
acquiring continuous multi-frame screen images of equipment to be detected in a preset test time period;
dividing the continuous multi-frame screen image into a plurality of basic video frame test units, each basic video frame test unit comprising: a preset frame number and continuous multi-frame screen images;
extracting effective detection areas of screen images of each frame in a basic video frame test unit;
and inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit.
Preferably, the effective detection area extracting method of the basic video frame testing unit includes: cutting at a fixed position of a basic video frame test unit through a preset fixed size to obtain an effective detection area; or firstly adopting a Canny edge detection algorithm to detect the edge of the image in the basic video frame test unit to obtain an edge image, acquiring the maximum circumscribed rectangle information in the edge image, and mapping the circumscribed rectangle back to the basic video frame test unit to obtain an effective detection area.
Preferably, the training process of the splash screen detection model is as follows:
acquiring screen image data, and respectively storing a first folder and a second folder which are preset according to whether screen images are in a screen flashing state or not to obtain a training data set, wherein the first folder and the second folder respectively store basic training units of corresponding types, and each basic training unit comprises a plurality of frames of screen images;
constructing a video classification model based on TSM;
and training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
Preferably, the basic training unit and the basic video frame test unit have the same number of screen image frames.
Preferably, the video classification model of the TSM uses a time transfer module to carry out displacement processing on two-dimensional screen images at different moments so as to obtain characteristic association of the images in time sequence.
Preferably, the specific steps of the screen-flashing detection model for the effective detection area are as follows:
judging whether the basic video frame test unit flashes;
if yes, saving iteration of the current basic video frame test unit for the screen-flash detection model;
if the screen is not in the flash screen, the detection of the next basic video frame test unit is continued until all the basic video frame test units in the test time period complete the detection.
The second aspect of the application provides a system for modeling splash screen detection based on time sequence characteristics, which comprises a screen image frame acquisition module, a test unit construction module, an effective detection area extraction module, a splash screen detection module and a message pushing module,
the screen image frame acquisition module is used for acquiring continuous multi-frame screen images of the equipment to be detected in a preset test time period;
the test unit construction module is used for dividing a multi-frame screen image into a plurality of basic video frame test units, and each basic video frame test unit comprises: a preset frame number and continuous multi-frame screen images;
the effective detection area extraction module is used for extracting effective detection areas of screen images of each frame in the basic video frame test unit;
the screen-flash detection module is used for inputting the effective detection area of each frame of screen image in each basic video frame test unit into a pre-trained screen-flash detection model and outputting a screen-flash detection result corresponding to each basic video frame test unit. The method comprises the steps of carrying out a first treatment on the surface of the
The message pushing module is used for pushing the screen flashing detection abnormal message.
Preferably, if the effective detection area appears to be in the splash screen, splash screen samples are collected and data labeling is carried out to construct a splash screen sample data set.
Preferably, the construction process of the splash screen detection model is as follows:
acquiring screen image data, and respectively storing the screen image data into a first folder and a second folder according to whether the screen image is flashed, wherein the first folder and the second folder respectively store basic training units of corresponding categories to obtain a training data set;
constructing a video classification model based on TSM;
and training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
A third aspect of the present application provides a computer readable storage medium, where the computer readable storage medium includes a program for modeling a splash screen detection method based on a time sequence feature, where the program for modeling the splash screen detection method based on the time sequence feature, when executed by a processor, implements the steps of the method for modeling the splash screen detection method based on the time sequence feature.
As can be seen from the above, the method, system and medium for detecting a splash screen based on time sequence feature modeling provided by the present application acquire continuous screen images of a plurality of frames of equipment within a preset test time; extracting continuous frame screen images from the screen images to form a basic video frame test unit; extracting an effective detection area of a basic video frame test unit; and finally, performing screen-flashing detection on the effective detection area by using a preset screen-flashing detection model, so as to ensure that the detection model has higher defect detection accuracy and robustness under a dynamic complex background.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for detecting a splash screen based on time sequence feature modeling according to an embodiment of the present application;
fig. 2 is a flowchart of construction of a splash screen detection model according to an embodiment of the present application;
FIG. 3 is a basic data format diagram of a training sample set according to an embodiment of the present application;
FIG. 4 (a) is a training sample showing a splash screen provided in an embodiment of the present application;
FIG. 4 (b) is a training sample for a splash screen display according to an embodiment of the present application;
FIG. 4 (c) is a sample of a training sample showing a slight splash screen provided by an embodiment of the present application;
FIG. 4 (d) is a sample of a training sample representing a splash screen provided by an embodiment of the present application;
FIG. 5 is a timing feature extraction schematic provided in an embodiment of the present application;
fig. 6 is a flowchart of performing a splash screen detection on an effective detection area by using a splash screen detection model provided in an embodiment of the present application;
fig. 7 is a block diagram of a system for detecting a splash screen based on timing characteristics modeling according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that like reference numerals and letters refer to like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Referring to fig. 1, a flowchart of a method for modeling splash screen detection based on time sequence features in some embodiments of the present application is shown. The method for detecting the screen flash based on the time sequence feature modeling is used for detecting the screen of the electronic terminal equipment, such as a computer, a mobile phone terminal and the like. The method for detecting the splash screen based on the time sequence feature modeling comprises the following steps:
s101, acquiring continuous multi-frame screen images of equipment to be detected in a preset test time period;
s102, dividing the continuous multi-frame screen image into a plurality of basic video frame test units, wherein each basic video frame test unit comprises: a preset frame number and continuous multi-frame screen images;
s103, extracting effective detection areas of screen images of all frames in a basic video frame test unit;
s104, inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit.
It should be noted that the present technology aims to acquire a plurality of continuous frame device screen images within a preset test period by using an image acquisition device; dividing the continuous multi-frame screen image into a plurality of basic video frame test units, each basic video frame test unit comprising: a preset frame number and continuous multi-frame screen images; extracting effective detection areas of screen images of each frame in a basic video frame test unit; and inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit. In the scheme, the high-definition image acquisition device can be used for shooting the screen of the electronic equipment to acquire screen images, the continuous shooting duration can be set according to the actual scene requirement, for example, the continuous shooting duration can be set to be 5 seconds or 6 seconds, if the frame rate is 30 frames/second, 150 frames of screen images are acquired within 5 seconds, and 180 frames of screen images are acquired within 6 seconds. And acquiring continuous screen images, then performing frame extraction, namely extracting a plurality of frames at certain intervals, and forming the extracted frames into a basic video frame test unit. In a specific embodiment, a test unit is composed of 8 consecutive frames, mainly for detecting a single black or white screen defect.
It should be noted that, after the splash screen detection model detects the splash screen abnormality in the present application, the splash screen sample is collected and data is labeled to construct a splash screen sample data set, and the data set can be used for training the splash screen detection model.
In addition, the detection abnormal information is sent to the server side, and the basic test unit with the screen flash is stored and displayed. In a specific embodiment, a front-end display device may be provided to push the image of the splash sample onto the display device and prompt for the splash type.
According to the embodiment of the invention, the effective detection area extraction method of the basic video frame test unit comprises the following steps: cutting at a fixed position of a basic video frame test unit through a preset fixed size to obtain an effective detection area; or firstly adopting a Canny edge detection algorithm to detect the edge of the image in the basic video frame test unit to obtain an edge image, acquiring the maximum circumscribed rectangle information in the edge image, and mapping the circumscribed rectangle back to the basic video frame test unit to obtain an effective detection area.
In a specific embodiment, firstly, image edge information is obtained according to a Canny edge algorithm, then, a maximum circumscribed rectangle is obtained according to an image edge, then, the size of the circumscribed rectangle is judged, and if an effective area is smaller than a preset size (for example, the length and the width are 500 x 500), preset size clipping is adopted as an effective detection area.
It should be noted that, in the application, the basic video frame test unit may be extracted by a plurality of methods, for example, the effective detection area may be obtained by presetting a fixed size and cutting the size at a fixed position of the basic video frame test unit. In addition, a Canny edge detection algorithm can be adopted to detect the edge of the image in the basic video frame test unit, an edge image is obtained, the maximum circumscribed rectangle information is obtained in the edge image, and the circumscribed rectangle is mapped back to the basic video frame test unit, so that an effective detection area is obtained. The method is not limited to the extraction mode of the effective detection area, and any mode capable of extracting the image area can be applied to the method.
Referring to fig. 2, a flowchart of the construction of a splash screen detection model according to an embodiment of the present application is shown. According to the embodiment of the invention, the training process of the splash screen detection model is as follows:
s201, acquiring screen image data, and respectively storing a first folder and a second folder which are preset according to whether screen images are in a screen flashing state or not to obtain a training data set, wherein the first folder and the second folder respectively store basic training units of corresponding types, and each basic training unit comprises a plurality of frames of screen images;
s202, constructing a video classification model based on TSM;
and S203, training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
It should be noted that, the preset first folder is an OK folder, i.e. a sample folder without screen flashing, the second folder is a ShanPing folder, i.e. a folder with screen flashing samples, and the first folder and the second folder respectively store basic training units of corresponding categories, where each basic training unit includes a plurality of screen images. In a specific embodiment, the basic training unit is composed of 8 consecutive frames of images, and the basic data format of the training sample set is shown in fig. 3 below.
Preferably, the basic training unit and the basic video frame test unit have the same number of screen image frames.
It should be noted that the basic training unit and the basic video frame test unit have the same number of video frame images. As shown in fig. 4 (a) -4 (d), the partial splash training samples are respectively, from top to bottom, that is, fig. 4 (a) shows splash black, fig. 4 (b) shows splash color, fig. 4 (c) shows slight splash white (i.e., splash white film), and fig. 4 (d) shows splash white. In order to reduce the difficulty of data collection, make the test scene that the screen shot detects the model can cover more simultaneously, this application has also adopted abundant data construction mode, through collecting the OK sample of general volume, then mix with the screen shot sample at random thereby make more screen shot defect samples. Through rich data construction modes, the time for constructing a sample set can be reduced, and model deployment is quickened.
According to the embodiment of the invention, the video classification model of the TSM uses a time transfer module to carry out displacement processing on two-dimensional screen images at different moments so as to obtain characteristic association of the images on time sequence.
It should be noted that, the TSM is a video classification neural network model, and the application of the TSM in the splash screen detection reconstructs its data input, and the video is changed into the basic video frame test unit described herein, so as to form a dense detection of the splash screen defect. The TSM backbone network selects a MobileNet V3 classification structure, and a time transfer module is used for carrying out shift processing on two-dimensional image features at different moments, so that feature association on time sequence is obtained. More specifically, 1,2,3, and 4 in fig. 5 can be regarded as feature maps of the first, second, third, and fourth images in time series, and then a feature shift operation (Temporal shift) is performed in time series (Temporal T) to fuse the Temporal features of the preceding and following frames. And in the model training, a front-back frame cross cyclic fusion (cyclic shift) mode is adopted, and in the online test, only the front frame and the current frame feature information are fused.
Referring to fig. 6, a flowchart of performing a splash screen detection on an effective detection area in the splash screen detection model provided in the embodiment of the present application is shown. According to the embodiment of the invention, the specific steps of the screen-flashing detection model for the effective detection area are as follows:
s601, judging whether a basic video frame test unit flashes;
s602, if the screen is in a screen-flash state, saving iteration of the current basic video frame test unit for the screen-flash detection model;
and S603, if the screen is not in the flash screen, continuing to detect the next basic video frame test unit until all the basic video frame test units in the test time period complete detection.
When the screen shot detection is performed, an output value of 0 of the screen shot detection model indicates OK, and an output value of 1 indicates screen shot.
Referring to fig. 7, a block diagram of a system for detecting a splash screen based on time sequence feature modeling is provided in an embodiment of the present application.
A second aspect of the present application provides a system for modeling splash screen detection based on timing characteristics, comprising: a screen image frame acquisition module 701, a test unit construction module 702, an effective detection area extraction module 703, a splash screen detection module 704, a message pushing module 705,
the screen image frame acquisition module 701 is configured to acquire continuous multi-frame screen images of the device to be detected in a preset test period;
the test unit construction module 702 is configured to divide the multi-frame screen image into a plurality of basic video frame test units, where each basic video frame test unit includes: a preset frame number and continuous multi-frame screen images;
the effective detection area extracting module 703 is configured to extract an effective detection area of each frame of screen image in the basic video frame testing unit;
the screen-flash detection module 704 is configured to input an effective detection area of each frame of screen image in each basic video frame test unit into a pre-trained screen-flash detection model, and output a screen-flash detection result corresponding to each basic video frame test unit. The method comprises the steps of carrying out a first treatment on the surface of the
The message pushing module 705 is configured to push a splash screen to detect an abnormal message.
Preferably, if the effective detection area appears to be in the splash screen, splash screen samples are collected and data labeling is carried out to construct a splash screen sample data set.
It should be noted that the present technology aims to acquire a plurality of continuous frame device screen images within a preset test period by using an image acquisition device; dividing the continuous multi-frame screen image into a plurality of basic video frame test units, each basic video frame test unit comprising: a preset frame number and continuous multi-frame screen images; extracting effective detection areas of screen images of each frame in a basic video frame test unit; and inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit. In the scheme, the high-definition image acquisition device can be used for shooting the screen of the electronic equipment to acquire screen images, the continuous shooting duration can be set according to the actual scene requirement, for example, the continuous shooting duration can be set to be 5 seconds or 6 seconds, if the frame rate is 30 frames/second, 150 frames of screen images are acquired within 5 seconds, and 180 frames of screen images are acquired within 6 seconds. And acquiring continuous screen images, then performing frame extraction, namely extracting a plurality of frames at certain intervals, and forming the extracted frames into a basic video frame test unit. In a specific embodiment, a test unit is composed of 8 consecutive frames, mainly for detecting a single black or white screen defect.
It should be noted that, after the splash screen detection model detects the splash screen abnormality in the present application, the splash screen sample is collected and data is labeled to construct a splash screen sample data set, and the data set can be used for training the splash screen detection model.
In addition, the detection abnormal information is sent to the server side, and the basic test unit with the screen flash is stored and displayed. In a specific embodiment, a front-end display device may be provided to push the image of the splash sample onto the display device and prompt for the splash type.
According to the embodiment of the invention, the effective detection area extraction method of the basic video frame test unit comprises the following steps: cutting at a fixed position of a basic video frame test unit through a preset fixed size to obtain an effective detection area; or firstly adopting a Canny edge detection algorithm to detect the edge of the image in the basic video frame test unit to obtain an edge image, acquiring the maximum circumscribed rectangle information in the edge image, and mapping the circumscribed rectangle back to the basic video frame test unit to obtain an effective detection area.
In a specific embodiment, firstly, image edge information is obtained according to a Canny edge algorithm, then, a maximum circumscribed rectangle is obtained according to an image edge, then, the size of the circumscribed rectangle is judged, and if an effective area is smaller than a preset size (for example, the length and the width are 500 x 500), preset size clipping is adopted as an effective detection area.
It should be noted that, in the application, the basic video frame test unit may be extracted by a plurality of methods, for example, the effective detection area may be obtained by presetting a fixed size and cutting the size at a fixed position of the basic video frame test unit. In addition, a Canny edge detection algorithm can be adopted to detect the edge of the image in the basic video frame test unit, an edge image is obtained, the maximum circumscribed rectangle information is obtained in the edge image, and the circumscribed rectangle is mapped back to the basic video frame test unit, so that an effective detection area is obtained. The method is not limited to the extraction mode of the effective detection area, and any mode capable of extracting the image area can be applied to the method.
Referring to fig. 2, a flowchart of the construction of a splash screen detection model according to an embodiment of the present application is shown. According to the embodiment of the invention, the training process of the splash screen detection model is as follows:
s201, acquiring screen image data, and respectively storing a first folder and a second folder which are preset according to whether screen images are in a screen flashing state or not to obtain a training data set, wherein the first folder and the second folder respectively store basic training units of corresponding types, and each basic training unit comprises a plurality of frames of screen images;
s202, constructing a video classification model based on TSM;
and S203, training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
It should be noted that, the preset first folder is an OK folder, i.e. a sample folder without screen flashing, the second folder is a ShanPing folder, i.e. a folder with screen flashing samples, and the first folder and the second folder respectively store basic training units of corresponding categories, where each basic training unit includes a plurality of screen images. In a specific embodiment, the basic training unit is composed of 8 consecutive frames of images, and the basic data format of the training sample set is shown in fig. 3 below.
Preferably, the basic training unit and the basic video frame test unit have the same number of screen image frames.
It should be noted that the basic training unit and the basic video frame test unit have the same number of video frame images. As shown in fig. 4 (a) -4 (d), the partial splash training samples are respectively, from top to bottom, that is, fig. 4 (a) shows splash black, fig. 4 (b) shows splash color, fig. 4 (c) shows slight splash white (i.e., splash white film), and fig. 4 (d) shows splash white. In order to reduce the difficulty of data collection, make the test scene that the screen shot detects the model can cover more simultaneously, this application has also adopted abundant data construction mode, through collecting the OK sample of general volume, then mix with the screen shot sample at random thereby make more screen shot defect samples. Through rich data construction modes, the time for constructing a sample set can be reduced, and model deployment is quickened.
According to the embodiment of the invention, the video classification model of the TSM uses a time transfer module to carry out displacement processing on two-dimensional screen images at different moments so as to obtain characteristic association of the images on time sequence.
It should be noted that, the TSM is a video classification neural network model, and the application of the TSM in the splash screen detection reconstructs its data input, and the video is changed into the basic video frame test unit described herein, so as to form a dense detection of the splash screen defect. The TSM backbone network selects a MobileNet V3 classification structure, and a time transfer module is used for carrying out shift processing on two-dimensional image features at different moments, so that feature association on time sequence is obtained. More specifically, 1,2,3, and 4 in fig. 5 can be regarded as feature maps of the first, second, third, and fourth images in time series, and then a feature shift operation (Temporal shift) is performed in time series (Temporal T) to fuse the Temporal features of the preceding and following frames. And in the model training, a front-back frame cross cyclic fusion (cyclic shift) mode is adopted, and in the online test, only the front frame and the current frame feature information are fused.
Referring to fig. 6, a flowchart of performing a splash screen detection on an effective detection area in the splash screen detection model provided in the embodiment of the present application is shown. According to the embodiment of the invention, the specific steps of the screen-flashing detection model for the effective detection area are as follows:
s601, judging whether a basic video frame test unit flashes;
s602, if the screen is in a screen-flash state, saving iteration of the current basic video frame test unit for the screen-flash detection model;
and S603, if the screen is not in the flash screen, continuing to detect the next basic video frame test unit until all the basic video frame test units in the test time period complete detection.
When the screen shot detection is performed, an output value of 0 of the screen shot detection model indicates OK, and an output value of 1 indicates screen shot.
Referring to fig. 7, a block diagram of a system for detecting a splash screen based on time sequence feature modeling is provided in an embodiment of the present application.
A third aspect of the present application provides a computer readable storage medium, where the computer readable storage medium includes a program for modeling a splash screen detection method based on a time sequence feature, where the program for modeling the splash screen detection method based on the time sequence feature, when executed by a processor, implements the steps of the method for modeling the splash screen detection method based on the time sequence feature.
According to the method, the system and the medium for detecting the screen flashing based on the time sequence feature modeling, continuous screen images of a plurality of frames of equipment are obtained in a preset test time; then, extracting frames from the screen image to form a basic video frame test unit; extracting an effective detection area of a basic video frame test unit; and finally, performing screen-flashing detection on the effective detection area by using a preset screen-flashing detection model, so as to ensure that the detection model has higher defect detection accuracy and robustness under a dynamic complex background.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, or the like, which can store program codes.
Alternatively, the above-described integrated units of the present invention may be stored in a readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
Claims (8)
1. A method for modeling splash screen detection based on time sequence features, the method comprising:
acquiring continuous multi-frame screen images of equipment to be detected in a preset test time period;
dividing the continuous multi-frame screen image into a plurality of basic video frame test units, each basic video frame test unit comprising: a preset frame number and continuous multi-frame screen images;
extracting effective detection areas of screen images of each frame in a basic video frame test unit;
inputting the effective detection areas of the screen images of each frame in each basic video frame test unit into a pre-trained screen flashing detection model, and outputting a screen flashing detection result corresponding to each basic video frame test unit;
the training process of the splash screen detection model is as follows:
acquiring screen image data, and respectively storing a first folder and a second folder which are preset according to whether screen images are in a screen flashing state or not to obtain a training data set, wherein the first folder and the second folder respectively store basic training units of corresponding types, and each basic training unit comprises a plurality of frames of screen images;
constructing a video classification model based on TSM;
and training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
2. The method for detecting the splash screen based on the time sequence feature modeling as claimed in claim 1, wherein the effective detection area extracting method of the basic video frame testing unit comprises the following steps: cutting at a fixed position of a basic video frame test unit through a preset fixed size to obtain an effective detection area; or firstly adopting a Canny edge detection algorithm to detect the edge of the image in the basic video frame test unit to obtain an edge image, acquiring the maximum circumscribed rectangle information in the edge image, and mapping the circumscribed rectangle back to the basic video frame test unit to obtain an effective detection area.
3. The method for detecting the splash screen based on the time sequence feature modeling as claimed in claim 1, wherein the basic training unit and the basic video frame testing unit have the same number of the image frames of the screen.
4. The method for detecting the splash screen based on the time sequence feature modeling according to claim 1, wherein the video classification model of the TSM uses a time transfer module to process the displacement of the two-dimensional screen images at different moments to obtain the feature association of the images in the time sequence.
5. The method for detecting the splash screen based on the time sequence feature modeling according to claim 1, wherein the specific steps of the splash screen detection model for detecting the splash screen of the effective detection area are as follows:
judging whether a current basic video frame test unit flashes;
if yes, saving iteration of the current basic video frame test unit for the screen-flash detection model;
if the screen is not in the flash screen, the detection of the next basic video frame test unit is continued until all the basic video frame test units in the test time period complete the detection.
6. A system for detecting a splash screen based on time sequence feature modeling is characterized in that a screen image frame acquisition module, a test unit construction module, an effective detection area extraction module, a splash screen detection module and a message pushing module,
the screen image frame acquisition module is used for acquiring continuous multi-frame screen images of the equipment to be detected in a preset test time period;
the test unit construction module is used for dividing a multi-frame screen image into a plurality of basic video frame test units, and each basic video frame test unit comprises: a preset frame number and continuous multi-frame screen images;
the effective detection area extraction module is used for extracting effective detection areas of screen images of each frame in the basic video frame test unit;
the screen-flashing detection module is used for inputting the effective detection area of each frame of screen image in each basic video frame test unit into a pre-trained screen-flashing detection model and outputting a screen-flashing detection result corresponding to each basic video frame test unit;
the message pushing module is used for pushing the screen flashing detection abnormal message;
the construction process of the splash screen detection model comprises the following steps:
acquiring screen image data, and respectively storing the screen image data into a first folder and a second folder according to whether the screen image is flashed, wherein the first folder and the second folder respectively store basic training units of corresponding categories to obtain a training data set;
constructing a video classification model based on TSM;
and training the TSM-based video classification model by using the training data set to obtain a splash screen detection model.
7. The system of claim 6, wherein if a splash occurs in the effective detection area, a splash sample is collected and labeled to construct a splash sample data set.
8. A computer readable storage medium, characterized in that a program of a method for detecting a time series feature based modeling splash screen is included in the computer readable storage medium, and when the program of the method is executed by a processor, the steps of the method for detecting a time series feature based modeling splash screen according to any one of claims 1 to 5 are implemented.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311293486.8A CN117036831B (en) | 2023-10-09 | 2023-10-09 | Method, system and medium for detecting splash screen based on time sequence feature modeling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311293486.8A CN117036831B (en) | 2023-10-09 | 2023-10-09 | Method, system and medium for detecting splash screen based on time sequence feature modeling |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117036831A CN117036831A (en) | 2023-11-10 |
| CN117036831B true CN117036831B (en) | 2024-03-22 |
Family
ID=88632245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311293486.8A Active CN117036831B (en) | 2023-10-09 | 2023-10-09 | Method, system and medium for detecting splash screen based on time sequence feature modeling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117036831B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017028722A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method and device for displaying splash screen image of application |
| CN111045876A (en) * | 2019-11-12 | 2020-04-21 | 广州视源电子科技股份有限公司 | Splash screen detection method and device, storage medium and electronic equipment |
| CN111290894A (en) * | 2020-01-20 | 2020-06-16 | Oppo（重庆）智能科技有限公司 | Splash screen detection method, device, terminal and storage medium |
| WO2021017859A1 (en) * | 2019-07-31 | 2021-02-04 | Oppo广东移动通信有限公司 | Screen state control method and apparatus, mobile terminal and storage medium |
| WO2021098362A1 (en) * | 2019-11-19 | 2021-05-27 | 腾讯科技（深圳）有限公司 | Video classification model construction method and apparatus, video classification method and apparatus, and device and medium |
| CN114339395A (en) * | 2021-12-14 | 2022-04-12 | 浙江大华技术股份有限公司 | Video jitter detection method, detection device, electronic equipment and readable storage medium |
| WO2022095359A1 (en) * | 2020-11-06 | 2022-05-12 | 平安科技（深圳）有限公司 | Anti-screen-capturing-based information security protection method and apparatus, electronic device and medium |
| CN115359747A (en) * | 2022-10-18 | 2022-11-18 | 无锡美科微电子技术有限公司 | Screen flashing test equipment and screen flashing test method |
-
2023
- 2023-10-09 CN CN202311293486.8A patent/CN117036831B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017028722A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method and device for displaying splash screen image of application |
| WO2021017859A1 (en) * | 2019-07-31 | 2021-02-04 | Oppo广东移动通信有限公司 | Screen state control method and apparatus, mobile terminal and storage medium |
| CN111045876A (en) * | 2019-11-12 | 2020-04-21 | 广州视源电子科技股份有限公司 | Splash screen detection method and device, storage medium and electronic equipment |
| WO2021098362A1 (en) * | 2019-11-19 | 2021-05-27 | 腾讯科技（深圳）有限公司 | Video classification model construction method and apparatus, video classification method and apparatus, and device and medium |
| CN111290894A (en) * | 2020-01-20 | 2020-06-16 | Oppo（重庆）智能科技有限公司 | Splash screen detection method, device, terminal and storage medium |
| WO2022095359A1 (en) * | 2020-11-06 | 2022-05-12 | 平安科技（深圳）有限公司 | Anti-screen-capturing-based information security protection method and apparatus, electronic device and medium |
| CN114339395A (en) * | 2021-12-14 | 2022-04-12 | 浙江大华技术股份有限公司 | Video jitter detection method, detection device, electronic equipment and readable storage medium |
| CN115359747A (en) * | 2022-10-18 | 2022-11-18 | 无锡美科微电子技术有限公司 | Screen flashing test equipment and screen flashing test method |
Non-Patent Citations (4)
| Title |
|---|
| 基于双向特征融合卷积神经网络的液晶面板缺陷检测算法;彭大芹;刘恒;许国良;邓柯;;广东通信技术(04);全文 * |
| 基于时空关系模型的交通信号灯的实时检测与识别;李宗鑫;秦勃;王梦倩;;计算机科学(06);全文 * |
| 基于时空关系模型的城市道路车辆实时检测;王元奎;秦勃;李伟;;计算机系统应用(10);全文 * |
| 视频分割中的层次化结构与匹配;候锐;马利庄;桑胜举;;计算机辅助设计与图形学学报(07);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117036831A (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108805047B (en) | Living body detection method and device, electronic equipment and computer readable medium | |
| CN107105310B (en) | Human image replacing method and device in video live broadcast and recording and broadcasting system | |
| CN113992861B (en) | Image processing method and image processing device | |
| CN111898701A (en) | Model training, frame image generation, frame interpolation method, device, equipment and medium | |
| CN100596165C (en) | Image processing apparatus and its method | |
| CN111191576A (en) | Personnel behavior target detection model construction method, intelligent analysis method and system | |
| CN105100784A (en) | Smoothness test method and device | |
| CN111401238B (en) | Method and device for detecting character close-up fragments in video | |
| CN112950738B (en) | Rendering engine processing method and device, storage medium and electronic equipment | |
| CN110717058A (en) | Information recommendation method and device and storage medium | |
| CN113592754A (en) | Image generation method and electronic equipment | |
| CN113688820A (en) | Stroboscopic stripe information identification method and device and electronic equipment | |
| CN112788329A (en) | Video static frame detection method and device, television and storage medium | |
| CN112801911B (en) | Method and device for removing text noise in natural image and storage medium | |
| CN117036831B (en) | Method, system and medium for detecting splash screen based on time sequence feature modeling | |
| CN108629254A (en) | A kind of detection method and device of moving target | |
| CN114066894A (en) | Detection method for display image reproduction, storage medium and processing equipment | |
| CN112565909B (en) | Video playing method and device, electronic equipment and readable storage medium | |
| CN113570509A (en) | Data processing method and computer device | |
| CN110782392B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN112801150A (en) | Method and device for expanding periodic video data | |
| WO2023001110A1 (en) | Neural network training method and apparatus, and electronic device | |
| CN109120856B (en) | Camera shooting method and device | |
| CN113365145B (en) | Video processing method, video playing method, video processing device, video playing device, computer equipment and storage medium | |
| CN115439386A (en) | Image fusion method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |