TWI697871B - Inspection system for image containing mosaic and method thereof - Google Patents

Abstract

The invention discloses an inspection system for image containing mosaic and method thereof. An image pre-processing module extracts pictures from a known picture set containing mosaic to obtain grayscale picture and original color picture, and the picture is used for mosaic recognition training in machine learning. A grayscale edge intersection extraction unit extracts feature information such as a horizontal edge, a vertical edge, an intersection point of the two edges, and a mergeable range of the intersection point respectively for the grayscale picture. A MB edge intersection point extraction unit performs MB smoothing angle and line feature extraction on RGB three colors of the original color picture to obtain feature information such as a horizontal edge, a vertical edge, an intersection point of the two edges, and a mergeable range of the intersection point. A mosaic feature analysis module analyzes mosaic features of the grayscale picture according to the feature information of the grayscale picture, and analyzes mosaic features of the original color picture according to the feature information of the RGB three colors of the original color picture.

Description

Inspection system and method for image containing mosaic

The present invention relates to a mosaic inspection technology, in particular to an inspection system and method for an image containing mosaics.

The presence of mosaics in movies often causes viewers to be unhappy, and affects the quality of film and television services, and also reduces the quality of the film. In addition, film inspectors usually use the subjective method of the human eye to find whether the image contains mosaics, which consumes a lot of manpower and working time. At the same time, in response to the video classification, video inspectors need to filter videos that contain mosaics to avoid inappropriate confusion between violent, pornographic videos and general videos, so as to check whether the video meets the set viewing level before the video is released.

In addition, the prior art proposes a video mosaic image detection method, which first converts the image to be detected into a grayscale image to perform cannae edge detection to obtain an image with only contour information; then, Use four templates to perform template matching on the contour image to obtain four matching images; then, perform thresholding processing on the four matching images to obtain an image with only matching points; then, use sliding window mosaic detection The algorithm detects whether the image has mosaics. However, this existing technology cannot be used in Marseille When the boundary of the gram range is not obvious (similar to the surrounding color), it is found that there is a mosaic, and the difference between the mesh object, the text and the real mosaic cannot be distinguished, and the red, green and blue (RGB) layer can not be found to contain the mosaic.

Therefore, how to provide a novel or innovative inspection system and method for mosaics in images has become a major research topic for those skilled in the art.

The present invention provides a novel or innovative inspection system and method for images containing mosaics, which can solve the problem that film inspectors use human eyes to subjectively find whether pictures or films contain mosaics and consume a lot of manpower and working time, thereby reducing film inspection Burden and improve the quality of the video.

The inspection system for images containing mosaics of the present invention includes: a picture pre-processing module, which extracts at least one picture from a set of known mosaic-containing pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein: The picture is used for the mosaic recognition training of machine learning; the edge line and intersection feature extraction module has a gray-scale edge-line intersection extraction unit and an MB (macro block) edge-line intersection extraction unit, where the gray-scale edge line intersection extraction unit The grayscale image converted by the image pre-processing module is extracted from the horizontal edge, vertical edge, intersection of horizontal edge and vertical edge of the grayscale image, and feature information of the mergeable range of the intersection, and the MB edgeline intersection extraction unit performs The red, green, and blue (RGB) colors of the original color image retained by the processing module are respectively extracted by MB smoothing angle and line features to obtain the horizontal, vertical, and horizontal edges of the red, green, and blue (RGB) colors of the original color image The intersection of the edge and the vertical edge, and the feature information of the mergeable range of the intersection; and a mosaic feature analysis module based on the grayscale map extracted by the grayscale edgeline intersection extraction unit Analyze the mosaic features of the grayscale image from the feature information, and analyze the red, green, and blue (RGB) three colors of the original color image based on the feature information of the original color image extracted by the MB edge line intersection extraction unit Mosaic features of color.

The method for checking mosaics in an image of the present invention includes: extracting at least one picture from a set of known mosaic-containing pictures by a picture pre-processing module to convert the picture into a grayscale picture and retain the original color picture of the picture. The picture Mosaic recognition training for machine learning; the grayscale image converted by the image preprocessing module by a grayscale edgeline intersection extraction unit extracts the horizontal edgeline, vertical edgeline, intersection of horizontal edgeline and vertical edgeline, and The intersection can merge the feature information of the range; an MB (macro block) edge intersection extraction unit performs MB smoothing of the red, green, and blue (RGB) colors of the original color image retained by the image preprocessing module respectively. Extract to obtain the original color image of the red, green and blue (RGB) three colors of the horizontal edge, vertical edge, intersection of horizontal and vertical edges, and feature information of the mergeable range of the intersection; and a mosaic feature analysis module based on gray The feature information of the grayscale map extracted by the edge line intersection extraction unit analyzes the mosaic feature of the grayscale map, and the red, green, and blue (RGB) three-color feature information of the original color image extracted by the MB edge line intersection extraction unit is analyzed The original color picture is a mosaic of red, green and blue (RGB) colors.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, embodiments are specifically described below in conjunction with the accompanying drawings. In the following description, the additional features and advantages of the present invention will be partially described, and these features and advantages will be partly known from the description, or can be learned by practicing the present invention. The features and advantages of the present invention are realized and achieved by means of the elements and combinations specifically pointed out in the scope of the patent application. It should be understood that the foregoing general description and the following detailed description are both exemplary and explanatory, and are not intended to limit the scope of the present invention. Surrounding.

1‧‧‧ Inspection system for mosaic in images

10‧‧‧Detection image capture module

11‧‧‧Picture title unit to be tested

12‧‧‧Picture Capture Unit

20‧‧‧Picture preprocessing module

21‧‧‧RGB2Y unit

22‧‧‧Mosaic range rough estimation unit

30‧‧‧Edge and intersection feature extraction module

31‧‧‧Gray scale edge line intersection extraction unit

32‧‧‧MB Edge Intersection Extraction Unit

40‧‧‧Mosaic feature analysis module

41‧‧‧Grayscale mosaic feature analysis unit

42‧‧‧MB mosaic feature analysis unit

50‧‧‧Characteristic parameter module

51‧‧‧Gray scale intersection and line feature parameter unit

52‧‧‧MB Smooth intersection and line feature parameter unit

60‧‧‧Picture Mosaic Evaluation Module

61‧‧‧Feature weighting parameter adjustment unit

62‧‧‧Mosaic Evaluation Unit

63‧‧‧Mosaic accuracy rate parameter unit

64‧‧‧Non-mosaic accuracy rate parameter unit

70‧‧‧Pictures to be detected

80‧‧‧A collection of known mosaic pictures

90‧‧‧Video File

100‧‧‧Feature weighting parameter group

P‧‧‧corner point

A01 to A12, B01 to B09, C01 to C04‧‧‧Steps

Figure 1 is a schematic diagram of the architecture of the inspection system for mosaics in images of the present invention; Figure 2 is a schematic flow diagram of the inspection method for mosaics in images of the present invention in training mode; Figures 3A to 3D are respectively A schematic diagram of the four right-angle forms of the mosaic features of the invention; Figure 4 is a schematic flow diagram of the inspection method of the present invention for an image containing mosaic in the inspection mode; Figures 5A to 5D are respectively an image containing mosaics in the present invention The grayscale map of the image, the vertical edge feature of the mosaic in the image, the horizontal edge feature of the mosaic in the image, the MB edge and intersection feature of the mosaic in the image; and Figure 6 is the inspection method for the mosaic in the image of the present invention Schematic diagram of the process in picture capture mode.

The following describes the implementation of the present invention with specific specific embodiments. Those familiar with this technology can understand the other advantages and effects of the present invention from the contents disclosed in this specification, and can also implement other different specific equivalent embodiments. Or apply.

The present invention provides an inspection system and method for mosaics in images, which can Use machine learning to find out whether the picture or video (picture set to be detected) contains a mosaic, and provide the location of the mosaic and the playback time relative to the beginning of the film, which can solve the problem of the examiner using the subjective method of human eyes to find whether the video contains non-artificial mosaics Existence, resulting in the problem of consuming a lot of manpower and working time, thereby reducing the burden of film inspection to ensure the quality of the film.

In addition, the present invention can find out pictures or videos that contain mosaics caused by damaged or incomplete information during transcoding, and collect some known mosaics from such videos that are known to contain mosaics. After the mosaic recognition training of machine learning is used, a feature weighting parameter group that can find the currently searched mosaic image is obtained, and then the feature weighting parameter group is used to check the mosaic image in the image to find the mosaic image. The position and relative playing time of the movie title.

At the same time, the present invention can find out the existence range of mosaic when the boundary of the mosaic image is not obvious (close to the surrounding color), and can also distinguish the difference between mesh objects, text and the real mosaic range, thereby solving the problem of the prior art problem.

Fig. 1 is a schematic diagram of the structure of the inspection system 1 with mosaics in images of the present invention, and its main technical content is as follows, and the rest of the technical content is the same as the detailed description of Figs. 2 to 6, which will not be repeated here.

As shown in Fig. 1, the inspection system 1 for detecting mosaics in images of the present invention may include a detection image capturing module 10, a image preprocessing module 20, a sideline and intersection feature extraction module 30, and a mosaic feature Analysis module 40, a characteristic parameter module 50, a picture mosaic evaluation module 60, a picture set 70 to be detected, a picture set known to contain mosaic 80 (store pictures known to contain a mosaic picture), a video file 90 And a feature weighting parameter group 100. Moreover, the detection picture capture module 10 may have a picture title unit 11 and a picture capture Unit 12, the image pre-processing module 20 may have an RGB2Y (red, green and blue to gray scale) unit 21 and a rough mosaic range estimation unit 22, and the edge line and intersection feature extraction module 30 may have a gray scale edge line intersection extraction unit 31 and An MB (Macroblock; macro block) edge line intersection extraction unit 32, the mosaic feature analysis module 40 may have a gray-scale mosaic feature analysis unit 41 and an MB mosaic feature analysis unit 42, and the feature parameter module 50 may have a gray Order intersection and line feature parameter unit 51 and an MB smooth intersection and line feature parameter unit 52. The image mosaic evaluation module 60 may have a feature weighting parameter adjustment unit 61, a mosaic evaluation unit 62, a mosaic accuracy parameter unit 63, and A non-mosaic accuracy parameter unit 64.

In short, the inspection system 1 and method for detecting mosaics in images of the present invention mainly includes: taking at least one picture from the known mosaic picture set 80 by the picture preprocessing module 20 to convert the picture into a grayscale image And retain the original color image of the image, where the image is used for mosaic recognition training of machine learning; the grayscale image converted by the grayscale edgeline intersection extraction unit 31 of the edge and intersection feature extraction module 30 to the image preprocessing module 20 Extract the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and feature information of the mergeable range of the grayscale image, and pre-process the image by the MB edge intersection extraction unit 32 of the edge and intersection feature extraction module 30 The red, green, and blue (RGB) colors of the original color image retained by the module 20 are respectively extracted by MB smoothing angle and line features to obtain the horizontal, vertical, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of the edge and the vertical edge, and the feature information of the mergeable range of the intersection; and the mosaic feature analysis module 40 analyzes the mosaic feature of the grayscale image based on the feature information of the grayscale image extracted by the grayscale edgeline intersection extraction unit 31, The red, green and blue (RGB) three-color mosaic features of the original color image are analyzed according to the feature information of the red, green and blue (RGB) colors of the original color image extracted by the MB edge line intersection extraction unit 32.

In addition, before searching for or checking mosaics in a movie, the present invention can collect pictures known to contain mosaic pictures from movies known to contain mosaic pictures in advance to store them in the known mosaic picture collection 80, and then set the machine to The training mode shown in Figure 2 below is used to train machine learning how to recognize mosaics.

Figure 2 is a flow diagram of the method for checking mosaics in images of the present invention in training mode. It can include the following steps A01 to A12; Figures 3A to 3D are the four right angles of the mosaic features of the present invention. Schematic diagram of the shape (please refer to Figure 1 together).

In step A01 of Figure 2, first query whether all the pictures in the set of known mosaic pictures 80 shown in Figure 1 have been detected (completed mosaic recognition training)?

If it is (representing the completion of the mosaic recognition training), proceed to step A02 in FIG. 2, and the image mosaic evaluation module 60 shown in FIG. 1 stores the value of the feature weighting parameter group 100 as a post-training feature weighting parameter group. On the contrary, if no (indicating that the mosaic recognition training has not been completed), proceed to step A03 in Fig. 2, and the picture pre-processing module 20 shown in Fig. 1 will take out a mosaic picture set 80 that has not yet been trained for recognition The image is then converted into a grayscale image by the RGB2Y (red-green-blue-to-grayscale) unit 21 of the image pre-processing module 20. In the aforementioned RGB2Y unit 21, RGB represents the red (Red)/green ( Green)/Blue (Blue) three colors, Y represents grayscale or brightness. For example, in step A03 in Fig. 2, the picture pre-processing module 20 takes out a picture that has not been identified and trained from the known mosaic picture set 80 shown in Fig. 1, and the RGB2Y unit 21 converts the picture to YUV (Brightness/chroma/density) conversion to retain the grayscale image of Y and the original color image.

In step A04 in Figure 2, extract the feature from the edge and intersection shown in Figure 1. Take the gray-scale edge line intersection extraction unit 31 of the module 30 to extract the horizontal edge line, vertical edge line, the intersection point of the horizontal edge and vertical edge line, and the mergeable range of the intersection point and other characteristic information of the gray-scale image, and combine the feature information of the gray-scale image Or related information is recorded or stored in the gray level intersection and line feature parameter unit 51 of the feature parameter module 50 shown in FIG. 1.

In step A05 in Figure 2, the edge line and the MB (macro block) edge line intersection extraction unit 32 of the feature extraction module 30 shown in Figure 1 separate the red, green, and blue (RGB) colors of the original color image. Perform MB (macro block) smooth corner and line feature extraction to obtain its horizontal edge, vertical edge, intersection point of horizontal edge and vertical edge, and the mergeable range of the intersection and other feature information, and combine the red, green, and blue ( The characteristic information of the three colors (RGB) is stored in the MB smooth intersection and line characteristic parameter unit 52 of the characteristic parameter module 50 shown in FIG. 1.

In step A06 of Figure 2, the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 analyzes the gray-scale intersection of the feature parameter module 50 and the gray level recorded by the line feature parameter unit 51 Whether the feature information of the picture has the four right-angle forms of mosaic features (see Fig. 3A to Fig. 3D), and the parameters of these feature information are based on the inside and outside of the angular feature range (shown in Fig. 3A to Fig. 3D 16 Take the range of each point as an example) corresponding to the pixel value of each point in the grayscale map as the component number, and then record the score and range information of each group of parameters obtained in the grayscale map quality score table.

Similarly, in step A07 in Figure 2, the MB mosaic feature analysis unit 42 of the mosaic feature analysis module 40 shown in Figure 1 analyzes the MB smooth intersection point of the feature parameter module 50 and the line feature parameter unit 52. Whether the feature information of the red, green, and blue (RGB) colors of the original color image has mosaic features, and the pixel value of each point in the range of six adjacent points along the right angle according to the four right-angle forms shown in Figure 3A to Figure 3D When the number of components, then the parameters obtained The information of the score and range is recorded in the MB (Macro Block) Quality Score Sheet.

In step A08 in Figure 2, the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 multiplies the value in the gray-scale image quality score table by the weighted value of the relevant parameter, and the mosaic The MB mosaic feature analysis unit 42 of the feature analysis module 40 multiplies the value in the MB quality score table by the relevant weighting value of the feature weighting parameter group, and then stores the multiplied value in the picture with the accuracy of mosaic and The accuracy of the picture without mosaic is in the two variables. For example, the gray-scale mosaic feature analysis unit 41 calculates the gray-scale accuracy of both the picture with mosaic and the picture without mosaic according to the gray-scale image quality score table and the related gray-scale weighting parameter values, so as to align the two gray levels. The accuracy rates are respectively stored in the gray-scale accuracy parameters and the gray-scale non-accuracy parameters. At the same time, the MB mosaic feature analysis unit 42 calculates the MB accuracy rates of both the picture with mosaic and the picture without mosaic according to the MB quality score table and the relevant MB weighting parameter values, so that the MB accuracy rates of the two are respectively stored in the MB standard. The accuracy parameters and MB non-accuracy parameters.

Then, the picture mosaic evaluation module 60 shown in Fig. 1 calculates the mosaic accuracy and non-accuracy respectively according to the gray-scale accuracy parameters, gray-scale non-accuracy parameters, MB accuracy parameters, MB non-accuracy parameters and related weighting parameter values. The mosaic accuracy rate is to record the mosaic accuracy rate and the non-mosaic accuracy rate in the mosaic accuracy rate parameter unit 63 and the non-mosaic accuracy rate parameter unit 64 of the picture mosaic evaluation module 60 respectively.

In step A09 in Fig. 2, the mosaic evaluation unit 62 of the picture mosaic evaluation module 60 shown in Fig. 1 compares the accuracy of the picture without mosaic and the accuracy of the picture with mosaic.

If the accuracy of the picture without mosaic is greater, proceed to step A10 in Figure 2. The image mosaic evaluation module 60 sequentially adjusts a weighting value in the feature weighting parameter group 100, and records that the feature weighting parameter group 100 has changed, and then according to the gray scale accuracy parameter, gray scale non-accuracy rate parameter, and MB accuracy rate parameter , MB non-accuracy rate parameters and related weighting parameter values to recalculate mosaic accuracy and non-mosaic accuracy. Conversely, if the accuracy of the picture with mosaic is greater or equal to the accuracy of the picture without mosaics (representing the picture with mosaics), proceed to step A11 in Figure 2, and the picture mosaic evaluation module 60 determines whether the feature weighting parameter set 100 has changed . If the feature weighting parameter set 100 has changed, the feature weighting parameter adjustment unit 61 of the image mosaic evaluation module 60 first clears the changed flags of the feature weighting parameter set 100, and clears the analyzed flags (setting from the first Start to remake the inspection and analysis of the mosaic in the picture), and then return to the place where the known mosaic-containing picture set 80 has been detected in step A01 of Figure 2 to determine whether the training is completed or the mosaic recognition training of the next picture . Or, if the feature weighting parameter group 100 has not changed, then directly return to step A01 in Figure 2 and continue to execute the next mosaic picture. After the mosaic recognition training is completed, the image mosaic evaluation module 60 stores the value of the feature weighting parameter group 100 as a post-training feature weighting parameter group.

Figure 4 is a schematic diagram of the flow of the inspection method for the image containing mosaics in the inspection mode of the present invention (please refer to Figure 1 together).

In short, in the inspection mode of Fig. 4, at least one picture can be taken from the picture set 70 to be inspected shown in Fig. 1 and used by the RGB2Y (red-green-blue-to-gray scale) unit of the picture preprocessing module 20 21. Convert the picture to YUV (brightness/chroma/density) to a grayscale image for inspection. The rest of the procedure is similar to the training mode shown in Figure 2, but the value of the feature weighting parameter group after training is no longer changed. At the same time, when the accuracy rate of the picture with mosaic is greater or equal to the accuracy rate of the picture without mosaic, the picture mosaic evaluation module 60 will determine that one of the pictures in the picture set 70 to be detected contains Marseilles. The position of the frame and the playing time relative to the movie title are recorded to continue the detection of the next picture in the picture set 70 to be detected until the content of the picture set to be detected 70 is checked.

For example, in step B01 in Fig. 4, first query whether the pictures in the picture set 70 to be inspected shown in Fig. 1 have all been inspected?

If it is (the detection has been completed), end. On the contrary, if not (the detection has not been completed), proceed to step B02 in Fig. 4, and take out an untested picture from the test picture capture module 10 shown in Fig. 1 from the test picture title unit 11 The detected pictures are converted to YUV to retain the Y grayscale image and the original color image.

In step B03 in Figure 4 (similar to step A04 in Figure 2), the gray-scale edge-line intersection extraction unit 31 of the edge and intersection feature extraction module 30 shown in Figure 1 extracts horizontal edges and vertical edges from the gray-scale image. The feature information of the edge, the intersection of the horizontal edge and the vertical edge, and the mergeable range of the intersection, and the feature information or related information is recorded or stored in the gray level intersection and line feature parameter unit 51 of the feature parameter module 50 shown in Figure 1 .

In step B04 in Fig. 4 (similar to step A05 in Fig. 2), the MB (macro block) edge intersection extraction unit 32 shown in Fig. 1 performs the red, green, and blue (RGB) colors of the original color image. Perform MB (macro block) smoothing angle and line feature extraction to obtain feature information such as horizontal edge, vertical edge, intersection of horizontal and vertical edge, and mergeable range of intersection, and store the feature information as shown in Figure 1. The MB smooth intersection point and line feature parameter unit 52 of the feature parameter module 50.

In step B05 in Figure 4, the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 analyzes whether the gray-scale intersection point and the line feature parameter unit 51 are within a range of the gray-scale map. The four or more corner points P (see Figure 3A to Figure 3D) and Whether the pixels in the range of four lines or more have the same value, and whether the edge of the range has the same value as outside the box to obtain the complex parameter, and the complex parameter is given scores according to the mosaic feature and the non-mosaic feature to obtain the group of parameters Score, and then record the score and range information of each group of parameters in the grayscale chart quality score table.

In step B06 in Figure 4, the MB mosaic feature analysis unit 42 of the mosaic feature analysis module 40 shown in Figure 1 analyzes the red, green, and blue (RGB) colors of the original color image by the MB smooth intersection point and the line feature parameter unit 52 Is there a pair of more than four corner points P (see Figures 3A to 3D) and more than four lines in a range, whether the pixels in the range have the same value, and whether the edges of the range are the same as those outside the frame The values of the four right-angle shapes shown in Figure 3A to Figure 3D along the six points adjacent to the right angle are combined with the scores obtained by the feature weighting parameter group 100 and recorded in the MB quality score table.

In step B07 in Figure 4 (similar to step A08 in Figure 2), the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 multiplies the values in the gray-scale image quality score table The related parameter weighted value is used, and the MB mosaic feature analysis unit 42 of the mosaic feature analysis module 40 multiplies the value in the MB quality score table by the related weighted value of the feature weighted parameter group, and then the multiplied value is accumulated Stored in the variables of the accuracy of the picture with mosaic and the accuracy of the picture without mosaic.

In step B08 in Fig. 4, the mosaic evaluation unit 62 of the picture mosaic evaluation module 60 shown in Fig. 1 compares the accuracy of the picture without mosaic and the accuracy of the picture with mosaic.

If the accuracy of the picture without mosaic is greater, proceed to step B09 in Fig. 4, and the picture mosaic evaluation module 60 records the position of the mosaic picture and the playback relative to the title. Time, then return to step B01. On the contrary, if the accuracy of the picture with mosaic is greater or equal to the accuracy of the picture without mosaic, then directly return to step B01 in Figure 4.

[First embodiment: training mode]

Please refer to Figures 1 to 2 and Figures 5A to 5D. Figures 5A to 5D are respectively the grayscale map with mosaic in the image and the vertical edge feature of the image with mosaic in the image of the present invention. , The image contains the horizontal edge feature of the mosaic, and the image contains the MB edge and intersection feature of the mosaic.

For example, as shown in Figure 2 the image contains mosaic inspection method in the training mode of the flow diagram, in step A01 of Figure 2, first query the known mosaic picture set 80 shown in Figure 1 Are all of the pictures tested?

If it is (representing the completion of the mosaic recognition training), proceed to step A02 in FIG. 2, and the image mosaic evaluation module 60 shown in FIG. 1 stores the value of the feature weighting parameter group 100 as a post-training feature weighting parameter group. On the contrary, if no (indicating that the mosaic recognition training has not been completed), proceed to step A03 in Fig. 2, and the picture pre-processing module 20 shown in Fig. 1 will take out a mosaic picture set 80 that has not yet been trained for recognition The picture is then converted by the RGB2Y (red-green-blue-to-grayscale) unit 21 of the picture preprocessing module 20 into a grayscale image containing mosaics as shown in FIG. 5A.

In step A04 in Figure 2, the gray-scale edge-line intersection extraction unit 31 of the edge and intersection feature extraction module 30 shown in Figure 1 performs a 3*3 matrix linear convolution on the gray-scale map shown in Figure 5A. The edge detection of the kernel and the horizontal line convolution kernel moves one point each time to generate the four points of the straight line and the horizontal line as shown in Figure 5B to Figure 5C as a set of information, and the intersection of the straight line and the horizontal line The corner feature information of 4 points on the straight line and the horizontal line of the point (intersection point), and then divided Do not record or store characteristic information or related information such as the horizontal edge, vertical edge, intersection of horizontal and vertical edges, and the mergeable range of the intersection, etc., in the gray-scale intersection and line feature parameter unit of the feature parameter module 50 shown in Figure 1. 51.

In step A05 in Fig. 2, the MB (macro block) edge line intersection extraction unit 32 of the edge line shown in Figure 1 and the intersection feature extraction module 30 performs the following procedures P1 to P3 to the picture (original color image) The red, green, and blue (RGB) of the MB (macro block) smooth corner and line feature extraction are performed separately.

In the procedure P1, the MB (macro block) edge line intersection extraction unit 32 respectively subtracts the pixel value of the (4*n)th line and the pixel value of the (4*n-1)th line in the x direction of the picture Take the absolute value (n=1) to obtain (picture width/4) horizontal picture information, and respectively compare the pixel value of the (4*n)th line in the y direction of the picture with the (4*n-1)th line The pixel value of the line is subtracted from the absolute value (n=1) to obtain the vertical image information in the y direction (image height/4), where n is a positive integer. The reason for choosing 4*n above is that the minimum MB (macro block) of h264 is 4x4, 8*8, 16*16, and the minimum coding tree unit (Coding Tree) of High Efficiency Video Coding (HEVC) Block; CTU) is 8*8, 64*64, so the minimum value is 4.

In procedure P2, the MB (macro block) edge intersection extraction unit 32 respectively finds out the same edge or difference interval for the picture information in the horizontal direction and the picture information in the vertical direction (threshold value can be set) Compare), and then find the intersection (corner) from the edge to get the result of the MB edge and intersection feature of the mosaic in the image shown in Figure 5D.

In the procedure P3, the MB (macro block) edge intersection extraction unit 32 performs color grouping confirmation on the result of the MB edge and intersection feature containing the mosaic in the image shown in Figure 5D, including: (i) a row along the edge Or whether the pixels in the left row and the edge pixels can become If the values of two different groups are marked as boundaries, store the information in the MB edge and intersection feature parameters; and (ii) around the intersection point can become two different gray-scale groups, as shown in Figures 3A to 3D The information is stored in the MB smooth intersection point and line feature parameter unit 52 of the feature parameter module 50 shown in FIG. 1 for the corner points P of the four right-angle forms (the thick black circle is the corner point P).

In step A06 of Figure 2, the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 analyzes the gray-scale intersection of the feature parameter module 50 and the gray level recorded by the line feature parameter unit 51 Whether the feature information of the picture has the four right-angle forms of mosaic features (see Fig. 3A to Fig. 3D), and the parameters of these feature information are based on the inside and outside of the angular feature range (shown in Fig. 3A to Fig. 3D 16 Take the range of each point as an example) corresponding to the pixel value of each point in the grayscale map as the component number, and then record the score and range information of each group of parameters obtained in the grayscale map quality score table.

Similarly, in step A07 of Figure 2, the MB mosaic feature analysis unit 42 of the mosaic feature analysis module 40 shown in Figure 1 analyzes whether the MB smooth intersection point of the feature parameter module 50 and the line feature parameter unit 52 information It has a mosaic feature, and the pixel value of each point range adjacent to each side of the six points at the right angle is used as the component number according to the four right-angle patterns shown in the four right-angle patterns shown in Figure 3A to Figure 3D, and then the scores and range information of each set of parameters are obtained Recorded in MB (macro block) quality score table.

In step A08 in Figure 2, the gray-scale mosaic feature analysis unit 41 of the mosaic feature analysis module 40 shown in Figure 1 multiplies the value in the gray-scale image quality score table by the weighted value of the relevant parameter, and compares it The multiplied values respectively progressively accumulate the accuracy rate of the picture with mosaic and the accuracy rate of the picture without mosaic, and respectively store them in the gray-scale accuracy parameters and gray-scale non-accuracy parameters of the mosaic accuracy parameter unit 63 of the picture mosaic evaluation module 60. . At the same time, as shown in Figure 1 Shows that the MB mosaic feature analysis unit 42 of the mosaic feature analysis module 40 multiplies the value in the MB quality score table by the weighted value of the relevant parameter, and the multiplied value is respectively progressively added to the accuracy of the picture with mosaic and the picture without mosaic The accuracy rates are respectively stored in the MB accuracy rate parameter and the MB non-accuracy rate parameter of the mosaic accuracy rate parameter unit 63 of the image mosaic evaluation module 60.

Then, the mosaic accuracy parameter unit 63 of the picture mosaic evaluation module 60 shown in Figure 1 adds the "gray scale accuracy parameter multiplied by the gray scale accuracy weighting value" plus "MB accuracy rate parameter multiplied by the MB accuracy weighting Value" to generate the mosaic accuracy, and the non-mosaic accuracy parameter unit 64 of the image mosaic evaluation module 60 shown in Figure 1 multiplies the gray-scale non-accuracy parameter by the gray-scale non-accuracy weighted value and adds " The MB non-accuracy rate parameter is multiplied by the MB non-accuracy rate weighted value" to generate the non-mosaic accuracy rate.

In step A09 in Fig. 2, the mosaic evaluation unit 62 of the picture mosaic evaluation module 60 shown in Fig. 1 compares the accuracy of the picture without mosaic and the accuracy of the picture with mosaic.

If the accuracy of the picture without mosaic is high, proceed to step A10 in Figure 2, and the picture mosaic evaluation module 60 sequentially adjusts a weighting value in the feature weighting parameter group 100, and records that the feature weighting parameter group 100 has changed. Recalculate the mosaic accuracy and non-mosaic accuracy according to the gray-scale accuracy parameters, gray-scale non-accuracy parameters, MB accuracy parameters, MB non-accuracy parameters and related weighted parameter values. Conversely, if the accuracy of the picture with mosaic is greater or equal to the accuracy of the picture without mosaics (representing the picture with mosaics), proceed to step A11 in Figure 2, and the picture mosaic evaluation module 60 determines whether the feature weighting parameter set 100 has changed . If the feature weighting parameter set 100 has changed, the feature weighting parameter adjustment unit 61 of the image mosaic evaluation module 60 first clears the changed flags of the feature weighting parameter set 100, and clears the analyzed flags (set Make sure to remake the inspection and analysis of the mosaic in the picture from the first picture), and then return to the place where the known mosaic picture set 80 has been tested as shown in step A01 in Figure 2 to determine whether the training is completed or the next Picture mosaic recognition training. On the contrary, if the feature weighting parameter group 100 has not changed, it will directly return to step A01 in Figure 2 and continue to execute the next mosaic picture. After the mosaic recognition training is completed, the image mosaic evaluation module 60 stores the value of the feature weighting parameter group 100 as a post-training feature weighting parameter group.

[Second Example: Inspection of Mosaic in Image]

Please refer to Fig. 1, Fig. 4 and Fig. 6. Fig. 6 is a flowchart of the method for checking mosaics in images of the present invention in the image capture mode, and Fig. 6 can include the following steps C01 to Step C04.

In short, in Figure 6, the image can be captured by the image capture unit 12 shown in Figure 1, and the image can be stored in the to-be-detected image collection 70 until the capture of the video is completed. The picture capturing unit 12 notifies the picture preprocessing module 20 that all the capturing has been completed. When the first picture is captured, the picture capturing unit 12 informs the picture pre-processing module 20 to start the inspection of the mosaic in the image.

For example, in step C01 in FIG. 6, the image capture unit 12 of the detection image capture module 10 shown in FIG. 1 captures and processes images of the video file. In step C02 of Fig. 6, the picture capturing unit 12 takes out the most recent I-frame (frame) into 1 picture every t minutes (t is an adjustable value, and the initial value of t is set to 1) Stored in the to-be-detected picture collection 70, and sequentially store the title of each picture in the to-be-tested picture title unit 11. When the first picture is generated, the picture capturing unit 12 informs the picture pre-processing module 20 to start execution . In step C03 in Figure 6, the picture capturing unit 12 determines whether the video is over (that is, the video is captured)?

If not (the movie is not over), return to step C02. On the contrary, if it is (the movie has ended), proceed to step C03, and the picture capturing unit 12 informs the picture preprocessing module 20 that all pictures of the movie have been captured.

Then, as shown in the flowchart of the inspection method of the image containing mosaic in the inspection mode as shown in Fig. 4, at least one picture is taken from the to-be-detected picture set 70 shown in Fig. 1, and the RGB2Y of the picture pre-processing module 20 Unit 21 (red, green and blue to grayscale) converts the picture into a YUV (brightness/chroma/concentration) image for inspection. The rest of the procedure is similar to the training mode shown in Figure 2, but the training is no longer changed The value of the post feature weighting parameter group. At the same time, when the accuracy rate of the picture with mosaic is greater or equal to the accuracy of the picture without mosaic, the picture mosaic evaluation module 60 records the position of the mosaic picture in the picture set 70 to be detected and the relative play time of the title to continue. The detection of a picture under the picture set 70 to be detected is performed until the content check of the picture set 70 to be detected is completed. For the detailed technical content of the inspection mode, please refer to the complete description of step B01 to step B09 in Figure 4 above, which will not be repeated here.

In summary, the inspection system and method for detecting mosaics in images of the present invention can at least have the following features, advantages or technical effects.

1. The present invention uses machine learning for mosaic recognition training and MB (macro block) smooth edge and intersection RGB (red, green, and blue) feature extraction, which can find when the boundary of the mosaic image is not obvious (similar to the surrounding color) It can also distinguish the difference between the mesh object, text and the real mosaic range, and also find the difference between the red, green and blue (RGB) layer of the mosaic.

2. The present invention can use machine learning to find whether a picture or movie (picture set to be detected) contains the position of the mosaic picture or the playing time relative to the title, which can solve the movie check Personnel use the subjective method of human eyes to find whether the film contains mosaics, which consumes a lot of manpower and working time, thereby reducing the burden of film inspection and improving the quality of the film.

3. The present invention can find out that the original picture or video has been damaged or the information is incomplete during the conversion. It can collect some known mosaic pictures from such videos that are known to contain mosaic pictures. After the mosaic recognition training of machine learning is used, a feature weighting parameter group that can find the currently searched mosaic image is obtained, and then the feature weighting parameter group is used to check the mosaic image in the image to find the mosaic image. The position and relative playing time of the movie title.

4. The present invention is in response to the classification of videos and can be used to filter videos containing (artificial) mosaic images to avoid violent, pornographic videos or improper confusion of general videos. It can also be used to check whether the videos meet the set viewing level before the videos are put on the shelves. , So as to ensure the quality of the film on the film or classification review.

5. The present invention can complete the film quality check or viewing level check before the film is filmed. For example, for various platforms such as MOD (multimedia platform for video on demand), OTT (providing audio and video content to users through the Internet) and other platforms, perform a video quality check or a viewing level check before the film is released.

The above embodiments are only illustrative of the principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone who is familiar with the art can comment on the above without departing from the spirit and scope of the present invention. Modifications and changes to the implementation form. Any equivalent changes and modifications made by using the contents disclosed in the present invention should still be covered by the scope of the patent application. Therefore, the protection scope of the present invention should be as listed in the scope of patent application.

1‧‧‧ Inspection system for mosaic in images

10‧‧‧Detection image capture module

11‧‧‧Picture title unit to be tested

12‧‧‧Picture Capture Unit

20‧‧‧Picture preprocessing module

21‧‧‧RGB2Y unit

22‧‧‧Mosaic range rough estimation unit

30‧‧‧Edge and intersection feature extraction module

31‧‧‧Gray scale edge line intersection extraction unit

32‧‧‧MB Edge Intersection Extraction Unit

40‧‧‧Mosaic feature analysis module

41‧‧‧Grayscale mosaic feature analysis unit

42‧‧‧MB mosaic feature analysis unit

50‧‧‧Characteristic parameter module

51‧‧‧Gray scale intersection and line feature parameter unit

52‧‧‧MB Smooth intersection and line feature parameter unit

60‧‧‧Picture Mosaic Evaluation Module

61‧‧‧Feature weighting parameter adjustment unit

62‧‧‧Mosaic Evaluation Unit

63‧‧‧Mosaic accuracy rate parameter unit

64‧‧‧Non-mosaic accuracy rate parameter unit

70‧‧‧Pictures to be detected

80‧‧‧A collection of known mosaic pictures

90‧‧‧Video File

100‧‧‧Feature weighting parameter group

Claims (9)

An image containing mosaic inspection system includes: a picture pre-processing module, which takes at least one picture from a set of known mosaic pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein The image is used for mosaic recognition training of machine learning; the edge line and intersection feature extraction module has: a gray-scale edge-line intersection extraction unit that extracts the gray-scale image converted by the image pre-processing module The feature information of the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and the mergeable range of the intersection of the grayscale image; and an MB (macro block) edge intersection extraction unit, which is used by the image pre-processing module The red, green, and blue (RGB) colors of the original color image are retained for MB smoothing angle and line feature extraction to obtain the horizontal, vertical, horizontal, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of vertical edges and the feature information of the mergeable range of intersections; a mosaic feature analysis module analyzes the mosaic features of the grayscale map based on the feature information of the grayscale map extracted by the grayscale edgeline intersection extraction unit, And analyze the red, green and blue (RGB) three-color mosaic feature of the original color image based on the feature information of the red, green and blue (RGB) three colors of the original color image extracted by the MB edge line intersection extraction unit; and a gray scale Image quality score table for a gray-scale mosaic feature analysis unit of the mosaic feature analysis module to analyze whether the feature information of the gray-scale image has four right-angle forms of mosaic features, and base the feature information on the parameters The inside and outside of the angular feature range corresponds to the pixel value of each point of the grayscale map as the component number, and the score and range information of each group of parameters are recorded in the grayscale map quality score table. For example, the inspection system described in item 1 of the scope of patent application, wherein the gray-scale mosaic feature analysis unit further calculates the gray-scale image quality score table and related gray-scale weighting parameter values respectively. Calculate the grayscale accuracy of both the picture with mosaic and the picture without mosaic, so that the grayscale accuracy of the two are stored in the grayscale accuracy parameter and the grayscale non-accuracy parameter. An image containing mosaic inspection system includes: a picture pre-processing module, which takes at least one picture from a set of known mosaic pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein The image is used for mosaic recognition training of machine learning; the edge line and intersection feature extraction module has: a gray-scale edge-line intersection extraction unit that extracts the gray-scale image converted by the image pre-processing module The feature information of the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and the mergeable range of the intersection of the grayscale image; and an MB (macro block) edge intersection extraction unit, which is used by the image pre-processing module The red, green, and blue (RGB) colors of the original color image are retained for MB smoothing angle and line feature extraction to obtain the horizontal, vertical, horizontal, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of vertical edges and the feature information of the mergeable range of intersections; a mosaic feature analysis module analyzes the mosaic features of the grayscale map based on the feature information of the grayscale map extracted by the grayscale edgeline intersection extraction unit, And analyze the red, green, and blue (RGB) mosaic characteristics of the original color image based on the feature information of the red, green, and blue (RGB) colors of the original color image extracted by the MB edge line intersection extraction unit; and an MB quality A score table for one of the MB mosaic feature analysis units of the mosaic feature analysis module to analyze whether the red, green, and blue (RGB) feature information of the original color image has mosaic features, and based on the four right-angle shapes The pixel value of each point range adjacent to the right-angle 6 points is used as the component number, and the score and range information of each group of parameters are recorded in the MB quality score table. For example, the inspection system described in item 3 of the scope of patent application, wherein the MB mosaic feature analysis unit further calculates the MB accuracy rates of both the picture with mosaic and the picture without mosaic according to the MB quality score table and the relevant MB weighting parameter value. , To store the MB accuracy rate of the two in the MB accuracy rate parameter and the MB non-accuracy rate parameter respectively. An image containing mosaic inspection system includes: a picture pre-processing module, which takes at least one picture from a set of known mosaic pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein The image is used for mosaic recognition training of machine learning; the edge line and intersection feature extraction module has: a gray-scale edge-line intersection extraction unit that extracts the gray-scale image converted by the image pre-processing module The feature information of the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and the mergeable range of the intersection of the grayscale image; and an MB (macro block) edge intersection extraction unit, which is used by the image pre-processing module The red, green, and blue (RGB) colors of the original color image are retained for MB smoothing angle and line feature extraction to obtain the horizontal, vertical, horizontal, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of vertical edges and the feature information of the mergeable range of intersections; a mosaic feature analysis module analyzes the mosaic features of the grayscale map based on the feature information of the grayscale map extracted by the grayscale edgeline intersection extraction unit, And analyze the red, green and blue (RGB) three-color mosaic feature of the original color image according to the feature information of the red, green and blue (RGB) color of the original color image extracted by the MB edge line intersection extraction unit; and a mosaic feature The image mosaic evaluation module of the accuracy parameter unit and the non-mosaic accuracy parameter unit, wherein the image mosaic evaluation module is based on the gray scale accuracy parameter, gray scale non-accuracy rate parameter, MB accuracy rate parameter, MB non-accuracy rate Parameters and related weighted parameter values respectively The mosaic accuracy rate and the non-mosaic accuracy rate are calculated to record the mosaic accuracy rate and the non-mosaic accuracy rate in the mosaic accuracy rate parameter unit and the non-mosaic accuracy rate parameter unit respectively. For example, in the inspection system described in item 5 of the scope of patent application, the image mosaic evaluation module has a mosaic evaluation unit that compares the accuracy rate of the image without mosaic and the accuracy rate of the image with mosaic. If the accuracy of the picture without mosaic is greater, the picture mosaic evaluation module adjusts a weighting value of the feature weighting parameter group, and records the change of the feature weighting parameter group, and then according to the gray scale accuracy parameter and gray scale non-precision The accuracy parameter, the MB accuracy parameter, the MB non-accuracy rate parameter and the relevant weighting parameter value are recalculated for the mosaic accuracy rate and the non-mosaic accuracy rate. An image containing mosaic inspection system includes: a picture pre-processing module, which takes at least one picture from a set of known mosaic pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein The image is used for mosaic recognition training of machine learning; the edge line and intersection feature extraction module has: a gray-scale edge-line intersection extraction unit that extracts the gray-scale image converted by the image pre-processing module The feature information of the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and the mergeable range of the intersection of the grayscale image; and an MB (macro block) edge intersection extraction unit, which is used by the image pre-processing module The red, green, and blue (RGB) colors of the original color image are retained for MB smoothing angle and line feature extraction to obtain the horizontal, vertical, horizontal, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of vertical edges and the feature information of the mergeable range of intersections; a mosaic feature analysis module analyzes the mosaic features of the grayscale map based on the feature information of the grayscale map extracted by the grayscale edgeline intersection extraction unit, And based on the intersection of the MB edge The feature information of the red, green and blue (RGB) colors of the original color image extracted by the extraction unit analyzes the mosaic features of the red, green and blue (RGB) colors of the original color image; and a gray-scale mosaic feature analysis unit and a The grayscale map quality score table, wherein the grayscale mosaic feature analysis unit analyzes whether there are more than four corner points and four lines in pairs in a range of the grayscale map, and whether the pixels in the range Have the same value and whether the edge of the range has the same value as outside the box to obtain the complex parameter, and score the complex parameter according to the mosaic feature and non-mosaic feature to obtain the score of each group of parameters, and then the group The score of the parameter and the information of the range are recorded in the grayscale image quality score table. An image containing mosaic inspection system includes: a picture pre-processing module, which takes at least one picture from a set of known mosaic pictures to convert the picture into a grayscale picture and retain the original color picture of the picture, wherein The image is used for mosaic recognition training of machine learning; the edge line and intersection feature extraction module has: a gray-scale edge-line intersection extraction unit that extracts the gray-scale image converted by the image pre-processing module The feature information of the horizontal edge, vertical edge, intersection of horizontal and vertical edge, and the mergeable range of the intersection of the grayscale image; and an MB (macro block) edge intersection extraction unit, which is used by the image pre-processing module The red, green, and blue (RGB) colors of the original color image are retained for MB smoothing angle and line feature extraction to obtain the horizontal, vertical, horizontal, and horizontal edges of the red, green, and blue (RGB) colors of the original color image. The intersection of vertical edges and the feature information of the mergeable range of intersections; a mosaic feature analysis module analyzes the mosaic features of the grayscale map based on the feature information of the grayscale map extracted by the grayscale edgeline intersection extraction unit, And based on the intersection of the MB edge The red, green and blue (RGB) three-color feature information of the original color image extracted by the extraction unit analyzes the red, green and blue (RGB) three-color mosaic features of the original color image; and an MB mosaic feature analysis unit and an MB The quality score table, where the MB mosaic feature analysis unit analyzes whether there are more than four corner points and more than four lines in pairs in one of the three colors of red, green and blue (RGB) of the original color image. Whether the pixels in the range have the same value, and whether the edge of the range has the same value as the outside of the frame, and the four right-angle shapes along the right-angle 6 points adjacent to the two sides of the value are combined with the feature weighting parameter group to obtain the score and record in this MB quality score sheet. A method for checking mosaics in an image includes: extracting at least one picture from a set of known mosaic-containing pictures by a picture pre-processing module to convert the picture into a grayscale image and retain the original color image of the picture, wherein: The image is used for mosaic recognition training of machine learning; the gray-scale image converted by the image pre-processing module by a gray-scale edge line intersection extraction unit extracts the horizontal edge, vertical edge, horizontal edge and vertical of the gray-scale image. The intersection of the edges and the feature information of the mergeable range of intersections; the red, green, and blue (RGB) colors of the original color image retained by the image preprocessing module by an MB (macro block) edge intersection extraction unit Perform MB smoothing angle and line feature extraction to obtain the original color image's red, green and blue (RGB) three-color horizontal edge, vertical edge, intersection of horizontal and vertical edges, and feature information of the mergeable range of intersections; The mosaic feature analysis module analyzes the mosaic feature of the grayscale image based on the feature information of the grayscale image extracted by the grayscale edge line intersection extraction unit, and based on the redness of the original color image extracted by the MB edgeline intersection extraction unit The characteristic information of the three colors of green and blue (RGB) analyzes the mosaic characteristics of the three colors of red, green and blue (RGB) of the original color image; and A gray-scale mosaic feature analysis unit of the mosaic feature analysis module analyzes whether the feature information of the gray-scale map has four right-angle forms of mosaic features, and corresponds the parameters of the feature information according to the inside and outside of the angular feature range The pixel value of each point in the grayscale map is used as the component number, and the information of the score and range of each group of parameters is recorded in a grayscale map quality score table.

Images