CN101853381B - Method and device for acquiring video subtitle information - Google Patents

Abstract

The present invention relates to a method and device for obtaining subtitle information. By performing subtitle detection based on wavelet on the luminance component image of a data frame in a video stream, the attribute information of the detected subtitle is obtained, and the detected subtitle is extracted according to the attribute information. subtitle. Thus, the subtitle information in the data frame can be accurately obtained. Since the subtitle detection based on wavelet does not need to limit the area where the subtitle is located, the embodiment of the present invention can accurately acquire the subtitle information in the video data without limiting the subtitle location area.

Description

Method and device for acquiring video subtitle information

technical field

The invention relates to the field of applied electronic technology, in particular to a method and device for acquiring video subtitle information.

Background technique

Video subtitles show people the content of video programs in an intuitive form, which can effectively assist people to grasp the theme of the program in video appreciation, and then understand the content of the video. In addition, the detection and recognition of video subtitle information can enrich text-based video content query. Therefore, effective acquisition of video subtitle information is a necessary link.

In the process of realizing the present invention, the inventor found that the existing technology for obtaining subtitle information is sensitive to the position information of the subtitle information appearing in the video screen, and usually, it is assumed that the subtitle area is static and the subtitle position is also fixed In the middle and lower part of the image, if the subtitle information is not within the specified detection range, then the subtitle information cannot be well acquired and applied.

Contents of the invention

Embodiments of the present invention provide a method and device for acquiring video subtitle information, so as to accurately acquire subtitle information in video data without limiting the subtitle location area.

The embodiment of the present invention provides a method for acquiring video subtitle information, including:

Carry out subtitle detection based on wavelet on the luminance component image of the data frame in the video stream;

Obtain the attribute information of the detected subtitles;

Based on the attribute information, the detected subtitles are extracted.

The embodiment of the present invention also provides a video subtitle information acquisition device, including:

The detection module is used to carry out subtitle detection based on wavelet to the luminance component image of the data frame in the video stream;

A first acquisition module, configured to acquire attribute information of the subtitles detected by the detection module;

An extraction module, configured to extract the subtitles detected by the detection module according to the subtitle attribute information acquired by the first acquisition module.

It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that in the embodiments of the present invention, by performing wavelet-based subtitle detection on the luminance component image of the data frame in the video stream, and obtaining the attribute information of the detected subtitles, according to the The above attribute information is used to extract the detected subtitles. Thus, the subtitle information in the data frame can be accurately obtained. Since the subtitle detection based on wavelet does not need to limit the area where the subtitle is located, the embodiment of the present invention can accurately acquire the subtitle information in the video data without limiting the subtitle location area.

Description of drawings

FIG. 1 is a first schematic flow diagram of the method provided by the embodiment of the present invention;

Fig. 2 is the second schematic flow diagram of the method provided by the embodiment of the present invention;

Fig. 3 is the third schematic flow diagram of the method provided by the embodiment of the present invention;

Fig. 4 is a schematic structural diagram of the device provided by the embodiment of the present invention;

FIG. 5 is a second structural schematic diagram of the device provided by the embodiment of the present invention;

FIG. 6 is a first structural schematic diagram of the detection module provided by the embodiment of the present invention;

FIG. 7 is a second structural schematic diagram of the detection module provided by the embodiment of the present invention;

FIG. 8 is a schematic structural diagram of the first acquisition module provided by an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of the extraction module provided by the embodiment of the present invention.

Detailed ways

The embodiment of the present invention provides a method for acquiring video subtitle information. As shown in FIG. The above attribute information is used to extract the detected subtitles. Thus, the subtitle information in the data frame can be accurately obtained. Since subtitle detection based on wavelet does not need to limit the subtitle area, the embodiment of the present invention can accurately acquire subtitle information in video data without limiting the subtitle location area.

A specific embodiment of the video subtitle information acquisition method provided by the embodiment of the present invention can be shown in Figure 2, and this embodiment can specifically include:

Step 21, acquiring the luminance component image of the specified data frame from the video data stream.

In order to speed up the speed of obtaining subtitle information, this embodiment of the present invention may specifically decode a specified data frame from a video data stream, and obtain a luminance component image of the specified data frame.

For example, only decode the intra-frame encoding with an odd (or even) frame number, that is, the code stream of the I frame (or other forms of video frames, such as predictive coding frames, that is, the P frame), and obtain the brightness component image of the I frame , while the chrominance component of the I frame and other frames are quickly skipped, thereby speeding up the speed of obtaining subtitle information.

It should be noted that the embodiment of the present invention does not limit the compression format of the video data stream.

Step 22, perform subtitle detection based on wavelet on the luminance component image of the selected data frame.

Specifically, in this step, subtitle detection based on wavelet is used for the luminance component image of the selected data frame.

In a specific embodiment, the specific execution process of this step can be as shown in Figure 3, including:

Step 221: Perform wavelet transform on the luminance component image of the data frame to obtain a horizontal high-frequency sub-band texture map, a vertical high-frequency sub-band texture map, and a diagonal high-frequency sub-band texture map.

The wavelet transform involved in the embodiments of the present invention may specifically be HAAR (Haar) wavelet transform, Mexican sombrero wavelet transform, 9-7 wavelet transform, 5-3 wavelet transform, and the like.

In this step, wavelet transform is performed on the luminance component image of the selected data frame to obtain a low-frequency sub-band and high-frequency sub-bands in the horizontal, vertical and diagonal directions, wherein the horizontal sub-high frequency sub-band It can be marked as H, the vertical high-frequency sub-band can be marked as V, and the diagonal high-frequency sub-band can be marked as D.

Calculate the absolute values of the coefficients of the three high-frequency subbands H, V, and D generated after wavelet transformation, and obtain the horizontal high-frequency sub-band texture map (CH), the vertical high-frequency sub-band texture map (CV) and the diagonal High frequency subband texture map (CD).

In this step, three high-frequency sub-band texture maps (CH, CV, CD) can also be combined to obtain a comprehensive high-frequency sub-band texture map (CS).

The value of each point in the integrated high-frequency subband texture image can be obtained by the following formula:

CS(i,j)=CH(i,j)+CV(i,j)+CD(i,j)

Step 222 , according to the horizontal high frequency subband texture map, the vertical high frequency subband texture map and the diagonal high frequency subband texture map, obtain the subtitle point image (TextPnt) of the data frame.

In a specific embodiment, this step may specifically include the following links:

First, an initial subtitle point image is generated according to the high-frequency subband texture map.

Taking the horizontal high-frequency subband texture map as an example, subtitle point detection is performed on the horizontal high-frequency subband texture map to obtain the initial subtitle point image (MAPH_ORG) of the horizontal high-frequency subband.

Wherein, the value of the horizontal high-frequency sub-band initial subtitle point image at coordinates (i, j) is calculated according to the following formula:

$\mathrm{<span\; class="notranslate">\; MAPH</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; ORG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; CH</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; TH</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; CH</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; TH</span>}\end{array}\right.$

It should be noted that the value "0" represents the background, and the value "1" represents the initial subtitle point. The calculation method of the threshold (TH) in the formula can be as follows:

$\mathrm{<span\; class="notranslate">\; TH</span>}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 50</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; 50</span>}\\ \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; 50</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 18</span>}\\ \mathrm{<span\; class="notranslate">\; 18</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 18</span>}\end{array}\right.$

MH in the formula is the average texture intensity in the horizontal high-frequency sub-band texture image.

Then, denoising is performed on the initial subtitle point image of the horizontal high-frequency subband to obtain the final subtitle point image (MAPH) in the horizontal direction.

The denoising processing involved in the embodiments of the present invention may specifically adopt mature technical processing schemes such as overlapping and sliding block filtering, which is not limited in the embodiments of the present invention.

Next, perform similar processing steps on the vertical high-frequency subband texture map and the diagonal high-frequency subband texture map to obtain the vertical subband initial subtitle point image (MAPV_ORG) and the diagonal subband initial subtitle point image (MAPD_ORG) , and de-noise the initial subtitle dot image in the vertical subband and the initial subtitle dot image in the diagonal subband to obtain the final subtitle dot image in the vertical direction (MAPV) and the final subtitle dot image in the diagonal direction (MAPD).

Finally, intersect the final caption point images (MAPH, MAPV, MAPD) in the three directions to obtain the caption point image (TextPnt) of the data frame.

It should be noted that, in the embodiment of the present invention, the initial subtitle point image (MAP_ORG) is removed from the subtitle noise point, and the specific implementation method flow of obtaining the subtitle area can be realized by the following procedures:

//h, w represent the height and width of the subband image respectively

block=4;//The size of the block

dis=3; //The distance of each square offset

h_num=(h/dis);//The number of times the block is offset in the vertical direction

w_num=(w/dis);//The number of times the block is offset in the horizontal direction

MAP = MAPH_ORG;

for(k=1:h_num)

for(I＝1:w_num)

If(((k-1)＊dis+1+block＞h)||((I-1)＊dis+1+block＞w))

continue;//If the sub-block moves out of the boundary of the image, jump out of the loop

else

          num=TextPntNum(); //Statistics of the number of subtitle points inside the box

if(num<(block＊block/2))

      StartH＝(k-1)＊dis;

    EndH＝StartH+block;

     StartW＝(I-1)＊dis;

    EndW＝StartW+block;

MAP(StartH:End H, StartW:EndW)＝0;

//If the number is less than (block*block/2), all pixels in this square area are internally set to 0,

// That is, the subtitle points in this box are noise points

Else//If the number is greater than (block＊block/2), this square area is the real subtitle point

  MAP(StartH:EndH, StartW:EndW)＝MAP_ORG(StartH:EndH, StartW:EndW)

end

end

It can be understood that the above examples are for illustration only, and do not limit the protection scope of the embodiments of the present invention in any way.

Step 223, generate a subtitle area image (TextArea) from the subtitle point image of the data frame.

In a specific embodiment, this step may specifically include the following steps:

Firstly, the closed operation and the open operation in the horizontal direction are respectively performed on the acquired subtitle point image to obtain a horizontal image (Verlmg).

Among them, the structural elements of the closing operation can be a 20*1 full "1" matrix, and the structural elements of the opening operation can be a 1*2 full "1" matrix. Of course, the structural elements used in the closing operation and the opening operation can be based on In practice, flexible arrangements are required.

Next, perform vertical closing and opening operations on the subtitle point image to obtain a vertical image (Horlmg).

Similarly, the structural elements of the closing operation can be a 1*20 full "1" matrix, and the structural elements of the opening operation can be a 2*1 full "1" matrix;

Then, perform a union set operation on the obtained horizontal image and vertical image to obtain the maximum point set image (lmg) including all subtitle regions. The specific obtaining method is as follows:

Next, the closed operation is performed on the maximum point set image to obtain the subtitle area image.

The structural elements of the closing operation may adopt a 6*6 all "1" matrix, or other matrices.

Step 224, determine the number of subtitles in the subtitle area image and the position information of the subtitle area.

In a specific embodiment, this step may specifically include the following links:

Firstly, distinguish whether the subtitles are arranged horizontally or vertically for each subtitle area in the subtitle area image.

The method of distinction is based on the relative size of the height and width of the subtitle area. Specifically, if the width of the subtitle area is greater than the height, then the subtitles in the subtitle area are arranged horizontally, and if the width of the subtitle area is smaller than the height, then the subtitles in the subtitle area are arranged vertically.

It should be noted that the method for confirming the subtitle area in the subtitle area image may be confirmed by using a labeling method in morphology or other mature methods, which is not limited in this embodiment of the present invention.

For the subtitle area where the subtitles are arranged horizontally, determine the corresponding area of the subtitle area in the horizontal image, and determine the subtitle area through the coordinate positions of the uppermost, lower, left, and right pixel points of the subtitle area in the horizontal image The position of the top border, bottom border, left border, and right border in the horizontal image.

For the subtitle area where the subtitles are vertically arranged, determine the corresponding area of the subtitle area in the vertical image, and use the same method as the subtitle area where the above subtitles are horizontally arranged to obtain the upper border and lower border of the subtitle area in the vertical image , the position of the left border and the right border.

Then, horizontally project the area corresponding to the corresponding integrated sub-band texture map (CS) in the subtitle area positioning frame, and determine the number of subtitles in the integrated sub-band texture map from the peak and valley information of the projection curve of the integrated sub-band texture map and the position of the top and bottom borders of each horizontal subtitle.

Specifically, the number of subtitles in the subtitle area can be determined by the number of troughs in the projection curve, and this process can specifically include:

Divide the texture mean in the integrated subband texture map by a parameter (alfa) to get the threshold. If the value of the projected curve is less than this threshold, it is a trough. Since the position of the trough is the middle position between two subtitles, the number of subtitles in the subtitle area can be determined by determining the number of troughs, that is, the number of troughs plus 1. It should be noted that, in the embodiment of the present invention, the value range of the parameter (alfa) may be [2, 3], after the actual operation test, the recommended parameter alfa=2.6.

In addition, since the positions of the upper and lower borders of the subtitles separated by the trough correspond to the coordinate positions of the top and end of the trough respectively, by determining the position of the trough, the upper border and the lower border of each horizontal subtitle in this subtitle area can be determined. The position of the bottom border.

For vertically arranged subtitles, perform vertical projection on the corresponding integrated subband texture map area in the subtitle area positioning frame, and determine the number of subtitles and the left and right border positions of each vertical subtitle from the peak-valley relationship of the projection curve, Its specific implementation method is the same as that of subtitles arranged horizontally.

Through the above operations, information such as the position where the subtitle appears in the video stream can be determined.

Optionally, in one embodiment, in order to improve the accuracy of detection, it may further include:

Step 225, check whether the subtitle area is a real subtitle area.

Since in the subtitle detection, there may be false detection, and an area that is not a subtitle is detected as a subtitle area, therefore, it is necessary to verify the authenticity of the confirmed subtitle area, which can effectively improve the performance of subtitle detection.

Specifically, it may be determined whether the detection area is a real subtitle area according to the subtitle texture distribution, the gray level distribution, and the distribution of the number of edge points.

When a subtitle area is a real subtitle area, the valleys in the projection on the corresponding integrated subband texture map and the valleys in the projection of the low-frequency component image after wavelet transformation are evenly distributed. The detection method of the trough is the same as that described in step 224, and the uniform measurement method is that the length of the trough does not exceed the peak, and the variance of the trough is small.

Step 23, acquiring the attribute information of the detected subtitles

Specifically, in this step, matching and tracking operations may be performed on the detected subtitles to determine subtitle attribute information.

The subtitle matching operation is to judge whether the subtitles detected match according to the subtitle detection situation of the previous I frame and the current I frame. If it matches, it indicates that the matched subtitles belong to the same subtitle, otherwise they belong to different subtitles.

Whether two adjacent I frames that need to perform subtitle detection need to carry out subtitle matching tracking is to judge according to the subtitle bar number detected in these two frames and according to the following possible situations:

1) If the number of subtitles in the previous I frame and the current I frame is 0, there is no need to perform matching and tracking operations;

2) If the number of subtitles in the previous I frame is 0, and the number of subtitles in the current I frame is not 0, then it can be determined that the number of subtitles in the current I frame is all new subtitles, so matching and tracking operations are required , to determine the start frame of the subtitle in the current I frame.

When judging the start frame, it first needs to be processed according to the subtitle matching situation in the current I frame and the next I frame and the determined subtitle attribute. If there is no subtitle in the next I frame, or there is a subtitle but does not match with the subtitle detected in the current I frame, then the subtitle detected in the current I frame is regarded as wrongly detected and removed, otherwise the new detected subtitle in the current I frame A subtitle bar appears for subtitle tracking.

3) if the subtitle bar quantity of previous I frame is not 0, and the subtitle bar quantity of current I frame is 0, then the subtitle bar of current I frame is disappearance subtitle bar, need to carry out matching and tracking operation so, to determine current I The ending frame of the subtitle in the frame.

4) If the number of subtitles in the previous I frame and the current I frame is not 0, then it is necessary to match and track the subtitles in the previous I frame and the current frame I to determine which subtitles in the previous I frame are matches, which ones are gone, and which subtitles are matches and which ones are new in the current I-frame. For the previous I frame, which I frames that disappeared between the previous I frame and the current I frame need to determine the end frame of the subtitle, and for the new subtitle bar in the current I frame, it needs to be from the previous I frame to the current I frame Frame Between Determines the frame in which the subtitle appears.

Then it can be seen that as long as the number of subtitles in one frame in the previous I frame or the current I frame is not zero, matching and tracking operations are required.

In the embodiment of the present invention, the subtitle matching operation can be realized by means of sampling matching, that is, calculating subtitle p to be matched in the current I frame and any unmatched subtitle q in the next I frame (1≤q≤n ) in sliding matching (MAD: Mean Absolute Difference), and then select the smallest MAD value from n subtitle matching as the best matching subtitle, and further judge whether the minimum MAD meets the minimum constraint threshold.

Specifically, for the subtitle q of the current I frame and the subtitle p of the next I frame, the positions of the upper, lower, left, and right borders where the subtitles are located are respectively U _IC ^q , D _IC ^q , L _IC ^q , R _IC ^q and U _IP ^p , D _IP ^p , L _IP ^p , R _IP ^p .

If the two I frames are arranged horizontally, then extract the subtitle q of the current I frame and the subtitle p of the next I frame, and in the common area in the horizontal direction, the maximum value of the left border $\mathrm{QUR}=\max \{L_{\mathrm{IP}}^p,L_{\mathrm{IC}}^q\},$ and the minimum value of the right border $\mathrm{QUR}=\min \{R_{\mathrm{IP}}^p,R_{\mathrm{IC}}^q\},$ If Rpq-Lpq is less than or equal to the threshold, then consider mismatch (the threshold here can be specifically 10); if greater than the threshold, then extract the center cy ( $\mathrm{cy}=\mathrm{round}[(u_{\mathrm{IP}}^p+{\mathrm{D.}}_{\mathrm{IP}}^p)/2],$ Among them, round[·] represents the pixel IP(cy, Lpq:Rpq) at the place of rounding), and the subtitle q between it and the subtitle q of the current I frame is determined by sliding matching and other methods, and the height of IC(y, Lpq:Rpq) at the height y The matching error MAD(y, q) of the pixel strip, and the best matching position Specifically, it can be calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; MAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; QUR</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; QUR</span>}<span\; class="notranslate">\; )</span>}\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; QUR</span>}}{\overset{\mathrm{<span\; class="notranslate">\; QUR</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; IP</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; cy</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; IC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; IC</span>}}^{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; IC</span>}}^{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; ]</span>$

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; MAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; q</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; MAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

下的MAD(q₀)≤MAD_th，则认为是匹配字幕。本发明实施例中，阈值MADth的较佳取值可以为MAD_th＝20。If in the best match position

If MAD(q 0 )≤MAD th under MAD(q ₀ )≤MAD _th , it is considered as matching subtitles. In the embodiment of the present invention, a preferred value of the threshold MADth may be MAD _th =20.

If they are all arranged vertically, then extract the subtitle q of the current I frame and the subtitle p of the next I frame, and in the common area in the vertical direction, the maximum value of the upper border $\mathrm{Upq}=\max \{u_{\mathrm{IP}}^p,u_{\mathrm{IC}}^q\},$ The minimum value of the border below $\mathrm{DpQ}=\min \{{\mathrm{D.}}_{\mathrm{IP}}^p,{\mathrm{D.}}_{\mathrm{IC}}^q\},$ If Dpq-Upq≤10, then it is considered that it does not match; if it is greater than the threshold, then extract in the common area in the vertical direction, the center cx( $\mathrm{cx}=\mathrm{round}[(L_{\mathrm{IP}}^p+R_{\mathrm{IP}}^p)/2]$ ) at the center pixel IP (Upq: Dpq, cx), determine it and the current I current frame subtitle q by methods such as sliding matching, and the matching error of the pixel strip of IC (Upq: Dpq, x) at the width of x is MAD (x, q), and the best matching position x0, the specific method is similar to the above-mentioned horizontal subtitle, and then select the subtitle corresponding to the minimum MAD value as the best matching, if the best matching position MAD(q ₀ )≤MAD _th is considered as a matching subtitle.

For the matching subtitles, a tracking operation can be performed on them to determine the positions of the start frame and the end frame in the subtitles.

Specifically, subtitles can be classified into two types: static subtitles and rolling subtitles according to the matching speed calculated from the relative position difference of subtitle matching. If the position of the matched subtitle in the two subtitle detection frames remains unchanged, it is judged as a static subtitle, otherwise it is judged as a rolling subtitle.

If it is a scrolling subtitle, according to the matching speed and the position of the scrolling subtitle in the current frame, it is determined that a certain frame before the current frame of the subtitle border just enters the image frame, and a certain frame after the current frame just exceeds the image frame The data frame of the frame number corresponding to the range is used as the appearing frame and the ending frame.

If it is a static subtitle, access the video stream of the group of pictures (GOP: group of pictures: group of pictures) where the previous frame is located, and decode the luminance component image of each frame, and obtain its subtitle area direct current (DC) Image, calculate the average absolute error MAD value of the subtitle area DC image in this GOP, and determine the appearance frame and end frame of the static subtitle according to the MAD value.

In the static subtitle bar tracking in the above steps, the average absolute error of the DC image in the subtitle area within a GOP is realized by extracting the DC lines in this area for matching. details as follows:

Firstly, realize partial decoding of the frame between the previous frame and the current frame and obtain the DC image.

Then, its corresponding coordinate position in the DC image is obtained according to the derived subtitle border position in the current frame, and the DC line at the center block of the area where the subtitle is located in the DC image is extracted.

Next, calculate the difference value of the DC line between the given frame i and the current frame.

The arrangement direction of subtitles should be considered when extracting DC lines. For horizontally arranged subtitles, the DC line difference MADDC(i) between the i-th frame and the current frame can be obtained according to the following formula:

$\mathrm{<span\; class="notranslate">\; MADDC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}\underset{\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; DC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; DCY</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; IC</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; DC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; DCY</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; IP</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; IC</span>}$

Where DC(y, x, i) represents the DC image corresponding to the i-th frame, and dcy represents the center position of the subtitle area in the DC image in the vertical direction.

The calculation method for vertically arranged subtitles is similar to the above method.

As for the judging method of appearing frame or terminating frame, it can be determined by looking for a sudden change point on the MADDC curve. The specific method is shown in the following formula:

Wherein th1 and th2 are the constraint thresholds for judging the mutation point, and the preferred constraint thresholds selected in the embodiment of the present invention are th1=3.5 and th2=9.

If the current frame is the center and the search radius is 2 GOP lengths, no mutation point is found, then the subtitle bar is removed as a wrongly detected subtitle; otherwise, find the data frame closest to the front or back of the current frame, as a present frame or an end frame.

The above formula is to calculate the difference value for horizontally arranged subtitles, and the calculation method for vertically arranged subtitles is obtained by a method similar to the above.

Step 24, extracting the detected subtitles according to the attribute information of the subtitles.

It should be noted that, in the video subtitle information acquisition method provided by the embodiment of the present invention, the acquired subtitle attribute information can be recorded in real time.

The subtitle attribute information may specifically include basic subtitle information, scene information, matching information, and the like.

The basic information may specifically include basic attribute information and detection information of the subtitle;

The scene information may specifically include the start frame and end frame of the subtitle, and whether the subtitle crosses the shot mark, etc.;

The matching information may specifically include a flag indicating whether to match, matching location information, and the like.

Among them, in the embodiment of the present invention, for the method of judging whether to cross a shot, a mature method such as scene change detection can be used in the interval where the recorded data frame before the start frame and the data frame after the end frame are located. The embodiment of the present invention does not limit this.

The subtitle attribute information involved in the embodiment of the present invention can be specifically shown in Table 1:

Table 1

/*Structure, used to describe the properties of the currently active subtitle bar*/typedef struct ActiveTextLine{/*Basic information*/int frameIndex; //Current frame number int textPos[4]; //4-dimensional array of position information, left Boundary, upper boundary, right boundary, lower boundary int rollingFlag; //Rolling and static flags, 0-stationary, 1-vertical scrolling, 2-horizontal scrolling int verVel; //vertical speed downward is positive speed, upward is Negative velocity int horVel; //horizontal velocity to the right is positive velocity, and to the left is negative velocity bool direction; //distribution direction, 0-horizontal distribution, 1-vertical distribution/*scene information*/int startFrame; //start Frame int startGOP; //Start GOPint endFrame; //End frame int duration; //The length of subtitles (number of frames) bool startAbrupt; //Whether there is a sudden change in the start frame, 0-no, 1-bool endAbrupt; //Whether there is a sudden change in the termination frame, 0 no, 1-bool crossScene appears; //subtitle cross-shot flag, 0-no, 1-appear int crossPos[10]; //record the frame number of the subtitle cross-shot/*matching information */bool matchFlag; //Subtitle match flag, 1, a match occurs in the next I frame, 0, no match int matchTextPos[4]; //The position of the matching subtitle}ATL;

In addition, the embodiment of the present invention can also save the subtitle attribute information acquired in real time in the form of text records. The text of record keeping can be shown in Table 2:

Table 2

/*Property record file format of subtitle bar*/TextNumIndex:#n //The nth subtitle in the video startFrame; //Start frame endFrame; //End frame rollingFlag; //Rolling and static flag direction; / /distribution direction, 0-horizontal distribution, 1-vertical distribution textPos[4]; //position information 4-dimensional array, left boundary, upper boundary, right boundary, lower boundary RollingMV[2]; //subtitle rolling speed 2-dimensional array , vertical speed, horizontal speed OCRString; //Recognition results after subtitle segmentation

Then, in this step, according to the recorded subtitle attribute information, including information such as the start frame, end frame and appearance position of the subtitle, the subtitle frame for segmentation is extracted, and then the subtitle segmentation of the fusion of multiple frames is performed, and the Segmentation results are identified, which may specifically include:

From the recorded subtitle attribute information, it is judged whether the subtitle is still or rolling.

For still subtitles, directly extract all I frames and P frames between the start and end frames, and the subtitle area images at the same position;

For rolling subtitles, according to the scrolling speed, all corresponding image areas of the I frame and P frame of the subtitle are extracted.

On the basis of the region determination, the subtitle area part of all I frames in the subtitle continuous frame is firstly segmented by adaptive threshold value binarization to obtain a binary image with pixel values of only 0 and 255; and then all the segmented I frames Subtitle area image, perform "AND operation" on the pixel values at the same position to obtain "I frame and image"; then average the subtitle area images of all I frames and P frames in the subtitle continuous frame for the pixel values at the same position Pixel value, that is, to find an average image of these images, and perform binary segmentation on this average image to obtain the "I-P frame average image"; finally, the obtained two images of "I frame and image" and "I-P frame average image" The effect map obtained by performing "AND operation" is used as the final segmentation result.

As for the segmentation result, in the subtitle recognition process, the text recognition (OCR: Optical Character Recognition) software can be used to recognize the segmented binary image.

It can be seen from the above description that the embodiment of the present invention provides a subtitle information acquisition method, by performing wavelet-based subtitle detection on the luminance component image of the data frame in the video stream, to obtain the attribute information of the detected subtitle, and extract the subtitle information according to the attribute information The detected subtitles are extracted, so as to accurately obtain the subtitle information in the data frame. Since subtitle detection based on wavelet does not need to limit the subtitle area, the subtitle information acquisition method provided by the embodiment of the present invention can acquire subtitle information in video data without limiting the subtitle location area. Moreover, since only the luminance component image of the specified data frame is acquired, the subtitle information acquisition method provided by the embodiment of the present invention can acquire subtitle information more quickly. Moreover, the subtitle information acquisition method provided by the embodiment of the present invention can also verify the authenticity of the subtitle area for the acquired subtitles, as well as match and track operations, so that the subtitle information acquisition method provided by the embodiment of the present invention can obtain more accurately Subtitle information, effectively improving the performance of subtitle detection. In addition, the method for acquiring subtitle information provided by the embodiment of the present invention can also perform segmentation operations on the acquired subtitles, thereby making it more convenient for users to use.

The embodiment of the present invention also provides a subtitle information acquisition device, as shown in FIG. 4 , the device includes a detection module 410 , a first acquisition module 420 and an extraction module 430 . in:

The detection module 410 is configured to perform subtitle detection based on wavelet on the luminance component image of the data frame in the video stream.

The first acquiring module 420 is configured to acquire the attribute information of the subtitles detected by the detecting module 410 .

The subtitle attribute information obtained by the first obtaining module 420 may specifically include basic subtitle information, scene information, matching information, and the like.

The basic information may specifically include basic attribute information and detection information of the subtitle;

The scene information may specifically include the start frame and end frame of the subtitle, and whether the subtitle crosses the shot mark, etc.;

The matching information may specifically include a flag indicating whether to match, matching location information, and the like.

Among them, in the embodiment of the present invention, for the method of judging whether to cross a shot, a mature method such as scene change detection can be used in the interval where the recorded data frame before the start frame and the data frame after the end frame are located. The embodiment of the present invention does not limit this.

The subtitle attribute information involved in the embodiment of the present invention may be specifically shown in Table 1.

In addition, the embodiment of the present invention can also save the subtitle attribute information acquired in real time in the form of text records. The text of the record keeping can be shown in Table 2.

The extraction module 430 is configured to extract the subtitles detected by the detection module 430 according to the subtitle attribute information acquired by the first acquisition module 420 .

In a specific embodiment of the device for obtaining subtitle information provided by the embodiment of the present invention, as shown in FIG. 5 , the device may further include a second obtaining module 440, configured to obtain a luminance component image of a specified data frame.

In order to speed up the speed of obtaining subtitle information, this embodiment of the present invention may specifically decode a specified data frame from a video data stream, and obtain a luminance component image of the specified data frame.

For example, only decode the intra-frame encoding with an odd (or even) frame number, that is, the code stream of the I frame (or other forms of video frames, such as predictive coding frames, that is, the P frame), and obtain the brightness component image of the I frame , while the chrominance component of the I frame and other frames are quickly skipped, thereby speeding up the speed of obtaining subtitle information.

It should be noted that the embodiment of the present invention does not limit the compression format of the video data stream.

The detection module 410 involved in the embodiment of the present invention may be specifically shown in FIG. 6 , including a first acquisition unit 411 , a second acquisition unit 412 , a generation unit 413 , and a determination unit 414 . in:

The first acquisition unit 411 is configured to perform wavelet transform on the luminance component image acquired by the second acquisition module 430 to acquire high-frequency sub-band texture maps in three directions: horizontal, vertical and diagonal.

The wavelet transform involved in the embodiments of the present invention may specifically be HAAR (Haar) wavelet transform, Mexican sombrero wavelet transform, 9-7 wavelet transform, 5-3 wavelet transform, and the like.

Specifically, the first acquiring unit 411 performs wavelet transform on the luminance component image of the selected data frame to acquire a low-frequency subband and high-frequency subbands in three directions: horizontal, vertical, and diagonal, wherein the horizontal The high-frequency sub-band is marked as H, the vertical high-frequency sub-band is marked as V, and the diagonal high-frequency sub-band is marked as D.

Then, calculate the absolute values of the obtained high-frequency subband coefficients in the three directions of horizontal, vertical and diagonal directions to obtain the horizontal high-frequency sub-band texture map, the vertical high-frequency sub-band texture map, and the diagonal high-frequency sub-band texture map. subband texture map.

The first obtaining unit 411 may also combine the obtained three high-frequency sub-band texture maps to obtain a comprehensive high-frequency sub-band texture map (CS).

The value of each point in the integrated high-frequency subband texture image can be obtained by the following formula:

CS(i,j)=CH(i,j)+CV(i,j)+CD(i,j)

The second acquiring unit 412 is configured to acquire the data frame subtitle point image (TextPnt) for the high-frequency sub-band texture maps in the horizontal, vertical and diagonal directions acquired by the first acquiring unit 411.

The second obtaining unit 412 specifically obtains the subtitle point image of the data frame through the following operations:

First, an initial subtitle point image is generated according to the high-frequency subband texture map.

Taking the horizontal high-frequency subband texture map as an example, subtitle point detection is performed on the horizontal high-frequency subband texture map to obtain the initial subtitle point image (MAPH_ORG) of the horizontal high-frequency subband.

Wherein, the value of the horizontal high-frequency sub-band initial subtitle point image at coordinates (i, j) is calculated according to the following formula:

$\mathrm{<span\; class="notranslate">\; MAPH</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; ORG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; CH</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; TH</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; CH</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; TH</span>}\end{array}\right.$

It should be noted that the value "0" represents the background, and the value "1" represents the initial subtitle point. The calculation method of the threshold (TH) in the formula can be as follows:

$\mathrm{<span\; class="notranslate">\; TH</span>}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 50</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; 50</span>}\\ \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; 50</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 18</span>}\\ \mathrm{<span\; class="notranslate">\; 18</span>}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; MH</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 18</span>}\end{array}\right.$

MH in the formula is the average texture intensity in the horizontal high-frequency sub-band texture image.

Then, denoising is performed on the initial subtitle point image of the horizontal high-frequency subband to obtain the final subtitle point image (MAPH) in the horizontal direction.

The denoising processing involved in the embodiments of the present invention may specifically adopt mature technical processing schemes such as overlapping and sliding block filtering, which is not limited in the embodiments of the present invention.

Next, perform similar processing steps on the vertical high-frequency subband texture map and the diagonal high-frequency subband texture map to obtain the vertical subband initial subtitle point image (MAPV_ORG) and the diagonal subband initial subtitle point image (MAPD_ORG) , and de-noise the initial subtitle dot image in the vertical subband and the initial subtitle dot image in the diagonal subband to obtain the final subtitle dot image in the vertical direction (MAPV) and the final subtitle dot image in the diagonal direction (MAPD).

Finally, intersect the final caption point images (MAPH, MAPV, MAPD) in the three directions to obtain the caption point image (TextPnt) of the data frame.

The generating unit 413 is configured to generate a subtitle area image according to the subtitle point image acquired by the second acquiring unit 412 .

The generating unit 413 may specifically generate the subtitle area image through the following operations:

Firstly, the closed operation and the open operation in the horizontal direction are respectively performed on the generated subtitle point image to obtain a horizontal image (Verlmg).

Among them, the structural elements of the closing operation can be a 20*1 full "1" matrix, and the structural elements of the opening operation can be a 1*2 full "1" matrix. Of course, the structural elements used in the closing operation and the opening operation can be based on Flexible arrangements are actually required;

Next, perform vertical closing and opening operations on the subtitle point image to obtain a vertical image (Horlmg).

Similarly, the structural elements of the closing operation can be a 1*20 full "1" matrix, and the structural elements of the opening operation can be a 2*1 full "1" matrix;

Then, perform a union set operation on the obtained horizontal image and vertical image to obtain the maximum point set image (lmg) including all subtitle regions. The specific obtaining method is as follows:

Next, the closed operation is performed on the maximum point set image to obtain the subtitle area image.

The structural elements of the closing operation may adopt a 6*6 all "1" matrix, or other matrices.

The determining unit 414 is configured to determine the number of subtitles and the position information of the subtitle area in the subtitle area image generated by the generating unit 413 .

The determining unit 414 can specifically determine the number of subtitles in the subtitle area image and the position information of the subtitle area through the following operations:

Firstly, distinguish whether the subtitles are arranged horizontally or vertically for each subtitle area in the subtitle area image.

The method of distinction is based on the relative size of the height and width of the subtitle area. Specifically, if the width of the subtitle area is greater than the height, then the subtitles in the subtitle area are arranged horizontally, and if the width of the subtitle area is smaller than the height, then the subtitles in the subtitle area are arranged vertically.

It should be noted that the method for confirming the subtitle area in the subtitle area image may be confirmed by using a labeling method in morphology or other mature methods, which is not limited in this embodiment of the present invention.

For the subtitle area where the subtitles are arranged horizontally, determine the corresponding area of the subtitle area in the horizontal image, and determine the subtitle area through the coordinate positions of the uppermost, lower, left, and right pixel points of the subtitle area in the horizontal image The position of the top border, bottom border, left border, and right border in the horizontal image.

For the subtitle area where the subtitles are vertically arranged, determine the corresponding area of the subtitle area in the vertical image, and use the same method as the subtitle area where the above subtitles are horizontally arranged to obtain the upper border and lower border of the subtitle area in the vertical image , the position of the left border and the right border.

Then, horizontally project the area corresponding to the corresponding integrated sub-band texture map (CS) in the subtitle area positioning frame, and determine the number of subtitles in the integrated sub-band texture map from the peak and valley information of the projection curve of the integrated sub-band texture map and the position of the top and bottom borders of each horizontal subtitle.

Specifically, the number of subtitles in the subtitle area can be determined by the number of troughs in the projection curve, and this process can specifically include:

Divide the texture mean in the integrated subband texture map by a parameter (alfa) to get the threshold. If the value of the projected curve is less than this threshold, it is a trough. Since the position of the trough is the middle position between two subtitles, the number of subtitles in the subtitle area can be determined by determining the number of troughs, that is, the number of troughs plus 1. It should be noted that, in the embodiment of the present invention, the value range of the parameter (alfa) may be [2, 3], after the actual operation test, the recommended parameter alfa=2.6.

In addition, since the positions of the upper and lower borders of the subtitles separated by the trough correspond to the coordinate positions of the top and end of the trough respectively, by determining the position of the trough, the upper border and the lower border of each horizontal subtitle in this subtitle area can be determined. The position of the bottom border.

For vertically arranged subtitles, perform vertical projection on the corresponding integrated subband texture map area in the subtitle area positioning frame, and determine the number of subtitles and the left and right border positions of each vertical subtitle from the peak-valley relationship of the projection curve, Its specific implementation method is the same as that of subtitles arranged horizontally.

Through the above operations, information such as the position where the subtitle appears in the video stream can be determined.

In another specific embodiment of the detection module 410 provided by the embodiment of the present invention, the detection module 410 may further include a detection unit 415 as shown in FIG. for detection of real subtitle regions.

Since in the subtitle detection, there may be false detection, and an area that is not a subtitle is detected as a subtitle area, therefore, it is necessary to verify the authenticity of the confirmed subtitle area, which can effectively improve the performance of subtitle detection.

Specifically, it may be determined whether the detection area is a real subtitle area according to the subtitle texture distribution, the gray level distribution, and the distribution of the number of edge points.

When a subtitle area is a real subtitle area, the valleys in the projection on the corresponding integrated subband texture map and the valleys in the projection of the low-frequency component image after wavelet transformation are evenly distributed. The measure of uniformity is that the length of the trough does not exceed the size of the peak, and the variance of the trough is small.

The first obtaining module 420 provided by the embodiment of the present invention may specifically be shown in FIG. 8 , including a judging unit 421 , a first determining unit 422 and a second determining unit 423 . in:

The judging unit 421 is configured to judge whether the current I frame where the subtitle detected by the detection module 410 is located matches the previous I frame of the current I frame.

The conditions for the judging unit 421 to judge specifically may include: whether the number of subtitles in the previous I frame and the current I frame are both zero.

If the number of subtitles in one I frame in the previous I frame and the current I frame is not zero, the judging unit 421 needs to perform a judging operation on whether they match.

It should be noted that the determination conditions of the determination unit 421 are not limited to the above conditions, and may be supplemented and adjusted according to the needs of practical applications.

The judging unit 421 can judge whether the current I frame where the subtitle detected by the detection module 410 is located matches with the previous I frame of the current I frame through a sampling matching method.

That is, calculate the minimum mean absolute error (MAD: Mean Absolute Difference) between the subtitle p to be matched in the current I frame and any unmatched subtitle q (1≤q≤n) in the next I frame in sliding matching, and then from n Select the subtitle with the smallest MAD value in the subtitle matching as the best matching subtitle, and further judge whether the minimum MAD meets the minimum constraint threshold.

Specifically, for the subtitle q of the current I frame and the subtitle p of the next I frame, the positions of the upper, lower, left, and right borders where the subtitles are located are respectively U _IC ^q , D _IC ^q , L _IC ^q , R _IC ^q and U _IP ^p , D _IP ^p , L _IP ^p , R _IP ^p .

具体可通过如下公式计算获取：If the two I frames are arranged horizontally, then extract the subtitle q of the current I frame and the subtitle p of the next I frame, and in the common area in the horizontal direction, the maximum value of the left border $\mathrm{QUR}=\max \{L_{\mathrm{IP}}^p,L_{\mathrm{IC}}^q\},$ and the minimum value of the right border $\mathrm{QUR}=\min \{R_{\mathrm{IP}}^p,R_{\mathrm{IC}}^q\},$ If Rpq-Lpq is less than or equal to the threshold, then consider mismatch (the threshold here can be specifically 10); if greater than the threshold, then extract the center cy ( $\mathrm{cy}=\mathrm{round}[(u_{\mathrm{IP}}^p+{\mathrm{D.}}_{\mathrm{IP}}^p)/2],$ Among them, round[·] represents the pixel IP(cy, Lpq:Rpq) at the place of rounding), and the subtitle q between it and the subtitle q of the current I frame is determined by sliding matching and other methods, and the height of IC(y, Lpq:Rpq) at the height y The matching error MAD(y, q) of the pixel strip, and the best matching position

Specifically, it can be calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; MAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; QUR</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; QUR</span>}<span\; class="notranslate">\; )</span>}\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; QUR</span>}}{\overset{\mathrm{<span\; class="notranslate">\; QUR</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; IP</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; cy</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; IC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; IC</span>}}^{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; IC</span>}}^{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; ]</span>$

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; MAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

下的MAD(q₀)≤MA_th，则认为是匹配字幕。本发明实施例中，阈值MADth的较佳取值可以为MAD_th＝20。 $q_0=\arg \underset{q}{\min }\left\{\mathrm{MAD}\left(q\right)\right\}$ If in the best match position

If MAD(q 0 )≤MA th under MAD(q ₀ )≤MA _th , it is considered as a matching subtitle. In the embodiment of the present invention, a preferred value of the threshold MADth may be MAD _th =20.

的MAD(q₀)≤MAD_th则认为是匹配字幕。If they are all arranged vertically, then extract the subtitle q of the current I frame and the subtitle p of the next I frame, and in the common area in the vertical direction, the maximum value of the upper border $\mathrm{Upq}=\max \{u_{\mathrm{IP}}^p,u_{\mathrm{IC}}^q\},$ The minimum value of the border below $\mathrm{DpQ}=\min \{{\mathrm{D.}}_{\mathrm{IP}}^p,{\mathrm{D.}}_{\mathrm{IC}}^q\},$ If Dpq-Upq≤10, then it is considered that it does not match; if it is greater than the threshold, then extract in the common area in the vertical direction, the center cx( $\mathrm{cx}=\mathrm{round}[(L_{\mathrm{IP}}^p+R_{\mathrm{IP}}^p)/2]$ ) at the center pixel IP (Upq: Dpq, cx), determine it and the current I current frame subtitle q by methods such as sliding matching, and the matching error of the pixel strip of IC (Upq: Dpq, x) at the width of x is MAD (x, q), and the best matching position x0, the specific method is similar to the above-mentioned horizontal subtitle, and then select the subtitle corresponding to the minimum MAD value as the best matching, if the best matching position

MAD(q ₀ )≤MAD _th is considered as a matching subtitle.

After determining the match, the judging unit triggers the first determining unit 422 .

The first determination unit 422 is configured to determine whether the detected subtitles are dynamic subtitles or static subtitles according to the matching speed calculated according to the relative position difference of subtitle matching when the judging result of the judging unit 421 is a match.

Specifically, the first determination unit 422 may classify subtitles into two types: static subtitles and rolling subtitles according to the matching speed calculated from the relative position difference of subtitle matching.

If the positions of the matching subtitles in the two data frames performing subtitle detection are unchanged, it is judged as a static subtitle, otherwise it is judged as a rolling subtitle.

The second determination unit 423 is used to determine the start frame and end frame of the dynamic subtitle according to the matching speed of the dynamic subtitle and the position of the current frame in the dynamic subtitle when the first determination unit 422 determines that the subtitle is a dynamic subtitle; When the first determining unit 422 determines that the subtitle is a static subtitle, it extracts DC lines in the static subtitle, performs a matching operation on the DC lines, and determines a start frame and an end frame of the static subtitle.

If it is a scrolling subtitle, the second determining unit 423 determines that the subtitle border just enters the image frame in a certain frame before the current frame, and in a certain frame after the current frame according to the matching speed and the position of the scrolling subtitle in the current frame. The data frame whose frame is just beyond the frame number corresponding to the image frame range is used as the appearing frame and the ending frame.

If it is a static subtitle, the second determining unit 423 accesses the video stream of the group of pictures (GOP: group of pictures: group of pictures) where the previous frame is located, and decodes the luminance component image of each frame, and obtains its subtitle at the same time Regional direct current (DC) image, calculate the average absolute error MAD value of the subtitle area DC image in this GOP, and determine the appearance frame and end frame of the static subtitle according to the MAD value.

In the static subtitle bar tracking in the above steps, the average absolute error of the DC image in the subtitle area within a GOP is realized by extracting the DC lines in this area for matching. details as follows:

Firstly, realize partial decoding of the frame between the previous frame and the current frame and obtain the DC image.

Then, its corresponding coordinate position in the DC image is obtained according to the derived subtitle border position in the current frame, and the DC line at the center block of the area where the subtitle is located in the DC image is extracted.

Next, calculate the difference value of the DC line between the given frame i and the current frame.

The arrangement direction of subtitles should be considered when extracting DC lines. For horizontally arranged subtitles, the DC line difference MADDC(i) between the i-th frame and the current frame can be obtained according to the following formula:

$\mathrm{<span\; class="notranslate">\; MADDC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}\underset{\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; DC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; DCY</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; IC</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; DC</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; DCY</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; dcx</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; IP</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; IC</span>}$

Where DC(y, x, i) represents the DC image corresponding to the i-th frame, and dcy represents the center position of the subtitle area in the DC image in the vertical direction.

The calculation method for vertically arranged subtitles is similar to the above method.

As for the judging method of appearing frame or terminating frame, it can be determined by looking for a sudden change point on the MADDC curve. The specific method is shown in the following formula:

Wherein th1 and th2 are the constraint thresholds for judging the mutation point, and the preferred constraint thresholds selected in the embodiment of the present invention are th1=3.5 and th2=9.

If the current frame is the center and the search radius is 2 GOP lengths, no mutation point is found, then the subtitle bar is removed as a wrongly detected subtitle; otherwise, find the data frame closest to the front or back of the current frame, as a present frame or an end frame.

The above formula is to calculate the difference value for horizontally arranged subtitles, and the calculation method for vertically arranged subtitles is obtained by a method similar to the above.

The extraction module 430 provided in the real-time example of the present invention may be specifically shown in FIG. 9 , including an extraction unit 431 , a segmentation unit 432 and an identification unit 433 . in:

The extracting unit 431 is configured to extract the subtitle frame for segmentation in the subtitle according to the start frame, end frame and appearance position information of the subtitle.

The segmentation unit 432 is configured to determine the subtitle area corresponding to the subtitle frame extracted by the extraction unit 431, and perform binary segmentation on the subtitle area to obtain a binary image.

Specifically, the segmentation unit 432 extracts subtitle frames for segmentation according to the recorded subtitle attribute information, including information such as the start frame, end frame, and appearance position of the subtitle, and then performs subtitle segmentation for merging multiple frames, and performs segmentation Identify the results, which may include:

From the recorded subtitle attribute information, it is judged whether the subtitle is still or rolling.

For still subtitles, directly extract all I frames and P frames between the start and end frames, and the subtitle area images at the same position;

For rolling subtitles, according to the scrolling speed, all corresponding image areas of the I frame and P frame of the subtitle are extracted.

On the basis of the region determination, the subtitle area part of all I frames in the subtitle continuous frame is firstly segmented by adaptive threshold value binarization to obtain a binary image with pixel values of only 0 and 255; and then all the segmented I frames Subtitle area image, perform "AND operation" on the pixel values at the same position to obtain "I frame and image"; then average the subtitle area images of all I frames and P frames in the subtitle continuous frame for the pixel values at the same position Pixel value, that is, to find an average image of these images, and perform binary segmentation on this average image to obtain the "I-P frame average image"; finally, the obtained two images of "I frame and image" and "I-P frame average image" The effect map obtained by performing "AND operation" is used as the final segmentation result.

The identification unit 433 is configured to identify the binary image obtained by the segmentation unit 432 and extract subtitles.

Specifically, the recognition unit 433 may use OCR (Optical Character Recognition) software to recognize the segmented binary image and extract subtitles therein.

It can be seen from the above description that the embodiment of the present invention provides a device for obtaining subtitle information, which detects subtitles based on wavelets on the luminance component images of data frames in video streams, and performs matching and tracking operations on the detected subtitles, thereby accurately determining Subtitle information for this data frame. Since subtitle detection based on wavelet does not need to limit the subtitle area, the subtitle information acquisition device provided by the embodiment of the present invention can acquire subtitle information in video data without limiting the subtitle location area. Moreover, since only the luminance component images of part of the specified data frame are acquired, and the acquired subtitles are verified for the authenticity of the subtitle area, as well as matching and tracking operations, the subtitle information acquisition device provided by the embodiment of the present invention can be faster and more accurate Acquiring subtitle information can effectively improve the performance of subtitle detection. In addition, the device for obtaining subtitle information provided by the embodiment of the present invention can also perform segmentation operations on the obtained subtitles, thereby making it more convenient for users to use.

It should be noted that the formulas or numerical values involved in the above-mentioned embodiments of the present invention have no limiting influence on the scope of protection of the embodiments of the present invention. When other wavelet transform and matching tracking techniques are used, corresponding transformations can be carried out completely. .

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be realized by means of software plus a necessary hardware platform, and of course all can be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, all or part of the contribution made by the technical solution of the present invention to the background technology can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic disks, optical disks, etc. , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present invention.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present invention can easily think of changes or Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (11)

1. A method for obtaining video subtitle information, characterized in that, comprising: Carry out wavelet transform to the brightness component image of data frame, generate horizontal subband, vertical subband and diagonal subband, calculate the absolute value of the coefficient of described horizontal subband, vertical subband and diagonal subband respectively, obtain Horizontal high frequency subband texture map, vertical high frequency subband texture map and diagonal high frequency subband texture map; Subtitle point detection is performed on the horizontal high frequency subband texture map, vertical high frequency subband texture map and diagonal high frequency subband texture map to generate initial subtitle point images in three directions: horizontal, vertical and diagonal , respectively performing denoising processing on the initial subtitle point images in the three directions to obtain the final subtitle point images in the three directions, and intersecting the final subtitle point images in the three directions to obtain the subtitle points in the data frame image; generating a subtitle area image from the subtitle point image of the data frame; Determine the number of subtitles in the subtitle area image and the position information of the subtitle area; Acquiring the attribute information of the detected subtitle, the attribute information including: basic information of the subtitle, scene information, and matching information; Based on the attribute information, the detected subtitles are extracted. 2 . The method according to claim 1 , further comprising: obtaining the luminance component image of the specified data frame before performing wavelet-based subtitle detection on the luminance component image of the data frame. 3 . 3. method according to claim 1, is characterized in that, described generating subtitle area image by the subtitle point image of described data frame comprises: Performing horizontal closing and opening operations on the subtitle point image to obtain a horizontal image, and performing vertical closing and opening operations on the subtitle point image to obtain a vertical image; Perform a union operation on the obtained horizontal image and vertical image to obtain the largest point set image including all subtitle regions; Perform a closing operation on the maximum point set image to obtain a subtitle area image. 4. method according to claim 3, is characterized in that, described determining the number of subtitles in the subtitle area image and subtitle area position information comprises: Distinguishing between a horizontal subtitle area and a vertical subtitle area in the subtitle area image; By the coordinate positions of the uppermost, lower, left and right pixel points of the horizontal subtitle area in the horizontal image, determine the position of the upper border, lower border, left border, and right border of the positioning frame in the horizontal image of the horizontal subtitle area ;By the coordinate positions of the uppermost, lower, left and right pixel points of the vertical subtitle area in the vertical image, determine the upper border, lower border, left border, and right border of the positioning frame in the vertical image of the vertical subtitle area Location; Perform horizontal projection and vertical projection respectively in the area corresponding to the integrated high-frequency sub-band texture map in the horizontal subtitle area positioning frame and the corresponding integrated high-frequency sub-band texture map in the vertical subtitle area positioning frame Projecting, determining the peak and valley information of the projection curve, and determining the number of subtitles in the subtitle area and the positions of the upper and lower borders of the subtitles according to the peak and valley information. 5. The method according to any one of claims 1 to 4, wherein the attribute information of the subtitle includes a start frame and an end frame of the subtitle, as well as appearance position information. 6. The method according to claim 5, wherein the acquisition of the detected subtitle start frame and end frame comprises: Judging whether the current I frame where the subtitles detected matches with the last I frame of the current frame; If they match, then determine that the subtitles are dynamic subtitles or static subtitles according to the calculated matching speed according to the relative position difference of subtitle matching; If the subtitle is a dynamic subtitle, then determine the start frame and end frame of the dynamic subtitle according to the matching speed of the dynamic subtitle and the position of the current frame in the dynamic subtitle; If the subtitle is a static subtitle, extracting DC lines in the static subtitle, performing a matching operation on the DC lines, and determining a start frame and an end frame of the static subtitle. 7. The method according to claim 1, wherein said extracting detected subtitles according to said attribute information comprises: Extracting subtitle frames for segmentation in the subtitles according to the start frame, end frame and occurrence position information of the subtitles; determining the subtitle area corresponding to the extracted subtitle frame, and performing binary segmentation on the subtitle area to obtain a binary image; Identify the binary image to obtain the subtitle. 8. A video subtitle information acquisition device, characterized in that, comprising: The detection module is used to carry out subtitle detection based on wavelet to the luminance component image of the data frame in the video stream; A first acquisition module, configured to acquire attribute information of the subtitles detected by the detection module; An extraction module, configured to extract the subtitles detected by the detection module according to the subtitle attribute information obtained by the first acquisition module: The detection module includes: The first acquisition unit performs wavelet transform on the brightness component image of the data frame, and acquires the horizontal high-frequency sub-band texture map, the vertical high-frequency sub-band texture map and the diagonal high-frequency sub-band texture map; The second acquisition unit is configured to acquire the subtitle point image of the data frame according to the horizontal, vertical and diagonal high-frequency sub-band texture maps acquired by the first acquisition unit; a generation unit, configured to generate a subtitle area image according to the subtitle point image of the data frame acquired by the second acquisition unit; A determining unit, configured to determine the number of subtitles in the subtitle area image generated by the generating unit and the position information of the subtitle area; The first acquisition module includes: A judging unit, configured to judge whether the current I frame where the subtitle detected by the detection module is matched with the previous frame of the current frame; The first determining unit is used to determine whether the subtitles are dynamic subtitles or static subtitles according to the matching speed calculated according to the relative position difference of subtitle matching when the judgment result of the judging unit is a match; The second determination unit is used to determine the start frame and end frame of the dynamic subtitle according to the matching speed of the dynamic subtitle and the position of the current frame in the dynamic subtitle when the subtitle is a dynamic subtitle; When the subtitle is a static subtitle, extract the DC lines in the static subtitle, and perform a matching operation on the DC lines to determine the start frame and end frame of the static subtitle. 9. The device according to claim 8, further comprising: The second obtaining module is used to obtain the brightness component image of the specified data frame. 10. The device according to claim 9, wherein the detection module further comprises: A detecting unit, configured to detect whether the subtitle area determined by the determining unit is a real subtitle area. 11. The device according to claim 8, wherein the extraction module comprises: An extracting unit, configured to extract a subtitle frame for segmentation in the subtitle according to the start frame, end frame and occurrence position information of the subtitle; A segmentation unit, configured to determine the subtitle area corresponding to the subtitle frame extracted by the extraction unit, and perform binary segmentation on the subtitle area to obtain a binary image; The identification unit is used to identify the binary image obtained by the segmentation unit, and extract the subtitle.

Images