CN107145888A - Video caption real time translating method - Google Patents
Video caption real time translating method Download PDFInfo
- Publication number
- CN107145888A CN107145888A CN201710345936.1A CN201710345936A CN107145888A CN 107145888 A CN107145888 A CN 107145888A CN 201710345936 A CN201710345936 A CN 201710345936A CN 107145888 A CN107145888 A CN 107145888A
- Authority
- CN
- China
- Prior art keywords
- mrow
- text
- mser
- msub
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000013519 translation Methods 0.000 claims abstract description 39
- 238000001914 filtration Methods 0.000 claims abstract description 26
- 238000000605 extraction Methods 0.000 claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 14
- 238000012216 screening Methods 0.000 claims description 19
- 238000012706 support-vector machine Methods 0.000 claims description 15
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 3
- 230000004069 differentiation Effects 0.000 abstract 1
- 238000001514 detection method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000007635 classification algorithm Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/635—Overlay text, e.g. embedded captions in a TV program
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/58—Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/158—Segmentation of character regions using character size, text spacings or pitch estimation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Image Analysis (AREA)
Abstract
The present invention provides a kind of video caption real time translating method, including：Multichannel extraction is carried out to the original image intercepted from video, multiple single channel images are obtained；Based on MSER algorithms, the MSER regions of original image and multiple single channel images are extracted respectively；The local contrast between each MSER region and its background area is calculated, and according to each local contrast, it is determined whether corresponding MSER regions are filtered out；Determine the border key point in each MSER region；Using border key point as category filter feature, to filtering out each rear remaining MSER region by the SVM progress category filters trained, obtain text filed；According on vertical direction per two neighboring the distance between text filed, to each text filed progress line of text differentiation, according to per two neighboring the distance between text filed, text filed classifying to one text row each on one text row；Based on each text filed progress video caption real time translation after classification.
Description
Technical Field
The invention belongs to the field of image processing, and particularly relates to a real-time translation method for video subtitles.
Background
In recent years, text detection and recognition in images of natural scenes has been a subject of intense research in the fields of computer vision, pattern recognition and even document analysis. Researchers have proposed a number of new ideas and methods for extracting textual information from images of natural scenes. However, when translating video subtitles at present, because the time complexity of extracting text information from an image is high, real-time translation of the video subtitles cannot be realized.
Disclosure of Invention
The invention provides a real-time translation method for video subtitles, which aims to solve the problem that the real-time translation of the video subtitles cannot be realized due to higher time complexity of extracting text information from an image when the video subtitles are translated at present.
According to a first aspect of the embodiments of the present invention, a method for real-time translating video subtitles is provided, which includes:
performing multi-channel extraction on an original image intercepted from a video to obtain a plurality of single-channel images;
respectively extracting MSER areas of the original image and the single-channel images based on a MSER algorithm of a maximum stable extremum area;
introducing local contrast text features, calculating the local contrast between each MSER region and the background region thereof, and determining whether to filter the corresponding MSER region according to each local contrast;
introducing text features of the boundary key points, and determining the boundary key points of each MSER region;
classifying and screening all MSER regions left after filtering by using the boundary key points as classification and screening characteristics through a trained Support Vector Machine (SVM) to obtain text regions;
according to the distance between every two adjacent text regions in the vertical direction, distinguishing each text region, and according to the distance between every two adjacent text regions in the same text line, classifying each text region in the same text line;
and performing real-time translation on the video subtitles based on the classified text regions.
In an optional implementation manner, before performing multi-channel extraction on an original image intercepted from a video to obtain a plurality of single-channel images, the method further includes: preprocessing the original image including sharpening and blurring.
In another optional implementation manner, the performing multi-channel extraction on the original image captured from the video to obtain a plurality of single-channel images includes: and respectively carrying out R, G, B, H, S, V six-channel image extraction on the original image and the preprocessed original image so as to obtain a plurality of single-channel images.
In another optional implementation manner, the calculating a local contrast between each MSER region and its background region, and determining whether to filter out the corresponding MSER region according to each local contrast includes:
the local contrast lc between each MSER region and its background is calculated according to the following formula:
wherein n represents the number of pixels corresponding to the MSER region, k represents the number of pixels corresponding to the background region, and Ri、Gi、BiRespectively representing the values of red, green and blue of three channels of the image corresponding to the MSER area, wherein i represents the ith pixel point corresponding to the MSER area, and j represents the jth pixel point corresponding to the background area;
and for each MSER area, if the local contrast of the MSER area is less than a first preset threshold, filtering the MSER area.
In another optional implementation manner, the determining the boundary key point of each MSER region includes:
aiming at each MSER region, setting the gray value of the pixel point of the MSER detected in the MSER region as 255 and setting the gray values of other pixel points as 0;
successively traversing each pixel point in the MSER area, and if the gray value of the pixel point is 255 and the gray value of at least one of the adjacent pixel points is 0, determining the pixel point as a contour point;
after all contour points of at least one MSER area are obtained, each contour point is compressed by adopting a Douglas-Pock algorithm, redundant points are removed, and boundary key points corresponding to the MSER area are obtained.
In another optional implementation manner, the aspect ratio, the area perimeter ratio, the convex hull area ratio and the stroke width area ratio of each remaining MSER region after filtering are used as classification screening features, and classification screening is performed on each remaining MSER region after filtering through a trained SVM.
In another alternative implementation manner, in the process of training the SVM, the number ratio of the positive samples to the negative samples is controlled to be 1: and 3, the positive sample is a letter and an Arabic number corresponding to the translation target language, and the negative sample is a non-text region which is obtained by manually identifying and marking the extracted MSER region after the MSER regions of the original image and the multiple single-channel images are respectively extracted.
In another optional implementation manner, the performing text line distinction on each text region according to a distance between every two adjacent text regions in the vertical direction includes:
the distance d between every two adjacent text areas in the vertical direction is calculated according to the following formulav：
Wherein, b1Y-axis coordinate, t, representing the bottom of a vertically adjacent upper text region2Representing top Y-axis coordinates, h, of vertically adjacent lower text regions2Representing a height of a lower text region adjacent in a vertical direction;
for every two adjacent text regions in the vertical direction, if the distance d between the two adjacent text regionsvIf the number of the adjacent text areas is larger than the second preset threshold value, the two adjacent text areas are classified into the same text line, otherwise, the two adjacent text areas are classified into different text lines.
In another optional implementation manner, the classifying the text regions of the same text line according to the distance between every two adjacent text regions on the same text line includes:
the distance d between every two adjacent text regions of the same text line is calculated according to the following formulah：
Wherein,represents the average value of all letter widths of the text line, and deltad represents the X-axis direction of adjacent letters in two adjacent text areasDifference in upward distance;
for every two adjacent text regions of the same text line, if the distance d between the two adjacent text regionshIf the number of the adjacent text areas is larger than the third preset threshold, the two adjacent text areas are classified into one type, otherwise, the two adjacent text areas are classified into a different type.
In another alternative implementation, when an original image is cut out from a video, a video picture is cut out according to frames, and two thirds of the area below the cut-out video image is used as the original image.
The invention has the beneficial effects that:
1. before the text is identified, a plurality of single-channel images are introduced, color information of an original image is effectively utilized, richer basic data are provided for text region extraction, then local contrast is introduced, threshold filtering is carried out on MSER regions extracted from the original image and the single-channel images, the accuracy of text region extraction can be improved, the time complexity of local contrast filtering is linear time, the filtering time is short, a foundation can be provided for real-time translation of video subtitles, boundary key points are introduced to serve as SVM classification screening features, non-text region interference in the MSER regions can be eliminated even when the images are rotated and scaled, the sensitivity of image rotation and scaling of text region extraction can be improved, after the MSER regions are subjected to threshold filtering based on the local contrast, the remaining MSER regions after filtering are screened by an SVM classifier, the accuracy of text region extraction can be improved, and aiming at a text region obtained after training and screening are finished, the invention adopts two layers of text classification algorithms of text line classification in the vertical direction and text region classification in the same text line in the horizontal direction, thereby greatly reducing time complexity, improving word recognition rate and providing a basis for realizing real-time translation of video captions, and therefore, the invention can realize real-time and accurate translation of the video captions;
2. according to the method, the original image is subjected to sharpening pretreatment, so that the sharpened original image can enhance the contrast between the text area and the surrounding background, and the text detection is facilitated;
3. according to the method, the quantity ratio of the positive samples to the negative samples is controlled to be 1:3 during SVM training, so that the screening effect can be optimized, and the accuracy of text region acquisition is further improved;
4. according to the invention, when the original image is intercepted from the video, the video image is intercepted according to the frame, and then the partial area of the video image is intercepted to be used as the original image, so that the identification precision can be improved, and the detection time can be reduced.
Drawings
FIG. 1 is a flow chart of an embodiment of a real-time video caption translation method according to the present invention;
FIG. 2 is a diagram of a Laplace operation template according to the present invention;
FIG. 3 is a schematic diagram of boundary keypoints;
FIG. 4 is a text line constraint parameter specification diagram;
FIG. 5 is a letter width to space ratio statistical chart.
Detailed Description
In order to make the technical solutions in the embodiments of the present invention better understood and make the above objects, features and advantages of the embodiments of the present invention more comprehensible, the technical solutions in the embodiments of the present invention are described in further detail below with reference to the accompanying drawings.
In the description of the present invention, unless otherwise specified and limited, it is to be noted that the term "connected" is to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, or a communication between two elements, or may be a direct connection or an indirect connection through an intermediate medium, and a specific meaning of the term may be understood by those skilled in the art according to specific situations.
Referring to fig. 1, a flowchart of an embodiment of a video subtitle real-time translation method according to the present invention is shown. The video subtitle real-time translation method can comprise the following steps:
and S101, performing multi-channel extraction on the original image intercepted from the video to obtain a plurality of single-channel images.
In this embodiment, the video resources may be divided into two types: one is local video which can be played off-line, and the other is online video which needs to be played on the internet. Aiming at a local video, corresponding software can be provided for a user, the software can comprise an offline translation database, when the user establishes connection between the local video and the software, the software can perform text recognition on subtitles in the local video according to the method in the patent, after the text recognition is completed, the software can automatically translate the recognized text by adopting the offline translation database, and the translation result is returned and transmitted to the local video for display; aiming at the online video, corresponding software can be provided for a user, a Web server can also be set up to provide Web online service for the user, when the user establishes a link between the online video and the Web server, the Web server can perform text recognition on subtitles in the online video according to the method in the patent, and after the text recognition is completed, the Web server can translate the recognized text and return the translation result to the online video for display.
In order to implement a real-time translation function of a video, a video image may be cut out by frames when an original image is cut out from the video, and in order to improve recognition accuracy and reduce detection time, the cut-out video image may be subjected to region cutting, for example, only the bottom two thirds of the region of the image is cut out as the original image.
After the original image is cut from the video, the original image may first be pre-processed, including sharpening and blurring. When the original image is subjected to sharpening preprocessing, sharpening processing can be performed according to formula (1):
g(x,y)＝f(x,y)+c[▽2f(x,y)](1)
wherein, g (x, y) and f (x, y) respectively represent the original image and the original image after the sharpening pretreatment, the value of c depends on the template used for the sharpening, and if the template shown in fig. 2(a) or fig. 2(b) is used, c is-1; if two templates shown in fig. 2(c) are used, c is 1. The sharpened original image can enhance the contrast of the text region and the surrounding background thereof, and is more favorable for text detection.
When the original image is subjected to blur preprocessing, the original image may be subjected to gaussian filtering according to equation (2):
wherein f (x) represents the original image after fuzzy preprocessing, mu represents the mean value of random variables following normal distribution, and sigma represents the average value of random variables2Representing the variance of the random variable x. According to the invention, the original image is subjected to fuzzy preprocessing, so that a text area under a complex background can be more prominent, and text detection is more facilitated.
After the original image is preprocessed, image extraction of six channels of R (red), G (green), B (blue), H (hue), S (saturation), and V (brightness) may be performed on the original image and the preprocessed original image, so as to obtain a plurality of single-channel images. The invention can effectively utilize the color information by extracting the multi-channel image, and provides richer basic data for extracting the text regions, thereby ensuring that the subtitles recognized and translated based on the text regions are more accurate.
And S102, respectively extracting MSER areas of the original image and the single-channel images based on the MSER algorithm.
In this embodiment, in order to accelerate the extraction rate, the following settings are made for the parameters involved in the MSER algorithm: the threshold was set at 5 steps, with a minimum MSER area of 80 and a maximum MSER area of 14400. Since the MSER algorithm is an image extraction method known in the art, detailed description of a specific extraction process of the MSER algorithm is omitted here.
Step S103, local contrast text features are introduced, the local contrast between each MSER region and the background region of each MSER region is calculated, and whether the corresponding MSER region is filtered or not is determined according to each local contrast.
In this embodiment, the MSER regions extracted in step S102 are not all text regions, and the applicant has found that the text to be identified must have a certain contrast with its background, and the contrast of the text region and its background region, and the contrast of the non-text region and its background region are not the same, and the contrast of the former is higher than that of the latter. Based on the characteristic, the invention introduces the characteristic of local contrast to filter out non-text areas. First, the local contrast lc between each MSER region and its background region can be calculated using the following equation (3):
wherein n represents the number of pixels corresponding to the MSER region, k represents the number of pixels corresponding to the background region, and Ri、Gi、BiThe values of three channels of the image corresponding to the MSER area are respectively represented, i represents the ith pixel point corresponding to the MSER area, and j represents the jth pixel point corresponding to the background area.
Then, whether to filter out the corresponding MSER regions may be determined according to the local contrast of each MSER region, where for each MSER region, if the local contrast of the MSER region is smaller than a first preset threshold, the MSER region is filtered out, otherwise, the MSER region is not filtered out. The applicant has found that the local contrast lc of the non-text region is generally less than 0.35, i.e. the first preset threshold may be 0.35. Although the multi-channel image extraction is carried out on the image in the step S101, rich data basis can be provided for more obtained text regions, more non-text regions are introduced at the same time, and partial non-text region interference items in the MSER region can be eliminated by adopting the local contrast to filter the MSER region, so that the text region extraction accuracy can be improved. In addition, the time complexity of filtering the non-text region according to the local contrast is linear time, so the time required by filtering is short, and a foundation can be provided for real-time translation of the video subtitles.
And step S104, introducing the text characteristics of the boundary key points, and determining the boundary key points of each MSER area.
In this embodiment, when determining the key point of the boundary of each MSER region, first, image binarization is performed on each MSER region, where for each MSER region, the gray value of the pixel point of the MSER detected in the MSER region is set to 255, and the gray values of other pixel points are set to 0. And then, successively traversing each pixel point in the MSER area, if the gray value of the pixel point is 255 and the gray value of at least one of the adjacent pixel points is 0, determining the pixel point as a contour point, wherein the pixel points in the MSER area can be successively traversed from top to bottom in a left-to-right sequence, if the gray value p (X, Y) of the pixel point is 255 and the gray value p (X +1, Y) of the adjacent pixel point on the right side, the gray value p (X-1, Y) of the adjacent pixel point on the left side, the gray value p (X, Y +1) of the adjacent pixel point on the upper side and a value p (X, Y-1) of the adjacent pixel point on the lower side are 0, determining the pixel point as the contour point, wherein X represents the X-axis coordinate of the pixel point and Y represents the Y-axis coordinate of the pixel point.
After all contour points of at least one MSER region are obtained, each contour point is compressed by adopting a Douglas-Pock algorithm, redundant points are removed, and boundary key points corresponding to the MSER region are obtained, as shown in FIG. 3. After all contour points of one MSER area are obtained, each contour point is compressed by adopting a Douglas-Pock algorithm, redundant points are removed, and boundary key points (namely contour points left after the redundant points are removed) of the MSER area are obtained; or after all contour points of a preset number of MSER regions are obtained, compressing each contour point by adopting a Douglas-Pock algorithm, removing redundant points and obtaining boundary key points of the preset number of MSER regions; or after all contour points of all MSER areas can be obtained, each contour point is compressed by adopting a Douglas-Pock algorithm, redundant points are removed, and boundary key points of all MSER areas are obtained. The applicant researches and discovers that the number of the boundary key points k of the english letters is generally between 5 and 16, namely, the preset numerical range is 5 to 16, and when the number of the boundary key points k of the MSER region is less than 5 or more than 16, the MSER region can be determined to be a non-text region when translating english.
And S105, taking the boundary key points as classification screening characteristics, and performing classification screening on each MSER region left after filtering through the trained SVM to obtain a text region.
In this embodiment, after the threshold filtering in step S103 is completed, the present invention selects the width-to-height ratio (w/h) and the area-to-perimeter ratio (MSER-to-perimeter ratio) of each remaining MSER region after filtering, in addition to using the boundary key points as the classification and screening featuresConvex hull area ratio (a)cA), stroke width area ratio (w)sA) as a classifying and screening feature, thereby obtaining a text region, wherein w represents the width of the MSER region, h represents the height of the MSER region,denotes the square of the area of the MSER region, p denotes the perimeter of the MSER region, acDenotes the area of the convex hull (the convex hull is a common term in image processing and is not described herein), a denotes the area of the MSER region, wsRepresentation diagramThe stroke width of the image. In order to optimize the screening effect, the training parameters may be set as follows: the kernel function adopts a radial basis function RBF, and the iteration times are selected to be 100 times. The invention classifies the MSER regions left after filtering by adopting the trained SVM classifier, thereby improving the accuracy of obtaining the text region. In addition, in order to achieve the best classification effect of the SVM, the quantity ratio of the positive samples to the negative samples is controlled to be 1:3, the positive sample is a letter corresponding to the translation target language (for example, when the translation target language is english, the corresponding letter may include ' a ' - ' Z ' and ' a ' - ' Z) and an arabic numeral (for example, the numeral ' 0 ' - ' 9 '), and the negative sample is a non-text region that is manually identified and marked on the extracted MSER region after the MSER regions of the original image and the multiple single-channel images are respectively extracted in step S102, so that the screening effect may be optimized, and the accuracy of obtaining the text region may be further improved.
In the outline pixel point set of a region, a part of points are connected in a certain sequence, so that the region can be restored to the maximum extent. Because the boundary key points of the image cannot be influenced even if the image is rotated and zoomed, the invention can eliminate non-text region interference items in the MSER region even if the image is rotated and zoomed by introducing the boundary key points as the classification screening characteristics, thereby improving the sensitivity of text region extraction on image rotation, size change and the like.
And S106, distinguishing each text region according to the distance between every two adjacent text regions in the vertical direction, and classifying each text region in the same text line according to the distance between every two adjacent text regions in the same text line.
In this embodiment, referring to fig. 4 and 5, when text line distinction is performed on each text region according to the distance between every two adjacent text regions in the vertical direction, each phase in the vertical direction may be calculated according to formula (4) firstDistance d between two adjacent text regionsv：
Wherein, b1Y-axis coordinate, t, representing the bottom of a vertically adjacent upper text region2Representing top Y-axis coordinates, h, of vertically adjacent lower text regions2Indicating the height of the text region adjacent to the lower side in the vertical direction, as shown in fig. 4. Then, for every two adjacent text regions in the vertical direction, if the distance d between the two adjacent text regions is smaller than the predetermined distancevIf the number of the adjacent text areas is larger than the second preset threshold value, the two adjacent text areas are classified into the same text line, otherwise, the two adjacent text areas are classified into different text lines. The applicant researches and discovers that the distance d between two adjacent text areas in the vertical directionvIf the second predetermined threshold is greater than 0.62, the two adjacent text regions are in the same text line, so the second predetermined threshold may be 0.62.
In addition, when the words of the same text line are distinguished according to the distance between every two adjacent text regions on the same text line, the distance d between every two adjacent text regions on the same text line can be calculated according to the formula (5) firsth：
Wherein,represents the average value of the widths of all the letters of the text line, and Δ d represents the distance difference between the adjacent letters in the two adjacent text areas in the X-axis direction, i.e., the interval between the two adjacent letters. Then, for every two adjacent text regions of the same text line, if the distance d between the two adjacent text regionshIf the value is larger than the third preset threshold value, the adjacency is representedAdjacent letters in the two text areas belong to the same word, and at the moment, the two adjacent text areas are classified into the same class; otherwise, the adjacent letters of the two adjacent text regions belong to different words, and the two adjacent text regions are classified into different categories. Because different letters of the same word may be divided into different text regions in the process of obtaining the text region, and the interval between the words and the interval between the letters in the words have obvious difference, as shown in fig. 5, the word recognition accuracy can be improved by distinguishing the words on the same text line after the text region is distinguished, and the two layers of text classification algorithms of text line classification in the vertical direction and text region classification in the same text line in the horizontal direction are adopted in the invention, so that the time complexity (the time complexity of the same type of algorithm is O (n) greatly2) The time complexity of the present invention is O (nlog)2n)), the word recognition rate is improved, and a foundation is provided for realizing real-time translation of the video subtitles. The applicant researches and discovers that when the distance d between two adjacent text areashIf the number of the adjacent letters in the adjacent text regions is greater than 2.33, the adjacent letters in the adjacent text regions belong to the same word, and therefore the third preset threshold may be 2.33.
And S107, performing real-time translation on the video subtitles based on the classified text regions.
In this embodiment, after obtaining each classified text region, text recognition may be performed by using an open-source framework Tesseract, and meanwhile, for unified management of the system, the Tesseract and the OpenCV image processing runtime library need to be integrated. After the text is recognized, the text can be transmitted to an interface provided by Google translation in the form of letter strings, a translation result is obtained and finally displayed to a user, and therefore real-time translation of the video subtitles is achieved.
It can be seen from the above embodiments that, before the text is identified, the present invention introduces a plurality of single-channel images, effectively utilizes the color information of the original image, provides richer basic data for the extraction of the text region, then introduces local contrast text features, performs threshold filtering on the MSER region extracted from the original image and the plurality of single-channel images, can improve the accuracy of the extraction of the text region, and the time complexity of the local contrast filtering is linear time, the filtering time is short, can provide a basis for real-time translation of video subtitles, introduces boundary key points as SVM classification filtering features, can eliminate non-text region interference in the MSER region even when the image is rotated and scaled, thereby can improve the sensitivity of the extraction of the text region to image rotation and scaling, after the MSER region is subjected to threshold filtering based on the local contrast, the invention adopts two layers of text classification algorithms of text line classification in the vertical direction and text region classification in the same text line in the horizontal direction aiming at the text region obtained after the training and screening are finished, thereby greatly reducing the time complexity, improving the word recognition rate, providing a foundation for realizing the real-time translation of the video captions, and realizing the real-time and accurate translation of the video captions.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (10)
1. A method for translating video subtitles in real time is characterized by comprising the following steps:
performing multi-channel extraction on an original image intercepted from a video to obtain a plurality of single-channel images;
respectively extracting MSER areas of the original image and the single-channel images based on a MSER algorithm of a maximum stable extremum area;
introducing local contrast text features, calculating the local contrast between each MSER region and the background region thereof, and determining whether to filter the corresponding MSER region according to each local contrast;
introducing text features of the boundary key points, and determining the boundary key points of each MSER region;
classifying and screening all MSER regions left after filtering by using the boundary key points as classification and screening characteristics through a trained Support Vector Machine (SVM) to obtain text regions;
according to the distance between every two adjacent text regions in the vertical direction, distinguishing each text region, and according to the distance between every two adjacent text regions in the same text line, classifying each text region in the same text line;
and performing real-time translation on the video subtitles based on the classified text regions.
2. The method for real-time translation of video subtitles according to claim 1, wherein before performing multi-channel extraction on an original image intercepted from a video to obtain a plurality of single-channel images, the method further comprises: preprocessing the original image including sharpening and blurring.
3. The method for real-time translation of video subtitles according to claim 2, wherein the step of performing multi-channel extraction on the original image captured from the video to obtain a plurality of single-channel images comprises: and respectively carrying out R, G, B, H, S, V six-channel image extraction on the original image and the preprocessed original image so as to obtain a plurality of single-channel images.
4. The method of claim 1, wherein the calculating the local contrast between each MSER region and its background region, and determining whether to filter out the corresponding MSER region according to each local contrast comprises:
the local contrast lc between each MSER region and its background is calculated according to the following formula:
<mrow> <mi>l</mi> <mi>c</mi> <mo>=</mo> <mfrac> <mrow> <mo>|</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mrow> <mo>(</mo> <mrow> <msub> <mi>R</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>G</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>B</mi> <mi>i</mi> </msub> </mrow> <mo>)</mo> </mrow> <mo>-</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>k</mi> </munderover> <mrow> <mo>(</mo> <mrow> <msub> <mi>R</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>G</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>B</mi> <mi>i</mi> </msub> </mrow> <mo>)</mo> </mrow> <mo>|</mo> </mrow> <mrow> <mi>max</mi> <mrow> <mo>(</mo> <mrow> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mrow> <mo>(</mo> <mrow> <msub> <mi>R</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>G</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>B</mi> <mi>i</mi> </msub> </mrow> <mo>)</mo> </mrow> <mo>,</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>k</mi> </munderover> <mrow> <mo>(</mo> <mrow> <msub> <mi>R</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>G</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>B</mi> <mi>i</mi> </msub> </mrow> <mo>)</mo> </mrow> </mrow> <mo>)</mo> </mrow> </mrow> </mfrac> </mrow>
wherein n represents the number of pixels corresponding to the MSER region, k represents the number of pixels corresponding to the background region, and Ri、Gi、BiRespectively representing the values of red, green and blue of three channels of the image corresponding to the MSER area, wherein i represents the ith pixel point corresponding to the MSER area, and j represents the jth pixel point corresponding to the background area;
for each MSER area, if the local contrast of the MSER area is smaller than a first preset threshold, filtering the MSER area.
5. The method for real-time translation of video subtitles according to claim 1, wherein the determining the key points of the boundary of each MSER region comprises:
aiming at each MSER region, setting the gray value of the pixel point of the MSER detected in the MSER region as 255 and setting the gray values of other pixel points as 0;
successively traversing each pixel point in the MSER area, and if the gray value of the pixel point is 255 and the gray value of at least one of the adjacent pixel points is 0, determining the pixel point as a contour point;
after all contour points of at least one MSER area are obtained, each contour point is compressed by adopting a Douglas-Pock algorithm, redundant points are removed, and boundary key points corresponding to the MSER area are obtained.
6. The method for real-time translation of video subtitles of claim 1, wherein the aspect ratio, the area-to-perimeter ratio, the convex hull area ratio and the stroke width-to-area ratio of each MSER region left after filtering are used as classification and screening features, and classification and screening are performed on each MSER region left after filtering through a trained SVM.
7. The method for real-time translation of video subtitles according to claim 6, wherein in the process of training the SVM, the ratio of the number of the positive samples to the number of the negative samples is controlled to be 1: and 3, the positive sample is a letter and an Arabic number corresponding to the translation target language, and the negative sample is a non-text region which is obtained by manually identifying and marking the extracted MSER region after the MSER regions of the original image and the multiple single-channel images are respectively extracted.
8. The method for translating video subtitles in real time according to claim 1, wherein the text line distinguishing of each text region according to the distance between every two adjacent text regions in the vertical direction comprises:
the distance d between every two adjacent text areas in the vertical direction is calculated according to the following formulav：
<mrow> <msub> <mi>d</mi> <mi>v</mi> </msub> <mo>=</mo> <mfrac> <mrow> <msub> <mi>b</mi> <mn>1</mn> </msub> <mo>-</mo> <msub> <mi>t</mi> <mn>2</mn> </msub> </mrow> <msub> <mi>h</mi> <mn>2</mn> </msub> </mfrac> </mrow>
Wherein, b1Y-axis coordinate, t, representing the bottom of a vertically adjacent upper text region2Representing top Y-axis coordinates, h, of vertically adjacent lower text regions2Representing vertically adjacent lower contextsThe height of the region;
for every two adjacent text regions in the vertical direction, if the distance d between the two adjacent text regionsvIf the number of the adjacent text areas is larger than the second preset threshold value, the two adjacent text areas are classified into the same text line, otherwise, the two adjacent text areas are classified into different text lines.
9. The method for real-time translation of video subtitles according to claim 1, wherein the classifying the text regions of the same text line according to the distance between every two adjacent text regions on the same text line comprises:
the distance d between every two adjacent text regions of the same text line is calculated according to the following formulah：
<mrow> <msub> <mi>d</mi> <mi>h</mi> </msub> <mo>=</mo> <mfrac> <mover> <mi>w</mi> <mo>&OverBar;</mo> </mover> <mrow> <mi>&Delta;</mi> <mi>d</mi> </mrow> </mfrac> </mrow>
Wherein,representing the average value of the widths of all letters of the text line, and delta d representing the distance difference of adjacent letters of two adjacent text areas in the X-axis direction;
for every two adjacent text regions of the same text line, if the distance d between the two adjacent text regionshIf the number of the adjacent text areas is larger than the third preset threshold, the two adjacent text areas are classified into one type, otherwise, the two adjacent text areas are classified into a different type.
10. The method of claim 1, wherein the video subtitle real-time translation method is characterized in that when an original image is cut out from a video, a video picture is cut out by frames, and two thirds of the area below the cut-out video image is used as the original image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710345936.1A CN107145888A (en) | 2017-05-17 | 2017-05-17 | Video caption real time translating method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710345936.1A CN107145888A (en) | 2017-05-17 | 2017-05-17 | Video caption real time translating method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107145888A true CN107145888A (en) | 2017-09-08 |
Family
ID=59778166
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710345936.1A Pending CN107145888A (en) | 2017-05-17 | 2017-05-17 | Video caption real time translating method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107145888A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108520254A (en) * | 2018-03-01 | 2018-09-11 | 腾讯科技（深圳）有限公司 | A kind of Method for text detection, device and relevant device based on formatted image |
| CN109284751A (en) * | 2018-10-31 | 2019-01-29 | 河南科技大学 | The non-textual filtering method of text location based on spectrum analysis and SVM |
| CN111797632A (en) * | 2019-04-04 | 2020-10-20 | 北京猎户星空科技有限公司 | Information processing method and device and electronic equipment |
| CN113287319A (en) * | 2019-01-09 | 2021-08-20 | 奈飞公司 | Optimizing encoding operations in generating buffer-constrained versions of media subtitles |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102054271A (en) * | 2009-11-02 | 2011-05-11 | 富士通株式会社 | Text line detection method and device |
| CN102542268A (en) * | 2011-12-29 | 2012-07-04 | 中国科学院自动化研究所 | Method for detecting and positioning text area in video |
| CN102750540A (en) * | 2012-06-12 | 2012-10-24 | 大连理工大学 | Morphological filtering enhancement-based maximally stable extremal region (MSER) video text detection method |
| CN103310439A (en) * | 2013-05-09 | 2013-09-18 | 浙江大学 | Method for detecting maximally stable extremal region of image based on scale space |
| WO2015105755A1 (en) * | 2014-01-08 | 2015-07-16 | Qualcomm Incorporated | Processing text images with shadows |
| CN105825216A (en) * | 2016-03-17 | 2016-08-03 | 中国科学院信息工程研究所 | Method of locating text in complex background image |
| CN105868758A (en) * | 2015-01-21 | 2016-08-17 | 阿里巴巴集团控股有限公司 | Method and device for detecting text area in image and electronic device |
| US9576196B1 (en) * | 2014-08-20 | 2017-02-21 | Amazon Technologies, Inc. | Leveraging image context for improved glyph classification |
-
2017
- 2017-05-17 CN CN201710345936.1A patent/CN107145888A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102054271A (en) * | 2009-11-02 | 2011-05-11 | 富士通株式会社 | Text line detection method and device |
| CN102542268A (en) * | 2011-12-29 | 2012-07-04 | 中国科学院自动化研究所 | Method for detecting and positioning text area in video |
| CN102750540A (en) * | 2012-06-12 | 2012-10-24 | 大连理工大学 | Morphological filtering enhancement-based maximally stable extremal region (MSER) video text detection method |
| CN103310439A (en) * | 2013-05-09 | 2013-09-18 | 浙江大学 | Method for detecting maximally stable extremal region of image based on scale space |
| WO2015105755A1 (en) * | 2014-01-08 | 2015-07-16 | Qualcomm Incorporated | Processing text images with shadows |
| US9576196B1 (en) * | 2014-08-20 | 2017-02-21 | Amazon Technologies, Inc. | Leveraging image context for improved glyph classification |
| CN105868758A (en) * | 2015-01-21 | 2016-08-17 | 阿里巴巴集团控股有限公司 | Method and device for detecting text area in image and electronic device |
| CN105825216A (en) * | 2016-03-17 | 2016-08-03 | 中国科学院信息工程研究所 | Method of locating text in complex background image |
Non-Patent Citations (1)
| Title |
|---|
| 何东健: "《数字图像处理》", 28 February 2015 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108520254A (en) * | 2018-03-01 | 2018-09-11 | 腾讯科技（深圳）有限公司 | A kind of Method for text detection, device and relevant device based on formatted image |
| CN109284751A (en) * | 2018-10-31 | 2019-01-29 | 河南科技大学 | The non-textual filtering method of text location based on spectrum analysis and SVM |
| CN113287319A (en) * | 2019-01-09 | 2021-08-20 | 奈飞公司 | Optimizing encoding operations in generating buffer-constrained versions of media subtitles |
| CN113287319B (en) * | 2019-01-09 | 2024-05-14 | 奈飞公司 | Method and apparatus for optimizing encoding operations |
| CN111797632A (en) * | 2019-04-04 | 2020-10-20 | 北京猎户星空科技有限公司 | Information processing method and device and electronic equipment |
| CN111797632B (en) * | 2019-04-04 | 2023-10-27 | 北京猎户星空科技有限公司 | Information processing method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109961049B (en) | Cigarette brand identification method under complex scene | |
| CN108562589B (en) | Method for detecting surface defects of magnetic circuit material | |
| CN104751142B (en) | A kind of natural scene Method for text detection based on stroke feature | |
| CN113724231B (en) | Industrial defect detection method based on semantic segmentation and target detection fusion model | |
| CN104408449B (en) | Intelligent mobile terminal scene literal processing method | |
| CN108305253B (en) | Pathological image classification method based on multiple-time rate deep learning | |
| CN106875546A (en) | A kind of recognition methods of VAT invoice | |
| CN107145888A (en) | Video caption real time translating method | |
| CN106570510A (en) | Supermarket commodity identification method | |
| CN111046872A (en) | Optical character recognition method | |
| CN110598566A (en) | Image processing method, device, terminal and computer readable storage medium | |
| CN113298809B (en) | Composite material ultrasonic image defect detection method based on deep learning and superpixel segmentation | |
| CN110987886A (en) | Full-automatic microscopic image fluorescence scanning system | |
| CN115082776A (en) | Electric energy meter automatic detection system and method based on image recognition | |
| CN109741273A (en) | A kind of mobile phone photograph low-quality images automatically process and methods of marking | |
| CN115578741A (en) | Mask R-cnn algorithm and type segmentation based scanned file layout analysis method | |
| CN116824608A (en) | Answer sheet layout analysis method based on target detection technology | |
| CN116416624A (en) | Document electronization method and device based on layout correction and storage medium | |
| CN113392819B (en) | Batch academic image automatic segmentation and labeling device and method | |
| CN106295627A (en) | For identifying the method and device of word psoriasis picture | |
| CN113065404B (en) | Method and system for detecting train ticket content based on equal-width character segments | |
| CN114419008A (en) | Image quality evaluation method and system | |
| CN111738310B (en) | Material classification method, device, electronic equipment and storage medium | |
| CN110276260B (en) | Commodity detection method based on depth camera | |
| CN112348026A (en) | Magnetic hard disk sequence code identification method based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170908 |
|
| RJ01 | Rejection of invention patent application after publication |