EP0720114A2 - Verfahren und Gerät zur Detektion und Interpretation von Untertiteln in digitalen Videosignalen - Google Patents

Verfahren und Gerät zur Detektion und Interpretation von Untertiteln in digitalen Videosignalen Download PDF

Info

Publication number
EP0720114A2
EP0720114A2 EP95117595A EP95117595A EP0720114A2 EP 0720114 A2 EP0720114 A2 EP 0720114A2 EP 95117595 A EP95117595 A EP 95117595A EP 95117595 A EP95117595 A EP 95117595A EP 0720114 A2 EP0720114 A2 EP 0720114A2
Authority
EP
European Patent Office
Prior art keywords
image
interpretation
identification
text
computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP95117595A
Other languages
English (en)
French (fr)
Other versions
EP0720114B1 (de
EP0720114A3 (de
Inventor
Arturo Pizano
Farshid Arman
Daniel Conrad Benson
Remi Depommier
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Corporate Research Inc
Original Assignee
Siemens Corporate Research Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Corporate Research Inc filed Critical Siemens Corporate Research Inc
Publication of EP0720114A2 publication Critical patent/EP0720114A2/de
Publication of EP0720114A3 publication Critical patent/EP0720114A3/de
Application granted granted Critical
Publication of EP0720114B1 publication Critical patent/EP0720114B1/de
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/635—Overlay text, e.g. embedded captions in a TV programme
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Definitions

  • the present invention relates to the detection and interpretation of textual captions in video signals and, more particularly, to the identification and interpretation of text captions in digital video signals.
  • Text captions are commonly used in domains such as television sports and news broadcasts, to convey information that complements or explains the contents of the audio and video information being presented.
  • captions might present game scores and player names in sporting events, or places, situations, and dates in newscasts.
  • Fig. 1 illustrates typical captions found in television broadcasts.
  • the "text captions” herein referred to should be distinguished from the "closed-captions” used, for example, in broadcast programs to assist the hearing impaired and other audiences interested in live textual transcripts of the words spoken in the sound channel of a broadcast. These captions are transmitted in a separate channel of the signal and are "overlaid” into the screen display of a television receiver by means of a special decoder.
  • the captions herein addressed are textual descriptions embedded in the video signal itself.
  • an automatic caption detector can be used to address the following two principal problems in digital video management.
  • the first problem relates to indexing ⁇ captions can be interpreted with an optical character recognition (OCR) system and used to generate indexes into the contents of the video., e.g. names of persons, places, dates, situations, etc. These indexes serve as references to specific videos or video segments from a large collection. For example, a person could request clips of a particular sports star or when a certain politician appears, or about a certain place.
  • the second problem relates to segmentation ⁇ captions can be used to partition the video into meaningful segments based on the visual cues they offer to the viewer. In Fig. 2a, for example, captions were used to segment a video of an interview including a number of individuals. Using the browser one can quickly identify the person of interest and begin playing the corresponding segment (Fig. 2b).
  • captions are regularly used to report the score of the game after every field-goal or free-throw.
  • the ability to identify and interpret these captions can then be used to generate triplets of the form (time, TeamAScore, TeamBScore), which could latter be used to answer queries such as "show me all the segments of the game where TeamB was ahead by more than 10".
  • they can be used to create a score browser which would enable a person to move directly to specific portions of the video; see Fig. 2c.
  • the present invention is intended to be implemented by programmable computing apparatus, preferably by a digital computer.
  • operational steps herein referred to are generally intended to signify machine operations
  • captions in the present context are those textual descriptors overlaid on a video by its producer. More specifically, captions are considered to exhibit the following characteristics. Captions do not move from frame to frame, i.e., they remain in the exact same location in each frame regardless of what is happening in the rest of the scene. Captions remain on the screen for at least a minimum period of time, i.e., they will appear in a plurality of consecutive frames. This is important because it enables sampling of the video to detect captions, and because the redundancy can be used to improve the accuracy of the method.
  • Captions are intended to be read from a distance. Thus, there are minimum character sizes that can be utilized in making a determination as to whether a video segment contains text.
  • non-caption text may appear in video.
  • street signs and advertisements will typically appear in outdoor scenes. Text is also often found in commercial broadcast material. In both cases if one or more of the afore-mentioned characteristics is violated (e.g., the text in a street sign may move in a fast action shot), the text will not be detected in accordance with the present invention.
  • a computer-implemented method for the identification and interpretation of text captions in an encoded video stream of digital video signals comprises sampling by selecting frames for video analysis; decoding by converting each of frames selected into a digitized color image; performing edge detection for generating a grey scale image; binarizing by converting the grey scale image into a bi-level image by apparatus of a thresholding operation; compressing groups of consecutive pixel values in the binary image; mapping the consecutive pixel values into a binary value; and separating groups of connected pixels and determining whether they are likely to be part of a text region in the image or not.
  • the sampling is at a sampling rate fixed at 1 frame per N, where N is the number of consecutive frames in which the same caption is expected to appear.
  • the decoding uses one of JPEG encoding and MPEG.
  • the format of the resulting color image is a 24-bit RGB format and the compressing groups is performed eight at a time.
  • the mapping of the pixel levels is performed by apparatus of a test wherein during this step a binary edge image of a frame, which size is defined to be WIDTH x HEIGHT, is converted into an image of size WIDTH/8 x HEIGHT by compressing each byte in the original image (8 continuous pixels) into a binary value in accordance with predetermined criteria.
  • the criteria comprise
  • a step of taking a binary image as input and producing a table of connected-components is formatted as follows: Component ID EnclosingRectangle Dimensions Density where Component ID is an integer, Enclosing rectangle are the coordinates of the smallest rectangle containing all the pixels in the component (minX, minY, maxX, maxY), dimensions are measurements of the width, height and area of the enclosing rectangle, and density is the ratio of black pixels, which are associated with edges, in the component.
  • a step of determining whether a connected component is likely to contain edges associated with text or not comprises the steps of:
  • the method comprises the steps of: projecting black pixels contained in connected components that passed the geometric and content tests into the Y-axis of the image, thereby producing a projection pattern; testing the resulting projection pattern to determine if its vertical runs, defined as sequences of consecutive lines having counts greater than zero, exceed a minimum height of the characters being sought in the caption.
  • the computer-implemented method for the identification and interpretation of text captions includes the step of defining the binary image, and its corresponding video frame, as having a caption if and only if it has at least one run satisfying the minimum text height condition.
  • the computer-implemented method for the identification and interpretation of text captions includes the step of confirming whether a collection of frames previously determined to have captions is further processed to confirm the result by the steps of: defining N to be a frame determined to have a caption; applying the decoding, edge detection and binarization steps above to frames N-D/2 and N+D/2 ; combining resulting images with a binary image of N using an AND operation so as to result in two new binary images in which some of the edges associated with the individual frames have been removed, but those associated within text remain; applying the compression, connected component analysis and projection evaluation steps above to each of the two new binary images; determining a frame to have a caption, if and only if, either one, or both of the two new images is determined to have a caption.
  • a computer-implemented method for the identification and interpretation of text captions in an encoded video stream of digital video signals comprises: sampling by selecting frames for video analysis; decoding by converting each of frames selected into a digitized color image;separating each the digitized color image into three color images corresponding to three color planes;performing edge detection on each of the color planes for generating a respective grey scale image for each of the color planes;applying a thresholding image to each of the grey scale images so as to produce three respective binary edge images; combining the three binary edge images to obtain a single combined binary edge image;compressing groups of consecutive pixel values in the combined binary image; mapping the consecutive pixel values into a binary value; and separating groups of connected pixels and determining whether they are likely to be part of a text region in the image or not.
  • the digitized color image is separated into three 8-bit color images corresponding to the three color planes.
  • the three color planes are respectively red, green, and blue (RGB).
  • a computer-implemented method for the identification and interpretation of text captions in a video stream wherein the frame sequence is compressed comprises the steps of: determining whether the frame number divided by a predetermined number N is an integer, discarding non-integers; decoding compressed frames so as to result in uncompressed frames; detecting edges so as to derive a corresponding greyscale image; binarizing the greyscale image so as to derive a binary image; compressing the binary image so as to derive a compressed binary image; performing a connected component analysis.
  • the connected components analysis is carried out by computing connected components using a standard 4-neighbor connectivity text.
  • each the computed connected component is subjected to two sets of tests involving its geometric properties and contents.
  • the geometric tests involve minimum and maximum boundaries on a respective connected component's width, height and area.
  • the content tests, applied to an area in the binary image following edge detection corresponding to the connected component include upper and lower boundaries on the proportion of black pixels contains therein, and a threshold on the number of vertical zero-runs, defined as collections of one or more columns in which no black pixels occur.
  • the computer-implemented method for the identification and interpretation of text captions comprises the steps of: separating out connected components that passed the tests; projecting values of corresponding pixels in the binary image following edge detection into the vertical axis of the image so as to result in a projection pattern.
  • the computer-implemented method for the identification and interpretation of text captions comprises the steps of: testing the projection pattern to determine if it contains runs that exceed a given threshold and thereby determine if a caption is present.
  • Opaque background the caption is printed on top of a solid block that completely obscures the scene (e.g., captions 4 and 5 in Fig. 1)
  • Translucid background the caption is printed on top of translucid background that obscures but does not eliminate the scene (e.g., caption 2 in Fig. 1).
  • Transparent background the caption is printed directly on top of the scene (captions 1, 3, 6, 7 and 8 in Fig. 1).
  • Such a classification is useful because identifying captions where there is no background requires additional processing, as will become apparent in the course of the present description of the invention and, more particularly, in the description of the algorithm and its application in the context of the invention.
  • the processing of digital video to detect and interpret text captions is carried out in two phases. First, frames in the video that contain text captions are identified. Then these captions are separated from the rest of the image. Each step is described separately below.
  • the objective of the caption detection phase is to identify the collection of video frames that contain text captions.
  • the task is accomplished in two steps as follows: initial detection during which the algorithm determines if a particular frame has a caption; and confirmation where neighboring frames are examined to confirm the presence of a caption.
  • the initial detection step involves generating intermediate binary representations of video frames (pre-processing) then determining whether it contains a caption.
  • the initial detection step of the algorithm is designed to use information contained in a single frame.
  • the size of the sampling rate is a function of the time the captions are expected to be present in the video.
  • the inventors have found that the shortest captions appear to be active for at least two seconds, which translates into a sampling rate of 1/60 if the video is digitized at 30 frames per second.
  • Frames selected during the sampling process are decoded and converted into individual color images.
  • the system uses motion JPEG (Joint Photographic Experts Group) video, but MPEG (Moving Pictures Experts Group) or other formats can be used, given an appropriate frame decoder.
  • MPEG Motion Pictures Experts Group
  • the preferred representation of the color image is a 24-bit RGB (red-green-blue) format. The use of other color representations or depths would require slight changes to the algorithms.
  • the first step performed in the processing of an individual frame is edge detection.
  • advantage is taken of the availability of color information by applying edge detection in each separate color plane and then combining the results during the binarization step.
  • the process has proven particularly useful in dealing with transparent-background captions in which the letters of the caption blend with the rest of the scene.
  • the edge detection technique used involves taking the Pythagorean sum of two Sobel gradient operators at 90 degrees to each other. The algorithm is described "Digital Image Processing" by Gonzalez and Wintz, chapter 7, and implemented in the set of operations made available with the portable bitmap (PBM) file format used in this project.
  • PBM portable bitmap
  • the gray scale images obtained in the edge detection step are binarized and combined into a single binary edge image.
  • the process involves threshholding and OR-ing the image.
  • Figs. 3a-c illustrates the pre-processing tasks.
  • Compression ⁇ during this step the binary edge image of a frame, which size is defined to be WIDTH x HEIGHT, is converted into an image of size WIDTH/8 x HEIGHT by compressing each byte in the original image (8 continuous pixels) into a binary value using the following criteria:
  • Fig. 3d illustrates a compressed image corresponding to that of Fig. 3c.
  • Geometric tests involve minimum and maximum boundaries on the connected component's width, height and area (minimum width, maximum width, minimum height, maximum height, minimum area, maximum area, minimum density, minimum black pixels, maximum black pixels).
  • Content tests applied to the area in the original binary edge detection image corresponding to the connected component, include upper and lower boundaries on the percentage of black pixels it contains, and a threshold on the number of vertical zero-runs, defined as collections of one or more columns in which no black pixels occur. The reason for the latter test is to account for the separation that exists between the text characters.
  • Fig. 3e shows the components that remain after applying the tests to the compressed image of Fig. 3d.
  • the threshold values used by way of example are as follows. minimum width 3 maximum height 40 min. black pixels 0.33 maximum width 40 minimum area 16 max. black pixels 0.66 minimum height 10 maximum area 640 It is noted that these values were set for a frame consisting of 240 lines of 320 pixels.
  • the final analysis task involves separating the connected components that passed the test and projecting the values of the corresponding pixels in the original binary edge image into the vertical axis of the image; see Fig. 3f.
  • the resulting projection pattern is then tested to determine if it contains runs that exceed a given threshold. This test is based on the observation that shapes corresponding to text will contain a high number of edges concentrated in an area having approximately the height of the expected text.
  • the caption detection described above uses the color and shape information contained in each individual frame.
  • motion information is utilized to increase the overall accuracy (recall rate) of the method.
  • the basic premise for this work is the fact that captions will remain in the same portion of the video while the background changes and thus it is possible to effectively remove background noise by combining the edges of two (or more) neighboring frames.
  • N be the frame being analyzed
  • E(N) be the corresponding edge image
  • the procedure described herein detects a caption using the three basic characteristics described above, i.e., captions do not move, they remain on the screen for at least a minimum period of time, and they are intended to be read from a distance. It is also possible to use additional information about the domain associated with the video to enhance the efficiency of the procedure. In particular, the information of the approximate location of the caption within the frame, enables the algorithm to focus on specific areas of the image (e.g., the bottom half) with the corresponding reduction in processing requirements.
  • This approach can be taken a step further if the type of caption being sought is known exactly or almost exactly. For example, in sporting events the score of a match is typically reported constantly throughout the broadcast. In this case a simpler matching algorithm can be used in conjunction with the confirmation step.
  • the next step is to interpret the text contained in the caption.
  • the task involves locating the area or areas of interest, separating them from the rest of the image, and applying OCR.
  • the first two steps are relatively straightforward given the information obtained from the connected component analysis performed during the detection phase.
  • applying OCR is a more difficult task, particularly if the caption background is transparent, or even translucid, because the letters in the text will blend with the scene.
  • the solution herein disclosed involves taking advantage of motion and color to remove objects in the background.
  • the basic premises upon which this solution is based are: (1) captions are displayed using saturated colors; and (2) there is motion on the scene behind the captions.
  • the technique is similar to that used in the confirmation step of the capture detection algorithm, but this time the color images themselves are considered.
  • motion and color information is utilized as follows:
  • N, N-x and N+x be three frames containing the same caption; the value of x should be as large as possible, but all three frames must contain the same caption.
  • RGB(N,i,j) be the rgb color values for the i-th pixel of row j in frame N.
  • HSV(N,i,j) be the corresponding values in the hue-saturation color model.
  • a new color image I is constructed by multiplying the HSV values of N, N-x and N+x, and then normalizing to the maximum value. The resulting image will highlight saturated values that remain in the same location, fading all other values.
  • Fig. 5 illustrates the process when applied to the frames shown in Fig. 4. The binary image at the end of the sequence was obtain by threshholding the saturation values of the results.
  • the input is in the form of an encoded video stream.
  • Step 1 sampling, involves selecting frames for video analysis.
  • a sampling rate is fixed at 1 frame per N, where N is the number of consecutive frames in which the same caption is expected to appear.
  • Step 2 decoding, involves converting each of the selected frames into a digitized color image.
  • a current implementation of the uses motion JPEG encoding. However, the same technique can be applied using MPEG or another other encoding mechanism.
  • the preferred format of the resulting color image is a 24-bit RGB format.
  • Step 3 edge detection uses the algorithm described in "Digital Image Processing", by Gonzales and Wintz.
  • Step 4 binarization, converts the greyscale image generated in Step 3 into a bi-level image by means of a thresholding operation.
  • Step 5 compression, groups consecutive pixel values in the binary image - eight at a time - and maps them into a binary value by means of the test described above in relation to compression, that is, during this step the binary edge image of a frame, which size is defined to be WIDTH x HEIGHT, is converted into an image of size WIDTH/8 x HEIGHT by compressing each byte in the original image (8 continuous pixels) into a binary value using the criteria given above.
  • Step 6 connected component analysis, separates groups of connected pixels and determines whether they are likely to be part of a text region in the image or not.
  • the algorithm takes a binary image as input and produces a table of connected-components formatted as follows: Component ID EnclosingRectangle Dimensions Density where Component ID is an integer, Enclosing rectangle are the coordinates of the smallest rectangle containing all the pixels in the component (minX, minY, maxX, maxY), dimensions are measurements of the width, height and area of the enclosing rectangle, and density is the ratio of black pixels, which are associated with edges, in the component.
  • the algorithm corresponding to the flow chart of Figure 7 is used to determine whether the connected component is likely to contain edges associated with text or not. The following steps are performed for each component:
  • the final step in this portion of the algorithm involves projecting the black pixels contained in the connected components that passed the geometric and content tests into the Y-axis of the image.
  • the resulting projection pattern is then tested to determine if its vertical runs, defined a sequences of consecutive lines having counts greater than zero, exceed the minimum height of the characters being sought in the caption.
  • the binary image, and its corresponding video frame, is considered to have a caption if and only if it has at least one run satisfying the minimum text height condition.
  • the flow chart in Figure 9 illustrates the confirmation step. During this step the collection of frames determined in phase I to have captions is further processed to confirm the result.
  • N be a frame thought to have a caption.
  • the first part of this test involves applying the decoding, edge detection and binarization steps described above to frames N-D/2 and N+D/2 .
  • the resulting images are then combined with the binary image of N using an AND operation.
  • the result of this operation are two new binary images in which some of the edges associated with the individual frames have been removed, but those associated within text remain.
  • the second part of the confirmation step involves applying the compression, connected component analysis and projection evaluation steps described above to each of the two new binary images.
  • a frame is said to have a caption, if and only if, either one, or both of the new images is determined to have a caption.
  • the flow chart in Figure 10 shows the process as applied to frames containing captions to separate the text from the rest of the scene.
  • the technique used also exploits motion in the image to improve the quality of the results.
  • the color images themselves, not the edges are combined using this time and addition operations that first adds the values of corresponding pixels in the two images and then normalizes them with respect to the highest pixel value.
  • the effect of this operation is to darken pixels with low intensity values, and to enhance pixels with high intensity values such as those typically found in captions.
  • the two images obtained in the first step are combined using the same technique to obtain a third color image in which the background is likely to be darkened even more.
  • This images is the converted to grey-scale and binarized and yielding a black and white image which can the enhanced and passed to an OCR system.
  • the resulting text strings serve as indices into the content of the video.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Analysis (AREA)
EP95117595A 1994-12-28 1995-11-08 Verfahren und Gerät zur Detektion und Interpretation von Untertiteln in digitalen Videosignalen Expired - Lifetime EP0720114B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US36527794A 1994-12-28 1994-12-28
US365277 1994-12-28

Publications (3)

Publication Number Publication Date
EP0720114A2 true EP0720114A2 (de) 1996-07-03
EP0720114A3 EP0720114A3 (de) 1997-02-26
EP0720114B1 EP0720114B1 (de) 2001-01-24

Family

ID=23438200

Family Applications (1)

Application Number Title Priority Date Filing Date
EP95117595A Expired - Lifetime EP0720114B1 (de) 1994-12-28 1995-11-08 Verfahren und Gerät zur Detektion und Interpretation von Untertiteln in digitalen Videosignalen

Country Status (3)

Country Link
US (1) US6101274A (de)
EP (1) EP0720114B1 (de)
DE (1) DE69519980T2 (de)

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0893779A1 (de) * 1997-07-22 1999-01-27 Sony International (Europe) GmbH Videogerät mit automatischem Internetzugang
WO2000005679A1 (en) * 1998-07-20 2000-02-03 Intel Corporation Display capture system
WO2000045291A1 (en) * 1999-01-28 2000-08-03 Koninklijke Philips Electronics N.V. System and method for analyzing video content using detected text in video frames
FR2794560A1 (fr) * 1999-06-01 2000-12-08 Thomson Csf Procede d'aide a la lecture notamment pour malvoyant
GB2352915A (en) * 1999-08-06 2001-02-07 Television Monitoring Services A method of retrieving text data from a broadcast image
WO2001037212A1 (en) * 1999-11-17 2001-05-25 Koninklijke Philips Electronics N.V. Video stream classifiable symbol isolation method and system
WO2001076231A3 (en) * 2000-03-31 2002-01-31 Koninkl Philips Electronics Nv Text detection
US6366699B1 (en) 1997-12-04 2002-04-02 Nippon Telegraph And Telephone Corporation Scheme for extractions and recognitions of telop characters from video data
WO2002095662A1 (fr) * 2001-05-23 2002-11-28 France Telecom (Sa) Procede de detection de zones de texte dans une image video
WO2003001810A1 (en) * 2001-06-21 2003-01-03 Wespot Ab Invariant filters
US6608930B1 (en) 1999-08-09 2003-08-19 Koninklijke Philips Electronics N.V. Method and system for analyzing video content using detected text in video frames
US7203337B2 (en) 2001-06-21 2007-04-10 Wespot Ab Adjusted filters
US7436979B2 (en) 2001-03-30 2008-10-14 Siemens Energy & Automation Method and system for image processing
US7689613B2 (en) 2006-10-23 2010-03-30 Sony Corporation OCR input to search engine
US7814524B2 (en) 2007-02-14 2010-10-12 Sony Corporation Capture of configuration and service provider data via OCR
US7966552B2 (en) 2006-10-16 2011-06-21 Sony Corporation Trial selection of STB remote control codes
US7991271B2 (en) 2007-02-14 2011-08-02 Sony Corporation Transfer of metadata using video frames
US8035656B2 (en) 2008-11-17 2011-10-11 Sony Corporation TV screen text capture
US8077263B2 (en) 2006-10-23 2011-12-13 Sony Corporation Decoding multiple remote control code sets
US8320674B2 (en) 2008-09-03 2012-11-27 Sony Corporation Text localization for image and video OCR
US8438589B2 (en) 2007-03-28 2013-05-07 Sony Corporation Obtaining metadata program information during channel changes
US8763038B2 (en) 2009-01-26 2014-06-24 Sony Corporation Capture of stylized TV table data via OCR
CN107545261A (zh) * 2016-06-23 2018-01-05 佳能株式会社 文本检测的方法及装置
US9888279B2 (en) 2013-09-13 2018-02-06 Arris Enterprises Llc Content based video content segmentation
CN109409359A (zh) * 2018-09-25 2019-03-01 天津大学 一种基于深度学习的视频字幕提取方法

Families Citing this family (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6587586B1 (en) * 1997-06-12 2003-07-01 Siemens Corporate Research, Inc. Extracting textual information from a video sequence
JP3011157B2 (ja) * 1997-10-30 2000-02-21 日本電気株式会社 画像読取方法及び装置
US6470094B1 (en) * 2000-03-14 2002-10-22 Intel Corporation Generalized text localization in images
US6473522B1 (en) * 2000-03-14 2002-10-29 Intel Corporation Estimating text color and segmentation of images
US6738512B1 (en) * 2000-06-19 2004-05-18 Microsoft Corporation Using shape suppression to identify areas of images that include particular shapes
US6753929B1 (en) * 2000-06-28 2004-06-22 Vls Com Ltd. Method and system for real time motion picture segmentation and superposition
US7657064B1 (en) 2000-09-26 2010-02-02 Digimarc Corporation Methods of processing text found in images
WO2002047390A1 (en) * 2000-12-04 2002-06-13 Isurftv E-mail, telephone number or url within tv frame
KR100386116B1 (ko) * 2000-12-15 2003-06-02 (주) 멀티비아 멀티미디어 데이터의 부호화 및 복호화 시스템
US6888893B2 (en) * 2001-01-05 2005-05-03 Microsoft Corporation System and process for broadcast and communication with very low bit-rate bi-level or sketch video
CA2330854A1 (en) * 2001-01-11 2002-07-11 Jaldi Semiconductor Corp. A system and method for detecting a non-video source in video signals
US20110013081A1 (en) * 2001-01-11 2011-01-20 Pixelworks, Inc. System and method for detecting a non-video source in video signals
US6778700B2 (en) * 2001-03-14 2004-08-17 Electronics For Imaging, Inc. Method and apparatus for text detection
US6993594B2 (en) * 2001-04-19 2006-01-31 Steven Schneider Method, product, and apparatus for requesting a resource from an identifier having a character image
US7916794B2 (en) * 2001-04-28 2011-03-29 Microsoft Corporation System and process for broadcast and communication with very low bit-rate bi-level or sketch video
WO2002089033A1 (en) * 2001-04-30 2002-11-07 Philip Tan Meng Ngee A multi-dimensional method and system of simulating and managing an alliance investment portfolio
DE60229632D1 (de) * 2001-05-15 2008-12-11 Koninkl Philips Electronics Nv Untertitelerkennung in anzeigbaren Bildbereich eines Videosignals
FR2825817B1 (fr) * 2001-06-07 2003-09-19 Commissariat Energie Atomique Procede de traitement d'images pour l'extraction automatique d'elements semantiques
US6985631B2 (en) * 2002-02-20 2006-01-10 Hewlett-Packard Development Company, L.P. Systems and methods for automatically detecting a corner in a digitally captured image
AU2002368117A1 (en) * 2002-07-26 2004-02-16 National Institute Of Information And Communications Technology Incorporated Administrative Agency Image recognition apparatus and image recognition program
US7136066B2 (en) * 2002-11-22 2006-11-14 Microsoft Corp. System and method for scalable portrait video
JP4112968B2 (ja) * 2002-12-26 2008-07-02 富士通株式会社 ビデオテキスト処理装置
JP4258385B2 (ja) * 2004-01-14 2009-04-30 株式会社デンソー 路面反射検出装置
KR100537520B1 (ko) * 2004-02-18 2005-12-19 삼성전자주식회사 동영상의 자막 검출 방법 및 장치
KR100647284B1 (ko) * 2004-05-21 2006-11-23 삼성전자주식회사 영상의 문자 추출 장치 및 방법
US20060045346A1 (en) 2004-08-26 2006-03-02 Hui Zhou Method and apparatus for locating and extracting captions in a digital image
WO2006072897A1 (en) * 2005-01-04 2006-07-13 Koninklijke Philips Electronics N.V. Method and device for detecting transparent regions
US8745687B2 (en) * 2005-02-15 2014-06-03 Sony Corporation Digital closed caption transport in standalone stream
US7630544B1 (en) 2005-04-06 2009-12-08 Seiko Epson Corporation System and method for locating a character set in a digital image
US20060294545A1 (en) * 2005-06-23 2006-12-28 Microsoft Corporation Dynamic media guide listings
US20070011012A1 (en) * 2005-07-11 2007-01-11 Steve Yurick Method, system, and apparatus for facilitating captioning of multi-media content
US20080091713A1 (en) * 2006-10-16 2008-04-17 Candelore Brant L Capture of television metadata via OCR
US20080098357A1 (en) * 2006-10-23 2008-04-24 Candelore Brant L Phantom information commands
US20080098433A1 (en) * 2006-10-23 2008-04-24 Hardacker Robert L User managed internet links from TV
US8296808B2 (en) * 2006-10-23 2012-10-23 Sony Corporation Metadata from image recognition
EP1870868B1 (de) * 2006-11-10 2008-10-08 Engine SRL System und Verfahren zur Detektion der Durchschnittsgeschwindigkeit von Fahrzeugen zur Kontrolle des Verkehrs
KR100836197B1 (ko) * 2006-12-14 2008-06-09 삼성전자주식회사 동영상 자막 검출 장치 및 그 방법
JPWO2008078717A1 (ja) * 2006-12-25 2010-04-22 株式会社Ptp 番組データ管理サーバ、識別子割当装置、番組データ管理方法及びプログラム
US8155444B2 (en) * 2007-01-15 2012-04-10 Microsoft Corporation Image text to character information conversion
US8929461B2 (en) * 2007-04-17 2015-01-06 Intel Corporation Method and apparatus for caption detection
US7912289B2 (en) * 2007-05-01 2011-03-22 Microsoft Corporation Image text replacement
US20080273114A1 (en) * 2007-05-04 2008-11-06 Hardacker Robert L STB channel reader
ES2374686T3 (es) 2007-05-14 2012-02-21 Historx, Inc. Separación en compartimentos por caracterización de píxel usando agrupamiento de datos de imágenes.
US7929764B2 (en) * 2007-06-15 2011-04-19 Microsoft Corporation Identifying character information in media content
EP2162728B1 (de) 2007-06-15 2016-07-27 Novartis AG Mikroskopisches system und verfahren zur erfassung von standardisierten daten
CA2604317C (en) 2007-08-06 2017-02-28 Historx, Inc. Methods and system for validating sample images for quantitative immunoassays
CA2596204C (en) 2007-08-07 2019-02-26 Historx, Inc. Method and system for determining an optimal dilution of a reagent
US20090041352A1 (en) * 2007-08-10 2009-02-12 Naoki Okamoto Image formation device, image formation method, and computer-readable recording medium recording image formation program
WO2009029810A1 (en) 2007-08-31 2009-03-05 Historx, Inc. Automatic exposure time selection for imaging tissue
US8230039B2 (en) * 2008-04-16 2012-07-24 Adobe Systems, Incorporated Systems and methods for accelerated playback of rich internet applications
AU2009293380B2 (en) 2008-09-16 2016-05-19 Novartis Ag Reproducible quantification of biomarker expression
WO2011008821A1 (en) * 2009-07-15 2011-01-20 Home Box Office, Inc. Identification of 3d format and graphics rendering on 3d displays
US9491398B1 (en) 2010-12-21 2016-11-08 Pixelworks, Inc. System and method for processing assorted video signals
US9251144B2 (en) 2011-10-19 2016-02-02 Microsoft Technology Licensing, Llc Translating language characters in media content
TWI462576B (zh) * 2011-11-25 2014-11-21 Novatek Microelectronics Corp 固定圖案的邊緣偵測方法與電路
US8494346B2 (en) * 2011-12-22 2013-07-23 Max Abecassis Identifying a performer during a playing of a video
US9329692B2 (en) 2013-09-27 2016-05-03 Microsoft Technology Licensing, Llc Actionable content displayed on a touch screen
US9779293B2 (en) * 2016-01-27 2017-10-03 Honeywell International Inc. Method and tool for post-mortem analysis of tripped field devices in process industry using optical character recognition and intelligent character recognition
CN114648751B (zh) * 2020-12-17 2026-01-30 武汉Tcl集团工业研究院有限公司 一种处理视频字幕的方法、装置、终端及存储介质
CN113887334B (zh) * 2021-09-13 2024-08-09 华中师范大学 一种视频知识点抽取方法及装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE3113555A1 (de) * 1981-04-03 1982-10-21 Siemens AG, 1000 Berlin und 8000 München Verfahren zum automatischen erkennen von weissbloecken sowie text-, graphik- und/oder graubildbereichen auf druckvorlagen
US4811407A (en) * 1986-01-22 1989-03-07 Cablesoft, Inc. Method and apparatus for converting analog video character signals into computer recognizable binary data
US5459587A (en) * 1991-05-02 1995-10-17 Minolta Camera Kabushiki Kaisha Processing apparatus capable of discriminating between pseudo half-tone/non-half-tone image data based upon the number of adjacencies of similar type of pixels within a block
US5249050A (en) * 1991-10-18 1993-09-28 Zenith Electronics Corporation Closed captioned data line detection system
US5392365A (en) * 1991-12-23 1995-02-21 Eastman Kodak Company Apparatus for detecting text edges in digital image processing
US5335297A (en) * 1992-01-17 1994-08-02 The United States Of Americas As Represented By The Secretary Of The Navy Target detection for vision systems
US5467410A (en) * 1992-03-20 1995-11-14 Xerox Corporation Identification of a blank page in an image processing system
GB9206651D0 (en) * 1992-03-26 1992-05-06 Solid State Logic Ltd Video processing
US5262860A (en) * 1992-04-23 1993-11-16 International Business Machines Corporation Method and system communication establishment utilizing captured and processed visually perceptible data within a broadcast video signal
US5379130A (en) * 1992-12-02 1995-01-03 Industrial Technology Research Institute Text/image separation method

Cited By (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0893779A1 (de) * 1997-07-22 1999-01-27 Sony International (Europe) GmbH Videogerät mit automatischem Internetzugang
US6501856B2 (en) 1997-12-04 2002-12-31 Nippon Telegraph And Telephone Corporation Scheme for extraction and recognition of telop characters from video data
US6366699B1 (en) 1997-12-04 2002-04-02 Nippon Telegraph And Telephone Corporation Scheme for extractions and recognitions of telop characters from video data
WO2000005679A1 (en) * 1998-07-20 2000-02-03 Intel Corporation Display capture system
KR100448557B1 (ko) * 1998-07-20 2004-09-13 인텔 코오퍼레이션 디스플레이 캡쳐 시스템
WO2000045291A1 (en) * 1999-01-28 2000-08-03 Koninklijke Philips Electronics N.V. System and method for analyzing video content using detected text in video frames
CN100342376C (zh) * 1999-01-28 2007-10-10 皇家菲利浦电子有限公司 用图像帧中检测到的文本信息分析视频内容的系统和方法
US6731788B1 (en) 1999-01-28 2004-05-04 Koninklijke Philips Electronics N.V. Symbol Classification with shape features applied to neural network
US6614930B1 (en) 1999-01-28 2003-09-02 Koninklijke Philips Electronics N.V. Video stream classifiable symbol isolation method and system
FR2794560A1 (fr) * 1999-06-01 2000-12-08 Thomson Csf Procede d'aide a la lecture notamment pour malvoyant
GB2352915A (en) * 1999-08-06 2001-02-07 Television Monitoring Services A method of retrieving text data from a broadcast image
EP1074926A3 (de) * 1999-08-06 2002-01-30 Television Monitoring Services Limited Verfahren und Gerät zum Ermitteln von Textdaten aus einem Videosignal
US6608930B1 (en) 1999-08-09 2003-08-19 Koninklijke Philips Electronics N.V. Method and system for analyzing video content using detected text in video frames
WO2001037212A1 (en) * 1999-11-17 2001-05-25 Koninklijke Philips Electronics N.V. Video stream classifiable symbol isolation method and system
WO2001076231A3 (en) * 2000-03-31 2002-01-31 Koninkl Philips Electronics Nv Text detection
US6842537B2 (en) * 2000-03-31 2005-01-11 Koninklijke Philips Electronics N.V. Text detection
US7436979B2 (en) 2001-03-30 2008-10-14 Siemens Energy & Automation Method and system for image processing
FR2825173A1 (fr) * 2001-05-23 2002-11-29 France Telecom Procede de detection de zones de texte dans une image video
WO2002095662A1 (fr) * 2001-05-23 2002-11-28 France Telecom (Sa) Procede de detection de zones de texte dans une image video
US7203337B2 (en) 2001-06-21 2007-04-10 Wespot Ab Adjusted filters
WO2003001810A1 (en) * 2001-06-21 2003-01-03 Wespot Ab Invariant filters
US7966552B2 (en) 2006-10-16 2011-06-21 Sony Corporation Trial selection of STB remote control codes
US8077263B2 (en) 2006-10-23 2011-12-13 Sony Corporation Decoding multiple remote control code sets
US7689613B2 (en) 2006-10-23 2010-03-30 Sony Corporation OCR input to search engine
US8629942B2 (en) 2006-10-23 2014-01-14 Sony Corporation Decoding multiple remote control code sets
US9124922B2 (en) 2007-02-14 2015-09-01 Sony Corporation Capture of stylized TV table data via OCR
US7991271B2 (en) 2007-02-14 2011-08-02 Sony Corporation Transfer of metadata using video frames
US7814524B2 (en) 2007-02-14 2010-10-12 Sony Corporation Capture of configuration and service provider data via OCR
US9241134B2 (en) 2007-02-14 2016-01-19 Sony Corporation Transfer of metadata using video frames
US8438589B2 (en) 2007-03-28 2013-05-07 Sony Corporation Obtaining metadata program information during channel changes
US8621498B2 (en) 2007-03-28 2013-12-31 Sony Corporation Obtaining metadata program information during channel changes
US8320674B2 (en) 2008-09-03 2012-11-27 Sony Corporation Text localization for image and video OCR
US8035656B2 (en) 2008-11-17 2011-10-11 Sony Corporation TV screen text capture
US8763038B2 (en) 2009-01-26 2014-06-24 Sony Corporation Capture of stylized TV table data via OCR
US9888279B2 (en) 2013-09-13 2018-02-06 Arris Enterprises Llc Content based video content segmentation
CN107545261A (zh) * 2016-06-23 2018-01-05 佳能株式会社 文本检测的方法及装置
CN109409359A (zh) * 2018-09-25 2019-03-01 天津大学 一种基于深度学习的视频字幕提取方法

Also Published As

Publication number Publication date
EP0720114B1 (de) 2001-01-24
DE69519980T2 (de) 2001-06-07
DE69519980D1 (de) 2001-03-01
EP0720114A3 (de) 1997-02-26
US6101274A (en) 2000-08-08

Similar Documents

Publication Publication Date Title
US6101274A (en) Method and apparatus for detecting and interpreting textual captions in digital video signals
Gllavata et al. A robust algorithm for text detection in images
CN100514363C (zh) 用于数字图像处理的方法和装置
US20070286499A1 (en) Method for Classifying Digital Image Data
JP5067310B2 (ja) 字幕領域抽出装置、字幕領域抽出方法および字幕領域抽出プログラム
US7403656B2 (en) Method and apparatus for recognition of character string in scene image
KR20010110416A (ko) 비디오 스트림 분류가능 심볼 고립 방법 및 시스템
EP1463301A1 (de) Verfahren zur Identifizierung von Marken in Videosequenzen
EP2259207B1 (de) Verfahren zur Erfassung und Erkennung von Logos in einem Videodatenstrom
JP4626886B2 (ja) デジタル画像におけるキャプションを位置特定及び抽出する方法及び装置
US7305128B2 (en) Anchor person detection for television news segmentation based on audiovisual features
US8311269B2 (en) Blocker image identification apparatus and method
JP5090330B2 (ja) 画像処理装置、画像処理方法およびプログラム
JP3655110B2 (ja) 映像処理方法及び装置並びに映像処理手順を記録した記録媒体
Jain et al. A hybrid approach for detection and recognition of traffic text sign using MSER and OCR
Aung et al. Automatic license plate detection system for myanmar vehicle license plates
JPH10222678A (ja) 物体検出装置および物体検出方法
Gao et al. Automatic news video caption extraction and recognition
EP1460835A1 (de) Verfahren zur Identifizierung von Marken in Videosequenzen
Zhang et al. A new edge-based text verification approach for video
Gllavata et al. Finding text in images via local thresholding
Zhao et al. A robust color-independent text detection method from complex videos
Zedan et al. Caption detection, localization and type recognition in Arabic news video
Arai et al. Text extraction from TV commercial using blob extraction method
CN100417201C (zh) 检测新闻主播的影音特征以将电视新闻分段的方法

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): DE FR GB

17P Request for examination filed

Effective date: 19970820

17Q First examination report despatched

Effective date: 19971105

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REF Corresponds to:

Ref document number: 69519980

Country of ref document: DE

Date of ref document: 20010301

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: GB

Ref legal event code: IF02

26N No opposition filed
REG Reference to a national code

Ref country code: GB

Ref legal event code: 732E

Free format text: REGISTERED BETWEEN 20110303 AND 20110309

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 69519980

Country of ref document: DE

Owner name: SIEMENS AKTIENGESELLSCHAFT, DE

Free format text: FORMER OWNER: SIEMENS CORP. RESEARCH, INC., PRINCETON, N.J., US

Effective date: 20110214

Ref country code: DE

Ref legal event code: R081

Ref document number: 69519980

Country of ref document: DE

Owner name: SIEMENS AKTIENGESELLSCHAFT, DE

Free format text: FORMER OWNER: SIEMENS CORP. RESEARCH, INC., PRINCETON, US

Effective date: 20110214

REG Reference to a national code

Ref country code: FR

Ref legal event code: TP

Owner name: SIEMENS CORPORATION, US

Effective date: 20110927

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 69519980

Country of ref document: DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 69519980

Country of ref document: DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: TP

Owner name: SIEMENS AKTIENGESELLSCHAFT, DE

Effective date: 20120803

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 69519980

Country of ref document: DE

Effective date: 20120807

Ref country code: DE

Ref legal event code: R082

Ref document number: 69519980

Country of ref document: DE

Effective date: 20120806

Ref country code: DE

Ref legal event code: R081

Ref document number: 69519980

Country of ref document: DE

Owner name: SIEMENS AKTIENGESELLSCHAFT, DE

Free format text: FORMER OWNER: SIEMENS CORP. (N. D. GES. D. STAATES DELAWARE), ISELIN, N.J., US

Effective date: 20120806

Ref country code: DE

Ref legal event code: R081

Ref document number: 69519980

Country of ref document: DE

Owner name: SIEMENS AKTIENGESELLSCHAFT, DE

Free format text: FORMER OWNER: SIEMENS CORP. (N. D. GES. D. STAATES DELAWARE), ISELIN, US

Effective date: 20120806

REG Reference to a national code

Ref country code: GB

Ref legal event code: 732E

Free format text: REGISTERED BETWEEN 20121213 AND 20121219

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20140120

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20141112

Year of fee payment: 20

Ref country code: GB

Payment date: 20141110

Year of fee payment: 20

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 69519980

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150602

REG Reference to a national code

Ref country code: GB

Ref legal event code: PE20

Expiry date: 20151107

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF EXPIRATION OF PROTECTION

Effective date: 20151107