WO2002103617A1 - Automatic natural content detection in video information - Google Patents
Automatic natural content detection in video information Download PDFInfo
- Publication number
- WO2002103617A1 WO2002103617A1 PCT/IB2002/002279 IB0202279W WO02103617A1 WO 2002103617 A1 WO2002103617 A1 WO 2002103617A1 IB 0202279 W IB0202279 W IB 0202279W WO 02103617 A1 WO02103617 A1 WO 02103617A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- natural
- line
- content
- video information
- distances
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/40—Document-oriented image-based pattern recognition
- G06V30/41—Analysis of document content
- G06V30/413—Classification of content, e.g. text, photographs or tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- the invention relates to a method, device and apparatus for distinguishing areas of natural and synthetic content in video information.
- CRT monitors are characterized on one side by a higher resolution than television screens and, on the other side, by a lower brightness. This is due to the fact that originally the content displayed on computer monitors was exclusively synthetic, and in particular, it was represented by text. This type of content clearly needs a high resolution to be enjoyable to the user but this causes a decrease in brightness.
- the basic idea for new-concept CRT monitors is that the monitors should be adaptable to the content of an image that in a particular moment is displayed.
- An example is to apply video-enhancement algorithms to the natural content in order to obtain a significant improvement in the quality of natural images displayed on the monitors.
- video-enhancement algorithms are applied to pure text or graphics, the overall result is a significant loss in image quality. From this point of view, the ability to distinguish between natural and synthetic content becomes important.
- Enhancement solutions are known that can improve significantly visual performances, if applied to specific areas of the screen in which natural contents are present.
- Window-based (which is application based) manual selection performed by the user is a simple though boring method for identifying these areas, which can be adapted when the whole window content is natural content.
- the same approach cannot be used in cases where composite contents are in the same window, as is typical of Web pages, because, as noted above, application of the video-enhancement algorithms to the pure text or graphics can cause significant loss in their perceived visual quality.
- the natural image content is distinguished from the synthetic image content by means of a statistical analysis aimed at extracting some features from an image and then performing a smart interpretation of the features.
- a statistical analysis aimed at extracting some features from an image and then performing a smart interpretation of the features.
- the video information is handled as a sequence of images, each of which are processed independently.
- the video information is analyzed.
- neighboring sections of the video information which contain similar features found during the analysis are then grouped together. Sections can be lines of rows or columns of the image, but can also be parts of lines.
- groups which have a first feature are designated as being natural content and any remaining groups are designated as being synthetic content.
- a luminance histogram of pixel values for each line of the matrix is created.
- the distances between each of the non-zero histogram values for each line are then determined.
- a line is classified as containing natural content if the majority of distances is less than or equal to a predetermined value. Neighboring lines containing natural content are then grouped together to create groups of lines with natural content.
- Fig. 1(a) illustrates a block diagram of a general algorithm philosophy
- Fig. 1(b) illustrates a block diagram of an algorithm according to the invention
- Figs. 2(a)-(c) illustrate a luminance histogram analysis of a synthetic case according to the embodiment of the invention
- Figs. 3(a)-(c) illustrate a luminance histogram of a middle synthetic case according to the embodiment of the invention
- Figs. 4(a)-(c) illustrate a luminance histogram analysis of a natural case according to the embodiment of the invention
- Fig. 5 illustrates a data tree for storing information related to coordinates of target areas according to the embodiment of the invention
- Fig. 6 is a representation of a number of sub-areas extracted from the target areas according to the embodiment of the invention.
- Figs. 7-10 illustrate screen shots which will be used to describe an illustrative example of the invention
- the invention can be regarded as a mix of segmentation and recognition. Many problems of signal recognition have been posed and solved in literature and then in applications, but most of them referred to mono-dimensional signals. Although these proposed solutions are very different, if a general analysis is done of all of them, some similarities can be pointed out. In fact most of these proposed solutions present a similar general structure which is illustrated in Fig. 1(a). First of all, a feature extraction block 100 that performs the so-called “feature extraction” is present, followed by a feature analysis block 102 that performs the "feature analysis”. Obviously, this description represents a very general abstraction because the term "feature" can mean many different objects.
- a key idea of the invention is that the "intelligence" of the algorithm has to be posed in the feature analysis block 102 which does not operate on the original data, but rather on a filtered (condensed) version of the original data.
- Original data could be corrupted from noise or extra information not useful or dangerous for the recognition.
- Features, instead, are regarded as a filtered version of data (in a general sense) which contain only the essential information.
- Fig. 1(b) illustrates a system for implementing one embodiment of the invention.
- the system comprises a luminance conversion unit 120, a controller 122, a histogram evaluator, 124, a classification unit 126 comprising an analyser 1108 and a rule application unit 1110 and a coordinate extractor 128.
- the operation of the system is described below.
- the luminance conversion unit 120 provides the required conversion as explained below.
- luminance is provided by the following formula:
- L(x, y) (0.2989* R(x, y) + 0.5870*G(x, y) + 0.1140*B(x, y)) with L,R,G,B in [0,1].
- R,G,B being the red, green and blue color components of the pixel in the array with coordinates x, y.
- L(x, y) (77 R(x, y) + 150 Gfx, v + 29 Bfx. vY) (integer division
- Histograms of the luminance values L(x, y) are evaluated in the histogram evaluator 124 as described below.
- a key idea is to evaluate the mono-dimensional histograms of the luminance values L(x, y) of every row of the image separately. The same kind of elaboration is iterated on the columns to obtain an additional set of histograms.
- the processing of the luminance values L(x, y) is the most resource consuming. It is necessary to scan the whole image pixel by pixel. However, as noted above, the objective is to analyze the whole image to obtain a set of features less numerous than the luminance data of the whole image.
- a key idea behind the classification unit 126 is to classify lines (being rows or columns) as natural image, if the corresponding histograms are peculiar for a natural image. From experimental tests, it has been noticed that histograms relating to a natural image have different characteristics compared to a histogram related to a synthetic image. These characteristics are composed of distances d that occur between consecutive non-zero elements of the luminance histograms L(x, y).
- the analyser 1108 analyses these distances d, making use of a distance histogram hist(d) of these distances.
- a key idea in this analysis is that in case a significant amount of natural image is present in the line, small distances are more probable than large distances.
- a classification rule is then used in the rule application unit 1110 to classify lines based on these distances.
- a clear-cut separation between distance histograms hist(d) representing natural content and those representing synthetic content is obtained using the following rule:
- the parenthesis In this case it extracts the distance (or distances) corresponding to the absolute maximum value of hist(d) (or maximum values if there are two or more equally large maximums).
- that line is considered to be containing a significant amount of pixels belonging to a natural image and therefore the line is classified as NATURAL, otherwise it is classified as SYNTHETIC.
- distance equal to one is considered to represent a line with natural content while all other distances are considered to represent lines with synthetic content.
- the invention is not limited to just this one rule and that the distance which delineates between natural and synthetic content can have values other than one.
- a fuzzy approach could be used to take into account other small distances and use for example more classes like: "probably synthetic", "probably natural", “very likely natural”.
- neighboring lines classified as NATURAL are grouped together.
- This grouping of lines uses in a preferred embodiment as rule that if less than three consecutive SYNTHETIC lines are present in between NATURAL lines, then these SYNTHETIC lines are included in the group of neighboring lines classified as NATURAL.
- the rule may use more or less than the mentioned 3 lines.
- the rule discards groups which comprise less than a predetermined number of natural lines. This predetermined number can be one, but larger numbers are also possible.
- areas formed by the cross-sections of groups of row-lines and groups of column-lines are determined. These areas are the likely NATURAL areas of the image.
- the coordinate extractor 128 determines the coordinates of the corners of these areas.
- the controller 122 determines whether the process of determining the NATURAL area should be repeated for a specific area. If yes, the steps indicated in Fig. lb by the blocks 124, 126, 128, 122 is repeated for these specific NATURAL areas of the image. This repetition is done preferably on a slightly larger area to secure that this larger area encloses the actual NATURAL area of the image. After a number of cycles, resulting in a more accurate determination of the NATURAL areas, the controller 122 delivers the final values of the coordinates of the corners of the NATURAL areas.
- Figs. 2(a)-(c) illustrate an extremely synthetic case, in which in Fig. 2(a) a uniform line (simulated with two pixels having a value of 100) is plotted on a constant background (simulated with pixels whose values are 255). As illustrated with the luminance histogram hist(L) in Fig. 2(b), the distances d between the luminance values of the pixels present in the line is 155. As can be noticed in this case, no distances d equal to one are present in the distances histogram hist(d) and distances tend to have great values as expected for lines whose content is synthetic. Figs.
- FIG. 4(a)-(c) illustrate a "natural" case.
- the line of Fig. 4(a) being analyzed contains softened values that are typical of natural images.
- the pixel values are grouped between 122-126 and the distances equal 2, 1, 1, respectively as shown in the histogram hist(L) in Fig. 4(b).
- Figs. 3(a)-(c) illustrate an intermediate case in which both clear cut values and softened value are present.
- some of the pixel values are grouped around 100 while other pixel values equal 155 and 255 as shown in Fig. 3(a).
- the resulting distances equal 1, 54, 100, respectively as shown in the histogram hist(L) in Fig. 3(b).
- both distances d equal to one and different from one are present, but since distances d different from one are not more numerous than the distances d equal to one as shown in hist(d) in Fig. 3(c), the line is classified as natural.
- a tree is used as a data structure to store information related to coordinates found and the classifier 126 is used on the images extracted in the previous cycle.
- the classifier 126 is used on the entire image extracting a list of 4xm coordinates relative to the areas in which it is more probable the presence of an image (where m is the number of target areas).
- the classifier is restarted on each of these target areas extracting a number of sub-areas in which images could be present, as illustrated in Fig. 6.
- This recursive process is repeated a number of times. It is experimentally obtained that repeating the process tliree times, delivers good results.
- the number of cycles could be dependent on a rule to stop iterations, for example, when having at the end of a cycle no or only one natural area inside the area that was evaluated during that cycle.
- Figs. 7-10 illustrate screen shots which will be used to describe an illustrative example of the invention.
- Fig. 7 illustrates the histogram evaluator 124 and classification unit 126 of this illustrative example.
- the histograms for the rows and columns of the screen 700 are evaluated separately for each row (as shown symbolically in a row bar 710) and column (a shown symbolically in a column bar 720).
- the most likely distance between a histogram value and the nearest value different from zero is found. If the most likely distance found is equal to one, that row (or column) 701 is considered to contain some natural content. Consequently, it is labeled as a probable row (or column) with natural content.
- At the end of this step there are two vectors containing the classification of the rows and columns previously analyzed.
- a "regularization" of the classification of rows and columns contained in the vectors is performed, as illustrated in Fig. 8.
- regularization the aggregation of rows and columns labeled as natural content is meant.
- Rows (or columns) which have a distance between each other which is less than a predetermined threshold are considered to comprise information of the same natural image and are aggregated together as illustrated by blocks 802.
- the rows and columns as natural content are aggregated together according to their "density".
- the position of areas 902 with natural image content are identified as the cross-sections of the aggregated rows and columns as shown in Fig. 9.
- the position of these areas 902 is known from the two vectors. However, this position is not precisely known. Therefore, as next step each area of the image is evaluated separately. Larger areas 904 are considered in this step with respect the areas detected in the previous step, to take into account that detection, made previously, is quite rough.
- the whole process of histogram evaluation 124, classification 126 and regularization is applied recursively. The advantage is that the histograms are evaluated on more specific areas and so their statistical content is more homogeneous.
- areas 904 which have rows and columns which do not meet the requirements for "natural content" are discarded.
- the resulting areas 1002 with natural content are illustrated in Fig. 10.
- a distance probability function can be determined, using the output of the histogram evaluator 124 in Fig. 11.
- the "Distances' Probability Function” (DPF) is calculated in analyser 1108.
- (j) is the j-th value of the luminance histogram for line i .
- This value represents the number of pixels in the line i that have a luminance equal to j.
- the line is classified as Synthetic and the rest of the steps for that line are skipped.
- the separate calculation of the vector ⁇ can be deleted by deriving & directly from h,.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Character Input (AREA)
Abstract
Description
Claims
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP02727979A EP1402463A1 (en) | 2001-06-15 | 2002-06-14 | Automatic natural content detection in video information |
US10/480,126 US20040161152A1 (en) | 2001-06-15 | 2002-06-14 | Automatic natural content detection in video information |
JP2003505863A JP2004530992A (en) | 2001-06-15 | 2002-06-14 | Automatic natural content detection in video information |
KR10-2003-7002194A KR20030027953A (en) | 2001-06-15 | 2002-06-14 | Automatic natural content detection in video information |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP01202279.4 | 2001-06-15 | ||
EP01202279 | 2001-06-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2002103617A1 true WO2002103617A1 (en) | 2002-12-27 |
Family
ID=8180475
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2002/002279 WO2002103617A1 (en) | 2001-06-15 | 2002-06-14 | Automatic natural content detection in video information |
Country Status (6)
Country | Link |
---|---|
US (1) | US20040161152A1 (en) |
EP (1) | EP1402463A1 (en) |
JP (1) | JP2004530992A (en) |
KR (1) | KR20030027953A (en) |
CN (1) | CN1692369A (en) |
WO (1) | WO2002103617A1 (en) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2006087666A1 (en) * | 2005-02-16 | 2006-08-24 | Koninklijke Philips Electronics N.V. | Method for natural content detection and natural content detector |
US7978922B2 (en) * | 2005-12-15 | 2011-07-12 | Microsoft Corporation | Compressing images in documents |
TW200820767A (en) * | 2006-06-26 | 2008-05-01 | Genesis Microchip Inc | Universal, highly configurable video and graphic measurement device |
US7920755B2 (en) * | 2006-06-26 | 2011-04-05 | Genesis Microchip Inc. | Video content detector |
US7826680B2 (en) * | 2006-06-26 | 2010-11-02 | Genesis Microchip Inc. | Integrated histogram auto adaptive contrast control (ACC) |
US7881547B2 (en) * | 2006-07-28 | 2011-02-01 | Genesis Microchip Inc. | Video window detector |
US20080162561A1 (en) * | 2007-01-03 | 2008-07-03 | International Business Machines Corporation | Method and apparatus for semantic super-resolution of audio-visual data |
US20080219561A1 (en) * | 2007-03-05 | 2008-09-11 | Ricoh Company, Limited | Image processing apparatus, image processing method, and computer program product |
US9973723B2 (en) * | 2014-02-24 | 2018-05-15 | Apple Inc. | User interface and graphics composition with high dynamic range video |
CN105760884B (en) | 2016-02-22 | 2019-09-10 | 北京小米移动软件有限公司 | The recognition methods of picture type and device |
US11546617B2 (en) | 2020-06-30 | 2023-01-03 | At&T Mobility Ii Llc | Separation of graphics from natural video in streaming video content |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4668995A (en) * | 1985-04-12 | 1987-05-26 | International Business Machines Corporation | System for reproducing mixed images |
EP0522702A2 (en) * | 1991-06-12 | 1993-01-13 | Hewlett-Packard Company | Spot color extraction |
US6195459B1 (en) * | 1995-12-21 | 2001-02-27 | Canon Kabushiki Kaisha | Zone segmentation for image display |
US20020031268A1 (en) * | 2001-09-28 | 2002-03-14 | Xerox Corporation | Picture/graphics classification system and method |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0567680B1 (en) * | 1992-04-30 | 1999-09-22 | International Business Machines Corporation | Pattern recognition and validation, especially for hand-written signatures |
JP3373008B2 (en) * | 1993-10-20 | 2003-02-04 | オリンパス光学工業株式会社 | Image area separation device |
CA2144793C (en) * | 1994-04-07 | 1999-01-12 | Lawrence Patrick O'gorman | Method of thresholding document images |
US6104833A (en) * | 1996-01-09 | 2000-08-15 | Fujitsu Limited | Pattern recognizing apparatus and method |
US6351558B1 (en) * | 1996-11-13 | 2002-02-26 | Seiko Epson Corporation | Image processing system, image processing method, and medium having an image processing control program recorded thereon |
US6594380B2 (en) * | 1997-09-22 | 2003-07-15 | Canon Kabushiki Kaisha | Image discrimination apparatus and image discrimination method |
US6731775B1 (en) * | 1998-08-18 | 2004-05-04 | Seiko Epson Corporation | Data embedding and extraction techniques for documents |
US6674900B1 (en) * | 2000-03-29 | 2004-01-06 | Matsushita Electric Industrial Co., Ltd. | Method for extracting titles from digital images |
-
2002
- 2002-06-14 KR KR10-2003-7002194A patent/KR20030027953A/en not_active Application Discontinuation
- 2002-06-14 EP EP02727979A patent/EP1402463A1/en not_active Withdrawn
- 2002-06-14 CN CNA028118928A patent/CN1692369A/en active Pending
- 2002-06-14 US US10/480,126 patent/US20040161152A1/en not_active Abandoned
- 2002-06-14 JP JP2003505863A patent/JP2004530992A/en active Pending
- 2002-06-14 WO PCT/IB2002/002279 patent/WO2002103617A1/en not_active Application Discontinuation
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4668995A (en) * | 1985-04-12 | 1987-05-26 | International Business Machines Corporation | System for reproducing mixed images |
EP0522702A2 (en) * | 1991-06-12 | 1993-01-13 | Hewlett-Packard Company | Spot color extraction |
US6195459B1 (en) * | 1995-12-21 | 2001-02-27 | Canon Kabushiki Kaisha | Zone segmentation for image display |
US20020031268A1 (en) * | 2001-09-28 | 2002-03-14 | Xerox Corporation | Picture/graphics classification system and method |
Non-Patent Citations (1)
Title |
---|
GOVINDARAJU V ET AL: "NEWSPAPER IMAGE UNDERSTANDING", KNOWLEDGE BASED COMPUTER SYSTEMS. INTERNATIONAL CONFERENCE KBCS, XX, XX, 1990, pages 375 - 384, XP000892729 * |
Also Published As
Publication number | Publication date |
---|---|
EP1402463A1 (en) | 2004-03-31 |
CN1692369A (en) | 2005-11-02 |
US20040161152A1 (en) | 2004-08-19 |
KR20030027953A (en) | 2003-04-07 |
JP2004530992A (en) | 2004-10-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9940655B2 (en) | Image processing | |
US6574354B2 (en) | Method for detecting a face in a digital image | |
US6101274A (en) | Method and apparatus for detecting and interpreting textual captions in digital video signals | |
US6731788B1 (en) | Symbol Classification with shape features applied to neural network | |
US6614930B1 (en) | Video stream classifiable symbol isolation method and system | |
JP3740065B2 (en) | Object extraction device and method based on region feature value matching of region-divided video | |
JP3966154B2 (en) | Method and computer system for segmenting multidimensional images | |
EP1831823B1 (en) | Segmenting digital image and producing compact representation | |
US6512848B2 (en) | Page analysis system | |
CN101453575B (en) | Video subtitle information extracting method | |
US6996272B2 (en) | Apparatus and method for removing background on visual | |
CN106937114B (en) | Method and device for detecting video scene switching | |
US20050002566A1 (en) | Method and apparatus for discriminating between different regions of an image | |
US8369407B2 (en) | Method and a system for indexing and searching for video documents | |
US6360002B2 (en) | Object extracting method using motion picture | |
KR20010033552A (en) | Detection of transitions in video sequences | |
WO2005066896A2 (en) | Detection of sky in digital color images | |
JP2000196895A (en) | Digital image data classifying method | |
KR100390866B1 (en) | Color image processing method and apparatus thereof | |
US11836958B2 (en) | Automatically detecting and isolating objects in images | |
EP1402463A1 (en) | Automatic natural content detection in video information | |
Fernando et al. | Fade-in and fade-out detection in video sequences using histograms | |
Gllavata et al. | Finding text in images via local thresholding | |
JP2005521169A (en) | Analysis of an image consisting of a matrix of pixels | |
EP4164222A1 (en) | Lossy compression of video content into a graph representation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2002727979 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020037002194 Country of ref document: KR |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWP | Wipo information: published in national office |
Ref document number: 1020037002194 Country of ref document: KR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2003505863 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 10480126 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 028118928 Country of ref document: CN |
|
WWP | Wipo information: published in national office |
Ref document number: 2002727979 Country of ref document: EP |
|
REG | Reference to national code |
Ref country code: DE Ref legal event code: 8642 |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 2002727979 Country of ref document: EP |