Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4092—Image resolution transcoding, e.g. by using client-server architectures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/95—Retrieval from the web
  • G06F16/957—Browsing optimisation, e.g. caching or content distillation
  • G06F16/9577—Optimising the visualization of content, e.g. distillation of HTML documents
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
  • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
  • G11B27/034—Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
  • G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
  • G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process

Definitions

• the present invention relates to a method and a device for video presentation, and more particularly to a method and a device for adaptive video presentation on a display with limited screen size.
• ACM MM' 03, 3003 which introduces three browsing modes: manual browsing method, full-automatic browsing method and semiautomatic browsing method.
• the present invention provides an adaptive video presentation solution for full-automatically representing videos on display devices with small sizes according to metadata information based on content analysis in order to provide an optimal video viewing experience for users .
• the method comprises steps of determining a salient object group containing at least one salient object based on perceptual interest value of macroblocks for each frame of the video, extracting a window having a minimum size containing the salient object group for a scene of the video, characterized in that it further comprises steps of comparing size of the extracted window with the display size; and presenting at least a selected area of the extracted window containing at least a part of the salient object group for the scene on the display based on the result of the comparison step.
• the extracted window is exposited on the display with an appropriate zoom-in operation; if size of the extracted window is larger than a predefined percentage of the display size and equal or smaller than the display size, the extracted window is exposited directly on the display; and if size of the extracted window is larger than the display size, the extracted window is exposited on the display with an appropriate zoom-out operation.
• the method further comprises steps of calculating a weighted average motion vector length MV act of macroblocks inside frames for the scene of the video, comparing the weighted average motion vector length MV act of the scene with a predefined threshold T mo ti on> determining whether the weighted average motion vector length MV ac t of the scene is less than the predefined threshold T mo tion, and presenting the extracted window for the scene of video on the display in a true motion exhibition mode corresponding to a fast motion status in case the average motion vector length MV act of the scene is determined less than the predefined threshold TWi 0n ; or else presenting the extracted window for the scene of the video on the display in a normal exhibition mode corresponding to a low motion status in case the average motion vector length MV act of the scene is determined less than the predefined threshold T mot i o n-
• the extracted window containing all the salient object groups for whole the scene of the video is presented on the display with a weighted average gravity point of the salient object group for the whole scene defined as a still focus centre of the extracted window, in such a way the true motion exhibition mode can be achieved in the fast motion status.
• the method in the normal exhibition mode corresponding to the low motion status, further comprises steps of comparing length of scene of the video with a predefined minimum perceptual time, wherein if the length of scene is less than the predefined minimum perceptual time, it further comprises steps of determining number of the salient objects existing in the scene, presenting the extracted window on the display with a gravity flowing show operation in case only one salient object exists, or else presenting the extracted window directly on the display in case multiple salient objects exist, wherein if the length of scene is no less than the predefined minimum perceptual time, it further comprises a' step of presenting the extracted window on the display with the gravity flowing show operation.
• FIG. 1 is a schematic view of a first embodiment of the system framework using the method in accordance with the present invention
• Fig. 2 is a schematic view of a second embodiment of the system framework using the method in accordance with the present invention.
• Fig. 3 is a schematic view of a third embodiment of the system framework using the method in accordance with the present invention.
• Fig. 4 is a schematic view of salient objects inside one frame
• Fig. 5 is a schematic view of salient object group inside one frame
• Fig. 6 is a flowchart of adaptive video presentation sample solution. DETAIL DESCRIPTION OF PREFERRED EMBODIMENTS
• the present invention directs to a method and a device of adaptive video presentation (AVP) for better viewing experience with stream-embedded metadata base on content analysis information.
• AVP adaptive video presentation
• decoder end solution decoder end solution
• Joint encoder-decoder end solution Joint encoder-decoder end solution
• encoder end solution encoder end solution
• the first type of the AVP framework solution leaves all the region processing and display work in the decoder end, where only a preanalysis module 11a is provided at an encoder end 10a, while other functional blocks are provided at a decoder end 20a.
• the pre-analysis module 11a includes operations of scene change detection, attention area extraction and content/motion analysis.
• the other four functional blocks includes an object group classification (OGC) module 12a, which classifies objects/object groups based on the scene and attention mask information from pre-analysis module 11a; a property calculation (PC) module 13a, which calculates statistics (e.g.
• a still focus decision (SFD) module 14a which decides candidate focus area in special image based on the statistics information derived from PC module (e.g. gravity points) 13a and other metadata information from pre-analysis module 11; a spatial- temporal processing module 15a, which does a spatial- temporal processing to guarantee the video smooth and acceptable and eliminates the artifacts .
• SFD still focus decision
• a pre-analysis module lib an object group classification module 12b, a property calculation module 13b and a still focus decision module 14b are included in the encoder end 10b to generate candidate focus area, and a spatial/temporal processing module 15b is included in a decoder end 20b to do optimal display based on candidate focus area with consideration of temporal and spatial quality tradeoff.
• AVP Salient Object
• a Salient Object is a set of attention area MacroBlocks (MB) connected each other, as shown by area of MBs with grey in Fig. 4.
• the salient objects are separated by the non-attention MBs, which is denoted by MBs with white .
• Salient Object group contains all salient Objects in the current frame. It can be described by following parameters:
• Video consists of a collection of video frames and the segmentation results can be frame, shot, scene, and video with granularity from small to large. Shot is a sequence of frames recorded in a single-camera operation.
• Scene is a collection of consecutive shots that have semantic similarity in object, person, space, and time. It's also defined to tell the switch of salient objects between two frames.
• the display scheme inside a scene should be definite and usually stays consistent.
• Configuration parameters are necessary parameters on help making decisions of adaptive display mode selection, such as display or not, scaling down or not, summarizing or not, etc. There are four conditions defined to assist the video viewing path programming.
• MPT is used as a threshold for the fixation duration when viewing a salient object. It's also used as the threshold of the weight of information value when watching a scene. If a salient object doesn't stay on the screen longer than a MPT threshold MPT 30 , it may not be perceptible enough to let users catch the information. If a scene doesn't last longer than a threshold MPT SC , only the most significant portion in it may be perceptible enough.
• MPT 30 and MPT SC can be selected according to different application scenarios and human visual property, which is usually set to 1/3 second and 2 second in our real application.
• the MPS is used as a threshold of the minimum spatial area of a salient object. Normally, if the size of a salient object SOi is less than a threshold MPT 30 , the salient object Oi should become marked as non- attention object or be merged into its neighbourhood attention object. But the MPS threshold is not always correct since a salient object with smaller spatial area may carry the most important information, and it cannot be merged or unmarked. So, additional configuration parameter of Weight Information (be introduced below) can be used. Usually MPT 30 can be set to 5 MacroBlocks or be set to 5%-10% of the largest salient object size.
• WSOi can be defined by the semantic importance of each salient object, which is depend on the content mode, third part's appointed semantic information, specifically user's experience, etc. Furthermore, the gravity of the salient object group is re-calculated.
• Operation Set includes all the possible operations needed for the requirement of Adaptive Video Presentation, Currently six operations are defined as Table-I shows:
• the adaptive video presentation operations can be classified into two categories: low motion exhibition and true motion exhibition respectively corresponding to low motion status and high motion status, which can be distinguished by low motion status and high motion status can be distinguished by the weighted average motion vector length of all MacroBlocks inside one frame MV AC ⁇ -
• a threshold T MOTION can be selected to do classification, if MV AC T is less than T M OTI O N, the low motion status is determined, or else the fast motion status is determined.
• the first one is to directly exposit the salient objects or salient object groups on the display
• the second one is called as gravity flowing show, which control the movement of display area by following the movement of the gravity point of the salient object group, usually tolerance of gravity change (TGC) parameters are used to keep a smooth display strategy
• TGC tolerance of gravity change
• the third one is basically a pan operation with the consideration of saliency distribution to display the salient area on limited display window, especially in case of large saliency object or multiple saliency objects exist.
• a predefined threshold T MOTION ⁇ the true motion exhibition is introduced to display the salient objects or salient object group.
• the gravity point of the OG Object Group
• the gravity point moves forwards and backwards, a weighted average gravity point for the scene of the video will be used as a still focus centre of the display, the middle of the forward gravity point and backward one is approximately shown in the center of the display, therefore the presentation of the video can be viewed with the true motion exhibition mode on the display, i.e.
• the viewer can see the OG moving forwards and backwards on the display window.
• the gravity point moves along a certain direction rapidly, then the weighted average gravity point for the scene of the video will be determined as the still focus centre of the display, the viewer can see the OG moving from one side to the other side of the display window.
• the video can be treated as an information gravity point flowing plane, in which different salient objects have different weights of importance of the information, and the MBs have the same characteristics inside each salient object. Therefore, it's the gravity point but not the center point of the salient object or group should be the center of the display.
• the small display should focus on the area centralized by the gravity point of the group or a salient object, or progressively display the area by using the panning operation, which depends on the density distribution of the information.
• the STP (spatial-temporal processing) module is the most important module in the AVP framework. Optimal spatial-temporal operations will be taken in the module to guarantee a smooth and acceptable video viewing experience .
• Table-II demonstrates a sample of decisions of AVP operations, and of course some other types of combination can be considered due to the detail requirement of real application.
• DS means display size of the corresponding display device.
• Fig. 6 demonstrates the flowchart of one exemplary scheme for decisions of the adaptive video presentation solution in accordance with the present invention.
• step 100 the motion status of the scene of the video is determined by comparing the weighted average motion vector length for frames MV AC ⁇ with the predefined threshold T MOTION - In case the MV AC ⁇ is less than the predefined threshold T MOTIO ⁇ O then the next step goes to step 200, or else to step 400.
• step 220 it will determine that if the RZG is equal or larger then the DS, if the RZG is less than the DS but larger than the DS/n, then in step 230 the extracted window with the RZG will be directly exposited on the display, if the RZG is larger than the DS, then in step 240, it will determine whether the length of scene LOS is less then the minimum perceptual time MPT. Then in step 250, it will determine whether the salient object group contains only one salient object.
• step 260 In a condition that only one salient object exists and the LOS is less than the MPT, the video will be presented on the display in a gravity flowing shown operation with appropriate zoom-out operation, in step 260.
• step 270 case multiple salient objects exist and the LOS is less than the MPT, the video will be directly exposited on the display, since in this condition the pan operation is forbidden to avoid frequent changing of the presentation operation so as to smooth the viewing experience.
• the video will be presented in the gravity flowing show operation along with the saliency driven pan operation without zoom-out.
• the decision scheme of true motion exhibition mode is made through steps 400 to 440.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Multimedia (AREA)
• Databases & Information Systems (AREA)
• General Engineering & Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Signal Processing (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Human Computer Interaction (AREA)
• Controls And Circuits For Display Device (AREA)
• Studio Circuits (AREA)
• Transforming Electric Information Into Light Information (AREA)

Priority Applications (8)

Applications Claiming Priority (1)

Publications (1)

Family

ID=39156807

Family Applications (2)

Family Applications After (1)

Country Status (6)

Families Citing this family (14)

Citations (3)

Family Cites Families (16)

• 2006
  • 2006-09-01 WO PCT/CN2006/002261 patent/WO2008028334A1/en not_active Ceased
• 2007
  • 2007-09-03 JP JP2009525901A patent/JP2010503006A/ja active Pending
  • 2007-09-03 US US12/310,461 patent/US8605113B2/en not_active Expired - Fee Related
  • 2007-09-03 EP EP07800849.7A patent/EP2057531A4/en not_active Withdrawn
  • 2007-09-03 CN CN2007800318436A patent/CN101535941B/zh not_active Expired - Fee Related
  • 2007-09-03 KR KR1020097004095A patent/KR101414669B1/ko not_active Expired - Fee Related
  • 2007-09-03 WO PCT/CN2007/002632 patent/WO2008040150A1/en not_active Ceased
• 2014
  • 2014-03-04 JP JP2014041700A patent/JP2014139681A/ja active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events