Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/47—End-user applications
  • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
  • H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
  • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display

Definitions

• the invention relates to a video indexing method, and a video indexing device,
• ROI regions of interest
• coding applications often decode regions of interest and deploy more resources for coding these regions.
• the detection of regions of interest is today principally used prior to coding in a such a manner as to privilege the regions of interest during coding by according them more bandwidth, for example by reducing the quantization step for these regions.
• the present invention is principally concerned not with the detection of regions of interest, but rather with the transmission of these regions of interest to the devices or applications that take them into account for different applications and can at least resolve the picture display problem on a terminal with a low display capacity, whether mobile or not.
• the present invention proposes a method for indexing a coded video data stream.
• the video data stream comprises information relative to the location of regions of interest of each picture, the method comprises steps of:
• the selected regions of interest are recorded in a temporary memory as they are being selected and decoded, - when all the selected regions of interest are recorded in the temporary memory, the selected regions of interest are transferred to a permanent memory support (503).
• the regions of interest are formatted in order to obtain a homogenous size for all the selected regions of interest.
• the method comprises a step of encrypting the location of the regions of interest thanks to an encryption key.
• the method comprises a step of obtaining a decryption key upon payment by the user.
• the video data stream is coded according to the coding standard H.264/AVC and the location information is contained in a Supplemental Enhancement Information (SEI) type message.
• SEI Supplemental Enhancement Information
• the SEI messages are encapsulated into real-time protocol packets (RTP), the RTP packets being encrypted.
• the Supplemental Enhancement Information type messages relative to regions of interest location information are inserted in the coded data before or after each picture to which they refer.
• the location information comprises information chosen from:
• the selection step of a region of interest per picture selects a region of interest according to the weight relative to the importance of the region of interest.
• the video coding standard uses flexible macro-bloc ordering, the regions of interest being coded into slice groups, independently from the other picture data, the location information of regions of interest comprising the slice group numbers in which the regions of interest are coded.
• the Supplemental Enhancement Information message comprises an identifier indicating for each slice group if it is related to one region of interest.
• the method comprises a further step of reading the SEI messages and in that the step of decoding of video data decodes only the slice groups containing the region of interests.
• the invention concerns also a device for indexing a coded video data stream.
• the video data stream comprises information relative to the location of regions of interest of each picture
• the device comprises means for:
• the detection of the regions of interest of a picture is made in general prior to coding. This data is then used to facilitate the encoding.
• the inventors realized that the location of regions of interest can also be of interest during the decoding of a picture and particularly during the display on a device whose display capacity is limited. In fact, the reception terminal can in fact choose to display the regions of interest only, which enables having a better visibility of these regions relative to the display of the complete picture.
• FIG. 1 shows a coding device according to a preferred embodiment of the invention
• - figure 2 shows a coding method according to a preferred embodiment of the invention
• - figure 3 shows a decoding device according to a preferred embodiment of the invention
• - figure 4 shows a decoding method according to another embodiment of the invention
• - figure 5 shows a personal recording type device according to another embodiment of the invention
• FIG. 6 shows an indexing method in a personal recording type device implementing an embodiment of the invention.
• Figure 1 shows a coding device in accordance with the coding standard H.264/AVC implementing a preferred embodiment of the invention.
• a video stream is coded.
• a current frame F n is presented at the coder input to be coded by it.
• This frame is coded in the form of slices, namely it is divided into sub-units which each contain a certain number of macroblocks corresponding to groups of 16x16 pixels.
• Each macroblock is coded in intra or inter mode. Whether in intra mode or inter mode, a macroblock is coded by being based on a reconstructed frame.
• a module 109 decides the coding mode in intra mode of the current picture, according to the content of the picture.
• P shown in figure 2 comprises samples of the current frame Fn that were previously coded, decoded and reconstructed (uF'n on figure 2, u meaning non-filtered).
• inter mode P is comprised from a motion estimation based on one or more F' n- i frames.
• a motion estimation module 101 establishes an estimation of motion between the current frame Fn and at least one preceding frame F'n-1. From this motion estimation, a motion compensation module 102 produces a frame P when the current picture Fn must be coded in inter mode. A subtractor 103 produces a signal Dn, the difference between the picture Fn to be coded and the picture P. Then this picture is transformed by a DCT transform in a module 104. The transformed picture is then quantized by a quantization module 105. Then, the pictures are reorganized by a module 111.
• a CABAC (Context-based Adaptive Binary Arithmetic Coding) type entropic coding module 112 then codes each picture.
• the modules 106 and 107 respectively of quantization and inverse transformation enable a difference D'n to be reconstituted after transformation and quantization then inverse quantization and inverse transformation.
• an intra prediction module 108 codes the picture.
• a uF'n picture is obtained at the adder output 114, as is the sum of the D'n signal and the P signal.
• This module 108 also receives at input the reconstructed non-filtered F'n picture.
• a filter module 110 can obtain an F'n picture reconstructed and filtered from a uF'n picture.
• the entropic decoding module 112 transmits the coded slices encapsulated in NAL type units.
• the NALs contain, as well as the slices, information relating to the headers for example.
• the NAL type units are transmitted to a module 113.
• a module 116 enables the regions of interest to be determined.
• the means 116 then establish a salience map for each picture of the video.
• parameters entered by the user can also be taken into account. For example, it is possible to define, according to the event to which the video is related, certain important objects of the filmed scene and particularly for sporting events to specify that it concerns a football match.
• this allows a salience map to be obtained that weights the salience zones according to the event. In a football match, it would be preferable to focus on the ball rather than on the terraces.
• the region of interest module therefore enables one or more salient zones to be extracted, also referred to as regions of interest. These regions of interest are then geographically located on the picture. They are identified by their coordinates according to the height and width of the picture. Their size can also be extracted for each of the regions of interest. It is also possible to associate them with an element of semantic information. In fact for a football match, one may require information on a region of interest if the user can select the regions of interest to be displayed from a choice of several regions of interest to be displayed.
• the module 115 receives information relating to the regions of interest in order to code them into an SEI ("Supplemental Enhancement Information") type message.
• SEI Supplemental Enhancement Information
• the SEI message is coded as indicated in the table below:
• uuid_iso_iec_11578 single word of 128 bits to indicate our message type to the decoder.
• user_data_payload_byte 8 bits comprising a part of the SEI massage.
• payloadSize 17 (bytes) thus 16 for the UUID and 1 for the proprietary data.
• number_of_ROI Number of regions of interest present in the picture
• roi_x_16 Position X in the picture of the region of interest, in multiples of 16 pixels.
• roi_y_16 Position Y in the picture of the region of interest, in multiples of 16 pixels.
• roi_w_16 Width in the picture of the region of interest, in multiples of
• semantic_information title characterizing the region of interest.
• Relative weights gives the weight of each region of interest of the picture in such a way to know which region of interest that has in principle the most interest.
• Macroblock_alignment gives the number of the starting macroblock in which the region of interest is found, as well as the size of the region of interest in number of macroblocks, in width and in height.
• the regions of interest are classified as salient if their salience is higher than a certain threshold predetermined by the method for obtaining salience maps.
• the regions of interest are classed in increasing order of salience for all regions where the salience is higher than a fixed threshold.
• the module 113 inserts the SEI message into the data stream and sends the video stream thus coded to the transmission network.
• An SEI message is transmitted before each picture to which it refers. In other embodiments, it is also possible to transmit the SEI message only when the location of at least one region of interest changes between two or more pictures. Hence, during decoding, the decoder takes into account the last SEI message received, whether it is immediately before the picture to be decoded or if it relates to a picture previously received if the current picture is not preceded by such an SEI message.
• Figure 2 shows a coding method in accordance with the coding standard H.264/AVC implementing a preferred embodiment of the invention.
• the salience map associated with the video to be broadcast is determined.
• information relating to the video content can also be received to take account of this information during the establishment of the salience map.
• the position of the ball corresponds to a region of interest for the user and in this case, privilege the zones of the picture in which the ball is situated.
• the presenter corresponds to a region of interest, and in this case, determine the regions of interest by privileging the zones containing the presenter by detecting for example the face using known picture processing techniques.
• one or more regions of interest relating to the video content are thus obtained.
• the coordinates of the regions of interest in the pictures are determined.
• the size of the regions of interest can also be determined in pixels and semantic information on the content can be associated with each region of interest.
• the video stream is coded according to the coding standard H.264.
• zones are privileged that were detected as regions of interest.
• a lower quantization step is applied to them.
• an SEI message is created from location and semantic information associated with the regions of interest.
• the SEI message thus created is in accordance with the SEI message previously described in tables 1 and 2.
• the stream is constituted by inserting SEI messages into the stream to obtain a coded stream according to the H.264 standard.
• the video stream thus coded is transmitted to decoding devices in real time or in a deferred manner during a step E6, the decoding devices can be local or remote.
• Figure 3 represents a preferred embodiment of a decoding device according to the invention, in accordance with the coding standard H.264/AVC.
• a 209 module receives SEI messages at the input. It extracts the different SEI messages.
• the NALs of useful data are transmitted to an entropic decoding module 201.
• the SEI messages are analyzed by a module 210. This module enables decoding of the content of SEI messages representative of the regions of interest. The regions of interest of each picture are thus identified at the level of the decoding device in a simple manner and prior to the decoding of each picture using information contained in the field macroblock_alignment.
• the macroblocks are transmitted to a re-ordering module 202 to obtain a set of coefficients. These coefficients undergo an inverse quantization in the module 203 and an inverse DCT transformation in the module 204 at the output of which D'n macroblocks are obtained, D'n being a deformed version of Dn.
• a predictive block P is added to D'n, by an adder 205, to reconstruct a macroblock uF'n.
• the block P is obtained after motion compensation, carried out by a module 208, of the preceding decoded frame, during a coding in inter mode or after intra prediction of the macroblock uF'n, by the module 207, in the case of coding in intra mode.
• a filter 206 is applied to the signal uF'n to reduce the effects of the distortion and the reconstructed frame F'n is created from a series of macroblocks.
• SEI messages the blocks representative of regions of interest are detected in the stream and prior to display, these blocks are identified and can be cropped according to the choice of the user and transmitted for display to a device such as a PDA, or mobile telephone.
• this region of interest is displayed in zoom on the screen to take up the full screen.
• the decoding device thus only decodes the macroblocks likely to contain information of interest to the user. In this way the decoding is faster and requires less resources at the level of the decoding device and therefore at reception. This is particularly advantageous when the receiving device is a mobile terminal comprising limited processing capacity.
• Figure 4 shows a decoding method in accordance with the coding standard H.264/AVC implementing a preferred embodiment of the invention.
• Such a method can be implemented in a mobile terminal having a limited display capacity.
• step S1 the type of display required is selected.
• the selection is made by means of the user interface present on the mobile terminal. Either it is decided to function in full picture mode and in this case the integrality of the video stream is displayed as it is transmitted by the transmitter. Or it is decided to display the only the regions of interest of the picture. This particular mode constitutes the particularity of the invention.
• step S2 if not it passes to step S8.
• step S8 it is understood that different types of SEI messages can be inserted into the video stream for other applications and in this case, prior to step S8 or during step S8, there can be a step of SEI message analysis.
• the user selects the use that he wants to make of the regions of interest. Particularly, he can select: - the maximum number of regions of interest that he wants to display.
• the regions of interest whose "semantic information" field comprises the keyword are also possible to specify whether it is required to display a single region of interest per picture comprising the keyword (and in this case those for which the salience is maximum) or several regions of interest comprising the key word.
• the SEI messages present in the stream are analyzed as they are being received.
• the SEI message is used to code the location of regions of interest of the picture as they were detected prior to the picture coding. Hence for each picture, there can be one or more regions of interest according to the visual properties of the picture or according to picture content or both.
• the SEI message is coded according to the tables 1 and 2 previously described. Information relating to SEI messages is recorded temporally up until the display of the corresponding picture.
• the pictures are all decoded in conformance with the decoding standard.
• the decoded regions of interest are processed according to those that the user selected during the S2 step. If the user selects a zoom of the principle region of interest of the picture, then during step S6, the zone is magnified so as to reach the maximum size of display. If the user has selected a mosaic of regions of interest then the picture is recomposed of regions of interest, each being magnified according to the screen size and the number of regions of interest selected for display. If the user has specified a keyword, then the regions of interest comprising the keyword are displayed and zoomed. During a step S7, the regions of interest are displayed on the screen of the mobile terminal, according to the user's desire.
• step a S8 following a non-selection by the user to display only the regions of interest, the entire video stream is decoded for display.
• Figure 5 shows a video indexing application of the invention.
• FIG. 5 partially shows a personal recorder (PVR) type device 500.
• PVR personal recorder
• the PVR 500 receives a compressed video stream at its input.
• this video data stream is in accordance with the coding standard H.264.
• the compressed video stream comprises particularly
• This video data stream is partly transmitted to a recording support 503.
• Recording support can be understood as hard disk, holographic support, memory card or "blue ray” disk. This recording support can be remote in other embodiments.
• the video data stream is transmitted in another part to a decoder 501 to be decoded in real time, this for example to be displayed on a television set.
• the stream is transmitted to the decoder 501 when the user wants to view it in real time. If not, it is not decoded but simply recorded, when recording is requested.
• the present invention offers to decode part of the video data stream, even when viewing in real time is not requested.
• a part of the video stream it is understood particularly the regions of interest or certain regions of interest.
• the decoder 501 When the decoder 501 receives a video stream for which a recording is requested, the data is transmitted to the recording support 503.
• the recording support 503 records the data as it is received.
• the decoder 501 receives the video data stream and progressively decodes the SEI messages.
• the decoded regions of interest are transmitted to the video indexing module 502 responsible for their temporary recording before transmitting them to the recording support 503.
• Figure 6 illustrates the method implemented by the decoder 501 and the indexing module 502.
• the video data stream is received by the decoder 501.
• the decoder 501 decodes the SEI messages present in the video data stream.
• the decoded SEI messages are SEI messages as previously described in the tables 1 and 2.
• the decoder can also decode other SEI messages but that is not the object of the present invention.
• Each SEI message can describe one or more regions of interest per picture as described in tables 1 and 2.
• the decoder 501 analyzes each SEI message and decodes each picture. During this step, the weight indicated in the SEI message is used to select which region of interest will be recorded for each picture. In a preferred embodiment, the region of interest with the maximum of salience is kept, i.e having the highest weight.
• the indexing module 502. decides which picture is used to index the video. According to the preferred embodiment described here, only about 10 pictures will be selected for a video of one and a half hours. It can be imagined that in other embodiments the number of pictures will be greater. These 10 pictures are taken at regular intervals. These selected pictures are recorded temporarily in a RAM type memory comprised in the indexing module 502 and not shown.
• the pictures are zoomed during a step T5, that is they are enlarged so that they are all the same size.
• this size can be the size of the picture.
• they are read in the temporary memory and re-recorded after their enlargement.
• the pictures are enlarged prior to their recording in the temporary memory.
• the images are presented as a mosaic on the display. Therefore, instead of being enlarged, the images are reduced to one single size, the same for all of them.
• the indexing pictures are also transferred from the temporary memory to the recording support 503 and recorded in a file.
• the regions of interest are used for the indexation or can also be used for display on a PVR type device when the user wants to consult the content of the database.
• this encrypting step in respect of figure 2, would be a step E4' (not shown) but inserted after the step E4.
• Obtaining of the decryption key could be the object of a paid for service from the programme broadcaster for example.
• the SEI messages relating to regions of interest are encapsulated in RTP (Real Time Protocol) type packets and transmitted on a different video port.
• Temporal CTS type labels can link the SEI messages relating to regions of interest with corresponding pictures.
• this transmission mode enables encrypting only RTP packets containing the SEI messages and not the video.
• the decryption is carried out at the level of the terminal receiver.
• the encrypting standard used is DVB-CSA and SEI messages relating to regions of interest are encapsulated in a different PID than that of the video.
• the SEI messages relating to regions of interest are linked to corresponding pictures via the PTS (timestamp) of the PES packet header. This transmission mode allows encryption only of the PIDs that contain SEI messages relating to regions of interest and not the video PID.
• the video data stream is coded in accordance with the coding standard H.264/AVC using FMO (Flexible Macroblock Ordering) which enables coding of different parts of the picture independently and so decoding of them independently.
• FMO Flexible Macroblock Ordering
• the FMO mode uses "slice groups".
• the "slice groups” are defined in the standard.
• the regions of interest are coded in groups different from the rest of the picture.
• a PPS type NAL comprises a map of "slice groups”. SEI messages are inserted such as those described hereafter indicating in which "slice groups" the regions of interest are coded.
• uuid_iso_iec_11578 single word of 128 bits to indicate our message type to the decoder.
• user_data_payload_byte 8 bits comprising a part of the SEI message.
• a semantic information For each slice_group representing a region of interest, a semantic information, a relative weight and which macroblock it concerns can be specified.

Applications Claiming Priority (2)

Publications (1)

Family

ID=39204994

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (24)

Citations (1)

Family Cites Families (11)

• 2008
  • 2008-06-25 WO PCT/EP2008/058050 patent/WO2009003885A2/en active Application Filing
  • 2008-06-25 EP EP08761351A patent/EP2174500A2/en not_active Ceased
  • 2008-06-25 CN CN200880022001.9A patent/CN101690228B/zh not_active Expired - Fee Related
  • 2008-06-25 JP JP2010513897A patent/JP5346338B2/ja not_active Expired - Fee Related
  • 2008-06-25 KR KR1020107002047A patent/KR101488548B1/ko not_active IP Right Cessation

Patent Citations (1)

Also Published As

Similar Documents

Legal Events