Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
  • H04N19/109—Selection of coding mode or of prediction mode among a plurality of temporal predictive coding modes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
  • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
  • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process

Definitions

• the present invention relates to the field of video stream encoding .
• pan/tilt/ zoom (PTZ) functions result in a loss of quality or resolution of the enlarged image portion.
• Known server-side pan/tilt/ zoom (PTZ) functions result in a massive amount of computation at the server side.
• the method comprising: obtaining picture element information pertaining to the selected spatial portion; obtaining encoding hints derived from a complementary spatial portion of the original video stream that is peripheral to the selected spatial portion; and encoding the selected spatial portion with use of the encoding hints.
• the encoding process for the selected portion can be made more efficient by taking into account information about the bigger picture, in particular, features of the original video stream appearing in a region outside the selected portion, but close enough to be of a nature to influence the coding of subsequent frames.
• Relevant peripheral features are preferably selected on the basis of the motion of these features, and of the "motion" of the selected portion; i.e., the motion of the image that is induced by any panning, tilting, or zooming with respect to the original video stream.
• the method according to the present invention further comprises obtaining the original video stream; obtaining selection information representing the selected spatial portion of the stand-alone video-stream; and extracting the picture element information pertaining to the selected spatial portion from the original video stream in accordance with the selection information.
• the information of the original video stream is used to generate the reduced video, such that the quality (e.g., resolution) of the original video stream can be retained to the greatest possible extent in the reduced video stream.
• the method according to the present invention further comprises obtaining motion and feature information pertaining to the original video stream;
• the motion adapter is configured to identify the relevant features by comparing a motion vector of a candidate feature to a motion vector
• peripheral to the selected portion can be detected as moving into the region of interest, in which case their appearance can be anticipated, and new values may be selected for variable encoding parameters so as to optimally deal with the anticipated presence of that feature.
• the encoding is performed by means of a scalable video codec.
• the original video stream is provided as the base layer for the encoding.
• the scalable video codec is an H.264 SVC codec.
• the obtaining of the picture element information pertaining to the selected spatial portion comprises
• the original video stream and/or the selected spatial portion may originally be available as encoded streams only. In that case, judicious transcoding would be more efficient than decoding and recoding of the raw video stream.
• a computer program adapted to perform, when executed, the method according to embodiments of the present invention.
• a computer readable storage medium comprising instructions to cause a data processing apparatus to carry out steps of the method according to embodiments of the present invention.
• an apparatus for encoding a selected spatial portion of an original video stream as a stand-alone video stream comprising: a video input interface for receiving the original video stream; a selection information input interface for receiving selection information representing the selected spatial portion; a motion and feature
• a panorama reframer operatively coupled to the video input interface and the selection information input interface, the panorama reframer being configured to extract picture element information pertaining to the selected spatial portion from the original video stream in accordance with the selection information;
• a motion adapter operatively coupled to the motion and feature information input
• the motion adapter being configured to identify relevant
• a hinted encoder operatively coupled to the panorama reframer and the motion adapter, the hinted encoder being configured to encode the selected spatial portion with use of the encoding hints.
• the hinted encoder is configured to encode the selected spatial portion by means of a scalable video codec.
• the hinted encoder is configured to provide the original video stream as the base layer for the encoding.
• the scalable video codec is an H.264 SVC codec.
• the video input interface is configured to obtain the video stream as an encoded video stream, and wherein the hinted encoder is configured to transcode the encoded video stream.
• the invention further provides a system for encoding a selected spatial portion of an original video stream as a stand-alone video stream, the system comprising a feature analyzer configured to extract motion information pertaining to features in the peripheral portion, and a number of apparatus according to any of claims 9-13, the feature analyzer being coupled to respective motion and feature information interfaces of the apparatus.
• Figure 1 provides a flow chart of a method according to an embodiment of the present invention
• Figure 2 provides a schematic illustration of an apparatus and a system according to embodiments of the present invention
• Figure 3 illustrates the motion vector comparison performed in embodiments of the present invention.
• the invention is related to a system composed of a proxy serving multiple users which can request a personalized region of interest (Rol), in a pan-tilt-zoom (PTZ)
• Rol personalized region of interest
• PTZ pan-tilt-zoom
• the invention is further related to the optimization of the computational cost at the proxy side of the video encoding of these personalized video streams from the same video panorama source.
• the term "panorama” is generally used herein to designate a high-resolution video resolution, typically above 4000 pixels wide, which may contain a cylindrical or spherical mapping of a wide-angle view or representation of a physical space as a video.
• the panorama may be composed from the different video sources which are integrated and fused together.
• Current solutions perform user requested cropping and PTZ operations in order to generate the desired Rol frame at time t in raw data and encode it using a H.264, or WebM alike codec with motion compensation and/or intra coding. Unfortunately, using this technique, motion
• Rol-relative motion can then be computed from these panorama motion vectors by compensating the possible Rol user-requested displacement and resolution changes.
• these compensated motion vectors are sent as hints to the encoder devoted to the coding of that Rol. The encoder can refine this motion vector "hint" if necessary.
• Figure 1 provides a flow chart of a method according to an embodiment of the present invention.
• the skilled person shall appreciate that the various illustrated steps are not necessarily performed by a single entity.
• steps shown as taking place in parallel may be conducted
• the original video stream is obtained 100 and used on the one hand to extract the relevant picture element information for the selected region 130, and on the other hand to extract motion and feature information 150.
• a region of interest Rol
• the selection action may be limited to panning, zooming, and tilting with respect to the original video stream, wherein the aspect ratio of the selected region is preferably limited to a fixed aspect ratio associated with the viewing device (e.g., 3x4 or 16x9) .
• Picture element information is meant to comprise any form of representation of the video images within the selected region. This may be a stream of uncompressed video images, or an encoded video stream. Embodiments of the method according to the invention present the advantage are that motion is computed at highest
• Embodiments of the method according to the invention present the further advantage that overlapping Rols or aligned Rols for different users do not require a duplication of the motion estimation effort for their respective encoders as motion data is pre-computed on the panorama .
• Encoding hints are derived 160 from the extracted motion and feature information. While the motion and feature
• the encoding hint derivation selects those features that are relevant for improving the encoding performance, as will be described in more detail below.
• the encoder obtains the picture element information
• the encoder encodes 180 the selected portion of the video stream.
• the encoding hints are bits of information derived from the non-selected portion of the original video stream, used in addition to the information about the selected portion, to improve the encoding of the latter.
• methods according to embodiments of the present invention track navigation and zoom commands from the client (at step 120) , and then scale the global motion data in response to zoom commands and add Rol motion in response to panning or tilting commands (at step 160) .
• the encoding (at step 180) takes place on the basis of the obtained set of motion vectors, which are used either as hints as or actual motion vectors: If the reference frame area is not available in the encoder, a predicted motion vector and a search window range are sent to the encoder or an intra prediction mode.
• the aforementioned method may in general be carried out by an apparatus for encoding a selected spatial portion of an original video stream as a stand-alone video stream, the apparatus comprising means for obtaining picture element information 140 pertaining to the selected spatial portion; means for obtaining encoding hints 170 derived from a complementary spatial portion of the original video stream that is peripheral to the selected spatial portion; and means for encoding the selected spatial portion 180 with use of the encoding hints.
• the apparatus may further comprise means for obtaining the original video stream 100; means for obtaining selection information 120 representing the selected spatial portion of the stand-alone video-stream; and means for extracting the picture element information 130 pertaining to the selected spatial portion from the original video stream in accordance with the selection information.
• the apparatus may further comprise means for obtaining motion and feature information 150 pertaining to the original video stream; means for identifying relevant features pertaining to a region that is peripheral to the selected spatial portion in the motion and feature information; and means for deriving the encoding hints 160 from the identified relevant features.
• the means for encoding 180 may operate with a scalable video codec. More specifically, the means for encoding 180 may provide the original video stream as the base layer for the encoding. Also more specifically, the scalable video codec may be an H.264 SVC codec. The means for obtaining the picture element information 140 pertaining to the selected spatial portion, may be adapted to obtain an encoded video stream, and the encoding means 180 may be adapted to transcode the encoded video stream.
• Figure 2 provides a schematic illustration of an apparatus and a system according to embodiments of the present invention.
• the apparatus 200 comprises a video input interface 211 for receiving said original video stream 199, a selection information input interface 232 for receiving selection information representing said selected spatial portion, a motion and feature information input interface 221 for receiving feature information derived from said original video stream, a panorama reframer 211, operatively coupled to said video input interface 211 and said selection information input interface 232, said panorama reframer 211 being configured to extract picture element information pertaining to said selected spatial portion from said original video stream in accordance with said selection information, a motion adapter 220, operatively coupled to said motion and feature information input interface 221 and said selection information input interface 232, said motion adapter 220 being configured to identify relevant features pertaining to a region that is peripheral to said selected spatial portion in said motion and feature information and to derive encoding hints from said identified relevant features, and a hinted encoder 230, operatively coupled to said panorama reframer 210 and said motion adapter 220, said hinted encoder 230 being configured to encode said selected spatial portion with use of said
• the apparatus 200 further has an output interface 231 for sending the encoded selected video stream to a client 300 over a network 250.
• the illustrated network 250 may consist of one or more network links. It typically includes an access link on the client side.
• interface designates the necessary hardware and software required to establish data communication
• An access interface may for instance include an interface for an access link such as an xDSL, xPON, WMAN, or 3G link.
• a LAN interface may for instance include an interface for one or more of an IEEE 802.3 "Ethernet" link, an IEEE 802.11
• a PAN interface may for instance include a USB interface or a Bluetooth interface.
• Higher- layer protocols for communication over several network segments are preferably protocols from the TCP/IP protocol suite .
• the client 300 comprises a standard decoder 310 for decoding the received video stream.
• the client 300 further comprises means 320 to allow the end user to select a region of interest, and to feed the resulting selection information back to the apparatus 200.
• a system according to the invention may comprise a motion and feature analyzer 240 operating on the original video stream 199.
• this motion and feature analyzer 240 is centralized, or located "in the cloud", and offers its output to one or several apparatus 200 according to the invention.
• Adaptive Rol Encoder 200 comprises a motion adaptation module 220, a panorama video reframing module 210 and a hinted encoder 230 delivering a client 300 as represented in Fig. 2.
• the panorama motion analyzer 240 receives a raw or encoded panorama video stream 199 and delivers motion information to the motion adaptation module 220 of the N Adaptive Rol encoders 200. If the panorama analyzer 240 performs its task in the compressed domain, the reframing module 210 shall allow for the necessary partial decoding. If the encoder 230 is an MPEG-7 compliant transcoder, hinting information may be provided as MPEG-7 transcoding hints to the encoder 230. Such a transcoder is known from Peter M. Kuhn, Teruhiko Suzuki and Anthony Vetro, "MPEG-7 Transcoding Hints for Reduced Complexity and Improved Quality, " in Proceeding of PacketVideo '01, 2001.
• Each motion adaptation module 220 receives motion and feature information from the panorama motion analyzer 240 and the Rol position change requests from its client 300. Based on this information, the module 220 outputs hints to its connected hinted encoder 230.
• the panorama reframing module 210 also reads the Rol request and re-samples and crops the requested region of the panorama and prepares it for the hinted encoder 230 in the requested position and resolution. If the panorama region needs to be mapped in 2D (e.g. cylindrical map for spherical video) then this is also done in the panorama reframing module 210.
• Each hinted encoder 230 receives hints from its motion adaptation module 220 as well as raw cropped video stream at the requested resolution and position.
• the hints are
• the hints can be composed of a direct motion vector, a mode decision (motion partition) , a predicted motion vector on which the search window is to be centered as well as the size of this search window, an intra mode prediction, a skip decision, etc.
• a mode decision motion partition
• An example of this can be found in Ralph A.
• C C r + C m with C m > C r (typically C m » C r ) .
• the computational complexity of the panorama analysis (step 150 of Figure 1; the function of the analyzer 240 in Figure 2) is denoted as C p with C p > C.
• serving N clients results in a complexity of only C p + N*C r , instead of the classical linearity in function of ( C p + C r ) ⁇ Adding one client only adds an increment of Cr which is much smaller than the term C m + C r in the traditional case .
• the motion information computed by the panorama motion and feature analyzer 240 needs to be adapted to the client request by the motion adaptation module 220. This requires detecting whether the available motion vectors are pointing to panorama areas that are available as reference frames in the hinted encoder 230. Such detection is represented on Figure 3, where the panorama raw video frames motion and feature information are depicted together with one example of Rol request for M frames. Each panorama frame n(t) at time t is analyzed so as to produce for each pixel feature information and motion information in the panorama motion and feature analysis module 240.
• Feature information is composed of e.g. edge position and orientation information, a segmentation of the panorama together with SIFT
• motion information is represented as one motion vector for each of the M previous frames n(t-l)... ⁇ ( t-M) to serve as reference frames.
• the motion adaptation module 220 receives the Ro l request and computes the Ro l position changes in the panorama for the previous M reference frames used by the hinted encoder 230.
• the Ro l motion changes are represented by the vectors denoted as r in Figure 3, where the current Ro l macroblocks are represented as a regular grid and previous Ro l frame positions are represented as dashed rectangular regions (with width w(t) and height h(t) ) .
• the motion adaptation module then checks for every other motion adaptation module.
• the Rol request from the client 300 consists in a Rol
• This information is used by the motion adaptation module 220 and by the panorama reframing module 210.
• This module 210 This module 210
• these clients 300 can be aggregated as virtual super-clients. This is for instance possible if the Rol displacement is guided by a script (e.g. tracking of an object or person or
• the hinted encoder uses a scalable encoding technique, such as the scalable extension of H.264, SVC. This enables adaptation of the stream to the needs of the client 300 while still reducing the scalable encoding technique
• the panorama motion analyzer 240 then specifically outputs multi- resolution motion and feature information so as to enable this.
• the method according to the invention includes an additional step to estimate whether it is more efficient to use the motion analysis on the full panorama than to encode each Rol separately without hints.
• a computation is performed to determine the minimum number N p of clients for which the panorama motion analysis (step 150) cost is compensated by the higher scalability.
• N p is given by the following formula:
• processors may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
• the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of
• controller should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non volatile storage.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array
• ROM read only memory
• RAM random access memory
• any switches shown in the FIGS are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
• program storage devices e.g., digital data storage media, which are machine or computer readable and encode machine-executable or computer-executable
• the program storage devices may be, e.g., digital memories, magnetic storage media such as a magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media.
• the embodiments are also intended to cover computers programmed to perform said steps of the above- described methods.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=47988985

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (24)

Citations (2)

Family Cites Families (16)

• 2012
  • 2012-03-30 EP EP20120305389 patent/EP2645713A1/en not_active Ceased
• 2013
  • 2013-03-25 US US14/389,033 patent/US20150117524A1/en not_active Abandoned
  • 2013-03-25 JP JP2015502259A patent/JP6121518B2/ja active Active
  • 2013-03-25 WO PCT/EP2013/056189 patent/WO2013144049A1/en not_active Ceased
  • 2013-03-25 CN CN201380018231.9A patent/CN104365095B/zh not_active Expired - Fee Related
  • 2013-03-25 KR KR1020147030447A patent/KR20150003776A/ko not_active Ceased

Patent Citations (2)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events