EA042523B1 - VIDEO ENCODING SCALED BY FRAME RATE - Google Patents

Description

Cross-references to related applications

This application claims priority to U.S. Provisional Application No. 62/816521 filed March 11, 2019, U.S. Provisional Application No. 62/850985 filed May 21, 2019, U.S. Provisional Application No. 62/883195, filed August 6, 2019, and U.S. Provisional Patent Application No. 62/904744, filed September 24, 2019, each of which is incorporated herein by reference in its entirety.

The field of technology to which the invention belongs

This document generally refers to images. More specifically, an embodiment of the present invention relates to frame rate scalable video coding.

State of the art

As used herein, the term dynamic range (DR) may refer to the ability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luminance signal) in an image, e.g., from the darkest grays (blacks) to the deepest bright white flowers (high lights). In this sense, DR refers to scene-related intensity. DR can also refer to the ability of a display device to adequately or approximately render a specific latitude range of intensity. In this sense, DR refers to display-related intensity. Unless a particular sense is explicitly indicated as having a particular meaning at any point in the description herein, it must be concluded that the term can be used in any sense, such as interchangeably.

As used herein, the term high dynamic range (HDR) refers to a DR breadth that spans 14-15 orders of magnitude of the human visual system (HVS). In practice, DR, over which a person can simultaneously perceive a wide breadth in the range of intensity, may be truncated to a certain extent, relative to HDR.

In practice, images comprise one or more color components (eg, a Y luminance signal and a Cb and Cr chrominance signal), with each color component represented by a precision of n bits per pixel (eg, n=8). When using linear luminance coding, images in which n<8 (eg, color 24-bit JPEG images) are considered standard dynamic range (SDR) images, while images in which n>8 may be considered enhanced dynamic range images. HDR images can also be stored and distributed using high-precision (eg, 16-bit) floating point formats such as the OpenEXR file format developed by Industrial Light and Magic.

Currently, distribution of high dynamic range video content such as Dolby Vision by Dolby Laboratories or HDR10 in Blu-Ray is limited to 4K resolution (e.g. 4096x2160 or 3840x2160 etc.) and 60 frames per second (fps) due to the characteristics of many playback devices. Future versions anticipate that content up to 8K resolution (eg 7680x4320) and 120fps may be available for distribution and playback. It is desirable that future content types be compatible with existing playback devices to simplify the ecosystem of HDR content for playback such as Dolby Vision. Ideally, content creators should be able to adapt and distribute future HDR technologies without having to also extract and distribute special versions of content that are compatible with existing HDR devices (such as HDR10 or Dolby Vision). The inventors should take into account here that improved technologies are required for the scalable distribution of video content, in particular HDR content.

The approaches described in this section are approaches that may be considered, but are not necessarily approaches that are preconceived or considered. Therefore, unless otherwise indicated, any of the approaches described in this section should not be assumed to be prior art simply by virtue of their inclusion in this section. Likewise, matters identified with respect to one or more of the approaches should not be assumed to be recognized in the prior art based on this section unless otherwise noted.

Brief description of the drawings

An embodiment of the present invention is illustrated by way of example and not limitation; in the accompanying drawings, like reference numbers are referred to as like elements.

Fig. 1 illustrates an exemplary process for a video delivery pipeline.

Fig. 2 illustrates an exemplary process for combining successive source frames to render a target frame rate at a target shutter angle according to an embodiment of this invention.

Fig. 3 illustrates an exemplary representation of a variable input frame rate input sequence with a variable shutter angle in a fixed frame rate container according to an embodiment of this invention.

- 1 042523

Fig. 4 illustrates an exemplary view for temporal scalability at various frame rates and backward compatible shutter angles according to an embodiment of this invention.

Detailed Description of Exemplary Embodiments

This document describes exemplary embodiments that relate to frame rate scalability for video coding. In the following description, for purposes of explanation, many specific details are set forth in order to provide a thorough understanding of the various embodiments of the present invention. However, it should be apparent that various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail so as to avoid undue ambiguity, obscuring understanding, or complicating embodiments of the present invention.

The essence of the invention.

The exemplary embodiments described herein relate to frame rate scalability in video coding. In an embodiment, the processor system receives an encoded bitstream containing encoded video frames, wherein one or more encoded frames are encoded at a first frame rate and at a first shutter angle. The processor receives a first flag indicating the presence of a group of encoded frames to be decoded at a second frame rate and a second shutter angle; it accesses the values of the second frame rate and the second shutter angle for the group of coded frames from the encoded bitstream, and generates decoded frames at the second frame rate and the second shutter angle based on the group of coded frames, the first frame rate, the first shutter angle, the second frame rate and a second shutter angle.

In a second embodiment, a decoder with a processor receives an encoded bitstream containing groups of encoded video frames, wherein all encoded video frames in the encoded bitstream are encoded at a first frame rate;

receives a number N of combined frames;

takes a value for the base frame rate;

accesses a group of N consecutive encoded frames, where the i-th encoded frame in the group of N consecutive encoded frames, where i=1, 2,..., N, represents the average of up to i input video frames encoded in the encoder on the base frame rate and at the i-th shutter angle, based on the first shutter angle and the first frame rate;

accesses, from the encoded bitstream or user input, the values of the second frame rate and the second shutter angle to decode a group of N consecutive encoded frames at the second frame rate and the second shutter angle; and generates decoded frames at the second frame rate and at the second shutter angle based on the group of N consecutive encoded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle.

In the third embodiment, the structure of the encoded video stream contains a section of encoded images, including encoding a sequence of video images; and a signaling section including encoding a timeline-based parameter regarding a shutter interval indicating a number of units of time elapsed in one second;

of the clock cycle-based parameter regarding the shutter interval indicating the number of units of time of the clock signal operating at the frequency of the time-based parameter regarding the shutter interval, wherein the clock-based parameter regarding the shutter interval divided by the time scale-based parameter regarding the interval shutter release, indicates the value of the exposure duration;

a shutter interval duration flag indicating whether or not the exposure duration information is fixed for all temporal sublayers in the coded picture section, wherein if the shutter interval duration flag indicates that the exposure duration information is fixed, then the decoded version of the video sequence for all temporal sublayers in the the coded image section is decoded by calculating the exposure duration value based on the timeline-based parameter about the shutter interval and the parameter based on the clock cycles about the shutter interval, otherwise the signaling section includes one or more sublayer parameter arrays, with values in one or more arrays of sublayer parameters, combined with a time-based parameter regarding the shutter interval, are used to calculate for each sublayer o the corresponding exposure duration value in the sublayer to display

- 2 042523 decoded version of the time sublayer of the video sequence.

An example video delivery processing pipeline.

Fig. 1 illustrates an exemplary process of a conventional video delivery pipeline (100) showing the various stages from video capture to display of video content. A sequence of video frames (102) is captured or generated using an imaging unit (105). Video frames (102) may be captured digitally (eg, by means of a digital camera) or generated by a computer (eg, using computer animation) to provide video data (107). Alternatively, video frames (102) may be captured on film by a film camera. The film is digitized to provide video data (107). In the production phase (110), the video data (107) is edited to provide a production video stream (112).

The video data of the production stream (112) is then provided to the processor in block (115) for post-production video editing. Video compositing in block (115) may include adjusting or modifying colors or brightness in specific areas of an image to improve image quality or achieve a specific appearance for an image in accordance with the creative intent of the video creator. This is sometimes called color sync or color grading. Other editing (e.g., selecting and arranging scenes, cropping an image, adding computer-generated visual effects, controlling jitter or blur, frame rate control, etc.) may be performed in block (115) to result in the final production version (117) for distribution. During the layout (115) video video images are viewed on the reference display (125). After assembling (115), the final production video data (117) can be delivered to the encoding unit (120) for downstream delivery to decoding and playback devices such as television receivers, set-top boxes, movie theaters, and the like. In some embodiments, the encoding unit (120) may include audio and video encoders, such as audio and video encoders specified by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate a coded stream (122) bits. At the receiver, the encoded bitstream (122) is decoded by a decoder (130) to generate a decoded signal (132) representing an identical or good approximation of signal (117). The receiver may be coupled to a target display (140) which may have completely different characteristics than the reference display (125). In this case, the display control block (135) may be used to link the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a signal associated with the display (137).

scalable coding.

Scalable coding already forms part of a number of video coding standards such as MPEG-2, AVC and HEVC. In embodiments of this invention, scalable encoding is extended to improve performance and flexibility, in particular as it relates to very high resolution HDR content.

As used herein, the term shutter angle refers to an adjustable shutter setting that controls the proportion of time that film is exposed to light during each frame interval. For example, in an embodiment, the shutter angle exposure time is 360 frame intervals (1)

The term comes from the legacy mechanical rotary shutters; however, modern digital cameras can also adjust their shutter electronically. Filmmakers can use the shutter angle to control the amount of motion blur or judder that is recorded in each frame. It should be noted that instead of using exposure time, alternative terms such as exposure time, shutter interval, and shutter speed can also be used. Similarly, instead of using a frame interval, the term frame duration can be used. Alternatively, you can replace the frame interval with 1/frame rate. The exposure time value is typically less than or equal to the frame duration. For example, a shutter angle of 180° indicates that the exposure time is half the frame duration. In some situations, the exposure time may exceed the frame duration of the encoded video, for example, when the encoded frame rate is 120 frames/s and the frame rate of the associated video content prior to encoding and display is 60 frames/s.

Consider, without limitation, an embodiment in which source content is filmed (or rendered) at a native frame rate (eg, 120 fps) with a shutter angle of 360°. The receiver can then render the video output at a plurality of frame rates equal to or lower than the original frame rate by a reasonable combination of source frames, such as averaging or other operations known in the art.

The combining process can be performed on non-linear coded signals (for example, using gamma, PQ, or HLG), but the best image quality is obtained by combining frames into the line light region first, converting the non-linear coded signals to line light representations after that. , combining the transformed frames, and finally re-encoding the output with a non-linear transfer function. This process provides a more accurate simulation of the physical exposure of the camera than combining in the non-linear domain.

In general terms, the frame combining process can be expressed in terms of source frame rate, target frame rate, target shutter angle, and number of frames to be combined as follows:

n_frames=(target_shutter_angle/360)*(original_frame_rate/target_frame_rate) (2) which is equivalent to the following:

target_shutter_angle=3 60* n_frames * (target_frame_rate/original_frame_rate) (3) where n_frames is the number of combined frames, original_frame_rate is the frame rate of the original content, target_frame_rate is the frame rate to be prepared by rendering (where target_frame_rate<original_frame_rate), and target_shutter_angle specifies the value required motion blur.

In this example, the maximum target_shutter_angle is 360°, which corresponds to the maximum motion blur. The minimum value of target_shutter_angle can be expressed as 360*(target_frame_rate/original_frame_rate) and corresponds to the minimum motion blur. The maximum value of n_frames can be expressed as (original_frame_rate/target_frame_rate). The target_frame_rate and target_shutter_angle values must be chosen such that the value of n_frames is a non-zero integer.

In the particular case in which the original frame rate is 120 fps, Equation (2) can be rewritten as:

n Jramcs=targcl_shutlcr_angle/(3 * target Jramc_ratc) (4) which is equivalent to: target_shutter_angle=3*njrames*target_frame_rate (5)

The relationships between the target_frame_rate, n_frames, and target_shutter_angle values are shown in Table 1. 1 for the case original_frame_rate=120 fps. In table. 1 NA indicates that the corresponding combination of the target frame rate and the number of combined frames is not allowed.

Table 1

Relationship between target frame rate, combined frames, and target shutter angle for a native frame rate of 120 fps

Target frame rate (fps) Number of combined frames 5 4 3 2 1 Target shutter angle (degrees) 24 thirty 40 60 360 NA NA NA 288 360 NA NA 216 270 360 NA 144 180 240 360 72 90 120 180

Fig. 2 illustrates an exemplary process for combining successive source frames to render a target frame rate at a target shutter angle according to an embodiment. Given an input sequence (205) at 120 fps and a shutter angle of 360°, the process generates an output video sequence (210) at 24 fps and a shutter angle of 216° by combining three input frames in a set of five consecutive frames (for example, the first three consecutive frames) and discarding the other two. It should be noted that in some embodiments, output frame 01 (210) may be generated by combining alternative input frames (205) such as frames 1, 3, and 5 or frames 2, 4, and 5, etc.; however, it is expected that combining successive frames should result in better quality video output.

It is desirable to maintain variable frame rate source content, for example, in order to control artistic and stylistic effect. It is also desirable if the variable input frame rate of the source content is packaged in a container that has a fixed frame rate to facilitate content production, exchange and distribution. By way of example, three embodiments are presented with respect to how to represent variable frame rate video data in a fixed frame rate container. For purposes of clarity and without limitation, the descriptions below use a container for a fixed 120 frames/s, but the approaches can easily be extended to a container for an alternative frame rate.

- 4 042523

First Embodiment (Variable Frame Rate).

The first embodiment is an explicit description of a variable (non-constant) frame rate of source content packaged in a container having a constant frame rate. For example, source content that has a different frame rate, say 24, 30, 40, 60, or 120 fps for different scenes, can be packaged in a container having a constant frame rate of 120 fps. According to this example, each input frame can be duplicated 5, 4, 3, 2, or 1 times to package it into a common 120 frame/s container.

Fig. 3 illustrates an example of an input video sequence A with a variable frame rate and a variable shutter angle, which is represented in a fixed frame rate codestream B of bits. Then, at the decoder, the decoder reconstructs the output video sequence C at the desired frame rate and at the desired shutter angle, which may vary depending on the scene. For example, as illustrated in FIG. 3, to construct sequence B, some input frames are duplicated, some are encoded as is (no duplication), and some are copied four times. Then, to construct sequence C, any frame from each set of duplicated frames is selected to generate output frames matching the original frame rate and shutter angle.

In this embodiment, metadata is inserted into the bitstream to indicate the original (base) frame rate and shutter angle. The metadata may be transmitted using a high-level syntax such as a sequence parameter set (SPS), a picture parameter set (PPS), a slice or tile group header, and the like. The presence of metadata enables encoders and decoders to perform useful functions such as the following.

a) The encoder can ignore duplicate frames, thereby increasing the encoding rate and simplifying processing. For example, all coding tree units (CTUs) in duplicated frames may be encoded using skip mode and reference index 0 in reference frame list 0, which refers to the decoded frame from which the duplicated frames are copied.

b) The decoder can bypass the decoding of duplicate frames, thereby simplifying processing. For example, the metadata in the bitstream may indicate that a frame is a copy of a previously decoded frame that a decoder can reproduce by copying and without decoding a new frame.

c) The playback device may optimize downstream processing by specifying a base frame rate, such as by adjusting frame rate conversion algorithms or noise reduction.

This embodiment allows the end user to view rendered content at frame rates intended by content creators. This embodiment does not provide backward compatibility with devices that do not support container frame rates, such as 120 fps.

Tab. 2 and 3 illustrate an exemplary Primary Data Byte Sequence (RBSB) syntax for a sequence parameter set and a tile group header, in which proposed new syntax elements are illustrated in boldface. The remaining syntax corresponds to the syntax in the proposed Universal Video Codec (VVC) specification (reference [2]).

As an example, a flag can be added to the SPS (see Table 2) in order to provide a variable frame rate.

sps_vfr_enabled_flag equal to 1 indicates that the coded video sequence (CVS) can contain variable frame rate content; sps_vfr_enabled_flag equal to 0 indicates that CVS contains fixed frame rate content.

In tile_group_header() (see Table 3), ti le_group_vrf_info_present_flag equal to 1 indicates that the tile_group_true_fr and tile_group_shutterangle syntax elements are present in the syntax;

ti le_group_vrf_info_present_flag equal to 0 indicates that the syntax elements tile_group_true_fr and tile_group_shutter_angle are not present in the syntax (when tile_group_vrf_info_present_flag is not present, it is logically inferred to be 0);

ti le_group_true_fr indicates the true frame rate of the video data carried in this bitstream;

ti le_group_shutter_angle indicates the shutter angle corresponding to the true frame rate of the video data carried in this bitstream;

ti le_group_skip_flag equal to 1 indicates that the current tile group is copied from another tile group;

ti le_group_skip_flag equal to 0 indicates that the current tile group is not copied from another tile group;

tile_group_copy_pic_order_cnt_lsb indicates the number in the modulo image sequence

- 5 042523

MaxPicOrderCntLsb for the previously decoded picture that the current picture copies from when tile_group_skip_flag is set to 1.

table 2

Sample RBSP Syntax for Parameter Set for Variable Frame Rate Content

seq_parameter_set_rbsp() { Descriptor sps_max_sub_layers_minus 1 u(3) sps_reserved_zero_5bits u(5) profile_tier_level(sps_max_sub_layers_minus 1) sps_seq_parameter_set_id ue(v) sp s_vfr_enabled_flag u(l) sps_extension_flag u(l) if(sps_extension_flag) while(more_rbsp_data()) sps_extension_data_flag u(l) rbsp_trailing_bits() }

Table 3

Example of tile group header syntax supporting variable frame rate content

tile_group_header() { Descriptor tile_group_pic_parameter_set_id ue(v) if(NumTiles!nPic>l) { tile_group_address u(v) num_tiles_in_tile_group_minus 1 ue(v) } tile_group_type ue(v) tile_group_pic_order_cnt_lsb u(v) if(sps_vfr_enabled_flag) { tile_group_vfr_info_present_flag u(D if(tile_group_vfr_info_present_flag) { tile_group_true_fr u(9) tile_group_shutterangle u(9) } tile_group_skip_flag u(D } if(tile_group_skip_flag) tile_group_copy_pic_order_cnt_lsb u(v) else{ ALL OTHER TILE_GROUP_SYNTAX } if(num_tiles_in_tile_group_minusl>0) { offset_len_minus 1 ue(v)

- 6 042523

for(i=0; i<num_tiles_in_tile_group_minusl; i++) entry_point_offset_minus 1[i] u(v) 1 byte_alignment() 1

embodiment: fixed frame rate container.

Second

The second embodiment provides a use case in which rendering of source content having a fixed frame rate and shutter angle can be performed by a decoder at an alternate frame rate and a variable simulated shutter angle, for example as illustrated in FIG. 2. For example, if the source content has a frame rate of 120 fps and a shutter angle of 360° (meaning the shutter is open for 1/120 s), the decoder may render multiple frame rates that are less than or are equal to 120 frames/s. For example, as described in Table. 1, in order to decode 24 frames/s with a simulated shutter angle of 216°, the decoder may combine the three decoded frames and display at 24 frames/s. Tab. 4 is based in detail on Table. 1 and illustrates how to combine different numbers of encoded frames to render at output target frame rates and desired target shutter angles. Frame combining can be done by simple pixel averaging, by weighted pixel averaging, in which pixels from a particular frame can be given higher weights than pixels from other frames and the sum of all weights is summed as one, or by other filtering interpolation schemes known in the art. this field of technology. In table. 4, the function Ce(a, b) denotes a combination of encoded frames a-b, where the combination may be performed by averaging, weighted averaging, filtering, or the like.

Table 4

Example of combining input frames at 120 fps to produce output frames at target fps and shutter angle

Entrance. si s2 s3 s4 s5 s6 s7 s8 s9 S10 Encoder el e2 e3 e4 e5 fuck e7 e8 e9 her Decode ir. 120 fps @360 el e2 e3 e4 e5 fuck e7 e8 e9 her Decode ir. 60 fps @360 Ce(1,2) Ce(3,4) Ce(5,6) Ce(7,8) Ce(9D0) @180 el e3 e5 e7 e9 Decode ir. 40 fps @360 Ce(1,3) Ce(4,6) Ce(7,9) Ce(10,12) @240 Ce(1,2) Ce(4,5) Ce(7,8) Ce(10,11) @120 el e4 e7 her

- 7 042523

Decode ir. 30 fps @360 Ce(1,4) Ce(5,8) Ce(9,12) @270 C(1,3) Ce(5,7) Ce(9,11) @180 Ce(1,2) Ce(5,6) Ce(9,10) @90 el e5 e9 Decode ir. 24 fps @360 Ce(1.5) Ce(6,10) @288 Ce(1,4) Ce(6,9) @216 Ce(1,3) Ce(6,8) @144 Ce(1,2) Ce(6,7) @72 el fuck

When the value of the target shutter angle is less than 360°, the decoder may combine different sets of decoded frames. For example, from Table. 1, given a source stream of 120 fps @ 360°, to generate a stream at 40 fps and a shutter angle of 240°, the decoder must combine two frames out of three possible frames. Thus, it can combine either the first and second frames, or the second and third frames. The choice of which frames to combine can be described in terms of a decoding phase expressed as follows:

decode_phase=decode_phase_idx * (360/n_frames) (6) where decode_phase_idx indicates the offset index in the set of consecutive frames having index values in [0, n_frames_max-1], where n_frames is given by equation (2), and njrames_max=orig_frame_rate/target_frame_rate ( 7)

In general, decode_phase_idx ranges from [0, n_frames_max-n_frames]. For example, for a source sequence at 120 fps and a shutter angle of 360°, for a target frame rate of 40 fps at a shutter angle of 240°, n_frames_max=120/40=3. From equation (2) n_frames=2, whereby decode_phase_idx varies within [0, 1]. Thus, decode_phase_idx=0 indicates selection of frames with index 0 and 1 and decode_phase_idx=1 indicates selection of frames with index 1 and 2.

In this embodiment, the rendered variable frame rate intended by the content creator may be signaled as metadata such as a side enhancement information (SEI) message or as video usability information (VUI). Optionally, the rendered frame rate may be controlled by the receiver or by the user. An example of a frame rate conversion SEI message exchange that indicates the content creator's preferred frame rate and shutter angle is shown in Table 1. 5. The SEI message may also indicate whether or not frame combining is performed in the encoded signal domain (eg, gamma, PQ, etc.) or in the linear light domain. It should be noted that post-processing requires a frame buffer in addition to the Decoder Picture Buffer (DPB). The SEI message may indicate how many additional frame buffers are required, or some alternative method for combining frames. For example, to reduce complexity, frames may be recombined at reduced spatial resolution.

As illustrated in Table. 4, at certain combinations of frame rates and shutter angles (for example, at 30 fps and 360° or at 24 fps and 288 or 360°), the decoder may need to combine more than three decoded frames, which increases the amount of buffer space, required by the decoder. To reduce the overhead of additional buffer space in the decoder, in some embodiments, certain combinations of frame rates and shutter angles may be prohibited for a set of allowed decoding parameters (eg, by setting appropriate profiles and coding levels).

Taking the case of playback at 24 fps again as an example, the decoder may determine to display the same frame five times so that it is displayed at an output frame rate of 120 fps. This is exactly identical to displaying a frame once per output.

- 8 042523 frame rate of 24 fps. The advantage of maintaining a constant output frame rate is that the display can be performed at a constant clock rate, which greatly simplifies the overall hardware. If the display can dynamically vary the clock rate, then it may make more sense to show a frame only once (within 1/24th of a second) instead of repeating an identical frame five times (every 1/120th of a second). The first approach may result in slightly better image quality, better optical efficiency, or better power efficiency. Similar considerations also apply to other frame rates.

Tab. 5 illustrates an example SEI messaging syntax for frame rate conversion according to an embodiment.

Table 5

Example SEI message syntax to provide frame rate conversion

framerate_conversion(payloadSize) { Descriptor framerate_conversion_cancel_flag u(l) if(!frame_conversion_cancel_flag) { base_frame_rate u(9) base_shutter_angle u(9) decode_phase_idx_present_flag u(l) if(decode_phase_idx_present_flag) { decode_phase_idx u(3) } conversion_domain_idc u(l) num_frame_buffer u(3) framerate_conversion_persistence_flag u(D } }

framerate_conversion_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of any previous SEI message for frame rate conversion in output order.

framerate_conversion_cancel_flag equal to 0 indicates that frame rate conversion information follows.

base_frame_rate specifies the required frame rate.

base_shutter_angle specifies the desired shutter angle.

decode_phase_idx_present_flag equal to 1 indicates that decoding phase information is present.

decode_phase_idx_present_flag equal to 0 indicates that no decoding phase information is present.

decode_phase_idx indicates the offset index in the set of consecutive frames having index values 0, ..., (n_frames_max-1), where n_frames_max=120/base_frame_rate. The value of decode_phase_idx must be in the range 0, ..., (n_frames_max-n_frames), where n_frames=base_shutter_angle/(3*base_frame_rate). When decode_phase_idx is not present, it is logically inferred to be 0.

conversion_domain_idc equal to 0 indicates that the frame combining is performed in the linear domain; conversion_domain_idc equal to 1 indicates that the frame combining is performed in a non-linear domain.

num_frame_buffers specifies the additional number of frame buffers (no DPB count).

framerate_conversion_persistence_flag indicates the persistence of the SEI message for frame rate conversion for the current layer.

framerate_conversion_persistence_flag equal to 0 indicates that the SEI message for frame rate conversion is applied to the current decoded picture only. Let picA be the current image.

framerate_conversion_persistence_flag equal to 1 indicates that the frame rate conversion SEI message is retained for the current layer in output order until one or more of the following conditions are true.

A new coded layered video sequence (CLVS) of the current layer starts.

The bit stream ends.

The picture picB in the current layer in the access unit containing the SEI message for frame rate conversion that is applicable to the current layer is output, with PicOrderCnt(picB) exceeding PicOrderCnt(picA), where PicOrderCnt(picB) and PicOrderCnt(picA) are the values PicOrderCntVal for picB and picA respectively, immediately after the activation of the decoding process

- 9 042523 for the number in the image sequence for picB.

Third Embodiment: Input encoded at multiple shutter angles.

The third embodiment is a coding scheme that allows subframe frequencies to be extracted from a bitstream, thereby maintaining backward compatibility. In HEVC, this is achieved through temporal scalability. Scalability in the time layer is provided by assigning different values to the temporal_id syntax element for decoded frames. The bit stream can therefore be derived simply from the temporalid values. However, the HEVC approach to temporal scalability does not render output frame rates with different shutter angles. For example, a base frame rate of 60 fps extracted from a 120 fps original should always have a shutter angle of 180°.

ATSC 3.0 describes an alternative method in which frames at 60 fps having 360° shutter angles are emulated as a weighted average of two frames at 120 fps. Emulated frames at 60 fps are assigned a temporalid value of 0, and this is combined with the assigned value of temporal_id of the source frame variables at 120 fps 1. When 60 fps is required, the decoder must decode only frames with temporal_id 0. When 120 fps required, the decoder may subtract each frame with temporal_id=1 (i.e. frame at 120 fps) from the scaled version of each corresponding frame with temporal_id=0 (i.e. emulated frame at 60 fps) to reconstruct the corresponding original frame at 120 fps that is not explicitly transmitted; thereby recreating all original frames at 120 fps.

Embodiments of this invention describe a new algorithm that supports multiple target frame rates and target shutter angles in a manner that is backward compatible (BC). The proposal is to preprocess source content at 120 fps at base frame rate at multiple shutter angles. Then, in the decoder, other frame rates at various other shutter angles can simply be extracted. The ATSC 3.0 approach can be considered as a special case of the proposed scheme, where frames with temporal_id=0 carry frames at 60fps@ 360° shutter angle and frames with temporal_id=1 carry frames at 60fps@ 180° shutter angle.

As a first example, as illustrated in FIG. 4, consider an input sequence at 120 fps and a 360° shutter angle that is used to encode the sequence with a base layer frame rate of 40 fps and shutter angles of 120, 240, and 360°. In this scheme, the encoder computes new frames by combining up to three original input frames. For example, encoded frame 2 (En-2) representing input at 40 fps and 240° is generated by combining input frames 1 and 2, and encoded frame 3 (En-3) representing input at 40 fps and 360 ° is generated by combining frame En-2 with input frame 3. In the decoder, to reconstruct the input sequence, decoded frame 2 (Dec-2) is generated by subtracting frame En-1 from frame En-2, and decoded frame 3 (Dec- 3) is formed by subtracting the En-2 frame from the En-3 frame. The three decoded frames represent output at a base frame rate of 120 fps and a 360° shutter angle. Additional frame rates and shutter angles may be extrapolated using the decoded frames, as illustrated in Table 1. 6. In the table. 6, the function Cs(a, b) denotes a combination of input frames a-b, where the combination may be performed by averaging, weighted averaging, filtering, or the like.

Table 6

An example of combining frames with a baseline of 40 fps

Input frames 120 fps @360 si s2 s3 s4 s5 s6 s7 s8 s9 Encoding other frames 120 fps el= -si e2= Cs(l,2) e3= Cs(l,3) e4=s4 e5= Cs(4,5) e6= Cs(4,6) e7=s7 e8= Cs(7,8) e9= Cs(7,9)

- 10 042523

Ovn decoder. 120 fps @360 e1= -si e2-el= -s2 e3-e2= s3 e4=s4 e5-e4= s5 e6- e4=s6 e7=s7 e8-e7= s8 e9-e8 =s9 Ovn decoder. 60 fps @360 e2 e3-e2+e4 =Cs(3,4) e6-e4=Cs(5,6) e8=Cs(7,8) e9- e8+el0 @180 el e3-e2=s3 e5-e4=s5 e7 e9-e8 Ovn decoder. 40 fps @360 e3=Cs(l,3) e6 e9 @240 e2=Cs(l,2) e5 e8 @120 el=sl e4 e7 Ovn decoder. 30 fps @360 e3+e4=Cs(l,4) e6-e4+e8=Cs(5,8) e9- e8+el2 @270 e3=Cs(l,3) e6-e5+e7=Cs(5,7) e9- e8+ell @180 e2=Cs(l,2) e6-e4=Cs(5,6) e9- e8+el0 @90 el e5-e4=s5 e9-e8 Ovn decoder. 24 fps @360 e3+e5=Cs(l,5) e6-e5+e9+el0=Cs(6,10) @288 e3+e4=Cs(l,4) e6-e5+e9=Cs(6,9) @216 e3=Cs(l,3) e6-e5+e8=Cs(6,8) @144 e2=Cs(l,2) e6-e5+e7=Cs(6,7) @72 el=sl e6-e5=s6

The advantage of this approach is that, as illustrated in Table. 6, all versions at 40 fps can be decoded without further processing. Another advantage is that different frame rates can be retrieved at different shutter angles. For example, consider decoding a decoder at 30 fps and a 360° shutter angle. From Table. 4, the output corresponds to a sequence of frames generated by Ce(1,4)=Cs(1,4), Cs(5,8), Cs(9,12) and the like, which matches the decoding sequence also illustrated in tab. 6; However, in Table 6 Cs(5,8)=e6-e4+e8. In an embodiment, lookup tables (LUTs) may be used to specify how decoded frames are to be combined to form an output sequence at a specified output frame rate and emulated shutter angle.

In another example, it is proposed to combine up to five frames in the encoder to facilitate extraction of the base layer at 24 fps at shutter angles of 72, 144, 216, 288, and 360°, as shown below. This is required for film content, which is best presented at 24 fps on legacy television receivers.

- 11 042523

Table 7

An example of combining frames with a baseline of 24 fps

Inputs 120 fps @360 frames si s2 s3 s4 s5 s6 s7 s8 s9 Encoder frames el -si e2=Cs (1,2) e3=Cs(l,3 ) e4=Cs(l,4 ) e5=Cs(l,5 ) e6= s6 e7=Cs (6.7) e8=Cs (6.8) e9=Cs( 6,9) Decoded n. 120 fps @360 el e2-el e3-e2 e4-e3 e5-e4 e6 e7-e6 e8-e7 e9-e8 Decoded and. 60 fps @360 e2 e4-e2 e5-e4+e6 e8-e6 el0-e8 @180 el e3-e2 e5-e4 e7-e6 e9-e8 Decoded and. 40 fps @360 e3 e5-e3+e6 e9-e6 @240 e2 e5-e3 e8-e6 @120 el e4-e3 e7-e6 Decoded and. 30 fps @360 e4 e5-e4+e8 elO- e8+el 2 @270 e3 e5-e4+e7 elO- e8+el 1 @180 e2 e5-e4+e6 el0-e8 @90 el e5-e4 e9-e8

- 12 042523

Decoded n. 24 fps @360 e5 her @288 e4 e9 @216 e3 e8 @144 e2 e7 @72 el fuck

As illustrated in Table. 7, if the decoding frame rate is the same as the base frame rate (24 fps), then in each group of five frames (e.g., e1 to e5), the decoder can simply select one frame at the desired shutter angle (e.g., e2 for a shutter angle in 144°). To decode at a different frame rate and at a particular shutter angle, the decoder must determine how to properly combine (say, by adding or subtracting) the decoded frames. For example, to decode at 30 fps and at a shutter angle of 180°, the following steps may be performed.

a) The decoder may consider the encoder's hypothetical transmission at 120 fps and at 360° without consideration for backwards compatibility, then (Table 1) the decoder shall combine 2 of the 4 frames to generate an output sequence at the required frame rate and at the required angle shutter. For example, as illustrated in Table. 4, the sequence includes, Ce(1,2)=Avg(s1,s2), Ce(5,6)=Avg(s5,s6) and the like, where Avg(s1,s2) may denote an average frames s1 and s2.

b) Provided that, by definition, encoded frames can be expressed as e1=s1, e2=Avg(s1,s2), e3=Avg(s1,s3), etc., it can be easily extracted that the sequence of frames at the stage a) can also be expressed as follows:

Ce( 1,2)=Avg(s 1 ,s2)=e2

Ce(5,6)=Avg(s5,s6)=Avg(sl,s5)-Avg(sl,s4)+s6=e5-e4+e6 etc.

As stated above, the proper combination of decoded frames may be pre-calculated and made available as a LUT.

The advantage of the proposed method is that it provides opportunities for both content creators and users; those. it provides director's/editor's choice and user's choice. For example, pre-processing content in an encoder provides the ability to generate a base frame rate with different shutter angles. Each shutter corner can be assigned a temporalid value in the range [0, (n_frames-1)], where n_frames has a value of 120 divided by the base frame rate. (For example, for a base frame rate of 24 fps, temporalid is in the range [0,4].) The choice may be made to optimize compression efficiency, or for aesthetic reasons. In some use cases, say for streaming over networks, multiple bit streams with different base layers may be encoded and stored and offered to users for selection.

In the second example of the disclosed methods, multiple backward compatible frame rates may be supported. Ideally one would want to be able to decode at 24 fps to get the base layer at 24 fps, at 30 fps to get a sequence at 30 fps, at 60 fps to get a sequence at 60 frames /s, etc. If the target shutter angle is not specified, a default shutter target angle is recommended, of those shutter angles allowed for source and target frame rates as close to 180° as possible. For example, for the values illustrated in Table. 7, the preferred target frame/s shutter angles of 120, 60, 40, 30, and 24 are 360, 180, 120, 180, and 216°.

From the above examples, it can be noted that the choice of how to encode the content may affect the difficulty of decoding particular frame rates of the base layer. One embodiment of this invention is to adaptively select a coding scheme based on a desired base layer frame rate. For movie content it may be 24 fps, for example, while for sports it may be 60 fps.

An exemplary syntax for the BC embodiment of the present invention is shown below and in Table 1. 8 and 9.

In SPS (Table 8), two syntax elements are added: sps_hfr_BC_enabled_flag and sps_base_framerate (if sps_hfr_BC_enabled_flag is set to 1).

sps_hfr_BC_enabled_flag equal to 1 indicates that a backward compatible high frame rate is provided in the Coded Video Sequence (CVS).

sps_hfr_BC_enabled_flag set to 0 indicates that high frame rate with backwards compatibility is not supported in CVS.

sps_base_framerate specifies the base framerate for the current CVS.

In the tile group header, if sps_hfr_BC_enabled_flag is set to 1, the number_avg_frames syntax is sent in the bitstream.

number_avg_frames indicates the number of frames at the highest frame rate (eg, 120 fps) that are combined to form the current image at the base frame rate.

Table 8

Sample RBSP syntax for input at various shutter angles

seq_parameter_set_rbsp() { Descriptor sps_max_sub_layers_minus 1 u(3) sps_reserved_zero_5bits u(5) profile_tier_level(sps_max_sub_layers_minus 1) sps_seq_parameter_set_id ue(v) sps_hfr_BC_enabled_flag u(l) if(sps_hfr_BC_enabled_flag) { u(l) sps_base_frame_rate u(9) } sps_extension_flag u(l) if(sps_extension_flag) while(more_rb sp_data()) sps_extension_data_flag u(D rb sp_trailing_bits() }

Table 9

Approximate RBSB syntax for a set of image parameters for input at various shutter angles

pic_parameter_set_rbsp() { Descriptor pp s_pic_parameter_set_id ue(v) pp s_seq_parameter_set_id ue(v) if(sps_hfr_BC_enabled_flag) number_avg_frames se(v) rbsp_trailing_bits() }

Variations of the second embodiment (fixed frame rate) of the HEVC coding standard (H-265) (ref. [1]) and the forthcoming universal video coding standard (commonly referred to as VVC, see ref. [2]) define a syntax element, pic_struct, which indicates whether or not the image should be displayed as a frame or as one or more fields and whether or not the decoded image should be repeated. A copy of Table D.2 The HEVC interpretation of pic_struct is provided for ease of reference in the application.

It is important to note that the inventors should take into account that the existing pic_struct syntax element can only support a particular subset of content frame rates when using a fixed frame rate encoding container. For example, when using a container for a fixed frame rate of 60 fps, the existing picstruct syntax, when fixed_pic_rate_within_cvs_flag is 1, can support 30 fps by using frame doubling and 24 fps by using frame doubling and frame tripling in the combination variable for every second frame. However, when using a container for a fixed frame rate of 120 fps, the current pic_struct syntax cannot support frame rates of either 24 fps or 30 fps. To alleviate this problem, it is proposed

- 14 042523 two new ways, one is an extension of the HEVC version and the other is not.

Method 1. Picstruct without backwards compatibility.

VVC is still being developed, so you can design the syntax with maximum freedom. In an embodiment, it is proposed to remove the options for frame doubling and frame tripling in picstruct, use a specific picstruct value to indicate random frame repetition, and add a new syntax element num_frame_repetition_minus2 that specifies the number of frames to repeat. An example of the proposed syntax is described in the following tables, with the table. 10 denotes changes for table. D.2.3 in HEVC and tab. 11 indicates the changes in the table. D.2 shown in the appendix.

Table 10

Sample SEI Message Syntax for Image Synchronization Method 1

pic_timing(payloadSize) { Descriptor if(frame_field_info_present_flag) { pic_struct u(4) if(pic_struct==7) u(4) num_frame_repetition_minu s2 u(4) source_scan_type u(2) duplicate_flag u(l) } .... (similar to the original)

num_frame_repetition_minus2 plus 2 indicates that when fixed_pic_rate_withrn_cvs_flag is 1, the frame should be displayed num_frame_repetition_minus2 plus 2 times consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[n] as given by Equation E-73.

Table 11

Example of corrected pic struct according to way 1

Meaning Specified display Images Restrictions 0 (Progressive) frame field_seq_flag must be 0 1 Top field fieldseqflag must be 1 2 bottom field fieldseqflag must be 1 3 Top margin, bottom margin, in that order field_seq_flag must be 0 4 Bottom margin, top margin, in that order field_seq_flag must be 0 5 Top margin, bottom margin, repeated top margin, in that order field_seq_flag must be 0 6 Bottom margin, top margin, repeated bottom margin, in that order field_seq_flag must be 0 7 Frame repetition field_seq_flag must be 0 fixed_pic_rate_within_c v s_flag must be be equal to 1 8 Top margin paired with previous bottom margin in output order field seq flag must be 1

- 15 042523

9 Bottom margin paired with previous top margin in output order fieldseqflag must be 1 10 Top margin paired with next bottom margin in output order fieldseqflag must be 1 eleven Bottom margin paired with next top margin in output order fieldseqflag must be 1

Method 2: An extended version of the HEVC version of picstruct.

AVC and HEVC decoders are already being deployed; as such, it may be desirable to simply extend the existing picstruct syntax without removing old variants. In an embodiment, a new pic_struct=13, a frame repetition extension value, and a new syntax element num_frame_repetition_minus4 are added. An example of the proposed syntax is described in Table. 12 and 13. For pic_struct values 0-12, the proposed syntax is identical to the syntax in Table. D.2 (as shown in the appendix), whereby these values are omitted for simplicity.

Table 12

Sample SEI Message Syntax for Image Synchronization Method 2

pic_timing(payloadSize) { Descriptor if(frame_field_info_present_flag) { pic_struct u(4) if(pic_struct==13) u(4) num_frame_repetition_minu s4 u(4) source_scan_type u(2) duplicate_flag u(l) } ... (similar to the original)

num_frame_repetition_minus4 plus 4 indicates that when fixed_pic_rate_within_cvs_flag is equal to 1, the frame should be displayed num_frame_repetition_minus4 plus 4 times consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[n] as given by Equation E-73.

Table 13

Fixed pic struct example, method 2

Meaning Specified display Images Restrictions 0-12 Similar to table D.2 Similar to table D.2 13 Frame repetition extension field_seq_flag must be 0 fixed_pic_rate_within_c v s_flag must be be equal to 1

In HEVC, the frame_field_info_present_flag parameter is present in the video applicability information (VUI), but the pic_struct, source_scan_type, and duplicate_flag syntax elements are in the pic_timing() SEI message. In an embodiment, it is proposed to move all associated syntax elements into the VUI, along with frame_field_info_present_flag. An example of the proposed syntax is illustrated in Table. 14.

- 16 042523

Table 14

Sample syntax for VUI parameters with support for the corrected pic struct syntax element

vui_parameters() { Descriptor u(D field_seq_flag u(D frame_field_info_present_flag u(D if(frame_field_info_present_flag) { pic_struct u(4) source_scan_type u(2) duplicate_flag u(D } }

Alternative signaling of shutter angle information.

When dealing with variable frame rate issues, it is desirable to identify both the required frame rate and the required shutter angle. In previous video coding standards, video usability information (VUI) provides essential information for proper display of video content, such as aspect ratio, primary colors, chroma subsampling, and so on. The VUI can also provide frame rate information if the fixed picture rate is set to 1; however, there is no support for shutter angle information. Embodiments allow different shutter angles to be used for different temporal layers, and the decoder can use the shutter angle information to improve the final appearance on the display.

For example, HEVC supports temporal sublayers that essentially use frame dropping techniques to traverse from a higher frame rate to a lower frame rate. The main problem with this is that the effective shutter angle decreases with each frame drop. As an example, 60 fps can be extracted from a 120 fps video by dropping every second frame; 30 frames/s can be extracted by discarding 3 out of 4 frames; and 24 fps can be extracted by discarding 4 out of 5 frames. Assuming a full 360° shutter for 120 Hz with simple frame drop, the shutter angles for 60 fps, 30 fps, and 24 fps are 180, 90, and 72°, respectively [3]. Experience has shown that shutter angles below 180° are generally unacceptable, in particular with frame rates below 50 Hz. By providing shutter angle information, for example, if the display is required to generate a cinematic effect from video at 120 Hz with a reduced shutter angle for each time layer, smart technologies can be applied to improve the final appearance.

In another example, one may want to support a different time layer (say, a 60 fps sub-bitstream within a 120 fps bitstream) with an identical shutter angle. In such a case, the main problem is that when a 120 fps video is displayed at 120 Hz, even/odd frames have a different effective shutter angle. If the display has associated information, smart technologies can be applied to improve the final appearance. An example of the proposed syntax is shown in Table. 15, in which the E.2.1 VUI parameter syntax table in HEVC (Ref. [1]) is modified to maintain shutter angle information as noted. It should be noted that in another embodiment, instead of expressing the shutter_angle syntax in absolute degrees, it may alternatively be expressed as the ratio of frame rate to shutter speed (see equation (1)).

- 17 042523

Table 15

Sample VUI Parameter Syntax with Shutter Angle Support

vui_parameters() { Descriptor vui_timing_info_present_flag u(D if(vui_timing_info_present_flag) { vui_num_units_in_tick u(32) vui_time_scale u(32) vui_poc_proportional_to_timing_flag u(l) if(sps_poc_proportional_to_timing_flag) vui_num_ticks_poc_diff_one_minu s 1 ue(v) vui_hrd_parameters_present_flag u(l) if(vcl_hrd_parameters_present_flag) hrd_parameters( 1, sps_max_subjayers_minus 1) } vui_shutter_angle_info_present_flag u(l) if(vui_shutter_angles_info_present_flag) { fixed_shutter_angle_within_cvs_flag u(l) if(fixed_shutter_angle_with_cvs_flag) fixed_shutter_angle u(9) else { for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_shutter_angle [i] u(9) } } }

vui_shutter_angle_info_present_flag equal to 1 indicates that the shutter angle information is present in the vui_parameters() syntax structure.

vui_shutter_angle_info_present_flag equal to 0 indicates that the shutter angle information is not present in the vui_parameters() syntax structure.

fixed_shutter_angle_within_cvs_flag equal to 1 indicates that the shutter angle information is the same for all time sublayers in CVS.

fixed_shutter_angle_within_cvs_flag equal to 0 indicates that the shutter angle information may not be identical for all time sublayers in CVS.

fixed_shutter_angle specifies the shutter angle in degrees in CVS. fixed_shutterangle must be between 0 and 360.

sub_layer_shutter_angle[i] specifies the shutter angle in degrees when HighestTid is i. sub_layer_shutter_angle[i] must be between 0 and 360.

Gradually update the frame rate in the coded video sequence (CVS).

Experiments demonstrate that for HDR content displayed on an HDR display, in order to perceive identical motion judder as standard dynamic range (SDR) playback on a 100 nit display, the frame rate must increase based on the brightness of the content. In most standards (AVC, HEVC, VVC, etc.), the video frame rate can be specified in the VUI (contained in the SPS) using the vui_time_scale, vui_num_units_in_tick, and elemental_duration_in_tc_minus1 [temporal_id_max] syntax elements, for example, as shown in the following table. 16 (see section E.2.1 in Ref. [1]).

- 18 042523

Table 16

Syntax VUI elements that indicate the frame rate in HEVC

vui_parameters() { Descriptor vui_timing_info_present_flag u(D if(vui_timing_info_present_flag) { vui_num_units_in_tick u(32) vui_time_scale u(32) vui_poc_proportional_to_timing_flag u(l) if(sps_poc_proportional_to_timing_flag) vui_num_ticks_poc_diff_one_minu s 1 ue(v) vui_hrd_parameters_present_flag u(l) if(vcl_hrd_parameters_present_flag) hrd_parameters( 1, sps_max_subjayers_minus 1) }

As explained in reference [1], the ClockTick variable is retrieved as follows and is called the clock tick:

ClockTick=vui_num_units_in_tick^vui_time_scale picture_duration=ClockTick* (elemental_duration_in_tc_minus 1[i]+1) frame_rate= 1 /pic_duration

However, the frame rate may only change at specific times, for example, in HEVC only in frames based on an internal random access point (IRAP) or at the beginning of a new CVS. For HDR playback, when a case of fading in or fading out occurs because the brightness of an image changes frame by frame, it may be necessary to change the frame rate or image duration for each image. In order to be able to update the frame rate or duration of a picture at any point in time (even on a frame-by-frame basis), in an embodiment, a new SEI message is proposed for a gradual update rate, as shown in Table 1. 17.

Table 17

Exemplary Syntax to Support Gradual Refresh Frame Rates in SEI Messaging

gradual_refresh_rate(payloadSize) { Descriptor num_units_in_tick u(32) time_scale u(32) }

The definition of the new syntax numunits_in_tick is identical to vui_numunits_in_tick and the definition of timescale is identical to that of vui_time_scale.

numunits_in_tick is the number of time units of the clock signal running at timescale Hz, which corresponds to one increment (called the clock tick) of the clock tick counter; numunits_in_tick must be greater than 0. The clock tick in units of seconds is equal to the quotient num_units_in_tick divided by the timescale. For example, when the video signal's image rate is 25 Hz, time_scale may be 27000000, num_units_in_tick may be 1080000, and thus the clock cycle may be 0.04 s.

time_scale is the number of units of time that pass in 1 s. For example, a time frame that measures time using a 27 MHz clock signal has a time_scale of 27000000. The value of time_scale must be greater than 0.

The picture time duration for a picture that uses the gradual_refresh_rate SEI message is set as follows:

picture_duration=num_units_in_tick^time_scale

Shutter angle information signaling via SEI messaging.

As explained above, Table. 15 provides an example syntax for VUI parameters with shutter angle support. By way of example and without limitation, Table 18 lists identical syntax elements, but now as part of the SEI message for shutter angle information. It should be noted that SEI messaging is used as an example only and similar messaging can be constructed in other layers of the high-level syntax such as Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Sequence Header, or groups of mosaic fragments, etc.

Table 18

Exemplary SEI Message Syntax for Gate Angle Information

shutter_angle_information(payloadSize) { Descriptor fixed_shutter_angle_within_cvs_flag u(l) if(fixed_shutter_angle_within_cvs_flag) fixed_shutter_angle u(9) else { for(i=0; i<=sps_max_sub_layers_minusl; i++) sub_layer_shutter_angle [i] u(9) } }

The shutter angle is typically expressed in degrees from 0° to 360°. For example, a shutter angle of 180° indicates that the exposure time is 1/2 of the frame time. The shutter angle can be expressed as follows:

shutter_angle=frame_rate*360*shutter_speed where shutter_speed is the exposure duration and frame_rate is the inverse of the frame duration.

frame_rate for a given time sublayer Tid may be indicated by num_units_in_tick, time_scale, elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag [Tid] is 1, frame_rate=time_scale/(num_units Jn^ick* (elemental_duration_in_tc_minus 1 [Tid]+1)).

In some embodiments, the shutter angle value (eg, fixed_shutter_angle) may not be an integer, eg, it may be 135.75°. To ensure greater accuracy, in table. 21, you can replace u(9) (unsigned 9 bits) with u(16) or some other suitable bit depth (eg, 12 bits, 14 bits, or more than 16 bits).

In some embodiments, it may be advantageous to express the shutter angle information in terms of clock cycles. In VVC, the ClockTick variable is retrieved like this:

ClockTick=num_units_in_tick and time scale (8)

In this case, you can express both the frame duration and the exposure duration as a multiple or fractional relative to the clock cycles:

exposure_duration=fN* C lockTick (9) frame_duration=fM*ClockTick (10) where fN and fM are floating point values and fN<fM.

In this case shutter_angle=frame_rate*360* shutter_speed= =(l/frame_duration)*360*exposure_duration= (11) =(expo sure_durati on *360)/frame_durati on= =(fN*ClockTick*360)/(fM*ClockTick )= =(fN/fM) *360=(N umerator/Denominator) *360, where Numerator and Denominator are integers approximating the ratio fN/fM.

Tab. 19 shows an example of SEI messaging indicated by Equation (11). In this example, the shutter angle must be greater than 0° for the real world camera.

- 20 042523

Table 19

Exemplary SEI messaging for clock-based shutter angle information

shutter_angle_information(payloadSize) { Descriptor fixed_shutter_angle_within_cvs_flag u(l) if(fixed_shutter_angle_within_cvs_flag) { fixed_shutter_angle_numer_minu s 1 u(16) fixed_shutter_angle_denom_minu s 1 u(16) } else { for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_shutter_angle_numer_minus 1 [i] u(16) sub_layer_shutter_angle_denom_minus 1 [i] u(16) } }

As explained above, the use of u(16) (unsigned 16 bits) for shutter angle accuracy is illustrated as an example and corresponds to an accuracy of 360/2 ¹⁶ =0.0055. Accuracy can be adjusted based on actual applications. For example, when using u(8), the precision is 360/2 ⁸ =1.4063.

Note. The shutter angle is expressed in degrees greater than 0 but less than or equal to 360°. For example, a shutter angle of 180° indicates that the exposure time is 1/2 of the frame time.

fixed_shutter_angle_within_cvs_flag equal to 1 indicates that the shutter angle value is the same for all time sublayers in CVS.

fixed_shutter_angle_within_cvs_flag equal to 0 indicates that the shutter angle value may not be identical for all time sublayers in CVS.

fixed_shutter_angle_numer_minus1 plus 1 specifies the numerator used to retrieve the shutter angle value. The fixed_shutter_angle_numer_minus1 value must be between 0 and 65535 inclusive.

fixed_shutter_angle_demom_minus1 plus 1 specifies the denominator used to retrieve the shutter angle value. The fixed_shutter_angle_demom_minus1 value must be between 0 and 65535 inclusive.

The value of fixed_shutter_angle_numer_minus1 must be less than or equal to the value of fixed_shutter_angle_demom_minus 1.

The variable shutterAngle in degrees is retrieved like this:

shutterAngle=3 60* (fixedshutterangl enumerminus 1+1) (fixed_shutter_angle_demom_minu s 1+1) sub_layer_shutter_angle_numer_minus1[i] plus 1 specifies the numerator used to retrieve the shutter angle value when HighestTid is i. The value of sub_layer_shutter_angle_numer_minus1[i] must be between 0 and 65535 inclusive.

sub_layer_shutter_angle_demom_minus1[i] plus 1 specifies the denominator used to retrieve the shutter angle value when HighestTid is i. The value of sub_layer_shutter_angle_demom_minus1[i] must be between 0 and 65535 inclusive.

The value of sub_layer_shutter_angle_numer_minus1[i] must be less than or equal to the value of sub_layer_shutter_angle_denom_minus I[i].

The variable subLayerShutterAngle[i] in degrees is retrieved as follows: subLayerShutterAngle[i]=360* (sub_layer_shutter_angle_numer_minus 1 [i] +1) (sub_layer_shutter_angle_demom_minusl[i] +1)

In another embodiment, the frame duration (eg, frameduration) may be indicated by some other means. For example, in DVB/ATSC, when fixed_pic_rate_within_cvs_flag [Tid] is 1, frame_rate=time_s with ale/(num_units_in_tick* (elemental_duration_in_tc_minu s 1 [Tid]+1)), frame_duration=l/frame_rate.

Syntax in Table. 19 and in some subsequent tables suggests that the shutter angle should always be greater than zero; however, shutter angle=0 can be used to signal the creative intent that content should be rendered without motion blur. This can be the case for moving graphics, animations, CGI textures and matte screens, etc. IN

- 21 042523 In this regard, for example, signaling shutter angle=0 can be useful for mode selection decision in a transcoder (for example, to select transcoding modes that preserve edges), as well as in a display that receives shutter angle metadata over STA- interface or 3GPP interface. For example, shutter angle=0 may be used to indicate to the display that it should not perform motion processing such as denoising, frame interpolation, and the like. In such an embodiment, the fixed_shutter_angle_numer_minus1 and sub_layer_shutter_angle_numer_minus1[i] syntax elements may be replaced by the fixed_shutter_angle_numer and sub_layer_shutter_angle_numer[i] syntax elements, where fixed_shutter_angle_numer indicates the numerator used to retrieve the shutter angle value (the value of fixed_shutter_angle_numer must be between 35 and inclusive );

sub_layer_shutter_angle_numer[i] specifies the numerator used to retrieve the shutter angle value when HighestTid is i (sub_layer_shutter_angle_numer[i] must be between 0 and 65535 inclusive).

Furthermore, in another embodiment, fixed_shutter_angle_denom_minus1 and sub_layer_shutter_angle_denom_minus1[i] may also be replaced by the fixed_shutter_angle_denom and sub_layer_shutter_angle_denom[i] syntax elements.

In an embodiment, as illustrated in Table. 20, it is possible to reuse the numunits_in_tick and timescale syntax given in SPS by setting general_hrd_parameters_present_flag to 1 in VVC. According to this scenario, the SEI message can be renamed as the exposure duration SEI message.

Table 20

Exemplary SEI message exchange for exposure duration signaling

exposure_duration_information(payloadSize) { Descriptor fixed_exposure_duration_within_cvs_flag u(l) if(fixed_shutter_angle_within_cvs_flag) { fixed_expo sure_duration_numer_minu s 1 u(16) fixed_expo sure_duration_denom_minu s 1 u(16) } else { for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_exposure_duration_numer_minus 1 [i] u(16) sub_layer_exposure_duration_denom_minus 1 [i] u(16) } }

fixed_exposure_duration_within_cvs_flag equal to 1 indicates that the effective exposure duration value is the same for all time sublayers in CVS.

fixed_exposure_duration_within_cvs_flag equal to 0 indicates that the effective exposure duration value may not be identical for all time sublayers in CVS.

fixed_exposure_duration_numer_minus1+1 specifies the numerator used to retrieve the exposure duration value. The fixed_exposure_duration_numer_minus1 value must be between 0 and 65535 inclusive.

fixed_exposure_duration_demom_minus1+1 specifies the denominator used to extract the exposure duration value. fixed_exposure_duration_demom_minus1 must be between 0 and 65535 inclusive.

The value of fixed_exposure_during_numer_minus1 must be less than or equal to the value of fixed_exposure_duration_demom_minus1.

The variable fixedExposureDuration is retrieved like this:

fixedExposureDuration=(fixed_exposure_duration_numer_minus 1+1) (fixed_exposure_duration_demom_minusl+l)*ClockTicks sub_layer_exposure_duration_numer_minus1[i]+1 specifies the numerator used to retrieve the exposure duration value when HighestTid is i. The value of sub_layer_exposure_duration_numer_minus1[i] must be between 0 and 65535 inclusive.

sub_layer_exposure_duration_demom_minus1[i]+1 specifies the denominator used to retrieve the exposure duration value when lighestTid is equal to i. The sub_layer_exposure_duration_demom_minus1[i] value must deliver between 0 and 65535 inclusive.

- 22 042523

The value of sub_layer_exposure_duration_numer_minus1[i] must be less than or equal to the value of sub_layer_exposure_duration_demom_minus 1[i].

The variable subLayerExposureDuration[i] for HigestTid equal to i is retrieved as follows:

subLayerExposureDuration[i] =(sub_layer_exposure_duration_numer_minus 1 [i] +1) (sub_layer_exposure_duration_demom_minusl[i] +l)*ClockTicks

In another embodiment, as shown in table. 21, clockTick can be explicitly specified using the expo_num_units_in_tick and expo_time_scale syntax elements. The advantage here is that it is not based on whether or not general_hrd_parameters_present_flag is set to 1 in VVC as in the previous embodiment; in this case clockTick=expo numunitsintick < expo time scale (12)

Exemplary exchange of SEI messages for time signaling! exposure_duration_information(payloadSize) { expo_num_units_in_tick expo_time_scale fixed_exposure_duration_within_cvs_flag if(!fixed_exposure_duration_within_cvs_flag) Table 21 3 and exposure handle u(32) u(32) u(l) for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_exposure_duration_numer_minus 1 [i] u(16) sub_layer_exposure_duration_denom_minus 1 [i] u(16) } }

expo_num_units_in_tick is the number of time units of the clock signal running at timescale Hz, which corresponds to one increment (called the clock tick) of the clock tick counter; expo_num_units_in_tick must be greater than 0. The clock tick specified by the clockTick variable, in units of seconds, is equal to the quotient expo_num_units_in_tick divided by expo_time_scale.

expo_time_scale is the number of units of time that pass in 1 s. clockTick=expo_num_units_in_tick < expo time scale.

Note. Two syntax elements: expo_num_units_in_tick and expo_time_scale are given to measure exposure duration.

The bitstream matching requirement is that clockTick must be less than or equal to ClockTick when num_units_in_tick and time_scale are present.

fixed_exposure_duration_within_cvs_flag equal to 1 indicates that the effective exposure duration value is the same for all time sublayers in CVS.

fixed_exposure_duration_within_cvs_flag equal to 0 indicates that the effective exposure duration value may not be identical for all time sublayers in CVS. When fixed_exposure_duration_within_cvs_flag is 1, fixedExposureDuration is set to clockTick.

sub_layer_exposure_duration_numer_minus1[i]+1 specifies the numerator used to retrieve the exposure duration value when the HighestTid is i. The value of sub_layer_exposure_duration_numer_minus1[i] must be between 0 and 65535 inclusive.

sub_layer_exposure_duration_demom_minus1[i]+1 specifies the denominator used to extract the exposure duration value when the HighestTid is equal to i. The value of sub_layer_exposure_duration_demom_minus1[i] must be between 0 and 65535 inclusive.

sub_layer_exposure_duration_numer_minus1[i] must be less than or equal to sub_layer_exposure_duration_demom_minus1[i].

The variable subLayerExposureDuration[i] for HigestTid equal to i is retrieved as follows: subLayerExposureDuration[i] =(sub_layer_exposure_duration_numer_minus 1 [i] +1) < (sub_layer_exposure_duration_denom_minusl[i] +1 )*clockTick

As explained above, the syntax parameters sub_layer_exposure_duration_numer_minus1[i] and sub_layer_exposure_duration_denom_minus1[i] can also be replaced by sub_layer_exposure_duration_numer[i] and sub_layer_exposure_duration_denom[i].

In another embodiment, as shown in table. 22, the Shutterlnterval parameter (i.e. exposure duration) can be specified using the siinumunitsinshutterinterval and siitimescale syntax elements, where

ShutterInterval=sii_num_units_in_shutter_interval sii_time_scale (13)

- 23 042523

Table 22

Exemplary SEI Message Exchange for Signaling Exposure Duration (Shutter Interval Information)

shutter_interval_information(payloadSize) { Descriptor sii_num_units_in_shutter_interval u(32) sii_time_scale u(32) fixed_shutter_interval_within_cvs_flag u(D if(!fixed_shutter_interval_within_cvs_flag) for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_shutter_interval_numer[i] u(16) sub_layer_shutter_interval_denom[i] u(16) }

Semantics of SEI messages with shutter interval information.

The shutter interval information SEI message indicates the shutter interval for associated video content prior to encoding and display, for example, for camera-captured content, the amount of time the image sensor is exposed to form an image.

sii_num_units_in_shutter_interval specifies the number of time units of the clock signal running at sii_time_scale Hz, which corresponds to one increment of the shutter clock clock counter. The shutter interval specified by the ShutterInterval variable, in units of seconds, is equal to the quotient sii_num_units_in_shutter_interval divided by sii_time_scale. For example, when ShutterInterval is 0.04s, sii_time_scale may be 27000000 and sii_num_units_in_shutter_interval may be 1080000.

sii_time_scale specifies the number of units of time that pass in one second. For example, a time frame that measures time using a 27 MHz clock signal has a sii_time_scale of 27000000.

When the sii_time_scale value is greater than 0, the ShutterInterval value is specified as follows:

ShutterInterval=sii_num_units_in_shutter_interval > sii_time_scale

Otherwise (sii_time_scale is 0) ShutterInterval shall be interpreted as unknown or unspecified.

Note 1: A ShutterInterval value of 0 may indicate that the associated video content contains screen capture content, machine-generated content, or other non-camera-captured content.

Note 2: A value of ShutterInterval greater than the encoded picture rate inversion value of the encoded picture interval may indicate that the encoded picture rate is greater than the picture rate at which the associated video content is generated, for example, when the encoded picture rate is 120 Hz and the picture rate of the associated video content before encoding and display. is 60 Hz. The coded interval for a given time sublayer Tid may be indicated by ClockTick and elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag [Tid] is equal to 1, the picture interval for a given temporal sublayer Tid, given by the variable PictureInterval[Tid], may be specified as follows:

PictureInterval[T id] =S1ocT ick* (elemental_duration_in_tc_minu s 1 [Tid]+1).

fixed_shutter_interval_withm_cvs_flag equal to 1 indicates that the value of ShutterInterval is the same for all time sublayers in CVS.

fixed_shutter_interval_within_cvs_flag equal to 0 indicates that the value of ShutterInterval may not be identical for all time sublayers in CVS.

sub_layer_shutter_interval_numer[i] indicates the numerator used to retrieve the shutter interval for the sublayer specified by the variable subLayerShutterInterval[i] in units of seconds when HighestTid is equal to i.

sub_layer_shutter_interval_denom[i] indicates the denominator used to retrieve the shutter interval for the sublayer specified by the variable subLayerShutterInterval[i] in units of seconds when HighestTid is equal to i.

The value of subLayerShutterInterval[i] for HighestTid equal to i is retrieved as follows. When fixed_shutter_interval_within_cvs_flag is 0 and sub_layer_shutter_interval_denom[i] is greater than 0,

- 24 042523 subLayerShutterIntervalfi] =ShutterInterval*sub_layer_shutter_interval_numer[i] sub l ay ershutterintervaldenom [i ]

Otherwise (sub_layer_shutter_interval_denom[i] is 0) subLayerShutterInterval[i] shall be interpreted as unknown or unspecified. When fixed_shutter_interval_within_cvs_flag is not 0, subLayerShutterInterval [i]=ShutterInterval.

In an alternative embodiment, instead of using the numerator and denominator to signal the shutter interval for the sublayer, a single value is used. An example of such syntax is shown in Table. 23.

Table 23

Exemplary SEI Message Exchange for Shutter Interval Signaling

shutter_interval_information(payloadSize) { Descriptor sii_num_units_in_shutter_interval u(32) sii_time_scale u(32) fixed_shutter_interval_within_cvs_flag u(l) if(!fixed_shutter_interval_within_cvs_flag) for(i=0; i<=vps_max_sub_layers_minusl; i++) { sub_layer_num_units_in_shutter_interval[i] u(32) } }

Semantics of the Shi-message with shutter interval information.

The shutter interval information SEI message indicates the shutter interval for associated video content prior to encoding and display, for example, for camera-captured content, the amount of time the image sensor is exposed to form an image.

sii_num_units_in_shutter specifies the number of time units of the clock signal running at sii_time_scale Hz, which corresponds to one increment of the shutter clock clock counter. The shutter interval specified by the ShutterInterval variable in units of seconds is equal to the quotient sii_num_units_in_shutter_interval divided by sii_time_scale. For example, when ShutterInterval is 0.04s, sii_time_scale may be 27000000 and sii_num_units_in_shutter_interval may be 1080000.

sii_time_scale specifies the number of units of time that pass in one second. For example, a time frame that measures time using a 27 MHz clock signal has a sii_time_scale of 27000000.

When the sii_time_scale value is greater than 0, the ShutterInterval value is specified as follows:

ShutterInterval=sii_num_units_in_shutter_interval siitimescale

Otherwise (sii_time_scale is 0) ShutterInterval shall be interpreted as unknown or unspecified.

Note 1: A ShutterInterval value of 0 may indicate that the associated video content contains screen capture content, machine-generated content, or other non-camera-captured content.

Note 2: A value of ShutterInterval greater than the inverse value of the encoded picture rate, the encoded picture interval, may indicate that the encoded picture rate is higher than the picture rate at which the associated video content is created, for example, when the encoded picture rate is 120 Hz and the picture rate of the associated video content before encoding. and display is 60Hz. The interval of the encoded picture for a given time sublayer Tid may be indicated by ClockTick and elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag [Tid] is equal to 1, the picture interval for a given temporal sublayer Tid, given by the variable PictureInterval[Tid], may be specified as follows:

PictureInterval[T id] =ClockTick* (elemental_duration_in_tc_minus 1 [Tid]+1).

fixed_shutter_interval_within_cvs_flag equal to 1 indicates that the value of ShutterInterval is the same for all time sublayers in CVS.

fixed_shutter_interval_within_cvs_flag equal to 0 indicates that the value of ShutterInterval may not be identical for all time sublayers in CVS.

sub_layer_num_units_in_shutter_interval[i] specifies the number of time units of the clock signal operating at sii_time_scale 25 042523 Hz, which corresponds to one increment of the shutter clock clock counter. The shutter interval for the sublayer, given by the variable subLayerShutterInterval[i] in units of seconds when the HighestTid is equal to i, is equal to the sub_layer_num_units_in_shutter_interval[i] divided by sii_time_scale.

When the value of fixed_shutter_interval_within_cvs_flag is 0 and the value of sii_time_scale is greater than 0, the value of subLayerShutterInterval[i] is specified as follows:

subLayerShutterInterval[i]=sub_layer_num_units_in_shutter_interval[i] ^sii_time_scale

Otherwise (sii_time_scale value is 0) subLayerShutterInterval[i] shall be interpreted as unknown or unspecified. When fixed_shutter_interval_within_cvs_flag is not 0, subLayerShutterInterval[i]=ShutterInterval.

In table. 24 is a summary of the six approaches explained in Table 2. 18-23 to provide SEI messaging related to shutter angle or exposure time.

Table 24

A Brief Introduction to SEI Messaging Approaches for Signaling Gate Angle Information

Table number Key elements of signaling and dependencies 18 Shutter angle (from 0 to 360) is passed explicitly 19 The shutter angle is expressed as the ratio of the Numerator and Denominator values, which should be scaled by 360 (assuming the clock clock value) 20 The exposure time is transmitted as a ratio of the Numerator and Denominator values (assuming the clock cycle value) 21 The exposure duration is passed as a ratio of the Numerator and Denominator values; clock cycle value is passed explicitly as a ratio of two values 22 The shutter interval information is transmitted as a ratio of two values: the number of exposure clock cycles and the exposure timeline. The sublayer-related exposure times are reported as the ratio of the two values 23 The shutter interval information or exposure duration is transmitted as a ratio of two values: the number of exposure clock cycles and the exposure timeline. Sublayer-related exposure times are transmitted as the number of units of clock cycles at exposure in each sublayer

Variable frame rate signaling.

As explained in U.S. Provisional Patent Application 62/883195, filed August 06, 2019, many applications require a decoder to support playback at variable frame rates. Frame rate adaptation typically forms part of the operations in a hypothetical reference decoder (HRD), as described, for example, in Appendix C of reference material [2]. In an embodiment, it is proposed to signal, via SEI messaging or other means, a syntax element specifying a picture presentation time (PPT) as a function of a 90 kHz clock signal. This is a specific type of repetition of the nominal Decoder Picture Buffer (DPB) output time as specified in the HRD, but now using ClockTicks precision of 90 kHz as specified in the MPEG-2 system. The advantage of this SEI message is that

a) if no HRD is provided, the PPT SEI message may still be used to indicate the timing for each frame;

b) it can facilitate the translation of synchronization over the bitstream and synchronization in the system.

Tab. 25 describes an example of the syntax of the proposed PPT synchronization message, which is the same as the Variable Presentation Timestamp (PTS) syntax used in the MPEG-2 transport (H 222) (reference [4]).

- 26 042523

Table 25

Sample Syntax for Image Representation Time Messaging

picture_presentation_time(payloadSize) { Descriptor PPT and(33) }

PPT (Picture Presentation Time).

Presentation times must be related to decoding times as follows. The PPT is a 33-bit number encoded in three separate fields. It indicates the presentation time tpn(k) in the system target decoder of element k representation of elementary stream n. The PPT value is specified in units of the system clock period divided by 300 (resulting in 90 kHz). The image presentation time is extracted from the PPT according to the following equation:

PPT (k)=((system_clock_frequency* )/30)%2 ³³ , where tp _n (k) is the presentation time of the presentation unit Pn(k).

Bibliographic list.

Each of the referenced materials listed in this document is incorporated by reference in its entirety.

[1] High efficiency video coding, H.265, Series H, Coding of moving video, ITU, (02/2018).

[2] B. Brass, J. Chen, and S. Liu, Versatile Video Coding (Draft 5), JVET output document, JVET-N1001, v5, uploaded 14 May 2019.

[3] C. Carbonara, J. DeFilippis, M. Korpi, High Frame Rate Capture and Production, SMPTE 2015 Annual Technical Conference and Exhibition, October 26-29, 2015.

[4] Infrastructure of audiovisual services - Transmission multiplexing and synchronization, H.222.0, Series H, Generic coding of moving pictures and associated audio information: Systems, ITU, 08/2018.

An exemplary implementation of a computer system.

Embodiments of the present invention may be implemented by computer system(s) configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, field programmable gate array (FPGA), or other configurable or programmable logic device (PLD), discrete time processor or digital signal processor (DSP), application specific IC (ASIC), and/or equipment that includes one or more such systems, devices or components. The computer and/or IC may execute, control, or execute instructions related to frame rate scalability, such as the instructions described herein. The computer and/or IC may calculate any of a variety of parameters or values that relate to frame rate scalability described herein. Image and video related embodiments may be implemented in hardware, software, firmware, and various combinations of the foregoing.

Certain implementations of the invention comprise computer processors that execute program instructions that instruct the processors to carry out the method of the invention. For example, one or more processors in a display, encoder, set-top box, transcoder, or the like. may implement frame rate scalability techniques as described above by executing program instructions in program storage accessible to processors. Embodiments of the invention may also be provided in the form of a software product. The software product may comprise any durable and tangible medium that carries a set of machine-readable signals containing instructions that, when executed by a data processor, instruct the data processor to carry out the method of the invention. The software products of the invention may take any of a wide variety of non-volatile and tangible forms. The software product may include, for example, physical media such as magnetic storage media including floppy disks, hard drives, optical storage media including CD-ROM, DVD, electronic storage media including ROM, flash RAM, etc. Machine-readable signals on the software product may optionally be compressed or encrypted.

When a component (e.g., a software module, processor, node, device, circuit, etc.) is mentioned above, unless otherwise noted, reference to that component (including a reference to a facility) is to be interpreted as including, as equivalents to that component any component that performs the function of the described component (eg, that is functionally equivalent), including components that are not structurally equivalent to the disclosed structure that performs the function in the illustrated exemplary embodiments of the invention.

Equivalents, additions, alternatives, and miscellaneous.

As such, exemplary embodiments that relate to frame rate scalability are described. In the above detailed description, embodiments of the present invention are described with reference to a variety of specific details, which may vary depending on the implementation. Thus, the sole and exclusive indication of what constitutes an invention and what applicants mean by invention is the claims that follow from this application, in the particular form in which the claims are issued, including any subsequent amendments. All definitions expressly set forth in this document for the terms contained in this claims, should dictate the meaning of these terms when used in the claims. Therefore, limitations, elements, properties, features, advantages, or attributes that are not expressly set forth in the claims should not limit the scope of the claims in any way. Therefore, the detailed description and drawings are to be considered in an illustrative and not a limiting sense.

Application.

This application provides a copy of the table. D.2 and the associated pic_struct related information from specification H 265 (ref. [1]).

Table D.2

Interpreting pic_struct

Meaning Specified display Images Restrictions 0 (Progressive) frame field_seq_flag must be 0 1 Top field fieldseqflag must be 1 2 bottom field fieldseqflag must be 1 3 Top margin, bottom margin, in that order field_seq_flag must be 0 4 Bottom margin, top margin, in that order field_seq_flag must be 0 5 Top margin, bottom margin, repeated top margin, in that order field_seq_flag must be 0 6 Bottom margin, top margin, repeated bottom margin, in that order field_seq_flag must be 0 7 Frame doubling field_seq_flag must be 0 fixed_pic_rate_within_cvs_flag must be be equal to 1 8 Frame tripling field_seq_flag must be 0 fixed_pic_rate_within_cvs_flag must be be equal to 1 9 Top margin paired with previous bottom margin in output order field seq flag must be 1 10 Bottom margin paired with previous top margin in output order field seq flag must be 1 eleven Top margin paired with next bottom margin in output order field seq flag must be 1 12 Bottom margin paired with next top margin in output order field seq flag must be 1

-

Claims (3)

The semantics of the pic_struct syntax element. pic_struct specifies whether or not the image should be displayed as a frame or as one or more fields, and for displaying frames when fixed_pic_rate_within_cvs_flag is 1, may specify a frame doubling or tripling repeat period for displays that use a fixed frame refresh interval equal to DpbOutputElementalInterval [n] as given by Equation E-73. The interpretation of pic_struct is indicated in Table. D.2. pic_struct values that are not listed in Table. D.2 are reserved for future use by ITU-T|ISO/IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders MUST ignore reserved pic_struct values. When present, the bitstream matching requirement is that the value of pic_struct must be constrained such that exactly one of the following conditions is true: pic_struct is 0, 7, or 8 for all images in CVS; the value of pic_struct is 1, 2, 9, 10, 11, or 12 for all images in CVS; the value of pic_struct is 3, 4, 5, or 6 for all images in CVS. When fixed_pic_rate_within_cvs_flag is 1, frame doubling is indicated by pic_struct equal to 7, which indicates that the frame is to be displayed twice consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[n] as given by Equation E-73, and frame tripling is indicated by pic_struct equal to 8, which indicates that the frame should be displayed three times consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[n] as given by Equation E-73. Note 3: Frame doubling can be used to make it easier to display, for example, 25 Hz progressive scan video on a 50 Hz progressive scan display, or 30 Hz progressive scan video on a 60 Hz progressive scan display. Using frame doubling and frame tripling in a variable combination for every second frame can be used to facilitate displaying 24 Hz progressive scan video on a 60 Hz progressive scan display. The nominal vertical and horizontal sample locations in the top and bottom fields for the 4:2:0, 4:2:2, and 4:4:4 chrominance signal formats are shown in FIG. D.1, D.2 and D.3 respectively. The association indicators for fields (pic_struct equal to 9-12) provide hints to associate complementary parity fields together as frames. The parity of a field can be high or low, and the parity of two fields is said to be complementary when the parity of one field is high and the parity of the other field is low. When frame_field_info_present_flag is 1, the bitstream matching requirement is that the constraints specified in the third column of Table 1 apply. D.2. Note 4.— When frame_field_info_present_flag is 0, then in many cases default values may be inferred or indicated by another means. In the absence of other indications of the intended image display type, the decoder shall logically infer the value of pic_struct as equal to 0 when frame_field_info_present_flag is equal to 0. CLAIM 1. Machine-readable storage medium that contains the structure of the encoded video stream, and the structure of the encoded video stream contains a section of encoded images, including encoding a sequence of video images; and a signaling section including encoding a first gate angle flag that indicates whether or not the gate angle information is fixed for all temporal sublayers in the coded picture section, wherein if the first gate angle flag indicates that the gate angle information is fixed, then the signaling section includes a fixed shutter angle value for displaying a decoded version of the video sequence for all temporal sublayers in the coded picture section using the fixed shutter angle value, otherwise the signaling section includes an array of sublayer shutter angle values, where for each of the temporal sublayers a value of array of sublayer shutter angle values indicates the corresponding shutter angle for displaying the decoded version of the temporal sublayer of the video sequence. 2. The computer-readable storage medium of claim 1, wherein the signaling section further includes encoding a frame repetition value indicating the number of times a decoded image in the video sequence is to be sequentially displayed. 3. The computer-readable storage medium according to claim 1, in which the signaling section further includes -