CN116320394A

CN116320394A - Adaptive transfer function for video encoding and decoding

Info

Publication number: CN116320394A
Application number: CN202310395840.1A
Authority: CN
Inventors: A·托拉皮斯; D·辛格
Original assignee: Apple Inc
Current assignee: Apple Inc
Priority date: 2014-02-25
Filing date: 2015-02-25
Publication date: 2023-06-23
Also published as: KR20220106177A; AU2018208733A1; CN106030503B; JP2021010171A; US20150245043A1; AU2023200732A1; KR102025037B1; AU2015223123A1; CN110365974A; AU2021204255A1; US20150243200A1; US20150245044A1; WO2015130793A1; AU2018208733B2; US10986345B2; KR20190109579A; KR102330316B1; JP2018207512A; US20150245050A1; US20190182487A1

Abstract

The present disclosure relates to adaptive transfer functions for video encoding and decoding. The present invention provides a video encoding and decoding system that implements an adaptive transfer function method within a codec for signal representation. The focus dynamic range used to represent the effective dynamic range of the human visual system may be dynamically determined for each scene, sequence, frame, or region of the input video. Video data may be cropped and quantized into the bit depth of the codec according to the transfer function used for encoding within the codec. The transfer function may be the same as the transfer function of the input video data or may be a transfer function internal to the codec. The encoded video data may be decoded and expanded into the dynamic range of one or more displays. The adaptive transfer function approach may enable a codec to use fewer bits for an internal representation of a signal while still representing the entire dynamic range of the signal in the output.

Description

Adaptive transfer function for video encoding and decoding

The present application is a divisional application of the invention patent application No.201580010095.8 entitled "adaptive transfer function for video encoding and decoding" with application date of 2015, 2, 25.

Technical Field

The present disclosure relates generally to digital video or image processing and display.

Background

Various devices, including but not limited to personal computer systems, desktop computer systems, laptop and notebook computers, tablet or tablet devices, digital cameras, digital video recorders, and mobile phones or smart phones, may include software and/or hardware that may implement one or more video processing methods. For example, an apparatus may include a device (e.g., an Integrated Circuit (IC), such as a system on a chip (SOC), or a subsystem of an IC) that may receive and process digital video input from one or more sources and output processed video frames according to one or more video processing methods. As another example, a software program may be implemented on a device that can receive and process digital video input from one or more sources according to one or more video processing methods and output processed video frames to one or more destinations.

For example, the video encoder may be implemented as a device or alternative software program in which the digital video input is encoded or converted according to a video encoding method into another format, such as a compressed video format, such as the h.264/Advanced Video Coding (AVC) format or the h.265 High Efficiency Video Coding (HEVC) format. As another example, a video decoder may be implemented as an apparatus or alternative software program in which video in a compressed video format (such as AVC or HEVC) is received and decoded or converted into another (decompressed) format, such as a display format used by a display device, according to a video decoding method. The H.264/AVC standard is promulgated by ITU-T in a document titled "ITU-T Recommendation H.264: advanced video coding for generic audiovisual services". The H.265/HEVC standard is promulgated by ITU-T in a document entitled "ITU-T Recommendation H.265: high Efficiency Video Coding".

In many systems, a device or software program may implement both a video encoder component and a video decoder component; such as a device or program, is commonly referred to as a codec. Note that the codec may encode/decode both visual/image data and audio/sound data in the video stream.

The dynamic range is generally defined as the ratio between the maximum possible value and the minimum possible value of the variable amounts in signals such as, for example, sound and light. In digital image and video processing, extended dynamic range or High Dynamic Range (HDR) imaging refers to a technique that produces a wider range of brightness in an electronic image (e.g., as displayed on a display screen or display device) than is obtained using standard digital imaging techniques, known as standard dynamic range or SDR imaging.

An electro-optical transfer function (EOTF) may map digital code values to light values, for example to luminance values. The inverse process, commonly referred to as the optical-electrical transfer function (OETF), maps optical values to electronic/digital values. EOTF and OETF may be collectively referred to as transfer functions. The SI unit of brightness is candela per square meter (cd/m) ² ). The non-SI term for this unit is "NIT". In Standard Dynamic Range (SDR) imaging systems, in order to simplify the encoding process and decoding process in the encoding/decoding system or codec, a fixed transfer function, such as a fixed power law gamma transfer function, is typically used for the internal representation of video image content. With the advent of High Dynamic Range (HDR) imaging techniques, systems, and displays, a need for more flexible transfer functions has arisen.

Disclosure of Invention

Embodiments of video encoding and decoding systems and methods are described that implement adaptive transfer functions for use in video image content internal representations within video encoding and decoding systems or codecs. These implementations may dynamically determine a focus dynamic range for a current scene, sequence, frame, or region of a frame in an input video based on one or more characteristics of the image data (e.g., luminance, texture, etc.), crop the input video dynamic range to a focus range, and then appropriately map (e.g., quantize) values within the crop range from a bit depth of the input video to a bit depth of the codec according to a transfer function used to represent the video data in the codec.

In an implementation, various transfer functions may be used to represent the input video data and focus range video data in the codec. In some implementations, the transfer function used to represent video data in the codec may be the same as the transfer function used to represent input video data. In some implementations, video data in a codec may be represented using a transfer function (referred to as an internal transfer function or a secondary transfer function) that is different from the transfer function used to represent input video data (referred to as a primary transfer function).

In an embodiment, used by an encoder for a scene, sequence, frame,

Or the focus range of the region, transfer function, quantization parameters and other formats of information may be signaled to the decoder component, for example by metadata embedded in the output bitstream. In a decoder, an encoded bitstream may be decoded and dynamically extended to the full dynamic range of a target device (e.g., a High Dynamic Range (HDR) -enabled display) according to one or more signaled focus ranges of a scene, sequence, frame, or video region.

By dynamically adapting the transfer function to the input video data, embodiments may allow the video data to be represented in the codec with fewer bits than are used to represent the input video data, while also allowing the codec to output the video data such that it extends and fills the dynamic range of an HDR device such as an HDR-enabled display. An implementation of the adaptive transfer function method may, for example, enable a video encoding and decoding system to use 10 or less bits for internal representation and processing of video data within a codec, while when outputting video to an HDR device such as an HDR-enabled display, for example, using 12 or more bits to represent an extended dynamic range or high dynamic range of video data. Thus, an implementation of an adaptive transfer function method for a video encoding and decoding system may simplify the implementation of, and thus its adoption by, HDR technology, especially in the consumer space.

Drawings

Fig. 1 illustrates an exemplary codec or video encoding and decoding system implementing an embodiment of an adaptive transfer function.

Fig. 2 illustrates an exemplary encoder that applies an adaptive transfer function method to video input data and generates encoded video data, according to some embodiments.

Fig. 3 illustrates an exemplary decoder that decodes encoded video data according to an adaptive transfer function method and expands the video data decoded to generate display format video data, according to some embodiments.

Fig. 4 illustrates an exemplary full range of input video data and illustrates an exemplary focus range of video data, according to some embodiments.

Fig. 5 illustrates an example of mapping N-bit input video data within a focus range to generate C-bit video data, according to some embodiments.

Fig. 6 graphically illustrates an example of extending C-bit decoded video into the full dynamic range of an HDR-enabled device to generate D-bit video data for the device, in accordance with some embodiments.

Fig. 7A to 7C graphically illustrate the application of different focus ranges to different portions of a video sequence or video frame according to an embodiment of the adaptive transfer function method.

Fig. 8 is a flow chart of a video encoding method that applies an adaptive transfer function method to video input data and generates encoded video data, according to some embodiments.

Fig. 9 is a flow chart of a video decoding method that decodes encoded video data according to an adaptive transfer function method and expands the decoded video data to generate display format video data, according to some embodiments.

FIG. 10 is a block diagram of one embodiment of a system on a chip (SOC) that may be configured to implement aspects of the systems and methods described herein.

FIG. 11 is a block diagram of one embodiment of a system that may include one or more SOCs.

FIG. 12 illustrates an example computer system that can be configured to implement the various aspects of the systems and methods described herein, in accordance with some embodiments.

Fig. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments.

Fig. 14 depicts a portable multifunction device in accordance with some embodiments.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention. As used throughout this application, the word "possible" is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the word "include" means "including but not limited to".

Various units, circuits, or other components may be described as "configured to" perform a task or tasks. In such contexts, "configured to" is a broad expression of structure generally meaning "having" circuitry "that performs one or more tasks during operation. As such, the unit/circuit/component may be configured to perform this task even when the unit/circuit/component is not currently on. In general, the circuitry forming the structure corresponding to "configured to" may comprise hardware circuitry. Similarly, for ease of description, various units/circuits/components may be described as performing one or more tasks. Such descriptions should be construed to include the phrase "configured to". Detailed description of a unit/circuit/component configured to perform one or more tasks is intended to explicitly not refer to 35u.s.c. ≡112, the explanation of this unit/circuit/component in the sixth paragraph.

Detailed Description

Embodiments of video encoding and decoding systems and methods are described that implement adaptive transfer functions for use in video image content internal representations within video encoding and decoding systems or codecs. These implementations may allow the dynamic range of the input video data to be adapted to the codec during the encoding and decoding process. In an embodiment, the transfer function may be dynamically adapted to individual scenes, sequences, or frames of the input video to the codec. In some implementations, the transfer function may be dynamically adapted within the region of the frame. These implementations may dynamically adapt the transfer function to the input video data, retain only information within a dynamically determined effective dynamic range of the human visual system (referred to herein as the focus dynamic range or just the focus range), and map data within the focus range from the bit depth of the input video to the bit depth of the codec according to the transfer function for processing within the codec. The output of the codec may be extended to fill the dynamic range of an output device or target device, including but not limited to an extended dynamic range device or a High Dynamic Range (HDR) device, such as an HDR-enabled display. Fig. 1-3 illustrate an exemplary video encoding and decoding system or codec in which embodiments of the adaptive transfer function method may be implemented.

The human visual system as a whole covers a significant dynamic range. However, the human visual system tends to adapt and limit the dynamic range based on the current scene or image being viewed, for example, according to the brightness (illuminance) and texture characteristics of the scene or image. Thus, while the overall dynamic range of the human visual system is quite large, the effective dynamic range of a given scene, sequence, frame, or region of video may be quite small in terms of image characteristics, including, but not limited to, brightness and texture. Implementations of the video encoding and decoding systems and methods described herein may take advantage of this feature of human vision to employ similar strategies based on the characteristics of the current scene, sequence, frame, or region for dynamically limiting the range within the codec. In an embodiment, the encoder component or process may dynamically limit the range (e.g., brightness) of an input video scene, sequence, frame or region to within the effective dynamic range of the human visual system and within the required range (e.g., bit depth) of the codec, depending on one or more characteristics (e.g., brightness, texture, etc.) of the scene, sequence, frame or region currently being processed. This may be accomplished, for example, by dynamically determining the area within the current scene, sequence, frame, or region in the input video from the sample values and focusing the bit depth (e.g., 10 bits) of the codec into that range. In some implementations, focusing may be performed in the encoder by clipping the input video dynamic range to that area, and then appropriately mapping (e.g., quantizing) values in the clipping range (referred to as the focus dynamic range or focus range) from the bit depth of the input video to the bit depth of the codec according to a transfer function used to represent the video data in the codec. Fig. 4 and 5 graphically illustrate exemplary focus ranges determined for input video data and mapping N-bit input video data to a bit depth of a codec according to a transfer function, according to some embodiments. As used herein, N bits refer to the bit depth of the input video, C bits refer to the bit depth used to represent video data within the codec, and D bits refer to the bit depth of a target device (e.g., display device) that decodes the video.

In an implementation, various transfer functions may be used to represent the input video data and focus range video data in the codec. Examples of transfer functions that may be used to represent video data in an embodiment may include, but are not limited to: a gamma-based power law transfer function, a logarithmic-based transfer function, and a human visual perception-based transfer function, such as the Perception Quantizer (PQ) transfer function proposed by Dolby Laboratories, inc. In some implementations, the transfer function used to represent video data in the codec may be the same as the transfer function used to represent input video data. However, in other implementations, video data in the codec may be represented using a transfer function (referred to as an internal or secondary transfer function) that is different from the transfer function used to represent the input video data (referred to as a primary transfer function). This may allow video data (which may also be referred to as video signals) to be represented with a higher accuracy within a codec, for example, than may be accomplished using a one-level transfer function.

In an embodiment, information for a scene, sequence, focus range of a frame or region, transfer function, quantization parameters, and other formats of an encoder may be signaled to a decoder component, for example, by metadata embedded in an output bitstream. At the decoder, the encoded bitstream may be decoded and dynamically extended to the full dynamic range of a target device, such as an HDR device (e.g., an HDR-enabled display), according to one or more signaled focus ranges of a scene, sequence, frame, or video region. Figure 6 graphically illustrates extending decoded video data into the full dynamic range of an HDR device, in accordance with some embodiments.

The adaptive transfer function method described herein may be applied to all color components of a video signal, or alternatively may be applied to one or both of the luminance component and the chrominance component of a video signal, respectively.

Fig. 1 illustrates an exemplary video encoding and decoding system or codec 100 implementing an embodiment of an adaptive transfer function. In an embodiment, according to the adaptive transfer function method, the adaptive transfer function 110 component or module of the codec 100 may convert N-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) video input data 102 to C-bit (e.g., 10-bit or less) video input data 112 and to the encoder 120 component or module of the codec 100. In some implementations, to convert the N-bit video input data 102 to the C-bit video input data 112, the adaptive transfer function 110 may dynamically determine an area from sample values and focus C-bits into the area within a current scene, sequence, frame, or region in the N-bit video input data 102. In some implementations, this focusing may be done in adaptive transfer function 110 by clipping N-bit video input data 102 into a determined focus range, and then appropriately mapping (e.g., quantizing) the N-bit values in the focus range to the bit depth of the codec according to the transfer function used to represent the video data in encoder 120, thereby generating C-bit video input data 112 for encoder 120.

In some implementations, the adaptive transfer function 110 may also generate the format metadata 114 as an output to the encoder 120. The format metadata 114 may describe the adaptive transfer function method as applied to the input video data 102. For example, format metadata 114 may indicate a determined focus range, parameters for mapping video data to encoder bit depth, and may also contain information about a transfer function applied to the focus range.

In some implementations, the encoder 120 may then encode the C-bit video input data 112 to generate as output an encoded stream 122. In some implementations, the encoder 120 may encode the C-bit video input data 112 according to a compressed video format, such as an h.264/Advanced Video Coding (AVC) format or an h.265/High Efficiency Video Coding (HEVC) format. However, other encoding formats may be used. In some implementations, the encoder 120 may, for example, embed the format metadata 114 into the output stream 122 such that the format metadata 114 may be provided to the decoder 130 component. The output encoded stream 122 may be stored to memory or alternatively may be sent directly to the decoder 130 component or module of the codec 100. Fig. 2 illustrates exemplary encoder components of video encoding and decoding system or codec 100 in more detail.

The decoder 130 may read or receive the encoded stream 122 and decode the encoded stream 122 to generate a C-bit decoded stream 132 as an output to an inverse adaptive transfer function 140 component or module of the codec 100. In some embodiments, the format metadata 134 describing the adaptive transfer function method as being applied to the input video data 102 by the adaptive transfer function 110 may be extracted from the input stream 122 and passed to the inverse adaptive transfer function 140. The inverse adaptive transfer function 140 may convert the C-bit decoded stream 132 according to the format metadata 134 and display information 144 to generate a D-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) HDR output 142 to one or more displays 190 or other devices. Fig. 3 illustrates exemplary decoder components of video encoding and decoding system or codec 100 in more detail.

Embodiments of a video encoding and decoding system or codec 100 implementing an adaptive transfer function method as described herein may be implemented, for example, in a device or system that includes one or more image capture devices and/or one or more display devices. The image capturing device may be any device comprising an optical sensor or light sensor capable of capturing digital images or video. Image capture devices may include, but are not limited to, video cameras and still image cameras, as well as image capture devices that may capture both video and single images. The image capture device may be a stand alone device or may be a camera integrated into other devices including, but not limited to, smart phones, mobile phones, PDAs, tablet or tablet devices, multi-function devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and the like. It is noted that the image capturing device may comprise a low profile camera suitable for small devices such as mobile phones, PDAs and tablet devices. The display or display device may include a display screen or panel integrated into other devices including, but not limited to, smart phones, mobile phones, PDAs, tablet or tablet devices, multi-function devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and the like. The display device may also include a video monitor, projector, or generally any device that can display or project digital images and/or digital video. The display or display device may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may also be used.

Fig. 10-14 illustrate non-limiting examples of devices in which embodiments may be implemented. A device or system comprising an image capturing device and/or a display device may comprise hardware and/or software for implementing at least some of the functions for processing video data as described herein. In some embodiments, a portion of the functionality as described herein may be implemented on one device, while another portion may be implemented on another device. For example, in some embodiments, a device including an image capture device may implement a sensor pipeline that processes and compresses (i.e., encodes) an image or video captured via a light sensor, while another device including a display panel or screen may implement a display pipeline that receives and processes the compressed image (i.e., decodes) for display on a display or screen. In some embodiments, at least some of the functions as described herein may be implemented by one or more components or modules of a system on a chip (SOC) that may be used in a device including, but not limited to, a multi-function device, a smart phone, a tablet or tablet device, and other portable computing devices such as a laptop computer, a notebook computer, and a netbook computer. Fig. 10 shows an exemplary SOC, and fig. 11 shows an exemplary device implementing the SOC. FIG. 12 illustrates an exemplary computer system in which the methods and apparatus described herein may be implemented. Fig. 13 and 14 illustrate exemplary multi-function devices that may implement the methods and apparatus described herein.

Implementations are now generally described herein as processing video. However, instead of or in addition to processing video frames or video sequences, embodiments may also be applied to processing single images or still images. Thus, it should be understood that as "video," "video frame," "frame," and the like are used herein, these terms may refer generally to a captured digital image.

Fig. 2 illustrates an exemplary encoder 200 that applies an adaptive transfer function method to video input data 202 and generates encoded video data 232 as output, according to some embodiments. As shown in fig. 1, the encoder 200 may be, for example, a component or module of the codec 100. In an embodiment, the adaptive transfer function component or module 210 of the encoder 200 may convert the N-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) video input data 202 into C-bit (e.g., 10-bit or less) data 212 according to a transfer function method and output the C-bit data 212 to a processing pipeline 220 component of the encoder 200. In some implementations, to convert the N-bit video input data 202 to the C-bit video data 212, the adaptive transfer function module 210 may dynamically determine an area from sample values and focus C-bits into the area within a current scene, sequence, frame, or region in the N-bit video input data 202. In some implementations, this focusing may be done in adaptive transfer function module 210 by clipping the dynamic range of the N-bit video input data 202 to a determined focus range, and then mapping (e.g., quantizing) the N-bit values in the focus range to the C-bits appropriately according to the transfer function used to represent the video data in encoder 200. In some embodiments, adaptive transfer function module 210 may also generate format metadata 214 as an output to encoder 220. The format metadata 214 may describe the adaptive transfer function method as applied to the input video data 202. For example, format metadata 214 may indicate the determined focus range parameters for mapping video data to encoder bit depth, and may also contain information about the transfer function applied to the focus range.

In some implementations, the encoder 200 may then encode the C-bit video input data 212 generated by the adaptive transfer function module 210 to generate the encoded stream 232 as an output. In some implementations, the encoder 120 may encode the C-bit video input data 112 according to a compressed video format, such as an h.264/Advanced Video Coding (AVC) format or an h.265/High Efficiency Video Coding (HEVC) format. However, other encoding formats may be used.

In some implementations, input video frames are subdivided into encoders 200 and processed therein according to blocks of picture elements (referred to as pixels or primitives). For example, a 16×16 block of pixels, referred to as a macroblock, may be used for h.264 encoding. HEVC coding uses blocks known as Coding Tree Units (CTUs) that may vary from 16 x 16 pixels to 64 x 64 pixel blocks. CTUs may be divided into Coding Units (CUs) and may be further subdivided into Prediction Units (PUs) that may vary within a size that decreases from 64 x 64 pixels to 4 x 4 pixel blocks. In some implementations, the video input data 212 may be separated into luma and chroma components, and the luma and chroma components may be processed separately at one or more components or stages of the encoder.

In the exemplary encoder 200 shown in fig. 2, the encoder 200 includes a processing pipeline 220 component and an entropy encoding 230 component. The processing pipeline 220 may include a plurality of components or stages that process the video input data 212. The processing pipeline 220 may, for example, perform intra and inter estimation 222, mode decision 224, motion compensation and reconstruction 226 operations as one or more stages or components.

The following describes operation of the exemplary processing pipeline 220 at a high level and is not intended to be limiting. In some implementations, intra and inter estimates 222 may determine previously encoded pixel blocks to be used in encoding blocks input to the pipeline. In some video encoding techniques, such as h.264 encoding, each input block may be encoded using a block of pixels that are already encoded within the current frame. The process of determining these blocks may be referred to as intra-frame estimation or simply as intra-estimation. In some video encoding techniques, such as h.264 and h.265 encoding, blocks may also be encoded using blocks of pixels from one or more previously reconstructed frames (referred to as reference frames, shown in fig. 2 as reference data 240). In some implementations, the reconstructed and encoded frames output by the encoder pipeline may be decoded and stored to reference data 240 for use as reference frames. In some embodiments, the reference frames stored in the reference data 240 for reconstructing the current frame may include one or more reconstructed frames occurring in the video being processed temporally prior to the current frame, and/or one or more reconstructed frames occurring in the video being processed temporally later than the current frame. The process of finding matching pixel blocks in a reference frame may be referred to as inter-frame estimation, or more generally as motion estimation. In some implementations, the mode decision 224 may receive the output for a given block from the inter and intra estimates 222 and determine the best prediction mode (e.g., inter prediction mode or intra prediction mode) and the corresponding motion vector for the block. This information is passed to motion compensation and reconstruction 226.

The operation of motion compensation and reconstruction 226 may depend on the best mode received from mode decision 224. If the best mode is inter prediction, the motion compensation section obtains reference frame blocks corresponding to motion vectors and combines the blocks into a prediction block. The motion compensation component then applies weighted prediction to the prediction block to generate a final block prediction to be passed to the reconstruction component. In weighted prediction, values from reference data may be weighted according to one or more weighting parameters and shifted by offset values to generate prediction data to be used for encoding the current block. If the best mode is intra prediction, intra prediction is performed using one or more neighboring blocks to generate a predicted block of the current block being processed at that stage of the pipeline.

The reconstruction component performs a block (e.g., macroblock) reconstruction operation of the current block based on the motion compensation output. The reconstruction operations may include, for example, forward Transform and Quantization (FTQ) operations. The motion compensation and reconstruction 226 may output the transformed and quantized data to an entropy encoding 230 component of the encoder 200.

The entropy encoding 230 component may, for example, apply entropy encoding techniques to compress the transformed and quantized data output by the pipeline 220, generating an encoded output stream 232. Exemplary entropy encoding techniques that may be used may include, but are not limited to, huffman encoding techniques, CAVLC (context adaptive variable length coding) and CABAC (context adaptive binary arithmetic coding). In some implementations, the encoder 200 may embed the format metadata 214 into the encoded output stream 232 such that the format metadata 214 may be provided to the decoder. The output encoded stream 232 may be stored to memory or alternatively may be sent directly to the decoder element.

Reference data 240 may also be output by pipeline 220 and stored to memory. Reference data 240 may include, but is not limited to, one or more previously encoded frames (referred to as reference frames) that may be accessed, for example, in motion estimation and motion compensation and reconstruction operations of pipeline 220.

In some embodiments, encoder 200 may include a reformatting 250 component that may be configured to reformat reference frame data obtained from reference data 240 for use by pipeline 220 components when processing a current frame. Reformatting 250 may, for example, involve converting the reference frame data from a focus range/transfer function used to encode the reference frame to a focus range/transfer function to be used to encode the current frame. For example, the focus range/transfer function mapping of reference frame brightness may be from 0.05cd/m in C (e.g., 10) bits ² To 1000cd/m ² . For the current frame, the focus range may be extended due to brighter image content; for example, the focus range may be extended or increased to 2000cd/m ² . Thus, the focus range/transfer function mapping for the current frame brightness may be 0.05cd/m ² To 2000cd/m ² . To use the reference frame data in the pipeline 220 (e.g., in a motion estimation operation), the reference frame data may be reformatted according to a focus range/transfer function used by the reformatting 250 component to encode the current frame.

In the given example, the reformatting 250 component may convert the reference frame data from 0.05cd/m ² Shift to 1000cd/m ² A range within which it is from 0.05cd/m ² Range reconstruction to 2000cd/m for current frame ² Range. In this example, since the reference frame contains only from 0.05cd/m ² To 1000cd/m ² Thus some code words in the C-bit representation of the reformatted reference frame data (e.g., representing greater than 1000 cd/m) ² Code words of values of (c) may not be used in the reformatted reference data. However, multiple values are mapped to the current focus range for prediction orOther operations.

As indicated by the arrow returning from the reference data 240 to the adaptive transfer function module 210, in some embodiments, the adaptive transfer function module 210 may use adaptive transfer function information from one or more previously processed frames to determine a focus range/transfer function for the current frame to be processed in the pipeline 220.

Fig. 3 illustrates an exemplary decoder 300 that decodes the encoded video data 302 according to an adaptive transfer function method and expands the decoded video data 322 to generate display format video data 332, according to some embodiments. As shown in fig. 1, the decoder 300 may be, for example, a component or module of the codec 100. In the exemplary decoder 300 shown in fig. 3, the decoder 300 includes an entropy decoding 310 component, an inverse quantization and transform 320 component, and an inverse adaptive transfer function module 330. The entropy decoding 310 component may, for example, apply entropy decoding techniques to decompress the encoded stream 302 output by an encoder (e.g., encoder 200 shown in fig. 2). The inverse quantization and transform 320 component performs inverse quantization and transform operations on the data output by the entropy decoding 310 to generate a C-bit decoded stream 322 as output to the inverse adaptive transfer function module 330 of the decoder 300. In some embodiments, format metadata 324 describing the adaptive transfer function method as being applied to the current scene, sequence, frame, or region may be extracted from the input stream 302 and passed to the inverse adaptive transfer function module 330. Inverse adaptive transfer function module 330 may extend C-bit decoded stream 322 in accordance with format metadata 324 and display information 392 to generate D-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) HDR output 332 to one or more displays 390 or other devices.

Fig. 4-6 graphically illustrate an exemplary focus range for determining N-bit input video data, mapping the input video data to C-bits available within a codec according to a transfer function, and expanding the decoded video to the full dynamic range of an HDR device to generate D-bit video data for the device, according to some embodiments.

FIG. 4 illustrates an exemplary full view of N-bit input video data according to some embodimentsDynamic range, and illustrates an exemplary focus range determined for input video data. In fig. 4, the vertical axis represents N-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) code values in the input video data. The horizontal axis represents the dynamic range of luminance in the input video data, which in this example is 0cd/m ² -10000cd/m ² Wherein cd/m ² (candelas per square meter) is the SI unit of luminance. The non-SI term for this unit is "NIT". The curve represents an exemplary transfer function of the input video data. Focusing range (2000 cd/m in this example) ² -4000cd/m ² ) Representing the effective dynamic range of the human visual system of a current scene, sequence, frame or region determined from one or more characteristics (e.g., luminance, texture, etc.) of the corresponding video data. As can be seen, in this example, the focus range is represented by a ratio of N-bit code values (about 1/8). Note that different focus ranges may be determined for different scenes, sequences, frames, or regions within a video stream, as shown in fig. 7A-7C.

Fig. 5 illustrates an example of mapping N-bit input video data within a determined focus range to generate C-bit video data in a codec, according to some embodiments. In fig. 5, the vertical axis represents a C-bit (e.g., 12-bit, 10-bit, 8-bit, etc.) code value available in the codec. The horizontal axis shows the focus range of luminance values to be mapped to the C-bit code value (2000 cd/m in this example ² -4000cd/m ² ). The curve represents an exemplary transfer function for representing focus range data within the code. In some implementations, the transfer function used to represent video data in the codec may be the same as the transfer function used to represent input video data. However, in other implementations, video data in the codec may be represented using a transfer function (referred to as an internal or secondary transfer function) that is different from the transfer function used to represent the input video data (referred to as a primary transfer function). This may allow, for example, the focus range of the video signal to be represented with a higher accuracy within the codec than if a first order transfer function was used.

FIG. 6 illustrates expanding a C-bit decoded video according to some embodimentsAn example of a D-bit video data that is spread into the dynamic range of an HDR-enabled device to generate the device. In fig. 6, the vertical axis represents D-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) code values in video data output from a decoder (e.g., decoder 300 shown in fig. 3). The horizontal axis represents the dynamic range of luminance supported by the display device to which the D-bit video data is output by the decoder; in this example, the dynamic range is shown to be 0cd/m ² -10000cd/m ² . The curve represents an exemplary transfer function of the display device. After the encoded video signal is decoded by the decoder, the focus range for the signal internal representation in the codec is remapped from the C-bit representation to a wider dynamic range and a D-bit representation of the display device. Note that different focus ranges may be encoded for different scenes, sequences, frames, or regions within a video stream, as shown in fig. 7A-7C. In some embodiments, format metadata may be provided to a decoder in or with an encoded video stream that indicates a focus range for encoding each scene, sequence, frame, or region. In some implementations, the metadata may also include one or more parameters for mapping video data to a bit depth of the codec. In some embodiments, the metadata may further include information about an internal transfer function used to represent the focus range data in the codec. The metadata may be used by a decoder to remap focus range data from a C-bit representation to a wider dynamic range and a D-bit representation of the display device.

Fig. 7A graphically illustrates the application of different focus ranges to different scenes or sequences of an input video according to some embodiments of the adaptive transfer function method. In this example, focus range 710A is dynamically determined and used to encode frames 700A-700E, and focus range 710B is dynamically determined and used to encode frames 700F-700G. Frames 700A-700E and frames 700F-700H may, for example, represent different scenes within a video, or may represent different sequences of frames within or across multiple scenes in a video.

Fig. 7B graphically illustrates the application of a different focus range to each frame of the input video according to some embodiments of the adaptive transfer function method. In this example, focus range 710C is dynamically determined and used to encode frame 700I, focus range 710D is dynamically determined and used to encode frame 700J, and focus range 710E is dynamically determined and used to encode frame 700K.

Fig. 7C graphically illustrates the application of different focus ranges to different regions of a video frame according to some embodiments of the adaptive transfer function method. As shown in fig. 7A and 7B, in some embodiments, an adaptive transfer function method may be applied to a frame, scene, or sequence. However, in some implementations, the adaptive transfer function method may be applied to two or more regions 702 within a frame, as shown in fig. 7C, in addition or instead. In this example, focus range 710F is dynamically determined and used to encode region 702A of frame 700L, focus range 710G is dynamically determined and used to encode region 702B of frame 700L, and focus range 710H is dynamically determined and used to encode region 702C of frame 700L. In various embodiments, the region 702 to be encoded according to the adaptive transfer function method may be rectangular or may have other shapes, including but not limited to randomly determined irregular shapes. In some implementations, there may be a fixed number of regions 702 in each frame to be encoded according to the adaptive transfer function method. In some implementations, the number of regions 702 and/or the shape of the regions 702 to be encoded according to the adaptive transfer function method may be determined for each frame, scene, or sequence. In some embodiments, format metadata generated by the encoder and passed to the decoder may indicate the focus range, quantization parameters, and other information used to encode each region 702 in one or more frames 700. In implementations where a secondary transfer function or an internal transfer function is used within the codec, the format metadata may contain information for converting from the secondary transfer function to the primary transfer function. In embodiments where the number of regions 702 may vary, an indication of the actual number of regions 702 may be included in metadata that is passed to the decoder. In some embodiments, the coordinates and shape information of region 702 may also be included in metadata that is passed to the decoder.

In video content, the various parameters used in the adaptive transfer function method (e.g., focus range, quantization parameters, etc.) may be similar from one region or frame to the next in the input video being processed, especially within the scene. Thus, in some embodiments, the adaptive transfer function method may provide intra (spatial) prediction and/or inter (temporal) prediction of one or more of these adaptive transfer function parameters in the encoder, for example by utilizing weighted prediction processes in codecs such as AVC codec and HEVC codec. In inter (temporal) prediction, previously processed reference data from one or more temporally past or future frames may be used to predict one or more parameters for content in the current frame. In intra (spatial) prediction, data from one or more neighboring blocks or regions within a frame may be used to predict one or more parameters for content in the current block or region.

Fig. 8 is a high level flow chart of a video encoding method that applies an adaptive transfer function method to video input data and generates encoded video data, according to some embodiments. As indicated at 800 of fig. 8, N-bit (e.g., 12-bit, 14-bit, 16-bit, etc.) video data may be received for encoding. In some embodiments, frames of the video stream are processed sequentially by an encoder. In some implementations, an input video frame is subdivided into encoders and processed by the encoder according to blocks of pixels (e.g., macroblocks, CUs, PUs, or CTUs).

As indicated at 802 of fig. 8, a focus range of the input video data may be determined. In some implementations, the focus range for each scene, sequence, or frame of video input to the encoder can be dynamically determined. In some embodimentsIn this case, the focus range of each of two or more regions within a frame may be dynamically determined. In some implementations, the focus range can be determined based on one or more characteristics (e.g., brightness, texture, etc.) of the current scene, sequence, frame, or region being processed. The focus range represents the effective dynamic range of the human visual system for image data (e.g., brightness) in a current scene, sequence, frame, or region. For example, if the luminance dynamic range in the input video data is 0cd/m ² -10000cd/m ² An exemplary focus range based on specific features of various scenes, sequences, frames, or regions may be 2000cd/m ² -4000cd/m ² 、0cd/m ² -1000cd/m ² 、1000cd/m ² -2500cd/m ² Etc. Fig. 4 graphically illustrates an exemplary focus range determined for N-bit input video data, in accordance with some embodiments.

As indicated at 804 of fig. 8, N-bit video data within the focus range may be mapped to C-bit video data according to a transfer function. In some implementations, the input video data may be cropped according to the determined focus range, and then the cropped data values may be appropriately mapped (e.g., quantized) to the available C bits in the encoder according to the transfer function used to represent the video data in the encoder. Fig. 5 graphically illustrates an exemplary mapping of N-bit input video data to available C-bits in an encoder according to an exemplary focus range of a transfer function, in accordance with some embodiments. In an implementation, various transfer functions may be used to represent N-bit input video data and C-bit video data in an encoder. Examples of transfer functions that may be used to represent video data in embodiments may include, but are not limited to, gamma-based power law transfer functions, logarithmic-based transfer functions, transfer functions based on human visual perception, such as PQ transfer functions. In some implementations, the transfer function used to represent the C-bit video data in the encoder may be the same as the transfer function used to represent the N-bit input video data. However, in other implementations, the input video data may be represented according to a first-order transfer function, and the video data in the encoder may be represented using a different (second-order) transfer function. The secondary transfer function may represent the video signal with a higher accuracy, for example, within the encoder than would be possible using the primary transfer function.

As shown by element 806 of fig. 8, the adaptive transfer function processing performed at elements 800 through 804 will iterate as long as there are N-bit input video data (e.g., scenes, sequences, frames, or regions) to be processed. As indicated at 810 of fig. 8, per unit C-bit video data (e.g., scene, sequence, frame, or region) output by the adaptive transfer function method at elements 800 through 804 is input to and processed by an encoder. As previously mentioned, in some implementations, the N-bit input video data may be subdivided into and processed by adaptive transfer function modules or components, and then passed to the encoder as C-bit video data from blocks of pixels (e.g., macroblocks, CUs, PUs, or CTUs).

As indicated at 810 of fig. 8, one or more components of the encoder may process the C-bit video data to generate an encoded (and compressed) video data output (e.g., CAVLC output or CABAC output). In some embodiments, the encoder may encode the C-bit video input data according to a compressed video format, such as the h.264/AVC format or the h.265/HEVC format. However, other encoding formats may be used. In some embodiments, the focus range, transfer function, quantization parameters, and other format information used to encode each scene, sequence, frame, or region may be embedded as metadata in the encoded output stream, or may be signaled to one or more decoders by other means. Encoder 200 of fig. 2 illustrates an exemplary encoder for processing C-bit video data to generate an encoded output stream. In some implementations, the encoded output stream including metadata may be written to memory, for example, via Direct Memory Access (DMA). In some implementations, instead of or in addition to writing the output stream and metadata to memory, the encoded output stream and metadata may be sent directly to at least one decoder. The decoder may be implemented on the same or different devices and apparatuses as the devices and apparatuses implementing the encoder.

Fig. 9 is a high-level flow chart of a video decoding method that decodes encoded video data according to an adaptive transfer function method and expands the decoded video data to generate display format video data, according to some embodiments. As indicated at 900 of fig. 9, the decoder may obtain encoded data (e.g., CAVLC or CABAC encoded and compressed data). The encoded data may be read from memory, received from an encoder, or otherwise acquired, for example. As indicated at 902 of fig. 9, the decoder may decode the encoded data to generate decoded C-bit video data and format metadata. In some implementations, the decoder may decode the encoded data according to a compressed video format, such as the h.264/AVC format or the h.265/HEVC format. However, other encoding/decoding formats may be used. The format metadata extracted from the encoded data may include, for example, focus ranges, transfer functions, quantization parameters, and other format information used to encode each scene, sequence, frame, or region.

As shown by element 904 of fig. 9, the decoding performed at 900 and 902 may proceed as long as encoded data is present. As indicated at 910 of fig. 9, each unit of decoded data (e.g., each scene, sequence, frame, or region) decoded by the decoder may be output and processed according to an inverse adaptive transfer function method. At least a portion of the format metadata extracted from the encoded data may also be output to an inverse adaptive transfer function method.

As indicated at 910 of fig. 9, the inverse adaptive transfer function method may dynamically extend the decoded C-bit video data from the focus range to generate full dynamic range D-bit video data for output to a target device, such as an HDR-enabled display device. For a given scene, sequence, frame or region, the expansion may be performed dynamically according to the corresponding format metadata extracted from the encoded data by the decoder. Figure 6 graphically illustrates extending decoded video data into the full dynamic range of an HDR device in accordance with some embodiments

First order transfer function and second order transfer function

Examples of transfer functions that may be used to represent video data in embodiments of the adaptive transfer function method as described herein may include, but are not limited to, gamma-based power law transfer functions, logarithmic (log-based) transfer functions, and transfer functions based on human visual perception, such as the Perception Quantizer (PQ) transfer functions proposed by Dolby Laboratories, inc.

In some implementations, the transfer function used to represent video data in the codec may be the same as the transfer function used to represent input video data (referred to as a first order transfer function). In these implementations, the input video data may be cropped according to the focus range determined for the scene, sequence, frame, or region and then mapped (e.g., quantized) into the available bits of the codec according to a first order transfer function. The focus range, clipping, and mapping (e.g., quantization) parameters may be signaled to the decoder, for example, as metadata in the output encoded stream, such that the inverse adaptive transfer function method may be performed on the decoder to generate full range video data for the device (e.g., HDR-enabled display).

However, in some implementations, a transfer function other than the primary transfer function (referred to as an internal transfer function or a secondary transfer function) may be used to represent video data within the codec. In these implementations, the input video data may be cropped to a focus range determined for a scene, sequence, frame, or region, and then mapped, scaled, or quantized into the available bits of the codec according to a secondary transfer function. The secondary transfer function may, for example, allow video signals to be represented with higher precision within the codec than would be possible using the primary transfer function. In these implementations, in addition to the focus range, clipping, and mapping (e.g., quantization) parameters, information about how to convert video data from the secondary transfer function to the primary transfer function may also be signaled to the decoder, e.g., as metadata in the output stream. The information may include, but is not limited to, the type of secondary transfer function (e.g., power law gamma transfer function, logarithmic transfer function, PQ transfer function, etc.) and one or more control parameters of the transfer function. In some implementations, information describing the first order transfer function may also be signaled.

In some embodiments of the adaptive transfer function method, the internal transfer function or secondary transfer function used within the codec may be a discontinuous transfer function representation, where one or more portions of the original transfer function or primary transfer function of the internal representation may be preserved, while the remainder of the internal representation employs a different transfer function. In these embodiments, one or more regions of the reserved primary transfer function may be accurately described and signaled in the output stream, for example, using a tabular representation. Information about the different transfer functions may also be signaled.

As shown in fig. 7A to 7C, an implementation of the adaptive transfer function method may be performed at a frame level or a region level within a frame. Thus, intra-frame adjacent or neighboring encoded and decoded video frames or regions may have significantly different transfer function representations, which may have, for example, a negative impact on encoding (e.g., motion estimation, motion compensation, and reconstruction), because the transfer function representation of the current frame or region being encoded may be significantly different from that of the adjacent frames or regions available for the encoding process. However, in some implementations, weighted prediction that may be provided in a codec using coding formats such as Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) formats may be used to substantially increase coding efficiency. In these implementations, weighting parameters may be provided for frames or regions that may be used in a weighted prediction process to adjust for differences in the transfer function representation. The weighting parameter may be signaled, for example, by including the parameter in frame or block metadata. For example, in AVC or HEVC encoders, weighting parameters may be signaled in the slice header metadata and may be selected by changing reference indices within macroblocks in AVC or within Prediction Units (PUs) in HEVC. In some implementations, the weighting information may be explicitly signaled from the block, either related to the slice header portion or not. For example, in some implementations, a primary or master weight may be signaled in the slice header and have delta or difference that may be used to adjust the primary weight signaled at the block level. In some implementations, the weighting parameters may also include color weighting information. In some embodiments, these weighting parameters may be used for intra prediction as well as inter prediction. For example, in intra prediction, instead of or in addition to using previously processed neighboring or neighboring data (e.g., pixels or blocks of pixels) as such, additional weighting parameters and offset parameters may be provided that may be used to adjust the prediction according to possibly different transfer function characteristics of neighboring or neighboring samples.

In some embodiments where an internal transfer function or a secondary transfer function is used to represent data within the codec, in addition to the weighting parameters, information about the secondary transfer function may also be signaled in the codec for transfer function prediction. In some implementations, the secondary transfer function information may be signaled, for example, in a slice header using a transfer function table, such that each reference data index is associated with one or more transfer function adjustment parameters. In some implementations, the secondary transfer function information may be signaled at the block level instead or in addition. In implementations using discontinuous transfer function representations, multiple weighting parameters (e.g., parameters having different effects on different luminance values or levels) may be signaled in the slice header and/or at the block level.

In some embodiments, where an internal or secondary transfer function is used and adjusted to represent data within the codec, the internal transfer function and adjustments for a scene, sequence, frame, or region may be dynamically determined. In some implementations, the internal transfer function may be determined based on one or more characteristics of the current video frame or one or more regions of the frame. The features may include, but are not limited to, minimum and peak luminance, motion, texture, color, histogram mode, percentage concentration, and the like. In some implementations, determining the internal transfer function of the current frame may also utilize temporally past and/or future frames as reference frames. In some implementations, for example, determining the internal transfer function may utilize windows of one, two, or more reference frames that occur before and/or after a current frame in a video or video sequence. The use of window-based information to determine the internal transfer function may result in a change in characteristics of each frame and may help avoid or smooth too pronounced jumps or discontinuities in the internal transfer function adjustment that may otherwise adversely affect encoding. The smooth, window-based adaptive transfer function approach may provide better transfer function selection for encoding purposes and for final signal reconstruction at full transfer function range/desired transfer function range. In some implementations, the window-based adaptive transfer function method may include estimating a peak luminance and a minimum luminance of frames within a window to determine an internal transfer function. In some implementations, the luminance histogram, histogram pattern, concentration of values and how to adjust them, and the partitioning of areas or regions within the window can be estimated by a window-based method for determining the internal transfer function. In some embodiments, information about the human visual system may be utilized in this selection to achieve improved performance.

In some embodiments, the decision windows used in the window-based adaptive transfer function approach may not overlap and the decision is made separately for each frame or region. In some implementations, there may be overlap at window boundaries while boundary frames outside the current window are considered in the estimation, but are not themselves adjusted. In some implementations, multiple windows may affect boundary frames through an adaptive process, such as a weighted average of internal transfer function information based on window distance, which may help ensure a smooth transition between adjacent windows. In some embodiments, the window-based decision process for internal transfer functions may be based on a "run window" approach, where the internal transfer function for each frame is determined from characteristics of the frame itself and characteristics of one or more temporally past and/or future frames and/or windows. In some embodiments, the internal transfer function of the current frame may also be determined using the internal transfer functions of past and/or future frames and/or windows. In some embodiments, a multi-channel approach may be implemented in which the behavior and performance of the internal transfer function pre-selected in the previous channel is taken into account to tune the subsequent internal transfer function to achieve higher coding efficiency and better presentation of the given final transfer function objective.

In some implementations, the internal transfer function (e.g., range, type, bit depth, etc.) may be determined and adjusted based on characteristics of the video data being encoded. In some implementations, the compressibility or characteristics of a particular transfer function of the signal of interest may also be considered in selecting and/or adjusting the internal transfer function. In some implementations, the internal transfer function may additionally or alternatively be determined and adjusted based on one or more target displays and their characteristics or limitations. For example, if the current target display is known to have a limited dynamic range compared to the dynamic range supported by the first order transfer function, it may not be significant to include values outside the target display range in the encoded compressed signal. Instead, an internal transfer function representation may be determined that may provide a best fit of the signal to the dynamic range of the display, which may allow for a better compressed representation of the signal for that particular display capability. In some implementations, if multiple displays are to be supported by the codec, a dynamic range may be selected for the displays, and an internal transfer function representation that provides a best fit to the selected dynamic range may be determined. For example, the best capable display (e.g., with the highest dynamic range) may be selected and its dynamic range may be used to adjust the internal transfer function to generate video output for all displays. As another example, the dynamic range may be selected based on a pricing model in which one or more characteristics of the display (e.g., dynamic range) may be weighted (e.g., based on a ranking of the display determined or indicated based on importance or other factors).

Implementations of adaptive transfer function methods for video encoding and decoding systems or codecs as described herein may provide functionality to signal (from the encoder) and support (from the encoder and decoder) extending dynamic range or high dynamic range while keeping the complexity (e.g., bit depth) of the codec within reasonable limits. This may be accomplished by dynamically determining the focus range at the region, frame, scene or sequence level, cropping the input data to the focus range, and mapping the cropped data from the bit depth of the input video to the bit depth of the codec, while signaling appropriate parameters (e.g., focus range, quantization parameters, etc.) from the encoder to the decoder at the region, frame, scene or sequence level so that the dynamic range may be extended to the full range of the HDR-enabled display. Additionally, in some implementations, when different transfer function representations are used to encode different frames, appropriate parameters (e.g., weights, etc.) for performing the weighting parameters may be signaled.

Exemplary apparatus and devices

Fig. 10-14 illustrate non-limiting examples of devices and apparatus in or with which embodiments or components of various digital video or image processing and display methods and apparatus as described herein may be implemented. Fig. 10 shows an exemplary SOC, and fig. 11 shows an exemplary device implementing the SOC. FIG. 12 illustrates an exemplary computer system in which the methods and apparatus described herein may be implemented. Fig. 13 and 14 illustrate exemplary multi-function devices that may implement the methods and apparatus described herein.

Exemplary System on a chip (SOC)

Referring now to FIG. 10, a block diagram of one embodiment of a System On Chip (SOC) 8000 that may be used in the embodiments is shown. SOC 8000 is illustrated as coupled to memory 8800. As its name implies, the components of SOC 8000 may be integrated onto a single semiconductor substrate as an integrated circuit "chip". In some embodiments, these components may be implemented on two or more discrete chips in a system. However, SOC 8000 will be used herein as an example. In the illustrated embodiment, the components of the SOC 8000 include a Central Processing Unit (CPU) complex 8020, on-chip peripheral components 8040A-8040C (more briefly referred to as "peripherals"), a Memory Controller (MC) 8030, and a communication structure 8010. The

components

8020,8030,8040A-8040C can all be coupled to a communication structure 8010. The memory controller 8030 may be coupled to the memory 8800 during use, and the peripheral 8040B may be coupled to the external interface 8900 during use. In the illustrated embodiment, the CPU complex 8020 includes one or more processors (P) 8024 and a level two (L2) cache 8022.

Peripherals 8040A-8040B may be any collection of additional hardware functions that are included in SOC 8000. For example, the peripherals 8040A-8040B may include video peripherals such as an image signal processor configured to process image capture data from a camera or other image sensor, a display controller configured to display video data on one or more display devices, a Graphics Processing Unit (GPU), a video encoder/decoder or codec, a scaler, a rotator, a mixer, and the like. The peripheral devices may include audio peripheral devices such as microphones, speakers, interfaces to microphones and speakers, audio processors, digital signal processors, mixers, and the like. The peripheral may include a peripheral interface controller (e.g., peripheral 8040B) for various interfaces 8900 external to SOC 8000, including interfaces such as Universal Serial Bus (USB) ports, peripheral Component Interconnect (PCI) ports (including PCI express (PCIe) ports), serial ports, parallel ports, and the like. The peripheral device may include a networked peripheral device such as a Media Access Controller (MAC). Any collection of hardware may be included.

The CPU complex 8020 may include one or more CPU processors 8024 that function as CPUs for the SOC 8000. The CPU of the system includes one or more processors that execute system main control software, such as an operating system. Typically, software executed by the CPU during use may control other components of the system to achieve the desired functionality of the system. The processor 8024 may also execute other software, such as applications. The application may provide functionality for the user and may rely on the operating system for lower level device control. Thus, the processor 8024 may also be referred to as an application processor. The CPU complex 8020 may further include other hardware, such as an L2 cache 8022 and/or interfaces to other components of a system (e.g., interfaces to a communication structure 8010). In general, a processor may include any circuitry and/or microcode configured to execute instructions defined in an instruction set architecture implemented by the processor. Instructions and data operated on by the processor in response to executing the instructions may generally be stored in memory 8800, although some instructions may be defined as also having direct processor access to peripheral devices. Processors may encompass processor cores implemented on other components having integrated circuits as a system on a chip (SOC 8000) or other level of integration. The processor may further include discrete microprocessors, processor cores and/or microprocessors integrated into a multi-chip module implementation, processors implemented as multiple integrated circuits, and the like.

The memory controller 8030 may generally include circuitry for receiving memory operations from other components of the SOC 8000 and for accessing the memory 8800 to complete the memory operations. The memory controller 8030 may be configured to access any type of memory 8800. For example, the memory 8800 can be Static Random Access Memory (SRAM), dynamic RAM (DRAM), such as Synchronous DRAM (SDRAM) including double data rate (DDR, DDR2, DDR3, etc.) DRAM. Low power/mobile versions of DDR DRAM (e.g., LPDDR, mDDR, etc.) may be supported. The memory controller 8030 may include a memory operation queue for ordering (and possibly reordering) the operations and presenting the operations to the memory 8800. The memory controller 8030 may also include a data buffer for storing write data waiting to be written to memory and read data waiting to be returned to the source of the memory operation. In some embodiments, the memory controller 8030 may include a memory cache for storing recently accessed memory data. For example, in SOC implementations, the memory cache may reduce power consumption in the SOC by avoiding re-accessing data from memory 8800 in anticipation of a quick to re-access. In some cases, this memory cache may also be referred to as a system cache, as opposed to a private cache (such as the L2 cache 8022 or a cache in the processor 8024), which only serves certain components. Furthermore, in some embodiments, the system cache need not be located within the memory controller 8030.

In one embodiment, the memory 8800 may be packaged with the SOC 8000 in a chip-on-chip configuration or a package-on-package configuration. A multi-chip module configuration of SOC 8000 and memory 8800 may also be used. Such a configuration may be relatively more secure (in terms of data observability) than transmissions to other components in the system (e.g., to endpoints 16A-16B). Thus, the protected data may reside unencrypted in memory 8800, while the protected data may be encrypted for exchange between SOC 8000 and external endpoints.

The communication structure 8010 may be any communication interconnect and protocol for communicating between components of the SOC 8000. The communication fabric 8010 may be bus-based, including shared bus configurations, crossbar configurations, and layered buses with bridges. The communication fabric 8010 may also be packet-based and may be a hierarchy with bridges, a crossbar, point-to-point, or other interconnectors.

It is noted that the number of components of SOC 8000 (and the number of sub-components of those components shown in fig. 10 (such as within CPU complex 8020) may be different in different embodiments. There may be more or less of each component/sub-component than shown in fig. 10.

Fig. 11 is a block diagram of one embodiment of a system 9000 comprising at least one instance of SOC 8000 coupled to an external memory 8800 and one or more external peripherals 9020. A Power Management Unit (PMU) 9010 is provided that supplies a supply voltage to the SOC 8000 and one or more supply voltages to the memory 8800 and/or the peripheral 9020. In some implementations, more than one instance of SOC 8000 may be included (more than one memory 8800 may also be included).

The peripheral device 9020 may include any desired circuitry, depending on the type of system 9000. For example, in one embodiment, the system 9000 can be a mobile device (e.g., personal Digital Assistant (PDA), smart phone, etc.), and the peripheral device 9020 can comprise a device for various types of wireless communications, such as wifi, bluetooth, cellular, global positioning system, etc. The peripheral device 9020 may also include additional storage, including RAM storage, solid state storage, or disk storage. The peripheral devices 9020 may include user interface devices (such as a display screen including a touch display screen or a multi-touch display screen), a keyboard or other input device, a microphone, a speaker, and the like. In other embodiments, the system 9000 may be any type of computing system (e.g., desktop personal computer, laptop, workstation, network set-top box, etc.).

The external storage 8800 can comprise any type of memory. For example, the external memory 8800 can be SRAM, dynamic RAM (DRAM), such as Synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.), RAMBUS DRAM, low power version of DDR DRAM (e.g., LPDDR, mDDR, etc.), etc. The external memory 8800 can include one or more memory modules to which the memory device can be mounted, such as single rank memory modules (SIMMs), dual rank memory modules (DIMMs), and the like. Alternatively, the external memory 8800 can include one or more memory devices mounted on the SOC 8000 in a chip-on-chip configuration or package-on-package implementation.

Examples of multifunction devices

Fig. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments. In some embodiments, the device is a portable communication device such as a mobile phone that also includes other functions such as PDA, camera, video capture and/or playback, and/or music player functions. Exemplary embodiments of the portable multifunction device include, but are not limited to, those from Apple inc (Cupertino, california)

Device, iPod->

Equipment and->

An apparatus. Other portable electronic devices may also be used, such as having A laptop, mobile phone, smart phone, tablet or tablet computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the device is not a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad). In some embodiments, the device is a gaming computer having an orientation sensor (e.g., an orientation sensor in a game controller). In other embodiments, the device is not a portable communication device, but a camera and/or video camera.

In the following discussion, an electronic device including a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device may include one or more other physical user interface devices, such as a physical keyboard, mouse, and/or joystick.

The device typically supports various applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, workout support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and/or digital video player applications.

Various applications executable on the device may use at least one common physical user interface device such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device may be adjusted and/or changed from one application to the next and/or within the respective application. In this way, a common physical architecture of the device (such as a touch-sensitive surface) may support various applications with a user interface that is intuitive and transparent to the user.

Device 2100 can include memory 2102 (which can include one or more computer-readable storage media), a memory controller 2122, one or more processing units (CPUs) 2120, a peripheral interface 2118, RF circuitry 2108, audio circuitry 2110, speakers 2111, a touch-sensitive display system 2112, a microphone 2113, an input/output (I/O) subsystem 2106, other input control devices 2116, and external ports 2124. Device 2100 can include one or more optical sensors or cameras 2164. These components may communicate via one or more communication buses or signal lines 2103.

It should be understood that device 2100 is only one example of a portable multifunction device and that device 2100 may have more or fewer components than shown, may combine two or more components, or may have different configurations or arrangements of the components. The various components shown in fig. 13 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application specific integrated circuits.

Memory 2102 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to the memory 2102 by other components of the device 2100 (e.g., the CPU 2120 and the peripheral interface 2118) may be controlled by a memory controller 2122.

Peripheral interfaces 2118 can be used to couple the input and output peripherals of the device to the CPU 2120 and memory 2102. The one or more processors 2120 run or execute various software programs and/or sets of instructions stored in the memory 2102 to perform various functions of the device 2100 and process data.

In some embodiments, the peripheral interface 2118, CPU 2120, and memory controller 2122 may be implemented on a single chip, such as chip 2104. In some other embodiments, they may be implemented on separate chips.

The RF (radio frequency) circuit 2108 receives and transmits RF signals, also known as electromagnetic signals. The RF circuit 2108 converts/converts an electrical signal into/from an electromagnetic signal and communicates with a communication network and other communication devices via the electromagnetic signal. The RF circuitry 2108 may include well-known circuitry for performing these functions including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder/decoder (codec) chipset, a Subscriber Identity Module (SIM) card, memory, and so forth. The RF circuitry 2108 may communicate with networks such as the internet (also known as the World Wide Web (WWW)), intranets, and/or wireless networks such as cellular telephone networks, wireless Local Area Networks (LANs), and/or Metropolitan Area Networks (MANs), among other devices via wireless communications. The wireless communication may use any of a variety of communication standards, communication protocols, and communication technologies, including, but not limited to, global system for mobile communications (GSM), enhanced Data GSM Environment (EDGE), high Speed Downlink Packet Access (HSDPA), high Speed Uplink Packet Access (HSUPA), wideband code division multiple access (W-CDMA), code Division Multiple Access (CDMA), time Division Multiple Access (TDMA), bluetooth, wireless fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and/or IEEE 802.11 n), voice over internet protocol (VoIP), wi-MAX, email protocol (e.g., internet Message Access Protocol (IMAP), and/or Post Office Protocol (POP)), instant messaging (e.g., extensible message processing field protocol (XMPP), session initiation protocol for instant messaging and field usage extension (SIMPLE), instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed at the date of this document.

The audio circuitry 2110, speaker 2111 and microphone 2113 provide an audio interface between a user and the device 2100. The audio circuit 2110 receives audio data from the peripheral interface 2118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 2111. The speaker 2111 converts electrical signals into sound waves audible to the human ear. The audio circuit 2110 also receives an electrical signal that is converted by a microphone 2113 from sound waves. The audio circuit 2110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 2118 for processing. Audio data may be retrieved from and/or transferred to the memory 2102 and/or the RF circuitry 2108 by the peripheral interface 2118. In some embodiments, the audio circuit 2110 also includes a headset jack. The headset jack provides an interface between the audio circuit 2110 and a removable audio input/output peripheral, such as an output-only earphone or a headset having both an output (e.g., a monaural earphone or a binaural earphone) and an input (e.g., a microphone).

The I/O subsystem 2106 couples input/output peripheral devices on the device 2100, such as a touch screen 2112 and other input control devices 2116 to the peripheral device interface 2118. The I/O subsystem 2106 may include a display controller 2156 and one or more input controllers 2160 for other input control devices 2116. The one or more input controllers 2160 receive electrical signals from/send electrical signals to other input control devices 2116. Other input control devices 2116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative implementations, one or more input controllers 2160 may be coupled (or not coupled) to any of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. The one or more buttons may include an up/down button for controlling the volume of the speaker 2111 and/or microphone 2113. The one or more buttons may include a push button.

The touch sensitive display 2112 provides an input interface and an output interface between the device and a user. The display controller 2156 receives electrical signals from and/or transmits electrical signals to the touch screen 2112. The touch screen 2112 displays visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some implementations, some or all of the visual output may correspond to a user interface object.

Touch screen 2112 has a touch-sensitive surface, sensor, or set of sensors that receives input from a user based on haptic and/or tactile contact. Touch screen 2112 and display controller 2156 (along with any associated modules and/or sets of instructions in memory 2102) detect contact (and any movement or interruption of the contact) on touch screen 2112 and translate the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on touch screen 2112. In one exemplary embodiment, the point of contact between the touch screen 2112 and the user corresponds to a finger of the user.

Touch screen 2112 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, but other display technologies may be used in other embodiments. Touch screen 2112 and display controller 2156 may detect contact and any movement or interruption thereof using any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 2112. In one exemplary embodiment, a projected mutual capacitance sensing technique is used, such as that from Apple inc (Cupertino, california)

iPod/>

And->

The technology found in (a) is provided.

The touch screen 2112 may have a video resolution of over 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user may use any suitable object or appendage, such as a stylus, finger, etc., to make contact with the touch screen 2112. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the large contact area of the finger on the touch screen. In some embodiments, the device translates the finger-based coarse input into a precise pointer/cursor position or command for performing the action desired by the user.

In some embodiments, device 2100 may include a touchpad (not shown) for activating or deactivating particular functions in addition to touchscreen 2112. In some implementations, the touch pad is a touch sensitive area of the device that, unlike the touch screen, does not display visual output. The touch pad may be a touch-sensitive surface separate from the touch screen 2112 or an extension of the touch-sensitive surface formed by the touch screen.

The device 2100 also includes a power system 2162 for powering the various components. Power system 2162 may include a power management system, one or more power sources (e.g., battery, alternating Current (AC)), a recharging system, power failure detection circuitry, a power converter or inverter, a power status indicator (e.g., light Emitting Diode (LED)), and any other components associated with the generation, management, and distribution of power in portable devices.

Device 2100 can also include one or more optical sensors or cameras 2164. Fig. 13 shows optical sensors coupled to optical sensor controller 2158 in I/O subsystem 2106. The optical sensor 2164 may include, for example, a Charge Coupled Device (CCD), or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor or photosensor. The optical sensor 2164 receives light projected through one or more lenses from the environment and converts the light into data representative of an image. In conjunction with an imaging module 2143 (also referred to as a camera module), the optical sensor 2164 may capture still images and/or video sequences. In some embodiments, at least one optical sensor may be located at a rear of the device 2100 opposite the touch screen display 2112 at the front of the device. In some embodiments, the touch screen display may be used as a viewfinder for capturing still and/or video images. In some embodiments, alternatively or additionally, at least one optical sensor may be located at the front of the device.

Device 2100 can also include one or more proximity sensors 2166. Fig. 13 illustrates a proximity sensor 2166 coupled to the peripheral interface 2118. Alternatively, the proximity sensor 2166 may be coupled to an input controller 2160 in the I/O subsystem 2106. In some implementations, the proximity sensor turns off and disables the touch screen 2112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

Device 2100 can also include one or more orientation sensors 2168. In some embodiments, the one or more orientation sensors include one or more accelerometers (e.g., one or more linear accelerometers and/or one or more rotational accelerometers). In some embodiments, the one or more orientation sensors include one or more gyroscopes. In some embodiments, the one or more orientation sensors comprise one or more magnetometers. In some embodiments, the one or more orientation sensors include one or more of a Global Positioning System (GPS), a global navigation satellite system (GLONASS), and/or other global navigation system receivers. GPS, GLONASS, and/or other global navigation system receivers may be used to obtain information regarding the position and orientation (e.g., longitudinal or lateral) of device 2100. In some embodiments, the one or more orientation sensors include any combination of orientation sensors/rotation sensors. Fig. 13 illustrates one or more orientation sensors 2168 coupled to the peripheral interface 2118. Alternatively, the one or more orientation sensors 2168 may be coupled to the input controller 2160 in the I/O subsystem 2106. In some embodiments, information is displayed in a portrait view or a landscape view on a touch screen display based on analysis of data received from one or more orientation sensors.

In some embodiments, device 2100 may also include one or more other sensors (not shown), including, but not limited to, ambient light sensors and motion detectors. These sensors may be coupled to the peripheral interface 2118 or may be coupled to the input controller 2160 in the I/O subsystem 2106. For example, in some embodiments, device 2100 can include at least one forward (away from the user) light sensor and at least one backward (toward the user) light sensor that can be used to collect ambient light metrics from the environment in which device 2100 is located for use by video and image acquisition, processing, and display applications.

In some embodiments, the software components stored in memory 2102 include an operating system 2126, a communication module 2128, a contact/motion module (or set of instructions) 2130, a graphics module 2132, a text input module 2134, a Global Positioning System (GPS) module 2135, and an application program 2136. Further, in some embodiments, memory 2102 stores device/global internal state 2157. The device/global internal state 2157 includes one or more of the following: an active application state indicating which applications (if any) are currently active; display status indicating what applications, views, or other information occupy various areas of the touch screen display 2112; a sensor state including information obtained from the respective sensors of the devices and the input control device 2116; and location information regarding the device's location and/or pose.

The operating system 2126 (e.g., darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between the various hardware components and software components.

The communication module 2128 facilitates communication with other devices via one or more external ports 2124 and also includes various software components for processing data received by the RF circuitry 2108 and/or the external ports 2124. External port 2124 (e.g., universal Serial Bus (USB), firewire, etc.) is adapted to be coupled directly to other devices or indirectly via a network (e.g., the internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and/or compatible with a 30-pin connector used on an iPod (trademark of Apple inc.) device.

The contact/motion module 2130 may detect contact with the touch screen 2112 (in conjunction with the display controller 2156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). The contact/motion module 2130 includes a number of software components for performing various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting a finger press event), determining whether movement of the contact exists and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether contact has terminated (e.g., detecting a finger lift event or a contact break). The contact/motion module 2130 receives contact data from a touch-sensitive surface. Determining movement of the point of contact may include determining a velocity (magnitude), a speed (magnitude and direction), and/or an acceleration (change in magnitude and/or direction) of the point of contact, the movement of the point of contact being represented by a series of contact data. These operations may be applied to a single contact (e.g., one finger contact) or multiple simultaneous contacts (e.g., "multi-touch"/multiple finger contacts). In some embodiments, the contact/motion module 2130 and the display controller 2156 detect contact on the touchpad.

The contact/motion module 2130 may detect gesture inputs of a user. Different gestures on the touch-sensitive surface have different contact patterns. Thus, gestures may be detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger-down event, and then detecting a finger-up (lift-off) event at the same location (or substantially the same location) as the finger-down event (e.g., at an icon location). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift off) event.

Graphics module 2132 includes various software components for rendering and displaying graphics on touch screen 2112 or other display, including components for changing the intensity of the graphics displayed. As used herein, the term "graphic" includes any object that may be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, video, animation, and the like.

In some embodiments, graphics module 2132 stores data to be used to represent graphics. Each graphic may be assigned a corresponding code. The graphic module 2132 receives one or more codes designating graphics to be displayed from an application program or the like, together with coordinate data and other graphic attribute data as necessary, and then generates screen image data to output to the display controller 2156.

The text input module 2134, which may be a component of the graphics module 2132, provides a soft keyboard for entering text in a variety of applications requiring text input.

The GPS module 2135 determines the location of the device and provides such information to various applications for use by it (e.g., to the phone module 2138 for use by it in a location-based dialing process, to the camera module 2143 as picture/video metadata, to applications for providing location-based services, such as map/navigation applications).

The application 2136 may include, but is not limited to, one or more of the following modules (or sets of instructions), or a subset or superset thereof:

a telephony module 2138;

video conference module 2139;

a camera module 2143 for still imaging and/or video imaging;

an image management module 2144;

a browser module 2147;

search module 2151;

a video and music player module 2152, which may be comprised of a video player module and a music player module; and/or

Online video module 2155.

Examples of other applications 2136 that may be stored in memory 2102 include, but are not limited to, other word processing applications, other image editing applications, drawing applications, presentation applications, communication/social media applications, map applications, JAVA-enabled applications, encryption applications, digital rights management applications, voice recognition applications, and voice replication applications.

In conjunction with the RF circuitry 2108, the audio circuitry 2110, the speaker 2111, the microphone 2113, the touch screen 2112, the display controller 2156, the contact module 2130, the graphics module 2132, and the text input module 2134, the telephone module 2138 can be used to input a sequence of characters corresponding to a telephone number, access one or more telephone numbers in an address book, modify the telephone number that has been entered, dial the corresponding telephone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As described above, the wireless communication may use any of a variety of communication standards, protocols, and techniques.

In conjunction with RF circuitry 2108, audio circuitry 2110, speaker 2111, microphone 2113, touch screen 2112, display controller 2156, optical sensor 2164, optical sensor controller 2158, contact/motion module 2130, graphics module 2132, text input module 2134, and telephony module 2138, videoconferencing module 2139 includes executable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants according to user instructions.

In conjunction with touch screen 2112, display controller 2156, one or more optical sensors 2164, optical sensor controller 2158, contact/motion module 2130, graphics module 2132, and image management module 2144, camera module 2143 includes executable instructions for: capturing still images or video (including video streams) and storing them in memory 2102, modifying features of still images or video, or deleting still images or video from memory 2102.

In conjunction with the touch screen 2112, the display controller 2156, the contact/motion module 2130, the graphics module 2132, the text input module 2134, and the camera module 2143, the image management module 2144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slide or album), and storing still and/or video images.

In conjunction with RF circuitry 2108, touch screen 2112, display system controller 2156, contact/motion module 2130, graphics module 2132 and text input module 2134, browser module 2147 includes executable instructions for browsing the internet (including searching, linking to, receiving and displaying web pages or portions thereof, and attachments and other files linked to web pages) according to user instructions.

In conjunction with touch screen 2112, display system controller 2156, contact/motion module 2130, graphics module 2132 and text input module 2134, search module 2151 includes executable instructions for searching text, music, sound, images, video and/or other files in memory 2102 that match one or more search criteria (e.g., one or more search terms specified by a user) according to user instructions.

In conjunction with the touch screen 2112, display system controller 2156, contact/motion module 2130, graphics module 2132, audio circuitry 2110, speaker 2111, RF circuitry 2108 and browser module 2147, the video and music player module 2152 includes executable instructions that allow a user to download and playback recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, rendering or otherwise playing back video (e.g., on the touch screen 2112 or on an external display connected via external port 2124). In some embodiments, device 2100 may include the functionality of an MP3 player such as an iPod (trademark of Apple inc.).

In conjunction with the touch screen 2112, display system controller 2156, touch/motion module 2130, graphics module 2132, audio circuitry 2110, speaker 2111, RF circuitry 2108, text input module 2134, and browser module 2147, the online video module 2155 includes instructions that allow a user to access, browse, receive (e.g., by streaming and/or downloading), play back (e.g., on a touch screen or on an external display connected via external port 2124), and otherwise manage online video in one or more video formats such as h.264/AVC format or h.265/HEVC format.

Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the functions described above, as well as the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some implementations, the memory 2102 can store a subset of the modules and data structures identified above. Further, the memory 2102 may store additional modules and data structures not described above.

In some implementations, the device 2100 is a device that performs the operation of a predefined set of functions on the device solely through a touch screen and/or touchpad. The number of physical input control devices (such as push buttons, dials, etc.) on the device 2100 may be reduced by using a touch screen and/or touchpad as the primary input control device for operation of the device 2100.

The set of predefined functions that may be performed exclusively by the touch screen and/or touch pad include navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates the device 2100 from any user interface that may be displayed on the device 2100 to a main menu, home menu, or root menu. In such implementations, the touch pad may be referred to as a "menu button". In some other embodiments, the menu buttons may be physical push buttons or other physical input control devices, rather than touch pads.

Fig. 14 illustrates a portable multifunction device 2100 with a touch screen 2112 in accordance with some embodiments. The touch screen may display one or more graphics within a User Interface (UI) 2200. In at least some embodiments of device 2100, a user can gesture on graphics using, for example, one or more fingers 2202 (not necessarily drawn to scale in the figures) or using one or more styluses 2203 (not necessarily drawn to scale in the figures) to select one or more of the graphics.

The device 2100 may also include one or more physical buttons, such as a "home" button or menu button 2204. As previously described, menu button 2204 may be used to navigate to any application 2136 of a set of applications executable on device 2100. Alternatively, in some embodiments, the menu buttons may be implemented as soft keys in a GUI displayed on the touch screen 2112.

In one embodiment, device 2100 includes a touch screen 2112, a home button or menu button 2204, a push button 2206 for powering the device on and off and locking the device, one or more volume adjustment buttons 2208, a Subscriber Identity Module (SIM) card slot 2210, a headset jack 2212, and a docking/charging external port 2124. Pressing button 2206 may be used to turn on/off power on the device by pressing the button and holding the button in a pressed state for a predetermined time interval, to lock the device by pressing the button and releasing the button before the predetermined time interval elapses, and/or to unlock the device or initiate an unlocking process. In alternative embodiments, device 2100 may also accept voice input through microphone 2113 for activating or deactivating some functions.

Device 2100 can also include one or more cameras 2164. The camera 2164 may include, for example, a Charge Coupled Device (CCD), or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor or photosensor. The camera 2164 receives light projected through one or more lenses from the environment and converts the light into data representing images or video frames. In some embodiments, at least one camera 2164 may be located at a rear of the device 2100 opposite the touchscreen display 2112 at the front of the device. In some embodiments, alternatively or additionally, at least one camera 2164 may be located in front of the device having the touch-screen display 2112, for example, so that a user may obtain images of other video conference participants for video conferencing while watching the user on the touch-screen display 2112. In some embodiments, at least one camera 2164 may be located in a front of device 2100, and at least one camera 2164 may be located in a rear of device 2100. In some implementations, the touch screen display 2112 can be used as a viewfinder and/or user interface for still image and/or video sequence acquisition applications.

Device 2100 may include video and image processing hardware and/or software including, but not limited to, video encoding and/or decoding components, codecs, modules, or pipelines that may be used to capture, process, convert, compress, decompress, store, modify, transmit, display, otherwise manage and manipulate still images and/or video frames or video sequences captured or otherwise acquired (e.g., via a network interface) via camera 2164. In some embodiments, device 2100 may also include one or more light sensors or other sensors that may be used to collect ambient light metrics or other metrics from the environment in which device 2100 is located for video and image acquisition, processing, and display use.

Exemplary computer System

FIG. 12 illustrates an exemplary computer system 2900 that can be configured to perform any or all of the embodiments described above. In different embodiments, computer system 2900 may be any of a variety of types of devices, including, but not limited to: personal computer systems, desktop computers, laptop computers, notebook computers, tablet computers, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, consumer devices, application servers, storage devices, video recording devices, peripheral devices (such as switches, modems, routers), or generally any type of computing device or electronic device.

Various embodiments as described herein may be performed within one or more computer systems 2900 that may interact with various other devices. It is noted that, according to various embodiments, any of the components, acts, or functions described above in connection with fig. 1-11 may be implemented on one or more computers configured as the computer system 2900 shown in fig. 12. In the illustrated embodiment, the computer system 2900 includes one or more processors 2910 coupled to a system memory 2920 via an input/output (I/O) interface 2930. The computer system 2900 also includes a network interface 2940 coupled to the I/O interface 2930, and one or more input/output devices or components 2950, such as a cursor control 2960, a keyboard 2970, one or more displays 2980, one or more cameras 2990, and one or more sensors 2992, including but not limited to light sensors and motion detectors. In some cases, it is contemplated that an embodiment may be implemented using a single instance of computer system 2900, while in other embodiments, multiple such systems or multiple nodes comprising computer system 2900 may be configured to host different portions or instances of an embodiment. For example, in one embodiment, some elements may be implemented via one or more nodes of the computer system 2900 that are different from those nodes implementing other elements.

In various embodiments, computer system 2900 may be a single processor system including one processor 2910, or a multi-processor system including several processors 2910 (e.g., two, four, eight, or other suitable number). Processor 2910 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2910 may be a general-purpose processor or an embedded processor implementing any of a variety of Instruction Set Architectures (ISA), such as the x829, powerPC, SPARC or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each processor 2910 may typically, but need not necessarily, implement the same ISA.

The system memory 2920 may be configured to store program instructions 2922 and/or data accessible to the processor 2910. In various embodiments, the system memory 2920 may be implemented using any suitable memory technology, such as Static Random Access Memory (SRAM), synchronous Dynamic RAM (SDRAM), non-volatile/flash memory, or any other type of memory. In the illustrated embodiment, the program instructions 2922 may be configured to implement any of the functions described herein. Further, memory 2920 may include any of the information structures or data structures described herein. In some embodiments, program instructions and/or data may be received, transmitted, or stored on a different type of computer-accessible medium or similar medium, independent of system memory 2920 or computer system 2900. Although computer system 2900 is described as implementing the functions of the functional blocks of the previous figures, any of the functions described herein may be implemented via such a computer system.

In one embodiment, the I/O interface 2930 may be configured to coordinate I/O communications between the processor 2910, the system memory 2920, and any peripheral devices in the device, including the network interface 2940 or other peripheral device interfaces, such as the input/output device 2950. In some implementations, the I/O interface 2930 may perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., the system memory 2920) into a format suitable for use by another component (e.g., the processor 2910). In some implementations, the I/O interface 2930 may include support for devices attached, for example, through various types of peripheral bus, such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I/O interface 2930 may be divided into two or more separate components, such as a north bridge and a south bridge, for example. Further, in some embodiments, some or all of the functionality of the I/O interface 2930 (such as an interface to the system memory 2920) may be incorporated directly into the processor 2910.

The network interface 2940 may be configured to allow data to be exchanged between the computer system 2900 and other devices (e.g., bearers or proxy devices) attached to the network 2985, or between nodes of the computer system 2900. In various embodiments, network 2985 may include one or more networks including, but not limited to: a Local Area Network (LAN) (e.g., ethernet or enterprise network), a Wide Area Network (WAN) (e.g., the internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, the network interface 2940 may support communication via a wired or wireless general-purpose data network, such as any suitable type of ethernet; communication is via a telecommunications/telephony network, such as an analog voice network or a digital fiber optic communication network; communication is via a storage area network, such as a fibre channel SAN, or via any other suitable type of network and/or protocol.

Input/output devices 2950 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or accessing data by one or more computer systems 2900. Multiple input/output devices 2950 may be present in the computer system 2900, or may be distributed across various nodes of the computer system 2900. In some embodiments, similar input/output devices may be separate from computer system 2900 and may interact with one or more nodes of computer system 2900 through a wired or wireless connection, such as through network interface 2940.

As shown in fig. 12, memory 2920 may contain program instructions 2922 that may be executable by a processor to implement any of the elements or acts described above. In one embodiment, the program instructions may implement the methods described above. In other embodiments, different elements and data may be included. Note that the data may include any of the data or information described above.

Those skilled in the art will appreciate that the computer system 2900 is merely illustrative and is not intended to limit the scope of the embodiments. In particular, the computer systems and devices may include any combination of hardware or software that can perform the indicated functions, including computers, network devices, internet devices, personal digital assistants, wireless telephones, pagers, and the like. The computer system 2900 may also be connected to other devices not shown or otherwise operate as a standalone system. Furthermore, the functionality provided by the illustrated components may be combined in fewer components or distributed in additional components in some embodiments. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided, and/or other additional functionality may be provided.

Those skilled in the art will also recognize that while various items are shown as being stored in memory or on storage during use, these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software components may execute in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article for reading by a suitable drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system 2900 may be transmitted to computer system 2900 via a transmission medium or signal (such as an electrical, electromagnetic, or digital signal) that is transmitted via a communication medium (such as a network and/or wireless link). Various embodiments may further include receiving, transmitting, or storing instructions and/or data implemented in accordance with the foregoing description on a computer-accessible medium. Generally, computer-accessible media may include non-transitory computer-readable storage media or memory media such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, and the like. In some embodiments, the computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals transmitted via a communication medium, such as a network and/or wireless link.

In various embodiments, the methods described herein may be implemented in software, hardware, or a combination thereof. Further, the order of the blocks of the method may be changed, various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and alterations will become apparent to those skilled in the art having the benefit of this disclosure. The various embodiments described herein are intended to be illustrative rather than limiting. Many variations, modifications, additions, and improvements are possible. Thus, multiple instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the claims that follow. Finally, structures and functions presented as discrete components in an exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of the embodiments as defined in the claims that follow.

Claims

1. A system, comprising:

a video decoder configured to perform operations comprising:

Receiving a bitstream comprising (i) encoded video data corresponding to a plurality of frames of source video data encoded by an encoder and (ii) format metadata representing a luminance value focus range of the source video data, the format metadata providing information about the focus range luminance values and a plurality of transfer functions for mapping a full dynamic range of the source video data to the luminance value focus range, wherein different transfer functions of the plurality of transfer functions are used corresponding to the plurality of frames;

obtaining the encoded video data and the format metadata from the bitstream; and

decoding a particular frame of the plurality of frames from the encoded video data, wherein the decoding comprises:

selecting a specific transfer function of the plurality of transfer functions corresponding to the feature frame using the information provided by the format metadata; and

a luminance value of the particular frame is determined using the luminance value focus range indicated by the format metadata and the particular transfer function.

2. The system of claim 1, wherein the plurality of transfer functions comprises (i) a first transfer function configured to clip source video data corresponding to a frame of the plurality of frames according to a focus range of the source video data corresponding to the frame, and (ii) a second transfer function configured to map the clipped source video data corresponding to the frame to a usable number of bits for representing the encoded video data.

3. The system of claim 2, wherein the source video data and the encoded video data are represented using different numbers of bits, and

wherein a first number of bits used to represent the source video data is greater than a second number of bits used to represent the encoded video data.

4. The system of claim 3, wherein the decoder is configured to decode the encoded video data obtained from the bitstream using a third number of bits different from the first number of bits and the second number of bits.

5. The system of claim 2, wherein a first number of frames of the plurality of frames are encoded using one or more portions of a first transfer function and a second number of frames of the plurality of frames are encoded using a second transfer function.

6. The system of claim 5, wherein the format metadata includes information indicative of the one or more portions of a first transfer function.

7. The system of claim 1, wherein a first transfer function of the plurality of transfer functions and a second, different transfer function are used to map a first frame and a second frame of the plurality of frames, respectively.

8. The system of claim 7, wherein the first frame and the second frame are adjacent frames in the source video data.

9. A method performed by a video decoder, the method comprising:

10. The method of claim 9, wherein the plurality of transfer functions comprises (i) a first transfer function configured to clip source video data corresponding to a frame of the plurality of frames according to a focus range of the source video data corresponding to the frame, and (ii) a second transfer function configured to map the clipped source video data corresponding to the frame to a usable number of bits for representing the encoded video data.

11. The method of claim 10, wherein the source video data and the encoded video data are represented using different numbers of bits, and

12. The method of claim 11, wherein the decoder is configured to decode the encoded video data obtained from the bitstream using a third number of bits different from the first number of bits and the second number of bits.

13. The method of claim 10, wherein a first number of frames of the plurality of frames are encoded using one or more portions of a first transfer function and a second number of frames of the plurality of frames are encoded using a second transfer function.

14. The method of claim 13, wherein the format metadata includes information indicative of the one or more portions of a first transfer function.

15. The method of claim 9, wherein a first transfer function of the plurality of transfer functions and a second, different transfer function are used to map a first frame and a second frame of the plurality of frames, respectively.

16. The method of claim 15, wherein the first frame and the second frame are adjacent frames in the source video data.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a video decoder to perform operations comprising:

18. The non-transitory computer-readable medium of claim 17, wherein the plurality of transfer functions comprises (i) a first transfer function configured to clip source video data corresponding to a frame of the plurality of frames according to a focus range of the source video data corresponding to the frame, and (ii) a second transfer function configured to map the clipped source video data corresponding to the frame to a usable number of bits to represent the encoded video data.

19. The non-transitory computer readable medium of claim 18, wherein the source video data and the encoded video data are represented using different numbers of bits,

Wherein a first number of bits for representing the source video data is greater than a second number of bits for representing the encoded video data, and

wherein the decoder is configured to decode the encoded video data obtained from the bitstream using a third number of bits different from the first number of bits and the second number of bits.

20. The non-transitory computer-readable medium of claim 18, wherein a first number of frames of the plurality of frames are encoded using one or more portions of a first transfer function and a second number of frames of the plurality of frames are encoded using a second transfer function, and wherein the format metadata includes information indicating the one or more portions of a first transfer function.

21. The non-transitory computer-readable medium of claim 17, wherein a first transfer function and a different second transfer function of the plurality of transfer functions are used to map a first frame and a second frame of the plurality of frames, respectively, and

wherein the first frame and the second frame are adjacent frames in the source video data.

22. A system, comprising:

a video decoder configured to:

Receiving encoded video data output by a video encoder;

processing the encoded video data to generate decoded C-bit video data representing a focal dynamic range of the input video data and extracted format metadata; and

expanding the decoded C-bit video data to generate D-bit video data covering a full dynamic range of a target device, wherein D is greater than C;

wherein, to extend the decoded C-bit video data, an inverse adaptive transfer function component of the video decoder maps a focus dynamic range of the decoded C-bit video data indicated in the format metadata to a full dynamic range of the target device, wherein the focus dynamic range of the decoded C-bit video data is less than a total dynamic range of input video data encoded by the video encoder.

23. A system, comprising:

a video encoder, comprising:

a processing pipeline configured to process video data according to an encoding format to generate encoded video data, wherein the encoded video data is represented within the processing pipeline with a bit depth of C bits using a code value of an internal transfer function; and

An adaptive transfer function component configured to:

receiving input video data using a code value of a transfer function at a bit depth of N bits, where N is greater than C;

determining a focus dynamic range of the input video data based at least in part on one or more characteristics of the input video data, wherein the focus dynamic range is less than a total dynamic range of the input video data;

mapping the input video data within the focus dynamic range from a bit depth of the input video data to a bit depth of the video encoder to generate C-bit video data; and is also provided with

Outputting the C-bit video data to the processing pipeline for processing; and

a video decoder configured to:

receiving encoded video data output by the processing pipeline of the video encoder;

processing the encoded video data to generate decoded C-bit video data representing a focus dynamic range determined for the input video data and extracted format metadata indicative of one or more parameters used in the mapping; and

the decoded C-bit video data is extended based at least in part on the format metadata to generate D-bit video data that covers a full dynamic range of a target device, where D is greater than C.

24. An apparatus, comprising:

a decoder configured to decode the encoded C-bit video data generated by the encoder to generate decoded C-bit video data representing a range of luminance values extracted from the N-bit video data, wherein N > C, wherein the range of luminance values represented in the encoded C-bit video data is less than a full dynamic range of luminance of the N-bit video data; and

an inverse adaptive transfer function component configured to expand the decoded C-bit video data to generate D-bit video data covering a full dynamic range of a target device, wherein

D>C；

Wherein to extend the decoded C-bit video data, the inverse adaptive transfer function component maps a range of luminance values extracted from the N-bit video data to a full dynamic range of the target device according to a transfer function of the target device.

25. An apparatus, comprising:

an adaptive transfer function component configured to:

receiving input video data using a code value of a transfer function at a bit depth of N bits;

Mapping the input video data within the focus dynamic range from a bit depth N to a bit depth C of the input video data to generate C-bit video data, where N > C; and

an encoder configured to process the C-bit video data according to an encoding format to generate encoded C-bit video data, wherein the video data is represented in the encoder using code values of an internal transfer function.