US20210235093A1 - Backward-compatible video capture and distribution - Google Patents
Backward-compatible video capture and distribution Download PDFInfo
- Publication number
- US20210235093A1 US20210235093A1 US17/233,099 US202117233099A US2021235093A1 US 20210235093 A1 US20210235093 A1 US 20210235093A1 US 202117233099 A US202117233099 A US 202117233099A US 2021235093 A1 US2021235093 A1 US 2021235093A1
- Authority
- US
- United States
- Prior art keywords
- video data
- dynamic range
- display
- pipeline
- mapping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013507 mapping Methods 0.000 claims abstract description 186
- 238000000034 method Methods 0.000 claims abstract description 157
- 230000007613 environmental effect Effects 0.000 claims description 7
- 238000003384 imaging method Methods 0.000 abstract description 40
- 238000012545 processing Methods 0.000 abstract description 34
- 239000003086 colorant Substances 0.000 abstract description 12
- 230000002093 peripheral effect Effects 0.000 description 40
- 238000004891 communication Methods 0.000 description 32
- 230000033001 locomotion Effects 0.000 description 21
- 230000008569 process Effects 0.000 description 20
- 230000006870 function Effects 0.000 description 19
- 238000005516 engineering process Methods 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 16
- 238000007726 management method Methods 0.000 description 13
- 210000003811 finger Anatomy 0.000 description 10
- 238000012546 transfer Methods 0.000 description 10
- 238000006243 chemical reaction Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 239000004744 fabric Substances 0.000 description 6
- 230000003044 adaptive effect Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 241000023320 Luma <angiosperm> Species 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 4
- 229920001690 polydopamine Polymers 0.000 description 4
- 238000003672 processing method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 239000004065 semiconductor Substances 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000881 depressing effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 229910044991 metal oxide Inorganic materials 0.000 description 2
- 150000004706 metal oxides Chemical class 0.000 description 2
- 229920000642 polymer Polymers 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 230000009849 deactivation Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000010454 slate Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 229920000638 styrene acrylonitrile Polymers 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 210000003813 thumb Anatomy 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N19/98—Adaptive-dynamic-range coding [ADRC]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G06T5/009—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
- G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
- G09G3/2007—Display of intermediate tones
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/003—Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto
- G09G5/005—Adapting incoming signals to the display format of the display terminal
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/10—Intensity circuits
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/46—Colour picture communication systems
- H04N1/56—Processing of colour picture signals
- H04N1/60—Colour correction or control
- H04N1/6058—Reduction of colour to a range of reproducible colours, e.g. to ink- reproducible colour gamut
- H04N1/6063—Reduction of colour to a range of reproducible colours, e.g. to ink- reproducible colour gamut dependent on the contents of the image to be reproduced
- H04N1/6066—Reduction of colour to a range of reproducible colours, e.g. to ink- reproducible colour gamut dependent on the contents of the image to be reproduced dependent on the gamut of the image to be reproduced
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/46—Colour picture communication systems
- H04N1/64—Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/124—Quantisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/177—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a group of pictures [GOP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/188—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
- H04N19/33—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the spatial domain
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/517—Processing of motion vectors by encoding
- H04N19/52—Processing of motion vectors by encoding by predictive encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42202—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4318—Generation of visual interfaces for content selection or interaction; Content or additional data rendering by altering the content in the rendering process, e.g. blanking, blurring or masking an image region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/85—Camera processing pipelines; Components thereof for processing colour signals for matrixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/20—Circuitry for controlling amplitude response
-
- H04N5/2355—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
- H04N5/50—Tuning indicators; Automatic tuning control
- H04N5/505—Invisible or silent tuning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
- H04N9/641—Multi-purpose receivers, e.g. for auxiliary information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
- H04N9/67—Circuits for processing colour signals for matrixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20208—High dynamic range [HDR] image processing
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/02—Improving the quality of display appearance
- G09G2320/0261—Improving the quality of display appearance in the context of movement of objects on the screen or movement of the observer relative to the screen
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/02—Improving the quality of display appearance
- G09G2320/0271—Adjustment of the gradation levels within the range of the gradation scale, e.g. by redistribution or clipping
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0613—The adjustment depending on the type of the information to be displayed
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0613—The adjustment depending on the type of the information to be displayed
- G09G2320/062—Adjustment of illumination source parameters
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0626—Adjustment of display parameters for control of overall brightness
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/066—Adjustment of display parameters for control of contrast
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0666—Adjustment of display parameters for control of colour parameters, e.g. colour temperature
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0673—Adjustment of display parameters for control of gamma adjustment, e.g. selecting another gamma curve
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0686—Adjustment of display parameters with two or more screen areas displaying information with different brightness or colours
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/06—Adjustment of display parameters
- G09G2320/0693—Calibration of display systems
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/08—Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/10—Special adaptations of display systems for operation with variable images
- G09G2320/103—Detection of image changes, e.g. determination of an index representative of the image change
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/02—Handling of images in compressed format, e.g. JPEG, MPEG
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/14—Detecting light within display terminals, e.g. using a single or a plurality of photosensors
- G09G2360/144—Detecting light within display terminals, e.g. using a single or a plurality of photosensors the light being ambient light
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/04—Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
- G09G2370/042—Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller for monitor identification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4223—Cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/485—End-user interface for client configuration
- H04N21/4854—End-user interface for client configuration for modifying image parameters, e.g. image brightness, contrast
Definitions
- This disclosure relates generally to digital video or image processing and display.
- a device may include an apparatus (e.g., an integrated circuit (IC), such as a system-on-a-chip (SOC), or a subsystem of an IC), that may receive and process digital video input from one or more sources and output the processed video frames according to one or more video processing methods.
- IC integrated circuit
- SOC system-on-a-chip
- a software program may be implemented on a device that may receive and process digital video input from one or more sources according to one or more video processing methods and output the processed video frames to one or more destinations.
- a video encoder may be implemented as an apparatus, or alternatively as a software program, in which digital video input is encoded or converted into another format according to a video encoding method, for example a compressed video format such as H.264/Advanced Video Coding (AVC) format, or H.265 High Efficiency Video Coding (HEVC) format.
- a video decoder may be implemented as an apparatus, or alternatively as a software program, in which video in a compressed video format such as AVC or HEVC is received and decoded or converted into another (decompressed) format according to a video decoding method, for example a display format used by a display device.
- the H.264/AVC standard is published by ITU-T in a document titled “ITU-T Recommendation H.264: Advanced video coding for generic audiovisual services”.
- the H.265/HEVC standard is published by ITU-T in a document titled “ITU-T Recommendation H.265: High Efficiency Video Coding”.
- an apparatus or software program may implement both a video encoder component and a video decoder component; such an apparatus or program is commonly referred to as a codec.
- a codec may encode/decode both visual/image data and audio/sound data in a video stream.
- dynamic range is the ratio between the largest and smallest possible values of a changeable quantity, such as in signals like sound and light.
- digital images e.g., video or still images
- SDR standard dynamic range
- SCG standard color gamut
- HDR high dynamic range imaging
- HDR imaging as well as wide color gamut (WCG) imaging.
- WCG wide color gamut
- These devices may be referred to as HDR-enabled devices or simply HDR devices.
- many video capture devices and display devices may not directly support HDR imaging and/or WCG imaging.
- a sensor pipeline may generate lower or standard dynamic range (SDR) image data from HDR image data captured by an image sensor, for example using a tone mapping technique.
- SDR standard dynamic range
- Information used to generate the SDR image data may be recorded as metadata with the generated SDR image data.
- the SDR image data may be provided to a display pipeline to be rendered for display on a target display device. If the target display device does not support HDR imaging, the SDR image data may be directly rendered by the display pipeline for display. If the target display does support HDR imaging, then an inverse mapping technique may be applied to the SDR image data by the display pipeline according to the metadata to render HDR image data for display.
- a local tone mapping (LTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data.
- GTM global tone mapping
- LTM local tone mapping
- an image or frame is divided into multiple regions, with a tone curve dynamically determined for each region.
- the regional or local tone curves can be parameterized and stored in the metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered in an inverse tone mapping technique.
- the local tone mapping metadata can be compressed spatially (e.g., within an image) and/or temporally (e.g., across two or more frames).
- the inverse tone mapping technique performed by the display pipeline may be tuned to match the dynamic range of the target display.
- the inverse tone mapping technique of the display pipeline may be an ambient adaptive process in which information about ambient light or other environmental conditions for the display may be used to adjust the inverse tone mapping process, for example by modifying or weighting one or more of the tone curves used in local tone mapping.
- information used in performing color gamut mapping on the sensor pipeline may be recorded in the metadata and used by the display pipeline in image reconstruction to recover a wider color gamut for displays that support WCG imaging.
- the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to the lower display color gamut supported by a respective display according to the metadata.
- FIG. 1 illustrates operations of a video pipeline.
- FIG. 2 illustrates an example video pipeline that may provide both standard dynamic range (SDR) and high dynamic range (HDR) video from source HDR content, according to some embodiments.
- SDR standard dynamic range
- HDR high dynamic range
- FIG. 3 illustrates an example video pipeline that provides SDR video from source HDR content, according to some embodiments.
- FIG. 4 illustrates local tone mapping (LTM), according to some embodiments.
- FIG. 5 is a high-level flowchart of a method that may be implemented in a sensor pipeline, according to some embodiments.
- FIG. 6 is a high-level flowchart of a method that may be implemented in a display pipeline, according to some embodiments.
- FIG. 7 is a high-level flowchart of a method that may be implemented in a video pipeline that supports HDR displays and that is backwards compatible with SDR displays, according to some embodiments.
- FIG. 8 illustrates an example video playback system in which an encoder generates output data in an intermediate color gamut and in which mapping to a display color gamut is performed on the decoder/display side using metadata generated by the encoder, according to some embodiments.
- FIG. 9 is a flowchart of a video playback method in which an encoder generates output data in an intermediate color gamut and mapping to a display color gamut is performed by a decoder according to metadata generated by the encoder, according to some embodiments.
- FIG. 10 is a block diagram of one embodiment of a system on a chip (SOC) that may be configured to implement aspects of the systems and methods described herein.
- SOC system on a chip
- FIG. 11 is a block diagram of one embodiment of a system that may include one or more SOCs.
- FIG. 12 illustrates an example computer system that may be configured to implement aspects of the systems and methods described herein, according to some embodiments.
- FIG. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments.
- FIG. 14 depicts a portable multifunction device in accordance with some embodiments.
- circuits, or other components may be described as “configured to” perform a task or tasks.
- “configured to” is a broad recitation of structure generally meaning “having circuitry that” performs the task or tasks during operation.
- the unit/circuit/component can be configured to perform the task even when the unit/circuit/component is not currently on.
- the circuitry that forms the structure corresponding to “configured to” may include hardware circuits.
- various units/circuits/components may be described as performing a task or tasks, for convenience in the description.
- Embodiments of video processing techniques and pipelines are described that support capture, distribution, and display of high dynamic range (HDR) image data to both HDR-enabled display devices and display devices that do not support HDR imaging.
- Embodiments of a video processing pipeline may include a sensor pipeline portion and a display pipeline portion.
- the sensor pipeline and display pipeline may be in the same device, or may be in different devices.
- the sensor pipeline may generate lower or standard dynamic range (SDR) image data from HDR image data captured by an image sensor or photosensor of a camera associated with the sensor pipeline, for example using a tone mapping technique.
- Information used to generate the SDR image data (e.g., tone mapping information such as parameterized tone curves) may be recorded as metadata with the generated SDR image data.
- the SDR image data may be provided to a display pipeline to be rendered for display on a target display device. If the target display device does not support HDR imaging, the SDR image data may be directly rendered by the display pipeline for display. If the target display does support HDR imaging, then an inverse mapping technique may be applied to the SDR image data by the display pipeline according to the metadata to render HDR image data for display.
- a global tone mapping (GTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data.
- GTM global tone mapping
- a global tone curve may be specified or determined for one or more video frames and used in converting the HDR image data to SDR.
- the global tone curve can be parameterized and stored in metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered in an inverse tone mapping technique when generating the HDR image data from the input SDR image data.
- a local tone mapping (LTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data.
- LTM local tone mapping
- an image or frame is divided into multiple regions, with a tone curve dynamically determined for each region.
- the regional or local tone curves can be parameterized and stored in the metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered by an inverse tone mapping technique when generating HDR image data from the input SDR image data.
- the local tone mapping metadata can be compressed spatially (e.g., within an image) and/or temporally (e.g., across two or more frames).
- the inverse tone mapping technique performed by the display pipeline may be tuned to match the dynamic range of the target display.
- the dynamic range captured by the sensor may cover a wider range than the dynamic range supported by the display, even if the display is HDR-compatible.
- the inverse tone mapping technique of the display pipeline may modify the tone map metadata according to the dynamic range of the display, or may otherwise modify the inverse tone mapping method applied to the lower dynamic range image data, to generate HDR image data that better matches the dynamic range of the display.
- the inverse tone mapping technique of the display pipeline may be an ambient adaptive process in which information about ambient light or other environmental conditions for the display may be obtained, e.g. using light, motion, or other sensors on a device, and based on that information the inverse tone mapping technique may be adjusted, for example by modifying or weighting one or more parameters of the tone curves used in local tone mapping when performing the inverse tone mapping. For example, if ambient light is bright, the inverse tone mapping technique may adjust the tone curves to generate a brighter image with more contrast.
- Devices such as image sensors and displays may support wide color gamut (WCG) imaging in addition to HDR imaging.
- WCG wide color gamut
- the sensor pipeline may generate lower color gamut image data from wider color gamut image data captured by an image sensor of a camera associated with the sensor pipeline using a color gamut mapping technique that may clip colors from the wider color gamut range of a color space (e.g., an RGB color space) that aren't available in the lower color gamut of another color space (e.g., a YUV color space).
- a color space e.g., an RGB color space
- another color space e.g., a YUV color space
- information used in performing the color gamut mapping may be recorded in the metadata and used by the display pipeline in image reconstruction to recover a wider color gamut for displays that support WCG imaging.
- the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to the lower display color gamut supported by a respective display according to the metadata.
- Embodiments of video pipelines including sensor pipelines and display pipelines as described herein may, for example, be implemented in devices or systems that include one or more image capture devices and/or one or more display devices.
- An image capture device may be any device that includes an optical sensor or photosensor that is capable of capturing digital images or video.
- Image capture devices may include, but are not limited to, video cameras and still image cameras, as well as image capture devices that can capture both video and single images.
- Image capture devices may be stand-alone devices or may be cameras that are integrated into other devices including but not limited to smartphones, cellphones, PDAs, tablet or pad devices, multifunction devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and so on.
- image capture devices may include small form factor cameras suitable for use in small devices such as cellphones, PDAs, and tablet devices.
- Displays or display devices may include display screens or panels that are integrated into other devices including but not limited to smartphones, cellphones, PDAs, tablet or pad devices, multifunction devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and so on.
- Display devices may also include video monitors, projectors, or in general any device that can display or project digital images and/or digital video.
- the displays or display devices may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used.
- FIGS. 10 through 14 show non-limiting examples of devices in which embodiments may be implemented.
- a device or system that includes an image capture device and/or a display device may include hardware and/or software that implements at least some of the functionality for processing video data as described herein.
- a portion of the functionality as described herein may be implemented on one device, while another portion may be implemented on another device.
- a device that includes an image capture device may implement a sensor pipeline that processes and compresses (i.e., encodes) images or video captured via a photosensor, while another device that includes a display panel or screen may implement a display pipeline that receives and processes the compressed images (i.e., decodes) for display to the panel or screen.
- At least some of the functionality as described herein may be implemented by one or more components or modules of a system on a chip (SOC) that may be used in devices including but not limited to multifunction devices, smartphones, pad or tablet devices, and other portable computing devices such as laptop, notebook, and netbook computers.
- SOC system on a chip
- FIG. 10 illustrates an example SOC
- FIG. 11 illustrates an example device implementing an SOC.
- FIG. 12 illustrates an example computer system that may implement the methods and apparatus described herein.
- FIGS. 13 and 14 illustrate example multifunction devices that may implement the methods and apparatus described herein.
- HDR high dynamic range
- an HDR image is an image that is produced using an HDR imaging technique that produces a wider range of luminosity than is obtained using standard digital imaging techniques.
- an HDR image may include more bits per channel (e.g., 10, 12, 14, or more bits per luminance (luma) and chrominance (chroma) channel), or more bits for luminosity (the luma channel), than are used in conventional image processing (typically, 8 bits per channel, e.g. 8 bits for color/chroma and for luma).
- tone mapping is a technique that maps one set of tonal image values (e.g., from HDR image data) to another (e.g., to SDR image data). Tone mapping may be used, for example, to approximate the appearance of HDR images in a medium that has a more limited dynamic range (e.g., SDR). Tone mapping may generally be applied to luma image data.
- color gamut refers to a particular subset of colors, for example the subset of colors which can be accurately represented in a given circumstance, such as within a given color space (e.g., an RGB color space) or by a display device.
- Color gamut may also refer to the complete set of colors found within an image. At least some image processing operations performed on an image may alter the image's color gamut, and in some cases some of the colors in the original image may be lost. In other words, an image's color gamut may be narrowed or clipped by an image processing technique, for example a tone mapping technique, thus reducing the number of colors that can be represented by or reproduced from the image data.
- a gamut mapping technique (which may also be referred to as color or chroma mapping) may be applied to image data (generally to chroma image data), and may in some cases narrow or clip an image's color gamut, or alternatively may be used to correct or adjust the color gamut or range of an image during or after tone mapping.
- Embodiments are generally described as using YUV Rec. 709, otherwise known as ITU-R Recommendation BT. 709 , as a base SDR layer for transmitting or transferring digital image data between devices, e.g. between a camera sensor pipeline and a display pipeline.
- YUV Rec. 709 defines a color space and format that may be used in color image or video pipelines. However, note that other color spaces and formats may be used in embodiments. As just one example, Digital Cinema Initiatives (DCI) P3 may be used.
- DCI Digital Cinema Initiatives
- Embodiments are generally described as processing video frames or sequences. However, embodiments may be applied to process single or still images instead of or in addition to video frames or sequences, as well as other digital images. Thus, when “video”, “video frame”, “frame”, or the like is used herein, it is to be understood that the terms may refer to captured digital images in general.
- FIG. 1 graphically illustrates operations of a video pipeline at a high level.
- a camera sensor may capture images or frames, shown as camera sensor RGB data 120 .
- the images may be captured at the sensor at a higher dynamic range (HDR image data), while a display pipeline 160 is configured to process image data at a lower dynamic range (SDR image data) for output to a display.
- the images may be captured by the sensor according to the RGB color model or according to other color models.
- An image signal processor (ISP) 122 may process the captured camera sensor RGB data 120 to generate linear RGB image data 124 as output (e.g., image data in the linear RGB color space).
- ISP image signal processor
- the linear RGB image data 124 may still be at the higher dynamic range of the sensor.
- a tone mapping 126 technique e.g., a local and/or global tone mapping technique
- SDR lower dynamic range
- a tone mapping 126 technique in addition to reducing the dynamic range of the image data, application of a tone mapping 126 technique may result in the clipping of color values in the image data, thus narrowing the color gamut of the image(s).
- a gamut mapping technique may also be applied to the image data, and may result in clipping of the color gamut of the image data.
- the YUV Rec. 709 image data 130 A may generally be compressed 132 to generate YUV Rec. 709 compressed data 150 for transfer or transmittal to a display pipeline 160 .
- the YUV Rec. 709 compressed data 150 generated by a sensor pipeline 110 may be decompressed 170 to generate YUV Rec. 709 image data 130 B at SDR.
- a color conversion 174 technique may then be applied to the YUV Rec. 709 image data 130 B to generate sRGB image data 176 , also at SDR.
- the image data is converted from the YUV Rec. 709 color space to the sRGB color space according to the color conversion 174 technique.
- the sRGB image data 176 is then passed to display management 178 hardware and/or software that renders SDR RGB image data 180 for display on a target panel or screen.
- HDR-enabled devices may be referred to as HDR-enabled devices or simply HDR devices.
- SDR image data 150 is passed from a sensor pipeline 110 to a display pipeline 160 for a display that supports HDR or for a display that does not support HDR.
- SDR image data would be passed to and rendered on an HDR-enabled display according to the lower, standard dynamic range even if the sensor is capable of capturing HDR image data.
- the wide color gamut of the image data as original captured may be narrowed or clipped by the sensor pipeline 110 processing. Thus, much of the tonal range, color, and luminance that is actually captured by the sensor may be lost.
- Embodiments of backward-compatible methods and apparatus that provide both standard dynamic range (SDR) and higher dynamic range (HDR) versions of video are described.
- Embodiments may implement methods and apparatus to support processing and distribution of HDR image data to both HDR-enabled display devices and devices that do not support HDR imaging.
- Embodiments may implement a video pipeline that includes a sensor or encoding pipeline portion and a display or decoding pipeline portion.
- the video pipeline may be implemented in a single device, for example a device that includes both an image capture device and a display panel.
- the sensor pipeline portion and the display pipeline portion may also be implemented in different devices, with compressed images/video transmitted from one device to the other, for example over a network, or otherwise transferred from one device to the other.
- the sensor pipeline portion may be implemented in a video camera, and the display pipeline portion may be implemented in a display device.
- the encoding pipeline and the display pipeline may be implemented at least in part on system on a chip (SOC).
- one or more encoding pipelines may be implemented on a device or system; the encoding pipeline(s) may be configured to encode input video to generate standard dynamic range (SDR) video, stream the SDR video to one or more target devices each implementing at least one decoding pipeline, and generate and provide mapping metadata used in encoding the video to the target device(s).
- the target devices may include both HDR-enabled devices and SDR devices that do not support HDR imaging.
- a decoding pipeline for an HDR-enabled device may receive and use the mapping metadata generated by the encoding pipeline to recover at least part of the HDR of the input video that was lost in the encoding process when decoding the SDR video.
- a decoding pipeline for a SDR device may receive and process the SDR video without applying the mapping metadata.
- encoding pipelines and decoding pipelines may communicate information that allows an encoding pipeline to generate and transmit mapping metadata to HDR-enabled devices, while not transmitting mapping metadata to SDR devices.
- the mapping metadata may also include information that may be used in a decoding pipeline to recover at least part of the wide color gamut (WCG) of the input video that was lost in a color gamut mapping technique that converts the WCG input video data to a narrower color gamut used in the color space of the encoded SDR video.
- WCG wide color gamut
- FIG. 5 is a high-level flowchart of a method that may be implemented in a sensor pipeline, according to some embodiments.
- a camera/sensor may capture image(s) at a high dynamic range.
- lower dynamic range image data may be generated from the captured image data, for example using a mapping technique such as a tone mapping technique.
- information used to generate the lower dynamic range image data may be recorded as metadata with the image data.
- the image data and/or metadata may be compressed.
- FIG. 6 is a high-level flowchart of a method that may be implemented in a display pipeline, according to some embodiments.
- lower dynamic range image(s) may be obtained, for example from a sensor pipeline as described in FIG. 5 and illustrated in FIGS. 2 and 3 .
- metadata for the image(s) may also be obtained that includes information on how the image data was mapped from a higher dynamic range to a lower dynamic range.
- an inverse mapping technique may be applied to the image(s) according to the metadata to recover higher dynamic range image data as captured by the sensor.
- FIG. 7 is a high-level flowchart of a method that may be implemented in a video pipeline that supports HDR displays and that is backwards compatible with SDR displays, according to some embodiments.
- a camera/sensor may capture image(s) at a high dynamic range.
- lower dynamic range image data may be generated from the captured image data, for example using a mapping technique such as a tone mapping technique.
- information used to generate the lower dynamic range image data may be recorded as metadata with the image data.
- the image data and/or metadata may be compressed.
- the image data may be provided to a display pipeline. If the target display supports HDR imaging, then the lower dynamic range image(s) may be provided to the display pipeline, as indicated at 1310 .
- metadata for the image(s) may also be provided that includes information on how the image data was mapped from a higher dynamic range to a lower dynamic range, as indicated at 1312 .
- an inverse mapping technique may be applied to the image(s) according to the metadata to render higher dynamic range image data as captured by the sensor.
- the lower dynamic range image(s) may be provided to the display pipeline, as indicated at 1320 .
- the metadata may not be provided.
- the display pipeline may then render and display SDR image data from the provided image data, as indicated at 1322 . Inverse mapping to recover higher dynamic range image data is not performed, as the display does not support HDR imaging.
- information used in performing color gamut mapping may be recorded in the metadata by the sensor pipeline and used by the display pipeline to recover a wider color gamut (WCG) for displays that support WCG imaging.
- the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to a lower display color gamut supported by a respective display according to the metadata.
- FIGS. 5 through 7 Elements of the methods of FIGS. 5 through 7 are further described below in reference to FIGS. 2 through 4 .
- FIG. 2 illustrates an example video pipeline 200 that may process HDR image data captured by a sensor and provide both standard dynamic range (SDR) image data to non-HDR-enabled display devices and high dynamic range (HDR) image data to HDR-enabled display devices, according to some embodiments.
- compressed SDR image data 250 may be passed from a sensor pipeline 210 to a display pipeline 260 for a display that supports HDR or for a display that does not support HDR.
- metadata 232 A describing the processing performed by the sensor pipeline 210 may also be passed to the display pipeline 260 .
- the metadata 232 A may allow much if not all of the tonal range, color, and/or luminance that was originally captured by the sensor but that was removed by the HDR-to-SDR conversion performed by the sensor pipeline 210 to be recovered and rendered by the display pipeline 260 and viewed on a target HDR-enabled display device.
- the SDR image data 250 can be processed as normal by the display pipeline 260 , for example according to an SDR processing technique as shown for display pipeline 160 in FIG. 1 .
- a camera sensor may capture images or frames, shown as HDR camera sensor RGB 220 data.
- the images may be captured by the sensor according to the RGB color model or according to other color models.
- An image signal processor (ISP) 222 may process the captured camera sensor RGB data 220 to generate HDR linear RGB image data 224 as output.
- a tone mapping module or component may apply a tone mapping 226 technique (e.g., a local (LTM) and/or global (GTM) tone mapping technique) to the linear RGB image data 224 to convert the HDR linear RGB image data 224 to SDR YUV Rec. 709 image data 230 A.
- FIG. 4 illustrates a local tone mapping technique that may be used in some embodiments.
- a color gamut mapping technique may also be applied to the image data in the sensor pipeline 210 (e.g., by ISP 222 ) to map the colors from one color space (e.g., RGB) to another (e.g., YUV), which may result in clipping of the colors in the processed image data 230 A.
- one color space e.g., RGB
- another e.g., YUV
- the sensor pipeline 210 records information used to perform the tone mapping 226 and/or color gamut mapping as metadata 232 A and stores, embeds, or otherwise associates the metadata with the SDR YUV Rec. 709 image data 230 A.
- the metadata 232 A may be compressed temporally and/or spatially.
- the YUV Rec. 709 image data 230 A may be compressed 234 .
- the compressed YUV Rec. 709 image data 230 A and metadata 232 A may be output as compressed YUV Rec. 709+ metadata 250 , which may be stored, transferred or transmitted to one or more display pipelines 260 .
- the compressed YUV Rec. 709+ metadata 250 may be decompressed 270 to generate metadata 232 B and decompressed SDR YUV Rec. 709 image data 230 B.
- the metadata 232 B may then be used by an inverse tone mapping module or component of the display pipeline 260 in performing an inverse global and/or local tone mapping (TM) technique 276 on the SDR YUV Rec. 709 image data 230 B to generate HDR image data 278 .
- TM local tone mapping
- the inverse tone mapping technique 276 recovers or reconstructs at least a portion of the tonal range and luminance that was lost by the tone mapping technique 226 applied in the sensor pipeline 210 .
- a color conversion technique may also be applied to the YUV data to convert to a different color space, for example to an xRGB (extended RGB) color space.
- the metadata 232 B may also include information used to perform color gamut mapping, and this information may be used to recover or reconstruct a portion or all of the color gamut range that was clipped in the color gamut mapping operation on the sensor pipeline 210 , thus providing a wider color gamut for the HDR image data 278 for display.
- the xRGB image data 278 may then be passed to display management 280 hardware and/or software of an HDR-enabled device that renders HDR RGB image data 282 for display on a target panel or screen.
- SDR, YUV Rec. 709 image data 230 may still be passed to display pipelines for displays that do not support HDR, and the SDR, YUV Rec. 709 image data 230 B may be decompressed and processed without performing the inverse tone mapping operation (e.g., as illustrated for display pipeline 160 in FIG. 1 ) so that the image data can be displayed as normal SDR data on non-SDR-enabled displays.
- the sensor pipeline 210 as illustrated in FIG. 2 works to provide HDR imaging for HDR-enabled displays, while also being backwards compatible with displays that do not support HDR imaging.
- handshaking may be used between the sensor pipeline 210 and the display pipeline 260 , or between the display pipeline 260 and another source of SDR, YUV Rec. 709 image data 250 , so that a target display's dynamic range can be determined. If the target display does not support HDR, the compressed metadata 232 A may not be transferred or transmitted to the display pipeline 260 with the compressed YUV Rec. 709 image data 230 A.
- the sensor pipeline 210 and the display pipeline 260 may be components of a same device or of a same system on a chip (SOC), and the compressed SDR, YUV Rec. 709 image data 250 may be transferred, for example, over a communications bus or other connection local to the device.
- the sensor pipeline 210 and the display pipeline 260 may instead be components of different devices, and the compressed SDR, YUV Rec. 709 image data 250 may be transferred or transmitted via other media, for example via a wired and/or wireless network, or over a connect (e.g., a USB connection) coupling the devices, or via a removable storage medium such as a thumb drive, CD, or DVD.
- a connect e.g., a USB connection
- a local tone mapping (LTM) technique may be used in the sensor pipeline 210 in which an image is divided into multiple regions, with each region having its own tone curve.
- the regional tone curves can be parameterized and stored in the metadata 232 A that is passed to the display pipeline 260 so that tone/luminance can be recovered in an inverse LTM technique 276 when generating the HDR image data 278 from the input SDR image data 230 B and the local tone mapping information indicated in the metadata 232 B.
- a LTM technique can be performed in RGB color space or in YUV color space; more generally, LTM as described herein may be performed in any color space.
- FIG. 4 graphically illustrates a local tone mapping (LTM) technique, according to some embodiments.
- LTM local tone mapping
- Embodiments of the LTM technique may, for example, be used in embodiments of the video pipelines as illustrated in FIGS. 1 through 3 .
- embodiments of the LTM technique as described herein may be applied in other image or video processing techniques in which tone mapping is used.
- an image 400 may be divided into multiple regions 402 , with a local tone curve 404 indicated or determined for each region 402 of the image 400 . While FIG. 4 shows the image 400 evenly subdivided into square or rectangular regions 402 , in some embodiments regions 402 of different shapes or of varying sizes and/or shapes may be used.
- the tone curves 404 for the regions 402 may be similar, but may differ at least slightly between regions 402 according to varying characteristics, for example, tone/brightness, of the regions 402 .
- tone/brightness For example, in dark regions such as region 402 A, the tone curve 404 A may boost the dark area to provide a broader range.
- region 402 B For bright regions such as region 402 B, the tone curve 404 B may not boost as much to preserve highlights.
- the tone curves 404 determined for the regions 402 of an image 400 can be parameterized and stored, for example as metadata 232 as illustrated in FIG. 2 , and passed or otherwise provided to a display pipeline 260 so that a wider range of tone or luminance can be recovered in an inverse LTM technique 276 to generate HDR image data 278 from the input SDR image data 230 B on the display pipeline 260 .
- additional information may be saved as metadata 232 A for use in the inverse LTM technique 276 of the display pipeline 260 .
- the LTM technique 226 may clip at least some image data values, and the metadata 232 A may include additional information that may be used in at least partially recovering the clipped values.
- information used in performing color gamut mapping on the sensor pipeline 210 may also be stored as metadata 232 A and provided to the display pipeline 260 .
- the color gamut mapping information may be used in image reconstruction on the display pipeline 260 to recover a wider color gamut for display.
- the tone mapping metadata 232 A can be compressed spatially (i.e., within an image 400 or frame) and/or temporally (e.g., across two or more images 400 or frames).
- the local tone curves 402 may not change much between adjacent or nearby regions 402 , and thus the tone mapping metadata 232 A may be compressed within the image 400 (spatially).
- the tone curves may not change much between the same regions 402 in two or more adjacent images 400 or frames, and thus the tone curve information in metadata 232 may be compressed temporally (across two or more images 400 or frames).
- the inverse tone mapping technique 276 applied by a display pipeline 260 may be tuned to match the dynamic range of a target display.
- the dynamic range captured by the sensor may, for example, cover a wider range than the dynamic range supported by the target display, even if the display is HDR-compatible.
- the inverse tone mapping technique 276 of the display pipeline 260 may modify the tone mapping metadata 232 B received with the SDR image data 230 B according to the dynamic range of a target display, or may otherwise modify the inverse tone mapping technique 276 applied to the SDR image data 230 B, to generate HDR image data 278 that matches the dynamic range of the target display.
- the inverse tone mapping technique 276 of the display pipeline may be an ambient adaptive process in which information about ambient light, distance to one or more viewers, or other ambient or environmental conditions at the target display may be obtained, e.g. using one or more sensors. Based on the obtained environmental information, the inverse tone mapping technique 276 may be dynamically adjusted, for example by modifying or weighting one or more of the tone curves 404 as indicated in the metadata 232 B for one or more regions 402 of the video frames being processed in the display pipeline 260 .
- the inverse tone mapping technique 276 may adjust one or more of the tone curves 404 as indicated in the metadata 232 B to generate a brighter image 400 (or brighter regions 402 within an image 400 ) and/or an image 400 or region(s) 402 with more contrast. If ambient light is detected to be dim at the display (e.g., below a specified threshold), the inverse tone mapping technique 276 may adjust one or more of the tone curves 404 as indicated in the metadata 232 B to tone down or darken the image 400 or region(s) 402 within the image 400 .
- tone mapping may involve applying a transfer function (e.g., an electro-optical transfer function (EOTF)) to input video data values represented at a higher dynamic range according to a first color space (e.g., an RGB color space) to generate output video data represented at a lower dynamic range according to a second color space (e.g., a YUV color space).
- EOTF electro-optical transfer function
- the transfer function may correspond to the tone curve (which may be referred to as a transfer curve), and may be represented by and stored in metadata as one or more transfer function parameter values.
- the inverse tone mapping technique may involve applying an inverse of the transfer function as represented in the metadata to the decompressed video data obtained from the sensor/encoding pipeline.
- a wider color gamut as captured by a sensor may also be preserved on the sensor side 210 and at least partially recovered on the display side 260 using metadata 232 .
- the sensor pipeline 210 may generate lower color gamut image data (e.g., YUV Rec.
- image data 230 A from wider color gamut image data (e.g., camera sensor RGB image data 220 ) captured by an image sensor of a camera associated with the sensor pipeline 210 using a color gamut mapping technique that may clip colors from the wider color gamut range of a color space (e.g., RGB color space) that aren't available in the lower color gamut of another color space (e.g., YUV color space).
- a color gamut mapping technique may clip colors from the wider color gamut range of a color space (e.g., RGB color space) that aren't available in the lower color gamut of another color space (e.g., YUV color space).
- information used in or resulting from performing the color gamut mapping may be recorded in the metadata 232 and used by the display pipeline 260 in image reconstruction to recover a wider color gamut for displays that support WCG imaging by at least partially restoring the clipped colors.
- FIG. 3 illustrates an example video pipeline providing standard dynamic range (SDR) video from source high dynamic range (HDR) content, according to some embodiments.
- a video pipeline 300 may generate and output video data in different formats by encoding camera sensor RGB data 320 to generate YUV, 10+ bit depth image data 330 A for output and transcoding the YUV, 10+ bit depth image data 330 A to generate sRGB, Rec. 709 image data 382 for output.
- a camera sensor may capture images or frames, shown as HDR camera sensor RGB 320 data.
- An image signal processor (ISP) 322 may process the captured camera sensor RGB data 320 to generate HDR linear RGB image data 324 as output.
- a tone mapping technique 326 e.g., a local (LTM) and/or global (GTM) tone mapping technique as described herein
- LTM local
- GTM global
- SDR standard dynamic range
- color gamut mapping may be applied to the image data to map the colors from one color space (e.g., RGB) to another (e.g., YUV).
- the YUV, 10+ bit depth image data 330 A may be compressed 334 .
- the video pipeline 300 may record information used to perform the tone mapping 326 and/or color gamut mapping as metadata 332 A.
- the compressed YUV, 10+ bit depth image data 330 A and metadata 332 A may be output, for example via a network 350 to one or more target devices.
- the YUV, 10+ bit depth image data 330 A may be decompressed and processed, and the metadata 332 may be used in an inverse tone mapping technique and/or gamut mapping technique to recover at least some of the dynamic range and/or color for HDR-enabled displays.
- the compressed YUV, 10+ bit depth image data 330 A and metadata 332 A may instead or also be sent on a transcoding path 340 to be transcoded into YUV Rect. 709 image data 382 for output, for example via a network 350 to one or more target devices.
- the compressed YUV, 10+ bit depth image data 330 A may be decompressed 370 to generate compressed YUV, 10+ bit depth image data 330 B.
- the metadata 332 B may be used in performing an inverse global and/or local tone mapping and color conversion operation 376 on the image data 330 B to generate HDR image data 378 .
- the inverse operation 376 may recover or reconstruct at least a portion of the luminance and/or color in image data 378 according to the metadata 332 B.
- Another global and/or local tone mapping operation 380 may then be applied to the HDR image data 378 to generate sRGB, Rec. 709 image data 382 for output, for example via a network 350 to one or more target devices.
- Metadata 332 B may not be sent on the transcoding path 340 .
- the transcoding path 340 may apply a tone mapping and color conversion technique directly to the decompressed image data 330 B to directly generate the Rec. 709 image data 382 from image data 330 B.
- Digital Cinema Initiatives (DCI) P3 may be used as a base layer rather than Rec. 709.
- the tone mapping 326 can be performed in RGB color space or in YUV color space, or in other color spaces if available.
- Example embodiments of video display and playback methods and apparatus for supporting high dynamic range (HDR) and wide color gamut (WCG) video processing, recovery, and display are described.
- embodiments of methods and apparatus for mapping video to target display panels using encoders and decoders are described.
- at least some of the functionality as described herein may be implemented by one or more components or modules of a system on a chip (SOC) that may be used in devices including but not limited to multifunction devices, smartphones, pad or tablet devices, and other portable computing devices such as laptop, notebook, and netbook computers.
- SOC system on a chip
- FIG. 10 illustrates an example SOC
- FIG. 11 illustrates an example device implementing an SOC.
- mapping to a target display panel is performed at least in part on the decoder side, with gamut mapping metadata generated on the encoder side and passed to a component on the decoder side used in recovering WCG video content.
- the methods and apparatus of FIGS. 8 and 9 may also include or implement the methods as shown in FIGS. 2 through 7 to provide both standard dynamic range (SDR) and high dynamic range (HDR) versions of video to display devices from a HDR video source.
- FIG. 9 is a flowchart of a video playback method in which an encoder generates output data in an intermediate color gamut and mapping to a display color gamut is performed by a decoder according to metadata generated by the encoder, according to some embodiments.
- an encoding pipeline maps video content from a wide color gamut to an intermediate color gamut.
- This intermediate color gamut is not necessarily the full range gamut of the source content on the encoding side, but is a color gamut that may lie between the full range color gamut and the color gamut of one or more target displays.
- the intermediate color gamut is represented at a bit depth (e.g., 10 bits) that is compatible with typical encoder/decoder hardware bit depth.
- the encoding pipeline then encodes the video content according to a compressed video format, for example H.264/AVC format. Encoding the video data may involve converting the video data from one color space to another color space, for example from CIE 1931 XYZ color space to a YUV Rec. 709 color space.
- the encoding pipeline records metadata describing at least the color gamut mapping that was performed.
- one or more decoding pipelines may each obtain the encoded video content and the metadata.
- each decoding pipeline decodes the encoded video content to generate decoded (uncompressed) video content at the intermediate color gamut.
- Decoding the video data may involve converting the video data from one color space to another color space, for example from YUV to XYZ.
- each decoding pipeline then maps the decoded video content from the intermediate color gamut to the color gamut of a respective target display panel according to the metadata. Mapping the video data may involve converting the video data from one color space to another color space, for example from XYZ to a YCC or YUV color space.
- the decoded video content may be further processed and provided to the respective display panel for display. Elements of the method of FIG. 9 are further described below in reference to FIG. 8 .
- FIG. 8 illustrates an example video playback system in which an encoder generates output data in an intermediate color gamut and in which mapping to a display color gamut is performed on the decoder/display side using metadata generated by the encoder, according to some embodiments.
- the example video playback method and system may involve H.264/AVC encoding in a wider or intermediate color gamut for distribution to displays, with decoder-side mapping to the display color gamut.
- FIG. 8 shows an encoding 800 module or pipeline including one or more components and a decoding 850 module or pipeline including one or more components.
- encoding 800 and decoding 850 may be implemented on an SOC.
- encoding 800 and decoding 850 may be implemented on the same device and/or SOC.
- encoding 800 and decoding 850 may be implemented on different devices or SOCs.
- one or more encoding 800 pipelines may be implemented on a device or system; the encoding 800 pipelines may be configured to encode and stream video to one or more target devices or systems each implementing at least one decoding 850 pipeline.
- color gamut mapping is performed at least in part on the decoding 850 side, with a video encoded stream (VES) 810 in a wide color gamut (WCG) and gamut mapping metadata 820 generated on the encoding 800 side and passed to a component on the decoding 850 side (e.g., to an ISP color pipe 854 , or alternatively to a GPU) for use in decoding 850 side gamut mapping operations.
- VES video encoded stream
- WCG wide color gamut
- gamut mapping metadata 820 generated on the encoding 800 side and passed to a component on the decoding 850 side (e.g., to an ISP color pipe 854 , or alternatively to a GPU) for use in decoding 850 side gamut mapping operations.
- the input video content may, for example, be encoded in (linear) CIE 1931 XYZ color space at a bit depth of 16 bits.
- a mapping component 802 may apply a 10-bit electro-optical transfer function (EOTF) operation to the input linear XYZ video to map the 16-bit input data to 10-bit log XYZ video data.
- EOTF electro-optical transfer function
- the EOTF may be a transfer function that maps to a color space that is wide enough to transfer to all target display panel color spaces.
- Mapping component 802 may also generate metadata 820 describing the mapping operation.
- An H.264 encoder component 804 encodes the 10-bit log XYZ video data to generate wide color gamut (WCG) H.264 compressed video 810 at a bit depth of 10 bits.
- WCG wide color gamut
- an H.264 decode component 852 decodes the H.264 compressed video 810 to generate 10-bit data in the XYZ color space.
- An Image Signal Processor (ISP) 854 color pipe, or alternatively a Graphics Processor Unit (GPU), may then be used to perform gamut mapping from the WCG of the 10-bit XYZ data to a display gamut according to the metadata 820 .
- the ISP 854 may generate 10-bit data in a YCC color space.
- a super-resolution technique 856 may be performed on the data, and the 10-bit YCC data may then be passed to a display pipe 858 for processing into display output data, for example 10-bit RGB data, at the color gamut of the display.
- encoding is done in a wider color gamut that fits into a bit depth supported by encoder/decoder hardware (e.g., 10 bits), and metadata 820 is passed to the decoding 850 to be used in color gamut mapping to a display gamut.
- FIG. 8 shows operations in the decoding 850 side that may leverage existing components of an SOC such as an H.264 decoder, ISP color pipe, and display pipe.
- Decoding may be performed at a bit depth of 10 bits, and conversion to the color gamut of the display may be performed using the metadata 820 information that describes how the gamut mapping should be done that is received from the encoding 800 side.
- some, but not all, of the color gamut processing is performed on the playback/decoding 850 side rather than on the server/encoding 800 side.
- the content is mapped into an intermediate color gamut.
- This intermediate color gamut is not necessarily the full range gamut of the source content on the encoding 800 side, but is a color gamut that may lie between the full range color gamut and the display gamut which can fit into 10-bit and that allows existing decoding 850 hardware that can support up to 10-bit decoding to be used.
- the metadata 820 may be used to map to the gamut of the target panel/display. In some embodiments, this mapping may be performed in two steps.
- the input video content 810 is mapped back to the original wider gamut according to the metadata 820 .
- This re-mapped content is then mapped down to the gamut of the display.
- the playback/decoding 850 side gamut mapping may be performed on, by, or in an Image Signal Processor (ISP) color pipe 854 which may, for example, perform gamma correction, color space conversion, etc.
- ISP Image Signal Processor
- one or more components of the ISP color pipe 854 e.g., 3 D color lookup tables (CLUTS)
- the playback/decoding 850 gamut mapping may instead or in addition be performed by or in one or more GPUs.
- Embodiments of a decoding 850 pipeline as shown in FIG. 8 may perform an initial decode with 10-bit data using a decoder 852 component, and may restore at least some of the color gamut based on metadata 820 received from encoding 800 either in an ISP color pipe 854 or in a GPU.
- the encoding 800 generates output encoded video stream 810 at a wider, intermediate color gamut that may attempt to capture a range of display gamut(s) that the video stream 810 could be targeted to.
- the metadata 820 generated by the encoding 800 and passed to one or more decoder(s) 850 defines how to map from the intermediate color gamut of the output stream 810 to the potentially narrower display gamuts.
- the metadata 820 may reference or map into one or more lookup tables (e.g., CLUTS) that define how to map from and to different color gamuts.
- one or more lookup tables referenced by the metadata 820 may be used by an ISP color pipe 854 or other decoding 850 component to map to a Rec. 709 gamut, to a P3 DCI D65, to a Rec. 2020 gamut, and so on.
- the metadata 820 may include equations and/or parameters (e.g., parameters for configuring a display pipe 858 , ISP color pipe 854 , and/or GPU(s)) for performing the decoder-side gamut mapping.
- metadata 820 may also include information used in a tone mapping technique (e.g., a local (LTM) and/or global (GTM) tone mapping technique) applied by the encoding 800 (e.g., by mapping 802 component) to map the input video data to a dynamic range of the output data stream 810 .
- the metadata 820 may then be used by an inverse tone mapping technique in the decoding 850 pipeline to recover at least some dynamic range that may have been clipped in the tone mapping applied by the encoding 800.
- a tone mapping module or component of an ISP color pipe may be used for tone mapping on the encoding 800 and/or decoding 850 side.
- a 3 D color lookup table (CLUT) of the ISP color pipe may be used for global and/or local tone mapping and for gamut mapping.
- encoding and decoding may be performed according to the H.265 High Efficiency Video Coding (HEVC) format instead of the H.264/AVC format.
- HEVC High Efficiency Video Coding
- FIGS. 10 through 14 show non-limiting examples of devices and apparatus in or with which embodiments or components of the various digital video or image processing and display methods and apparatus as described herein may be implemented.
- FIG. 10 illustrates an example SOC
- FIG. 11 illustrates an example device implementing an SOC.
- FIG. 12 illustrates an example computer system that may implement the methods and apparatus described herein.
- FIGS. 13 and 14 illustrate example multifunction devices that may implement the methods and apparatus described herein.
- FIG. 10 a block diagram of one embodiment of a system-on-a-chip (SOC) 8000 that may be used in embodiments.
- SOC 8000 is shown coupled to a memory 8800 .
- the components of the SOC 8000 may be integrated onto a single semiconductor substrate as an integrated circuit “chip.” In some embodiments, the components may be implemented on two or more discrete chips in a system. However, the SOC 8000 will be used as an example herein.
- the components of the SOC 8000 include a central processing unit (CPU) complex 8020 , on-chip peripheral components 8040 A- 8040 C (more briefly, “peripherals”), a memory controller (MC) 8030 , and a communication fabric 8010 .
- the components 8020 , 8030 , 8040 A- 8040 C, may all be coupled to the communication fabric 8010 .
- the memory controller 8030 may be coupled to the memory 8800 during use, and the peripheral 8040 B may be coupled to an external interface 8900 during use.
- the CPU complex 8020 includes one or more processors (P) 8024 and a level two (L2) cache 8022 .
- the peripherals 8040 A- 8040 B may be any set of additional hardware functionality included in the SOC 8000 .
- the peripherals 8040 A- 8040 B may include video peripherals such as an image signal processor configured to process image capture data from a camera or other image sensor, display controllers configured to display video data on one or more display devices, graphics processing units (GPUs), video encoder/decoders or codecs, scalers, rotators, blenders, etc.
- the peripherals may include audio peripherals such as microphones, speakers, interfaces to microphones and speakers, audio processors, digital signal processors, mixers, etc.
- the peripherals may include peripheral interface controllers for various interfaces 8900 external to the SOC 8000 (e.g.
- peripheral 8040 B including interfaces such as Universal Serial Bus (USB), peripheral component interconnect (PCI) including PCI Express (PCIe), serial and parallel ports, etc.
- the peripherals may include networking peripherals such as media access controllers (MACs). Any set of hardware may be included.
- the CPU complex 8020 may include one or more CPU processors 8024 that serve as the CPU of the SOC 8000 .
- the CPU of the system includes the processor(s) that execute the main control software of the system, such as an operating system. Generally, software executed by the CPU during use may control the other components of the system to realize the desired functionality of the system.
- the processors 8024 may also execute other software, such as application programs. The application programs may provide user functionality, and may rely on the operating system for lower level device control. Accordingly, the processors 8024 may also be referred to as application processors.
- the CPU complex 8020 may further include other hardware such as the L2 cache 8022 and/or and interface to the other components of the system (e.g. an interface to the communication fabric 8010 ).
- a processor may include any circuitry and/or microcode configured to execute instructions defined in an instruction set architecture implemented by the processor.
- the instructions and data operated on by the processors in response to executing the instructions may generally be stored in the memory 8800 , although certain instructions may be defined for direct processor access to peripherals as well.
- Processors may encompass processor cores implemented on an integrated circuit with other components as a system on a chip (SOC 8000 ) or other levels of integration. Processors may further encompass discrete microprocessors, processor cores and/or microprocessors integrated into multichip module implementations, processors implemented as multiple integrated circuits, etc.
- the memory controller 8030 may generally include the circuitry for receiving memory operations from the other components of the SOC 8000 and for accessing the memory 8800 to complete the memory operations.
- the memory controller 8030 may be configured to access any type of memory 8800 .
- the memory 8800 may be static random access memory (SRAM), dynamic RAM (DRAM) such as synchronous DRAM (SDRAM) including double data rate (DDR, DDR2, DDR3, etc.) DRAM.
- DDR dynamic RAM
- SDRAM synchronous DRAM
- Low power/mobile versions of the DDR DRAM may be supported (e.g. LPDDR, mDDR, etc.).
- the memory controller 8030 may include queues for memory operations, for ordering (and potentially reordering) the operations and presenting the operations to the memory 8800 .
- the memory controller 8030 may further include data buffers to store write data awaiting write to memory and read data awaiting return to the source of the memory operation.
- the memory controller 8030 may include a memory cache to store recently accessed memory data.
- the memory cache may reduce power consumption in the SOC by avoiding re-access of data from the memory 8800 if it is expected to be accessed again soon.
- the memory cache may also be referred to as a system cache, as opposed to private caches such as the L2 cache 8022 or caches in the processors 8024 , which serve only certain components. Additionally, in some embodiments, a system cache need not be located within the memory controller 8030 .
- the memory 8800 may be packaged with the SOC 8000 in a chip-on-chip or package-on-package configuration.
- a multichip module configuration of the SOC 8000 and the memory 8800 may be used as well.
- Such configurations may be relatively more secure (in terms of data observability) than transmissions to other components in the system (e.g. to the end points 16 A- 16 B). Accordingly, protected data may reside in the memory 8800 unencrypted, whereas the protected data may be encrypted for exchange between the SOC 8000 and external endpoints.
- the communication fabric 8010 may be any communication interconnect and protocol for communicating among the components of the SOC 8000 .
- the communication fabric 8010 may be bus-based, including shared bus configurations, cross bar configurations, and hierarchical buses with bridges.
- the communication fabric 8010 may also be packet-based, and may be hierarchical with bridges, cross bar, point-to-point, or other interconnects.
- the number of components of the SOC 8000 (and the number of subcomponents for those shown in FIG. 10 , such as within the CPU complex 8020 ) may vary from embodiment to embodiment. There may be more or fewer of each component/subcomponent than the number shown in FIG. 10 .
- FIG. 11 a block diagram of one embodiment of a system 9000 that includes at least one instance of an SOC 8000 coupled to one or more external peripherals 9020 and the external memory 8800 .
- a power management unit (PMU) 9010 is provided which supplies the supply voltages to the SOC 8000 as well as one or more supply voltages to the memory 8800 and/or the peripherals 9020 .
- PMU power management unit
- more than one instance of the SOC 8000 may be included (and more than one memory 8800 may be included as well).
- the peripherals 9020 may include any desired circuitry, depending on the type of system 9000 .
- the system 9000 may be a mobile device (e.g. personal digital assistant (PDA), smart phone, etc.) and the peripherals 9020 may include devices for various types of wireless communication, such as wifi, Bluetooth, cellular, global positioning system, etc.
- the peripherals 9020 may also include additional storage, including RAM storage, solid state storage, or disk storage.
- the peripherals 9020 may include user interface devices such as a display screen, including touch display screens or multitouch display screens, keyboard or other input devices, microphones, speakers, etc.
- the system 9000 may be any type of computing system (e.g. desktop personal computer, laptop, workstation, net top etc.).
- the external memory 8800 may include any type of memory.
- the external memory 8800 may be SRAM, dynamic RAM (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM, RAMBUS DRAM, low power versions of the DDR DRAM (e.g. LPDDR, mDDR, etc.), etc.
- DRAM dynamic RAM
- the external memory 8800 may include one or more memory modules to which the memory devices are mounted, such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc.
- the external memory 8800 may include one or more memory devices that are mounted on the SOC 8000 in a chip-on-chip or package-on-package implementation.
- FIG. 12 illustrates an example computer system 2900 that may be configured to execute any or all of the embodiments described above.
- computer system 2900 may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, tablet, slate, pad, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, a consumer device, application server, storage device, a video recording device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device.
- computer system 2900 may be executed in one or more computer systems 2900 , which may interact with various other devices.
- any component, action, or functionality described above with respect to FIGS. 1-11 may be implemented on one or more computers configured as computer system 2900 of FIG. 12 , according to various embodiments.
- computer system 2900 includes one or more processors 2910 coupled to a system memory 2920 via an input/output (I/O) interface 2930 .
- I/O input/output
- Computer system 2900 further includes a network interface 2940 coupled to I/O interface 2930 , and one or more input/output devices or components 2950 , such as cursor control 2960 , keyboard 2970 , display(s) 2980 , camera(s) 2990 , and sensor(s) 2992 including but not limited to light sensors and motion detectors.
- embodiments may be implemented using a single instance of computer system 2900 , while in other embodiments multiple such systems, or multiple nodes making up computer system 2900 , may be configured to host different portions or instances of embodiments.
- some elements may be implemented via one or more nodes of computer system 2900 that are distinct from those nodes implementing other elements.
- computer system 2900 may be a uniprocessor system including one processor 2910 , or a multiprocessor system including several processors 2910 (e.g., two, four, eight, or another suitable number).
- processors 2910 may be any suitable processor capable of executing instructions.
- processors 2910 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x829, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA.
- ISAs instruction set architectures
- each of processors 2910 may commonly, but not necessarily, implement the same ISA.
- System memory 2920 may be configured to store program instructions 2922 and/or data accessible by processor 2910 .
- system memory 2920 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory.
- program instructions 2922 may be configured to implement any of the functionality described herein.
- memory 2920 may include any of the information or data structures described herein.
- program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory 2920 or computer system 2900 . While computer system 2900 is described as implementing the functionality of functional blocks of previous Figures, any of the functionality described herein may be implemented via such a computer system.
- I/O interface 2930 may be configured to coordinate I/O traffic between processor 2910 , system memory 2920 , and any peripheral devices in the device, including network interface 2940 or other peripheral interfaces, such as input/output devices 2950 .
- I/O interface 2930 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 2920 ) into a format suitable for use by another component (e.g., processor 2910 ).
- I/O interface 2930 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example.
- PCI Peripheral Component Interconnect
- USB Universal Serial Bus
- I/O interface 2930 may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface 2930 , such as an interface to system memory 2920 , may be incorporated directly into processor 2910 .
- Network interface 2940 may be configured to allow data to be exchanged between computer system 2900 and other devices attached to a network 2985 (e.g., carrier or agent devices) or between nodes of computer system 2900 .
- Network 2985 may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof.
- LANs Local Area Networks
- WANs Wide Area Networks
- wireless data networks some other electronic data network, or some combination thereof.
- network interface 2940 may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.
- general data networks such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.
- Input/output devices 2950 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by one or more computer systems 2900 .
- Multiple input/output devices 2950 may be present in computer system 2900 or may be distributed on various nodes of computer system 2900 .
- similar input/output devices may be separate from computer system 2900 and may interact with one or more nodes of computer system 2900 through a wired or wireless connection, such as over network interface 2940 .
- memory 2920 may include program instructions 2922 , which may be processor-executable to implement any element or action described above.
- the program instructions may implement the methods described above.
- different elements and data may be included. Note that data may include any data or information described above.
- computer system 2900 is merely illustrative and is not intended to limit the scope of embodiments.
- the computer system and devices may include any combination of hardware or software that can perform the indicated functions, including computers, network devices, Internet appliances, PDAs, wireless phones, pagers, etc.
- Computer system 2900 may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system.
- the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components.
- the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.
- instructions stored on a computer-accessible medium separate from computer system 2900 may be transmitted to computer system 2900 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link.
- Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium.
- a computer-accessible medium may include a non-transitory, computer-readable storage medium or memory medium such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc.
- a computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link.
- FIG. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments.
- the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA, camera, video capture and/or playback, and/or music player functions.
- portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, Calif.
- Other portable electronic devices such as laptops, cell phones, smartphones, pad or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and/or touch pads), may also be used.
- the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).
- the device is a gaming computer with orientation sensors (e.g., orientation sensors in a gaming controller).
- the device is not a portable communications device, but is a camera and/or video camera.
- an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.
- the device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
- applications such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
- the various applications that may be executed on the device may use at least one common physical user-interface device, such as the touch-sensitive surface.
- One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device may be adjusted and/or varied from one application to the next and/or within a respective application.
- a common physical architecture (such as the touch-sensitive surface) of the device may support the variety of applications with user interfaces that are intuitive and transparent to the user.
- Device 2100 may include memory 2102 (which may include one or more computer readable storage mediums), memory controller 2122 , one or more processing units (CPU's) 2120 , peripherals interface 2118 , RF circuitry 2108 , audio circuitry 2110 , speaker 2111 , touch-sensitive display system 2112 , microphone 2113 , input/output (I/O) subsystem 2106 , other input control devices 2116 , and external port 2124 .
- Device 2100 may include one or more optical sensors or cameras 2164 . These components may communicate over one or more communication buses or signal lines 2103 .
- device 2100 is only one example of a portable multifunction device, and that device 2100 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of the components.
- the various components shown in FIG. 13 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.
- Memory 2102 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 2102 by other components of device 2100 , such as CPU 2120 and the peripherals interface 2118 , may be controlled by memory controller 2122 .
- Peripherals interface 2118 can be used to couple input and output peripherals of the device to CPU 2120 and memory 2102 .
- the one or more processors 2120 run or execute various software programs and/or sets of instructions stored in memory 2102 to perform various functions for device 2100 and to process data.
- peripherals interface 2118 , CPU 2120 , and memory controller 2122 may be implemented on a single chip, such as chip 2104 . In some other embodiments, they may be implemented on separate chips.
- RF (radio frequency) circuitry 2108 receives and sends RF signals, also called electromagnetic signals.
- RF circuitry 2108 converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals.
- RF circuitry 2108 may include well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder/decoder (codec) chipset, a subscriber identity module (SIM) card, memory, and so forth.
- codec coder/decoder
- RF circuitry 2108 may communicate with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication.
- networks such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication.
- networks such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication.
- WLAN wireless local area network
- MAN metropolitan area network
- the wireless communication may use any of a variety of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSDPA), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any
- Audio circuitry 2110 , speaker 2111 , and microphone 2113 provide an audio interface between a user and device 2100 .
- Audio circuitry 2110 receives audio data from peripherals interface 2118 , converts the audio data to an electrical signal, and transmits the electrical signal to speaker 2111 .
- Speaker 2111 converts the electrical signal to human-audible sound waves.
- Audio circuitry 2110 also receives electrical signals converted by microphone 2113 from sound waves.
- Audio circuitry 2110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 2118 for processing. Audio data may be retrieved from and/or transmitted to memory 2102 and/or RF circuitry 2108 by peripherals interface 2118 .
- audio circuitry 2110 also includes a headset jack.
- the headset jack provides an interface between audio circuitry 2110 and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone).
- removable audio input/output peripherals such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone).
- I/O subsystem 2106 couples input/output peripherals on device 2100 , such as touch screen 2112 and other input control devices 2116 , to peripherals interface 2118 .
- I/O subsystem 2106 may include display controller 2156 and one or more input controllers 2160 for other input control devices 2116 .
- the one or more input controllers 2160 receive/send electrical signals from/to other input control devices 2116 .
- the other input control devices 2116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth.
- input controller(s) 2160 may be coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse.
- the one or more buttons may include an up/down button for volume control of speaker 2111 and/or microphone 2113 .
- the one or more buttons may include a push button.
- Touch-sensitive display 2112 provides an input interface and an output interface between the device and a user.
- Display controller 2156 receives and/or sends electrical signals from/to touch screen 2112 .
- Touch screen 2112 displays visual output to the user.
- the visual output may include graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some embodiments, some or all of the visual output may correspond to user-interface objects.
- Touch screen 2112 has a touch-sensitive surface, sensor or set of sensors that accepts input from the user based on haptic and/or tactile contact.
- Touch screen 2112 and display controller 2156 (along with any associated modules and/or sets of instructions in memory 2102 ) detect contact (and any movement or breaking of the contact) on touch screen 2112 and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on touch screen 2112 .
- user-interface objects e.g., one or more soft keys, icons, web pages or images
- a point of contact between touch screen 2112 and the user corresponds to a finger of the user.
- Touch screen 2112 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other embodiments.
- Touch screen 2112 and display controller 2156 may detect contact and any movement or breaking thereof using any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 2112 .
- touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 2112 .
- projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, Calif.
- Touch screen 2112 may have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi.
- the user may make contact with touch screen 2112 using any suitable object or appendage, such as a stylus, a finger, and so forth.
- the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen.
- the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.
- device 2100 may include a touchpad (not shown) for activating or deactivating particular functions.
- the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output.
- the touchpad may be a touch-sensitive surface that is separate from touch screen 2112 or an extension of the touch-sensitive surface formed by the touch screen.
- Power system 2162 for powering the various components.
- Power system 2162 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices.
- power sources e.g., battery, alternating current (AC)
- AC alternating current
- a recharging system e.g., a recharging system
- a power failure detection circuit e.g., a power failure detection circuit
- a power converter or inverter e.g., a power converter or inverter
- a power status indicator e.g., a light-emitting diode (LED)
- Device 2100 may also include one or more optical sensors or cameras 2164 .
- FIG. 13 shows an optical sensor coupled to optical sensor controller 2158 in I/O subsystem 2106 .
- Optical sensor 2164 may, for example, include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors or photosensors.
- CMOS complementary metal-oxide semiconductor
- Optical sensor 2164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image.
- imaging module 2143 also called a camera module
- optical sensor 2164 may capture still images and/or video sequences.
- at least one optical sensor may be located on the back of device 2100 , opposite touch screen display 2112 on the front of the device.
- the touch screen display may be used as a viewfinder for still and/or video image acquisition.
- at least one optical sensor may instead or also be located on the front of the device.
- Device 2100 may also include one or more proximity sensors 2166 .
- FIG. 13 shows proximity sensor 2166 coupled to peripherals interface 2118 .
- proximity sensor 2166 may be coupled to input controller 2160 in I/O subsystem 2106 .
- the proximity sensor turns off and disables touch screen 2112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).
- Device 2100 may also include one or more orientation sensors 2168 .
- the one or more orientation sensors include one or more accelerometers (e.g., one or more linear accelerometers and/or one or more rotational accelerometers).
- the one or more orientation sensors include one or more gyroscopes.
- the one or more orientation sensors include one or more magnetometers.
- the one or more orientation sensors include one or more of global positioning system (GPS), Global Navigation Satellite System (GLONASS), and/or other global navigation system receivers. The GPS, GLONASS, and/or other global navigation system receivers may be used for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device 2100 .
- GPS global positioning system
- GLONASS Global Navigation Satellite System
- the GPS, GLONASS, and/or other global navigation system receivers may be used for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device 2100 .
- the one or more orientation sensors include any combination of orientation/rotation sensors.
- FIG. 13 shows the one or more orientation sensors 2168 coupled to peripherals interface 2118 .
- the one or more orientation sensors 2168 may be coupled to an input controller 2160 in I/O subsystem 2106 .
- information is displayed on the touch screen display in a portrait view or a landscape view based on an analysis of data received from the one or more orientation sensors.
- device 2100 may also include one or more other sensors (not shown) including but not limited to ambient light sensors and motion detectors. These sensors may be coupled to peripherals interface 2118 or, alternately, may be coupled to an input controller 2160 in I/O subsystem 2106 .
- device 2100 may include at least one forward-facing (away from the user) and at least one backward-facing (towards the user) light sensors that may be used to collect ambient lighting metrics from the environment of the device 2100 for use in video and image capture, processing, and display applications.
- the software components stored in memory 2102 include operating system 2126 , communication module 2128 , contact/motion module (or set of instructions) 2130 , graphics module 2132 , text input module 2134 , Global Positioning System (GPS) module 2135 , and applications 2136 .
- memory 2102 stores device/global internal state 2157 .
- Device/global internal state 2157 includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, views or other information occupy various regions of touch screen display 2112 ; sensor state, including information obtained from the device's various sensors and input control devices 2116 ; and location information concerning the device's location and/or attitude.
- Operating system 2126 e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks
- Operating system 2126 includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
- Communication module 2128 facilitates communication with other devices over one or more external ports 2124 and also includes various software components for handling data received by RF circuitry 2108 and/or external port 2124 .
- External port 2124 e.g., Universal Serial Bus (USB), FIREWIRE, etc.
- USB Universal Serial Bus
- FIREWIRE FireWire
- the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with the 30-pin connector used on iPod (trademark of Apple Inc.) devices.
- Contact/motion module 2130 may detect contact with touch screen 2112 (in conjunction with display controller 2156 ) and other touch sensitive devices (e.g., a touchpad or physical click wheel).
- Contact/motion module 2130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact).
- Contact/motion module 2130 receives contact data from the touch-sensitive surface.
- Determining movement of the point of contact may include determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts).
- contact/motion module 2130 and display controller 2156 detect contact on a touchpad.
- Contact/motion module 2130 may detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Thus, a gesture may be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.
- Graphics module 2132 includes various software components for rendering and displaying graphics on touch screen 2112 or other display, including components for changing the intensity of graphics that are displayed.
- graphics includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.
- graphics module 2132 stores data representing graphics to be used. Each graphic may be assigned a corresponding code. Graphics module 2132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller 2156 .
- Text input module 2134 which may be a component of graphics module 2132 , provides soft keyboards for entering text in various applications that need text input.
- GPS module 2135 determines the location of the device and provides this information for use in various applications (e.g., to telephone module 2138 for use in location-based dialing, to camera module 2143 as picture/video metadata, and to applications that provide location-based services such as map/navigation applications).
- applications e.g., to telephone module 2138 for use in location-based dialing, to camera module 2143 as picture/video metadata, and to applications that provide location-based services such as map/navigation applications.
- Applications 2136 may include one or more of, but are not limited to, the following modules (or sets of instructions), or a subset or superset thereof:
- Examples of other applications 2136 that may be stored in memory 2102 include but are not limited to other word processing applications, other image editing applications, drawing applications, presentation applications, communication/social media applications, map applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.
- telephone module 2138 may be used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in an address book, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed.
- the wireless communication may use any of a variety of communications standards, protocols and technologies.
- videoconferencing module 2139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.
- camera module 2143 includes executable instructions to capture still images or video (including a video stream) and store them into memory 2102 , modify characteristics of a still image or video, or delete a still image or video from memory 2102 .
- image management module 2144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.
- modify e.g., edit
- present e.g., in a digital slide show or album
- browser module 2147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
- search module 2151 includes executable instructions to search for text, music, sound, image, video, and/or other files in memory 2102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
- search criteria e.g., one or more user-specified search terms
- video and music player module 2152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on touch screen 2112 or on an external, connected display via external port 2124 ).
- device 2100 may include the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
- online video module 2155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port 2124 ), and otherwise manage online videos in one or more video formats, such as the H.264/AVC format or the H.265/HEVC format.
- modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein).
- modules i.e., sets of instructions
- memory 2102 may store a subset of the modules and data structures identified above.
- memory 2102 may store additional modules and data structures not described above.
- device 2100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad.
- a touch screen and/or a touchpad as the primary input control device for operation of device 2100 , the number of physical input control devices (such as push buttons, dials, and the like) on device 2100 may be reduced.
- the predefined set of functions that may be performed exclusively through a touch screen and/or a touchpad include navigation between user interfaces.
- the touchpad when touched by the user, navigates device 2100 to a main, home, or root menu from any user interface that may be displayed on device 2100 .
- the touchpad may be referred to as a “menu button.”
- the menu button may be a physical push button or other physical input control device instead of a touchpad.
- FIG. 14 illustrates a portable multifunction device 2100 having a touch screen 2112 in accordance with some embodiments.
- the touch screen may display one or more graphics within user interface (UI) 2200 .
- UI user interface
- a user may select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 2202 (not drawn necessarily drawn to scale in the figure) or one or more styluses 2203 (not necessarily drawn to scale in the figure).
- Device 2100 may also include one or more physical buttons, such as a “home” or menu button 2204 .
- menu button 2204 may be used to navigate to any application 2136 in a set of applications that may be executed on device 2100 .
- the menu button is may be implemented as a soft key in a GUI displayed on touch screen 2112 .
- device 2100 includes touch screen 2112 , home or menu button 2204 , push button 2206 for powering the device on/off and locking the device, volume adjustment button(s) 2208 , Subscriber Identity Module (SIM) card slot 2210 , head set jack 2212 , and docking/charging external port 2124 .
- Push button 2206 may be used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process.
- device 2100 also may accept verbal input for activation or deactivation of some functions through microphone 2113 .
- Device 2100 may also include one or more cameras 2164 .
- a camera 2164 may, for example, include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors or photosensors.
- CCD charge-coupled device
- CMOS complementary metal-oxide semiconductor
- a camera 2164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image or video frame.
- at least one camera 2164 may be located on the back of device 2100 , opposite touch screen display 2112 on the front of the device.
- at least one camera 2164 may instead or also located on the front of the device with the touch screen display 2112 , for example so that the user's image may be obtained for videoconferencing while the user views the other video conference participants on the touch screen display 2112 .
- At least one camera 2164 may be located on the front of the device 2100 , and at least one camera 2164 may be located on the back of the device 2100 .
- the touch screen display 2112 may be used as a viewfinder and/or user interface for still image and/or video sequence acquisition applications.
- Device 2100 may include video and image processing hardware and/or software, including but not limited to video encoding and/or decoding components, codecs, modules, or pipelines, that may be used to capture, process, convert, compress, decompress, store, modify, transmit, display, and otherwise manage and manipulate still images and/or video frames or video sequences captured via camera 2164 or otherwise acquired (e.g., via a network interface).
- device 2100 may also include one or more light or other sensors that may be used to collect ambient lighting or other metrics from the environment of the device 2100 for use in video and image capture, processing, and display.
- the methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments.
- the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
- Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure.
- the various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Ecology (AREA)
- Remote Sensing (AREA)
- Biodiversity & Conservation Biology (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Environmental & Geological Engineering (AREA)
- Environmental Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Controls And Circuits For Display Device (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
Video processing techniques and pipelines that support capture, distribution, and display of high dynamic range (HDR) image data to both HDR-enabled display devices and display devices that do not support HDR imaging. A sensor pipeline may generate standard dynamic range (SDR) data from HDR data captured by a sensor using tone mapping, for example local tone mapping. Information used to generate the SDR data may be provided to a display pipeline as metadata with the generated SDR data. If a target display does not support HDR imaging, the SDR data may be directly rendered by the display pipeline. If the target display does support HDR imaging, then an inverse mapping technique may be applied to the SDR data according to the metadata to render HDR data for display. Information used in performing color gamut mapping may also be provided in the metadata and used to recover clipped colors for display.
Description
- This application is a continuation of U.S. patent application Ser. No. 16/277,892, filed Feb. 15, 2019, which is a continuation of U.S. patent application Ser. No. 14/631,401, filed Feb. 25, 2015, which claims benefit of priority of U.S. Provisional Application Ser. No. 61/944,484 entitled “DISPLAY PROCESSING METHODS AND APPARATUS” filed Feb. 25, 2014, and U.S. patent application Ser. No. 14/631,401 claims benefit of priority of U.S. Provisional Application Ser. No. 61/946,638 entitled “DISPLAY PROCESSING METHODS AND APPARATUS” filed Feb. 28, 2014, and U.S. patent application Ser. No. 14/631,401 claims benefit of priority of U.S. Provisional Application Ser. No. 61/946,633 entitled “ADAPTIVE METHODS AND APPARATUS” filed Feb. 28, 2014, the contents of which are incorporated by reference herein in their entirety.
- This disclosure relates generally to digital video or image processing and display.
- Various devices including but not limited to personal computer systems, desktop computer systems, laptop and notebook computers, tablet or pad devices, digital cameras, digital video recorders, and mobile phones or smart phones may include software and/or hardware that may implement video processing method(s). For example, a device may include an apparatus (e.g., an integrated circuit (IC), such as a system-on-a-chip (SOC), or a subsystem of an IC), that may receive and process digital video input from one or more sources and output the processed video frames according to one or more video processing methods. As another example, a software program may be implemented on a device that may receive and process digital video input from one or more sources according to one or more video processing methods and output the processed video frames to one or more destinations.
- As an example, a video encoder may be implemented as an apparatus, or alternatively as a software program, in which digital video input is encoded or converted into another format according to a video encoding method, for example a compressed video format such as H.264/Advanced Video Coding (AVC) format, or H.265 High Efficiency Video Coding (HEVC) format. As another example, a video decoder may be implemented as an apparatus, or alternatively as a software program, in which video in a compressed video format such as AVC or HEVC is received and decoded or converted into another (decompressed) format according to a video decoding method, for example a display format used by a display device. The H.264/AVC standard is published by ITU-T in a document titled “ITU-T Recommendation H.264: Advanced video coding for generic audiovisual services”. The H.265/HEVC standard is published by ITU-T in a document titled “ITU-T Recommendation H.265: High Efficiency Video Coding”.
- In many systems, an apparatus or software program may implement both a video encoder component and a video decoder component; such an apparatus or program is commonly referred to as a codec. Note that a codec may encode/decode both visual/image data and audio/sound data in a video stream.
- Generally defined, dynamic range is the ratio between the largest and smallest possible values of a changeable quantity, such as in signals like sound and light. In digital image and video processing, conventionally, digital images (e.g., video or still images) are captured, rendered, and displayed at a limited dynamic range, referred to as standard dynamic range (SDR) imaging. In addition, images are conventionally rendered for display using a relatively narrow color gamut, referred to as standard color gamut (SCG) imaging. Extended or high dynamic range (HDR) imaging refers to technology and techniques that produce a wider range of luminance in electronic images (e.g., as displayed on display screens or devices) than is obtained using standard digital imaging technology and techniques (referred to as standard dynamic range, or SDR, imaging). Many new devices such as image sensors and displays support HDR imaging as well as wide color gamut (WCG) imaging. These devices may be referred to as HDR-enabled devices or simply HDR devices. At the same time, many video capture devices and display devices may not directly support HDR imaging and/or WCG imaging.
- Embodiments of video processing techniques and pipelines are described that support capture, distribution, and display of high dynamic range (HDR) image data to both HDR-enabled display devices and display devices that do not support HDR imaging. In embodiments, a sensor pipeline may generate lower or standard dynamic range (SDR) image data from HDR image data captured by an image sensor, for example using a tone mapping technique. Information used to generate the SDR image data may be recorded as metadata with the generated SDR image data. The SDR image data may be provided to a display pipeline to be rendered for display on a target display device. If the target display device does not support HDR imaging, the SDR image data may be directly rendered by the display pipeline for display. If the target display does support HDR imaging, then an inverse mapping technique may be applied to the SDR image data by the display pipeline according to the metadata to render HDR image data for display.
- In some embodiments, instead of or in addition to a global tone mapping (GTM) technique, a local tone mapping (LTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data. In an LTM technique, an image or frame is divided into multiple regions, with a tone curve dynamically determined for each region. The regional or local tone curves can be parameterized and stored in the metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered in an inverse tone mapping technique. In some embodiments, the local tone mapping metadata can be compressed spatially (e.g., within an image) and/or temporally (e.g., across two or more frames).
- In some embodiments, the inverse tone mapping technique performed by the display pipeline may be tuned to match the dynamic range of the target display. In some embodiments, the inverse tone mapping technique of the display pipeline may be an ambient adaptive process in which information about ambient light or other environmental conditions for the display may be used to adjust the inverse tone mapping process, for example by modifying or weighting one or more of the tone curves used in local tone mapping.
- In some embodiments, in addition to tone mapping information, information used in performing color gamut mapping on the sensor pipeline may be recorded in the metadata and used by the display pipeline in image reconstruction to recover a wider color gamut for displays that support WCG imaging. In some embodiments, the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to the lower display color gamut supported by a respective display according to the metadata.
-
FIG. 1 illustrates operations of a video pipeline. -
FIG. 2 illustrates an example video pipeline that may provide both standard dynamic range (SDR) and high dynamic range (HDR) video from source HDR content, according to some embodiments. -
FIG. 3 illustrates an example video pipeline that provides SDR video from source HDR content, according to some embodiments. -
FIG. 4 illustrates local tone mapping (LTM), according to some embodiments. -
FIG. 5 is a high-level flowchart of a method that may be implemented in a sensor pipeline, according to some embodiments. -
FIG. 6 is a high-level flowchart of a method that may be implemented in a display pipeline, according to some embodiments. -
FIG. 7 is a high-level flowchart of a method that may be implemented in a video pipeline that supports HDR displays and that is backwards compatible with SDR displays, according to some embodiments. -
FIG. 8 illustrates an example video playback system in which an encoder generates output data in an intermediate color gamut and in which mapping to a display color gamut is performed on the decoder/display side using metadata generated by the encoder, according to some embodiments. -
FIG. 9 is a flowchart of a video playback method in which an encoder generates output data in an intermediate color gamut and mapping to a display color gamut is performed by a decoder according to metadata generated by the encoder, according to some embodiments. -
FIG. 10 is a block diagram of one embodiment of a system on a chip (SOC) that may be configured to implement aspects of the systems and methods described herein. -
FIG. 11 is a block diagram of one embodiment of a system that may include one or more SOCs. -
FIG. 12 illustrates an example computer system that may be configured to implement aspects of the systems and methods described herein, according to some embodiments. -
FIG. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments. -
FIG. 14 depicts a portable multifunction device in accordance with some embodiments. - While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.
- Various units, circuits, or other components may be described as “configured to” perform a task or tasks. In such contexts, “configured to” is a broad recitation of structure generally meaning “having circuitry that” performs the task or tasks during operation. As such, the unit/circuit/component can be configured to perform the task even when the unit/circuit/component is not currently on. In general, the circuitry that forms the structure corresponding to “configured to” may include hardware circuits. Similarly, various units/circuits/components may be described as performing a task or tasks, for convenience in the description. Such descriptions should be interpreted as including the phrase “configured to.” Reciting a unit/circuit/component that is configured to perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112, paragraph six, interpretation for that unit/circuit/component.
- Embodiments of video processing techniques and pipelines are described that support capture, distribution, and display of high dynamic range (HDR) image data to both HDR-enabled display devices and display devices that do not support HDR imaging. Embodiments of a video processing pipeline may include a sensor pipeline portion and a display pipeline portion. The sensor pipeline and display pipeline may be in the same device, or may be in different devices. In embodiments, the sensor pipeline may generate lower or standard dynamic range (SDR) image data from HDR image data captured by an image sensor or photosensor of a camera associated with the sensor pipeline, for example using a tone mapping technique. Information used to generate the SDR image data (e.g., tone mapping information such as parameterized tone curves) may be recorded as metadata with the generated SDR image data. The SDR image data may be provided to a display pipeline to be rendered for display on a target display device. If the target display device does not support HDR imaging, the SDR image data may be directly rendered by the display pipeline for display. If the target display does support HDR imaging, then an inverse mapping technique may be applied to the SDR image data by the display pipeline according to the metadata to render HDR image data for display.
- In some embodiments, a global tone mapping (GTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data. In a GTM technique, a global tone curve may be specified or determined for one or more video frames and used in converting the HDR image data to SDR. In some embodiments, the global tone curve can be parameterized and stored in metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered in an inverse tone mapping technique when generating the HDR image data from the input SDR image data.
- In some embodiments, instead of or in addition to a GTM technique, a local tone mapping (LTM) technique may be used in the sensor pipeline in converting the input HDR image data to SDR image data. In an LTM technique, an image or frame is divided into multiple regions, with a tone curve dynamically determined for each region. The regional or local tone curves can be parameterized and stored in the metadata that is passed to the display pipeline so that tone or luminance can be at least partially recovered by an inverse tone mapping technique when generating HDR image data from the input SDR image data. In some embodiments, since the local tone curves may be similar among neighboring regions on a frame or between the same region on neighboring frames, the local tone mapping metadata can be compressed spatially (e.g., within an image) and/or temporally (e.g., across two or more frames).
- In some embodiments, the inverse tone mapping technique performed by the display pipeline may be tuned to match the dynamic range of the target display. The dynamic range captured by the sensor may cover a wider range than the dynamic range supported by the display, even if the display is HDR-compatible. Thus, in some embodiments, the inverse tone mapping technique of the display pipeline may modify the tone map metadata according to the dynamic range of the display, or may otherwise modify the inverse tone mapping method applied to the lower dynamic range image data, to generate HDR image data that better matches the dynamic range of the display.
- In some embodiments, the inverse tone mapping technique of the display pipeline may be an ambient adaptive process in which information about ambient light or other environmental conditions for the display may be obtained, e.g. using light, motion, or other sensors on a device, and based on that information the inverse tone mapping technique may be adjusted, for example by modifying or weighting one or more parameters of the tone curves used in local tone mapping when performing the inverse tone mapping. For example, if ambient light is bright, the inverse tone mapping technique may adjust the tone curves to generate a brighter image with more contrast.
- Devices such as image sensors and displays may support wide color gamut (WCG) imaging in addition to HDR imaging. In addition to supporting HDR imaging, embodiments may also support wide color gamut (WCG) imaging. In some embodiments, the sensor pipeline may generate lower color gamut image data from wider color gamut image data captured by an image sensor of a camera associated with the sensor pipeline using a color gamut mapping technique that may clip colors from the wider color gamut range of a color space (e.g., an RGB color space) that aren't available in the lower color gamut of another color space (e.g., a YUV color space). In some embodiments, in addition to tone mapping information, information used in performing the color gamut mapping may be recorded in the metadata and used by the display pipeline in image reconstruction to recover a wider color gamut for displays that support WCG imaging. In some embodiments, the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to the lower display color gamut supported by a respective display according to the metadata.
- Embodiments of video pipelines including sensor pipelines and display pipelines as described herein may, for example, be implemented in devices or systems that include one or more image capture devices and/or one or more display devices. An image capture device may be any device that includes an optical sensor or photosensor that is capable of capturing digital images or video. Image capture devices may include, but are not limited to, video cameras and still image cameras, as well as image capture devices that can capture both video and single images. Image capture devices may be stand-alone devices or may be cameras that are integrated into other devices including but not limited to smartphones, cellphones, PDAs, tablet or pad devices, multifunction devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and so on. Note that image capture devices may include small form factor cameras suitable for use in small devices such as cellphones, PDAs, and tablet devices. Displays or display devices may include display screens or panels that are integrated into other devices including but not limited to smartphones, cellphones, PDAs, tablet or pad devices, multifunction devices, computing devices, laptop computers, notebook computers, netbook computers, desktop computers, and so on. Display devices may also include video monitors, projectors, or in general any device that can display or project digital images and/or digital video. The displays or display devices may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used.
-
FIGS. 10 through 14 show non-limiting examples of devices in which embodiments may be implemented. A device or system that includes an image capture device and/or a display device may include hardware and/or software that implements at least some of the functionality for processing video data as described herein. In some embodiments, a portion of the functionality as described herein may be implemented on one device, while another portion may be implemented on another device. For example, in some embodiments, a device that includes an image capture device may implement a sensor pipeline that processes and compresses (i.e., encodes) images or video captured via a photosensor, while another device that includes a display panel or screen may implement a display pipeline that receives and processes the compressed images (i.e., decodes) for display to the panel or screen. In some embodiments, at least some of the functionality as described herein may be implemented by one or more components or modules of a system on a chip (SOC) that may be used in devices including but not limited to multifunction devices, smartphones, pad or tablet devices, and other portable computing devices such as laptop, notebook, and netbook computers.FIG. 10 illustrates an example SOC, andFIG. 11 illustrates an example device implementing an SOC.FIG. 12 illustrates an example computer system that may implement the methods and apparatus described herein.FIGS. 13 and 14 illustrate example multifunction devices that may implement the methods and apparatus described herein. - Generally defined, dynamic range is the ratio between the largest and smallest possible values of a changeable quantity, such as in signals like sound and light. In digital image processing, a high dynamic range (HDR) image is an image that is produced using an HDR imaging technique that produces a wider range of luminosity than is obtained using standard digital imaging techniques. For example, an HDR image may include more bits per channel (e.g., 10, 12, 14, or more bits per luminance (luma) and chrominance (chroma) channel), or more bits for luminosity (the luma channel), than are used in conventional image processing (typically, 8 bits per channel, e.g. 8 bits for color/chroma and for luma). An image produced using standard digital imaging techniques may be referred to as having a standard dynamic range (SDR), and typically uses 8 bits per channel. Generally defined, tone mapping is a technique that maps one set of tonal image values (e.g., from HDR image data) to another (e.g., to SDR image data). Tone mapping may be used, for example, to approximate the appearance of HDR images in a medium that has a more limited dynamic range (e.g., SDR). Tone mapping may generally be applied to luma image data.
- Generally defined, color gamut refers to a particular subset of colors, for example the subset of colors which can be accurately represented in a given circumstance, such as within a given color space (e.g., an RGB color space) or by a display device. Color gamut may also refer to the complete set of colors found within an image. At least some image processing operations performed on an image may alter the image's color gamut, and in some cases some of the colors in the original image may be lost. In other words, an image's color gamut may be narrowed or clipped by an image processing technique, for example a tone mapping technique, thus reducing the number of colors that can be represented by or reproduced from the image data. In addition, a gamut mapping technique (which may also be referred to as color or chroma mapping) may be applied to image data (generally to chroma image data), and may in some cases narrow or clip an image's color gamut, or alternatively may be used to correct or adjust the color gamut or range of an image during or after tone mapping.
- Embodiments are generally described as using YUV Rec. 709, otherwise known as ITU-R Recommendation BT.709, as a base SDR layer for transmitting or transferring digital image data between devices, e.g. between a camera sensor pipeline and a display pipeline. YUV Rec. 709 defines a color space and format that may be used in color image or video pipelines. However, note that other color spaces and formats may be used in embodiments. As just one example, Digital Cinema Initiatives (DCI) P3 may be used.
- Embodiments are generally described as processing video frames or sequences. However, embodiments may be applied to process single or still images instead of or in addition to video frames or sequences, as well as other digital images. Thus, when “video”, “video frame”, “frame”, or the like is used herein, it is to be understood that the terms may refer to captured digital images in general.
- Backward-Compatible HDR Image Capture, Distribution, and Display Systems
-
FIG. 1 graphically illustrates operations of a video pipeline at a high level. As shown inFIG. 1 , in asensor pipeline 110 portion of avideo pipeline 100, a camera sensor may capture images or frames, shown as camerasensor RGB data 120. The images may be captured at the sensor at a higher dynamic range (HDR image data), while adisplay pipeline 160 is configured to process image data at a lower dynamic range (SDR image data) for output to a display. The images may be captured by the sensor according to the RGB color model or according to other color models. An image signal processor (ISP) 122 may process the captured camerasensor RGB data 120 to generate linearRGB image data 124 as output (e.g., image data in the linear RGB color space). The linearRGB image data 124 may still be at the higher dynamic range of the sensor. Atone mapping 126 technique (e.g., a local and/or global tone mapping technique) may be applied to the linearRGB image data 124 to convert the higher dynamic range linearRGB image data 124 to YUV Rec. 709image data 130A at a lower dynamic range (SDR) for processing by thedisplay pipeline 160. - In some embodiments, in addition to reducing the dynamic range of the image data, application of a
tone mapping 126 technique may result in the clipping of color values in the image data, thus narrowing the color gamut of the image(s). In at least some cases, a gamut mapping technique may also be applied to the image data, and may result in clipping of the color gamut of the image data. - The YUV Rec. 709
image data 130A may generally be compressed 132 to generate YUV Rec. 709compressed data 150 for transfer or transmittal to adisplay pipeline 160. - As shown in
FIG. 1 , in adisplay pipeline portion 160 of avideo pipeline 100, the YUV Rec. 709compressed data 150 generated by asensor pipeline 110 may be decompressed 170 to generate YUV Rec. 709image data 130B at SDR. Acolor conversion 174 technique may then be applied to the YUV Rec. 709image data 130B to generatesRGB image data 176, also at SDR. In other words, the image data is converted from the YUV Rec. 709 color space to the sRGB color space according to thecolor conversion 174 technique. ThesRGB image data 176 is then passed to displaymanagement 178 hardware and/or software that renders SDRRGB image data 180 for display on a target panel or screen. - However, devices such as image sensors and displays may support high dynamic range (HDR) imaging as well as wide color gamut (WCG) imaging. These devices may be referred to as HDR-enabled devices or simply HDR devices. Using the video pipeline as shown in
FIG. 1 ,SDR image data 150 is passed from asensor pipeline 110 to adisplay pipeline 160 for a display that supports HDR or for a display that does not support HDR. SDR image data would be passed to and rendered on an HDR-enabled display according to the lower, standard dynamic range even if the sensor is capable of capturing HDR image data. In addition, the wide color gamut of the image data as original captured may be narrowed or clipped by thesensor pipeline 110 processing. Thus, much of the tonal range, color, and luminance that is actually captured by the sensor may be lost. - Embodiments of backward-compatible methods and apparatus that provide both standard dynamic range (SDR) and higher dynamic range (HDR) versions of video are described. Embodiments may implement methods and apparatus to support processing and distribution of HDR image data to both HDR-enabled display devices and devices that do not support HDR imaging. Embodiments may implement a video pipeline that includes a sensor or encoding pipeline portion and a display or decoding pipeline portion. The video pipeline may be implemented in a single device, for example a device that includes both an image capture device and a display panel. However, the sensor pipeline portion and the display pipeline portion may also be implemented in different devices, with compressed images/video transmitted from one device to the other, for example over a network, or otherwise transferred from one device to the other. For example, the sensor pipeline portion may be implemented in a video camera, and the display pipeline portion may be implemented in a display device. In some embodiments, one or both of the encoding pipeline and the display pipeline may be implemented at least in part on system on a chip (SOC).
- In some embodiments, one or more encoding pipelines may be implemented on a device or system; the encoding pipeline(s) may be configured to encode input video to generate standard dynamic range (SDR) video, stream the SDR video to one or more target devices each implementing at least one decoding pipeline, and generate and provide mapping metadata used in encoding the video to the target device(s). The target devices may include both HDR-enabled devices and SDR devices that do not support HDR imaging. A decoding pipeline for an HDR-enabled device may receive and use the mapping metadata generated by the encoding pipeline to recover at least part of the HDR of the input video that was lost in the encoding process when decoding the SDR video. A decoding pipeline for a SDR device may receive and process the SDR video without applying the mapping metadata. In some embodiments, encoding pipelines and decoding pipelines may communicate information that allows an encoding pipeline to generate and transmit mapping metadata to HDR-enabled devices, while not transmitting mapping metadata to SDR devices. In some embodiments, the mapping metadata may also include information that may be used in a decoding pipeline to recover at least part of the wide color gamut (WCG) of the input video that was lost in a color gamut mapping technique that converts the WCG input video data to a narrower color gamut used in the color space of the encoded SDR video.
-
FIG. 5 is a high-level flowchart of a method that may be implemented in a sensor pipeline, according to some embodiments. As illustrated at 1100, a camera/sensor may capture image(s) at a high dynamic range. As indicated at 1102, lower dynamic range image data may be generated from the captured image data, for example using a mapping technique such as a tone mapping technique. As illustrated at 1104, information used to generate the lower dynamic range image data may be recorded as metadata with the image data. The image data and/or metadata may be compressed. -
FIG. 6 is a high-level flowchart of a method that may be implemented in a display pipeline, according to some embodiments. As indicated at 1200, lower dynamic range image(s) may be obtained, for example from a sensor pipeline as described inFIG. 5 and illustrated inFIGS. 2 and 3 . As indicated at 1202, metadata for the image(s) may also be obtained that includes information on how the image data was mapped from a higher dynamic range to a lower dynamic range. As indicated at 1204, an inverse mapping technique may be applied to the image(s) according to the metadata to recover higher dynamic range image data as captured by the sensor. -
FIG. 7 is a high-level flowchart of a method that may be implemented in a video pipeline that supports HDR displays and that is backwards compatible with SDR displays, according to some embodiments. As illustrated at 1300, a camera/sensor may capture image(s) at a high dynamic range. As indicated at 1302, lower dynamic range image data may be generated from the captured image data, for example using a mapping technique such as a tone mapping technique. As illustrated at 1304, information used to generate the lower dynamic range image data may be recorded as metadata with the image data. The image data and/or metadata may be compressed. - At 1306, the image data may be provided to a display pipeline. If the target display supports HDR imaging, then the lower dynamic range image(s) may be provided to the display pipeline, as indicated at 1310. In addition, metadata for the image(s) may also be provided that includes information on how the image data was mapped from a higher dynamic range to a lower dynamic range, as indicated at 1312. As indicated at 1314, an inverse mapping technique may be applied to the image(s) according to the metadata to render higher dynamic range image data as captured by the sensor.
- At 1306, if the target display does not support HDR imaging, then the lower dynamic range image(s) may be provided to the display pipeline, as indicated at 1320. However, the metadata may not be provided. The display pipeline may then render and display SDR image data from the provided image data, as indicated at 1322. Inverse mapping to recover higher dynamic range image data is not performed, as the display does not support HDR imaging.
- In some embodiments of the methods of
FIGS. 5 through 7 , in addition to tone mapping information, information used in performing color gamut mapping may be recorded in the metadata by the sensor pipeline and used by the display pipeline to recover a wider color gamut (WCG) for displays that support WCG imaging. In some embodiments, the sensor pipeline may generate image data in an intermediate color gamut that a display pipeline can map to a lower display color gamut supported by a respective display according to the metadata. - Elements of the methods of
FIGS. 5 through 7 are further described below in reference toFIGS. 2 through 4 . -
FIG. 2 illustrates anexample video pipeline 200 that may process HDR image data captured by a sensor and provide both standard dynamic range (SDR) image data to non-HDR-enabled display devices and high dynamic range (HDR) image data to HDR-enabled display devices, according to some embodiments. Using thevideo pipeline 200 ofFIG. 2 , compressedSDR image data 250 may be passed from asensor pipeline 210 to adisplay pipeline 260 for a display that supports HDR or for a display that does not support HDR. To support HDR-enabled display devices,metadata 232A describing the processing performed by thesensor pipeline 210 may also be passed to thedisplay pipeline 260. Themetadata 232A may allow much if not all of the tonal range, color, and/or luminance that was originally captured by the sensor but that was removed by the HDR-to-SDR conversion performed by thesensor pipeline 210 to be recovered and rendered by thedisplay pipeline 260 and viewed on a target HDR-enabled display device. For non-HDR-enabled display devices, theSDR image data 250 can be processed as normal by thedisplay pipeline 260, for example according to an SDR processing technique as shown fordisplay pipeline 160 inFIG. 1 . - As shown in
FIG. 2 , in asensor pipeline 210 portion of avideo pipeline 200, a camera sensor may capture images or frames, shown as HDRcamera sensor RGB 220 data. The images may be captured by the sensor according to the RGB color model or according to other color models. An image signal processor (ISP) 222 may process the captured camerasensor RGB data 220 to generate HDR linearRGB image data 224 as output. A tone mapping module or component may apply atone mapping 226 technique (e.g., a local (LTM) and/or global (GTM) tone mapping technique) to the linearRGB image data 224 to convert the HDR linearRGB image data 224 to SDR YUV Rec. 709image data 230A.FIG. 4 illustrates a local tone mapping technique that may be used in some embodiments. - In some embodiments, in addition to reducing the dynamic range of the image data, application of the
tone mapping 226 technique may result in the clipping of color values in the image data, thus narrowing the color gamut of the image(s). In at least some cases, a color gamut mapping technique may also be applied to the image data in the sensor pipeline 210 (e.g., by ISP 222) to map the colors from one color space (e.g., RGB) to another (e.g., YUV), which may result in clipping of the colors in the processedimage data 230A. - In the
video pipeline 200 shown inFIG. 2 , thesensor pipeline 210 records information used to perform thetone mapping 226 and/or color gamut mapping asmetadata 232A and stores, embeds, or otherwise associates the metadata with the SDR YUV Rec. 709image data 230A. In some embodiments, themetadata 232A may be compressed temporally and/or spatially. The YUV Rec. 709image data 230A may be compressed 234. The compressed YUV Rec. 709image data 230A andmetadata 232A may be output as compressed YUV Rec. 709+metadata 250, which may be stored, transferred or transmitted to one ormore display pipelines 260. - At the
display pipeline 260, the compressed YUV Rec. 709+metadata 250 may be decompressed 270 to generate metadata 232B and decompressed SDR YUV Rec. 709image data 230B. The metadata 232B may then be used by an inverse tone mapping module or component of thedisplay pipeline 260 in performing an inverse global and/or local tone mapping (TM)technique 276 on the SDR YUV Rec. 709image data 230B to generateHDR image data 278. The inversetone mapping technique 276 recovers or reconstructs at least a portion of the tonal range and luminance that was lost by thetone mapping technique 226 applied in thesensor pipeline 210. - In some embodiments, a color conversion technique may also be applied to the YUV data to convert to a different color space, for example to an xRGB (extended RGB) color space. In some embodiments, the metadata 232B may also include information used to perform color gamut mapping, and this information may be used to recover or reconstruct a portion or all of the color gamut range that was clipped in the color gamut mapping operation on the
sensor pipeline 210, thus providing a wider color gamut for theHDR image data 278 for display. ThexRGB image data 278 may then be passed todisplay management 280 hardware and/or software of an HDR-enabled device that renders HDRRGB image data 282 for display on a target panel or screen. - Using the
sensor pipeline 210 as shown inFIG. 2 , SDR, YUV Rec. 709 image data 230 may still be passed to display pipelines for displays that do not support HDR, and the SDR, YUV Rec. 709image data 230B may be decompressed and processed without performing the inverse tone mapping operation (e.g., as illustrated fordisplay pipeline 160 inFIG. 1 ) so that the image data can be displayed as normal SDR data on non-SDR-enabled displays. Thus, thesensor pipeline 210 as illustrated inFIG. 2 works to provide HDR imaging for HDR-enabled displays, while also being backwards compatible with displays that do not support HDR imaging. - In some embodiments, handshaking may be used between the
sensor pipeline 210 and thedisplay pipeline 260, or between thedisplay pipeline 260 and another source of SDR, YUV Rec. 709image data 250, so that a target display's dynamic range can be determined. If the target display does not support HDR, thecompressed metadata 232A may not be transferred or transmitted to thedisplay pipeline 260 with the compressed YUV Rec. 709image data 230A. - In some embodiments, in the video pipeline as shown in
FIG. 2 , thesensor pipeline 210 and thedisplay pipeline 260 may be components of a same device or of a same system on a chip (SOC), and the compressed SDR, YUV Rec. 709image data 250 may be transferred, for example, over a communications bus or other connection local to the device. In some embodiments, thesensor pipeline 210 and thedisplay pipeline 260 may instead be components of different devices, and the compressed SDR, YUV Rec. 709image data 250 may be transferred or transmitted via other media, for example via a wired and/or wireless network, or over a connect (e.g., a USB connection) coupling the devices, or via a removable storage medium such as a thumb drive, CD, or DVD. - Local Tone Mapping
- In some embodiments, a local tone mapping (LTM) technique may be used in the
sensor pipeline 210 in which an image is divided into multiple regions, with each region having its own tone curve. The regional tone curves can be parameterized and stored in themetadata 232A that is passed to thedisplay pipeline 260 so that tone/luminance can be recovered in aninverse LTM technique 276 when generating theHDR image data 278 from the inputSDR image data 230B and the local tone mapping information indicated in the metadata 232B. In various embodiments, a LTM technique can be performed in RGB color space or in YUV color space; more generally, LTM as described herein may be performed in any color space. -
FIG. 4 graphically illustrates a local tone mapping (LTM) technique, according to some embodiments. Embodiments of the LTM technique may, for example, be used in embodiments of the video pipelines as illustrated inFIGS. 1 through 3 . However, embodiments of the LTM technique as described herein may be applied in other image or video processing techniques in which tone mapping is used. - As shown in
FIG. 4 , in a LTM technique implemented in asensor pipeline 210 as illustrated inFIG. 2 , animage 400 may be divided intomultiple regions 402, with alocal tone curve 404 indicated or determined for eachregion 402 of theimage 400. WhileFIG. 4 shows theimage 400 evenly subdivided into square orrectangular regions 402, in someembodiments regions 402 of different shapes or of varying sizes and/or shapes may be used. - The tone curves 404 for the
regions 402 may be similar, but may differ at least slightly betweenregions 402 according to varying characteristics, for example, tone/brightness, of theregions 402. For example, in dark regions such asregion 402A, thetone curve 404A may boost the dark area to provide a broader range. For bright regions such asregion 402B, thetone curve 404B may not boost as much to preserve highlights. - The tone curves 404 determined for the
regions 402 of animage 400 can be parameterized and stored, for example as metadata 232 as illustrated inFIG. 2 , and passed or otherwise provided to adisplay pipeline 260 so that a wider range of tone or luminance can be recovered in aninverse LTM technique 276 to generateHDR image data 278 from the inputSDR image data 230B on thedisplay pipeline 260. - In some embodiments, additional information may be saved as
metadata 232A for use in theinverse LTM technique 276 of thedisplay pipeline 260. For example, in some embodiments, theLTM technique 226 may clip at least some image data values, and themetadata 232A may include additional information that may be used in at least partially recovering the clipped values. - In some embodiments, information used in performing color gamut mapping on the
sensor pipeline 210 may also be stored asmetadata 232A and provided to thedisplay pipeline 260. The color gamut mapping information may be used in image reconstruction on thedisplay pipeline 260 to recover a wider color gamut for display. - In various embodiments, the
tone mapping metadata 232A can be compressed spatially (i.e., within animage 400 or frame) and/or temporally (e.g., across two ormore images 400 or frames). For example, within animage 400, the local tone curves 402 may not change much between adjacent ornearby regions 402, and thus thetone mapping metadata 232A may be compressed within the image 400 (spatially). In addition, the tone curves may not change much between thesame regions 402 in two or moreadjacent images 400 or frames, and thus the tone curve information in metadata 232 may be compressed temporally (across two ormore images 400 or frames). - Matching Tone Mapping to the Display
- In some embodiments, the inverse
tone mapping technique 276 applied by adisplay pipeline 260 may be tuned to match the dynamic range of a target display. The dynamic range captured by the sensor may, for example, cover a wider range than the dynamic range supported by the target display, even if the display is HDR-compatible. Thus, in some embodiments, the inversetone mapping technique 276 of thedisplay pipeline 260 may modify the tone mapping metadata 232B received with theSDR image data 230B according to the dynamic range of a target display, or may otherwise modify the inversetone mapping technique 276 applied to theSDR image data 230B, to generateHDR image data 278 that matches the dynamic range of the target display. - Ambient Adaptive Tone Mapping
- In some embodiments, the inverse
tone mapping technique 276 of the display pipeline may be an ambient adaptive process in which information about ambient light, distance to one or more viewers, or other ambient or environmental conditions at the target display may be obtained, e.g. using one or more sensors. Based on the obtained environmental information, the inversetone mapping technique 276 may be dynamically adjusted, for example by modifying or weighting one or more of the tone curves 404 as indicated in the metadata 232B for one ormore regions 402 of the video frames being processed in thedisplay pipeline 260. For example, if ambient light is detected to be bright at the display (e.g., above a specified threshold), the inversetone mapping technique 276 may adjust one or more of the tone curves 404 as indicated in the metadata 232B to generate a brighter image 400 (orbrighter regions 402 within an image 400) and/or animage 400 or region(s) 402 with more contrast. If ambient light is detected to be dim at the display (e.g., below a specified threshold), the inversetone mapping technique 276 may adjust one or more of the tone curves 404 as indicated in the metadata 232B to tone down or darken theimage 400 or region(s) 402 within theimage 400. - Inverse Tone Mapping Methods
- As previously mentioned, a display/decoding pipeline may apply an inverse tone mapping technique to recover at least part of the high dynamic range of the original video data as input to the sensor/encoding pipeline. In some embodiments, tone mapping may involve applying a transfer function (e.g., an electro-optical transfer function (EOTF)) to input video data values represented at a higher dynamic range according to a first color space (e.g., an RGB color space) to generate output video data represented at a lower dynamic range according to a second color space (e.g., a YUV color space). The transfer function may correspond to the tone curve (which may be referred to as a transfer curve), and may be represented by and stored in metadata as one or more transfer function parameter values. In some embodiments, the inverse tone mapping technique may involve applying an inverse of the transfer function as represented in the metadata to the decompressed video data obtained from the sensor/encoding pipeline.
- Color Gamut Recovery
- As previously mentioned, in some embodiments, in addition to recovering tone mapping information on the display side using metadata 232 as illustrated in
FIG. 2 , a wider color gamut as captured by a sensor may also be preserved on thesensor side 210 and at least partially recovered on thedisplay side 260 using metadata 232. In some embodiments, thesensor pipeline 210 may generate lower color gamut image data (e.g., YUV Rec. 709image data 230A) from wider color gamut image data (e.g., camera sensor RGB image data 220) captured by an image sensor of a camera associated with thesensor pipeline 210 using a color gamut mapping technique that may clip colors from the wider color gamut range of a color space (e.g., RGB color space) that aren't available in the lower color gamut of another color space (e.g., YUV color space). In some embodiments, in addition to tone mapping information, information used in or resulting from performing the color gamut mapping may be recorded in the metadata 232 and used by thedisplay pipeline 260 in image reconstruction to recover a wider color gamut for displays that support WCG imaging by at least partially restoring the clipped colors. - Providing SDR Video
-
FIG. 3 illustrates an example video pipeline providing standard dynamic range (SDR) video from source high dynamic range (HDR) content, according to some embodiments. InFIG. 3 , avideo pipeline 300 may generate and output video data in different formats by encoding camerasensor RGB data 320 to generate YUV, 10+ bitdepth image data 330A for output and transcoding the YUV, 10+ bitdepth image data 330A to generate sRGB, Rec. 709image data 382 for output. - As shown in
FIG. 3 , in avideo pipeline 300, a camera sensor may capture images or frames, shown as HDRcamera sensor RGB 320 data. An image signal processor (ISP) 322 may process the captured camerasensor RGB data 320 to generate HDR linearRGB image data 324 as output. A tone mapping technique 326 (e.g., a local (LTM) and/or global (GTM) tone mapping technique as described herein) may be applied to the linearRGB image data 324 to convert the HDR linearRGB image data 324 to standard dynamic range (SDR) image data in the YUV color space represented at a bit depth of 10 or more bits (YUV, 10+ bitdepth image data 330A). In some embodiments, color gamut mapping may be applied to the image data to map the colors from one color space (e.g., RGB) to another (e.g., YUV). The YUV, 10+ bitdepth image data 330A may be compressed 334. Thevideo pipeline 300 may record information used to perform thetone mapping 326 and/or color gamut mapping asmetadata 332A. The compressed YUV, 10+ bitdepth image data 330A andmetadata 332A may be output, for example via anetwork 350 to one or more target devices. At a target device, the YUV, 10+ bitdepth image data 330A may be decompressed and processed, and the metadata 332 may be used in an inverse tone mapping technique and/or gamut mapping technique to recover at least some of the dynamic range and/or color for HDR-enabled displays. - In some embodiments, the compressed YUV, 10+ bit
depth image data 330A andmetadata 332A may instead or also be sent on atranscoding path 340 to be transcoded into YUV Rect. 709image data 382 for output, for example via anetwork 350 to one or more target devices. On thetranscoding path 340, the compressed YUV, 10+ bitdepth image data 330A may be decompressed 370 to generate compressed YUV, 10+ bitdepth image data 330B. Themetadata 332B may be used in performing an inverse global and/or local tone mapping andcolor conversion operation 376 on theimage data 330B to generateHDR image data 378. Theinverse operation 376 may recover or reconstruct at least a portion of the luminance and/or color inimage data 378 according to themetadata 332B. Another global and/or localtone mapping operation 380 may then be applied to theHDR image data 378 to generate sRGB, Rec. 709image data 382 for output, for example via anetwork 350 to one or more target devices. - In some embodiments,
metadata 332B may not be sent on thetranscoding path 340. In some embodiments, instead of applying inverse global and/or local tone mapping andcolor conversion operation 376 on the decompressedimage data 330B to generateHDR image data 378, and then applying anothertone mapping operation 380 to convert theHDR image data 378 to Rec. 709image data 382, thetranscoding path 340 may apply a tone mapping and color conversion technique directly to the decompressedimage data 330B to directly generate the Rec. 709image data 382 fromimage data 330B. In some embodiments, Digital Cinema Initiatives (DCI) P3 may be used as a base layer rather than Rec. 709. In various embodiments, thetone mapping 326 can be performed in RGB color space or in YUV color space, or in other color spaces if available. - Example Encoder/Decoder Implementation
- Example embodiments of video display and playback methods and apparatus for supporting high dynamic range (HDR) and wide color gamut (WCG) video processing, recovery, and display are described. In particular, embodiments of methods and apparatus for mapping video to target display panels using encoders and decoders are described. In some embodiments, at least some of the functionality as described herein may be implemented by one or more components or modules of a system on a chip (SOC) that may be used in devices including but not limited to multifunction devices, smartphones, pad or tablet devices, and other portable computing devices such as laptop, notebook, and netbook computers.
FIG. 10 illustrates an example SOC, andFIG. 11 illustrates an example device implementing an SOC. - In the methods and apparatus as illustrated in
FIGS. 8 and 9 , mapping to a target display panel is performed at least in part on the decoder side, with gamut mapping metadata generated on the encoder side and passed to a component on the decoder side used in recovering WCG video content. In some embodiments, the methods and apparatus ofFIGS. 8 and 9 may also include or implement the methods as shown inFIGS. 2 through 7 to provide both standard dynamic range (SDR) and high dynamic range (HDR) versions of video to display devices from a HDR video source. -
FIG. 9 is a flowchart of a video playback method in which an encoder generates output data in an intermediate color gamut and mapping to a display color gamut is performed by a decoder according to metadata generated by the encoder, according to some embodiments. Referring toFIG. 9 , as indicated at 1450, an encoding pipeline maps video content from a wide color gamut to an intermediate color gamut. This intermediate color gamut is not necessarily the full range gamut of the source content on the encoding side, but is a color gamut that may lie between the full range color gamut and the color gamut of one or more target displays. In some embodiments, the intermediate color gamut is represented at a bit depth (e.g., 10 bits) that is compatible with typical encoder/decoder hardware bit depth. As indicated at 1452 ofFIG. 9 , the encoding pipeline then encodes the video content according to a compressed video format, for example H.264/AVC format. Encoding the video data may involve converting the video data from one color space to another color space, for example from CIE 1931 XYZ color space to a YUV Rec. 709 color space. As indicated at 1454 ofFIG. 9 , the encoding pipeline records metadata describing at least the color gamut mapping that was performed. - As indicated at 1456 of
FIG. 9 , one or more decoding pipelines may each obtain the encoded video content and the metadata. As indicated at 1458 ofFIG. 9 , each decoding pipeline decodes the encoded video content to generate decoded (uncompressed) video content at the intermediate color gamut. Decoding the video data may involve converting the video data from one color space to another color space, for example from YUV to XYZ. As indicated at 1460, each decoding pipeline then maps the decoded video content from the intermediate color gamut to the color gamut of a respective target display panel according to the metadata. Mapping the video data may involve converting the video data from one color space to another color space, for example from XYZ to a YCC or YUV color space. The decoded video content may be further processed and provided to the respective display panel for display. Elements of the method ofFIG. 9 are further described below in reference toFIG. 8 . -
FIG. 8 illustrates an example video playback system in which an encoder generates output data in an intermediate color gamut and in which mapping to a display color gamut is performed on the decoder/display side using metadata generated by the encoder, according to some embodiments. The example video playback method and system may involve H.264/AVC encoding in a wider or intermediate color gamut for distribution to displays, with decoder-side mapping to the display color gamut. -
FIG. 8 shows anencoding 800 module or pipeline including one or more components and adecoding 850 module or pipeline including one or more components. In some embodiments, one or both ofencoding 800 anddecoding 850 may be implemented on an SOC. In some embodiments, encoding 800 anddecoding 850 may be implemented on the same device and/or SOC. In some embodiments, encoding 800 anddecoding 850 may be implemented on different devices or SOCs. In some embodiments, one ormore encoding 800 pipelines may be implemented on a device or system; theencoding 800 pipelines may be configured to encode and stream video to one or more target devices or systems each implementing at least onedecoding 850 pipeline. - In this embodiment, color gamut mapping is performed at least in part on the
decoding 850 side, with a video encoded stream (VES) 810 in a wide color gamut (WCG) andgamut mapping metadata 820 generated on theencoding 800 side and passed to a component on thedecoding 850 side (e.g., to anISP color pipe 854, or alternatively to a GPU) for use indecoding 850 side gamut mapping operations. The input video content may, for example, be encoded in (linear) CIE 1931 XYZ color space at a bit depth of 16 bits. Amapping component 802 may apply a 10-bit electro-optical transfer function (EOTF) operation to the input linear XYZ video to map the 16-bit input data to 10-bit log XYZ video data. In some embodiments, the EOTF may be a transfer function that maps to a color space that is wide enough to transfer to all target display panel color spaces.Mapping component 802 may also generatemetadata 820 describing the mapping operation. An H.264encoder component 804 encodes the 10-bit log XYZ video data to generate wide color gamut (WCG) H.264compressed video 810 at a bit depth of 10 bits. - At decoding 850, an H.264
decode component 852 decodes the H.264compressed video 810 to generate 10-bit data in the XYZ color space. An Image Signal Processor (ISP) 854 color pipe, or alternatively a Graphics Processor Unit (GPU), may then be used to perform gamut mapping from the WCG of the 10-bit XYZ data to a display gamut according to themetadata 820. TheISP 854 may generate 10-bit data in a YCC color space. Asuper-resolution technique 856 may be performed on the data, and the 10-bit YCC data may then be passed to adisplay pipe 858 for processing into display output data, for example 10-bit RGB data, at the color gamut of the display. - In this embodiment, encoding is done in a wider color gamut that fits into a bit depth supported by encoder/decoder hardware (e.g., 10 bits), and
metadata 820 is passed to thedecoding 850 to be used in color gamut mapping to a display gamut. FIG. 8 shows operations in thedecoding 850 side that may leverage existing components of an SOC such as an H.264 decoder, ISP color pipe, and display pipe. Decoding may be performed at a bit depth of 10 bits, and conversion to the color gamut of the display may be performed using themetadata 820 information that describes how the gamut mapping should be done that is received from theencoding 800 side. - In
FIG. 8 , some, but not all, of the color gamut processing is performed on the playback/decoding 850 side rather than on the server/encoding 800 side. Instead of mapping directly into the target display color gamut on theencoding 800 side, the content is mapped into an intermediate color gamut. This intermediate color gamut is not necessarily the full range gamut of the source content on theencoding 800 side, but is a color gamut that may lie between the full range color gamut and the display gamut which can fit into 10-bit and that allows existingdecoding 850 hardware that can support up to 10-bit decoding to be used. At decoding 850, themetadata 820 may be used to map to the gamut of the target panel/display. In some embodiments, this mapping may be performed in two steps. As a first step, theinput video content 810 is mapped back to the original wider gamut according to themetadata 820. This re-mapped content is then mapped down to the gamut of the display. In some embodiments, the playback/decoding 850 side gamut mapping may be performed on, by, or in an Image Signal Processor (ISP)color pipe 854 which may, for example, perform gamma correction, color space conversion, etc. In some embodiments, one or more components of the ISP color pipe 854 (e.g., 3D color lookup tables (CLUTS)) may be used in performing gamut mapping. However, the playback/decoding 850 gamut mapping may instead or in addition be performed by or in one or more GPUs. - Embodiments of a
decoding 850 pipeline as shown inFIG. 8 may perform an initial decode with 10-bit data using adecoder 852 component, and may restore at least some of the color gamut based onmetadata 820 received from encoding 800 either in anISP color pipe 854 or in a GPU. Theencoding 800 generates output encodedvideo stream 810 at a wider, intermediate color gamut that may attempt to capture a range of display gamut(s) that thevideo stream 810 could be targeted to. Themetadata 820 generated by theencoding 800 and passed to one or more decoder(s) 850 defines how to map from the intermediate color gamut of theoutput stream 810 to the potentially narrower display gamuts. In some embodiments, themetadata 820 may reference or map into one or more lookup tables (e.g., CLUTS) that define how to map from and to different color gamuts. For example, one or more lookup tables referenced by themetadata 820 may be used by anISP color pipe 854 orother decoding 850 component to map to a Rec. 709 gamut, to a P3 DCI D65, to a Rec. 2020 gamut, and so on. Alternatively, themetadata 820 may include equations and/or parameters (e.g., parameters for configuring adisplay pipe 858,ISP color pipe 854, and/or GPU(s)) for performing the decoder-side gamut mapping. - In some embodiments,
metadata 820 may also include information used in a tone mapping technique (e.g., a local (LTM) and/or global (GTM) tone mapping technique) applied by the encoding 800 (e.g., by mapping 802 component) to map the input video data to a dynamic range of theoutput data stream 810. Themetadata 820 may then be used by an inverse tone mapping technique in thedecoding 850 pipeline to recover at least some dynamic range that may have been clipped in the tone mapping applied by theencoding 800. In some embodiments, a tone mapping module or component of an ISP color pipe may be used for tone mapping on theencoding 800 and/or decoding 850 side. In some embodiments, a 3D color lookup table (CLUT) of the ISP color pipe may be used for global and/or local tone mapping and for gamut mapping. - Note that the various video formats, color spaces, and so on shown in
FIG. 8 are given by way of example and are not intended to be limiting. For example, other cOor spaces than CIE 1931 XYZ may be used for input video. As another example, in some embodiments, encoding and decoding may be performed according to the H.265 High Efficiency Video Coding (HEVC) format instead of the H.264/AVC format. - Example Devices and Apparatus
-
FIGS. 10 through 14 show non-limiting examples of devices and apparatus in or with which embodiments or components of the various digital video or image processing and display methods and apparatus as described herein may be implemented.FIG. 10 illustrates an example SOC, andFIG. 11 illustrates an example device implementing an SOC.FIG. 12 illustrates an example computer system that may implement the methods and apparatus described herein.FIGS. 13 and 14 illustrate example multifunction devices that may implement the methods and apparatus described herein. - Example System on a Chip (SOC)
- Turning now to
FIG. 10 , a block diagram of one embodiment of a system-on-a-chip (SOC) 8000 that may be used in embodiments.SOC 8000 is shown coupled to amemory 8800. As implied by the name, the components of theSOC 8000 may be integrated onto a single semiconductor substrate as an integrated circuit “chip.” In some embodiments, the components may be implemented on two or more discrete chips in a system. However, theSOC 8000 will be used as an example herein. In the illustrated embodiment, the components of theSOC 8000 include a central processing unit (CPU) complex 8020, on-chipperipheral components 8040A-8040C (more briefly, “peripherals”), a memory controller (MC) 8030, and acommunication fabric 8010. Thecomponents communication fabric 8010. Thememory controller 8030 may be coupled to thememory 8800 during use, and the peripheral 8040B may be coupled to anexternal interface 8900 during use. In the illustrated embodiment, theCPU complex 8020 includes one or more processors (P) 8024 and a level two (L2)cache 8022. - The
peripherals 8040A-8040B may be any set of additional hardware functionality included in theSOC 8000. For example, theperipherals 8040A-8040B may include video peripherals such as an image signal processor configured to process image capture data from a camera or other image sensor, display controllers configured to display video data on one or more display devices, graphics processing units (GPUs), video encoder/decoders or codecs, scalers, rotators, blenders, etc. The peripherals may include audio peripherals such as microphones, speakers, interfaces to microphones and speakers, audio processors, digital signal processors, mixers, etc. The peripherals may include peripheral interface controllers forvarious interfaces 8900 external to the SOC 8000 (e.g. the peripheral 8040B) including interfaces such as Universal Serial Bus (USB), peripheral component interconnect (PCI) including PCI Express (PCIe), serial and parallel ports, etc. The peripherals may include networking peripherals such as media access controllers (MACs). Any set of hardware may be included. - The
CPU complex 8020 may include one ormore CPU processors 8024 that serve as the CPU of theSOC 8000. The CPU of the system includes the processor(s) that execute the main control software of the system, such as an operating system. Generally, software executed by the CPU during use may control the other components of the system to realize the desired functionality of the system. Theprocessors 8024 may also execute other software, such as application programs. The application programs may provide user functionality, and may rely on the operating system for lower level device control. Accordingly, theprocessors 8024 may also be referred to as application processors. TheCPU complex 8020 may further include other hardware such as theL2 cache 8022 and/or and interface to the other components of the system (e.g. an interface to the communication fabric 8010). Generally, a processor may include any circuitry and/or microcode configured to execute instructions defined in an instruction set architecture implemented by the processor. The instructions and data operated on by the processors in response to executing the instructions may generally be stored in thememory 8800, although certain instructions may be defined for direct processor access to peripherals as well. Processors may encompass processor cores implemented on an integrated circuit with other components as a system on a chip (SOC 8000) or other levels of integration. Processors may further encompass discrete microprocessors, processor cores and/or microprocessors integrated into multichip module implementations, processors implemented as multiple integrated circuits, etc. - The
memory controller 8030 may generally include the circuitry for receiving memory operations from the other components of theSOC 8000 and for accessing thememory 8800 to complete the memory operations. Thememory controller 8030 may be configured to access any type ofmemory 8800. For example, thememory 8800 may be static random access memory (SRAM), dynamic RAM (DRAM) such as synchronous DRAM (SDRAM) including double data rate (DDR, DDR2, DDR3, etc.) DRAM. Low power/mobile versions of the DDR DRAM may be supported (e.g. LPDDR, mDDR, etc.). Thememory controller 8030 may include queues for memory operations, for ordering (and potentially reordering) the operations and presenting the operations to thememory 8800. Thememory controller 8030 may further include data buffers to store write data awaiting write to memory and read data awaiting return to the source of the memory operation. In some embodiments, thememory controller 8030 may include a memory cache to store recently accessed memory data. In SOC implementations, for example, the memory cache may reduce power consumption in the SOC by avoiding re-access of data from thememory 8800 if it is expected to be accessed again soon. In some cases, the memory cache may also be referred to as a system cache, as opposed to private caches such as theL2 cache 8022 or caches in theprocessors 8024, which serve only certain components. Additionally, in some embodiments, a system cache need not be located within thememory controller 8030. - In an embodiment, the
memory 8800 may be packaged with theSOC 8000 in a chip-on-chip or package-on-package configuration. A multichip module configuration of theSOC 8000 and thememory 8800 may be used as well. Such configurations may be relatively more secure (in terms of data observability) than transmissions to other components in the system (e.g. to the end points 16A-16B). Accordingly, protected data may reside in thememory 8800 unencrypted, whereas the protected data may be encrypted for exchange between theSOC 8000 and external endpoints. - The
communication fabric 8010 may be any communication interconnect and protocol for communicating among the components of theSOC 8000. Thecommunication fabric 8010 may be bus-based, including shared bus configurations, cross bar configurations, and hierarchical buses with bridges. Thecommunication fabric 8010 may also be packet-based, and may be hierarchical with bridges, cross bar, point-to-point, or other interconnects. - It is noted that the number of components of the SOC 8000 (and the number of subcomponents for those shown in
FIG. 10 , such as within the CPU complex 8020) may vary from embodiment to embodiment. There may be more or fewer of each component/subcomponent than the number shown inFIG. 10 . -
FIG. 11 a block diagram of one embodiment of asystem 9000 that includes at least one instance of anSOC 8000 coupled to one or moreexternal peripherals 9020 and theexternal memory 8800. A power management unit (PMU) 9010 is provided which supplies the supply voltages to theSOC 8000 as well as one or more supply voltages to thememory 8800 and/or theperipherals 9020. In some embodiments, more than one instance of theSOC 8000 may be included (and more than onememory 8800 may be included as well). - The
peripherals 9020 may include any desired circuitry, depending on the type ofsystem 9000. For example, in one embodiment, thesystem 9000 may be a mobile device (e.g. personal digital assistant (PDA), smart phone, etc.) and theperipherals 9020 may include devices for various types of wireless communication, such as wifi, Bluetooth, cellular, global positioning system, etc. Theperipherals 9020 may also include additional storage, including RAM storage, solid state storage, or disk storage. Theperipherals 9020 may include user interface devices such as a display screen, including touch display screens or multitouch display screens, keyboard or other input devices, microphones, speakers, etc. In other embodiments, thesystem 9000 may be any type of computing system (e.g. desktop personal computer, laptop, workstation, net top etc.). - The
external memory 8800 may include any type of memory. For example, theexternal memory 8800 may be SRAM, dynamic RAM (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM, RAMBUS DRAM, low power versions of the DDR DRAM (e.g. LPDDR, mDDR, etc.), etc. Theexternal memory 8800 may include one or more memory modules to which the memory devices are mounted, such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc. Alternatively, theexternal memory 8800 may include one or more memory devices that are mounted on theSOC 8000 in a chip-on-chip or package-on-package implementation. - Example Computer System
-
FIG. 12 illustrates anexample computer system 2900 that may be configured to execute any or all of the embodiments described above. In different embodiments,computer system 2900 may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, tablet, slate, pad, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, a consumer device, application server, storage device, a video recording device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device. - Various embodiments as described herein, may be executed in one or
more computer systems 2900, which may interact with various other devices. Note that any component, action, or functionality described above with respect toFIGS. 1-11 may be implemented on one or more computers configured ascomputer system 2900 ofFIG. 12 , according to various embodiments. In the illustrated embodiment,computer system 2900 includes one or more processors 2910 coupled to asystem memory 2920 via an input/output (I/O)interface 2930.Computer system 2900 further includes anetwork interface 2940 coupled to I/O interface 2930, and one or more input/output devices orcomponents 2950, such ascursor control 2960,keyboard 2970, display(s) 2980, camera(s) 2990, and sensor(s) 2992 including but not limited to light sensors and motion detectors. In some cases, it is contemplated that embodiments may be implemented using a single instance ofcomputer system 2900, while in other embodiments multiple such systems, or multiple nodes making upcomputer system 2900, may be configured to host different portions or instances of embodiments. For example, in one embodiment some elements may be implemented via one or more nodes ofcomputer system 2900 that are distinct from those nodes implementing other elements. - In various embodiments,
computer system 2900 may be a uniprocessor system including one processor 2910, or a multiprocessor system including several processors 2910 (e.g., two, four, eight, or another suitable number). Processors 2910 may be any suitable processor capable of executing instructions. For example, in various embodiments processors 2910 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x829, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors 2910 may commonly, but not necessarily, implement the same ISA. -
System memory 2920 may be configured to storeprogram instructions 2922 and/or data accessible by processor 2910. In various embodiments,system memory 2920 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment,program instructions 2922 may be configured to implement any of the functionality described herein. Additionally,memory 2920 may include any of the information or data structures described herein. In some embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate fromsystem memory 2920 orcomputer system 2900. Whilecomputer system 2900 is described as implementing the functionality of functional blocks of previous Figures, any of the functionality described herein may be implemented via such a computer system. - In one embodiment, I/
O interface 2930 may be configured to coordinate I/O traffic between processor 2910,system memory 2920, and any peripheral devices in the device, includingnetwork interface 2940 or other peripheral interfaces, such as input/output devices 2950. In some embodiments, I/O interface 2930 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 2920) into a format suitable for use by another component (e.g., processor 2910). In some embodiments, I/O interface 2930 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface 2930 may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface 2930, such as an interface tosystem memory 2920, may be incorporated directly into processor 2910. -
Network interface 2940 may be configured to allow data to be exchanged betweencomputer system 2900 and other devices attached to a network 2985 (e.g., carrier or agent devices) or between nodes ofcomputer system 2900.Network 2985 may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments,network interface 2940 may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol. - Input/
output devices 2950 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by one ormore computer systems 2900. Multiple input/output devices 2950 may be present incomputer system 2900 or may be distributed on various nodes ofcomputer system 2900. In some embodiments, similar input/output devices may be separate fromcomputer system 2900 and may interact with one or more nodes ofcomputer system 2900 through a wired or wireless connection, such as overnetwork interface 2940. - As shown in
FIG. 12 ,memory 2920 may includeprogram instructions 2922, which may be processor-executable to implement any element or action described above. In one embodiment, the program instructions may implement the methods described above. In other embodiments, different elements and data may be included. Note that data may include any data or information described above. - Those skilled in the art will appreciate that
computer system 2900 is merely illustrative and is not intended to limit the scope of embodiments. In particular, the computer system and devices may include any combination of hardware or software that can perform the indicated functions, including computers, network devices, Internet appliances, PDAs, wireless phones, pagers, etc.Computer system 2900 may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system. In addition, the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available. - Those skilled in the art will also appreciate that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components may execute in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or a portable article to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from
computer system 2900 may be transmitted tocomputer system 2900 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link. Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Generally speaking, a computer-accessible medium may include a non-transitory, computer-readable storage medium or memory medium such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link. - Multifunction Device Examples
-
FIG. 13 illustrates a block diagram of a portable multifunction device in accordance with some embodiments. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA, camera, video capture and/or playback, and/or music player functions. Example embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as laptops, cell phones, smartphones, pad or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and/or touch pads), may also be used. It should also be understood that, in some embodiments, the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad). In some embodiments, the device is a gaming computer with orientation sensors (e.g., orientation sensors in a gaming controller). In other embodiments, the device is not a portable communications device, but is a camera and/or video camera. - In the discussion that follows, an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.
- The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
- The various applications that may be executed on the device may use at least one common physical user-interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device may be adjusted and/or varied from one application to the next and/or within a respective application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device may support the variety of applications with user interfaces that are intuitive and transparent to the user.
-
Device 2100 may include memory 2102 (which may include one or more computer readable storage mediums),memory controller 2122, one or more processing units (CPU's) 2120,peripherals interface 2118,RF circuitry 2108,audio circuitry 2110,speaker 2111, touch-sensitive display system 2112,microphone 2113, input/output (I/O)subsystem 2106, otherinput control devices 2116, andexternal port 2124.Device 2100 may include one or more optical sensors orcameras 2164. These components may communicate over one or more communication buses orsignal lines 2103. - It should be appreciated that
device 2100 is only one example of a portable multifunction device, and thatdevice 2100 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of the components. The various components shown inFIG. 13 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits. -
Memory 2102 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access tomemory 2102 by other components ofdevice 2100, such asCPU 2120 and theperipherals interface 2118, may be controlled bymemory controller 2122. - Peripherals interface 2118 can be used to couple input and output peripherals of the device to
CPU 2120 andmemory 2102. The one ormore processors 2120 run or execute various software programs and/or sets of instructions stored inmemory 2102 to perform various functions fordevice 2100 and to process data. - In some embodiments,
peripherals interface 2118,CPU 2120, andmemory controller 2122 may be implemented on a single chip, such aschip 2104. In some other embodiments, they may be implemented on separate chips. - RF (radio frequency)
circuitry 2108 receives and sends RF signals, also called electromagnetic signals.RF circuitry 2108 converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals.RF circuitry 2108 may include well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder/decoder (codec) chipset, a subscriber identity module (SIM) card, memory, and so forth.RF circuitry 2108 may communicate with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication. The wireless communication may use any of a variety of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSDPA), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document. -
Audio circuitry 2110,speaker 2111, andmicrophone 2113 provide an audio interface between a user anddevice 2100.Audio circuitry 2110 receives audio data fromperipherals interface 2118, converts the audio data to an electrical signal, and transmits the electrical signal tospeaker 2111.Speaker 2111 converts the electrical signal to human-audible sound waves.Audio circuitry 2110 also receives electrical signals converted bymicrophone 2113 from sound waves.Audio circuitry 2110 converts the electrical signal to audio data and transmits the audio data toperipherals interface 2118 for processing. Audio data may be retrieved from and/or transmitted tomemory 2102 and/orRF circuitry 2108 byperipherals interface 2118. In some embodiments,audio circuitry 2110 also includes a headset jack. The headset jack provides an interface betweenaudio circuitry 2110 and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone). - I/
O subsystem 2106 couples input/output peripherals ondevice 2100, such astouch screen 2112 and otherinput control devices 2116, toperipherals interface 2118. I/O subsystem 2106 may includedisplay controller 2156 and one ormore input controllers 2160 for otherinput control devices 2116. The one ormore input controllers 2160 receive/send electrical signals from/to otherinput control devices 2116. The otherinput control devices 2116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 2160 may be coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons may include an up/down button for volume control ofspeaker 2111 and/ormicrophone 2113. The one or more buttons may include a push button. - Touch-
sensitive display 2112 provides an input interface and an output interface between the device and a user.Display controller 2156 receives and/or sends electrical signals from/totouch screen 2112.Touch screen 2112 displays visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some embodiments, some or all of the visual output may correspond to user-interface objects. -
Touch screen 2112 has a touch-sensitive surface, sensor or set of sensors that accepts input from the user based on haptic and/or tactile contact.Touch screen 2112 and display controller 2156 (along with any associated modules and/or sets of instructions in memory 2102) detect contact (and any movement or breaking of the contact) ontouch screen 2112 and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed ontouch screen 2112. In an example embodiment, a point of contact betweentouch screen 2112 and the user corresponds to a finger of the user. -
Touch screen 2112 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other embodiments.Touch screen 2112 anddisplay controller 2156 may detect contact and any movement or breaking thereof using any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact withtouch screen 2112. In an example embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, Calif. -
Touch screen 2112 may have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user may make contact withtouch screen 2112 using any suitable object or appendage, such as a stylus, a finger, and so forth. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user. - In some embodiments, in addition to the
touch screen 2112,device 2100 may include a touchpad (not shown) for activating or deactivating particular functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad may be a touch-sensitive surface that is separate fromtouch screen 2112 or an extension of the touch-sensitive surface formed by the touch screen. -
Device 2100 also includespower system 2162 for powering the various components.Power system 2162 may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices. -
Device 2100 may also include one or more optical sensors orcameras 2164.FIG. 13 shows an optical sensor coupled to optical sensor controller 2158 in I/O subsystem 2106.Optical sensor 2164 may, for example, include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors or photosensors.Optical sensor 2164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module 2143 (also called a camera module),optical sensor 2164 may capture still images and/or video sequences. In some embodiments, at least one optical sensor may be located on the back ofdevice 2100, oppositetouch screen display 2112 on the front of the device. In some embodiments, the touch screen display may be used as a viewfinder for still and/or video image acquisition. In some embodiments, at least one optical sensor may instead or also be located on the front of the device. -
Device 2100 may also include one ormore proximity sensors 2166.FIG. 13 showsproximity sensor 2166 coupled toperipherals interface 2118. Alternately,proximity sensor 2166 may be coupled toinput controller 2160 in I/O subsystem 2106. In some embodiments, the proximity sensor turns off and disablestouch screen 2112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call). -
Device 2100 may also include one ormore orientation sensors 2168. In some embodiments, the one or more orientation sensors include one or more accelerometers (e.g., one or more linear accelerometers and/or one or more rotational accelerometers). In some embodiments, the one or more orientation sensors include one or more gyroscopes. In some embodiments, the one or more orientation sensors include one or more magnetometers. In some embodiments, the one or more orientation sensors include one or more of global positioning system (GPS), Global Navigation Satellite System (GLONASS), and/or other global navigation system receivers. The GPS, GLONASS, and/or other global navigation system receivers may be used for obtaining information concerning the location and orientation (e.g., portrait or landscape) ofdevice 2100. In some embodiments, the one or more orientation sensors include any combination of orientation/rotation sensors.FIG. 13 shows the one ormore orientation sensors 2168 coupled toperipherals interface 2118. Alternately, the one ormore orientation sensors 2168 may be coupled to aninput controller 2160 in I/O subsystem 2106. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on an analysis of data received from the one or more orientation sensors. - In some embodiments,
device 2100 may also include one or more other sensors (not shown) including but not limited to ambient light sensors and motion detectors. These sensors may be coupled toperipherals interface 2118 or, alternately, may be coupled to aninput controller 2160 in I/O subsystem 2106. For example, in some embodiments,device 2100 may include at least one forward-facing (away from the user) and at least one backward-facing (towards the user) light sensors that may be used to collect ambient lighting metrics from the environment of thedevice 2100 for use in video and image capture, processing, and display applications. - In some embodiments, the software components stored in
memory 2102 include operating system 2126, communication module 2128, contact/motion module (or set of instructions) 2130, graphics module 2132, text input module 2134, Global Positioning System (GPS) module 2135, and applications 2136. Furthermore, in someembodiments memory 2102 stores device/globalinternal state 2157. Device/globalinternal state 2157 includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, views or other information occupy various regions oftouch screen display 2112; sensor state, including information obtained from the device's various sensors andinput control devices 2116; and location information concerning the device's location and/or attitude. - Operating system 2126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
- Communication module 2128 facilitates communication with other devices over one or more
external ports 2124 and also includes various software components for handling data received byRF circuitry 2108 and/orexternal port 2124. External port 2124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with the 30-pin connector used on iPod (trademark of Apple Inc.) devices. - Contact/motion module 2130 may detect contact with touch screen 2112 (in conjunction with display controller 2156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module 2130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module 2130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, may include determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some embodiments, contact/motion module 2130 and
display controller 2156 detect contact on a touchpad. - Contact/motion module 2130 may detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Thus, a gesture may be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.
- Graphics module 2132 includes various software components for rendering and displaying graphics on
touch screen 2112 or other display, including components for changing the intensity of graphics that are displayed. As used herein, the term “graphics” includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like. - In some embodiments, graphics module 2132 stores data representing graphics to be used. Each graphic may be assigned a corresponding code. Graphics module 2132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display
controller 2156. - Text input module 2134, which may be a component of graphics module 2132, provides soft keyboards for entering text in various applications that need text input.
- GPS module 2135 determines the location of the device and provides this information for use in various applications (e.g., to
telephone module 2138 for use in location-based dialing, to camera module 2143 as picture/video metadata, and to applications that provide location-based services such as map/navigation applications). - Applications 2136 may include one or more of, but are not limited to, the following modules (or sets of instructions), or a subset or superset thereof:
-
-
telephone module 2138; - video conferencing module 2139;
- camera module 2143 for still and/or video imaging;
- image management module 2144;
-
browser module 2147; -
search module 2151; - video and music player module 2152, which may be made up of a video player module and a music player module; and/or
- online video module 2155.
-
- Examples of other applications 2136 that may be stored in
memory 2102 include but are not limited to other word processing applications, other image editing applications, drawing applications, presentation applications, communication/social media applications, map applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication. - In conjunction with
RF circuitry 2108,audio circuitry 2110,speaker 2111,microphone 2113,touch screen 2112,display controller 2156, contact module 2130, graphics module 2132, and text input module 2134,telephone module 2138 may be used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in an address book, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed. As noted above, the wireless communication may use any of a variety of communications standards, protocols and technologies. - In conjunction with
RF circuitry 2108,audio circuitry 2110,speaker 2111,microphone 2113,touch screen 2112,display controller 2156,optical sensor 2164, optical sensor controller 2158, contact/motion module 2130, graphics module 2132, text input module 2134, andtelephone module 2138, videoconferencing module 2139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions. - In conjunction with
touch screen 2112,display controller 2156, optical sensor(s) 2164, optical sensor controller 2158, contact/motion module 2130, graphics module 2132, and image management module 2144, camera module 2143 includes executable instructions to capture still images or video (including a video stream) and store them intomemory 2102, modify characteristics of a still image or video, or delete a still image or video frommemory 2102. - In conjunction with
touch screen 2112,display controller 2156, contact/motion module 2130, graphics module 2132, text input module 2134, and camera module 2143, image management module 2144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images. - In conjunction with
RF circuitry 2108,touch screen 2112,display system controller 2156, contact/motion module 2130, graphics module 2132, and text input module 2134,browser module 2147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages. - In conjunction with
touch screen 2112,display system controller 2156, contact/motion module 2130, graphics module 2132, and text input module 2134,search module 2151 includes executable instructions to search for text, music, sound, image, video, and/or other files inmemory 2102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions. - In conjunction with
touch screen 2112,display system controller 2156, contact/motion module 2130, graphics module 2132,audio circuitry 2110,speaker 2111,RF circuitry 2108, andbrowser module 2147, video and music player module 2152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., ontouch screen 2112 or on an external, connected display via external port 2124). In some embodiments,device 2100 may include the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.). - In conjunction with
touch screen 2112,display system controller 2156, contact/motion module 2130, graphics module 2132,audio circuitry 2110,speaker 2111,RF circuitry 2108, text input module 2134, andbrowser module 2147, online video module 2155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port 2124), and otherwise manage online videos in one or more video formats, such as the H.264/AVC format or the H.265/HEVC format. - Each of the above identified modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments,
memory 2102 may store a subset of the modules and data structures identified above. Furthermore,memory 2102 may store additional modules and data structures not described above. - In some embodiments,
device 2100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using a touch screen and/or a touchpad as the primary input control device for operation ofdevice 2100, the number of physical input control devices (such as push buttons, dials, and the like) ondevice 2100 may be reduced. - The predefined set of functions that may be performed exclusively through a touch screen and/or a touchpad include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates
device 2100 to a main, home, or root menu from any user interface that may be displayed ondevice 2100. In such embodiments, the touchpad may be referred to as a “menu button.” In some other embodiments, the menu button may be a physical push button or other physical input control device instead of a touchpad. -
FIG. 14 illustrates aportable multifunction device 2100 having atouch screen 2112 in accordance with some embodiments. The touch screen may display one or more graphics within user interface (UI) 2200. In at least some embodiments of adevice 2100, a user may select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 2202 (not drawn necessarily drawn to scale in the figure) or one or more styluses 2203 (not necessarily drawn to scale in the figure). -
Device 2100 may also include one or more physical buttons, such as a “home” ormenu button 2204. As described previously,menu button 2204 may be used to navigate to any application 2136 in a set of applications that may be executed ondevice 2100. Alternatively, in some embodiments, the menu button is may be implemented as a soft key in a GUI displayed ontouch screen 2112. - In one some embodiments,
device 2100 includestouch screen 2112, home ormenu button 2204,push button 2206 for powering the device on/off and locking the device, volume adjustment button(s) 2208, Subscriber Identity Module (SIM)card slot 2210, head setjack 2212, and docking/chargingexternal port 2124.Push button 2206 may be used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process. In an alternative embodiment,device 2100 also may accept verbal input for activation or deactivation of some functions throughmicrophone 2113. -
Device 2100 may also include one ormore cameras 2164. Acamera 2164 may, for example, include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors or photosensors. Acamera 2164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image or video frame. In some embodiments, at least onecamera 2164 may be located on the back ofdevice 2100, oppositetouch screen display 2112 on the front of the device. In some embodiments, at least onecamera 2164 may instead or also located on the front of the device with thetouch screen display 2112, for example so that the user's image may be obtained for videoconferencing while the user views the other video conference participants on thetouch screen display 2112. In some embodiments, at least onecamera 2164 may be located on the front of thedevice 2100, and at least onecamera 2164 may be located on the back of thedevice 2100. In some embodiments, thetouch screen display 2112 may be used as a viewfinder and/or user interface for still image and/or video sequence acquisition applications. -
Device 2100 may include video and image processing hardware and/or software, including but not limited to video encoding and/or decoding components, codecs, modules, or pipelines, that may be used to capture, process, convert, compress, decompress, store, modify, transmit, display, and otherwise manage and manipulate still images and/or video frames or video sequences captured viacamera 2164 or otherwise acquired (e.g., via a network interface). In some embodiments,device 2100 may also include one or more light or other sensors that may be used to collect ambient lighting or other metrics from the environment of thedevice 2100 for use in video and image capture, processing, and display. - The methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments. In addition, the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of claims that follow. Finally, structures and functionality presented as discrete components in the example configurations may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of embodiments as defined in the claims that follow.
Claims (20)
1. A system comprising:
an encoding pipeline configured to:
receive input video data represented at a high dynamic range in a first color space;
apply a tone mapping technique to the input video data to convert the input video data to video data represented at a lower dynamic range in a second color space; and
generate mapping metadata indicating information used in the tone mapping technique to convert the input video data to the lower dynamic range video data;
a decoding pipeline configured to:
obtain the lower dynamic range video data and the mapping metadata;
convert the lower dynamic range video data from the second color space to a third color space to generate video data represented in the third color space; and
apply an inverse tone mapping technique to the video data represented in the third color space to generate video data represented at a higher dynamic range in the third color space, wherein the inverse tone mapping technique uses the mapping metadata to recover at least part of the high dynamic range of the input video data.
2. The system as recited in claim 1 ,
wherein the encoding pipeline is further configured to:
apply color gamut mapping to the input video data to convert the video data from the first color space to the second color space; and
record color gamut mapping information in the mapping metadata;
wherein, to convert the lower dynamic range video data from the second color space to the third color space, the decoding pipeline is configured to recover color that was clipped by the color gamut mapping according to the color gamut mapping information in the mapping metadata.
3. The system as recited in claim 1 , wherein the tone mapping technique includes a local tone mapping technique in which each image is divided into a plurality of regions, a local tone curve is dynamically determined for each region, and each region is tone mapped according to its respective local tone curve.
4. The system as recited in claim 3 , wherein the mapping metadata includes information indicating the local tone curves as applied to the regions in one or more images.
5. The system as recited in claim 4 , wherein the information indicating the local tone curves is compressed spatially within each image.
6. The system as recited in claim 4 , wherein the information indicating the local tone curves is compressed temporally across two or more images.
7. The system as recited in claim 1 , wherein the encoding pipeline is further configured to encode the lower dynamic range video data according to a compressed video format to generate compressed video data, and wherein the decoding pipeline is further configured to decode the compressed video data to obtain the lower dynamic range video data.
8. The system as recited in claim 7 , wherein the compressed video format is one of H.264/Advanced Video Coding (AVC) format or H.265 High Efficiency Video Coding (HEVC) format.
9. The system as recited in claim 1 , wherein the decoding pipeline is further configured to adjust the inverse tone mapping technique as applied to the video data according to dynamic range of a target display to generate the video data represented at the dynamic range of the target display.
10. The system as recited in claim 1 , wherein the decoding pipeline is further configured to dynamically adjust the inverse tone mapping technique as applied to the video data according to current environmental conditions at a target display to generate the video data represented at a dynamic range adapted for viewing in the current environmental conditions of the target display.
11. The system as recited in claim 10 , wherein the environmental conditions include ambient lighting conditions and distance to one or more viewers.
12. The system as recited in claim 1 , wherein the first and third color spaces are RGB color spaces, and wherein the second color space is a YUV color space.
13. The system as recited in claim 1 , wherein the encoding pipeline and the decoding pipeline are implemented on a single device.
14. The system as recited in claim 1 , wherein the encoding pipeline and the decoding pipeline are implemented on different devices.
15. The system as recited in claim 1 , wherein the encoding pipeline and the decoding pipeline are components of a system on a chip (SOC).
16. A method, comprising:
performing, by an encoding pipeline:
receiving input video data represented at a high dynamic range;
applying tone mapping to the input video data to convert the input video data to video data represented at a lower dynamic range;
generating mapping metadata indicating information used in the tone mapping to map the input video data to the lower dynamic range;
encoding the lower dynamic range video data according to a compressed video format to generate compressed video data; and
providing the compressed video data and the mapping metadata to one or more display pipelines.
17. The method as recited in claim 16 , further comprising:
performing, by a display pipeline:
decoding the compressed video data to obtain decompressed video data represented at the lower dynamic range;
applying inverse tone mapping to the decompressed video data to generate video data represented at a higher dynamic range, wherein the inverse tone mapping uses the mapping metadata to recover at least part of the high dynamic range of the input video data; and
displaying the video data represented at the higher dynamic range to a high dynamic range (HDR)-enabled display.
18. The method as recited in claim 16 , wherein applying tone mapping to the input video data comprises:
for each video frame, dynamically determining a local tone curve for each of a plurality of regions in the frame; and
tone mapping each region according to its respective local tone curve;
wherein the mapping metadata includes information indicating the local tone curves as applied to the regions.
19. An apparatus, comprising:
an encoding module configured to:
apply tone mapping to input video data represented at a high dynamic range to generate video data represented at a lower dynamic range;
encode the lower dynamic range video data according to a compressed video format to generate compressed video data; and
output the compressed video data and metadata including information used in said tone mapping;
a decoding module configured to:
decode the compressed video data to obtain decompressed video data represented at the lower dynamic range;
apply inverse tone mapping to the decompressed video data to generate video data represented at a higher dynamic range, wherein the inverse tone mapping technique uses the mapping metadata to recover at least part of the high dynamic range of the input video data.
20. The apparatus as recited in claim 19 , wherein the decoding module is further configured to apply tone mapping to the video data represented at a higher dynamic range to generate standard dynamic range (SDR) video data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/233,099 US20210235093A1 (en) | 2014-02-25 | 2021-04-16 | Backward-compatible video capture and distribution |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201461944484P | 2014-02-25 | 2014-02-25 | |
US201461946633P | 2014-02-28 | 2014-02-28 | |
US201461946638P | 2014-02-28 | 2014-02-28 | |
US14/631,401 US10212429B2 (en) | 2014-02-25 | 2015-02-25 | High dynamic range video capture with backward-compatible distribution |
US16/277,892 US10986345B2 (en) | 2014-02-25 | 2019-02-15 | Backward-compatible video capture and distribution |
US17/233,099 US20210235093A1 (en) | 2014-02-25 | 2021-04-16 | Backward-compatible video capture and distribution |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/277,892 Continuation US10986345B2 (en) | 2014-02-25 | 2019-02-15 | Backward-compatible video capture and distribution |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210235093A1 true US20210235093A1 (en) | 2021-07-29 |
Family
ID=53882764
Family Applications (10)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/631,394 Active US10271054B2 (en) | 2014-02-25 | 2015-02-25 | Display-side adaptive video processing |
US14/631,401 Active 2035-07-10 US10212429B2 (en) | 2014-02-25 | 2015-02-25 | High dynamic range video capture with backward-compatible distribution |
US14/631,405 Active 2035-05-10 US10264266B2 (en) | 2014-02-25 | 2015-02-25 | Non-linear display brightness adjustment |
US14/631,398 Active US10880549B2 (en) | 2014-02-25 | 2015-02-25 | Server-side adaptive video processing |
US14/631,410 Active 2035-04-06 US10812801B2 (en) | 2014-02-25 | 2015-02-25 | Adaptive transfer function for video encoding and decoding |
US16/277,892 Active US10986345B2 (en) | 2014-02-25 | 2019-02-15 | Backward-compatible video capture and distribution |
US17/073,263 Active US11445202B2 (en) | 2014-02-25 | 2020-10-16 | Adaptive transfer function for video encoding and decoding |
US17/132,922 Abandoned US20210227226A1 (en) | 2014-02-25 | 2020-12-23 | Server-side adaptive video processing |
US17/233,099 Pending US20210235093A1 (en) | 2014-02-25 | 2021-04-16 | Backward-compatible video capture and distribution |
US17/897,079 Pending US20220408096A1 (en) | 2014-02-25 | 2022-08-26 | Adaptive Transfer Function for Video Encoding and Decoding |
Family Applications Before (8)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/631,394 Active US10271054B2 (en) | 2014-02-25 | 2015-02-25 | Display-side adaptive video processing |
US14/631,401 Active 2035-07-10 US10212429B2 (en) | 2014-02-25 | 2015-02-25 | High dynamic range video capture with backward-compatible distribution |
US14/631,405 Active 2035-05-10 US10264266B2 (en) | 2014-02-25 | 2015-02-25 | Non-linear display brightness adjustment |
US14/631,398 Active US10880549B2 (en) | 2014-02-25 | 2015-02-25 | Server-side adaptive video processing |
US14/631,410 Active 2035-04-06 US10812801B2 (en) | 2014-02-25 | 2015-02-25 | Adaptive transfer function for video encoding and decoding |
US16/277,892 Active US10986345B2 (en) | 2014-02-25 | 2019-02-15 | Backward-compatible video capture and distribution |
US17/073,263 Active US11445202B2 (en) | 2014-02-25 | 2020-10-16 | Adaptive transfer function for video encoding and decoding |
US17/132,922 Abandoned US20210227226A1 (en) | 2014-02-25 | 2020-12-23 | Server-side adaptive video processing |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/897,079 Pending US20220408096A1 (en) | 2014-02-25 | 2022-08-26 | Adaptive Transfer Function for Video Encoding and Decoding |
Country Status (8)
Country | Link |
---|---|
US (10) | US10271054B2 (en) |
EP (2) | EP3829171A1 (en) |
JP (5) | JP6383425B2 (en) |
KR (7) | KR101898643B1 (en) |
CN (7) | CN111526350B (en) |
AU (5) | AU2015223123A1 (en) |
DE (2) | DE112015000950T5 (en) |
WO (3) | WO2015130793A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11445202B2 (en) | 2014-02-25 | 2022-09-13 | Apple Inc. | Adaptive transfer function for video encoding and decoding |
US20220351341A1 (en) * | 2021-04-30 | 2022-11-03 | Realnetworks, Inc. | Intelligent metadata service for video enhancement |
WO2023055075A1 (en) * | 2021-09-30 | 2023-04-06 | Samsung Electronics Co., Ltd. | Standard dynamic range (sdr) / hybrid log-gamma (hlg) with high dynamic range (hdr) 10+ |
Families Citing this family (221)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9955084B1 (en) * | 2013-05-23 | 2018-04-24 | Oliver Markus Haynold | HDR video camera |
JP2015005878A (en) | 2013-06-20 | 2015-01-08 | ソニー株式会社 | Reproduction device, reproduction method and recording medium |
MX353125B (en) * | 2013-06-20 | 2017-12-20 | Sony Corp | Reproduction device, reproduction method, and recording medium. |
US20150228106A1 (en) * | 2014-02-13 | 2015-08-13 | Vixs Systems Inc. | Low latency video texture mapping via tight integration of codec engine with 3d graphics engine |
JP2015195439A (en) * | 2014-03-31 | 2015-11-05 | ソニー株式会社 | image processing apparatus, image processing method and program |
WO2015174109A1 (en) * | 2014-05-12 | 2015-11-19 | ソニー株式会社 | Communication apparatus, communication method, and computer program |
US10217438B2 (en) * | 2014-05-30 | 2019-02-26 | Apple Inc. | User interface and method for directly setting display white point |
WO2015190045A1 (en) * | 2014-06-10 | 2015-12-17 | パナソニックIpマネジメント株式会社 | Conversion method and conversion device |
EP3739894A1 (en) * | 2014-06-27 | 2020-11-18 | Panasonic Intellectual Property Management Co., Ltd. | Data output device, data output method, and data generation method |
US10163408B1 (en) * | 2014-09-05 | 2018-12-25 | Pixelworks, Inc. | LCD image compensation for LED backlighting |
EP3223530A4 (en) * | 2014-09-22 | 2018-01-10 | Panasonic Intellectual Property Management Co., Ltd. | Playback method and playback device |
US9729801B2 (en) * | 2014-10-02 | 2017-08-08 | Dolby Laboratories Licensing Corporation | Blending images using mismatched source and display electro-optical transfer functions |
JP6601729B2 (en) * | 2014-12-03 | 2019-11-06 | パナソニックIpマネジメント株式会社 | Data generation method, data reproduction method, data generation device, and data reproduction device |
TW201633779A (en) * | 2014-12-16 | 2016-09-16 | 湯姆生特許公司 | Method and device of converting a HDR version of a picture to a SDR version of said picture |
US9860504B2 (en) | 2015-01-09 | 2018-01-02 | Vixs Systems, Inc. | Color gamut mapper for dynamic range conversion and methods for use therewith |
US9589313B2 (en) * | 2015-01-09 | 2017-03-07 | Vixs Systems, Inc. | Dynamic range converter with pipelined architecture and methods for use therewith |
US9560330B2 (en) * | 2015-01-09 | 2017-01-31 | Vixs Systems, Inc. | Dynamic range converter with reconfigurable architecture and methods for use therewith |
US9544560B2 (en) * | 2015-01-09 | 2017-01-10 | Vixs Systems, Inc. | Dynamic range converter with generic architecture and methods for use therewith |
US9558538B2 (en) * | 2015-01-09 | 2017-01-31 | Vixs Systems, Inc. | Dynamic range converter with frame by frame adaptation and methods for use therewith |
US10475361B2 (en) | 2015-02-02 | 2019-11-12 | Apple Inc. | Adjustable display illumination |
KR102423434B1 (en) * | 2015-02-17 | 2022-07-22 | 소니그룹주식회사 | Transmitting device, sending method, receiving device and receiving method |
KR102423610B1 (en) * | 2015-02-27 | 2022-07-22 | 소니그룹주식회사 | Transmitting device, sending method, receiving device and receiving method |
US20160182853A1 (en) * | 2015-03-20 | 2016-06-23 | Mediatek Inc. | Dynamic Content Adaptive Frame Rate Conversion |
US10109228B2 (en) * | 2015-04-10 | 2018-10-23 | Samsung Display Co., Ltd. | Method and apparatus for HDR on-demand attenuation control |
US10735755B2 (en) * | 2015-04-21 | 2020-08-04 | Arris Enterprises Llc | Adaptive perceptual mapping and signaling for video coding |
JP6476057B2 (en) * | 2015-04-28 | 2019-02-27 | 日本放送協会 | Quantization table correction device, gradation reduction device, and program |
KR102322709B1 (en) * | 2015-04-29 | 2021-11-08 | 엘지디스플레이 주식회사 | Image processing method, image processing circuit and display device using the same |
KR102337159B1 (en) * | 2015-05-21 | 2021-12-08 | 삼성전자주식회사 | Apparatus and method for outputting content, and display apparatus |
US9942489B2 (en) * | 2015-06-02 | 2018-04-10 | Samsung Electronics Co., Ltd. | Adaptive tone mapping based on local contrast |
US10007412B2 (en) * | 2015-06-24 | 2018-06-26 | Samsung Electronics Co., Ltd. | Tone mastering system with creative intent metadata |
JP6530854B2 (en) | 2015-07-16 | 2019-06-12 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Signal shaping and coding for HDR and wide gamut signals |
JP2018525905A (en) * | 2015-07-22 | 2018-09-06 | アリス エンタープライジズ エルエルシーArris Enterprises Llc | System for encoding high dynamic range and wide color gamut sequences |
US10609395B2 (en) * | 2015-07-28 | 2020-03-31 | Vid Scale, Inc. | High dynamic range video coding architectures with multiple operating modes |
JP6548516B2 (en) * | 2015-08-26 | 2019-07-24 | キヤノン株式会社 | IMAGE DISPLAY DEVICE, IMAGE PROCESSING DEVICE, CONTROL METHOD OF IMAGE DISPLAY DEVICE, AND CONTROL METHOD OF IMAGE PROCESSING DEVICE |
US10687080B2 (en) * | 2015-08-28 | 2020-06-16 | Arris Enterprises Llc | Color volume transforms in coding of high dynamic range and wide color gamut sequences |
US10140694B2 (en) | 2015-08-31 | 2018-11-27 | Lg Electronics Inc. | Image display apparatus |
JP6986670B2 (en) * | 2015-09-11 | 2021-12-22 | パナソニックIpマネジメント株式会社 | Video receiving method and video receiving device |
US10798422B2 (en) * | 2015-10-20 | 2020-10-06 | Intel Corporation | Method and system of video coding with post-processing indication |
US10715772B2 (en) * | 2015-11-09 | 2020-07-14 | Netflix, Inc. | High dynamic range color conversion correction |
CN106878694B (en) * | 2015-12-10 | 2018-12-18 | 瑞昱半导体股份有限公司 | high dynamic range signal processing system and method |
CN108370442B (en) | 2015-12-15 | 2020-02-14 | 华为技术有限公司 | Method and device for processing high dynamic range image and computer readable storage medium |
WO2017110823A1 (en) * | 2015-12-25 | 2017-06-29 | シャープ株式会社 | Display device, method for controlling display device, control program, and recording medium |
US20170186403A1 (en) * | 2015-12-28 | 2017-06-29 | Printing Technology Research Institute | Method of simulating print color on mobile devices |
CN105657539B (en) * | 2015-12-30 | 2019-05-31 | 深圳Tcl数字技术有限公司 | Video broadcasting method and device |
JP6233424B2 (en) * | 2016-01-05 | 2017-11-22 | ソニー株式会社 | Imaging system and imaging method |
JP6237797B2 (en) * | 2016-01-05 | 2017-11-29 | ソニー株式会社 | Video system, video processing method, program, and video converter |
JP6652301B2 (en) * | 2016-01-27 | 2020-02-19 | キヤノン株式会社 | Image processing apparatus, image processing method, and program |
KR20170090824A (en) * | 2016-01-29 | 2017-08-08 | 삼성전자주식회사 | Electronic apparatus and the contorl method thereof |
US10257394B2 (en) | 2016-02-12 | 2019-04-09 | Contrast, Inc. | Combined HDR/LDR video streaming |
US10264196B2 (en) | 2016-02-12 | 2019-04-16 | Contrast, Inc. | Systems and methods for HDR video capture with a mobile device |
US20180358048A1 (en) * | 2016-02-26 | 2018-12-13 | Sony Corporation | Information processing device, display device, information recording medium, information processing method, and program |
US20170279866A1 (en) * | 2016-03-22 | 2017-09-28 | Intel Corporation | Adaptation of streaming data based on the environment at a receiver |
JP2017175459A (en) * | 2016-03-24 | 2017-09-28 | シャープ株式会社 | Signal converter, signal conversion method, television receiver, program, and recording medium |
US10152947B2 (en) | 2016-04-06 | 2018-12-11 | Microsoft Technology Licensing, Llc | Display brightness updating |
US10956766B2 (en) | 2016-05-13 | 2021-03-23 | Vid Scale, Inc. | Bit depth remapping based on viewing parameters |
CN107395950B (en) * | 2016-05-16 | 2021-11-09 | 西安电子科技大学 | Media data processing method, device and system |
WO2018000126A1 (en) * | 2016-06-27 | 2018-01-04 | Intel Corporation | Method and system of multi-dynamic range multi-layer video blending with alpha channel sideband for video playback |
EP3477933B1 (en) * | 2016-06-27 | 2022-03-16 | Sony Group Corporation | Signal processing device, signal processing method, camera system, video system and server |
EP4336850A3 (en) * | 2016-07-08 | 2024-04-17 | InterDigital Madison Patent Holdings, SAS | Systems and methods for region-of-interest tone remapping |
US9916638B2 (en) * | 2016-07-20 | 2018-03-13 | Dolby Laboratories Licensing Corporation | Transformation of dynamic metadata to support alternate tone rendering |
US10587818B2 (en) * | 2016-08-02 | 2020-03-10 | Magna Electronics Inc. | Vehicle vision system with enhanced camera brightness control |
US10554901B2 (en) | 2016-08-09 | 2020-02-04 | Contrast Inc. | Real-time HDR video for vehicle control |
US10277829B1 (en) | 2016-08-12 | 2019-04-30 | Apple Inc. | Video capture in low-light conditions |
CN106210921B (en) * | 2016-08-12 | 2019-10-11 | 深圳创维-Rgb电子有限公司 | A kind of image effect method for improving and its device |
CN106101679B (en) * | 2016-08-23 | 2018-01-12 | 青岛海信电器股份有限公司 | The processing method and processing system of high dynamic range signal |
US10757385B2 (en) * | 2016-08-24 | 2020-08-25 | Qualcomm Incorporated | Color gamut adaptation with feedback channel |
US11153594B2 (en) | 2016-08-29 | 2021-10-19 | Apple Inc. | Multidimensional quantization techniques for video coding/decoding systems |
KR102594201B1 (en) * | 2016-09-22 | 2023-10-27 | 삼성디스플레이 주식회사 | Method of processing image and display apparatus performing the same |
US9883112B1 (en) * | 2016-09-22 | 2018-01-30 | Pinnacle Imaging Corporation | High dynamic range imaging |
US10631002B2 (en) | 2016-09-30 | 2020-04-21 | Qualcomm Incorporated | Frame rate up-conversion coding mode |
US20180098041A1 (en) * | 2016-09-30 | 2018-04-05 | Sean J. Lawrence | Adaptive chroma subsampling based on display brightness |
EP3312798A1 (en) * | 2016-10-20 | 2018-04-25 | Thomson Licensing | Method and device for inverse tone mapping |
US10244244B2 (en) * | 2016-10-26 | 2019-03-26 | Dolby Laboratories Licensing Corporation | Screen-adaptive decoding of high dynamic range video |
US9942513B1 (en) * | 2016-10-31 | 2018-04-10 | Cisco Technology, Inc. | Automated configuration of behavior of a telepresence system based on spatial detection of telepresence components |
JP2018081595A (en) * | 2016-11-17 | 2018-05-24 | キヤノン株式会社 | Image processing device and method and imaging device |
US10218952B2 (en) | 2016-11-28 | 2019-02-26 | Microsoft Technology Licensing, Llc | Architecture for rendering high dynamic range video on enhanced dynamic range display devices |
CN110050292B (en) | 2016-12-12 | 2023-08-01 | 杜比实验室特许公司 | System and method for adjusting video processing curve of high dynamic range image |
JP6822122B2 (en) * | 2016-12-19 | 2021-01-27 | ソニー株式会社 | Image processing equipment, image processing methods and programs |
JP6822121B2 (en) * | 2016-12-19 | 2021-01-27 | ソニー株式会社 | Image processing equipment, image processing methods and programs |
JP6822123B2 (en) * | 2016-12-19 | 2021-01-27 | ソニー株式会社 | Image processing equipment, image processing methods and programs |
GB2558000B (en) | 2016-12-21 | 2020-06-10 | Apical Ltd | Display control |
GB2558236B (en) | 2016-12-22 | 2021-03-31 | Apical Ltd | Image data comprising encoded image data and additional data for modification of image |
CN106710505A (en) * | 2017-01-16 | 2017-05-24 | 努比亚技术有限公司 | Display device and method |
CN108320721B (en) | 2017-01-17 | 2022-05-24 | 华硕电脑股份有限公司 | Brightness adjusting device and brightness adjusting method thereof |
US10902601B2 (en) | 2017-01-18 | 2021-01-26 | Dolby Laboratories Licensing Corporation | Segment-based reshaping for coding high dynamic range video |
WO2018136432A1 (en) * | 2017-01-18 | 2018-07-26 | Dolby Laboratories Licensing Corporation | Segment-based reshaping for coding high dynamic range video |
CN106791865B (en) * | 2017-01-20 | 2020-02-28 | 杭州当虹科技股份有限公司 | Self-adaptive format conversion method based on high dynamic range video |
US10104334B2 (en) | 2017-01-27 | 2018-10-16 | Microsoft Technology Licensing, Llc | Content-adaptive adjustment of display device brightness levels when rendering high dynamic range content |
US10176561B2 (en) | 2017-01-27 | 2019-01-08 | Microsoft Technology Licensing, Llc | Content-adaptive adjustments to tone mapping operations for high dynamic range content |
US11765406B2 (en) | 2017-02-17 | 2023-09-19 | Interdigital Madison Patent Holdings, Sas | Systems and methods for selective object-of-interest zooming in streaming video |
US11310532B2 (en) | 2017-02-24 | 2022-04-19 | Interdigital Vc Holdings, Inc. | Method and device for reconstructing image data from decoded image data |
JP7086587B2 (en) * | 2017-02-24 | 2022-06-20 | インターデジタル ヴイシー ホールディングス, インコーポレイテッド | Method and device for reconstructing image data from decoded image data |
EP3591968A4 (en) * | 2017-02-28 | 2020-01-22 | Sony Corporation | Image processing device, and image processing method |
CN108540781A (en) * | 2017-03-03 | 2018-09-14 | 扬智科技股份有限公司 | Image signal processor and video-signal processing method |
JP7083837B2 (en) * | 2017-03-06 | 2022-06-13 | イー インク コーポレイション | Methods and equipment for rendering color images |
KR102628139B1 (en) | 2017-03-07 | 2024-01-22 | 인터디지털 매디슨 페턴트 홀딩스 에스에이에스 | Customized video streaming for multi-device presentations |
US10701390B2 (en) | 2017-03-14 | 2020-06-30 | Qualcomm Incorporated | Affine motion information derivation |
CN106911903A (en) * | 2017-03-15 | 2017-06-30 | 四川长虹电器股份有限公司 | A kind of SDR signals turn the image quality data processing method of HDR signal effects |
WO2018174387A1 (en) * | 2017-03-20 | 2018-09-27 | 엘지전자 주식회사 | Method for transmitting 360 video, method for receiving 360 video, 360 video transmitting device, and 360 video receiving device |
US11158032B2 (en) | 2017-03-20 | 2021-10-26 | Dolby Laboratories Licensing Corporation | Perceptually preserving scene-referred contrasts and chromaticities |
CN108737877B (en) * | 2017-04-21 | 2020-06-02 | 华为技术有限公司 | Image processing method and device and terminal equipment |
CN108810531B (en) * | 2017-05-03 | 2019-11-19 | 腾讯科技(深圳)有限公司 | Video coding processing method, device and electronic equipment |
US10847118B2 (en) * | 2017-05-12 | 2020-11-24 | Apple Inc. | Electronic devices with tone mapping engines |
US10403214B2 (en) | 2017-05-12 | 2019-09-03 | Apple Inc. | Electronic devices with tone mapping to accommodate simultaneous display of standard dynamic range and high dynamic range content |
US10242628B2 (en) * | 2017-05-18 | 2019-03-26 | Dell Products, L.P. | Light Emitting Diode (LED) backlight control for reproduction of High Dynamic Range (HDR) content using Standard Dynamic Range (SDR) Liquid Crystal Display (LCD) panels |
GB2563413B (en) * | 2017-06-14 | 2021-12-22 | Displaylink Uk Ltd | Processing display data |
GB2563411B (en) * | 2017-06-14 | 2022-10-05 | Displaylink Uk Ltd | Processing display data |
CN110770787B (en) * | 2017-06-16 | 2023-04-07 | 杜比实验室特许公司 | Efficient end-to-end single-layer reverse display management coding |
US10778978B2 (en) * | 2017-08-21 | 2020-09-15 | Qualcomm Incorporated | System and method of cross-component dynamic range adjustment (CC-DRA) in video coding |
CN107656454A (en) * | 2017-09-21 | 2018-02-02 | 深圳市晟达机械设计有限公司 | A kind of efficient cell monitoring management system |
KR102444156B1 (en) * | 2017-09-22 | 2022-09-16 | 삼성디스플레이 주식회사 | Display device and mehthod for driving the same |
US11533450B2 (en) * | 2017-09-25 | 2022-12-20 | Comcast Cable Communications, Llc | Anti-piracy video transmission and display |
US10609372B2 (en) | 2017-09-29 | 2020-03-31 | Dolby Laboratories Licensing Corporation | Up-conversion to content adaptive perceptual quantization video signals |
CN107682673B (en) * | 2017-10-09 | 2019-11-12 | 成都索贝数码科技股份有限公司 | A method of 4KHDR and HDSDR signal is supervised based on GPU and video IO card simultaneously |
US10972767B2 (en) * | 2017-11-01 | 2021-04-06 | Realtek Semiconductor Corp. | Device and method of handling multiple formats of a video sequence |
KR102413839B1 (en) | 2017-11-15 | 2022-06-28 | 삼성전자 주식회사 | Apparatus for providing content, method for controlling thereof and recording media thereof |
CN108024032A (en) * | 2017-11-24 | 2018-05-11 | 深圳Tcl新技术有限公司 | Image enhancement processing method, TV, server and computer-readable recording medium |
CN108288278B (en) * | 2018-01-22 | 2020-11-24 | 海信视像科技股份有限公司 | Image brightness processing method and device and electronic equipment |
CN108200441B (en) * | 2018-01-22 | 2020-10-13 | 海信视像科技股份有限公司 | Image brightness processing method and device and electronic equipment |
US10880531B2 (en) * | 2018-01-31 | 2020-12-29 | Nvidia Corporation | Transfer of video signals using variable segmented lookup tables |
JP2019140430A (en) * | 2018-02-06 | 2019-08-22 | 船井電機株式会社 | Reproduction device |
US11025942B2 (en) * | 2018-02-08 | 2021-06-01 | Samsung Electronics Co., Ltd. | Progressive compressed domain computer vision and deep learning systems |
EP3525463A1 (en) * | 2018-02-13 | 2019-08-14 | Koninklijke Philips N.V. | System for handling multiple hdr video formats |
ES2724212B2 (en) * | 2018-03-02 | 2021-03-31 | Visionapp Solutions S L | METHOD IMPLEMENTED IN THE COMPUTER AND SYSTEM FOR THE PREVENTION OF VISION IMPAIRMENT CAUSED BY THE PROLONGED USE OF ELECTRONIC DISPLAYS IN LOW LIGHTING CONDITIONS. |
US10609424B2 (en) * | 2018-03-09 | 2020-03-31 | Dolby Laboratories Licensing Corporation | Single-layer progressive coding for supporting multi-capability HDR composition |
JP7090730B2 (en) | 2018-04-05 | 2022-06-24 | キヤノン株式会社 | Methods and devices for encapsulating images in files |
US11120725B2 (en) | 2018-04-24 | 2021-09-14 | Advanced Micro Devices, Inc. | Method and apparatus for color gamut mapping color gradient preservation |
TWI808971B (en) * | 2018-04-30 | 2023-07-21 | 圓剛科技股份有限公司 | Video signal conversion device |
TWI808970B (en) | 2018-04-30 | 2023-07-21 | 圓剛科技股份有限公司 | Video signal conversion device |
TW201946430A (en) * | 2018-04-30 | 2019-12-01 | 圓剛科技股份有限公司 | Video signal conversion device and method thereof |
TWI811217B (en) * | 2018-04-30 | 2023-08-11 | 圓剛科技股份有限公司 | Video signal conversion device |
US10692467B2 (en) | 2018-05-04 | 2020-06-23 | Microsoft Technology Licensing, Llc | Automatic application of mapping functions to video signals based on inferred parameters |
US20190356891A1 (en) * | 2018-05-16 | 2019-11-21 | Synaptics Incorporated | High dynamic range (hdr) data conversion and color space mapping |
US11030728B2 (en) * | 2018-05-29 | 2021-06-08 | Apple Inc. | Tone mapping techniques for increased dynamic range |
US10951888B2 (en) * | 2018-06-04 | 2021-03-16 | Contrast, Inc. | Compressed high dynamic range video |
JP7171768B2 (en) * | 2018-06-28 | 2022-11-15 | アップル インコーポレイテッド | Video coding system |
US11115563B2 (en) * | 2018-06-29 | 2021-09-07 | Ati Technologies Ulc | Method and apparatus for nonlinear interpolation color conversion using look up tables |
KR102572432B1 (en) * | 2018-07-03 | 2023-08-30 | 삼성전자주식회사 | Display apparatus, control method thereof and recording media |
EP3763129A4 (en) | 2018-07-12 | 2021-07-21 | Samsung Electronics Co., Ltd. | Method and apparatus for rendering contents for vision accessibility |
US11962819B2 (en) | 2018-07-17 | 2024-04-16 | Dolby Laboratories Licensing Corporation | Foviation and HDR |
JP7443325B2 (en) * | 2018-07-17 | 2024-03-05 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Foveation and HDR |
US20200045341A1 (en) * | 2018-07-31 | 2020-02-06 | Ati Technologies Ulc | Effective electro-optical transfer function encoding for limited luminance range displays |
US10930022B2 (en) * | 2018-08-09 | 2021-02-23 | Nvidia Corporation | Motion adaptive rendering using variable rate shading |
CN109192145A (en) * | 2018-09-21 | 2019-01-11 | 合肥惠科金扬科技有限公司 | A kind of subregion light adjusting circuit and display device |
JP7509761B2 (en) * | 2018-09-26 | 2024-07-02 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Projector Light Source Dimming Using Future Frame Metadata |
CN109167946B (en) * | 2018-09-30 | 2021-02-19 | Oppo广东移动通信有限公司 | Video processing method, video processing device, electronic equipment and storage medium |
CN109274985B (en) * | 2018-10-12 | 2020-04-28 | 腾讯科技(深圳)有限公司 | Video transcoding method and device, computer equipment and storage medium |
CN109151574B (en) | 2018-10-15 | 2020-03-24 | Oppo广东移动通信有限公司 | Video processing method, video processing device, electronic equipment and storage medium |
WO2020081305A1 (en) | 2018-10-17 | 2020-04-23 | Corning Incorporated | Methods for achieving, and apparatus having, reduced display device energy consumption |
US10957024B2 (en) | 2018-10-30 | 2021-03-23 | Microsoft Technology Licensing, Llc | Real time tone mapping of high dynamic range image data at time of playback on a lower dynamic range display |
US11503310B2 (en) * | 2018-10-31 | 2022-11-15 | Ati Technologies Ulc | Method and apparatus for an HDR hardware processor inline to hardware encoder and decoder |
WO2020100200A1 (en) * | 2018-11-12 | 2020-05-22 | Eizo株式会社 | Image processing system, image processing device, and computer program |
CN109327734B (en) * | 2018-11-27 | 2020-11-13 | 成都索贝数码科技股份有限公司 | Method for down-converting HDR video to SDR video based on dynamic photometry |
US10860504B2 (en) | 2018-12-05 | 2020-12-08 | New Century Technologies Ltd. | Peripheral device with embedded video codec functionality |
US11587526B2 (en) * | 2018-12-20 | 2023-02-21 | Dolby Laboratories Licensing Corporation | Luminance adaption to minimize discomfort and improve visibility |
EP3672267A1 (en) * | 2018-12-20 | 2020-06-24 | InterDigital VC Holdings, Inc. | Methods for processing audio and/or video contents and corresponding signal, devices, electronic assembly, system, computer readable program products and computer readable storage media |
CN111479164A (en) * | 2019-01-23 | 2020-07-31 | 上海哔哩哔哩科技有限公司 | Hardware decoding dynamic resolution seamless switching method and device and storage medium |
KR20200095651A (en) * | 2019-02-01 | 2020-08-11 | 삼성전자주식회사 | An electronic device for playing high dynamic range content and method thereof |
JP7263053B2 (en) | 2019-02-28 | 2023-04-24 | キヤノン株式会社 | IMAGING DEVICE, IMAGE PROCESSING DEVICE, CONTROL METHOD THEREOF, AND PROGRAM |
US10509991B1 (en) | 2019-03-18 | 2019-12-17 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing infrared light |
US10534948B1 (en) | 2019-03-18 | 2020-01-14 | Capital One Services, Llc | Optimizing detection of images in relation to targets based on colorspace transformation techniques |
US10496862B1 (en) | 2019-03-18 | 2019-12-03 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing ultraviolet light |
US10496911B1 (en) | 2019-03-18 | 2019-12-03 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing ultraviolet and infrared light |
CN111754412B (en) * | 2019-03-29 | 2024-04-19 | Tcl科技集团股份有限公司 | Method and device for constructing data pair and terminal equipment |
US11404025B2 (en) | 2019-04-10 | 2022-08-02 | Mediatek Inc. | Video processing system for performing artificial intelligence assisted picture quality enhancement and associated video processing method |
EP3928524A4 (en) | 2019-04-15 | 2022-06-22 | Beijing Bytedance Network Technology Co., Ltd. | Temporal prediction of parameters in non-linear adaptive loop filter |
US10523420B1 (en) | 2019-04-18 | 2019-12-31 | Capital One Services, Llc | Transmitting encoded data along transmission mediums based on colorspace schemes |
US10504013B1 (en) | 2019-04-24 | 2019-12-10 | Capital One Services, Llc | Colorspace encoding multimedia data on a physical page |
WO2020219401A1 (en) * | 2019-04-25 | 2020-10-29 | Dolby Laboratories Licensing Corporation | Content-aware pq range analyzer and tone mapping in live feeds |
CN111916031B (en) * | 2019-05-10 | 2023-03-21 | 京东方科技集团股份有限公司 | Display method and display device |
EP3742728B1 (en) * | 2019-05-24 | 2022-09-21 | Axis AB | A method and bitrate controller for controlling output bitrate of a video encoder |
US11425379B2 (en) | 2019-05-31 | 2022-08-23 | Hfi Innovation Inc. | Method and apparatus of latency reduction for chroma residue scaling |
US11115602B2 (en) * | 2019-06-11 | 2021-09-07 | Watchguard Video, Inc. | Emulating light sensitivity of a target |
US10529300B1 (en) | 2019-06-20 | 2020-01-07 | Capital One Services, Llc | Adaptive image display based on colorspace conversions |
US10614635B1 (en) | 2019-07-25 | 2020-04-07 | Capital One Services, Llc | Augmented reality system with color-based fiducial marker |
US11438584B2 (en) * | 2019-08-16 | 2022-09-06 | Apple Inc. | Adaptive bilateral filtering using look-up tables |
US10937385B1 (en) * | 2019-08-20 | 2021-03-02 | Apple Inc. | Frame replay with bit depth considerations |
US10833852B1 (en) | 2019-10-03 | 2020-11-10 | Capital One Services, Llc | Encoded data along tape based on colorspace schemes |
US11159737B2 (en) * | 2019-10-14 | 2021-10-26 | Google Llc | Exposure change control in low light environments |
CN110798689B (en) * | 2019-10-18 | 2021-01-19 | 西安交通大学 | Method for converting NV12+ format into P010 format |
US10715183B1 (en) | 2019-10-25 | 2020-07-14 | Capital One Services, Llc | Data encoding with error-correcting code pursuant to colorspace schemes |
US10867226B1 (en) | 2019-11-04 | 2020-12-15 | Capital One Services, Llc | Programmable logic array and colorspace conversions |
US10762371B1 (en) | 2019-11-14 | 2020-09-01 | Capital One Services, Llc | Object detection techniques using colorspace conversions |
CN110868634B (en) * | 2019-11-27 | 2023-08-22 | 维沃移动通信有限公司 | Video processing method and electronic equipment |
TWI739227B (en) * | 2019-12-03 | 2021-09-11 | 智成電子股份有限公司 | System-on-chip module to avoid redundant memory access |
US10878600B1 (en) | 2019-12-10 | 2020-12-29 | Capital One Services, Llc | Augmented reality system with color-based fiducial marker utilizing local adaptive technology |
WO2021118556A1 (en) * | 2019-12-11 | 2021-06-17 | Google Llc | Color calibration of display modules using a reduced number of display characteristic measurements |
KR20210083840A (en) * | 2019-12-27 | 2021-07-07 | 삼성전자주식회사 | Electronic device for video editing with dynamic tone metadata and operating method thereof |
NO346123B1 (en) * | 2020-02-05 | 2022-03-07 | Movi Tech As | Video decoding method and device enabling improved user interaction with video content |
CN111294655B (en) | 2020-02-20 | 2021-10-22 | 深圳创维-Rgb电子有限公司 | HDR (high-definition multimedia System) film source playing method and device and storage medium |
CN111415393B (en) * | 2020-03-16 | 2023-08-01 | 安博思华智能科技有限责任公司 | Method and device for adjusting display of multimedia blackboard, medium and electronic equipment |
CN111524077B (en) * | 2020-04-17 | 2023-11-17 | 三维六度(北京)文化有限公司 | Method and system for repairing image data |
CN111540324B (en) * | 2020-05-20 | 2021-12-24 | Tcl华星光电技术有限公司 | Display device and pixel compensation method and device thereof |
TWI754970B (en) * | 2020-06-22 | 2022-02-11 | 鴻海精密工業股份有限公司 | Device, method and storage medium for accelerating operation of an activation function |
US11302036B2 (en) | 2020-08-19 | 2022-04-12 | Capital One Services, Llc | Color conversion between color spaces using reduced dimension embeddings |
CN111970564B (en) * | 2020-08-26 | 2023-03-24 | 展讯通信(上海)有限公司 | Optimization method and device for HDR video display processing, storage medium and terminal |
US11425502B2 (en) | 2020-09-18 | 2022-08-23 | Cisco Technology, Inc. | Detection of microphone orientation and location for directional audio pickup |
CN112118437B (en) * | 2020-09-24 | 2021-08-17 | 上海松鼠课堂人工智能科技有限公司 | Virtual reality classroom simulation method and system |
US11769234B2 (en) | 2020-10-07 | 2023-09-26 | Qualcomm Incorporated | Methods and apparatus for histogram based tone mapping |
US11538136B2 (en) * | 2020-10-28 | 2022-12-27 | Qualcomm Incorporated | System and method to process images of a video stream |
CN112351264B (en) * | 2020-11-04 | 2023-05-23 | 深圳Tcl数字技术有限公司 | Display precision adjusting method, device, terminal equipment and storage medium |
US20220191381A1 (en) * | 2020-12-11 | 2022-06-16 | Contrast, Inc. | Dynamic transfer function for reversible encoding |
US11115625B1 (en) | 2020-12-14 | 2021-09-07 | Cisco Technology, Inc. | Positional audio metadata generation |
CN112927273A (en) * | 2021-01-28 | 2021-06-08 | 北京字节跳动网络技术有限公司 | Three-dimensional video processing method, equipment and storage medium |
US11470162B2 (en) * | 2021-01-30 | 2022-10-11 | Zoom Video Communications, Inc. | Intelligent configuration of personal endpoint devices |
CN113160342B (en) * | 2021-05-14 | 2023-08-25 | 杭州网易智企科技有限公司 | Encoding method and device based on feedback, storage medium and electronic equipment |
CN113489930B (en) * | 2021-06-10 | 2024-03-19 | 中央广播电视总台 | Video signal processing method, device and storage medium |
US11612812B1 (en) | 2021-06-29 | 2023-03-28 | Amazon Technologies, Inc. | Video game streaming with dynamic range conversion |
US11666823B1 (en) * | 2021-06-29 | 2023-06-06 | Amazon Technologies, Inc. | Video game streaming with dynamic range conversion |
US11617946B1 (en) | 2021-06-29 | 2023-04-04 | Amazon Technologies, Inc. | Video game streaming with dynamic range conversion |
CN113395551A (en) * | 2021-07-20 | 2021-09-14 | 珠海极海半导体有限公司 | Processor, NPU chip and electronic equipment |
EP4388741A1 (en) * | 2021-08-16 | 2024-06-26 | InterDigital CE Patent Holdings, SAS | Systems and methods for processing video content based on viewing conditions |
CN113676773B (en) * | 2021-08-16 | 2023-11-14 | 广州虎牙信息科技有限公司 | Video playing method, system, device, computer equipment and storage medium |
US20230127327A1 (en) * | 2021-10-22 | 2023-04-27 | Samsung Electronics Co., Ltd. | System and method for learning tone curves for local image enhancement |
US11734806B2 (en) | 2021-11-24 | 2023-08-22 | Roku, Inc. | Dynamic tone mapping |
CN114363507A (en) * | 2021-12-02 | 2022-04-15 | 维沃移动通信有限公司 | Image processing method and device |
CN116540960A (en) * | 2022-01-25 | 2023-08-04 | 苏州佳世达电通有限公司 | Uniformity adjusting method, uniformity adjusting device and electronic device using uniformity adjusting device |
CN114422829A (en) * | 2022-01-30 | 2022-04-29 | 杭州雾联科技有限公司 | HDR cloud video processing method, system and equipment |
CN114466246A (en) * | 2022-02-14 | 2022-05-10 | 维沃移动通信有限公司 | Video processing method and device |
US12002189B2 (en) * | 2022-03-02 | 2024-06-04 | Nvidia Corporation | Remastering lower dynamic range content for higher dynamic range displays |
CN114727029B (en) * | 2022-03-21 | 2024-03-22 | 北京百度网讯科技有限公司 | Video processing method and device, electronic equipment and storage medium |
Citations (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5731818A (en) * | 1994-04-19 | 1998-03-24 | Eastman Kodak Company | Method and apparatus for constrained gamut clipping |
US20020196470A1 (en) * | 2001-05-24 | 2002-12-26 | Hiroyuki Kawamoto | Image processing method and apparatus and image forming apparatus for reducing moire fringes in output image |
US20040109180A1 (en) * | 2002-12-04 | 2004-06-10 | Eastman Kodak Company | Color gamut mapping using a cost function |
US20070035751A1 (en) * | 2005-08-15 | 2007-02-15 | Microsoft Corporation | Gamut shape dependent color management system |
US20070104378A1 (en) * | 2003-03-05 | 2007-05-10 | Seadragon Software, Inc. | Method for encoding and serving geospatial or other vector data as images |
US20080043268A1 (en) * | 2006-08-16 | 2008-02-21 | Samsung Electronics Co., Ltd. | Apparatus, method and medium outputting wide gamut space image |
US20080112024A1 (en) * | 2006-11-09 | 2008-05-15 | Seiko Epson Corporation | Image processing apparatus, image processing method, image processing program, recording medium storing image processing program, and image display apparatus |
US20080137990A1 (en) * | 2006-12-06 | 2008-06-12 | Brightside Technologies Inc. | Representing and reconstructing high dynamic range images |
US20090027558A1 (en) * | 2007-07-27 | 2009-01-29 | Rafal Mantiuk | Apparatus and Method for Rendering High Dynamic Range Images for Standard Dynamic Range Display |
US7508981B2 (en) * | 2004-12-15 | 2009-03-24 | Samsung Electronics Co., Ltd. | Dual layer bus architecture for system-on-a-chip |
US20100053651A1 (en) * | 2008-08-27 | 2010-03-04 | Seiko Epson Corporation | Image Processing Device and Image Processing Method |
US20100157078A1 (en) * | 2008-12-19 | 2010-06-24 | Qualcomm Incorporated | High dynamic range image combining |
US20100157154A1 (en) * | 2007-08-02 | 2010-06-24 | Sony Corporation | Image processing apparatus |
US20100172411A1 (en) * | 2006-01-23 | 2010-07-08 | Alexander Efremov | High dynamic range codecs |
US20100245857A1 (en) * | 2009-03-27 | 2010-09-30 | Konica Minolta Systems Laboratory, Inc. | Systems and Methods for Processing Color Data |
US20110194618A1 (en) * | 2009-03-13 | 2011-08-11 | Dolby Laboratories Licensing Corporation | Compatible compression of high dynamic range, visual dynamic range, and wide color gamut video |
US20110243473A1 (en) * | 2010-04-01 | 2011-10-06 | Himax Media Solutions, Inc. | Image enhancement method and apparatuses utilizing the same |
US20110292992A1 (en) * | 2010-05-28 | 2011-12-01 | Microsoft Corporation | Automating dynamic information insertion into video |
US8115787B2 (en) * | 2001-06-07 | 2012-02-14 | Genoa Color Technologies Ltd. | Device, system and method of data conversion for wide gamut displays |
US20120314944A1 (en) * | 2011-06-13 | 2012-12-13 | Dolby Laboratories Licensing Corporation | High dynamic range, backwards-compatible, digital cinema |
US20130003086A1 (en) * | 2011-06-28 | 2013-01-03 | Konica Minolta Laboratory U.S.A., Inc. | Method for processing high dynamic range images using tone mapping to extended rgb space |
US20130076763A1 (en) * | 2010-06-08 | 2013-03-28 | Dolby Laboratories Licensing Corporation | Tone and Gamut Mapping Methods and Apparatus |
US20130076974A1 (en) * | 2011-09-26 | 2013-03-28 | Dolby Laboratories Licensing Corporation | Image Formats and Related Methods and Apparatuses |
US20130129214A1 (en) * | 2010-08-04 | 2013-05-23 | Nec Corporation | Image processing method, image processing apparatus, and image processing program |
US20130308027A1 (en) * | 2012-05-03 | 2013-11-21 | Aptina Imaging Corporation | Systems and methods for generating metadata in stacked-chip imaging systems |
US20130321671A1 (en) * | 2012-05-31 | 2013-12-05 | Apple Inc. | Systems and method for reducing fixed pattern noise in image data |
US20130335439A1 (en) * | 2012-06-18 | 2013-12-19 | Samsung Display Co., Ltd. | System and method for converting color gamut |
US20140022460A1 (en) * | 2011-04-12 | 2014-01-23 | Dolby Laboratories Licensing Corporation | Quality Assessment for Images that Have Extended Dynamic Ranges or Wide Color Gamuts |
US20140241418A1 (en) * | 2011-11-09 | 2014-08-28 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | Inter-layer prediction between layers of different dynamic sample value range |
US20140267822A1 (en) * | 2013-03-12 | 2014-09-18 | Stmicroelectronics (Grenoble 2) Sas | Tone mapping method |
US8866975B1 (en) * | 2013-05-02 | 2014-10-21 | Dolby Laboratories Licensing Corporation | Backwards-compatible delivery of digital cinema content with higher dynamic range and related preprocessing and coding methods |
US9076224B1 (en) * | 2012-08-08 | 2015-07-07 | Dolby Laboratories Licensing Corporation | Image processing for HDR images |
US20160156965A1 (en) * | 2013-07-14 | 2016-06-02 | Lg Electronics Inc. | Method and apparatus for transmitting and receiving ultra high-definition broadcasting signal for expressing high-quality color in digital broadcasting system |
Family Cites Families (117)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US1020068A (en) | 1909-04-29 | 1912-03-12 | Singer Mfg Co | Feeding mechanism for sewing-machines. |
US1027105A (en) | 1911-01-31 | 1912-05-21 | George Crawford | Spring-wheel. |
US1021245A (en) | 1911-02-16 | 1912-03-26 | Morton Arthur George | Detachable hinge for windows. |
US1026426A (en) | 1911-05-15 | 1912-05-14 | Fred Kestler | Cinder-leveler. |
US5042077A (en) * | 1987-10-02 | 1991-08-20 | General Electric Company | Method of highlighting subtle contrast in graphical images |
US5357278A (en) | 1993-01-14 | 1994-10-18 | Sony Electronics, Inc. | Integrated linear/non-linear static and controllable dynamic companding |
US6141047A (en) * | 1993-12-22 | 2000-10-31 | Canon Kabushiki Kaisha | Image signal processing apparatus and image pickup device |
US5815206A (en) | 1996-05-03 | 1998-09-29 | Lsi Logic Corporation | Method for partitioning hardware and firmware tasks in digital audio/video decoding |
US6118820A (en) | 1998-01-16 | 2000-09-12 | Sarnoff Corporation | Region-based information compaction as for digital images |
US6829301B1 (en) | 1998-01-16 | 2004-12-07 | Sarnoff Corporation | Enhanced MPEG information distribution apparatus and method |
US6560285B1 (en) * | 1998-03-30 | 2003-05-06 | Sarnoff Corporation | Region-based information compaction as for digital images |
CA2269651A1 (en) * | 1998-05-12 | 1999-11-12 | Lucent Technologies, Inc. | Transform domain image watermarking method and system |
US6396508B1 (en) | 1999-12-02 | 2002-05-28 | Matsushita Electronics Corp. | Dynamic low-level enhancement and reduction of moving picture disturbance for a digital display |
US7072523B2 (en) | 2000-09-01 | 2006-07-04 | Lenovo (Singapore) Pte. Ltd. | System and method for fingerprint image enhancement using partitioned least-squared filters |
US6762741B2 (en) | 2000-12-22 | 2004-07-13 | Visteon Global Technologies, Inc. | Automatic brightness control system and method for a display device using a logarithmic sensor |
US7062445B2 (en) * | 2001-01-26 | 2006-06-13 | Microsoft Corporation | Quantization loop with heuristic approach |
GB0110748D0 (en) | 2001-05-02 | 2001-06-27 | Apical Ltd | Image enhancement methods and apparatus therefor |
US6757648B2 (en) * | 2001-06-28 | 2004-06-29 | Microsoft Corporation | Techniques for quantization of spectral data in transcoding |
US20030086595A1 (en) * | 2001-11-07 | 2003-05-08 | Hui Hu | Display parameter-dependent pre-transmission processing of image data |
US7120293B2 (en) | 2001-11-30 | 2006-10-10 | Microsoft Corporation | Interactive images |
US7433532B2 (en) | 2002-05-01 | 2008-10-07 | Kestrel Corporation | Max entropy optimized retinal camera |
US6778183B1 (en) * | 2002-07-10 | 2004-08-17 | Genesis Microchip Inc. | Method and system for adaptive color and contrast for display devices |
US7550701B2 (en) | 2003-02-25 | 2009-06-23 | Omnivision Cdm Optics, Inc. | Non-linear wavefront coding systems and methods |
US20050117799A1 (en) * | 2003-12-01 | 2005-06-02 | Chiou-Shann Fuh | Method and apparatus for transforming a high dynamic range image into a low dynamic range image |
GB0402640D0 (en) | 2004-02-06 | 2004-03-10 | Pelikon Ltd | Ambient light sensor |
US20050219252A1 (en) * | 2004-03-30 | 2005-10-06 | Buxton Mark J | Two-dimensional buffer, texture and frame buffer decompression |
US8218625B2 (en) * | 2004-04-23 | 2012-07-10 | Dolby Laboratories Licensing Corporation | Encoding, decoding and representing high dynamic range images |
US20050243176A1 (en) * | 2004-04-30 | 2005-11-03 | James Wu | Method of HDR image processing and manipulation |
EP1769459B1 (en) | 2004-05-31 | 2014-11-26 | Phase One A/S | Image compression for rapid high-quality imaging |
US7457477B2 (en) | 2004-07-06 | 2008-11-25 | Microsoft Corporation | Digital photography with flash/no flash extension |
US8050511B2 (en) * | 2004-11-16 | 2011-11-01 | Sharp Laboratories Of America, Inc. | High dynamic range images from low dynamic range images |
US7593024B2 (en) | 2005-01-15 | 2009-09-22 | International Business Machines Corporation | Screen calibration for display devices |
US7693304B2 (en) * | 2005-05-12 | 2010-04-06 | Hewlett-Packard Development Company, L.P. | Method and system for image quality calculation |
WO2007026283A2 (en) | 2005-08-31 | 2007-03-08 | Koninklijke Philips Electronics, N.V. | Ambient lighting derived from video content using adaptive extraction regions |
US7809200B2 (en) * | 2005-11-15 | 2010-10-05 | Teledyne Licensing, Llc | Dynamic range compression of high dynamic range imagery |
US8014445B2 (en) | 2006-02-24 | 2011-09-06 | Sharp Laboratories Of America, Inc. | Methods and systems for high dynamic range video coding |
US8194997B2 (en) | 2006-03-24 | 2012-06-05 | Sharp Laboratories Of America, Inc. | Methods and systems for tone mapping messaging |
US9106914B2 (en) | 2006-05-25 | 2015-08-11 | Thomson Licensing | Method and system for weighted encoding |
US7730043B1 (en) * | 2006-06-29 | 2010-06-01 | Adobe Systems Incorporated | Incremental batch-mode editing of digital media objects |
US8888592B1 (en) | 2009-06-01 | 2014-11-18 | Sony Computer Entertainment America Llc | Voice overlay |
CN101119497A (en) | 2006-08-04 | 2008-02-06 | 广达电脑股份有限公司 | Image processing apparatus and method thereof |
TWI333383B (en) | 2006-09-25 | 2010-11-11 | Ind Tech Res Inst | Color adjustment circuit, digital color adjustment device and a multimedia apparatus using the same |
KR101318081B1 (en) | 2006-11-21 | 2013-10-14 | 엘지디스플레이 주식회사 | LCD and drive method thereof |
US7899267B2 (en) | 2007-05-23 | 2011-03-01 | Zoran Corporation | Dynamic range compensation by filter cascade |
US20080316372A1 (en) | 2007-06-20 | 2008-12-25 | Ning Xu | Video display enhancement based on viewer characteristics |
US8004538B2 (en) * | 2007-09-21 | 2011-08-23 | Global Oled Technology Llc | Preferential tone scale for electronic displays |
TW200928860A (en) | 2007-12-31 | 2009-07-01 | Htc Corp | Method and device for adjusting output frame |
US9584710B2 (en) | 2008-02-28 | 2017-02-28 | Avigilon Analytics Corporation | Intelligent high resolution video system |
US8159616B2 (en) * | 2008-03-27 | 2012-04-17 | Bang & Olufsen A/S | Histogram and chrominance processing |
US9384535B2 (en) | 2008-06-13 | 2016-07-05 | Imax Corporation | Methods and systems for reducing or eliminating perceived ghosting in displayed stereoscopic images |
ES2389458T3 (en) * | 2008-07-10 | 2012-10-26 | The University Of Warwick | Methods and devices for compression of HDR video data |
EP2200268B1 (en) * | 2008-12-22 | 2012-02-08 | Thomson Licensing | Method of calibration of a target color reproduction device |
US8406569B2 (en) * | 2009-01-19 | 2013-03-26 | Sharp Laboratories Of America, Inc. | Methods and systems for enhanced dynamic range images and video from multiple exposures |
JP5274287B2 (en) * | 2009-02-09 | 2013-08-28 | 三菱電機株式会社 | Display device and display system |
KR101256030B1 (en) * | 2009-03-10 | 2013-04-23 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Extended dynamic range and extended dimensionality image signal conversion |
JP5566048B2 (en) | 2009-04-28 | 2014-08-06 | キヤノン株式会社 | Display device and control method thereof |
WO2010132237A1 (en) | 2009-05-11 | 2010-11-18 | Dolby Laboratories Licensing Corporation | Light detection, color appearance models, and modifying dynamic range for image display |
JP2011022481A (en) | 2009-07-17 | 2011-02-03 | Panasonic Corp | Liquid crystal display apparatus |
US8897377B2 (en) | 2009-12-31 | 2014-11-25 | Broadcom Corporation | Transcoding multiple media elements for independent wireless delivery |
EP2360926A1 (en) | 2010-01-19 | 2011-08-24 | Fraunhofer-Gesellschaft zur Förderung der Angewandten Forschung e.V. | Image encoder and image decoder |
WO2011089917A1 (en) | 2010-01-22 | 2011-07-28 | パナソニック株式会社 | Control device, display device, eyeglasses device and video system |
KR101428504B1 (en) * | 2010-02-22 | 2014-08-11 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Video display with rendering control using metadata embedded in the bitstream |
US8890793B2 (en) * | 2010-03-26 | 2014-11-18 | Hong Kong Applied Science and Technology Research Institute, Co. Ltd. | Adjusting a brightness level of a backlight of a display device |
US8493316B2 (en) * | 2010-04-22 | 2013-07-23 | Qualcomm Incorporated | Adjusting backlight intensity based on a progress of a task |
EP2591602A1 (en) * | 2010-07-06 | 2013-05-15 | Koninklijke Philips Electronics N.V. | Generation of high dynamic range images from low dynamic range images |
KR101537510B1 (en) * | 2010-07-22 | 2015-07-17 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Display management server |
US8677428B2 (en) * | 2010-08-20 | 2014-03-18 | Disney Enterprises, Inc. | System and method for rule based dynamic server side streaming manifest files |
TWI479898B (en) | 2010-08-25 | 2015-04-01 | Dolby Lab Licensing Corp | Extending image dynamic range |
US8704859B2 (en) * | 2010-09-30 | 2014-04-22 | Apple Inc. | Dynamic display adjustment based on ambient conditions |
EP2445214A1 (en) | 2010-10-19 | 2012-04-25 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Video coding using temporally coherent dynamic range mapping |
US8462221B2 (en) | 2010-11-03 | 2013-06-11 | Eastman Kodak Company | Method for producing high dynamic range images |
PL4053786T3 (en) * | 2010-11-23 | 2024-04-08 | Dolby Laboratories Licensing Corporation | Display of high dynamic range images on a local dimming display using bit depth reduction mapping metadata |
US9024951B2 (en) | 2011-02-16 | 2015-05-05 | Apple Inc. | Devices and methods for obtaining high-local-contrast image data |
US20140003527A1 (en) | 2011-03-10 | 2014-01-02 | Dolby Laboratories Licensing Corporation | Bitdepth and Color Scalable Video Coding |
US8621320B2 (en) * | 2011-04-07 | 2013-12-31 | Apple Inc. | Per-image forward error correction |
US8971408B2 (en) | 2011-04-14 | 2015-03-03 | Dolby Laboratories Licensing Corporation | Piecewise cross color channel predictor |
TWI513327B (en) | 2011-04-15 | 2015-12-11 | Dolby Lab Licensing Corp | Encoding, decoding, and representing high dynamic range images |
CN103493490B (en) | 2011-04-25 | 2016-12-28 | 杜比实验室特许公司 | Non-linear VDR residual quantizer |
EP2702766B1 (en) * | 2011-04-28 | 2017-06-14 | Koninklijke Philips N.V. | Apparatuses and methods for hdr image encoding and decoding |
US10097822B2 (en) | 2011-05-10 | 2018-10-09 | Koninklijke Philips N.V. | High dynamic range image signal generation and processing |
HUE058712T2 (en) | 2011-05-27 | 2022-09-28 | Dolby Laboratories Licensing Corp | Scalable systems for controlling color management comprising varying levels of metadata |
US20120324492A1 (en) | 2011-06-20 | 2012-12-20 | Microsoft Corporation | Video selection based on environmental sensing |
JP5747711B2 (en) * | 2011-07-22 | 2015-07-15 | 富士通株式会社 | Network evaluation device |
WO2013039730A2 (en) | 2011-09-15 | 2013-03-21 | Dolby Laboratories Licensing Corporation | Method and system for backward compatible, extended dynamic range encoding of video |
CN103827956B (en) | 2011-09-27 | 2017-06-13 | 皇家飞利浦有限公司 | For the apparatus and method of the dynamic range transform of image |
JP5829758B2 (en) | 2011-10-20 | 2015-12-09 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Method and system for video equalization |
KR101812469B1 (en) | 2011-12-06 | 2017-12-27 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Method of improving the perceptual luminance nonlinearity-based image data exchange across different display capabilities |
US8773543B2 (en) | 2012-01-27 | 2014-07-08 | Nokia Corporation | Method and apparatus for image data transfer in digital photographing |
KR101793628B1 (en) | 2012-04-08 | 2017-11-06 | 삼성전자주식회사 | Transparent display apparatus and method thereof |
JP5598769B2 (en) | 2012-04-16 | 2014-10-01 | コニカミノルタ株式会社 | Color adjustment method and color adjustment program |
US20140092108A1 (en) | 2012-05-24 | 2014-04-03 | Popslate Media, Inc. | Dynamically Changeable Decorative-Protective Accessory for a Mobile Computing Device |
US9258641B2 (en) * | 2012-06-07 | 2016-02-09 | Qbiz, Llc | Method and system of audio capture based on logarithmic conversion |
US9330606B2 (en) | 2012-06-08 | 2016-05-03 | Apple Inc. | Electronic device with display brightness control |
AR091515A1 (en) | 2012-06-29 | 2015-02-11 | Sony Corp | DEVICE AND METHOD FOR IMAGE PROCESSING |
US9516351B2 (en) * | 2012-07-13 | 2016-12-06 | Koninklijke Philips N.V. | HDR image encoding and decoding methods and devices |
US8821281B2 (en) | 2012-07-17 | 2014-09-02 | International Business Machines Corporation | Detection of an orientation of a game player relative to a screen |
RU2615224C2 (en) | 2012-09-28 | 2017-04-04 | Интел Корпорейшн | Methods and systems of dynamic control of display device brightness basing on content |
US9064313B2 (en) * | 2012-09-28 | 2015-06-23 | Intel Corporation | Adaptive tone map to a region of interest to yield a low dynamic range image |
CN103077696B (en) | 2013-01-15 | 2015-08-12 | Opera软件股份公司 | For regulating method and apparatus and the display system of brightness of display screen |
CN104956670B (en) * | 2013-01-25 | 2017-09-08 | 杜比实验室特许公司 | Light modulation based on global display management |
US9058667B2 (en) | 2013-01-30 | 2015-06-16 | Apple Inc. | Detail preserving image transfer operations |
RU2633128C2 (en) * | 2013-02-21 | 2017-10-11 | Конинклейке Филипс Н.В. | Improved methods and devices for coding and decoding hdr images |
TWI473039B (en) | 2013-03-05 | 2015-02-11 | Univ Tamkang | Method and image processing device for image dynamic range compression with local contrast enhancement |
GB201304221D0 (en) | 2013-03-08 | 2013-04-24 | Univ Warwick | High Dynamic Range Imaging Systems |
TWI639151B (en) | 2013-03-15 | 2018-10-21 | 美商傲思丹度科技公司 | Dynamic gamut display systems, methods, and applications thereof |
US20140321561A1 (en) | 2013-04-26 | 2014-10-30 | DDD IP Ventures, Ltd. | System and method for depth based adaptive streaming of video information |
KR102031580B1 (en) | 2013-05-10 | 2019-11-08 | 엘지디스플레이 주식회사 | Display apparatus and display apparatus control method |
CN103391435A (en) * | 2013-07-02 | 2013-11-13 | 广东工业大学 | Coding method for high dynamic range (HDR) image compatible with low dynamic range (LDR) and decoding method thereof |
JP6351313B2 (en) | 2013-07-11 | 2018-07-04 | キヤノン株式会社 | Image encoding device, image decoding device, image processing device, and control method thereof |
CN105393525B (en) | 2013-07-18 | 2019-04-12 | 皇家飞利浦有限公司 | Create the method and apparatus of the code mapping function for encoding to HDR image and the method and apparatus using such coded image |
US20150130967A1 (en) * | 2013-11-13 | 2015-05-14 | Nvidia Corporation | Adaptive dynamic range imaging |
US9973723B2 (en) | 2014-02-24 | 2018-05-15 | Apple Inc. | User interface and graphics composition with high dynamic range video |
EP3829171A1 (en) | 2014-02-25 | 2021-06-02 | Apple Inc. | Adaptive transfer function for video encoding and decoding |
JP6731722B2 (en) * | 2015-05-12 | 2020-07-29 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Display method and display device |
ES2684432T3 (en) * | 2015-12-17 | 2018-10-02 | Koninklijke Philips N.V. | Dynamic range coding for images and video |
US10200687B2 (en) * | 2017-06-02 | 2019-02-05 | Apple Inc. | Sample adaptive offset for high dynamic range (HDR) video compression |
US10212456B2 (en) * | 2017-06-02 | 2019-02-19 | Apple Inc. | Deblocking filter for high dynamic range (HDR) video |
-
2015
- 2015-02-25 EP EP20201343.9A patent/EP3829171A1/en active Pending
- 2015-02-25 KR KR1020167022526A patent/KR101898643B1/en active IP Right Grant
- 2015-02-25 US US14/631,394 patent/US10271054B2/en active Active
- 2015-02-25 AU AU2015223123A patent/AU2015223123A1/en not_active Abandoned
- 2015-02-25 KR KR1020197027139A patent/KR20190109579A/en not_active Application Discontinuation
- 2015-02-25 JP JP2016548661A patent/JP6383425B2/en active Active
- 2015-02-25 DE DE112015000950.7T patent/DE112015000950T5/en active Pending
- 2015-02-25 KR KR1020237008260A patent/KR102654563B1/en active IP Right Grant
- 2015-02-25 CN CN202010303152.4A patent/CN111526350B/en active Active
- 2015-02-25 CN CN202010008012.4A patent/CN111246050B/en active Active
- 2015-02-25 US US14/631,401 patent/US10212429B2/en active Active
- 2015-02-25 CN CN201580010095.8A patent/CN106031172B/en active Active
- 2015-02-25 CN CN201580010103.9A patent/CN106030503B/en active Active
- 2015-02-25 CN CN201910771191.4A patent/CN110365974A/en active Pending
- 2015-02-25 US US14/631,405 patent/US10264266B2/en active Active
- 2015-02-25 KR KR1020207028453A patent/KR102330316B1/en active IP Right Grant
- 2015-02-25 US US14/631,398 patent/US10880549B2/en active Active
- 2015-02-25 KR KR1020187025884A patent/KR102025037B1/en active IP Right Grant
- 2015-02-25 KR KR1020227021690A patent/KR102509533B1/en active IP Right Grant
- 2015-02-25 DE DE112015000959.0T patent/DE112015000959T5/en active Pending
- 2015-02-25 EP EP15710636.0A patent/EP3111644A1/en not_active Withdrawn
- 2015-02-25 CN CN201580008321.9A patent/CN106233706B/en active Active
- 2015-02-25 US US14/631,410 patent/US10812801B2/en active Active
- 2015-02-25 CN CN202310395840.1A patent/CN116320394A/en active Pending
- 2015-02-25 KR KR1020217037741A patent/KR102414567B1/en active IP Right Grant
- 2015-02-25 WO PCT/US2015/017536 patent/WO2015130793A1/en active Application Filing
- 2015-02-25 WO PCT/US2015/017539 patent/WO2015130796A1/en active Application Filing
- 2015-02-25 WO PCT/US2015/017540 patent/WO2015130797A1/en active Application Filing
-
2018
- 2018-07-27 AU AU2018208733A patent/AU2018208733B2/en active Active
- 2018-08-03 JP JP2018146796A patent/JP7038021B2/en active Active
-
2019
- 2019-02-15 US US16/277,892 patent/US10986345B2/en active Active
-
2020
- 2020-02-13 AU AU2020201023A patent/AU2020201023B2/en active Active
- 2020-09-23 JP JP2020158908A patent/JP2021010171A/en active Pending
- 2020-10-16 US US17/073,263 patent/US11445202B2/en active Active
- 2020-12-23 US US17/132,922 patent/US20210227226A1/en not_active Abandoned
-
2021
- 2021-04-16 US US17/233,099 patent/US20210235093A1/en active Pending
- 2021-06-23 AU AU2021204255A patent/AU2021204255B2/en active Active
- 2021-10-11 JP JP2021166907A patent/JP7429676B2/en active Active
-
2022
- 2022-08-26 US US17/897,079 patent/US20220408096A1/en active Pending
-
2023
- 2023-02-10 AU AU2023200732A patent/AU2023200732B2/en active Active
- 2023-10-13 JP JP2023177736A patent/JP2024009972A/en active Pending
Patent Citations (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5731818A (en) * | 1994-04-19 | 1998-03-24 | Eastman Kodak Company | Method and apparatus for constrained gamut clipping |
US20020196470A1 (en) * | 2001-05-24 | 2002-12-26 | Hiroyuki Kawamoto | Image processing method and apparatus and image forming apparatus for reducing moire fringes in output image |
US8115787B2 (en) * | 2001-06-07 | 2012-02-14 | Genoa Color Technologies Ltd. | Device, system and method of data conversion for wide gamut displays |
US20040109180A1 (en) * | 2002-12-04 | 2004-06-10 | Eastman Kodak Company | Color gamut mapping using a cost function |
US20070104378A1 (en) * | 2003-03-05 | 2007-05-10 | Seadragon Software, Inc. | Method for encoding and serving geospatial or other vector data as images |
US7508981B2 (en) * | 2004-12-15 | 2009-03-24 | Samsung Electronics Co., Ltd. | Dual layer bus architecture for system-on-a-chip |
US20070035751A1 (en) * | 2005-08-15 | 2007-02-15 | Microsoft Corporation | Gamut shape dependent color management system |
US20100172411A1 (en) * | 2006-01-23 | 2010-07-08 | Alexander Efremov | High dynamic range codecs |
US20080043268A1 (en) * | 2006-08-16 | 2008-02-21 | Samsung Electronics Co., Ltd. | Apparatus, method and medium outputting wide gamut space image |
US20080112024A1 (en) * | 2006-11-09 | 2008-05-15 | Seiko Epson Corporation | Image processing apparatus, image processing method, image processing program, recording medium storing image processing program, and image display apparatus |
US20080137990A1 (en) * | 2006-12-06 | 2008-06-12 | Brightside Technologies Inc. | Representing and reconstructing high dynamic range images |
US20090027558A1 (en) * | 2007-07-27 | 2009-01-29 | Rafal Mantiuk | Apparatus and Method for Rendering High Dynamic Range Images for Standard Dynamic Range Display |
US20100157154A1 (en) * | 2007-08-02 | 2010-06-24 | Sony Corporation | Image processing apparatus |
US20100053651A1 (en) * | 2008-08-27 | 2010-03-04 | Seiko Epson Corporation | Image Processing Device and Image Processing Method |
US20100157078A1 (en) * | 2008-12-19 | 2010-06-24 | Qualcomm Incorporated | High dynamic range image combining |
US20110194618A1 (en) * | 2009-03-13 | 2011-08-11 | Dolby Laboratories Licensing Corporation | Compatible compression of high dynamic range, visual dynamic range, and wide color gamut video |
US20100245857A1 (en) * | 2009-03-27 | 2010-09-30 | Konica Minolta Systems Laboratory, Inc. | Systems and Methods for Processing Color Data |
US20110243473A1 (en) * | 2010-04-01 | 2011-10-06 | Himax Media Solutions, Inc. | Image enhancement method and apparatuses utilizing the same |
US20110292992A1 (en) * | 2010-05-28 | 2011-12-01 | Microsoft Corporation | Automating dynamic information insertion into video |
US20130076763A1 (en) * | 2010-06-08 | 2013-03-28 | Dolby Laboratories Licensing Corporation | Tone and Gamut Mapping Methods and Apparatus |
US20130129214A1 (en) * | 2010-08-04 | 2013-05-23 | Nec Corporation | Image processing method, image processing apparatus, and image processing program |
US20140022460A1 (en) * | 2011-04-12 | 2014-01-23 | Dolby Laboratories Licensing Corporation | Quality Assessment for Images that Have Extended Dynamic Ranges or Wide Color Gamuts |
US20120314944A1 (en) * | 2011-06-13 | 2012-12-13 | Dolby Laboratories Licensing Corporation | High dynamic range, backwards-compatible, digital cinema |
US20130003086A1 (en) * | 2011-06-28 | 2013-01-03 | Konica Minolta Laboratory U.S.A., Inc. | Method for processing high dynamic range images using tone mapping to extended rgb space |
US20130076974A1 (en) * | 2011-09-26 | 2013-03-28 | Dolby Laboratories Licensing Corporation | Image Formats and Related Methods and Apparatuses |
US20140241418A1 (en) * | 2011-11-09 | 2014-08-28 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | Inter-layer prediction between layers of different dynamic sample value range |
US20130308027A1 (en) * | 2012-05-03 | 2013-11-21 | Aptina Imaging Corporation | Systems and methods for generating metadata in stacked-chip imaging systems |
US20130321671A1 (en) * | 2012-05-31 | 2013-12-05 | Apple Inc. | Systems and method for reducing fixed pattern noise in image data |
US20130335439A1 (en) * | 2012-06-18 | 2013-12-19 | Samsung Display Co., Ltd. | System and method for converting color gamut |
US9076224B1 (en) * | 2012-08-08 | 2015-07-07 | Dolby Laboratories Licensing Corporation | Image processing for HDR images |
US20140267822A1 (en) * | 2013-03-12 | 2014-09-18 | Stmicroelectronics (Grenoble 2) Sas | Tone mapping method |
US8866975B1 (en) * | 2013-05-02 | 2014-10-21 | Dolby Laboratories Licensing Corporation | Backwards-compatible delivery of digital cinema content with higher dynamic range and related preprocessing and coding methods |
US20160156965A1 (en) * | 2013-07-14 | 2016-06-02 | Lg Electronics Inc. | Method and apparatus for transmitting and receiving ultra high-definition broadcasting signal for expressing high-quality color in digital broadcasting system |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11445202B2 (en) | 2014-02-25 | 2022-09-13 | Apple Inc. | Adaptive transfer function for video encoding and decoding |
US20220351341A1 (en) * | 2021-04-30 | 2022-11-03 | Realnetworks, Inc. | Intelligent metadata service for video enhancement |
US11544826B2 (en) * | 2021-04-30 | 2023-01-03 | Realnetworks, Inc. | Intelligent metadata service for video enhancement |
WO2023055075A1 (en) * | 2021-09-30 | 2023-04-06 | Samsung Electronics Co., Ltd. | Standard dynamic range (sdr) / hybrid log-gamma (hlg) with high dynamic range (hdr) 10+ |
US11991412B2 (en) | 2021-09-30 | 2024-05-21 | Samsung Electronics Co., Ltd. | Standard dynamic range (SDR) / hybrid log-gamma (HLG) with high dynamic range (HDR) 10+ |
Also Published As
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10986345B2 (en) | Backward-compatible video capture and distribution |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |