EP4702531A1

EP4702531A1 - Enhancement of image data based on inference images

Info

Publication number: EP4702531A1
Application number: EP24705867.0A
Authority: EP
Inventors: Bang Sian LIU; Shang Chih Chuang; Jyun-Kai Hu
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-01-10
Filing date: 2024-01-10
Publication date: 2026-03-04
Also published as: WO2025151116A1; CN121909483A; TW202531124A

Abstract

This document describes systems and techniques for using an inference image generated by down-scaling image data as a basis for generating enhancement data for selectively adjusting one or more characteristics of at least one aspect of the image data. The inference image has a reduced resolution comprising inference regions that each represent multiple pixels of the image data. The inference regions are analyzed to identify one or more aspects of the image data to one or more enhancement interfaces to generate enhancement data to adjust at least one characteristic of the one or more aspects. The inference image is analyzed with an image recognition module to generate semantic data that is used by an image enhancement interface to generate enhancement data to adjust at least one characteristic of at least one aspect of the image data.

Description

ENHANCEMENT OF IMAGE DATA BASED ON INFERENCE IMAGES

BACKGROUND

[0001] Individuals increasingly use their mobile telephones and other mobile devices for viewing images and watching video. Mobile devices are readily available to access image and video data wherever the user may be. Further, improving wireless communications speeds, increasing battery life, and improving display resolution all sen e to make viewing image video data on mobile devices increasingly more inviting.

[0002] Mobile devices may employ techniques to increase brightness, heighten sharpness, or adjust colors in an attempt to improve image quality of video data. However, the use of automatic image enhancement in mobile devices may present some disadvantages. For example, applying image enhancement to all aspects of a frame of video data may result in undesirable effects. For example, brightening an image may improve the appearance of some colors but may result in other colors taking an unnatural tone. Similarly, sharpening features of an image may cause some aspects of the image to appear more vivid while sharpening features of other subjects, such as human faces, may result in those human faces having an unnatural appearance.

[0003] At least some of these issues may be solved by using artificial intelligence (A.I.) models or other image recognition modules to differentiate aspects of a frame of image data to support selective enhancement of different aspects of the image to enhance some aspects of the image, sharpen or otherwise enhance some of the same or other aspects, etc. However, image recognition modules may be processing-intensive. On a mobile device, such as a mobile telephone, execution of image recognition modules may consume significant processing resources. Correspondingly, execution of image recognition modules may consume significant quantities of electrical power. In a mobile device in which electrical power is provided by a battery, electrical power may be a limited resource that should be conserved when possible.

SUMMARY

[0004] This document describes systems and techniques for using an inference image generated by down-scaling image data as a basis for generating enhancement data for selectively adjusting one or more characteristics of at least one aspect of the image data. The inference image has a reduced resolution including inference regions that each represent multiple pixels of the image data. The inference regions are analyzed to determine an aspect of the image data that is useful to an enhancement interface. These aspects are then passed to the enhancement interface to generate enhancement data to adjust a characteristic of the aspect. [0005] For example, a method includes receiving a frame of image data and down-scaling the frame of image data to generate an inference image. The inference image has a reduced resolution compared to the frame of image data and has a number of inference regions representing portions of the frame of image data. The inference image is analyzed with an image recognition module configured to detect at least one aspect for enhancement and to generate semantic data identifying the at least one aspect in one or more of the inference regions of the inference image. The semantic data is presented to an image enhancement interface configured to generate enhancement data to adjust at least one characteristic of the at least one aspect. The enhancement data is presented to one or more image enhancement modules associated with a display processing unit, the one or more image enhancement modules being configured to adjust the frame of image data responsive to the enhancement data. In other examples, a system includes means for performing the method and a program includes computer-executable instructions to cause a computing system to perform the method.

[0006] This Summary is provided to introduce systems and techniques for using an inference image as a basis for generating enhancement data for selectively adjusting a characteristic of an aspect of the image data, as further described below in the Detailed Description and Drawings. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The details of one or more aspects of systems and techniques for using an inference image generated by down-scaling image data as a basis for generating enhancement data for selectively adjusting one or more characteristics of at least one aspect of the image data are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:

[0008] FIG. 1 is a block diagram of a system for enhancing image data based on an inference image generated by down-scaling the image data using a common interface;

[0009] FIG. 2 is a schematic diagram of a frame of image data being down-scaled to generate an inference image;

[0010] FIG. 3 is a schematic diagram showing inference regions of an inference image corresponding to portions of the frame of image data if FIG. 2 from which the inference image was derived;

[0011] FIG. 4 is a schematic diagram showing enhancement of a characteristic of an aspect identified in the inference image derived from the frame of image data; [0012] FIG. 5 is a block diagram of the system of FIG. 1 adapted for performing local tone mapping;

[0013] FIG. 6 is a schematic diagram of the system of FIG. 5 performing local tone mapping;

[0014] FIG. 7 is a block diagram of the system of FIG. 1 adapted for performing sharpness enhancement;

[0015] FIG. 8 is a schematic diagram of the system of FIG. 7 performing sharpness enhancement;

[0016] FIG. 9 is a schematic diagram of a mask used in FIG. 8 being adapted to exclude portions of a human face from sharpness enhancement;

[0017] FIG. 10 is a block diagram of the system of FIG. 1 adapted for performing color look-up table enhancement;

[0018] FIG. 11 is a schematic diagram of the system of FIG. 10 performing color look-up table enhancement;

[0019] FIG. 12 is a block diagram of the system of FIG. 1 in which a common interface includes multiple image enhancement interfaces;

[0020] FIG. 13 is a schematic diagram of the system of FIG. 12 applying multiple image enhancement processes to adjust characteristics of aspects of the image data;

[0021] FIG. 14 is a block diagram of the common interface of FIG. 12 and image enhancement modules configured to apply additional image enhancement techniques;

[0022] FIG. 15 is a block diagram of a hardware-based image enhancement interface;

[0023] FIG. 16 is a block diagram of a software-based image enhancement interface configured to execute in a general-purpose computing system;

[0024] FIG. 17 is a block diagram of the system of FIG. 1 adapted for latency synchronization in generating image enhancement data; and

[0025] FIG. 18 is a flow diagram of an example method of enhancing image data using an inference image.

DETAILED DESCRIPTION

OVERVIEW

[0026] Some computing devices provide the capability to automatically enhance frames of image data. For example, these computing devices can automatically adjust colors to make the sky bluer or grass greener or may automatically sharpen textures of plants or other objects. In automatically adjusting these aspects within the image data, the resulting images may present images with more vivid and eye-catching colors in sharp, non-blurry images. [0027] However, if these image enhancement techniques are not applied selectively, the resulting images may be changed for the worse. For example, enhancement of human faces presents a challenge. While enhancing the blues of the sky or the green of the grass may make an image more vivid, correspondingly deepening or brightening the color of a human face may result in the human face appearing in a noticeably unnatural tone. Similarly, while sharpening the texture of a tree may present a more-appealing image, sharpening of eyes or other features of a human face may yield an unnatural or unpleasant rendition of the subject.

[0028] To avoid the appearance of potentially unnatural colors or potentially unwanted sharpening, image data may be preprocessed to identify human faces in the image to selectively control the application of enhancement to the human face. However, a frame of high-resolution image data may be many megabytes in size. On a mobile telephone or other mobile device, processing a frame of image data of that size - let alone a sequence of such frames included in a video stream - consumes significant processing resources and electrical power.

[0029] This document describes systems and techniques for using an inference image generated by down-scaling image data as a basis for generating enhancement data by which to selectively adjust a characteristics of an aspect of the image data. The inference image has a reduced resolution including inference regions that each represent multiple pixels of the image data. The inference regions are analyzed to identify an aspect of the image data, which is then used by an enhancement interface to generate enhancement data by which to adjust a characteristic of the aspect.

SYSTEM FOR SELECTIVE IMAGE ENHANCEMENT

[0030] FIG. 1 illustrates a system 100 that processes image data 102, including one or more frames of image data, to facilitate application of image enhancement to the image data 102. The image data 102 may be stored in an image buffer 104 for processing by a display processing unit 106, which includes one or more image enhancement modules 108. The image enhancement modules 108 adjusts one or more characteristics, such as changing a color, sharpening a feature, or performing other operations. By so doing, the techniques present the image data 102 in a visually more-pleasing way. However, as previously described, it may be desirable to control how the image enhancement modules 108 are applied to aspects of the image data 102 to provide desirable image enhancement without causing unnatural skin tones, undesirably sharpened facial features, or other potentially undesirable results. In implementations, the system 100 may be operated on a mobile telephone, or other mobile device. The mobile device may be a battery' powered mobile device. [0031] In implementations, the system 100 includes a preprocessing system 110 that includes a down -seal er 112 configured to down-scale the image data 102 to a lower-resolution inference image 114. The inference image 114 represents each of the aspects included in the image data 102, but at a lower resolution. The resulting inference image 114 results in a body of data that is significantly smaller than the original image data 102. The inference image 114 is then stored in memory, e.g., an inference buffer 116.

[0032] As previously described, tasking an image recognition module 118 with processing large files represented by full-resolution image data 102 is a resource-intensive operation that consumes significant processing and electrical power. An object of using the down-scaler 112 to generate the inference image 1 14 is to provide a much smaller data file for the image recognition module 118 to process. The inference image 114, although lower in resolution than the original image data 102, is representative of the image data 102 and represents the same aspects captured in the image data 102 with generally the same characteristics, such as brightness and color, as in the image data 102.

[0033] By analog}', the inference image 114 may be regarded as a thumbnail image representing an image file stored in data storage. A thumbnail image does not show a whole image file but represents aspects included in the image file to a degree that enables a user perusing the data storage to be able to identify an image file of interest. The user may not have to see the entire image file in order to identify which image file is of interest; the scaled-down thumbnail shows enough aspects of image files stored in the data storage to identify image files of interest.

[0034] In a corresponding manner, the inference image 114 represents the image data 102 in a way that allows the image recognition module 118 to identify aspects in the image data 102 for enhancement without the image recognition module 118 having to evaluate the image data 102 in full. The image recognition module 118 can identify' aspects for potential enhancement in the scaled-down inference image 114 just as a user can identify' an image of interest from a scaled- down thumbnail image (which may be low er resolution, smaller in size, or both). Then, from the inference image 114, the image recognition module 118 may generate semantic data 120 that provides a basis for the image enhancement modules 108 of the display processing unit 106 to desirably enhance various aspects included in the image data 102. As further described below', elements of the inference image 114 are mapped to elements of the image data 102, enabling image enhancement to be applied to aspects of the image according to the semantic data 120 generated by the image recognition module 118.

[0035] The image recognition module 118 may include any form of inference system, artificial intelligence (A.I.) system, or machine learning system that is configured to identify' aspects included in the image data 102. The image recognition module 118 may be configured or programmed to identify data values and data patterns in the image data 102 that the image recognition module 118 is configured to equate with particular aspects that may be included in the image data 102, such as human faces, plant life, bodies of water, the sky, or other aspects for which representative values and patterns have been equated with the different aspects. As will be understood by those ordinarily skilled in the art. machine learning is performed by data collection to gather training images that may include representations of aspects to be identified in the image data 102 and performing model training by labelling aspects in the training images. Through iteration, the image recognition module 118 thus may be trained to identify that, when aspects in received image data 102 include particular values (or values within particular ranges) and/or in particular patterns, the presence of the particular values and/or particular patterns in the image data 102 indicate the presence of an aspect the image recognition module 118 should recognize.

[0036] Actual implementation of the image recognition module 118 may be performed using various data processing configurations, such as convolutional neural networks, support vector machines, feature extraction based on oriented gradients, texture analysis based on local binary patterns, and other programmable recognition technologies. The image recognition module 118 may be implemented in the form of computer-executable instructions to be performed by a general-purpose computing system or the image recognition module may be implemented in the form of an application-specific processing device configured to identify or infer the presence of various aspects that may be included in the image data 102.

[0037] In implementations, the down-scaler 112 may represent each of the picture elements (pixels) at a lower bit representation (e.g., reducing each pixel to an 8-bit representation from a higher representation). For example, implementations may also reduce the overall image resolution to a 320-by-240-point image from original, higher-resolution image data, with elements of the lower resolution mapping to multiple elements of the original image data, as described further below.

[0038] The inference image 114 may be presented to the image recognition module 118 to enable the image recognition module 118 to detect at least one aspect for enhancement in the image data 102 by the image enhancement modules 108 of the display processing unit 106. The image recognition module 118 generates semantic data 120 that identifies aspects in the inference image 114, such as identifying of human faces or other aspects, for potential image enhancement. As previously, described, in processing the image data 102, the image recognition module 118 may consume significant amounts of processing and electrical power in generating the semantic data 120. However, the image recognition module 118 consumes much less processing power and electrical power in processing the inference image 114 because the inference image 114 is a much smaller body of data than the image data 102. [0039] One or more image enhancement interfaces 124 may be included in a common interface 122. The common interface 122 receives the semantic data 120 and, in response to content of the semantic data 120, generates enhancement data 126 that will be presented to one or more of the image enhancement modules 108 to enhance the image data 102 stored in the image buffer 104. The one or more image enhancement interfaces 124 direct adjustment of a characteristic, such as a brightness, color, or sharpness of one or more aspects of the image data 102, by the corresponding image enhancement modules 108.

[0040] As described further below, the system 100 also may include a synchronization capability that enables the image enhancement interfaces 124 to communicate with the image buffer 104 to correlate the image data 102 with corresponding enhancement data 126. Preparation of enhancement data 126 by the image enhancement interfaces 124 may take a non-negligible amount of time. Accordingly, the image enhancement interfaces 124 may cause image data 102 stored in the image buffer 104 to be held until the corresponding enhancement data 126 is presented to the display processing unit 106 for processing by the image enhancement modules 108. Synchronizing the image buffer 104 with the image enhancement interface 124 ensures that the enhancement data 126 is applied by the image enhancement module 108 to corresponding image data 102, as further described below;

GENERATION AND USE OF AN INFERENCE IMAGE

[0041] FIG. 2 illustrates an example of how- the image data 102 is dow n-scaled to generate the inference image 114 that enables the image recognition module 118 to identify aspects of the image data 102 in the inference image 114. The image data 102 is represented as a grid 200 of image elements 202. The inference image 114 is represented as an inference grid 204 of inference elements 206. Each of the inference elements 206 in the inference grid 204 of the inference image 114 encompasses multiple image elements 202 of the grid 200 of the image data 102. Nonetheless, the dow-n-scaled inference grid 204 still spans or represents an entirety of the image data 102, with various inference elements of the inference image 114 corresponding with or mapping to groups of image elements of the image data 102.

[0042] For example, a first inference element 208 in the inference grid 204 of the inference image 114 corresponds with a first group of image elements 210 in the grid 200 of the image data 102. Similarly, a second inference element 212 in the inference grid 204 corresponds with a second group of image elements 214 in the grid 200, and a third inference element 216 in the inference grid 204 corresponds with a third group of image elements 218 in the grid 200.

[0043] In the example of FIG. 2, each of the inference elements 206, 208, 212, and 214 encompasses or corresponds with a four-by-four array of image elements 202, but this representation is selected purely for the sake of illustration. A 16: 1 dow n-scale ratio applied by the down-scaler 112 may be selected to balance granularity of the inference image 114 against a desired use of processing and electrical power resources. Even in this simple example, the five- by-five grid 204 of the inference image 114 includes 25 inference elements, which is one-sixteenth the number of 400 image elements 202 included in the 25-by-25 grid 200 of the image data 102. For a high-resolution image, the down-scale ratio may be higher so that the image recognition module 118 operates on an inference image 114 in which each of the inference elements 206 represents a larger group of image elements 202, resulting in the inference image 114 being much less than one-sixteenth the size of the image data 102. It will be appreciated that the larger the down-scale ratio used by the down-scaler 112, the smaller the inference image 114 will be compared to the image data 102 and the greater the reduction in processing and electrical power resources used by the image recognition module 118 in processing the inference image 114. The down-scale ratio may be selected to enable the image recognition module 118 to identify aspects of a particular size in proportion to the image data 102 as a whole while still seeking to minimize the use of processing and electrical power resources.

[0044] FIG. 3 shows an example of an image file 300, which, as in the example of FIG. 2, includes a grid 302 of image elements 304. The image file 300 is down-scaled to an inference image 306 that includes an inference grid 308 of inference elements 310. The image file 300 includes two identifiable aspects: a human face 312 and a tree 314. As previously described, it may be desirable to enhance the color or sharpness of some aspects in the image data 300 in a particular way. For example, it may be desirable to sharpen features of the tree 314 and/or to make the tree 314 a darker or more vibrant shade of green to make the appearance of the image data 300 more pleasing. However, sharpening, coloring, or otherwise changing attributes of the human face 312 may result in eyes of the human face 312 appearing unnatural or the human face 312 being presented in an unnatural color. The image recognition module 118 (see FIGS. 1 and 2) may be programmed to identify and differentiate the different aspects 312 and 314 of the image data 300 to sharpen and enhance the color of the tree 314 while avoiding similar, potentially undesirable enhancement of the human face 312.

[0045] The inference image 306 reduces the processing and electrical powder resources used by the image recognition module 118 by presenting the image recognition module 118 with a smaller data file to process that is still representative of the image data 300. For example, a first inference element 316 corresponds to a first group of image elements 318 that encompasses the human face 312. A second inference element 320 corresponds to a second group of image elements 322 that encompasses the tree 314. The human face 312 or the tree 314 may each span multiple inference elements 310 (although each are shown in FIG. 3 as being substantially encompassed in just a single, respective, inference element 316, 320). However, the number of inference elements 310 in the inference image 306 will still be less than (or at a lower resolution than) the number of image elements 304 included in the image data 300, thereby resulting in potentially orders of magnitude less data for the image recognition module 118 to process in identifying which of the aspects 312 and 314 should be subjected to image enhancement and/or how the aspects 312 and 314 should be enhanced (and which are not enhanced or are enhanced differently).

[0046] Referring to FIG. 4, the image data 300 is processed to generate enhanced image data 400 based on the image recognition module 118 (see FIGS. 1 and 2) processing the inference image 306 to determine whether to and/or how to enhance the aspects 312 and 314 in the enhanced image data 400. In a simplified example, based on the inference image 306, the image recognition module 118 is able to recognize the tree 314 as a suitable aspect for sharpening and enhancement. As a result, the image recognition module 118 generates semantic data 120 that differentiates the tree 314 from other aspects of the image and, thus, causes the image enhancement modules 108 to both sharpen and adjust a color of the tree 314 to present an enhanced view 402 of the tree 314 (in which an outline of the tree 314 is darkened and the tree 314 is cross-hatched in the enhanced view 402 to represent its adjusted color). At the same time, based on the inference image 306, in this example, the image recognition module 118 determines that the human face 312 should neither be sharpened nor have its color adjusted. Thus, the enhanced image data 400 presents the human face 312 in its original form. The enhanced image data 400 may include the same enhancements (or exclude the same areas for enhancement) as if the image recognition module 118 analyzed the image data 102 in full. However, by analyzing the inference image 114 generated by the down-scaler 112, fewer processing and electrical power resources are consumed in generating the enhanced image data 400.

IMAGE ENHANCEMENT IMPLEMENTATIONS

[0047] Based on the image recognition module 118 using the inference image 114 to generate the semantic data 120, one or more of multiple different image enhancements may be applied to the image data 102. As previously described, although some image enhancement techniques may be unsuitable for human faces, implementations may employ local tone mapping to desirably enhance human faces in the image data 102. Sharpness enhancement may be selectively applied to other aspects of the image data 102 to improve the appearance of the image data 102. Similarly, color look-up enhancement may be used to adjust colors within the image data 102, particularly with aspects other than human faces. One or more of these enhancements or other enhancements may be used. [0048] Referring to FIG. 5, to selectively enhance human faces identified by the A.I module 118 (see FIGS. 3, 4, 6, 8, 9, 11, and 13) in the inference image 114, the common interface 122 in a system 500 includes a local tone-mapping interface 502 and the display processing unit 106 includes a local tone-mapping module 504. The local tone-mapping interface 502 uses the semantic data 120 to generate enhancement data 126 that adjusts aspects of the image data 102 that, from the inference image 114, are determined to include human faces. The enhancement data 126 is then presented to the local tone-mapping module 504 to cause the local tone-mapping module 504 to enhance the image data 102 stored in the image buffer 104.

[0049] FIG. 6 shows the enhancement of a frame of image data 600 to enhance a human face 602 included in the image data 600 using the local tone-mapping interface 502 and the local tone-mapping module 504. The image data 600 also includes a tree 604 that may be identified by the image recognition module 118. The image recognition module 118 may identify the tree 604 for enhancement by one or more other image enhancement interfaces and image enhancement modules. However, for purposes of this example, the local tone-mapping interface 502 ignores the tree 604 and other aspects of the image data 600 other than the human face 602.

[0050] From the image data 600, a reduced-size inference image 606 is generated by the down-scaler 112 (see FIG. 5). From the inference image 606, the image recognition module 118 identifies a bounding block 608 of inference regions 610 that includes the face 602 and a block 612 of inference regions 614 that includes the tree 604, although, as previously described, the local tone-mapping interface 502 ignores aspects of the image data 600 other than the human face 602. The image recognition module 118 thus flags the bounding block 608 of the inference regions 610 for processing by the local tone-mapping interface 502 in semantic data 616 that may be used by the local tone-mapping interface 502 to adjust the image data 600.

[0051] In the example of FIG. 6, the human face 602 includes a shaded region 618 that is darker than the rest of the human face 602, which may be an artifact of lighting conditions represented in the image data 600. It may be desirable to brighten or lighten the shaded region 618, which is a function of the local tone-mapping interface 502 and the local tone-mapping module 504.

[0052] In one implementation, the local tone-mapping interface 502 employs bilateral grid local tone mapping to brighten the shaded region 618 of the human face 602 in the image data 600. Directed to the bounding block 608 of the image data 600 in which the human face 602 is located, the local tone-mapping interface 502 generates a histogram 620 according to a brightness index 622, a luma index 624, and a width index 626 for various aspects 628, 630, and 632 in the image data 600 in which, for example, the aspect 632 represents the human face 602. A cumulative distribution function 634 is then generated that represents a portion of pixels with an intensity that is equal to or less than a specific value.

[0053] For example, in the bounding block 608 corresponding to the location of the human face 602, the cumulative distribution function 634 includes values 636 and 638 that correspond to normally illuminated portions of the human face 602 and values 640, 642, 644. and 646 that correspond to the shaded region 618 of the human face. The local tone-mapping interface 502 may generate enhancement data 126 (FIG. 5) to be presented to the local tone-mapping module 504 to adjust the values 640, 642, 644, and 646 corresponding to the shaded region 618 of the human face 602. A skin-tone labeling technique may be applied to control the local tone-mapping to ensure that the human face 602 remains within a range determined to present the human face 602 in realistic shades.

[0054] As a result, in adjusted image data 648, the previously shaded region 618 is lightened to match the rest of the human face 602. However, because the image recognition module 118 was seeking to identify human faces in the image data 600, the tree 604 was not flagged for adjustment and, thus, the appearance of the tree 604 remains the same in the adjusted image data 648.

[0055] Referring to FIG. 7, to selectively sharpen features of human faces identified by the A.I module 118 in the inference image 114, the common interface 122 in a system 700 includes a sharpness enhancement interface 702 and the display processing unit 106 includes a sharpness enhancement module 704. The sharpness enhancement interface 702 uses the semantic data 120 to generate enhancement data 126 that adjusts aspects of the image data 102 that, from the inference image 114. are determined to include human faces. The enhancement data 126 is then presented to the sharpness enhancement module 704 to cause the sharpness enhancement module 704 to enhance the image data 102 stored in the image buffer 104.

[0056] FIG. 8 shows an inference image 800 (generated from image data 102, as previously described with reference to FIGS. 5 and 6) including representations of a human face 802 and a tree 804 for which sharpness enhancement is desired. The image recognition module 118 may identify' both the human face 802 and the tree 804, marking each with bounding blocks 806 and 808, respectively. The human face 802 includes some attributes that could be sharpened, such as lips 810, a nose 812, and eyebrows 814. Similarly, edges 816 of the tree 804 are not sharply defined (as represented with the edges 816 being in dotted lines in FIG. 8).

[0057] In implementations, the sharpness enhancement interface 702 generates a grid 818 in which inference regions 820, 822, 824, 826, 828, and 830 including the bounding block 806 encompassing the human face 802 and inference regions 832, 834, 836, 838, 840, and 842 including the bounding block 808 encompassing the tree 804 are analyzed to identify what portion of the respective inference regions is filled with the human face 802 or the tree 804. For example, the inference regions 836 and 838 specify 100%, indicating that each is filled with the human face 802, while the inference regions 840 and 842 specify 50%, indicating that each is only half-filled with the human face 802.

[0058] The sharpness enhancement interface 702 translates the grid 818 into a guidance map 844 that represents the extent to which each of the aspects 802 and 804 fill the respective inference regions. The resulting guidance map 844 includes masks 846 and 848 for the human face 802 and the tree 804, respectively. The masks 846 and 848 are included in the enhancement data 126 presented to the sharpness enhancement module 704 and, thus, direct a degree of sharpness that the sharpness enhancement module 704 applies to the aspects 802 and 804, respectively. The more the inference regions 820, 822, 824, 826, 828, and 830 around the human face 802 or the inference regions 832, 834, 836, 838, 840, and 842 around the tree 804 are determined to be filled by the respective aspects 802 and 804, the higher a degree of sharpness the sharpness enhancement module 704 applies. By applying the masks 846 and 848, enhanced image data 850 generated by the sharpness enhancement module 704 includes an enhanced human face 852 including features wi th visually sharpened features, including enhanced lips 854, an enhanced nose 856, and enhanced eyebrows 858. Similarly, an enhanced tree 860 is presented with a sharpened texture 862 and a sharpened outline 864 of the enhanced tree 860.

[0059] As previously described, it may be undesirable to enhance a sharpness of human eyes because such sharpening may result in an undesirable appearance. Accordingly, referring to FIG. 9, in implementations, the image recognition module 118 (see FIG. 7) may flag areas in the semantic data 120 in which eyes 900 appear in the human face 802 in the image data 800 for exclusion from image sharpening. As a result, a guidance map 902 may include a same mask 848 for the tree 804 (see FIG. 8) but may generate an adjusted mask 904 in which a mask 906 for the human face 802 includes exclusion zones 908 for the eyes 900. As a result, the sharpness enhancement module 704 does not sharpen the eyes 900 from how the eyes 900 appear in the image data 800. The sharpness enhancement module 704 may generate enhanced image data 910 that includes other visually sharpened features, such as the enhanced lips 854, enhanced nose 856, and enhanced eyebrows 858, but without enhancing the appearance of the eyes 900.

[0060] Referring to FIG. 10, in a system 1000, to selectively enhance colors in the image data 102. the common interface 122 includes a color look-up table enhancement interface 1002 and the display processing unit 106 includes a color look-up enhancement module 1004. The color look-up table enhancement interface 1002 uses the semantic data 120 to generate enhancement data 126 that adjusts some colors appearing in the image data 102 without unnaturally adjusting colors determined from the inference image 114 to include human faces or other aspects of human subjects. The enhancement data 126 is then presented to the color lookup table enhancement module 1004 to enhance the image data 102 stored in the image buffer 104.

[0061] FIG. 11 shows an inference image 1100 (generated from image data 102, as previously described with reference to FIGS. 5 and 6) including representations of a human face 1102 and a tree 1104. Color enhancement is desired for the image data 102 (see FIG. 10) from which the inference image 1 100 is derived but without unnaturally adjusting colors of the human face 1102, while enhancing colors of the tree 1104, clothing 1106, and a field 1108 and sky 1110 in the background. In a portrait segmentation map 1112, the image recognition module 118 may identify the human face 1102 and generate a portrait map 1114 that flags the human face 1102 to exclude or limit color enhancement of the human face 1102 included in the portrait map 1 114. The portrait segmentation map 1112 may be included in the semantic data 120 presented to the color look-up table enhancement interface 1002 to prevent the colors of the human face 1102 from being adjusted in an unnatural way.

[0062] In a color boost map 1116 used by the color look-up table enhancement interface 1002, different inference regions are associated with colors identified in the inference image 1100. For example, inference regions 1118 that correspond with the tree 1104 are identified as “G” for green. Inference regions 1120 that correspond with the field 1108 are identified as “Y” for yellow. Inference regions 1122 that correspond with the clothing 1106 are identified as “R” for red. Inference regions 1124 that correspond with the sky 1110 are identified as “B” for blue. Inference regions 1126 including the human face 1102 as identified by the portrait map 1114 are labeled as “F’’ for flesh and, responsive to the portrait segmentation map 1112, are excluded from the ty pes of color enhancement that will be applied to other aspects in the inference regions 1118, 1120, 1122, and 1124. The color boost map 1116 is provided by the color look-up interface 1002 to the color look-up enhancement module 1004.

[0063] In enhanced image data 1128 generated by the color look-up interface module 1004, colors of the inference regions 1118, 1120, 1122, and 1124 not including the human face 1102 are adjusted by, for example, cross-referencing the colors identified in the color boost map 1116 and associating them with enhanced colors that are substituted for the original colors included in the image data 102 and represented in the inference image 1100. Thus, for example, the tree 1104 is presented in an enhanced shade of green 1130 (as represented, along with other enhanced colors in the enhanced image data 1128, with a different fill pattern than that used in the inference image 1100 derived from the image data 102). The clothing 1106 is presented in an enhanced shade of red 1 132. The field 1108 is represented in an enhanced shade of yellow 1 134. The sky 1110 is presented in an enhanced shade of blue 1136. However, to avoid making the human face 1102 appear unnatural, the color of the human face 1102 is unchanged. IMPLEMENTATIONS OF IMAGE ENHANCEMENT INTERFACES

[0064] FIG. 12 shows a system in which the common interface 122 includes multiple image enhancement interfaces 124, including the local tone-mapping interface 502 (see FIG. 5), the sharpness enhancement interface 702 (see FIG. 7). and the color look-up table interface 1002 (see FIG. 10). Correspondingly, the display processing unit 106 includes multiple, corresponding image enhancement modules 108 including the local tone-mapping module 504, the sharpness enhancement module 704, and the color look-up table module 1004. The image enhancement interfaces 502, 702, and 1002 receive the semantic data 120 generated by the image recognition module 118 and perform respective processing tasks to generate the enhancement data 126 that is used by the image enhancement modules 504, 704, and 1004 as described with respect to FIG. 6, 8-9, and 11. The common interface 122 thus receives the semantic data 120 from the image recognition module 118 and provides data to multiple image enhancement interfaces 124 for each of multiple image enhancement modules 108. The enhancement data 126 may include a combined data file including the enhancement data 126 generated by each of the multiple image enhancement interfaces 124 or may generate separate data files directed to corresponding modules of the image enhancement modules 108.

[0065] The image recognition module 118 may process the inference image 114 to efficiently provide the semantic data 120 that ultimately is used to perform multiple enhancements of the image data 102 stored in the image buffer 104. Referring to FIG. 13, image data 1300 includes aspects that merit being enhanced as described with reference to FIGS. 6, 8, 9, and 11. For example, the image data 1300 includes ahuman face 1302 that includes the shaded region 618 of the image data 600 of FIG. 6. The human face 1302 includes the lips 810, the nose 812. and the eyebrows 814 that should be sharpened, as well as a tree 1304 with an indistinct form or the edges 816 that should be sharpened, as described with reference to FIG. 8. Colors of aspects of the image data 1300, such as the color of the tree 1104, the clothing 1106, the field 1108, and the sky 1110, as described with reference to FIG. 11, should be enhanced.

[0066] The image enhancement interfaces 502, 702. and 1002 of the common interface 122 receive the semantic data 120 generated by the image recognition module 118 and each perform respective processing tasks to generate the enhancement data 126 that is used by the respective image enhancement modules 504, 704, and 1004 as described with respect to FIG. 6, 8-9. and 11. As a result, enhanced image data 1306 includes a human face 1308 that no longer includes the shaded region 618, as described with reference to FIG. 6. The enhanced image data 1306 includes visually sharpened features, including the enhanced lips 854, enhanced nose 856, and enhanced eyebrows 858 (but optionally without sharpening the eyes 900 themselves, as described with reference to FIG. 9), and a tree 1310 has the enhanced texture 862 and outline 864 as described with reference to FIG. 8. The enhanced image data 1306 features the enhanced colors of the tree 1130, the clothing 1132, the field 1 134, and the sky 1136. Thus, the common interface 122 enables each of the image enhancement interfaces 124 to act individually or in parallel with others of the image enhancement interfaces 124 to receive the semantic data 120 from the image recognition module 118 - which may operate using the down-scaled inference image 114 - to generate the enhancement data 126 to be provided to one or more of the image enhancement modules 108 to generate enhanced image data.

[0067] The common interface 122 may include any number of image enhancement interfaces 124 to support any number of image enhancement modules 108. As shown in FIG. 14, like the example of FIG. 12, the common interface 122 may include image enhancement interfaces 124 that include the local tone-mapping interface 502, the sharpness enhancement module 702, and the color look-up table interface 1002 that provide enhancement data 126 with the local tonemapping module 404, the sharpness enhancement module 704, and the color look-up table module 1004 of the image enhancement modules 108. In addition, the tap-in module 122 may include a signal -to-noise ratio control interface 1400 that provides enhancement data 126 to a signal -to- noise ratio control module 1402. A convolutional sparse coding interface 1404 may be included to provide enhancement data 126 to a convolutional sparse coding module 1406. A color correction matrix interface 1408 may be included to provide enhancement data 126 to a color correction matrix module 1410. A local contrast enhancement interface 1412 may be included to provide enhancement data 126 to a local contrast enhancement module 1414. A low-pass filter blur reduction interface 1416 may be included to provide enhancement data 126 to a low-pass filter blur reduction module 1418. The common interface 122 also may include other image enhancement interfaces 124 that may provide enhancement data 126 to other image enhancement modules 108.

[0068] The common interface 122 may be either hardware-based or software-based. FIG. 15 depicts a hardware-based common interface 1500 including image enhancement interface hardware 1502. The image enhancement interface hardware 1502 may include a specially configured system-on-chip (SoC) device that includes processing capabilities, memory, storage, and supporting logic to process the semantic data 120 generated by the image recognition module 118 (see FIGS. 1, 5, 7, 10, and 12) and generate the enhancement data 126 to direct the one or more image enhancement modules 108. The image enhancement interface hardware 1502 also may include a graphical processing unit, a tensor processing unit, or another processing system and supporting components to perform the processing capabilities to process the semantic data 120 generated by the image recognition module 118 and generate the enhancement data 126 as previously described. The image enhancement interface hardware 1502 may be configured to perform the functions of the local tone-mapping interface 502, the sharpness enhancement module 602, and the color look-up table interface 1002, or other image enhancement interfaces 124 described with reference to FIG. 14.

[0069] Alternatively, as shown in FIG. 16. functions of the common interface 122 may include a software-based common interface 1600 that executes on a general -purpose computing system 1602, such as a main processing system of a mobile device. The computing system 1602 includes a processor 1604, memory' 1606, and storage 1608 where the storage 1608 stores image enhancement interface instructions 1610 executable by the processor 1604. In the software-based common interface 1600, the instructions 1610 are configured to direct the processor 1604 to process the semantic data 120 generated by the image recognition module 118 and generate the enhancement data 126 to direct the one or more image enhancement modules 108. Like the hardware-based common interface 1500, the software-based common interface 1600 may be configured to perform the functions of the local tone-mapping interface 502, the sharpness enhancement module 602, and the color look-up table interface 1002, or other image enhancement interfaces described with reference to FIG. 14. The processor 1604 may be a multiple-core, highspeed processor that enables the processor 1 04 to perform other functions of the device in which it is included as well as those of the software-based common interface 1600.

SYNCHRONIZATION OF IMAGE ENHANCEMENT

[0070] Processing time may be allowed for the image recognition module 118 (see FIGS. 1 , 5, 7, 10, and 12) to process the image data 102 to generate the semantic data 120 and for the one or more image enhancement interfaces 124 of the common interface 122 (see FIG. 1) to generate the enhancement data 126. As a result, a system 1700 that processes a frame of image data for frame N (fx) 1702 may include an image buffer with latency synchronization 1704 to selectively delay presentation of the frame of image data for frame N (fx) 1702 until the enhancement data 126 for the frame of image data for frame N (fx) 1702 is ready to be applied at the one or more image enhancement interfaces 124. Accordingly, the common interface 122 may include image enhancement interfaces with latency synchronization 1706 that are configured to provide a latency signal 1708 to the image buffer with latency synchronization 1704.

[0071] For example, when the image enhancement interfaces 1706 complete processing of enhancement data 1710 for a frame of image data for frame N -X (fx-x) 1712, which is one or more frames earlier in the sequence than the frame of image data for frame N (fx) 1702, the one or more image enhancement interfaces 1706 may communicate via the latency signal 1708 that enhancement data for a frame of image dataN-X (fx-x) 1712 is ready. Responsive to the latency signal 1708, the image buffer with latency synchronization 1704 presents the image data for the frame of image dataN-X (fN-x) 1712 so that the image enhancement modules 108 may apply the enhancement data 1710 to the corresponding frame of image data 1712. The image buffer 1704 will hold the image data for the frame N (fx) 1702 until the latency signal 1708 indicates that the enhancement data 1710 for the frame of image data for frame N (fx) 1702 is ready to be applied at the one or more image enhancement modules 108. Thus, delivery of the image data 1712 to the image enhancement modules 108 of the display processing unit 106 is synchronized to correspond with presentation of the enhancement data 1710 for the corresponding frame of image data 1712.

EXAMPLE METHOD OF ENHANCING IMAGE DATA USING AN INFERENCE IMAGE

[0072] FIG. 18 illustrates an example method 1800 of enhancing image data using an inference image. At a block 1802, a frame of image data 102 (see FIG. 1) is received. The frame of image data 102 may be received from a codec or other input for presentation via a display processing unit 106. At a block 1804, the frame of image data 102 is down-scaled to generate an inference image 114, the inference image 114 having a reduced resolution compared to the image data 102 and having a number of inference regions representing portions of the image data 102. As previously described, use of the inference image 114 presents a smaller body of data for processing by an image recognition module 118, reducing consumption of processing power (and corresponding electrical power) used to generate semantic data 120.

[0073] At a block 1806, the inference image 114 is analyzed with the image recognition 118 module configured to detect at least one aspect for enhancement and to generate the semantic data 120 identifying the at least one aspect in one or more of the inference regions of the inference image 114. As previously described, the semantic data 120 may identify a presence of a human face or another aspect for enhancement such as sharpening, color adjustment, or other enhancements. At a block 1808, the semantic data 120 is presented to an image enhancement interface 124 configured to generate enhancement data 126 to adjust at least one characteristic of the at least one aspect. The characteristic may include brightness, a degree of sharpening, a color, or another characteristic of the at least one aspect. At a block 1810, the enhancement data 126 is presented to one or more image enhancement modules 108 associated with a display processing unit 106, the one or more image enhancement modules 108 being configured to adjust the frame of image data 102 responsive to the enhancement data 126.

[0074] Unless context dictates otherwise, use herein of the word "‘of’ may be considered use of an “inclusive or,” or a term that permits inclusion or application of one or more items that are linked by the word “or” (e.g., a phrase “A or B” may be interpreted as permitting just “A,” as permitting j ust “B,” or as permitting both “A" and “B”). Also, as used herein, a phrase referring to ‘'at least one of’ a list of items refers to any combination of those items, including single members. For instance, “at least one of a, b, or c” can cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c. or any other ordering of a. b, and c). Further, items represented in the accompanying figures and terms discussed herein may be indicative of one or more items or terms, and thus reference may be made interchangeably to single or plural forms of the items and terms in this written description.

ADDITIONAL EXAMPLES

[0075] In the following section, additional examples are provided.

[0076] Example 1 : A method comprising receiving a frame of image data; down-scaling the frame of image data to generate an inference image, the inference image having a reduced resolution compared to the frame of image data and having a number of inference regions representing portions of the frame of image data: analyzing the inference image with an image recognition module configured to detect at least one aspect for enhancement and to generate semantic data identifying the at least one aspect in one or more of the inference regions of the inference image; presenting the semantic data to at least one image enhancement interface configured to generate enhancement data to adjust at least one characteristic of the at least one aspect; and presenting the enhancement data to one or more image enhancement modules associated with a display processing unit, the one or more image enhancement modules being configured to adjust the frame of image data responsive to the enhancement data.

[0077] Example 2: The method of example 1, wherein the at least one image enhancement interface includes a local tone-mapping interface.

[0078] Example 3: The method of example 2, wherein the semantic data presented to the local tone-mapping interface includes a bounding block to identify a human face and the local tone-mapping interface applies bilateral grid tone mapping to adjust a brightness of one or more regions of the human face.

[0079] Example 4: The method of any preceding example, wherein the at least one image enhancement interface includes a sharpness enhancement interface.

[0080] Example 5: The method of example 4, wherein the sharpness enhancement interface determines an extent to which the at least one aspect fills the one or more inference regions in which the at least one aspect is located to generate a guidance map indicating a degree of sharpness to be applied to the at least one aspect. [0081] Example 6: The method of example 4 or 5, wherein the sharpness enhancement interface is configured to exclude one or more portions of the at least one aspect from sharpness enhancement.

[0082] Example 7: The method of any one of examples 1 to 6, wherein the image enhancement interface includes a color look-up table interface.

[0083] Example 8: The method of example 7, wherein the semantic data includes a portrait segmentation map including a portrait map identifying a human face and the color look-up table interface is responsive to the portrait segmentation map to maintain a color of the human face during color enhancement.

[0084] Example 9: The method of any one of examples 1 to 8, wherein the at least one image enhancement interface and the one or more image enhancement modules include one or more of: a signal-to-noise ratio control interface in communication with a signal-to-noise control module associated with the display processing unit; a convolutional sparse coding interface in communication with a convolutional sparse coding module associated with the display processing unit; a color correction matrix interface in communication with a color correction matrix module associated with the display processing unit; a local contrast enhancement interface in communication with a local contrast enhancement module associated with the display processing unit; and a low-pass filter blur reduction interface in communication with a low-pass filter blur reduction module associated with the display processing unit.

[0085] Example 10: The method of any one of examples 1 to 9, wherein the at least one image enhancement interface includes a plurality⁷ of image enhancement interfaces collected in a common interface configured to receive the semantic data from the image recognition module and to generate the enhancement data for the one or more image enhancement modules.

[0086] Example 11 : The method of example 10, wherein the common interface includes a dedicated hardware device configured to generate the enhancement data or includes computerexecutable instructions configured to operate in a computing system to generate the enhancement data.

[0087] Example 12: The method of any preceding example, further comprising synchronizing the enhancement data generated by the at least one image enhancement interface to correspond with the frame of image data for which the enhancement data is generated.

[0088] Example 13: The method of example 12, further comprising the at least one image enhancement interface generating a latency signal to identify a frame of image data stored in an image buffer for which the enhancement data is generated by the at least one image enhancement interface. [0089] Example 14: The method of any preceding example, comprising presenting the semantic data to a plurality of image enhancement interfaces, each being configured to generate enhancement data to adjust at least one characteristic of the at least one aspect; and presenting the enhancement data to a corresponding plurality of image enhancement modules, each of the plurality of image enhancement modules being configured to adjust the frame of image data responsive to the enhancement data.

[0090] Example 15: A system comprising means for performing a method of any one of examples 1 through 14.

[0091] Example 16: The system of example 15, comprising: a down-scaler configured to receive a frame of image data and to down-scale the frame of image data to generate an inference image, the inference image having a reduced resolution compared to the frame of image data and having a number of inference regions representing portions of the frame of image data; an image recognition module configured to analyze the inference image and to detect at least one aspect for enhancement and to generate semantic data identifying the at least one aspect in one or more of the inference regions of the inference image; at least one image enhancement interface configured to receive the semantic data and to generate enhancement data to adjust at least one characteristic of the at least one aspect; and one or more image enhancement modules associated with a display processing unit configured to receive the enhancement data and to adjust the frame of image data responsive to the enhancement data.

[0092] Example 17: The system of example 16, comprising an image buffer configured to store frames of image data.

[0093] Example 18: The system of example 16 or 17, comprising a display processing unit, the display processing unit comprising the one or more image enhancement modules, and being configured to process frames of image data.

[0094] Example 19: The system of any of examples 16 to 18, comprising a plurality⁷ of image enhancement interfaces, and a corresponding plurality of image enhancement modules.

[0095] Example 20: A program for causing a computer to execute the method recited in any one of examples 1 through 14.

CONCLUSION

[0096] Although implementations of systems and techniques for using an inference image as a basis for generating enhancement data for selectively adjusting a characteristic of an aspect of image data have been described in language specific to certain features and/or methods, the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of the described systems and techniques.

Claims

CLAIMS What is claimed is:

1. A method comprising: receiving a frame of image data; down-scaling the frame of image data to generate an inference image, the inference image having a reduced resolution compared to the frame of image data and having a number of inference regions representing portions of the frame of image data; analyzing the inference image with an image recognition module configured to detect at least one aspect for enhancement and to generate semantic data identifying the at least one aspect in one or more of the inference regions of the inference image; presenting the semantic data to at least one image enhancement interface configured to generate enhancement data to adjust at least one characteristic of the at least one aspect; and presenting the enhancement data to one or more image enhancement modules associated with a display processing unit, the one or more image enhancement modules being configured to adjust the frame of image data responsive to the enhancement data.

2. The method of claim 1, wherein the at least one image enhancement interface includes a local tone-mapping interface.

3. The method of claim 2, wherein the semantic data presented to the local tonemapping interface includes a bounding block to identify a human face and the local tone-mapping interface applies bilateral grid tone mapping to adjust a brightness of one or more regions of the human face.

4. The method of any preceding claim, wherein the at least one image enhancement interface includes a sharpness enhancement interface.

5. The method of claim 4, wherein the sharpness enhancement interface determines an extent to which the at least one aspect fills the one or more inference regions in which the at least one aspect is located to generate a guidance map indicating a degree of sharpness to be applied to the at least one aspect.

6. The method of claim 4 or 5, wherein the sharpness enhancement interface is configured to exclude one or more portions of the at least one aspect from sharpness enhancement.

7. The method of any one of claims 1-6, wherein the image enhancement interface includes a color look-up table interface.

8. The method of claim 7, wherein the semantic data includes a portrait segmentation map including a portrait map identifying a human face and the color look-up table interface is responsive to the portrait segmentation map to maintain a color of the human face during color enhancement.

9. The method of any one of claims 1-8, wherein the at least one image enhancement interface and the one or more image enhancement modules include one or more of: a signal-to-noise ratio control interface in communication with a signal-to-noise control module associated with the display processing unit; a convolutional sparse coding interface in communication with a convolutional sparse coding module associated with the display processing unit; a color correction matrix interface in communication with a color correction matrix module associated with the display processing unit; a local contrast enhancement interface in communication with a local contrast enhancement module associated with the display processing unit; and a low-pass filter blur reduction interface in communication with a low-pass filter blur reduction module associated with the display processing unit.

10. The method of any one of claims 1-9, wherein the at least one image enhancement interface includes a plurality of image enhancement interfaces collected in a common interface configured to receive the semantic data from the image recognition module and to generate the enhancement data for the one or more image enhancement modules.

11. The method of claim 10, wherein the common interface includes a dedicated hardware device configured to generate the enhancement data or includes computer-executable instructions configured to operate in a computing system to generate the enhancement data.

12. The method of any preceding claim, further comprising synchronizing the enhancement data generated by the at least one image enhancement interface to correspond with the frame of image data for which the enhancement data is generated.

13. The method of claim 12, further comprising the at least one image enhancement interface generating a latency signal to retrieve a frame of image data stored in an image buffer for which the enhancement data is generated by the at least one image enhancement interface.

14. A system comprising means for performing a method of any one of claims 1 through 13.

15. A program for causing a computer to execute the method recited in any one of claims 1 through 13.