AU2015271935A1 - Measure of image region visual information - Google Patents

Abstract

- 27 MEASURE OF IMAGE REGION VISUAL INFORMATION A computer-implementable method (420, 300) of determining visual information of regions of an image receives receiving (310) the image and a corresponding depth map for the image. The method generates (330) a background-degraded image from the received image by degrading background regions of the image based on the depth in the corresponding depth map, and generates (320) a foreground-degraded image from the received image by degrading foreground regions of the image based on the depth in the corresponding depth map. The method then compares (340) each of the background-degraded image and the foreground degraded image with the image to calculate visual information in the background and foreground regions of the image respectively. 10R2S55 1 P1S0340 snr Iodge 500 (620) Start Receive image, depth 510 map, desired filters and filter strengths Apply filters to 520 generate modified image I (display modified image) Calculate features 530 Estimate impact on 540 foreground/background separation D Ip 550 Display estimated impact Fig5599 End Fig. 5

Description

-1- 2015271935 18 Dec 2015

MEASURE OF IMAGE REGION VISUAL INFORMATION

TECHNICAL FIELD

[001] This invention relates to the field of image processing and, in particular, to methods of modifying images to improve perceptual or aesthetic quality.

BACKGROUND

[002] The prevalence of digital photography has made image processing technology which improves the visual and aesthetic quality of images available to an increasing range of users. Improving an image however is still a time consuming task and one which requires a great deal of knowledge to perfect. Many different image filters are available in existing photo editing tools, and even in the cases where the behaviour of such filters can be easily described, their aesthetic impact on a particular image may be far from obvious without much trial and error. The increasing number of different image filters available further increases the complexity, as combinations of filters may interact in unexpected ways both computationally or aesthetically.

[003] Typically, a user would easily be able to describe their aesthetic goal for an image in natural language. However, translating this into an appropriate combination of filters for a particular image can be difficult. To aid this, tools are being created with increasingly abstract descriptions, thereby making the tools tied more to aesthetic goals the user can understand, at the potential cost of obscuring the fine details of what is being done to the image. These tools help users more efficiently achieve their aesthetic goals by simplifying the translation of those goals into concepts provided by the tools.

[004] One valuable high level concept is the sense of foreground/background separation in an image. Many images have a single subject, to which the viewer is meant to be drawn, with the background simply providing context. For some of these images, processing the images to make the foreground subject stand out more from the background, while ideally still retaining a natural appearance, improves the aesthetic value of the image. This is not limited to images of a single subject. Some images may have multiple foreground objects, or a range of objects between the foreground and background, but still benefit from having the perceived separation between depth layers increased. 10825256 1 P150340_spec_lodge -2- 2015271935 18 Dec 2015 [005] In order to provide high level tools which efficiently increase the perceived foreground/background separation, an image processing system needs to understand the impact of the available filters on this perception. This requires models to be created which understand the relationship between filters, properties of the images they are applied to and perceived foreground background/separation of the resulting filtered images. To aid the creation of these models, features are needed which can capture the effects of a filter that relate to the sense of foreground/background separation for a particular image.

SUMMARY

[006] Presently disclosed is a method of calculating an amount of visual information in a region. An image and a region map are received, the region map describing the location of the region of the image for which to calculate the amount of visual information. A region-degraded image is created by degrading the image in the received region, for example by taking the weighted average between the image and neutral grey, the weight corresponding to the level of inclusion in the region for each pixel. The loss of visual information resulting from the degradation is calculated with an image quality measure such as Visual Information Fidelity. The measured loss of information resulting from the degradation indicates the amount of visual information present in the image in the given region.

[007] The visual information in a region may in turn be used to estimate a foreground/background information difference for an image. This may be performed by measuring the visual information in the foreground region, followed by measuring the visual information in the background region, and comparing the two.

[008] Calculating a foreground/background visual information difference for both an image and a fdtered version of that image allows the two foreground/background visual information differences to be compared. This combined with other measures indicates the impact of the fdter on perceived foreground/background separation in the image.

[009] According to one aspect of the present disclosure there is provided a processor-implementable method of determining visual information of regions of an image, the method comprising: receiving the image and a corresponding depth map for the image; generating a background-degraded image from the received image by degrading background regions of the image based on the depth in the corresponding depth map; generating a foreground-degraded 10825256 1 P150340_spec_lodge -3- 2015271935 18 Dec 2015 image from the received image by degrading foreground regions of the image based on the depth in the corresponding depth map; and comparing each of the background-degraded image and the foreground-degraded image with the image to calculate visual information in the background and foreground regions of the image respectively.

[0010] Preferably the degrading is performed in proportion to the depth of the region in the corresponding depth map.

[0011] Desirably, the degrading comprises greying out the corresponding region in proportion to the depth in the corresponding depth map. More specifically, the greying out comprises modifying the image by a weighted average of the image with neutral grey.

[0012] More generally, the degrading can be one of greying out, blurring, whitening, and darkening [0013] A specific implementation further comprises: (i) applying a filter to the image to generate a filtered image; (ii) generating a background-degraded filtered image and a foreground-degraded filtered image by greying out background and foreground regions of the filtered image respectively, the greying out of the regions being proportional to the depth in the corresponding depth map; (iii) comparing each of the background-degraded filtered image and the foreground-degraded filtered image to the filtered image to calculate visual information in the background and foreground of the filtered image respectively; and (iv) determining an effect of the applied filter by comparing the visual information of the image and the visual information of the filtered image.

[0014] In other implementations, the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.

[0015] In other implementations, the determining an effect of the applied filter is performed by adding the filtered foreground image comparison value to the background image comparison value, and subtracting the filtered background image comparison and the foreground image comparison values. More desirably, the comparing of the background-degraded image and the foreground degraded image is performed using the Visual 10825256 1 P150340_spec_lodge -4- 2015271935 18 Dec 2015

Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.

[0016] In other arrangements, the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.

[0017] Other configurations further comprises choosing a different level of strength of the filter to apply to the image based on the determined effect of the filter applied.

[0018] Some configurations further comprise (v) applying steps (i) to (iv) for at least one additional filter; and (vi) choosing which filter to apply based on the determined effects of each filter.

[0019] Other configurations further comprise (v) applying steps (i) to (iv) for at least one additional filter; and (vi) choosing different levels of strengths to apply with each filter based on the determined effects of each filter.

[0020] Other aspects are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0021] At least one embodiment of the present invention will now be described with reference to the following drawings, in which: [0022] Figs. 1A and IB collectively form a schematic block diagram representation of a computer system upon which the presently disclosed arrangements may be implemented; [0023] Fig. 2 is a schematic flow diagram illustrating a method of calculating the amount of visual information in a region of an image according to one implementation; [0024] Fig. 3 is a schematic flow diagram illustrating a method of calculating a visual information difference between the foreground and background of an image; 10825256 1 P150340_spec_lodge -5- 2015271935 18 Dec 2015 [0025] Fig. 4 is a schematic flow diagram illustrating a method of estimating the change in foreground/background visual information difference between an original and a modified image; [0026] Fig. 5 is a schematic flow diagram illustrating a method of calculating and displaying the impact of a filter on an image; [0027] Fig. 6 is a schematic flow diagram illustrating a method of choosing filters to apply to an image; [0028] Fig. 7 is an illustration of an interface for modifying an image, including a number of filters as well as a display indicating the estimated sense of background/foreground separation resulting from applying those filters; [0029] Figs. 8A and 8B show an example of an image and an associated depth map, the depth map indicating approximate and relative depth rather than a physical distance; [0030] Figs. 9 A and 9B show an example of an image and a degraded version of that image created in accordance with the depth map in Fig. 8B; [0031] Fig 10 is an illustration of an interface for modifying an image to affect the sense of foreground/background separation.

DETAILED DESCRIPTION INCLUDING BEST MODE

Physical Implementation [0032] Figs. 1A and IB depict a general-purpose computer system 100, upon which the various arrangements described can be practiced.

[0033] As seen in Fig. 1A, the computer system 100 includes: a computer module 101; input devices such as a keyboard 102, a mouse pointer device 103, a scanner 126, a camera 127, and a microphone 180; and output devices including a printer 115, a display device 114 and loudspeakers 117. An external Modulator-Demodulator (Modem) transceiver device 116 may be used by the computer module 101 for communicating to and from a communications network 120 via a connection 121. The communications network 120 may 10825256 1 P150340_spec_lodge -6 - 2015271935 18 Dec 2015 be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 121 is a telephone line, the modem 116 may be a traditional “dial-up” modem. Alternatively, where the connection 121 is a high capacity (e.g., cable) connection, the modem 116 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 120.

[0034] The computer module 101 typically includes at least one processor unit 105, and a memory unit 106. For example, the memory unit 106 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 101 also includes an number of input/output (I/O) interfaces including: an audiovideo interface 107 that couples to the video display 114, loudspeakers 117 and microphone 180; an I/O interface 113 that couples to the keyboard 102, mouse 103, scanner 126, camera 127 and optionally a joystick or other human interface device (not illustrated); and an interface 108 for the external modem 116 and printer 115. In some implementations, the modem 116 may be incorporated within the computer module 101, for example within the interface 108. The computer module 101 also has a local network interface 111, which permits coupling of the computer system 100 via a connection 123 to a local-area communications network 122, known as a Local Area Network (LAN). As illustrated in Fig. 1 A, the local communications network 122 may also couple to the wide network 120 via a connection 124, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 111 may comprise an Ethernet circuit card, a Bluetooth™ wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 111.

[0035] The I/O interfaces 108 and 113 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 109 are provided and typically include a hard disk drive (HDD) 110. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 112 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e g., CD-ROM, DVD, Blu-ray Disc™), USB-RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the system 100. 10825256 1 P150340_spec_lodge -7- 2015271935 18 Dec 2015 [0036] The components 105 to 113 of the computer module 101 typically communicate via an interconnected bus 104 and in a manner that results in a conventional mode of operation of the computer system 100 known to those in the relevant art. For example, the processor 105 is coupled to the system bus 104 using a connection 118.

Likewise, the memory 106 and optical disk drive 112 are coupled to the system bus 104 by connections 119. Examples of computers on which the described arrangements can be practised include IBM-PC’s and compatibles, Sun Sparcstations, Apple Mac™ or a like computer systems.

[0037] The methods of determining visual information of regions of an image and associated image processing to be described may be implemented using the computer system 100 wherein the processes of Figs. 2 to 10, to be described, may be implemented as one or more software application programs 133 executable within the computer system 100.

In particular, the steps of the image processing methods are effected by instructions 131 (see Fig. IB) in the software 133 that are carried out within the computer system 100. The software instructions 131 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the visual information determinations and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0038] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 100 from the computer readable medium, and then executed by the computer system 100. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 100 preferably effects an advantageous apparatus for determining visual information of regions of an image.

[0039] The software 133 is typically stored in the HDD 110 or the memory 106. The software is loaded into the computer system 100 from a computer readable medium, and executed by the computer system 100. Thus, for example, the software 133 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 125 that is read by the optical disk drive 112. A computer readable medium having such software or computer program recorded 10825256 1 P150340_spec_lodge -8- 2015271935 18 Dec 2015 on it is a computer program product. The use of the computer program product in the computer system 100 preferably effects an apparatus for determining visual information of regions of an image.

[0040] In some instances, the application programs 133 may be supplied to the user encoded on one or more CD-ROMs 125 and read via the corresponding drive 112, or alternatively may be read by the user from the networks 120 or 122. Still further, the software can also be loaded into the computer system 100 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and/or data to the computer system 100 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray Disc™, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 101. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computer module 101 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0041] The second part of the application programs 133 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 114. Through manipulation of typically the keyboard 102 and the mouse 103, a user of the computer system 100 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 117 and user voice commands input via the microphone 180.

[0042] Fig. IB is a detailed schematic block diagram of the processor 105 and a “memory” 134. The memory 134 represents a logical aggregation of all the memory modules (including the HDD 109 and semiconductor memory 106) that can be accessed by the computer module 101 in Fig. 1A. 10825256 1 P150340_spec_lodge -9- 2015271935 18 Dec 2015 [0043] When the computer module 101 is initially powered up, a power-on self-test (POST) program 150 executes. The POST program 150 is typically stored in a ROM 149 of the semiconductor memory 106 of Fig. 1A. A hardware device such as the ROM 149 storing software is sometimes referred to as firmware. The POST program 150 examines hardware within the computer module 101 to ensure proper functioning and typically checks the processor 105, the memory 134 (109, 106), and a basic input-output systems software (BIOS) module 151, also typically stored in the ROM 149, for correct operation. Once the POST program 150 has run successfully, the BIOS 151 activates the hard disk drive 110 of Fig. 1A. Activation of the hard disk drive 110 causes a bootstrap loader program 152 that is resident on the hard disk drive 110 to execute via the processor 105. This loads an operating system 153 into the RAM memory 106, upon which the operating system 153 commences operation. The operating system 153 is a system level application, executable by the processor 105, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[0044] The operating system 153 manages the memory 134 (109, 106) to ensure that each process or application running on the computer module 101 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 100 of Fig. 1A must be used properly so that each process can run effectively. Accordingly, the aggregated memory 134 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 100 and how such is used.

[0045] As shown in Fig. IB, the processor 105 includes a number of functional modules including a control unit 139, an arithmetic logic unit (ALU) 140, and a local or internal memory 148, sometimes called a cache memory. The cache memory 148 typically includes a number of storage registers 144 - 146 in a register section. One or more internal busses 141 functionally interconnect these functional modules. The processor 105 typically also has one or more interfaces 142 for communicating with external devices via the system bus 104, using a connection 118. The memory 134 is coupled to the bus 104 using a connection 119. 10825256 1 P150340_spec_lodge -10- 2015271935 18 Dec 2015 [0046] The application program 133 includes a sequence of instructions 131 that may include conditional branch and loop instructions. The program 133 may also include data 132 which is used in execution of the program 133. The instructions 131 and the data 132 are stored in memory locations 128, 129, 130 and 135, 136, 137, respectively. Depending upon the relative size of the instructions 131 and the memory locations 128-130, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 130. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 128 and 129.

[0047] In general, the processor 105 is given a set of instructions which are executed therein. The processor 105 waits for a subsequent input, to which the processor 105 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 102, 103, data received from an external source across one of the networks 120, 122, data retrieved from one of the storage devices 106, 109 or data retrieved from a storage medium 125 inserted into the corresponding reader 112, all depicted in Fig. 1 A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 134.

[0048] The disclosed image processing arrangements use input variables 154, which are stored in the memory 134 in corresponding memory locations 155, 156, 157. The image processing arrangements produce output variables 161, which are stored in the memory 134 in corresponding memory locations 162, 163, 164. Intermediate variables 158 may be stored in memory locations 159, 160, 166 and 167.

[0049] Referring to the processor 105 of Fig. IB, the registers 144, 145, 146, the arithmetic logic unit (ALU) 140, and the control unit 139 work together to perform sequences of micro-operations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 133. Each fetch, decode, and execute cycle comprises: (i) a fetch operation, which fetches or reads an instruction 131 from a memory location 128, 129, 130; (ii) a decode operation in which the control unit 139 determines which instruction 10825256 1 P150340_spec_lodge -11- 2015271935 18 Dec 2015 has been fetched; and (iii) an execute operation in which the control unit 139 and/or the ALU 140 execute the instruction.

[0050] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 139 stores or writes a value to a memory location 132.

[0051] Each step or sub-process in the processes of Figs. 2 to 10 is associated with one or more segments of the program 133 and is performed by the register section 144, 145, 146, the ALU 140, and the control unit 139 in the processor 105 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 133.

[0052] The arrangements discussed generally herein may typically be implemented in a stand-alone computer system such as that described above and as illustrated. However other implementations include the computing arrangements being cloud-based in a server configuration accessible to network coupled computers or devices acting as a source of the original image and a destination for a final processed (separated) image.

[0053] The methods may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of image processing. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories. Such arrangement may be suited to implementation with a camera whereby the optimal separation may be tested prior to or during image capture. Subject to the detail and complexity pf the image processing load (discussed below), the camera may transfer the image to a cloud-based processing environment where one or more of the processing steps to be described may be performed.

First Implementation [0054] Fig. 7 shows an example of a GUI 700 of a system for modifying an image according to the present disclosure. The GUI 700 is typically reproduced upon the video display device 114 by operation of the processor 105 and an image 710 is displayed within the GUI 700. Slider controls 720 to 723 are formed and reproduced on the right of the image 710 10825256 1 P150340_spec_lodge -12- 2015271935 18 Dec 2015 and may be manipulated by a user to modify the image, for example by operation of the mouse pointer 103 or the keyboard 102. Each of the sliders 720-723 is configured to modify a different aspect of the image 710. The sliders 720-723, whilst only four (4) are illustrated in Fig. 7, each control corresponding common image processing methods such as sharpening, contrast adjustment, saturation, white balance, brightness, and so on. A separation display portion 730 of the GUI 700 is used to show an automatically determined estimate of the sense of foreground/background separation in the image 710 as a result of the processing applied, compared to the original, unprocessed image. The separation display 730 assists the user in finding combinations of processing obtained via the sliders 720-723 which improve the sense of foreground/background separation in the image 710, while still allowing the user to adjust the image 710 to a desired form.

[0055] In using the system, the user first loads a source image, for example retrieved from the HDD 110 by manipulating the computer 101, both to be modified and to be shown in the GUI 700 as the display image 710. The source image may be a photograph the user has taken, or another image has accessed or obtained that the user wishes to modify. Associated with the source image is a depth map, which is used for some of the calculations required for the separation display 730. The depth map may also be used by some image filters associated with the sliders 720 to 723. For example, one filter may apply sharpening to foreground regions while blurring background regions, based on the depth map provided. The depth map is provided by the user associated with the source image, for example by storing the depth map as an alpha channel alongside colour (e g. RGB, YCrCb) data of the source image, or by providing a file with a matching name in a location related to the location of the source image.

[0056] The depth information and thus the depth map may be calculated in a number of ways using techniques known in the art. These include depth from stereo (requiring at least a second image from a different but similar viewpoint), depth from defocus, depth information captured with the image using direct depth capture technology such as laser scanners, or may even have been manually created by the user.

[0057] The values of the depth map as used by the image processing filters, and as used by the methods described herein, do not need to correspond to literal depth values in the form of physical distances from the camera (e.g. camera 127) which captured the source image. The depth map is intended to make a distinction between the foreground objects 10825256 1 P150340_spec_lodge -13- 2015271935 18 Dec 2015 which are generally the subject of a captured photographic image, and the background objects which provide a setting to the subject. The distinction may be achieved with a depth map operating in a simple manner such as a binary segmentation into foreground and background.

[0058] To calculate the values reproduced as the display image 710 from the source image, a method 500 described in the flowchart of Fig. 5 is used. The method 500 is preferably implemented as software, for example as part of the application 133 which generates the GUI 700. The method 500 is executed by the processor 105. In step 510, the source (original unmodified) image is obtained by the processor 105 and for example loaded to the memory 106 ready for processing, and can form an initial image 710 displayed in the GUI 700. In step 510 the processor 105 also obtains the corresponding depth map, desired filters and level of change. The desired filters correspond to the sliders 720 to 723, and the level of change corresponds to the positions of each of the sliders 720-723.

[0059] In step 520, the processor 105 operates to apply the filters to the image to generate the modified image. This is performed according to each of the filters, in the same manner as in existing photo editing software. This results in a modified image, which is displayed as the image 710 in the GUI 700 reproduced on the display 114. The image 710 is also temporarily stored in the memory 106 for use with following calculations.

[0060] In step 530, features are calculated from the original and modified images, both of which are temporarily stored in the memory 106. Step 530 will be described in more detail further below as a preferred method 400. Although these features are described as calculated during this step, for efficiency the parts of the calculation of features which depend only on the original image are most desirably performed when the (source) image is initially loaded, rather than every time a slider is moved and the level of desired change changes. This allows a faster response time of the image processing system.

[0061] In step 540, the processor 105 calculates the impact on foreground/background separation of the filters, based on the features provided in step 530. This calculation requires a model of the perceived separation in an image. This model may be created from data collected in a psychophysical experiment in which observers are for example shown pairs of images, one original image and one modified by a combination of filters, and asked to rate which image they believe has a greater sense of separation. A rating may be provided from a 10825256 1 P150340_spec_lodge -14- 2015271935 18 Dec 2015 number of options including a small improvement in sense of separation for the left or right image, to a large improvement in sense of separation. Given a sufficient amount of data from this experiment, these ratings may be converted to a perceptual scale, using known methods such as MOS (Mean Opinion Score), or according to Torgerson’s Law of Categorical Judgment, which dates from about 1958. A relationship can be modelled between features of the original and modified images and these perceptual scale values. Analysis of one such experiment conducted by the inventors has shown that a feature representing the change, between the modified and original images, in the information difference between foreground and background, is valuable in such a model.

[0062] The output of such a model as used in step 540 is a perceptual scale value based on the features of the original and modified images.

[0063] In step 550, this perceptual scale value determined in step 540 is displayed to the user via the GUI 700, as seen in Fig. 7 as the separation display 730. The method 500 then ends at step 599.

[0064] At first the display separation values 730 may have little meaning to the user, as the range of perceptual scale values may depend on the experiment performed and there are no existing familiar units with which to describe a “sense of foreground/background separation”. However, where the user continues to modify the image 710, the method 500 repeats with each change of any one of the sliders 720-723, resulting in image changes being displayed in the image 710 of the GUI 700 and changes in the separation value 730 will become apparent. After some time, familiarity of the user can be gained with the range of values displayed, where a larger number is indicative of a higher level of image separation, and effectively forming a figure of merit for foreground/background separation is the modified instance 710 of the original source image.

[0065] The step 530 “Calculate features” will now be described in more detail, with reference to a preferred method 400 shown in Fig. 4. In step 410, the original image, modified image, and depth map are received or retrieved by the processor 105 into the memory 106 ready for processing.

[0066] In step 420, a value representing the information difference between foreground and background in the original image is calculated by the processor 105. This 10825256 1 P150340_spec_lodge -15- 2015271935 18 Dec 2015 value can be stored for example as the intermediate variable 159 and is useful as a feature for estimating the sense of separation between foreground and background. The calculation will be described in more detail as method 300 further below.

[0067] In step 430, a corresponding value representing the information difference between foreground and background in the modified image is calculated by the processor 105. This is identical to step 420 other than working with the modified image instead of the original image, and where the calculated value may be stored as the intermediate variable 160.

[0068] In step 440, the values (variables 159, 160) from steps 420 and 430 are combined to form a feature representing the change, in foreground/background information difference, between the original and modified images. This may be performed by subtracting one value from the other. Alternative comparisons such as the ratio of the values could be used. Such a feature value may be stored by the processor 105 as the intermediate variable 166. The method 400 then ends at step 499.

[0069] The steps 420 and 430 of calculating a foreground/background information difference for an image will now be described as the method 300 with reference to Fig. 3. In step 310, the image to be processed along with its associated depth map are received by the processor 105.

[0070] In step 320, a value is calculated representing the amount of visual information in the foreground regions of the image. For this calculation the depth map is used as a region map of the foreground. To do this, the depth map is normalised so that the depth map is in the range 0 to 1, and possibly inverted to ensure that foreground pixels have values close to 1 and background pixels have values close to 0. The calculation which is then performed will be described further below as method 200.

[0071] In step 330, a value is calculated representing the amount of visual information in the background regions of the image. This calculation is performed in the same manner as in step 320, but with a region map of the background rather than of the foreground. The background region map is found by inverting the region map used to represent the foreground in step 320, specifically by subtracting each region inclusion value in the map from 1. As a result, background pixels have values close to 1 and foreground pixels have values close to 0. 10825256 1 P150340_spec_lodge -16- 2015271935 18 Dec 2015 [0072] Steps 320 and 330 are independent of each other and hence may be interchanged in order in the method 300.

[0073] In step 340, the information difference between foreground and background is calculated by the processor 105. This may be performed by subtracting one value from the other. Alternative comparisons such as the ratio of the values could be used. The method 300 then ends at step 399.

[0074] The method 200 used in steps 320 and 330 will now be described, with reference to Fig. 2. In step 210, the image under consideration is provided to the processor 105 along with corresponding region map of the image to calculate the information within.

[0075] The following description treats the image as a greyscale image. For a colour image, this calculation may be performed by first converting the image to greyscale, for example by converting the image to a luminance and chrominance representation such as LCH or YUV and taking the luminance channel. Alternatively, the processing could be performed on each channel separately to produce a result for each colour, followed by combining the results. If a colour image based information metric is used in step 230, then the processing could also be performed on the full colour image.

[0076] In step 220, a partially degraded image is generated. This is performed by degrading the image in the region provided, either background or foreground. One approach to degrading the image is to take the weighted average of the image with neutral grey.

Neutral grey could be the midpoint of the luminance representation space that the image is in, or alternatively the mean luminance value of the image itself could be used. The weighting in the weighted average is from the region map provided. The region map is represented as a value between 0 and 1 for each pixel of the image, where 1 represents being fully included in the region, 0 fully excluded from the region, and other values representing partial inclusion in the region. The weighted average then gives neutral grey for the pixels with value 1, the unchanged image value for pixels with value 0, and a linear mix of the two for regions with values in between, resulting in a greying out of the included region.

[0077] An example of this is shown in Figs. 8A and 8B. Fig. 8A shows an artificial image 810 containing a stick figure (foreground) and a tree (background). Fig. 8B shows an associated region representation 820, with pixels of value 1 shown as white and pixels of 10825256 1 P150340_spec_lodge -17- 2015271935 18 Dec 2015 value 0 shown as black. Fig. 9A shows the image again as image 910, while Fig. 9B is the result 920 of degrading the image according to step 220, in which the foreground portion is degraded. The foreground region is degraded to become neutral grey, as represented by the dotted lines. The remaining parts of the image remain unchanged. This process effectively results in a greying out of the foreground regions of the image 910. A corresponding approach may alternatively be applied to grey out the background regions of the image.

[0078] Alternative information reducing degradations may be used, such as blurring the foreground region, whitening, or darkening. The degradation is desirably chosen to reduce the information in the image, particularly statistical information, as the goal is to measure the loss of information resulting from the degradation.

[0079] In step 230, the information difference between the image and the degraded image is calculated by the processor 105. This is calculated using the known method of Visual Information Fidelity. Alternatively, other image quality measures may be used. This calculates the information lost by degrading the image in the manner of step 220. As the degradation is known and not dependent on the image content, the amount of information lost by degrading the image in the given region represents the amount of information contained in that region to begin with.

[0080] In step 240, the value calculated in step 230 is normalised to the area of the region, which for a region defined by a region map would be the total sum (or average) of the values in the region map. A large region will mean a larger degradation, in turn leading to a larger value, the image itself not the size of the region is what is interesting for a feature helping to estimate perceived separation between foreground and background, so the impact of the size of the region on the value should be removed.

[0081] The method 200 then ends at step 299.

Second Implementation [0082] An alternative implementation relates to a system for improving the sense of separation in an image. Fig. 10 shows an example graphical user interface 1000 of such a system and which may be reproduced by the display 114 as part of the application 133 executable within the computer 101. The GUI 1000 includes a display of an image 1010 for 10825256 1 P150340_spec_lodge -18- 2015271935 18 Dec 2015 which the user desires to improve the perception of foreground/background separation. As before, a depth map is provided with the image 1010 to the processor 105. The user modifies the image to affect the perceived foreground foreground/background separation by moving a slider 1020, for example by manipulation of the mouse pointer 103 within the GUI 1000. As the slider 1020 moves to the right, the image 1010 is processed in such a way that the perceived foreground/background separation increases, while moving the slider 1020 to the left processes the image to reduce perceived foreground/background separation.

[0083] A method 600 may be used to produce a modified image each time the slider 1020 is moved will now be described with reference to Fig. 6. The method 600, as before, is desirably implemented in software for example as part of the application 133 executing within the computer system 100. In step 610, the image 1010, associated depth map, and desired change in foreground/background separation (based on the position of the slider 1020) are received by the processor 105 and temporarily stored in the memory 106.

[0084] In step 620, a set of image filters are considered, along with a set of combinations of filter strengths. This preferably involves the processor 105 identifying a list of image filters forming part of the application 133 and which are selectively executable by the processor 105 to apply a predetermined processing to the image 1010 that in some way alters the reproduction of the image 1010. These filters should be filters which are expected to impact the perceived foreground/background separation of at least some images. These filters may make use of the depth map provided, in order to modify the background and foreground of the image in a differential manner. Some examples of filters which modify the perceived foreground/background separation of at least some images when applied differentially to the foreground and background include sharpening/blurring, saturation boost/decrease, brightness, red/blue colour shift, contrast and unsharp masking.

[0085] Each combination of filter strength levels generates a modified image. Sufficient combinations of filter strengths should be chosen to cover the range of modifications which may be required to produce the desired change in perceived foreground/background separation. Sufficiently close combinations of filter strengths should also be chosen to cover changes in perceived impact at particular filter strength values. 10825256 1 P150340_spec_lodge -19- 2015271935 18 Dec 2015 [0086] For example, an implementation of this arrangement may be performed using three (3) filters, each with three (3) levels of gradation (e g. off, half on, full on), giving 27 combinations of filters. Relatively simple approaches of this kind, particularly if optimised, may lend themselves to integration within a camera for example, potentially for operation in near real-time. More complex approaches may involve more filters some with more levels of gradation, generally necessitating offline processing on a device with substantial computing resources, such as a desktop computer or server computer, for example operating in a cloud computing environment. Further, it is noted that the number of levels for each filter need not be the same. For example some filters may have two levels (on, off), whereas others may have five levels.

[0087] For each combination of filter strengths, as part of step 620, the impact on the perceived foreground/background separation of the image is then calculated. This calculation is performed using method 500 for each modified image, as previously described, potentially excluding step 550 as each individual calculated separation value 730 does not need to be displayed. The net effect of step 620 is the choosing of a different level of strength of one or more of the available filers to apply to the image based on the determined effect of the filter(s) applied.

[0088] In step 630, a model is formed relating the combinations of filter strengths to the resulting estimated perceived foreground/background separation 730, determined as a result of executing the method 500 for each of the filter strength combinations. This model may be formed by globally fitting a polynomial in the filter strengths to the estimated impact of each filter strength combination. For the example above with 3 filters, such would necessitate fitting the polynomial in a 3-dimensional space. Alternatively, the model may be formed by locally interpolating the estimated impact from the closest filter strength combinations evaluated. A simple model may just consider a few of the most effective filters for this image rather than including all available filters, and may consider the filters separately rather than modeling their interactions, and may even consider the case of those filters being applied in a fixed ratio based on their effectiveness.

[0089] Although they are described as happening in response to a change in the slider position, steps 620 and 630 do not require obtaining the desired change in foreground/background separation. Further, steps 620 and 630 may typically consume 10825256 1 P150340_spec_lodge -20- 2015271935 18 Dec 2015 significant computational resources and also take considerable time to execute. For improved efficiency, steps 620 and 630 are desirably performed just once for any given image, and the results stored, for example in the HDD 110 in association with the original image, for example alongside the corresponding depth map. Step 620 and 630 may also or alternatively be performed in advance when the image is initially loaded, or when the slider 1020 is first adjusted.

[0090] In step 640, the processor 105 chooses an optimal combination of the available filters to apply to the image 1010 based on the model produced in step 630. The model can be used to estimate filter strength combinations which produce the desired change in foreground/background separation. If, due to having many effective filters, multiple combinations are available which would provide the desired change, then any of these combinations may be chosen. The choice may be based on minimising the visibility of the processing for the same perceptual impact. The method 600 then ends at step 699.

[0091] The process 600 provides for a higher level user engagement with the image processing compared to that of the process 500. As such, the process 600 may be more suited to less skilled (amateur) users, whereas more professional users suitably adept in image attributes and processing and may more readily apply the method 500 to achieve an desired result in less time.

INDUSTRIAL APPLICABILITY

[0092] The arrangements described are applicable to the computer and data processing industries, and particularly for the processing of digital images to vary foreground/background separation.

[0093] The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0094] (Australia Only) In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not 10825256 1 P150340_spec_lodge -21- 2015271935 18 Dec 2015 “consisting only of’. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings. 10825256 1 P150340_spec_lodge

Claims (20)

1. CLAIMS The claims defining the invention are as follows:

   1. A computer-implementable method of determining visual information of regions of an image, the method comprising: receiving the image and a corresponding depth map for the image; generating a background-degraded image from the received image by degrading background regions of the image based on the depth in the corresponding depth map; generating a foreground-degraded image from the received image by degrading foreground regions of the image based on the depth in the corresponding depth map; and comparing each of the background-degraded image and the foreground-degraded image with the image to calculate visual information in the background and foreground regions of the image respectively.
2. 2. A method according to claim 1, wherein the degrading is performed in proportion to the depth of the region in the corresponding depth map.
3. 3. A method according to claim 1, wherein the degrading comprises greying out the corresponding region in proportion to the depth in the corresponding depth map.
4. 4. A method according to claim 3, wherein the greying out comprises modifying the image by a weighted average of the image with neutral grey.
5. 5. A method according to claim 1, wherein the degrading is one of greying out, blurring, whitening, and darkening
6. 6. The method according to claim 1 further comprising: (i) applying a filter to the image to generate a filtered image; (ii) generating a background-degraded filtered image and a foreground-degraded filtered image by greying out background and foreground regions of the filtered image respectively, the greying out of the regions being proportional to the depth in the corresponding depth map; (iii) comparing each of the background-degraded filtered image and the foreground-degraded filtered image to the filtered image to calculate visual information in the background and foreground of the filtered image respectively; and (iv) determining an effect of the applied filter by comparing the visual information of the image and the visual information of the filtered image.
7. 7. The method according to claim 1, wherein the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.
8. 8. The method according to claim 6, wherein determining an effect of the applied filter is performed by adding the filtered foreground image comparison value to the background image comparison value, and subtracting the filtered background image comparison and the foreground image comparison values.
9. 9. The method according to claim 8, wherein the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.
10. 10. The method according to claim 6, wherein the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.
11. 11. The method according to claim 6, further comprising: choosing a different level of strength of the filter to apply to the image based on the determined effect of the filter applied.
12. 12. The method according to claim 6, further comprising: (v) applying steps (i) to (iv) for at least one additional filter; and (vi) choosing which filter to apply based on the determined effects of each filter.
13. 13. The method according to claim 6, further comprising: (v) applying steps (i) to (iv) for at least one additional filter; and (vi) choosing different levels of strengths to apply with each filter based on the determined effects of each filter.
14. 14. A non-transitory computer readable storage medium having a program recorded thereon, the program being executable to determine visual information of regions of an image, the program comprising: code for receiving the image and a corresponding depth map for the image; code for generating a background-degraded image from the received image by degrading background regions of the image based on the depth in the corresponding depth map; code for generating a foreground-degraded image from the received image by degrading foreground regions of the image based on the depth in the corresponding depth map; and code for comparing each of the background-degraded image and the foreground-degraded image with the image to calculate visual information in the background and foreground regions of the image respectively.
15. 15. A non-transitory computer readable storage medium according to claim 14, wherein the degrading is performed in proportion to the depth of the region in the corresponding depth map, and the degrading comprises greying out the corresponding region in proportion to the depth in the corresponding depth map, the greying out comprising modifying the image by a weighted average of the image with neutral grey.
16. 16. A non-transitory computer readable storage medium according to claim 14 further comprising: (i) code for applying a filter to the image to generate a filtered image; (ii) code for generating a background-degraded filtered image and a foreground-degraded filtered image by greying out background and foreground regions of the filtered image respectively, the greying out of the regions being proportional to the depth in the corresponding depth map; (iii) code for comparing each of the background-degraded filtered image and the foreground-degraded filtered image to the filtered image to calculate visual information in the background and foreground of the filtered image respectively; and (iv) code for determining n effect of the applied filter by comparing the visual information of the image and the visual information of the filtered image.
17. 17. A non-transitory computer readable storage medium according to claim 14, wherein the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.
18. 18. A non-transitory computer readable storage medium according to claim 16, wherein determining an effect of the applied filter is performed by adding the filtered foreground image comparison value to the background image comparison value, and subtracting the filtered background image comparison and the foreground image comparison values,.
19. 19. A non-transitory computer readable storage medium according to claim 18, wherein: (a) the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image; and (b) the comparing of the background-degraded image and the foreground degraded image is performed using the Visual Information Fidelity measure, once to compare the foreground degraded image to the image and again to compare the background degraded image to the image.
20. 20. A non-transitory computer readable storage medium according to claim 16, further comprising: code for choosing a different level of strength of the filter to apply to the image based on the determined effect of the filter applied; and one of: (v) applying the code (i) to (iv) for at least one additional filter; and (vi) choosing which filter to apply based on the determined effects of each filter; or (vii) applying the code (i) to (iv) for at least one additional filter; and (viii) choosing different levels of strengths to apply with each filter based on the determined effects of each filter.