WO2017077533A1 - Real-time alpha compositing for high-resolution image stream - Google Patents

Real-time alpha compositing for high-resolution image stream Download PDF

Info

Publication number
WO2017077533A1
WO2017077533A1 PCT/IL2016/051184 IL2016051184W WO2017077533A1 WO 2017077533 A1 WO2017077533 A1 WO 2017077533A1 IL 2016051184 W IL2016051184 W IL 2016051184W WO 2017077533 A1 WO2017077533 A1 WO 2017077533A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
pixel
low
background
foreground
Prior art date
Application number
PCT/IL2016/051184
Other languages
French (fr)
Inventor
Ofri MASAD
Ariel Shamir
Original Assignee
Artincam Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Artincam Ltd. filed Critical Artincam Ltd.
Publication of WO2017077533A1 publication Critical patent/WO2017077533A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform

Definitions

  • the present invention relates to the field of image processing.
  • Alpha compositing is a common image processing technique for combining one or more elements, such as a foreground and a background, to create a final image, or composite.
  • each element has an associated matte that includes the element's image information, such as the shape, geometry, and color of the element.
  • an alpha channel provides the degree of transparency, or opacity for a given pixel, where an alpha (a) value of 1 indicates the pixel is opaque, or fully visible, an alpha value of 0 indicates the pixel is transparent, or invisible and an alpha between 0 and 1 allows combining the two mattes onto the same pixel.
  • an image processing method comprising using at least one hardware processor for: receiving a stream of high- resolution digital images at a computing device; for each of the high-resolution digital images, in real time: creating a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; making a low-resolution hint map from each of the low -resolution images by performing the following steps: a) segmenting the low -resolution image into a low-resolution foreground and a low -resolution background, wherein each pixel in the low -resolution image is categorized as belonging to either the low -resolution foreground or the low -resolution background; b) defining a low- resolution buffer zone comprising pixels that are within a predefined threshold distance of at
  • making the low-resolution hint map comprising steps a), b), and c) are performed a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
  • CPU central processing unit
  • GPU graphics processing unit
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
  • the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
  • the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
  • the method further comprises creating a new high- resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a , Cf or e g round and C new ⁇ background, wherein Cforegrmmd is the color of the associated foreground parent pixel, and wherein C new background is the color of the new corresponding background pixel.
  • the method further comprises rendering the new high- resolution image on a display of the computing device.
  • a computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor to: receive a stream of high-resolution digital images at a computing device; for each of the high-resolution digital images, in real time: create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; make a low-resolution hint map from each of the low-resolution images by performing the following steps: a) segmenting the low-resolution image into a low-resolution foreground and a low-resolution background, wherein each pixel in the low-resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background; b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low -resolution foreground and at least one pixel
  • making the low-resolution hint map comprising steps a), b), and c) are performed a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
  • CPU central processing unit
  • GPU graphics processing unit
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
  • the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
  • the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
  • the program code is further executable to create a new high- resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a , Cf or e g round and C n ew_back g round, wherein Cf or e g round is the color of the associated foreground parent pixel, and wherein C n ew_backgmund is the color of the new corresponding background pixel.
  • the program code is further executable to render the new high-resolution image on a display of the computing device.
  • an apparatus comprising: a computing device, configured to: receive a stream of high-resolution digital images; for each of the high-resolution digital images, in real time: create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; make a low-resolution hint map from each of the low-resolution images by performing the following steps: a) segmenting the low-resolution image into a low- resolution foreground and a low-resolution background, wherein each pixel in the low- resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background; b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low-resolution background; and c) for each pixel in the low-resolution buffer zone, associating a foreground parent
  • the apparatus further comprises: a serially-operative processor configured to perform steps a), b), and c); and a parallel-operative processor configured to perform steps d) and e).
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
  • the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
  • the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
  • the graphic processor is further configured to create a new high-resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high- resolution image is determined as a function of a , Cforeground and C new _ background-, wherein Cforeground is the color of the associated foreground parent pixel, and wherein Cnewjbackground is the color of the new corresponding background pixel.
  • the apparatus further comprises a display configured to render the new high-resolution image.
  • the apparatus further comprises a camera configured to capture the stream of high-resolution digital images.
  • FIG. 1 illustrates a system for replacing a background of an image in real-time, in accordance with an embodiment
  • Fig. 2A shows a zoomed-in portion of a low-resolution image corresponding to high-resolution image of Fig. 1, in accordance with an embodiment
  • Fig. 2B shows a zoomed in portion of an image with a low-resolution hint map, in accordance with an embodiment
  • Fig. 3 shows a high-resolution hint map scaled up from a low-resolution hint map, in accordance with an embodiment
  • FIGs. 4A-C illustrate an exemplary replacement of the background of an image, in accordance with an embodiment
  • FIGs. 5A-B which, together, is a flowchart of a method for replacing the background of a high-resolution image of an image stream, in real time, in accordance with an embodiment.
  • the computations for segmenting the image into foreground and background, and replacing the background with a different background are distributed to optimize the overall processing time.
  • a buffer zone is defined, marking the transition zone between the foreground and background sets. Calculation on the buffer zone are distributed between high and low-resolution to reduce latency, to allow outputting the new images with decreased delay.
  • the term "real time”, as referred to herein with respect to performing computing operations, may relate to performing the computing operations immediately, e.g., once an event occurs, guaranteeing response within strict time constraints.
  • the time constraints may include a timeframe of hundreds of milliseconds, such as 300, 250, 200, 150 and 100 milliseconds or less.
  • the term "camera”, as referred to herein, may relate to a digital camera including at least an image sensor, such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor; a lens assembly; and a processor.
  • the camera also incorporates, whether in its housing or externally, a digital display on which images captured by the image sensor, and/or other information, may be displayed.
  • CPU Central Processing Unit
  • 'Central Processing Unit' may relate to a general purpose microprocessor operating at a relatively high operating frequency.
  • GPU Graphics Processing Unit
  • 'Graphics Processing Unit' may relate to a special purpose microprocessor comprising multiple parallel processing cores that are optimized for performing calculations commonly required for computer graphics.
  • a computer 150 receives one or more high-resolution images 100, comprising a high- resolution video-stream captured via camera 102, and store the high-resolution images in a memory unit 104 of computer 100.
  • a processor 110 that may be suitable for serial processing, such as a CPU of computer 100, receives a low-resolution version of image 100, such as may have been down-scaled by a factor N to 1 , where N may be any integer, such as 10.
  • Any of serial processor 110, memory unit 104, camera 102, a parallel processor 118 and a display 120 of device 150 may communicated via one or more communication links, such as a communications bus 122 of device 150.
  • Fig. 2A shows a zoomed-in portion of a low- resolution image 200 corresponding to high-resolution image 100 of Fig. 1.
  • Processor 110 performs an image segmentation on low-resolution image 200 using any suitable image segmentation algorithm, to differentiate pixels of a low -resolution foreground set 206 from pixels of a low-resolution background set 208. Additionally, processor 110 defines a buffer zone 214 comprising pixels lying along and/or near the border of foreground and background sets 206 and 208.
  • a buffer zone 214 comprising pixels lying along and/or near the border of foreground and background sets 206 and 208.
  • Fig. 2B shows a zoomed in portion of image 200 having a low-resolution hint map 212, in accordance with an embodiment.
  • Processor 110 creates low-resolution hint map 212 by comparing the pixels lying in buffer zone 214 to the pixels of foreground and background sets 206 and 208 and that do not lie in buffer zone 214.
  • the pixels of low-resolution buffer zone 214 are indicated by a cross-hatch pattern, and foreground and background sets 206 and 208 are each indicated by oppositely oriented single-directional hatch patterns.
  • buffer zone 214 is illustrated as being relatively narrow, having a width of between one or two pixels, this is exemplary only, and buffer zone 214 may be wider, in accordance with the ratio between high-resolution and low-resolution images size, the accuracy of segmentation process and color variance over image space.
  • the pixels included in buffer zone 214 may be positioned within a predefined threshold distance of at least one pixel belonging to foreground set 206, and at least one pixel belonging to the background set 208.
  • Processor 110 assigns each pixel of buffer zone 214 a foreground parent pixel from foreground set 206 and a background parent pixel from background set 208.
  • the foreground parent pixel may be the closest geometrically positioned pixel from foreground set 206 that is not within buffer zone 214
  • the background parent pixel may be the closest geometrically positioned pixel from background portion 208 that is not within buffer zone 214.
  • Hint map 212 shown in Fig. 2B as a set of arrows, has one such association between a pixel of buffer zone 214 and its foreground and background parent pixels.
  • the pixel 230, positioned at coordinates (x,y) of image 200 is mapped to parent pixels, 232 and 234, positioned at coordinates (x+Axi, y+Ayi) and ( ⁇ + ⁇ 2, y+Ay2), respectively, where ( ⁇ , Ayi) may be the minimum integer distance, such as measured in pixels, between position (x,y) and any pixel from foreground set 206 , and (Zh3 ⁇ 4 Ayi) may be the minimum integer distance, such as measured in pixels, between position (x,y) and any pixel from background set 208.
  • (Axi, Ayi) may be a certain (non-minimum) integer distance between position (x,y) and any pixel from foreground set 206
  • (Zh3 ⁇ 4 Ayi) may be a certain (non-minimum) integer distance between position (x,y) and any pixel from background set 208.
  • hint map 212 includes associations for each of the pixels of low-resolution buffer-zone 214 with a pair of parent pixels, comprising one background parent pixel and one foreground parent pixel.
  • hint map 212 may comprise a table having three columns listing the coordinates of the pixels included in buffer zone 216, and the coordinates of the foreground parent and background parent pixels, respectively, and where the rows of the table may indicate the relationship of the pixels.
  • low-resolution hint map 212 may be scaled up to a high- resolution hint map 312, for example by a factor of 1 to N corresponding to the downscale factor previously applied to transform image 100 to low-resolution image.
  • the high- resolution hint map thus created may define a high-resolution foreground 106 and high- resolution background 108, corresponding to the low-resolution foreground set 206 and low-resolution background set 208 determined by the segmentation.
  • each pixel of hint map 212 may be mapped to N pixels in hint map 312.
  • high-resolution hint map 312 is illustrated here with arrows, high-resolution hint map 312 may comprise a table, as described above with respect to low-resolution hint map 212.
  • Hint map 312 may associate the pixels from a high-resolution buffer zone 314, corresponding to low-resolution buffer zone 214, with parent pixels from each of high-resolution foreground and background sets 106 and 108, corresponding to low-resolution foreground and background sets 206 and 208, respectively.
  • high-resolution hint map 312 is shown upscaled from low-resolution hint map 212 by a factor of 2: 1, however, any suitable upscaling, such as a factor of 10: 1, may be applied.
  • Upscaling low-resolution hint map 212 to high-resolution hint map 312 may comprise mapping each of the pixels of hint map 212 to N pixels having the same attributes, such as color, intensity, and/or saturation, and additionally, associating each of the N pixels with a foreground and background parent pixel from each of foreground set 306, and background set 308, in accordance with their positional proximity.
  • attributes such as color, intensity, and/or saturation
  • the low -resolution buffer-zone pixel 230 positioned at (x,y) from Fig. 2B is mapped to two high-resolution buffer-zone pixels 330a, and 330b, positioned at (x',y') and (x'+l,y'), low-resolution foreground parent pixel, 232, is mapped to two high-resolution foreground parent pixels 332a, and 332b, and low-resolution background parent pixel, 232, is mapped to two high-resolution foreground parent pixels 332a, and 332b, accordingly, where the coordinates of the upscaled pixels are meant to be illustrative only.
  • (x,y) may be mapped to N pixels defined by ( ⁇ + ⁇ , y+ ⁇ ), for ⁇ , ⁇ e ⁇ -1,1 ⁇ and n,m e ⁇ 0,1,2,3... ⁇ .
  • Hint map 312 may be received at a processor 118 that may be suitable for parallel processing, such as a graphic processing unit (GPU), or alternatively, multiple serial-type processors configured to operate in parallel.
  • processors 110 and 118 are the same processing unit.
  • Processor 118 may compute an alpha (a) channel for the pixels of buffer zone 314 in accordance with their similarity of a display attribute to the display attribute of the respective parent pixels.
  • the alpha channel may optionally be added as a fourth column of the table comprising hint map 312.
  • the attributes, such as color, lightness, or saturation of the pixel of buffer zone 314 positioned at (x',y') may have a 20% correlation with the attributes of its foreground parent pixel positioned at coordinates (x'+Ax'i, y'+Ay'i) and an 80% correlation with the attributes of its background parent pixel positioned at coordinates (x'+Ax'2, y'+Ay'2).
  • Processor 118 may assign that pixel an alpha value (a) of 0.2.
  • processor 118 may calculate the alpha channel by comparing a neighborhood of pixels about position ( ⁇ ', ⁇ ') to ' neighborhoods of pixels about foreground position ( ⁇ '+ ⁇ , y'+Ayi) and background position ( ⁇ '+ ⁇ '2, y'+Ay2)- Processor 118 may perform any of the comparisons between the pixels and/or neighborhood of pixels of hint map 318 with their respective foreground and background parent pixels or neighborhood of parent pixels, and determine any of the alpha values in parallel.
  • processor 118 may assign an a of 0 (transparent) to the pixels of background set 106, and an a of 1 (opaque) to the pixels of foreground set 108.
  • processor 118 may receive a definition for a new background coloring criterion, such as a new background image 408 that may have a resolution corresponding to image 100.
  • the background coloring criterion may be one or more colors that are defined numerically, such as according to a function.
  • Processor 118 may create a new image 400 having the same foreground 106 as image 100, but having background 408 in place of background 108, where the transition between foreground 106 and background 408 may be smoothed via hint map 312.
  • Background 408 may be a solid color, alternatively, background 408 may be a scene having many colors.
  • Processor 118 may compute the display attributes, such as color and/or intensity, for the pixels of image 400 by combining foreground set 106 with background 408 and determining a buffer zone 414 therebetween using the alpha channel.
  • processor 118 may use the a values of high-resolution hint map 312 to replace background set 108 with a corresponding portion of background 408.
  • Processor 118 may map the pixels of background 408 using a 1 : 1 correspondence with background 108, and thus, the background parent pixel positioned at (x'+Ax'2, y'+Ay2), may be assigned the color C new background ( ⁇ '+ ⁇ '2, y'+Ay ) corresponding to background 408.
  • processor 110 may apply a corrective scheme to scale background 408 to accord with the scale of background 108.
  • Fig. 4B shows the pixels of background 408 having a dotted pattern, corresponding to C new background, in place of the hatched pattern of background set 308 of Fig. 3, corresponding to background 108.
  • processor 118 For each pixel positioned at coordinates ( ⁇ ', ⁇ ') of image 400, processor 118 compute the color, C new image (x',y') > for the pixel as:
  • Cnew image (x',y') *Cfore g round (x'+Ax'], y'+Ay']) + (1- ) *Cnew_background(x'+Ax2, y'+Ay'2) and thereby overlay the mask with foreground set 106, where C oregmund ( ⁇ '+ ⁇ , y'+Ayi) is the color of the foreground parent pixel corresponding to foreground set 106, and Cnew_background(x'+Ax2, y'+Ay2) is the color of the pixel of background set 408 corresponding to the background parent pixel.
  • C oregmund ( ⁇ '+ ⁇ , y'+Ayi) is the color of the foreground parent pixel corresponding to foreground set 106
  • Cnew_background(x'+Ax2, y'+Ay2) is the color of the pixel of background set 408 corresponding to the background parent pixel.
  • the display attributes for each pixel of buffer zone 414 are determined as a combination of the display attribute of the associated foreground and background parent pixels of foreground set 106 and background set 408, in accordance with the computed for each pixel, and indicated by a combination of the dot pattern corresponding to background set 408 and the hatched pattern corresponding to foreground set 106.
  • image 400 created using the method above is shown having foreground 106 segued with background 408.
  • Processor 118 may provide image 400 for rendering on a display 120 of computer 150.
  • Processor 118 may perform the calculations described above with a high degree of parallelism, allowing the replacement of background 108 with background 408 in real time.
  • each captured frame of a high-resolution stream of images 100 may be replaced, in real time, with a high-resolution image 400, to yield a high-resolution stream 400 that may be rendered on display 120.
  • processor 110 and processor 118 may replace high-resolution image stream 100 having a frame rate of 60 frames per second, with high- resolution image stream 400 having a frame rate of 60 frames per second real time.
  • FIGs. 5A-B is a flowchart of a method for replacing the background of a high-resolution image of an image stream, in real time.
  • a stream of high-resolution images such as may be captured by a camera of a computing device, is received at the device (Step 500).
  • a low-resolution image corresponding to the high-resolution image is created by down-sampling the high- resolution image (Step 502).
  • a low-resolution hint map is created, optionally by a processor operative for serial processing, by performing the following steps: An image segmentation is performed on the low-resolution image, to categorize each pixel of the image as belonging to either a low-resolution foreground set or a low-resolution background set (Step 504).
  • a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground set and at least one pixel of the low- resolution background set is defined (Step 506).
  • Each pixel in the low-resolution buffer zone is associated with each of a foreground parent pixel that is the closest geometrically positioned pixel from the low -resolution foreground set, and a background parent pixel that is the closest geometrically positioned pixel from the low-resolution background set (Step 508), thereby creating the low-resolution hint map.
  • a high-resolution hint map is created from the low-resolution hint map by upscaling the low-resolution hint map, where the high- resolution hint map includes a high-resolution buffer zone corresponding to the low- resolution buffer zone (Step 510).
  • the high-resolution hint map thus created may define a high-resolution foreground and high-resolution background, corresponding to the low- resolution foreground and low -resolution background determined by the segmentation.
  • the high-resolution hint map is used to perform the following steps, optionally by the processor operative for parallel processing: a value for an alpha channel, a, is determined for each pixel of the high-resolution buffer zone in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map (Step 512).
  • the display attribute may be any combination of a color, shade, saturation, intensity, or lightness characteristic.
  • the display attribute may be a statistical evaluation of the display attributes of pixels situated in a neighborhood surrounding the pixel, and the similarity may be based on the similarity between the display attributes of each respective neighborhood.
  • the alpha channel value may range between 0 and 1 , where a value of 0 indicates the pixel is transparent, and a value of 1 indicates the pixel is opaque.
  • the high-resolution pixels in the buffer zone are recolored as a function of alpha and a new coloring criterion, such as a new background pixel of a new high-resolution background, where the position of the new background pixel in the new background pixel corresponds to the position of the background parent pixel (Step 514).
  • a new image is created from the high-resolution foreground, the new high-resolution background, and the recolored pixels of the high-resolution buffer zone (Step 516).
  • the new image is rendered on a display of the computing device (Step 518). Alternatively, the new image is stored at the computing device for subsequent display or processing.
  • the present invention may be a system, a method, and/or a computer program product.
  • the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
  • the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
  • the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • a non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or Flash memory erasable programmable read-only memory
  • SRAM static random access memory
  • CD-ROM compact disc read-only memory
  • DVD digital versatile disk
  • memory stick a floppy disk
  • mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon
  • a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
  • Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
  • the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
  • Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
  • the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the block may occur out of the order noted in the figures.
  • two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration can be implemented by special purpose hardware -based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

An image processing method comprising receiving a stream of high-resolution digital images at a computing device, downsampling each high-resolution digital image to create a low-resolution image, making a low-resolution hint map by segmenting the low- resolution image into a low-resolution foreground and a low-resolution background, and defining a low-resolution buffer zone, where each pixel in the low-resolution buffer zone is associated with a foreground parent pixel and a background parent pixel, creating a high- resolution hint map from the low-resolution hint map, determining an α value for each pixel in the high-resolution buffer zone, and recoloring the pixel as a function of α and a recoloring criterion.

Description

REAL-TIME ALPHA COMPOSITING FOR HIGH-RESOLUTION IMAGE
STREAM
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Patent Application No. 62/249,412, filed November 2, 2015, entitled "Real-Time Alpha Compositing for High- Resolution Image Stream", the contents of which are incorporated herein by reference in their entirety.
FIELD OF THE INVENTION
[0002] The present invention relates to the field of image processing.
BACKGROUND
[0003] Alpha compositing (Alpha Matting) is a common image processing technique for combining one or more elements, such as a foreground and a background, to create a final image, or composite. Typically, each element has an associated matte that includes the element's image information, such as the shape, geometry, and color of the element. To blend the mattes for the final composite image, an alpha channel provides the degree of transparency, or opacity for a given pixel, where an alpha (a) value of 1 indicates the pixel is opaque, or fully visible, an alpha value of 0 indicates the pixel is transparent, or invisible and an alpha between 0 and 1 allows combining the two mattes onto the same pixel.
[0004] The foregoing examples of the related art and limitations related therewith are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of skill in the art upon a reading of the specification and a study of the figures.
SUMMARY
[0005] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope. [0006] There is provided, in accordance with an embodiment, an image processing method comprising using at least one hardware processor for: receiving a stream of high- resolution digital images at a computing device; for each of the high-resolution digital images, in real time: creating a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; making a low-resolution hint map from each of the low -resolution images by performing the following steps: a) segmenting the low -resolution image into a low-resolution foreground and a low -resolution background, wherein each pixel in the low -resolution image is categorized as belonging to either the low -resolution foreground or the low -resolution background; b) defining a low- resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low- resolution background; and c) for each pixel in the low-resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low -resolution foreground, and a background parent pixel comprising the closest pixel from the low -resolution background; creating a high-resolution hint map from the low-resolution hint map by upscaling the low-resolution hint map, wherein the high-resolution hint map comprises a high-resolution buffer zone corresponding to the low -resolution buffer zone; for each pixel in the high-resolution buffer zone of the high-resolution hint map: d) determining an a value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and e) recoloring the pixel as a function of a and a recoloring criterion.
[0007] In some embodiments, making the low-resolution hint map comprising steps a), b), and c) are performed a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
[0008] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
[0009] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel. [0010] In some embodiments, the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
[0011] In some embodiments, the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
[0012] In some embodiments, the method further comprises creating a new high- resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a , Cforeground and Cnew ^background, wherein Cforegrmmd is the color of the associated foreground parent pixel, and wherein Cnew background is the color of the new corresponding background pixel.
[0013] In some embodiments, the method further comprises rendering the new high- resolution image on a display of the computing device.
[0014] There is provided, in accordance with an embodiment, a computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor to: receive a stream of high-resolution digital images at a computing device; for each of the high-resolution digital images, in real time: create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; make a low-resolution hint map from each of the low-resolution images by performing the following steps: a) segmenting the low-resolution image into a low-resolution foreground and a low-resolution background, wherein each pixel in the low-resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background; b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low -resolution foreground and at least one pixel of the low -resolution background; and c) for each pixel in the low-resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low-resolution foreground, and a background parent pixel comprising the closest pixel from low-resolution background; create a high-resolution hint map from the low-resolution hint map by upscaling the low-resolution hint map, wherein the high-resolution hint map comprises a high-resolution buffer zone corresponding to the low-resolution buffer zone; for each pixel in the high-resolution buffer zone of the high -resolution hint map: d) determine an a value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and e) recolor the pixel as a function of a and a recoloring criterion.
[0015] In some embodiments, making the low-resolution hint map comprising steps a), b), and c) are performed a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
[0016] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
[0017] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
[0018] In some embodiments, the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
[0019] In some embodiments, the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
[0020] In some embodiments, the program code is further executable to create a new high- resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a , Cforeground and Cnew_background, wherein Cforeground is the color of the associated foreground parent pixel, and wherein Cnew_backgmund is the color of the new corresponding background pixel.
[0021] In some embodiments, the program code is further executable to render the new high-resolution image on a display of the computing device.
[0022] There is provided, in accordance with an embodiment, an apparatus, comprising: a computing device, configured to: receive a stream of high-resolution digital images; for each of the high-resolution digital images, in real time: create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image; make a low-resolution hint map from each of the low-resolution images by performing the following steps: a) segmenting the low-resolution image into a low- resolution foreground and a low-resolution background, wherein each pixel in the low- resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background; b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low-resolution background; and c) for each pixel in the low-resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low-resolution foreground, and a background parent pixel comprising the closest pixel from low-resolution background; create a high-resolution hint map from the low-resolution hint map by upscaling the low-resolution hint map, wherein the high- resolution hint map comprises a high-resolution buffer zone corresponding to the low- resolution buffer zone; for each pixel in the high-resolution buffer zone of the high- resolution hint map: d) determine an value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and e) recolor the pixel as a function of and a recoloring criterion.
[0023] In some embodiments, the apparatus further comprises: a serially-operative processor configured to perform steps a), b), and c); and a parallel-operative processor configured to perform steps d) and e).
[0024] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU). [0025] In some embodiments, the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
[0026] In some embodiments, the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
[0027] In some embodiments, the graphic processor is further configured to create a new high-resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high- resolution image is determined as a function of a , Cforeground and Cnew_ background-, wherein Cforeground is the color of the associated foreground parent pixel, and wherein Cnewjbackground is the color of the new corresponding background pixel.
[0028] In some embodiments, the apparatus further comprises a display configured to render the new high-resolution image.
[0029] In some embodiments, the apparatus further comprises a camera configured to capture the stream of high-resolution digital images.
[0030] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the figures and by study of the following detailed description.
BRIEF DESCRIPTION OF THE FIGURES
[0031] Exemplary embodiments are illustrated in referenced figures. Dimensions of components and features shown in the figures are generally chosen for convenience and clarity of presentation and are not necessarily shown to scale. The figures are listed below.
[0032] Fig. 1 illustrates a system for replacing a background of an image in real-time, in accordance with an embodiment;
[0033] Fig. 2A shows a zoomed-in portion of a low-resolution image corresponding to high-resolution image of Fig. 1, in accordance with an embodiment; [0034] Fig. 2B shows a zoomed in portion of an image with a low-resolution hint map, in accordance with an embodiment;
[0035] Fig. 3 shows a high-resolution hint map scaled up from a low-resolution hint map, in accordance with an embodiment;
[0036] Figs. 4A-C illustrate an exemplary replacement of the background of an image, in accordance with an embodiment; and
[0037] Figs. 5A-B which, together, is a flowchart of a method for replacing the background of a high-resolution image of an image stream, in real time, in accordance with an embodiment.
DETAILED DESCRIPTION
[0038] Disclosed herein are systems, methods and computer program products for replacing the background in a stream of high-resolution images, in real time. The computations for segmenting the image into foreground and background, and replacing the background with a different background are distributed to optimize the overall processing time. In particular, a buffer zone is defined, marking the transition zone between the foreground and background sets. Calculation on the buffer zone are distributed between high and low-resolution to reduce latency, to allow outputting the new images with decreased delay.
[0039] The term "real time", as referred to herein with respect to performing computing operations, may relate to performing the computing operations immediately, e.g., once an event occurs, guaranteeing response within strict time constraints. The time constraints may include a timeframe of hundreds of milliseconds, such as 300, 250, 200, 150 and 100 milliseconds or less.
[0040] The term "camera", as referred to herein, may relate to a digital camera including at least an image sensor, such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor; a lens assembly; and a processor. Optionally, the camera also incorporates, whether in its housing or externally, a digital display on which images captured by the image sensor, and/or other information, may be displayed.
[0041] The term "CPU", or 'Central Processing Unit', as referred to herein, may relate to a general purpose microprocessor operating at a relatively high operating frequency.
[0042] The term "GPU", or 'Graphics Processing Unit', as referred to herein, may relate to a special purpose microprocessor comprising multiple parallel processing cores that are optimized for performing calculations commonly required for computer graphics.
[0043] Reference is now made to Fig. 1 which illustrates a system for replacing a background of an image in real-time, in accordance with an embodiment. As seen in Fig. 1 , a computer 150 receives one or more high-resolution images 100, comprising a high- resolution video-stream captured via camera 102, and store the high-resolution images in a memory unit 104 of computer 100. A processor 110 that may be suitable for serial processing, such as a CPU of computer 100, receives a low-resolution version of image 100, such as may have been down-scaled by a factor N to 1 , where N may be any integer, such as 10. Any of serial processor 110, memory unit 104, camera 102, a parallel processor 118 and a display 120 of device 150 may communicated via one or more communication links, such as a communications bus 122 of device 150.
[0044] Reference is now made to Fig. 2A, which shows a zoomed-in portion of a low- resolution image 200 corresponding to high-resolution image 100 of Fig. 1. Processor 110 performs an image segmentation on low-resolution image 200 using any suitable image segmentation algorithm, to differentiate pixels of a low -resolution foreground set 206 from pixels of a low-resolution background set 208. Additionally, processor 110 defines a buffer zone 214 comprising pixels lying along and/or near the border of foreground and background sets 206 and 208. By using a low-resolution version 200 of image 100 the segmentation of image 200 is completed significantly faster than if a similar segmentation were performed on high-resolution image 100.
[0045] Reference is now made to Fig. 2B which shows a zoomed in portion of image 200 having a low-resolution hint map 212, in accordance with an embodiment. Processor 110 creates low-resolution hint map 212 by comparing the pixels lying in buffer zone 214 to the pixels of foreground and background sets 206 and 208 and that do not lie in buffer zone 214. For illustrative purposes, the pixels of low-resolution buffer zone 214 are indicated by a cross-hatch pattern, and foreground and background sets 206 and 208 are each indicated by oppositely oriented single-directional hatch patterns. Although buffer zone 214 is illustrated as being relatively narrow, having a width of between one or two pixels, this is exemplary only, and buffer zone 214 may be wider, in accordance with the ratio between high-resolution and low-resolution images size, the accuracy of segmentation process and color variance over image space. Thus, the pixels included in buffer zone 214 may be positioned within a predefined threshold distance of at least one pixel belonging to foreground set 206, and at least one pixel belonging to the background set 208. Processor 110 assigns each pixel of buffer zone 214 a foreground parent pixel from foreground set 206 and a background parent pixel from background set 208. The foreground parent pixel may be the closest geometrically positioned pixel from foreground set 206 that is not within buffer zone 214, and the background parent pixel may be the closest geometrically positioned pixel from background portion 208 that is not within buffer zone 214. Hint map 212 shown in Fig. 2B as a set of arrows, has one such association between a pixel of buffer zone 214 and its foreground and background parent pixels. The pixel 230, positioned at coordinates (x,y) of image 200 is mapped to parent pixels, 232 and 234, positioned at coordinates (x+Axi, y+Ayi) and (χ+Δχ2, y+Ay2), respectively, where (Δχι, Ayi) may be the minimum integer distance, such as measured in pixels, between position (x,y) and any pixel from foreground set 206 , and (Zh¾ Ayi) may be the minimum integer distance, such as measured in pixels, between position (x,y) and any pixel from background set 208. Alternatively, (Axi, Ayi) may be a certain (non-minimum) integer distance between position (x,y) and any pixel from foreground set 206, and (Zh¾ Ayi) may be a certain (non-minimum) integer distance between position (x,y) and any pixel from background set 208.
[0046] In other embodiments, instead of a geometrical distance between pixels, as discussed above, other distance criteria may be used to determine the foreground and background parent pixels for the pixels in the buffer zone. For example, the distance may be determined as a function of a display attribute, such as color, hue, intensity, gray level, and the like. [0047] In a similar manner, hint map 212 includes associations for each of the pixels of low-resolution buffer-zone 214 with a pair of parent pixels, comprising one background parent pixel and one foreground parent pixel. Although hint map 212 is indicated here by arrows, hint map 212 may comprise a table having three columns listing the coordinates of the pixels included in buffer zone 216, and the coordinates of the foreground parent and background parent pixels, respectively, and where the rows of the table may indicate the relationship of the pixels.
[0048] Referring to Fig. 3, low-resolution hint map 212 may be scaled up to a high- resolution hint map 312, for example by a factor of 1 to N corresponding to the downscale factor previously applied to transform image 100 to low-resolution image. The high- resolution hint map thus created may define a high-resolution foreground 106 and high- resolution background 108, corresponding to the low-resolution foreground set 206 and low-resolution background set 208 determined by the segmentation. Thus, each pixel of hint map 212 may be mapped to N pixels in hint map 312. Although high-resolution hint map 312 is illustrated here with arrows, high-resolution hint map 312 may comprise a table, as described above with respect to low-resolution hint map 212. Hint map 312 may associate the pixels from a high-resolution buffer zone 314, corresponding to low-resolution buffer zone 214, with parent pixels from each of high-resolution foreground and background sets 106 and 108, corresponding to low-resolution foreground and background sets 206 and 208, respectively.
[0049] For illustrative purposes, high-resolution hint map 312 is shown upscaled from low-resolution hint map 212 by a factor of 2: 1, however, any suitable upscaling, such as a factor of 10: 1, may be applied. Upscaling low-resolution hint map 212 to high-resolution hint map 312 may comprise mapping each of the pixels of hint map 212 to N pixels having the same attributes, such as color, intensity, and/or saturation, and additionally, associating each of the N pixels with a foreground and background parent pixel from each of foreground set 306, and background set 308, in accordance with their positional proximity. A seen in the example of Fig. 3, the low -resolution buffer-zone pixel 230 positioned at (x,y) from Fig. 2B, is mapped to two high-resolution buffer-zone pixels 330a, and 330b, positioned at (x',y') and (x'+l,y'), low-resolution foreground parent pixel, 232, is mapped to two high-resolution foreground parent pixels 332a, and 332b, and low-resolution background parent pixel, 232, is mapped to two high-resolution foreground parent pixels 332a, and 332b, accordingly, where the coordinates of the upscaled pixels are meant to be illustrative only. For example, (x,y) may be mapped to N pixels defined by (χ+ηδ, y+ηιε), for δ,ε e {-1,1 } and n,m e {0,1,2,3... }.
[0050] Hint map 312 may be received at a processor 118 that may be suitable for parallel processing, such as a graphic processing unit (GPU), or alternatively, multiple serial-type processors configured to operate in parallel. Optionally, processors 110 and 118 are the same processing unit. Processor 118 may compute an alpha (a) channel for the pixels of buffer zone 314 in accordance with their similarity of a display attribute to the display attribute of the respective parent pixels. The alpha channel may optionally be added as a fourth column of the table comprising hint map 312. For example, the attributes, such as color, lightness, or saturation of the pixel of buffer zone 314 positioned at (x',y') may have a 20% correlation with the attributes of its foreground parent pixel positioned at coordinates (x'+Ax'i, y'+Ay'i) and an 80% correlation with the attributes of its background parent pixel positioned at coordinates (x'+Ax'2, y'+Ay'2). Processor 118 may assign that pixel an alpha value (a) of 0.2. Alternatively, processor 118 may calculate the alpha channel by comparing a neighborhood of pixels about position (χ',γ') to ' neighborhoods of pixels about foreground position (Χ'+ΔΧΊ, y'+Ayi) and background position (χ'+Δχ'2, y'+Ay2)- Processor 118 may perform any of the comparisons between the pixels and/or neighborhood of pixels of hint map 318 with their respective foreground and background parent pixels or neighborhood of parent pixels, and determine any of the alpha values in parallel.
[0051] Optionally, processor 118 may assign an a of 0 (transparent) to the pixels of background set 106, and an a of 1 (opaque) to the pixels of foreground set 108.
[0052] Reference is now made to Figs. 4A-C which illustrate an exemplary replacement of the background of an image, in accordance with an embodiment. Referring to Fig. 4A, processor 118 may receive a definition for a new background coloring criterion, such as a new background image 408 that may have a resolution corresponding to image 100. Alternatively, the background coloring criterion may be one or more colors that are defined numerically, such as according to a function. Processor 118 may create a new image 400 having the same foreground 106 as image 100, but having background 408 in place of background 108, where the transition between foreground 106 and background 408 may be smoothed via hint map 312. Background 408 may be a solid color, alternatively, background 408 may be a scene having many colors.
[0053] Processor 118 may compute the display attributes, such as color and/or intensity, for the pixels of image 400 by combining foreground set 106 with background 408 and determining a buffer zone 414 therebetween using the alpha channel.
[0054] Referring to Fig. 4B, processor 118 may use the a values of high-resolution hint map 312 to replace background set 108 with a corresponding portion of background 408. Processor 118 may map the pixels of background 408 using a 1 : 1 correspondence with background 108, and thus, the background parent pixel positioned at (x'+Ax'2, y'+Ay2), may be assigned the color Cnew background (χ'+Δχ'2, y'+Ay ) corresponding to background 408. Alternatively, if background 408 and 108 are not scaled equally, processor 118 or alternatively, processor 110 may apply a corrective scheme to scale background 408 to accord with the scale of background 108. Fig. 4B shows the pixels of background 408 having a dotted pattern, corresponding to Cnew background, in place of the hatched pattern of background set 308 of Fig. 3, corresponding to background 108.
[0055] For each pixel positioned at coordinates (χ',γ') of image 400, processor 118 compute the color, Cnew image (x',y')> for the pixel as:
Cnew image (x',y') = *Cforeground (x'+Ax'], y'+Ay']) + (1- ) *Cnew_background(x'+Ax2, y'+Ay'2) and thereby overlay the mask with foreground set 106, where C oregmund (Χ'+ΑΧΊ, y'+Ayi) is the color of the foreground parent pixel corresponding to foreground set 106, and Cnew_background(x'+Ax2, y'+Ay2) is the color of the pixel of background set 408 corresponding to the background parent pixel. As may be seen in Fig. 4B, the display attributes for each pixel of buffer zone 414 are determined as a combination of the display attribute of the associated foreground and background parent pixels of foreground set 106 and background set 408, in accordance with the computed for each pixel, and indicated by a combination of the dot pattern corresponding to background set 408 and the hatched pattern corresponding to foreground set 106. Referring to Fig. 4C, image 400 created using the method above, is shown having foreground 106 segued with background 408. Processor 118 may provide image 400 for rendering on a display 120 of computer 150.
[0056] Processor 118 may perform the calculations described above with a high degree of parallelism, allowing the replacement of background 108 with background 408 in real time. Thus, each captured frame of a high-resolution stream of images 100 may be replaced, in real time, with a high-resolution image 400, to yield a high-resolution stream 400 that may be rendered on display 120. For example, processor 110 and processor 118 may replace high-resolution image stream 100 having a frame rate of 60 frames per second, with high- resolution image stream 400 having a frame rate of 60 frames per second real time.
[0057] Reference is now made to Figs. 5A-B which, together, is a flowchart of a method for replacing the background of a high-resolution image of an image stream, in real time. A stream of high-resolution images, such as may be captured by a camera of a computing device, is received at the device (Step 500). For each image, in real time, a low-resolution image corresponding to the high-resolution image is created by down-sampling the high- resolution image (Step 502). A low-resolution hint map is created, optionally by a processor operative for serial processing, by performing the following steps: An image segmentation is performed on the low-resolution image, to categorize each pixel of the image as belonging to either a low-resolution foreground set or a low-resolution background set (Step 504). A low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground set and at least one pixel of the low- resolution background set is defined (Step 506). Each pixel in the low-resolution buffer zone is associated with each of a foreground parent pixel that is the closest geometrically positioned pixel from the low -resolution foreground set, and a background parent pixel that is the closest geometrically positioned pixel from the low-resolution background set (Step 508), thereby creating the low-resolution hint map. A high-resolution hint map is created from the low-resolution hint map by upscaling the low-resolution hint map, where the high- resolution hint map includes a high-resolution buffer zone corresponding to the low- resolution buffer zone (Step 510). The high-resolution hint map thus created may define a high-resolution foreground and high-resolution background, corresponding to the low- resolution foreground and low -resolution background determined by the segmentation. The high-resolution hint map is used to perform the following steps, optionally by the processor operative for parallel processing: a value for an alpha channel, a, is determined for each pixel of the high-resolution buffer zone in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map (Step 512). For example, the display attribute may be any combination of a color, shade, saturation, intensity, or lightness characteristic. Alternatively, for any of the buffer zone pixel, and/or the foreground and background pixel, the display attribute may be a statistical evaluation of the display attributes of pixels situated in a neighborhood surrounding the pixel, and the similarity may be based on the similarity between the display attributes of each respective neighborhood. The alpha channel value may range between 0 and 1 , where a value of 0 indicates the pixel is transparent, and a value of 1 indicates the pixel is opaque. The high-resolution pixels in the buffer zone are recolored as a function of alpha and a new coloring criterion, such as a new background pixel of a new high-resolution background, where the position of the new background pixel in the new background pixel corresponds to the position of the background parent pixel (Step 514). The new color may be determined by dewjmage = * Cforegroimd + (a -1) * new_background, where Cforeground is the color of the associated foreground parent pixel of the high-resolution buffer ZOnC, cincl Cnew_backg round IS the color of the new background pixel corresponding to the associated background parent pixel of the high- resolution buffer zone. A new image is created from the high-resolution foreground, the new high-resolution background, and the recolored pixels of the high-resolution buffer zone (Step 516). The new image is rendered on a display of the computing device (Step 518). Alternatively, the new image is stored at the computing device for subsequent display or processing.
[0058] The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
[0059] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
[0060] Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
[0061] Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
[0062] Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
[0063] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
[0064] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
[0065] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware -based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
[0066] The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

CLAIMS claimed is:
An image processing method comprising using at least one hardware processor for: receiving a stream of high-resolution digital images at a computing device; for each of the high-resolution digital images, in real time:
creating a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image;
making a low-resolution hint map from each of the low-resolution images by performing the following steps:
a) segmenting the low-resolution image into a low-resolution foreground and a low-resolution background, wherein each pixel in the low- resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background;
b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low-resolution background; and c) for each pixel in the low-resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low-resolution foreground, and a background parent pixel comprising the closest pixel from the low-resolution background;
creating a high-resolution hint map from the low-resolution hint map by upscaling the low-resolution hint map, wherein the high-resolution hint map comprises a high-resolution buffer zone corresponding to the low-resolution buffer zone;
for each pixel in the high-resolution buffer zone of the high-resolution hint map: d) determining an a value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and
e) recoloring the pixel as a function of a and a recoloring criterion.
2. The image-processing method of claim 1 , wherein the making of the low-resolution hint map is performed by a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
3. The image processing method of claim 2, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
4. The image processing method of claim 2, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
5. The image-processing method of claim 1, wherein the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
6. The image -processing method of claim 1, wherein the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
7. The image-processing method of claim 5, further comprising creating a new high- resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a, Cforeground and Cnew ^background, wherein Cforegrmmd is the color of the associated foreground parent pixel, and wherein Cnew background is the color of the new corresponding background pixel.
8. The image-processing method of claim 7, further comprising rendering the new high-resolution image on a display of the computing device.
9. A computer program product comprising a non -transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor to:
receive a stream of high-resolution digital images at a computing device; for each of the high-resolution digital images, in real time:
create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image;
make a low-resolution hint map from each of the low-resolution images by performing the following steps:
a) segmenting the low-resolution image into a low-resolution foreground and a low -resolution background, wherein each pixel in the low-resolution image is categorized as belonging to either the low-resolution foreground or the low-resolution background; b) defining a low -resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low- resolution background; and
c) for each pixel in the low-resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low-resolution foreground, and a background parent pixel comprising the closest pixel from low-resolution background; create a high-resolution hint map from the low-resolution hint map by upscaling the low-resolution hint map, wherein the high-resolution hint map comprises a high-resolution buffer zone corresponding to the low -resolution buffer zone;
for each pixel in the high-resolution buffer zone of the high-resolution hint map:
d) determine an a value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and e) recolor the pixel as a function of and a recoloring criterion.
10. The computer program product of claim 9, wherein the making of the low-resolution hint map is performed by a serially-operative processor, and steps d) and e) are performed by a parallel-operative processor.
11. The computer program product of claim 10, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
12. The computer program product of claim 10, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel -operative processor comprises multiple processing units configured to operate in parallel.
13. The computer program product of claim 9, wherein the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
14. The computer program product of claim 9, wherein the display attribute of any of the pixels comprising: the pixel in the high-resolution buffer zone, the background parent pixel, and the foreground parent pixel, comprises a display attribute of a neighborhood surrounding the respective pixel.
15. The computer program product of claim 13, wherein the program code is further executable to create a new high-resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high-resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a, Cforegmund and Cnewjbackground, wherein Cforegmund is the color of the associated foreground parent pixel, and wherein Cnew background is the color of the new corresponding background pixel.
16. The computer program product of claim 15, wherein the program code is further executable to render the new high-resolution image on a display of the computing device.
An apparatus, comprising:
a computing device, configured to:
receive a stream of high-resolution digital images;
for each of the high-resolution digital images, in real time:
create a low-resolution image corresponding to the high-resolution digital image by downsampling the high-resolution digital image;
make a low-resolution hint map from each of the low-resolution images by performing the following steps:
a) segmenting the low-resolution image into a low-resolution foreground and a low-resolution background, wherein each pixel in the low-resolution image is categorized as belonging to either the low-resolution foreground or the low -resolution background; b) defining a low-resolution buffer zone comprising pixels that are within a predefined threshold distance of at least one pixel of the low-resolution foreground and at least one pixel of the low-resolution background; and
c) for each pixel in the low -resolution buffer zone, associating a foreground parent pixel comprising the closest pixel from the low- resolution foreground, and a background parent pixel comprising the closest pixel from low-resolution background;
create a high-resolution hint map from the low -resolution hint map by upscaling the low-resolution hint map, wherein the high-resolution hint map comprises a high-resolution buffer zone corresponding to the low- resolution buffer zone;
for each pixel in the high-resolution buffer zone of the high- resolution hint map:
d) determine an a value for the pixel in accordance with a similarity of a display attribute of the pixel to a display attribute of each of the associated background parent pixel and foreground parent pixel of the high-resolution hint map; and
e) recolor the pixel as a function of a and a recoloring criterion.
18. The apparatus of claim 17, further comprising:
a serially-operative processor configured to perform steps a), b), and c); and a parallel -operative processor configured to perform steps d) and e).
19. The apparatus of claim 18, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor is a graphics processing unit (GPU).
20. The apparatus of claim 18, wherein the serially-operative processor is a central processing unit (CPU), and wherein the parallel-operative processor comprises multiple processing units configured to operate in parallel.
21. The apparatus of claim 17, wherein the recoloring criterion comprises a new color obtained from a new background pixel of a new background, wherein the position of the new background pixel corresponds to the position of the background parent pixel.
22. The apparatus of claim 21, wherein the graphic processor is further configured to create a new high-resolution image from: a) a high-resolution foreground defined by the high-resolution hint map, b) the new background, and c) the recolored pixels of the high- resolution buffer zone, wherein the color of each pixel of the high-resolution buffer zone of the new high-resolution image is determined as a function of a, Cforegmund and Cnew_ background, wherein Cforegmund is the color of the associated foreground parent pixel, and wherein Cnewjbackgmund is the color of the new corresponding background pixel.
23. The apparatus of claim 22, further comprising a display configured to render the new high-resolution image.
24. The apparatus of claim 17, further comprising a camera configured to capture the stream of high-resolution digital images.
25. A method comprising:
receiving a stream of digital images; using a serially-operative computer processor, creating a low-resolution hint map from each of the digital images, wherein the low-resolution hint map defines a low- resolution buffer zone between a background and a foreground;
creating a high-resolution hint map from the low -resolution hint map, such that the high-resolution hint map includes a high-resolution buffer zone corresponding to the low- resolution buffer zone; and
using a parallel-operative processor, determining an alpha value for each pixel in the high-resolution buffer zone.
26. An apparatus comprising:
a serially-operative computer processor; and
a parallel-operative computer processor,
wherein:
(a) at least one of said computer processors is configured to receive a stream of digital images;
(b) said serially-operative computer processor is configured to create a low- resolution hint map from each of the digital images, wherein the low -resolution hint map defines a low-resolution buffer zone between a background and a foreground;
(c) at least one of said computer processors is configured to create a high-resolution hint map from the low-resolution hint map, such that the high-resolution hint map includes a high-resolution buffer zone corresponding to the low -resolution buffer zone; and
(d) said parallel-operative processor is configured to determine an alpha value for each pixel in the high-resolution buffer zone.
PCT/IL2016/051184 2015-11-02 2016-11-02 Real-time alpha compositing for high-resolution image stream WO2017077533A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562249412P 2015-11-02 2015-11-02
US62/249,412 2015-11-02

Publications (1)

Publication Number Publication Date
WO2017077533A1 true WO2017077533A1 (en) 2017-05-11

Family

ID=58662509

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2016/051184 WO2017077533A1 (en) 2015-11-02 2016-11-02 Real-time alpha compositing for high-resolution image stream

Country Status (1)

Country Link
WO (1) WO2017077533A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5438651A (en) * 1991-10-30 1995-08-01 Fujitsu Limited Color adjustment for smoothing a boundary between color images
US20080107344A1 (en) * 2000-12-22 2008-05-08 Microsoft Corporation System and method providing subpixel-edge-offset-based determination of opacity
US20120120281A1 (en) * 2007-04-03 2012-05-17 Lifetouch Inc. Method and apparatus for background replacement in still photographs
US20130265382A1 (en) * 2012-04-09 2013-10-10 Futurewei Technologies, Inc. Visual Conditioning for Augmented-Reality-Assisted Video Conferencing
US20140348425A1 (en) * 2013-05-23 2014-11-27 Futurewei Technologies, Inc. Delta Interpolation for Upsampling Imaging Solution

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5438651A (en) * 1991-10-30 1995-08-01 Fujitsu Limited Color adjustment for smoothing a boundary between color images
US20080107344A1 (en) * 2000-12-22 2008-05-08 Microsoft Corporation System and method providing subpixel-edge-offset-based determination of opacity
US20120120281A1 (en) * 2007-04-03 2012-05-17 Lifetouch Inc. Method and apparatus for background replacement in still photographs
US20130265382A1 (en) * 2012-04-09 2013-10-10 Futurewei Technologies, Inc. Visual Conditioning for Augmented-Reality-Assisted Video Conferencing
US20140348425A1 (en) * 2013-05-23 2014-11-27 Futurewei Technologies, Inc. Delta Interpolation for Upsampling Imaging Solution

Similar Documents

Publication Publication Date Title
Zhao et al. Simultaneous color-depth super-resolution with conditional generative adversarial networks
US10198624B2 (en) Segmentation-guided real-time facial performance capture
Park et al. High-quality depth map upsampling and completion for RGB-D cameras
CN109660783B (en) Virtual reality parallax correction
EP3104332B1 (en) Digital image manipulation
JP6998388B2 (en) Methods and equipment for processing image property maps
US9105117B2 (en) Methods and apparatus for coherent manipulation and stylization of stereoscopic images
AU2017324923A1 (en) Predicting depth from image data using a statistical model
US20130071041A1 (en) High-Quality Denoising of an Image Sequence
US20230051960A1 (en) Coding scheme for video data using down-sampling/up-sampling and non-linear filter for depth map
US9153068B2 (en) Clipless time and lens bounds for improved sample test efficiency in image rendering
US20110229024A1 (en) Devices and Methods for Processing Images Using Scale Space
CN107622504B (en) Method and device for processing pictures
JP2017091298A (en) Image processing device, image processing method and image processing program
CN111192190A (en) Method and device for eliminating image watermark and electronic equipment
CN110390657B (en) Image fusion method
US10650488B2 (en) Apparatus, method, and computer program code for producing composite image
EP2908527A1 (en) Device, program, and method for reducing data size of multiple images containing similar information
Shafieyan et al. Image seam carving using depth assisted saliency map
Abate et al. An image based approach to hand occlusions in mixed reality environments
US12086931B2 (en) Methods of 3D clothed human reconstruction and animation from monocular image
JP5473096B2 (en) Modeling concave surfaces in image-based visual hulls
US9589324B1 (en) Overshoot protection of upscaled images
WO2017077533A1 (en) Real-time alpha compositing for high-resolution image stream
CN112929562B (en) Video jitter processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16861724

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16861724

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28/09/2018)

122 Ep: pct application non-entry in european phase

Ref document number: 16861724

Country of ref document: EP

Kind code of ref document: A1