JP2023507589A - Noise synthesis for digital images - Google Patents

Abstract

An apparatus and method for providing a software and hardware based solution to the problem of noise synthesis for digital images. According to one aspect, a probability image is generated and noise blocks are randomly placed at locations within the probability image, the locations having probability values that are compared to nine threshold criteria to generate a composite noise image. Embodiments include generating a synthetic film grain image and a synthetic digital camera noise image.

Description

This disclosure relates generally to images. More particularly, one embodiment of the present disclosure relates to combining film grain and camera noise for images.

This application claims the benefit of priority to U.S. Provisional Patent Application No. 62/950466, filed December 19, 2019, and European Patent Application No. 19218144.4, filed December 19, 2019 , which are hereby incorporated by reference in their entirety.

As used herein, the term noise synthesis means producing noise effects on a digital image (including video frames) by means other than how that noise is normally produced. At any point in the description herein, the terms may be used in either sense, e.g. should be inferred.

The photographic filmstrip was the canvas for traditional photography before the advent of digital technology. The film is a thin plastic sheet covered with a photosensitive material (eg, silver halide) that absorbs light emitted or reflected from the object being filmed. After processing these films, the captured light can be converted into images. As used herein, "film grain" means the development in processed images due to dye clouds (or silver particles) receiving sufficient photons. It refers to the visual texture developed, which is an optical phenomenon caused by the chemical processes that go on during the processing of a film, and the definition in which the film is observed and used. Both depend on the type of film stock.

Digital photography is inherently devoid of film grain effects. However, it is still desired for artistic and qualitative reasons. For example, Hollywood studios prefer movie content with more film grain. For digital images, the film grain pattern needs to be synthesized and superimposed on the image during post-processing.

“Camera noise” refers to variations in brightness and/or color (colored specks in an image) in a digital image due to camera electronics such as sensor size, exposure time, ISO settings, etc. Typically, camera noise is removed from digital images by various techniques to avoid problems in compression, but in some cases the original image had when it was captured. Camera noise is desired to preserve the "look".

As used herein, the term "metadata" means any aid that is transmitted as part of a coded bitstream and that assists a decoder to render a decoded image. See relevant information. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

In practice, a digital image contains one or more color components (e.g. luma Y, chroma Cb and Cr), where each color component has a precision of n-bits per pixel (e.g. n=8). expressed. Using linear luminance coding, images with n<8 (e.g., color 24-bit JPEG images) are considered Standard Dynamic Range (SDR) images, while n Images that are >8 may be considered enhanced dynamic range (EDR) images. EDR and HDR images can also be stored and distributed using high precision floating point formats (eg, 16-bit), such as the OpenEXR file format developed by Industrial Light and Magic.

As used herein, the term "noise synthesis" includes, but is not limited to, noise normally caused by film/video/image capture or development, such as film grain or camera noise. within the digital image. The term may also refer to any noise artificially inserted into a digital image for any purpose.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Accordingly, unless otherwise indicated, any technique described in this section should not be considered to qualify as prior art merely because of its inclusion in this section. Similarly, the problems identified with respect to one or more of the approaches should not be assumed to have been recognized in any prior art based on this section unless otherwise indicated.

Apparatus and methods according to the present disclosure provide a solution to the problem of noise composition for digital images. According to one embodiment of the present disclosure, synthesized film grain is added to a clean (ie, noise free) digital sequence.

The process of synthesizing noise as presented here has many advantages in image transmission and display. For example, the ability to synthesize noise enables transmission of noise-free compressed images, which can provide increased transmission efficiency (data rate, bandwidth, etc.). Adding synthesized noise to the uncompressed image can reduce the effects of compression artifacts in the displayed image. The use of noise blocks allows control over film grain size, sharpness, and type (noise source emulation), and their random placement in the image is induced by tiled placement. prevent possible blocking artifacts. Additionally, for hardware-based implementations, the use of per-line processing allows resource-efficient implantation of noise synthesis.

According to a first aspect of the present disclosure, a computer-based method is disclosed for generating a synthetic noise image for a digital image. The method comprises providing a plurality of noise block patterns, initializing a noise image, and generating a probability image, the probability image including randomly generated probability values at each pixel. and comparing each probability value in the probability image to a threshold criterion. selecting a noise block pattern from the plurality of noise block patterns if the probability value satisfies the threshold criterion, the noise block pattern having an anchor point, and if the probability value satisfies the threshold criterion; and generating a synthesized noise image by placing the noise block pattern within the noise image such that the anchor points correspond to pixels in the probability image that meet the threshold criteria.

The noise grain block pattern can be randomly selected. The probability values of the probability image may be uniformly distributed probability values. A noise block pattern may include a film grain block pattern. The method may also include aggregating values in regions of the noisy image where the noise block patterns overlap. The method may include generating the noise block pattern prior to generating the noise image. Here, the noise block pattern is generated through an inverse DCT technique. This generation may include removing low frequency DCT coefficients and/or removing high frequency DCT coefficients. The probability image and noise image may be sized to match the noise image to the video frame. The method may include editing the content image by adding to the channel of the content image. The method may include multiplying the values in the noise image by the modulation depth before adding the noise image to the content image. This value can be the luma intensity value of the noise image. The method may include generating a YUV content image based on the edited content image. The method may include generating an RGB content image based on the YUV content image. The method may be executed on a computer program product containing data encoded for performing the method.

The method may be performed by a noise synthesis module to create a synthetic noise image for the device. The device may include a decoder that decodes the encoded image and produces a decoded clean sequence. Here the decoder is configured to combine the synthesized noise image with the decoded clean sequence to produce a rendered image for display. The device may include a modulation curve module for modulating the synthetic noise image before combining it with the decoded clean sequence.

According to a second aspect of the disclosure, a device is disclosed for generating synthetic noise in an image. The device includes a line buffer containing lines from the image, firmware containing noise block patterns, a pseudo-random number generator for providing random values for pixels in the lines, and using the random values as threshold criteria. logic for comparing; a data store containing a noise block management table; logic for generating the noise block management table based on random noise block patterns from the firmware associated with pixels in the row; logic for determining that the random value associated with a pixel meets the threshold criteria; logic for calculating a noise value for each pixel location in the row of the image based on the noise block management table; and logic to modify the row.

The device may include logic for determining the modulation curve from the image, calculating the modulation depth, and applying the modulation depth to the noise values prior to row modification. A pseudorandom noise generator may generate a random number based on the row index and the random offset value. The images can be luma channel images or chroma channel images. The device may include multiple pseudo-random number generators and multiple line buffers to process images in parallel.

According to a third aspect of the disclosure, a method of generating metadata for a decoder is disclosed. The method includes setting a noise category parameter, setting a block size parameter, setting a threshold parameter, and setting a modulation depth curve parameter.

The method of generating metadata may include setting a noise variance parameter, setting a plurality of block pattern parameters, and/or setting a noise block management table entry parameter.

FIG. 1 shows an exemplary decoder system including a noise synthesis module, according to one embodiment. FIG. 2A shows an exemplary DCT matrix for generating film grain block patterns, according to one embodiment. FIG. 2B shows an exemplary DCT matrix for generating film grain block patterns, according to one embodiment. FIG. 3 shows an exemplary decoder system that utilizes synthetic noise generation, according to one embodiment. FIG. 4 shows an exemplary flowchart for adding noise to the luma channel of an image, according to one embodiment. FIG. 5 shows an example luma modulation curve with a fixed alpha, according to one embodiment. FIG. 6 shows an example luma modulation curve with fixed beta, according to one embodiment. FIG. 7 shows an example of film grain compositing using color transforms, according to one embodiment. FIG. 8 shows an example of camera noise synthesis using color transforms, according to one embodiment. FIG. 9 illustrates an exemplary block diagram for generating content infused with synthetic film grain via hardware, according to one embodiment. FIG. 10 shows an example of how each pixel of an image can be evaluated with respect to film grain in hardware, according to one embodiment. FIG. 11 shows an exemplary 16-bit register for use with a hardware implementation, according to one embodiment. FIG. 12 shows an example method of picking block ID bits from a pseudo-random number generator, according to one embodiment. FIG. 13 shows an example NBMT to pixel value logic, according to one embodiment. FIG. 14 shows exemplary plots comparing software-based (original) and hardware-based (piecewise linear) modulation curves, according to one embodiment. FIG. 15 shows an exemplary block diagram of a hardware implementation of digital camera noise synthesis, according to one embodiment. FIG. 16 shows an example of a simplified camera noise synthesis system/method for chroma channels, according to one embodiment. FIG. 17 shows an example unified noise injection architecture for hardware implementation, according to one embodiment.

Throughout this disclosure, a probability image is defined as a digital image made up of random values.

Noise refers to random variations in luminance (luma) and/or color (chroma) information in a digital image caused by effects other than the imaged environment.

Film grain refers to random optical texture in an image, such as may be caused by film processing. Film grain is noise in the luma channel.

Particle size refers to the average diameter of individual particles of the film grain.

Camera noise refers to random optical textures in an image, such as may be caused by digital camera hardware. Camera noise is noise in both luma and chroma channels. Camera noise, uncombined, comes from many sources, including sensor size, thermal effects, long exposure times, high ISO settings, and so on.

DCT refers to Discrete Cosine Transform and IDCT refers to Inverse Discrete Cosine Transform.

A digital image refers to an image or frame of video that is represented numerically.

Systems and methods for compositing noise, such as film grain or camera noise, onto digital images (including video) are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the presently claimed subject matter may be practiced without these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obscuring the present disclosure.

FIG. 1 shows an exemplary architecture for a decoder implementing noise synthesis. A bitstream (110) carrying encoded images (111) along with corresponding metadata (112) is received by the system. A decoder (120) decodes the image into a decoded clean sequence (125). Since the encoded image (111) is produced by removing any noise prior to encoding, the decoded clean sequence (125) is also (almost) free of noise. The metadata (112) provides parameters for the noise synthesis module (130) and, optionally, the modulation curve module (140). A noise synthesis module (130) uses the metadata (112) to generate a synthesized noise image (135). This synthesized noise image (135) may be run through a modulation curve module (140) to produce a modulated noise image (145). the decoded clean sequence (125) and the modulated noise image (145) (or the synthesized noise image (135) if no modulation is performed) are combined (150) in one or more channels; Produces a rendered image for display (155) or color conversion prior to display.

Noise synthesis does not need to be tied to the decoder. For example, a video producer can use noise synthesis for a video production pre-release. For non-decoder implementations, the decoded clean sequence (125) can be replaced with the content image to be edited. For example, a noise image can be composited by a colorist and then later modulated before adding to the content image.

The noise may be luma channel, chroma channel, or both, depending on the desired effect. For example, for film grain emulation, noise can only be present in the luma channel. Noise may be present in the luma and chroma channels to emulate the noise caused by digital camera hardware. Various types of noise emulation are possible by properly adjusting the metadata parameters.

Noise in a digital image can be synthesized by the process of adding noise block patterns to the image through a probability image. The concept is to add a pixel-by-pixel block pattern instead of dividing the original image into blocks.

To synthesize a film grain block pattern of size B×B, start with a B×B matrix M of standard normal random numbers (eg, B=4, 8, etc.). The matrix M represents the 2D DCT coefficients, the top left being the DC coefficients (frequency=0) and the bottom right being the highest frequency AC (frequency>0). The DC coefficients are labeled 210 as 1 and have increasing labels in a zigzag order as shown in FIG. 2A for the 4×4 block example.

These frequency coefficients can be broadly grouped into three bands. That is, low, medium, and high, as shown in FIG. 2B. The mid-frequency band (230) in M is preserved, while the low-frequency (220) and high-frequency (240) components are removed (or alternatively, only the low-frequency component (220) or components are removed ). For example, in a 4×4 block, frequency coefficients labeled 4-10 may be preserved and coefficients 1-3 and 11-16 removed. The range of frequency coefficients that can be stored is customizable, with at least coefficient 1 removed.

式1 Let _Mmid denote the matrix M in which only the mid-frequency coefficients are preserved and the low/high coefficients are set to zero. The film grain block G _raw may be constructed by taking the inverse DCT of M _mid as given in Equation 1.

formula 1

式2 Blocks are normalized by dividing each element by the energy E. It is then scaled to the desired multiplier V as shown in Equation 2.

formula 2

Th _norm is the threshold normalized to the interval [0,1]. An example value for Th _norm is 0.1. σ _n ² is the variance of the noise. Noise variance is an adjustable parameter. An exemplary range for a 16-bit image may be 1000-6000.

式3 Rounding is performed toward the ends to remove any fractional parts, as shown in Equation 3.

formula 3

Pseudocode for calculating the film grain matrix G is outlined below. The function zigzag() gives the frequency label for each position in the matrix. The range of frequency coefficients to be stored is [f _s ,f _e ].

Multiple noise patterns with different block dimensions can be constructed to be used later to combine the synthetic noise image. 4x4, 8x8, 16x16, and 32x32 are some typical block sizes.
The film grain size can be changed by changing the frequency range. If the proportion of frequency coefficients close to the LF region is significant in the frequency range [f _s ,f _e ], the grain size will be large. Conversely, if the portion of the frequency coefficients closer to the HF region is significant, it produces smaller particles. For the 8x8 block example, the range of frequency coefficients [4 - 10,] produces larger particles than [4 - 36].

（以降、Ｇ^－）に保管することができる。このデータベースは、多くの2Dノイズブロック数が次々にスタックされた3D行列として見ることができる。L×Lがフィルムグレインブロックの最大サイズである場合、Ｇ^－のディメンションはL×L×Qである。Q個の要素を有するベクトルＳは、パターンデータセット内の各ブロックの対応するサイズを保管するように維持され得る。すなわち、i∈[0,Q-1]についてＳ(i)≦L。 Pre-compute some noise blocks with different sizes, and block pattern dataset

(hereinafter referred to as ^G- ). This database can be viewed as a 3D matrix with many 2D noise block numbers stacked one after the other. The dimensions of ^G- are L×L×Q, where L×L is the maximum size of a film grain block. A vector S with Q elements may be maintained to store the corresponding size of each block in the pattern data set. That is, S(i)≤L for iε[0,Q-1].

FIG. 3 shows one example of constructing a synthetic noise image. In this example, the noise simulates film grain. To construct a synthetic noise image (302) of dimension H×W from block patterns (301), we start with the same size probability image (300). Typical noise images of film grain are FHD (Full High Definition, eg 1080p), UHD (Ultra High Definition, eg 4K), and 8K resolutions. A probability image is, for example, a matrix with uniform random numbers in the interval [0,1]. A threshold Th _norm (eg, 0.1) is selected and all entries in the probability image (300) are compared against the threshold. In FIG. 3, circles (310) indicate locations that have a probability below the threshold. Alternatively, locations may be selected at pixels that have a probability greater than, less than, or greater than the threshold. Basically, there is a criterion that compares the probability value at a given pixel to a threshold. Only a few locations are shown for simplicity, but a real probability image may have many more locations. Noise blocks are arranged in a circle. These locations are indicated by the set Ψ.

The noise block can be positioned such that the location in the probability image is at the upper left corner (320) of the noise block, as shown in FIG. Alternatively, the location can be set to the center, other corners, or any designated location for the noise block. The point to which a noise block corresponds to a location is the noise block's “anchor point” and can be the same for all noise blocks.

A block pattern selector (305) randomly selects a film grain block pattern to be inserted at a location within the set Ψ. This block selector can be a uniform random number generator. After the blocks are in place, they can overlap as shown in FIG. Cells from collocated cells in overlapping blocks are added together. This aggregate value goes into the output noise image (302). Since the overlap regions are summed instead of averaged, the film grain blocks can be scaled by the multiplier V instead of the noise variance σ _n ² (see Equation 2). This prevents the aggregate noise pixel value from becoming too large.

To synthesize a film grain image I, consider block pattern data set G ⁻ . Everywhere within the set Ψ, blocks can be randomly picked from the dataset to be inserted into the film grain image. Let r be the index of a randomly chosen block for the current location. The dimension of this block G ^' is S(r)*S(r), so that G ^' = ^G- (0:S(r)-1,0:S(r)-1,r) . A different random number r may be selected each time there is a new block to be inserted at the current location. The virtual code for this is shown below.

G ^' denotes a random block selected for location i,j based on the random number r. In other words, for each location i,j, a random number r is chosen and then G ^' is generated as a function of r.

For synthesizing film grain or similar noise, the noise can be very annoying and intrusive in bright areas of the image. The intensity-based luma modulation curve reduces the intensity of noise in these bright regions, making film grain injected images more desirable to viewers. As explained in the previous section, the film grain image has been pre-generated. These images are then modulated based on the co-located luma intensities and added to the luma channel.

FIG. 4 illustrates, for one embodiment and without limitation, a general method for adding noise to the luma channel (eg, Y channel) of an image. First, a noise image I is initialized (405). The noise image can have the same dimensions as the Y channel image. Initialization can set all pixel values of the noise image to 0, or some other set value. A probability image P is then generated (410), where at each location in P a random value is generated. The probability image can have the same dimensions as the noise image. The random value can range from 0 to 1 or between any two values. A location map Ψ (415) is then generated by comparing each probability value in P for each pixel location to a threshold criterion (eg, being less than or equal to a set threshold). For each location in Ψ identified as meeting the threshold criteria, a randomly selected noise block is added to the corresponding pixel location in I. If the block is larger than 1×1, the block may be a given anchor point (eg, the pixel in the upper left corner of the block), where the block is anchored at the corresponding pixel location in I. Once all identified locations in Ψ have been processed, the aggregate of added noise blocks form the composite noise image in I (425). The synthetic noise image can be modulated by a modulation index (435) that can be calculated from the luma channel (430). The modulated synthetic noise image can then be added to the luma channel (440) of the image.

Examples of luma modulation curves are shown in FIGS. These graphs show the modulation depth for different values of luma intensity. The modulation factor is a rational number in the range [0,1]. In some embodiments, without limitation, the factor is close to unity for low strength codewords and decreases as strength increases. The system may multiply the film grain value by its modulation depth before adding the film grain pixels to the co-located pixels in the luma channel. The modulation index is derived from the co-located luma intensity value and the modulation curve. Due to the attenuation trend of the curve, the film grain intensity is reduced for bright areas in the image.

（式4）
式4において、αおよびβは、曲線パラメータを示している。 Assume that the intensity value of the luma channel is denoted by l in the codeword range of B _t bits. As an example and without limitation, the modulation depth curve f(·) can be calculated as shown in Equation 4.

(Formula 4)
In Equation 4, α and β indicate curve parameters.

FIG. 5 shows an example of a luma modulation curve for variables α and β=2 (B _t =16). Curves are shown for α=0.15 (501), 0.25 (502), 0.4 (503).

FIG. 6 shows one example of a luma modulation curve for variables α=0.25 and β=2 (B _t =16). Curves are shown for β=1.0 (601), 2.0 (602), 4.0 (603).

The freedom to choose α or β allows using the same equations for different Electro-Optical Transfer Functions (EOTF).

Y(i,j)=y where _ij is the luma intensity at location (i,j) in the Y channel image, the noise value at location (i,j) in the noise image is the noise injected luma value Multiplied by the luma channel and added to the luma channel to obtain

FIG. 7 shows an example of film grain compositing using color transformation. A probability image (705) is generated by comparing the value at each pixel of the probability image (705) to a threshold, and a location is selected (710). In this example, locations are pixels with values below a threshold. A set of film grain patterns (715) is pre-generated, and blocks are randomly selected (720) to be injected into the probability image (725) to generate the film grain image (730). . The Y (luma) channel (735) of the image is modulated (740) to produce an image with modulation to noise (745). The film grain image (730) and the modulation depth image (745) are combined (750) to produce the luma modulated film grain image (755). The luma modulated film grain image (755) is combined (760) with the Y channel image (735) to produce a film grain added Y channel image (765). A YUV image containing the film grain added Y channel can be color converted (770) to produce an RGB image with film grain noise added for all channels (775).

Digital camera noise can be synthesized using Gaussian random numbers. Random numbers can be generated from uniform random numbers using, for example, the Box Muller transform or the Marsaglia Polar method.

は、平均0(zero)および標準偏差1（unit）のガウス乱数である。条件が満たされる前に、プロセスがあまりにも長い間ループすることを許さないように注意する必要がある。 For the Marsaglia Polar method for generating Gaussian random numbers from uniform random numbers, suppose u ₁ and u ₂ are two uniform random numbers chosen from the range [0,1]. If the condition 0<u ₁ ² +u ₂ ² < 1 is satisfied,

is a Gaussian random number with mean 0 (zero) and standard deviation 1 (unit). Care must be taken not to allow the process to loop too long before the condition is met.

Since digital camera noise can be simulated using Gaussian random numbers, synthesizing a camera noise image results in generating a 2D matrix that has the dimensions of the image and contains zero-mean Gaussian random numbers. . The noise image is modulated with luma intensity as described above. A noise image is then added to the Y, U, and V channels of the content, optionally followed by a color space conversion (eg, to RGB).

FIG. 8 shows an example of camera noise synthesis using color transformation. This is a further embodiment of noise synthesis, and features common to both embodiments may be omitted from the description, but should still be understood to be common to both. The Y (luma) channel (810) of the image is modulated (805) to produce a Modulation Intensity Image (“MFI”, 815). Three random Gaussian images (“RGI”, 820), one for each channel, are generated, and the modulation depth image is used (815) to generate three luma modulated noise images (830 ). These noise images (830) are added (835) to respective channels (840) in the original image. These channels (840) can then be converted to different color encoding schemes (850), such as RGB. The luma modulated noise image (830) represents the pixel-by-pixel magnification (825) of the MFI image (815) using the RGI image (820) for the corresponding channel.

When implementing noise synthesis in hardware, as opposed to software solutions, four factors should be considered for optimization.

1. In a software implementation, the noise image is synthesized separately and then added to the luma channel. Due to memory limitations in the hardware, only the film grain image pixels are calculated and added directly to the content luma channel. This approach eliminates any intermediate memory buffers for storing the entire film grain image and saves a significant amount of on-chip memory.

2. In a hardware implementation, images are accessed through line buffers. A line buffer can store rows of an image. Because hardware memory is expensive, the number of line buffers is typically limited, so the entire image cannot be loaded into these few line buffers. Reading and writing to memory also takes time. Thus, a hardware solution should read a line of the image into a line buffer once, perform the entire process, and write the result to memory only once. Multiple reads and writes to the line buffer of the same portion of the image is not optimal.

3. Floating point representation is not hardware friendly, so values are handled more efficiently in integer format for hardware implementation. Noise block patterns, image pixels, thresholds, etc. can be expressed in integer form to allow for simpler hardware implementations. For example, instead of setting the probabilities from 0 to 1 (real numbers), you can set them from 0 to 255 (integers).

4. Block sizes in hardware implementations are fixed for consistency across all decoder hardware chips (eg, all blocks are 8x8). For the rest of the description, the fixed block size is denoted by B×B.

The noise synthesis algorithm can be adjusted to take these factors into account so as to optimize the system for hardware implementation.

FIG. 9 shows an exemplary block diagram for generating noise, in this case film grain, content injected via hardware. Film grain (945) is added only to the Y (luma) channel (955), but other noise injection techniques can include V and U channels as well. The processor clock (timestamp/date/time/etc) can be used as the frame-level random number seed (905). This random seed is used to initialize all row-level pseudo-random number generators (910). Each pixel is evaluated with respect to film grain (920) and then added (940) to the luma channel (925) of the content after modulation (930, 935). The film grain added content (956) is finally converted to RGB (965) for display (970).

FIG. 10 shows an example of how each pixel can be evaluated with respect to film grain. A uniform block size of B×B is assumed for the film grain block pattern data set. Different block sizes serve to change the size of the film grain, but the same effect can be achieved by changing the frequency range for blocks of the same size. Therefore, to reduce the number of variables, the block size of the film grain pattern can be set to a fixed size when implemented in hardware. A film grain block pattern can be generated using the same scheme as described for the software implementation. FIG. 10 shows the steps for generating the film grain value at column C in row R.

There is no explicit probability image in hardware solutions. Instead, there is a pseudo-random number (PRN) sequence generator (1010-1015) for each row of the image. For each entry (pixel) in row R, the PRN module generates a new random number. The PRN can be seeded (1005) from a function of row index and random offset. film grain pattern such that if the random number is less than the threshold (1020) and there is enough room in the noise block management table (NBMT) (1090), the film grain block (1070) is inserted Randomly selected from the set of (1030), starting from this position (1065). The block is not added to the image immediately, rather the system puts an entry (1025) for that block in the NBMT (1090). The NBMT-to-pixel-value logic block (1035) calculates the noise value at the current location in the luma channel image (1045). Due to the hardware implementation, the threshold Th is not normalized, thereby reducing computational cost. The non-normalized threshold is denoted without the subscript "norm" in the symbol and is related to the normalized threshold by the equation Th=round(Th _norm ×2 ^Bt ).

The reason for using the pseudo-random number generator from the previous row while calculating the noise value at the current location is one of efficiency. If B (noise block size) is significantly large, reading/writing B rows in each pass can reduce memory input/output (I/O) and hardware costs by requiring B-line buffers in memory. increase significantly.

In contrast, our solution only requires a one-line buffer, regardless of block size. For a given position there may be overlapping blocks from previous rows. These blocks are indicated in FIG. 10 by the dashed squares (1070) in the image (1060). Since the random number seed for each PRN generator is deterministic, the order of random numbers passed by earlier rows can be reproduced. These sequences provide the precise location of blocks and block patterns used in earlier rows. This information allows calculation of the correct film grain value for any position with only one line buffer, thereby reducing I/O latency and hardware costs. A noise block management table (990) is maintained for each row to track overlapping blocks.

Although the original and grain-implanted luma channel rows R are shown separately in FIG. 10, the original rows R can be updated in place to obtain the film grain injection rows R.

A film grain block pattern is generated using a process similar to that described for the software-based solution. Since the mechanism of computing these block patterns requires the use of complex mathematical operations, this task can be handed over to the firmware. The synthesized blocks can later be stored in on-chip SRAM for easy hardware access.

Another major challenge for hardware implementation is the random number generator. For faster implementation, a linear feedback shift register based pseudo-random number generator can be used. Some bits in these random numbers are used to select film grain blocks from the block pattern data set.

Linear feedback shift registers are commonly used to generate sequences of random numbers within hardware chips. Consider the exemplary 16-bit register shown in FIG. The next random number is obtained by right shifting the contents of the register by 1 and inserting the new bit at the most significant bit (MSB) location (1110). The new bit is the exclusive OR (XOR) of the bits at a given predefined location (1120) in the register. In the example shown in FIG. 11, the bits at index locations 3, 12, 14, and 15 are XORed. The output of the XOR gate (1115) is fed to the MSB location (1110).

If the tap point location is chosen correctly, the register goes through every different 16-bit number once before returning to the same number. In other words, the sequence repeats after a predetermined period of time. It can be seen that the next random number is a function of the current state of the register. Therefore, care must be taken to ensure that registers are not initialized to all zeros. If all zeros, the process is locked in that state.

The hardware implementation does not explicitly generate probability images of H×W dimensions. As a workaround, random numbers for each row can be generated using a linear feedback shift register. The shift registers assigned to each row are initialized with different random seeds. This seed is a function of row index R and random offset.
Seed _R = (Const + R x Offset) mod ^2D
where D is the length of the register. A random offset is derived from the current timestamp, and a constant is set to an arbitrarily large number, such as 2 ¹⁴ for a 16-bit shift register. The offset is randomly chosen but fixed for all rows in the current frame. There is a different pseudo-random number generator for each row, but there are no more than B shift registers in hardware at any given time. Here, B×B denotes the dimension of blocks in the block pattern data set.

The seed is deterministic because the constant, offset, and row index are known. The idea behind making this seed deterministic is to reproduce the sequence of random numbers that earlier rows passed through when dealing with the current row R. This allows us to determine the location of overlapping film grain blocks from earlier rows and use that knowledge to calculate the film grain value for the current location.

Along with the random sequence generator, a table can be used to store information about the location of blocks in previous rows. This table is called a Noise Block Management Table.

テーブル１：ノイズブロック管理テーブル NBMT stores the information of the film grain blocks injected into the luma channel for a particular row. Every PRN sequence generator (see Figure 10) has a corresponding NBMT. The number of entries in the table is a configurable parameter, as shown in Table 1, and each entry holds three columns:

Table 1: Noise block management table

To compute the final value at each location, the system needs to consider all blocks that overlap with that location. The noise block management table keeps track of these overlapping blocks by storing the block ID b _i , the origin x-coordinate x _i , and the origin y-coordinate y _i of the blocks in the image. For speed and computational efficiency, the maximum number of entries in the table is limited to ceil(B/4).

For example, suppose the size of the film grain block pattern is B×B, and the system is calculating the film grain injection value at location (C,R) in the noise injected luma channel image, then up B−1 A row or block originating in column B-1 to the left can still affect the current location.
For this reason, while processing the current row R, the system considers B-1 rows above it.

As shown in Figure 10, the system starts a pseudo-random number generator for B-1 rows above the current row R to find overlapping blocks. The current row can also have its own random number generator, denoted PRN Row R. Since the random number seed is deterministic, the system can always reproduce the order through which the previous rows passed. Apart from a row specific random number generator, each of these B rows can also have its own NBMT.

Let C denote the current column in the row being processed. Row R of the luma channel image is read into the line buffer. Column C is incremented in steps of 1 from C=0 to C=W-1. For each value of C, the system can perform the following steps.

1. Each of the B random number generators generates a random number. If the numerical value is greater than or equal to the threshold Th, nothing is done.

2. If the random number is less than the threshold,
a. Remove any entries in the NBMT that do not affect the current location. For example, if the block size is B×B and x _i <C-B+1 or y _i <C-B+1 are deleted.
b. If the NBMT is not full for that row, select a random block from the film grain pattern data set. Bits from the row's pseudo-random number generator are used to select the block ID b _i . FIG. 12 represents one example of how the block ID bits are selected from the pseudo-random number generator. For a dataset of 16 different block patterns, block ID [15:12]=7 refers to block ID 7, which is the 7th pattern in the dataset. Ordering starts at index 0.
c. An entry for the block is added to the NBMT with x _i =C and the Y coordinate equals the row index. Block ID is derived from the previous step.

3. Compute the film grain value at location (C,R) from the NBMTs of all R rows. This step is called NBMT to pixel value logic and is illustrated in FIG.
a. For each entry in all NBMTs, check if the block overlaps the current location (1310). Blocks overlap if x _i ≤C≤B+x _i -1 and y _i ≤R≤B+y _i -1.
b. Using the block ID extracted from the PRN generator, select the corresponding block from the block pattern data set (1320). Pick the value in the block that overlaps the current location.
c. Since there may be multiple such overlapping blocks, the values are added together (1330).
d. This value is passed to the next stage which computes the luma modulation index (1340) for the noise value. The luma modulated noise value is added (1350) to the original luma (1355) and written to the line buffer in column (1360).

4. Each row goes through this process one at a time to obtain a film grain injected luma channel image. For rows that do not have a B-1 row on them, only available rows are considered.

For hardware-implemented luma intensity-based noise modulation techniques, software-implemented nonlinear curves should be replaced by piecewise linear segments to avoid wasting memory cycles in nonlinear operations. These line segment parameters can be signaled to the decoder in metadata. The system can then calculate the modulation index for each luma intensity value from the piecewise linear curve.

における値は、区分的線形近似の前に、B_c=8ビット整数にスケール化され得る。すなわち、0≦l≦2^Bc-1についてf_hw(l)=round(f(l)×2^Bc)である。曲線f_hw(・)は、次いで、T+1個のピボット点を使用して、T個のセグメントへパーティション分割される。ピボット点の場所をベクトルΩで表されるようにする。つまり、0≦k≦Tについて0≦Ω(k)≦2^Bc-1である。 Furthermore, the values in the curve for software implementation are fractional. Fractions are cumbersome in hardware. Therefore the curve

The values in can be scaled to B _c =8-bit integers before piecewise linear approximation. That is, f _hw (l)=round(f(l)×2 ^Bc ) for 0≦l≦2 ^Bc −1 . The curve f _hw (·) is then partitioned into T segments using T+1 pivot points. Let the location of the pivot point be represented by the vector Ω. That is, 0≦Ω(k)≦2 ^Bc −1 for 0≦k≦T.

式５

式６ For _every two pivots on the curve, i.e. x1=Ω(k) and x2= _Ω (k+1), the kth Compute the line segment formula. The line equation has three parameters. Slope, X-intercept (x1), Y-intercept (y1).

Equation 5

Formula 6

変調度がB_cビットのコードワード範囲内なので、フィルムグレイン値と変調度の積は、正しいルマ変調フィルムグレイン値を得るために、B_cビットだけ右シフトされ得る。 The slope, pivot, and Y-intercept parameters of each segment are sent to the decoder. The curve is reconstructed at the decoder to compute the modulation index for each luma intensity. For illustration, FIG. 14 plots the original (software-based) piecewise linear curve and the piecewise linear curve for α=0.25, β=2. If Y(C,R)=y _CR is the luma intensity at location (C,R) in the Y channel, then the noise value, n _CR , is f(y _CR ) to obtain the noise injected luma value and added to the luma channel.

Since the modulation index is within the codeword range of B _c bits, the product of the film grain value and the modulation index can be right-shifted by B _c bits to obtain the correct luma modulated film grain value.

Similar to film grain noise, the hardware implementation of digital camera noise also has some limitations.

1. Gaussian random number synthesis in hardware is expensive due to the log function involved. As a workaround, these numbers can be generated in firmware and stored as the digital camera's noise block.

2. Due to memory limitations, storing digital camera noise images in on-chip memory is expensive. Noise pixels can be calculated and added to the Y, U, and V channels directly after modulation.

3. Only a few lines of the Y, U and V channels are read into memory as line buffers for processing. Processing the entire image is I/O and memory intensive as the image size can be huge. Line buffers help alleviate latency and memory requirements. However, the architecture can vary significantly to achieve similar results as software solutions.

4. For hardware implementation, floating point arithmetic is more complex and all values are converted to fixed point format. When moving from floating-point to fixed-point arithmetic, some changes to the architecture are required.

5.The block size of the noise pattern of the digital camera can be fixed to 1×1 (B=1). This essentially means that the system is adding only one Gaussian random number everywhere. This is in line with software solutions.

One embodiment of digital camera noise synthesis in hardware limits the block size of the digital camera noise pattern to 1×1, and the probability threshold for row R is set to ^2D , while the other rows are , has a threshold of zero. Setting the probability threshold to ^2D ensures that blocks are added at every pixel location in the Y channel. Since the block size is limited to 1x1, there is no overlap between blocks. Therefore, there is only one entry in the NMBT of row R.

There is a pseudo-random number generator for each row. While processing row R and column C, if the pseudo-random number is less than ^2D , the system adds a digital camera block to that pixel. Since these random numbers are always less than ^2D , the system adds Gaussian random numbers at each location. Blocks are randomly selected using the leading bits of a pseudo-random number generator. The information for that block can be inserted into the NBMT for that row. NBMT's logic for pixel values computes the digital camera's noise value. This value is modulated using the modulation index and added to column C of the line buffer. For size 1×1, there is no overlap from adjacent blocks, so B=1 and the logic of NBMT for pixel values is simple. FIG. 15 shows one exemplary block diagram of a hardware implementation of digital camera noise synthesis, which can be described as a special case of FIG.

Digital camera noise is also added to the chroma channel, as opposed to film grain, which is only added to the luma channel. The architecture of camera noise injection into luma and chroma channels is essentially the same. The design for the chroma channel removes the superfluous pseudo-random number generator, threshold block, NBMT and NBMT from the pixel value logic from the hardware architecture of Fig. 15, as shown in the example of Fig. 16. can be simplified by doing

For example, assume there are 1024 block patterns for digital camera noise. Then the first 10 bits of PRN Row R can be used to calculate the block index b _i . A 1×1 block is taken from the dataset, modulated with luma intensity, and added to the chroma channel line buffer at the current location.

For YUV 444 sequences, the modulation index can be calculated from the co-located intensities in the luma channel. A noise value is added to the chroma channel after modulation. In other words, the noise value is first modulated with the luma intensity and then added to the chroma channel.

For YUV 422 sequences, the Y channel can be subsampled by half in width, and then the modulation index is calculated from the co-located pixels within the Y channel. The noise value is modulated with the downsampled luma intensity and then added to the chroma channel. Downsampling approaches using conventional filtering approaches can be expensive in terms of hardware. However, the modulation index can work with both crude and approximate downsampling. To reduce hardware complexity, the luma channel is subsampled by simple averaging of adjacent elements in the row of the luma channel. This process is detailed in the pseudocode below. The symbol Y _hh denotes a horizontally downsampled version of the luma channel Y by averaging adjacent elements in the same row.

FIG. 17 shows an example of a unified noise injection architecture for hardware implementation, allowing the synthesis of multiple types of noise (film grain and camera in this example). A pseudo-random number generator (1710) can generate a sequence of random numbers. If these random numbers satisfy a threshold condition, then aggregate using the film grain (1721) or digital camera noise (1722) block data sets, depending on which type of noise is determined (1715) needed. A noise value is calculated (1720). This value is modulated 1730 with luma intensity and added to the original luma or chroma channel. Since 1740 one line is processed at a time, the line buffer need only be large enough to accommodate a line of YUV content. The noise injected YUV content is converted 1750 to RGB for display.

There are two different scenarios in which the noise injection model is used to inject noise into content. In the first scenario, the content provider can control the look of the noise added to the video. Parameters are set at the encoder side and sent to the decoder as part of the metadata. For the second scenario, the decoder can synthesize noise based on viewing conditions or display specifications. Content provider settings may have higher priority than local adjustments. And vice versa.

The noise synthesis parameters can be set at different stages of the workflow: the content provider's preference settings on the encoder side or the local adaptive settings on the decoder side. Decoder parameters may be configured into firmware or hardware.

テーブル２ For artistic reasons, a content provider may wish to add film grain or camera noise to a video sequence for visual effects. The noise is compressed during encoding and as a result the cinematic effect is reduced at the decoder side. Furthermore, since a higher bitrate is required to compress a noisy sequence, transmitting the noiseless sequence along with the parameters for generating the noise at the decoder side reduces the bitrate requirement, and It still retains its cinematic effect. For that purpose there is an option to set the parameters on the encoder side and send them to the decoder via metadata. The parameters are shown in Table 2.

table 2

Parameters can be read from the metadata to initialize the noise generation module on the decoder side. Similarly, parameters may be adjusted at the decoder side based on viewing environment or display specifications. Either way, the decoder is given a set of parameters and needs to synthesize the noise. A decoder consists of a firmware or processor and a hardware system. Firmware can compute complex mathematical functions at the expense of speed, while hardware can perform low-level logic operations at a faster speed.

Based on speed requirements and computational complexity, the task of noise image generation needs to be split between firmware and hardware. Synthesizing film grain or camera noise blocks is computationally intensive for hardware and can be handed off to firmware. Since there are only a few block patterns to build, the firmware can easily build them during system bootup without much processing overhead.

exemplary embodiment
EE1. A computer-based method for generating a composite noise image for a digital image, the method comprising the steps of providing a plurality of noise block patterns, initializing the noise image, and generating a probability image. wherein the probability image includes a randomly generated probability value at each pixel; and comparing each probability value in the probability image to a threshold criterion. selecting a noise block pattern from the plurality of noise block patterns if the probability value satisfies the threshold criterion, the noise block pattern having an anchor point, and if the probability value satisfies the threshold criterion; and generating a synthesized noise image by placing the noise block pattern within the noise image such that the anchor points correspond to pixels in the probability image that meet the threshold criteria.

EE2. The method of EE1, wherein the noise grain block pattern is randomly selected.

EE3. The method of EE1 or EE2, wherein said probability values of said probability image are uniformly distributed probability values.

EE4. The method of any of EE1-EE3, wherein said noise block pattern comprises a film grain block pattern.

EE5. The method of EE1-EE4, wherein the method further comprises summarizing values in regions of the noise image where the noise block patterns overlap.

EE6. The method of any of EE1-EE5, wherein the method further comprises generating the noise block pattern prior to generating the noise image, wherein the noise block pattern is generated through an inverse DCT technique.

EE7. The method of EE6, wherein the generating step further includes removing low frequency DCT coefficients.

EE8. The method of EE6 or EE7, wherein the step of generating further includes removing high frequency DCT coefficients.

EE9. The method of any of EE1-EE8, wherein said probability image and said noise image are sized to match a video frame.

EE10. The method of any of EE1-EE9, further comprising editing the content image by adding the noise image to a channel of the content image.

EE11. The method of EE10, further comprising multiplying values in the noise image by a modulation index before adding the noise image to the content image.

EE12. The method of EE11, wherein said values are luma intensity values of said noise image.

EE13. The method of any of EE10-EE12, wherein the method further comprises generating a YUV content image based on the edited content image.

EE14. The method of EE13, wherein the method further comprises generating an RGB content image based on the YUV content image.

EE15. A decoder for decoding the encoded image to produce a decoded clean sequence and a noise synthesis module performing any method EE1-EE14 to produce a synthesized noise image; , wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to produce a rendered image for display.

EE16. The device of EE15, wherein said device further comprises a modulation curve module for modulating said synthesized noise image prior to combining with said decoded clean sequence.

EE17. A computer program comprising data, said data encoded to perform any of the methods EE1-EE14 when executed on a processor.

EE18. A device for generating synthetic noise in an image, said device providing a line buffer containing lines from said image, firmware containing noise block patterns, and random values for pixels in said lines. logic for comparing the random value to a threshold criterion; a data store containing a noise block management table; and a random noise block pattern from the firmware associated with pixels in the row. wherein the random value associated with the pixel satisfies the threshold criterion; and logic for generating the noise block management table based on the noise block management table for each pixel in the row of the image. A device comprising logic to calculate a noise value for a location and logic to modify the row based on the noise value.

EE19. The device of EE18, wherein the device further comprises logic for determining a modulation curve from the image, calculating a modulation index, and applying the modulation index to the noise value before modifying the row.

EE20. A pseudo-random noise generator is an EE18 or EE19 device that generates a random number based on a row index and a random offset value.

EE21. The device of any of EE18-EE20, wherein said image is a luma channel image.

EE22. The device of any of EE18-EE20, wherein said image is a chroma channel image.

EE23. The device of any of EE18-EE22, wherein said device further comprises multiple pseudo-random number generators and multiple line buffers for processing said images in parallel.

EE24. A decoder for decoding the encoded image to produce a decoded clean sequence, and a noise synthesis module comprising any device EE18-EE23 for producing a synthesized noise image; wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to generate a rendered image for display.

EE25. A method of generating metadata for a decoder, said method comprising the steps of: setting a noise category parameter; setting a block size parameter; setting a threshold parameter; and setting.

EE26. The method of EE25, wherein the method further comprises setting a noise variance parameter.

EE27. The method of EE25 or EE26, wherein the method further comprises setting a discrete cosine transform frequency range parameter.

EE28. The method of any of EE25-EE27, wherein the method further comprises setting a plurality of block pattern parameters.

EE29. The method of any of EE25-EE28, wherein the method further comprises the step of setting entry parameters of a noise block management table.

A number of embodiments of the disclosure have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other aspects are within the scope of the following claims. Any salient feature of the embodiments, where appropriate, is common to all embodiments.

The above examples are provided as a complete disclosure and illustration of how to make and use the embodiments of the present disclosure, and are intended as the basis for the inventor's use of these disclosures. is not intended to limit the scope of what is considered

Modifications of the above-described modes for implementing the methods and systems disclosed herein that are obvious to those skilled in the art are intended to be within the scope of the following claims. All patents and publications mentioned in the specification are indicative of the level of skill of those skilled in the art to which this disclosure pertains. All references cited in this disclosure are incorporated by reference to the same extent as if each reference were individually incorporated by reference in its entirety.

It is to be understood that this disclosure is not limited to a particular method or system, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an", and "the" do not expressly prescribe otherwise. as long as it encompasses more than one referent. The term “plurality” includes two or more referents unless the content clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

Claims (29)

A computer-based method for generating a synthetic noise image for a digital image, the method comprising:
providing a plurality of noise block patterns;
initializing the noise image;
generating a probability image, the probability image comprising a randomly generated probability value at each pixel;
comparing each probability value in the probability image to a threshold criterion;
including
if the probability value meets the threshold criterion,
selecting a noise block pattern from the plurality of noise block patterns, the noise block pattern having an anchor point; and
if the probability value meets the threshold criterion,
generating a synthesized noise image by placing the noise block pattern within the noise image such that the anchor points correspond to pixels in the probability image that meet the threshold criteria;
Method. A noise grain block pattern is randomly selected,
The method of claim 1. the probability values of the probability image are uniformly distributed probability values;
3. A method according to claim 1 or 2. wherein the noise block pattern comprises a film grain block pattern;
4. A method according to any one of claims 1-3. The method further comprises:
aggregating values in regions of the noise image where the noise block patterns overlap;
5. A method according to any one of claims 1-4. The method further comprises:
generating the noise block pattern prior to generating the noise image;
the noise block pattern is generated through an inverse DCT technique;
6. A method according to any one of claims 1-5. The generating step further comprises:
removing low frequency DCT coefficients;
7. The method of claim 6. The generating step further comprises:
removing high frequency DCT coefficients;
8. A method according to claim 6 or 7. the probability image and the noise image are sized to match a video frame;
9. A method according to any one of claims 1-8. The method further comprises:
editing the content image by adding the noise image to a channel of the content image;
10. A method according to any one of claims 1-9. The method further comprises:
multiplying values in the noise image by a modulation index before adding the noise image to the content image;
11. The method of claim 10. the value is the luma intensity value of the noise image;
12. The method of claim 11. The method further comprises:
generating a YUV content image based on the edited content image;
13. A method according to any one of claims 10-12. The method further comprises:
generating an RGB content image based on the YUV content image;
14. The method of claim 13. a decoder for decoding the encoded image to produce a decoded clean sequence;
a noise synthesis module for performing the method of any one of claims 1 to 14 to generate a synthetic noise image;
a device comprising
the decoder is configured to combine the synthesized noise image with the decoded clean sequence to produce a rendered image for display;
device. The device further comprises:
a modulation curve module for modulating the synthetic noise image prior to combining with the decoded clean sequence;
16. Device according to claim 15. A computer program containing data,
The data are encoded so as to perform the method of any one of claims 1 to 14 when executed on a processor.
computer program. A device for generating synthetic noise in an image, said device comprising:
a line buffer containing lines from the image;
firmware containing noise block patterns;
a pseudo-random number generator for providing random values for pixels in the row;
logic for comparing the random value to a threshold criterion;
a data store containing a noise block management table;
logic to generate the noise block management table based on random noise block patterns from the firmware associated with pixels in the row, wherein the random values associated with the pixels satisfy the threshold criteria; and,
logic to calculate a noise value for each pixel location in the row of the image based on the noise block management table;
logic to modify the row based on the noise value;
device, including The device further comprises:
logic for determining a modulation curve from the image, calculating a modulation depth, and applying the modulation depth to the noise values before modifying the row;
19. Device according to claim 18. a pseudorandom noise generator generating a random number based on the row index and the random offset value;
20. Device according to claim 18 or 19. wherein the image is a luma channel image;
21. A device according to any one of claims 18-20. wherein the image is a chroma channel image;
21. A device according to any one of claims 18-20. The device further comprises:
comprising multiple pseudo-random number generators and multiple line buffers for processing the image in parallel;
23. A device according to any one of claims 18-22. a decoder for decoding the encoded image to produce a decoded clean sequence;
a noise synthesis module, comprising a device according to any one of claims 18 to 23, for generating a synthetic noise image;
A decoder system comprising:
the decoder is configured to combine the synthesized noise image with the decoded clean sequence to produce a rendered image for display;
decoder system. A method of generating metadata for a decoder, the method comprising:
setting noise category parameters;
setting a block size parameter;
setting a threshold parameter;
setting modulation depth curve parameters;
A method, including The method further comprises:
setting a noise variance parameter;
26. The method of claim 25. The method further comprises:
setting a discrete cosine transform frequency range parameter;
27. A method according to claim 25 or 26. The method further comprises:
setting a plurality of block pattern parameters;
28. The method of any one of claims 25-27. The method further comprises:
setting entry parameters of the noise block management table;
29. The method of any one of claims 25-28.

Images