CN114945944A - Noise synthesis for digital images - Google Patents

Abstract

Apparatus and methods for providing a software and hardware based solution to the problem of noise in synthetic digital images. According to one aspect, a probability image is generated and noise blocks are randomly placed at locations in the probability image, thereby creating a composite noise image, where the locations have probability values that are compared to a threshold criterion. Embodiments include generating a composite film grain image and a composite digital camera noise image.

Description

Noise synthesis for digital images

Cross Reference to Related Applications

This application claims priority to U.S. provisional patent application No. 62/950,466 filed on 12/19/2019 and european patent application No. 19218144.4 filed on 12/19/2019, both of which are incorporated herein by reference in their entirety.

Technical Field

The present disclosure relates generally to images. More particularly, embodiments of the present disclosure relate to film grain and camera noise for composite images.

Background

As used herein, the term noise synthesis refers to the generation of noise effects for digital images (including video frames) by means other than the way noise normally occurs. Unless a particular meaning is explicitly specified in the description herein to have a particular meaning at any time, it should be inferred that the terms may be used in either sense, e.g., interchangeably.

Prior to the advent of digital technology, photographic film strips were the canvas for traditional photography. Such a film is a thin plastic sheet covered with a photosensitive material (e.g., silver halide) to absorb light emitted or reflected from an object to be photographed. After processing the films, the captured light can be converted into an image. As used herein, "film grain" refers to the visual texture formed in the processed image due to the dye cloud (or silver particles) receiving sufficient photons. It is an optical phenomenon caused by chemical processes that occur during processing of the film and depends on both the clarity with which the film is viewed and the type of film library used.

Digital photography inherently lacks film grain effects. It is still desirable for artistic and quality reasons. For example, hollywood studios prefer movie content with film grain. For digital images, the film grain pattern must be synthesized and superimposed on the image during post-processing.

"camera noise" refers to changes in brightness and/or color (color speckles on an image) in a digital image due to camera electronics (e.g., sensor size, exposure time, ISO settings, etc.). Typically, camera noise is removed from digital images by different techniques in order to avoid the problem of compression, but in some cases it is desirable for the camera noise to cause the image to retain the original "look" the image had when it was captured.

As used herein, the term "metadata" relates to any auxiliary information that is transmitted as part of the encoded bitstream and that assists the decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

In practice, a digital image includes one or more color components (e.g., luminance Y and chrominance Cb and Cr), where each color component is represented with an n-bit per pixel precision (e.g., n-8). Linear luminance coding is used, where an image with n <8 (e.g., a color 24-bit JPEG image) is considered a Standard Dynamic Range (SDR) image, while an image with n >8 may be considered an Enhanced Dynamic Range (EDR) image. EDR and High Dynamic Range (HDR) images may also be stored and distributed using a high precision (e.g., 16-bit) floating point format, such as the OpenEXR file format developed by Industrial Light and Magic.

As used herein, the term "noise synthesis" includes, but is not limited to, synthesizing noise (such as film grain or camera noise) in digital images that is typically caused by film/video/image capture or development. The term may also refer to any noise that is artificially inserted into a digital image for any purpose.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Accordingly, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the identified problems with one or more methods should not be considered as having been recognized in any prior art based on this section unless otherwise indicated.

Disclosure of Invention

The apparatus and method according to the present disclosure provide a solution to the problem of noise in the synthesized digital image. According to an embodiment of the present disclosure, the synthesized film grain is added to a clean (i.e., noise-free) digital sequence.

The process of synthesizing noise as presented herein has many advantages in image transmission and display. For example, the ability to synthesize noise after transmission allows for the transmission of noise-free compressed images, which may provide improved transmission efficiency (data speed, bandwidth, etc.). Adding synthesized noise to an uncompressed image may reduce the effect of compression artifacts in the displayed image. The use of noise blocks allows control over film grain size, sharpness, and type (noise source emulation) to be achieved, and the random placement of noise blocks in the image prevents blocking artifacts that would result from tiling placement. Furthermore, for hardware-based implementations, using inline processing allows for resource-efficient implantation of noise synthesis.

According to a first aspect of the present disclosure, there is disclosed a computer-based method for creating a composite noisy image for a digital image, the method comprising: providing a plurality of noise block patterns; initializing a noise image; generating a probability image comprising randomly generated probability values at each pixel of the probability image, comparing each probability value in the probability image to a threshold criterion; selecting a noise block pattern from the plurality of noise block patterns if the probability value satisfies the threshold criterion, the noise block pattern having an anchor point; and if the probability value satisfies the threshold criterion, creating a composite noise image by placing the noise block pattern in the noise image such that the anchor point corresponds to pixels in the probability image that meet the threshold criterion.

The noise particle patch pattern may be randomly selected. The probability value of the probability image may be a uniformly distributed probability value. The noise block pattern may comprise a film grain block pattern. The method may further include aggregating values in a region of the noise image where the noise block patterns overlap. The method may comprise generating the noise block pattern prior to generating the noise image, wherein the noise block pattern is generated by an inverse DCT technique. This generation may include removing low frequency DCT coefficients and/or removing high frequency DCT coefficients. The size of the probability image and the noise image may be determined to match a video frame. The method may include editing a content image by adding the noise image to a channel of the content image. The method may comprise multiplying values in the noise image by a modulation factor before adding the noise image to the content image. The value may be a luminance intensity value of the noise image. The method may include generating a YUV content image based on the edited content image. The method may include generating an RGB content image based on the YUV content image. The method may be run on a computer program product comprising data encoded to perform the method.

The method may be run by a noise synthesis module to create a synthesized noise image for the device. The apparatus may include a decoder to decode an encoded image to create a decoded clean sequence, wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to create a rendered image for display. The apparatus may include a modulation profile module to modulate the synthesized noise image prior to combination with the decoded cleaning sequence.

According to a second aspect of the present disclosure, an apparatus for creating synthesized noise in an image, the apparatus comprising: a line buffer comprising rows from the image; firmware including a noise block pattern; a pseudo-random number generator that provides random values for pixels in the row; logic for comparing the random value to a threshold criterion; an NBMT data storage area including a noise block management table; logic to populate the noise block management table based on a random noise block pattern from the firmware associated with pixels in the row, wherein the random values associated with the pixels meet the threshold criteria; logic that calculates a noise value for each pixel location in a row of the image based on the noise block management table; and logic to modify the row based on the noise value.

The apparatus may include logic to determine a modulation curve from the image, calculate a modulation factor, and apply the modulation factor to the noise value prior to modifying the row. The pseudo random noise generator may generate a random number based on the row index and a random offset value. The image may be a luminance channel image or a chrominance channel image. The apparatus may include a plurality of pseudo-random number generators and a plurality of line buffers to process the image in parallel.

According to a third aspect of the present disclosure, a method for generating metadata for a decoder, the method comprising: setting a noise category parameter; setting a block size parameter; setting a threshold parameter; and setting a modulation factor curve parameter.

The method for generating metadata may include setting a noise variance parameter, setting a plurality of block pattern parameters, and/or setting a noise block management table entry parameter.

Drawings

Fig. 1 depicts an example decoder system including a noise synthesis module, according to an embodiment.

Fig. 2A and 2B illustrate an exemplary DCT matrix for generating a film grain block pattern, in accordance with an embodiment.

Fig. 3 depicts an example decoder system utilizing synthesized noise generation, according to an embodiment.

Fig. 4 depicts an example flow diagram of a process of adding noise to a luminance channel of an image, according to an embodiment.

Fig. 5 shows an example of a brightness modulation curve with a fixed alpha according to an embodiment.

Fig. 6 shows an example of a brightness modulation curve with a fixed beta according to an embodiment.

Fig. 7 illustrates an example of film grain synthesis with color conversion, according to an embodiment.

Fig. 8 illustrates an example of camera noise synthesis with color conversion, according to an embodiment.

Fig. 9 illustrates an example block diagram for generating content for injection of synthesized film grain by hardware, in accordance with an embodiment.

Fig. 10 shows an example of how each pixel of an image can be evaluated in terms of film grain in hardware, according to an embodiment.

FIG. 11 illustrates an example 16-bit register for use with a hardware implementation, according to an embodiment.

FIG. 12 illustrates an example of how block ID bits may be picked from a pseudo random number generator, according to an embodiment.

Fig. 13 illustrates an example of NBMT to pixel value logic, according to an embodiment.

Fig. 14 illustrates an example graph comparing a software-based (raw) modulation profile and a hardware-based (piecewise linear) modulation profile, in accordance with an embodiment.

Fig. 15 illustrates an example block diagram of a hardware implementation of digital camera noise synthesis, according to an embodiment.

Fig. 16 illustrates an example simplified camera noise synthesis system/method for a chroma channel, according to an embodiment.

Fig. 17 illustrates an example unified noise injection architecture for a hardware implementation, according to an embodiment.

Definition of

Throughout this disclosure, a probability image is defined as a digital image composed of random values.

Noise refers to random variations in brightness (luminance) and/or color (chrominance) information in a digital image caused by effects other than imaging the environment.

Film grain refers to random optical texture on an image as would result from film processing. Film grain is noise in the luminance channel.

Grain size refers to the average diameter of the individual grains of the film grain.

Camera noise refers to random optical texture on an image as would be caused by digital camera hardware. Camera noise is noise in both the luminance and chrominance channels. Non-synthetic camera noise is caused by a number of reasons including sensor size, thermal effects, long exposure times, high ISO settings, etc.

DCT refers to discrete cosine transform, and IDCT refers to inverse discrete cosine transform.

A digital image refers to a frame of an image or video that is represented digitally.

Detailed Description

Systems and methods for synthesizing noise, such as film grain or camera noise, onto digital images, including video, are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It may be evident, however, that the presently claimed subject matter may be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, or obscuring the present disclosure.

Fig. 1 depicts an example architecture of a decoder implementing noise synthesis. A bitstream (110) that transmits the encoded image (111) and its corresponding metadata (112) is received by the system. The decoder (120) decodes the image into a decoded clean sequence (125). Since the encoded image (111) is produced from an image that is pre-encoded with any noise removed, the decoded cleaning sequence (125) is also (in most cases) noise-free. The metadata (112) provides parameters to a noise synthesis module (130) and optionally to a modulation curve module (140). A noise synthesis module (130) uses the metadata (112) to create a synthesized noise image (135). This composite noise image (135) may be passed through a modulation curve module (140) to produce a modulated noise image (145). The decoded clean sequence (125) and the modulated noise image (145) (or the synthesized noise image (135) if no modulation is performed) are combined (150) in one or more channels to produce a rendered image (155) for display or for color conversion prior to display.

The noise synthesis does not need to be tied to the decoder. For example, a video producer may use noise synthesis for video production pre-distribution. For the case of non-decoder implementations, the decoded clean sequence (125) may be replaced with a content image to be edited. For example, the noise image may be synthesized by a colorist and then modulated later before being added to the content image.

Depending on the desired effect, the noise may be in the luminance channel, the chrominance channel, or both. For example, to mimic film grain, noise may be in the luminance channel only. To mimic the noise induced by digital camera hardware, the noise may be in the luminance and chrominance channels. Various types of noise emulation can be performed by appropriately adjusting the metadata parameters.

The noise of the digital image may be synthesized by: noise block patterns are added to the image by means of the probability image. The idea is to add a block pattern on a pixel-by-pixel basis instead of dividing the original image into blocks.

To synthesize a film grain block pattern of size B × B, a B × B matrix M of standard normal random numbers (e.g., B ═ 4, 8, etc.) is started. The matrix M represents 2D DCT coefficients, where the upper left is the DC (frequency ═ 0) coefficients and the lower right is the highest frequency AC (frequency >0) coefficients. The DC coefficient is labeled 1(210) and the labels increase in a zig-zag order, as shown for the 4 x 4 block example in fig. 2A.

These frequency coefficients can be classified into three bands, namely low, medium and high, in general, as shown in fig. 2B. The mid-band (230) in M is preserved, while the low frequency components (220) and high frequency components (240) are removed (or alternatively, only a plurality or one of the low frequency components (220) is removed). For example, in a 4 x 4 block, the frequency coefficients labeled 4 to 10 may be retained and coefficients 1 to 3 and 11 to 16 may be removed. The range of frequency coefficients that can be preserved is customizable, with at least the coefficient 1 removed.

Let M _mid A matrix M is represented in which only the medium frequency coefficients are retained and the low/high coefficients are set to zero. Film grain block G _raw To pass through the pair M _mid An inverse DCT is performed to construct as shown in equation 1.

G _raw ＝iDCT(M _mid ) Equation 1

The blocks are normalized by dividing each element by the energy E and then scaling the blocks to the desired multiplier V, as shown in equation 2.

Th _norm Is a threshold value normalized to the interval 0, 1). Th _norm An example value of (d) is 0.1.

Is the noise variance. The noise variance is an adjustable parameter. An example range for a 16-bit image may be 1000 to 6000.

Rounding is performed at the end to remove any fractional part as shown in equation 3.

The pseudo code for calculating the film grain matrix G is outlined below. The function zigzag () gives the frequency label at each position in the matrix. The range of frequency coefficients to be preserved is f _s ,f _e ]。

Multiple noise patterns with different block sizes may be constructed for later use in putting together the synthesized noise images. 4 × 4, 8 × 8, 16 × 16, and 32 × 32 are some typical block sizes. The size of the film grain can be changed by modifying the frequency range. If the proportion of the frequency coefficients closer to the LF region is in the frequency range f _s ,f _e ]Is significant, the particle size will be larger. Conversely, if the portion of the frequency coefficient closer to the HF region is significant, it will produce smaller particles. For the 8 x 8 block example, and 4-36]By comparison, the frequency coefficient range [4-10 ]]Larger particles will be produced.

Several noise blocks with different sizes may be pre-computed and stored in the block pattern dataset

In (1). This database can be viewed as a 3D matrix with Q number of 2D noise blocks stacked one on top of the other. If LxL is the maximum size of a film grain block, then

The size of (b) is L × L × Q. A vector S of Q elements may be maintained to store a corresponding size of each block in the pattern dataset, i.e., S (i) ≦ L, where i ∈ [0, Q-1]。

Fig. 3 shows an example of constructing a composite noise image. In this example, the noise mimics film grain. To construct a composite noise image (302) of size H x W from a block pattern (301), we start with probability images (300) of the same size. Typical noise images of film grain are FHD (full high definition, e.g. 1080p), UHD (ultra high definition, e.g. 4K) and 8K resolution. The probability image is a matrix with uniform random numbers, for example, in the interval [0, 1). Selection threshold Th _norm (e.g., 0.1) and each entry in the probability image (300) is compared to a threshold. In fig. 3, a circle (310) represents a location having a probability less than a threshold. Alternatively, the location may be selected at a pixel having a probability greater than, less than, or equal to, or greater than or equal to a threshold. Basically, there is a criterion that compares the probability value at a given pixel to a threshold. For simplicity, only a few locations are shown, but the actual probability image may have more locations. Noise blocks are placed at the circles. These positions are represented by the set Ψ.

The noise block may be placed such that the position in the probability image is in the upper left corner of the noise block (320), as shown in fig. 3. Alternatively, the location may be set at the center, other corners, or any designated location associated with the noise block. The point at which the noise block corresponds to a location is the "anchor" of the noise block and may be the same for all noise blocks.

Block patternA selector (305) will randomly pick a film grain block pattern to be inserted at a location in the set Ψ. This block selector may be a uniform random number generator. After the blocks are put in place, there is a probability that the blocks may overlap, as shown in fig. 3. The cells from the collocated cells in the overlapping block are added together. This aggregate value becomes the output noise image (302). Since the overlapping regions are aggregated rather than averaged, the film grain block can be scaled by a multiplier V rather than by the noise variance

(see equation 2). This prevents the aggregate noise pixel value from becoming too large.

To synthesize film grain image I, consider a block pattern dataset

At each position in the set Ψ, a block to be inserted into the film grain image can be randomly chosen from the data set. Let r be the index of the randomly selected block for the current location, where r is Uniform [0, Q-1%]. The size of this block G' is S (r) x S (r) such that

A different random number r may be selected each time there is a new block to be inserted at the current position. This pseudo code can be shown as:

g' represents a random block selected for position i, j based on the random number r. In other words, a random number r is selected for each position i, j, and then G' is generated as a function of r.

For the case of synthetic film grain or similar noise, in bright areas of the image, the noise can be quite annoying and intrusive. The intensity-based brightness modulation profile reduces the intensity of noise in these bright areas, making the film grain-injected image more desirable to viewers. The film grain image is generated in advance as described in the previous section. These images are then modulated based on the collocated luminance intensities and added to the luminance channel.

Fig. 4 illustrates a general method for adding noise to a luminance channel (e.g., Y channel) of an image for an embodiment and not by way of limitation. First, a noise image I is initialized (405). The noise image may have the same size as the Y-channel image. Initialization may set all pixel values of the noise image to 0 or some other set value. Next, a probability image P is generated (410), where random values are generated at each location in P. The probability image may have the same size as the noise image. The random value may range from 0 to 1, or between any two values. The position map Ψ is then generated by comparing each probability value for each pixel position in P to a threshold criterion (e.g., less than or equal to a set threshold) (415). For each location in Ψ that is identified as meeting the threshold criterion, a randomly selected noise block is added to the corresponding pixel location in I. If the block is larger than 1 x1, the block may be assigned an anchor point (e.g., the top left pixel of the block), where the block will anchor to the corresponding pixel location in I. When all identified locations in Ψ have been processed, the aggregation of the added noise blocks forms a composite noise image in I (425). The resulting noise image (435) may be modulated by a modulation factor, which may be calculated (430) from the luminance channel. The modulated composite noise image may then be added to the luminance channel of the image (440).

Sample brightness modulation curves are shown in fig. 5 and 6. These graphs give modulation factors for different values of the brightness intensity. The system may multiply the film grain value by its modulation factor before adding the film grain pixel to a collocated pixel in the luminance channel.

Suppose that the intensity value of the luminance channel is represented by B _t The bit code range is denoted by l. By way of example, without limitation, the modulation factor curve f (-) may be calculated as shown in equation 4.

Wherein

Equation 4

In equation 4, α and β represent curve parameters.

Fig. 5 shows the variation of α and β ═ 2 (B) _t 16) of the luminance modulation curve. Curves are shown where α is 0.15(501), 0.25(502) and 0.4 (503).

Fig. 6 shows α ═ 0.25 and varying β (B) _t 16) of the luminance modulation curve. Curves are shown where β ═ 1.0(601), 2.0(602), and 4.0 (603).

The freedom to choose either a or β makes it possible to use the same equation for different electro-optical transfer functions (EOTFs).

If Y (i, j) is Y _ij Is the intensity of the luminance at position (i, j) in the Y channel image, the noise value at position (i, j) in the noise image can be multiplied by f (Y) _ij ) And added to the luminance channel to derive the luminance value of the injected noise

Fig. 7 shows an example of film grain synthesis with color conversion. A probability image (705) is created and locations (710) are selected by comparing the value at each pixel of the probability image (705) to a threshold, in this example, those pixels having a value below the threshold. A set of film grain patterns is pre-generated (715) and randomly selected (720), which are to be injected into the probabilistic image (725), thereby creating a film grain image (730). The Y (brightness) channel (735) of the image is modulated (740) to create an image with a modulation factor (745) of the noise. The film grain image (730) is combined (750) with the modulation factor image (745) to produce a brightness modulated film grain image (755). The luminance modulated film grain image (755) is combined (760) with the Y-channel image (735) to produce a film grain added Y-channel image (765). The YUV image including the Y channel with film grain added can be color converted (770) to produce an RGB image with film grain noise added for all channels (775).

The digital camera noise may be synthesized using gaussian random numbers. For example, random numbers can be generated from uniform random numbers using the Box Muller Transform (Box Muller Transform) or the mazary Polar Method (Marsaglia Polar Method).

For the Maxalian polar coordinates method for generating Gaussian random numbers from uniform random numbers, u is assumed ₁ And u ₂ Is selected from the range [0, 1]]Two uniform random numbers. If the condition 0 < u is satisfied ₁ ² +u ₂ ² Is less than 1, then

Is a gaussian random number with zero mean and unit standard deviation. Care should be taken to avoid cycling the process too long before the conditions are met.

Since digital camera noise can be simulated with gaussian random numbers, the composite camera noise image is summarized as generating a 2D matrix having the size of the image and containing gaussian random numbers with zero mean. With respect to the brightness intensity modulated noise image, as previously discussed. The noise image may then be added to the Y, U and V channels of the content, followed by optional color space conversion (e.g., to RGB).

Fig. 8 shows an example of camera noise synthesis with color conversion. This is a further embodiment of noise synthesis and therefore features common to both embodiments may be omitted from the description but should be understood to remain common to both embodiments. The Y (brightness) channel (810) of the image is modulated (805) to produce a modulation factor image ("MFI", 815). Three random gaussian images ("RGI", 820) are generated, one for each channel, and three luminance-modulated noise images (830) are generated using the modulation factor image (815). These noisy images (830) are added (835) to their respective channels (840) in the original image. These channels (840) may then be converted (845) to a different color coding scheme (850), such as RGB. The luminance modulated noise image (830) represents a pixel-by-pixel multiplication (825) of the MFI image (815) with the RGI image (820) of the corresponding channel.

When noise synthesis is implemented in a hardware (as opposed to software) solution, four factors should be considered for optimization.

1. In a software implementation, the noise image is synthesized separately and then added to the luminance channel. Due to memory limitations in hardware, only one pixel of the film grain image is computed and added directly to the luminance channel of the content. This approach removes any intermediate memory buffer for storing the entire film grain image and saves a significant amount of on-chip memory.

2. In a hardware implementation, the image is accessed through a line buffer. The line buffers may store lines of the image. Since hardware memory is expensive, the number of line buffers is typically limited, so that the entire image cannot be loaded into these few line buffers. Reading and writing to the memory is also time consuming. For this reason, a hardware solution should read a line of the image into the line buffer once, do the entire process, and then write the result out to memory only once. Reading and writing the same portion of the image into the line buffer multiple times is sub-optimal.

3. Floating point representations are not hardware friendly, so values are more efficient to process when in integer format for hardware implementations. Noise block patterns, image pixels, thresholds, etc. may be represented in integer format to allow for simpler hardware implementations. For example, instead of setting the probabilities from 0 to 1 (real numbers), they may be set from 0 to 255 (integer).

4. The block size in a hardware implementation is fixed (e.g., all blocks are 8 x 8) to achieve consistency across all decoder hardware chips. In the rest of the description, the fixed block size will be denoted B × B.

To optimize the system for hardware implementation, the noise synthesis algorithm may be adjusted to take these factors into account.

Fig. 9 shows an example block diagram of the generation of noise-injected (in this case film grain) content by hardware. Film grain (945) is added only to the Y (luminance) channel (955), but other noise injection techniques may include both the V and U channels. A processor clock (timestamp/date/time of day/etc.) may be utilized as the frame-level random seed (905). All row level pseudo random number generators are initialized 910 using this random seed. Each pixel (920) is evaluated in terms of film grain and then added (940) to the luminance channel (925) of the content after modulation (930, 935). The film grain-added content (956) is finally converted (960) to RGB (965) for display (970).

Fig. 10 shows an example of how each pixel can be evaluated in terms of film grain. For the film grain block pattern dataset, a uniform block size of B × B is assumed. Different block sizes will help to vary the size of the film grain, but the same effect can be achieved by varying the frequency range of equally sized blocks. Thus, to reduce the number of variables, the block size of the film grain pattern can be set to a fixed size when implemented in hardware. The film grain block pattern can be generated using the same approach described for the software implementation. Fig. 10 shows the steps of generating film grain values at C column (1050) in R rows.

There is no explicit probability image in the hardware solution. Instead, there is a pseudo-random number (PRN) sequence generator (1010 to 1015) for each row of the image. For each entry (pixel) in the R row, the PRN module generates a new random number. A PRN may be selected 1005 from a function of the row index and the random offset. If the random number is less than the threshold (1020) and there is sufficient space in the Noise Block Management Table (NBMT) (1090), then a film grain block (1070) to be inserted starting at this location (1065) is randomly selected from the set of film grain patterns (1030). Blocks not directly added to the imageInstead, the system places the block's entry (1025) in NBMT (1090). NBMT to pixel value logic block (1035) calculates a noise value at the current location in the luminance channel image (1045). For a hardware implementation, the threshold Th is not normalized, thus reducing the computational cost. The non-normalized threshold is expressed as having no subscript "norm" in the symbol, and it is passed through the equation

To correlate with the normalized threshold.

The reason for using a pseudo random number generator from a previous row when calculating the noise value at the current position is an efficiency issue. If B (noise block size) is significantly large, reading/writing B rows on each pass would significantly add memory input/output (I/O) and hardware cost by requiring B line buffers in memory.

In contrast, current solutions only require one line buffer, regardless of block size. For a given location, there may be overlapping blocks from previous rows. These blocks are represented by dashed squares (1070) in the image (1060) in fig. 10. Since the random seed of each PRN generator is deterministic, the sequence of random numbers traversed by the previous rows can be regenerated. These sequences provide the exact locations of the blocks and block patterns used in the previous rows. The information allows the correct film grain value to be calculated anywhere using only one line buffer, thus reducing I/O latency and hardware cost. A noise block management table (990) is maintained for each row to record overlapping blocks.

Although the raw and injected film grain luminance channel rows R are shown separately in fig. 10, the raw rows R can be updated in-situ to yield film grain injected rows R.

The film grain block pattern is generated using a process similar to that described for the software-based solution. This task can be turned over to firmware, since the mechanisms used to compute these block patterns require the use of complex mathematical operations. The synthesized block can be stored later in on-chip SRAM for easy access to the hardware.

Another major challenge of hardware implementation is the random number generator. For faster implementations, a pseudo-random number generator based on a linear feedback shift register may be used. Some bits of those random numbers are used to select a block of film grain from the data set of block patterns.

Linear feedback shift registers are commonly used to generate random number sequences in hardware chips. Consider the example 16-bit register shown in fig. 11. The next random number is obtained by shifting the contents of the register to the right by 1 and inserting a new bit at the Most Significant Bit (MSB) position (1110). The new bit is the exclusive or (XOR) of the bits at some predefined location (1120) in the register. For the example shown in FIG. 11, the bits at index positions 3, 12, 14, and 15 are XOR'd. The output of xor gate (1115) is fed into MSB position (1110).

If the tap point location is properly selected, the register passes through all the different 16-bit numbers once before returning to the same number. In other words, the sequence repeats itself after a certain period. It can be seen that the next random number is a function of the current state of the register. Therefore, care must be taken to ensure that the registers are not initialized to all zeros. For an all-zero case, the process will be locked in that state.

The hardware implementation does not explicitly produce a probability image of H x W size. As a temporary solution, a linear feedback shift register may be used to generate the random number in each row. The shift registers assigned to each row are initialized with a different random seed. This Seed is a function of the row index R and the random offset, Seed _R ＝(Const+R×Offset)mod 2 ^D Where D is the length of the register. The random offset is derived from the current timestamp and the constant is set to any large number, e.g. 2 for a 16-bit shift register ¹⁴ . The offset is chosen randomly, but it is fixed for each row in the current frame. Although there is a different pseudo-random number generator per row, there are no more than B shift registers in hardware at any given point in time. Here, B × B is the size of a block in the block pattern data set.

The seed is deterministic as a constant, with offsets and row indices known. The idea behind making this seed deterministic is to regenerate the sequence of random numbers that the previous few rows have passed when processing the current row R. This allows the position of overlapping film grain blocks from the first few lines to be determined and the knowledge used to calculate the film grain value for the current position.

Along with the random sequence generator, a table may be used to store information about the location of the blocks in the previous row. This table is called the Noise Block Management Table (NBMT).

The NBMT stores information of the film grain blocks injected into the luminance channel of a particular row. Each PRN sequence generator (see fig. 10) has a corresponding NBMT. The number of entries in the table is a configurable parameter and each entry holds the following three columns, as shown in table 1.

Table 1: noise block management table

Block ID	Position-x	Position-y
			b i	x i	y i

To calculate the final value at each location, the system needs to consider all blocks that overlap the location. Noise block management table by combining block idb _i The origin X coordinate of the block X _i And the origin Y coordinate Y _i Stored in the image to record these overlapping blocks. For speed and computational efficiency, the maximum number of entries in the table is limited to ceil (B/4).

For example, assuming that the size of the film grain block pattern is B × B and the system is calculating the film grain injection value at location (C, R) in the luminance channel image where noise is injected, blocks originating in B-1 rows above or B-1 columns to the left can still affect the current location. For this reason, when processing the current row R, the system considers B-1 rows above the current row.

As illustrated in FIG. 10, the system initiates a pseudo-random number generator for B-1 rows above the current row R to find overlapping blocks. The current row may also have its own random number generator represented by PRN row R. Since the random seed is deterministic, the system can always reproduce the sequence that the first few rows passed through. In addition to the row-specific random number generator, each of these B rows may also have its own NBMT.

Let C denote the current column in the row R being processed. The row R of the luminance channel image is read into the line buffer. Column C is incremented from C-0 to C-W-1 in steps of 1. For each value of C, the system may perform the following steps:

each of the B random number generators generates a random number. If the number is greater than or equal to the threshold Th, no action is taken.

2. Further, if the random number is less than the threshold value, then

a. Any entries in NBMT that do not affect the current location are removed. For example, remove block size B, where x _i <C-B +1 or y _i <R-B+1。

b. If the NBMT of the row is not full, a random block is selected from the film grain pattern dataset. Bits from the pseudo random number generator of the row are used to select the block ID b _i . FIG. 12 presents an example of how block ID bits are chosen from a pseudo-random number generator. For a data set of 16 different block patterns, the block ID PRN [15:12 ]]And 7 refers to block ID 7, which is the 7 th pattern in the data set. The sorting starts with index 0.

c. An entry for the block is added to NBMT, where x _i C and the origin Y coordinate is equal to the row index. The block ID source is derived from the previous step.

3. The film grain values at positions (C, R) of NBMT from all R rows are calculated. This step is referred to as NBMT to pixel value logic illustrated in fig. 13 by way of example.

a. For each entry in each NBMT, it is checked whether the block overlaps with the current location (1310). When x is _i ≤C≤B+x _i -1 and y _i ≤R≤B+y _i At-1, the blocks overlap.

b. The block ID extracted from the PRN generator is used to select a corresponding block from the block pattern dataset (1320). The values in the block that overlap the current position are picked.

c. There may be multiple such overlapping blocks, so the values are added together (1330).

d. The value is handed over to the next stage of the luminance modulation factor (1340) where the noise value is calculated. The luminance modulated noise values are added 1350 to the original luminance 1355 and written to the line buffers 1360 at column C.

4. Each row goes through this process, one at a time, to obtain a brightness channel image of the injected film grain. For rows that do not have B-1 rows above it, only the available rows are considered.

For a hardware implemented luminance intensity based noise modulation technique, the software implemented non-linear curve should be replaced with piecewise linear segments to avoid wasting memory cycles on non-linear arithmetic. The parameters of these line segments may be signaled to the decoder in the metadata. The system can then calculate a modulation factor for each luminance intensity value from the piecewise linear curve.

Furthermore, the values in the curves for the software implementation are fractional numbers. In hardware, the operation decimal is troublesome. Thus, curve

The value of (A) can be approximated in a piecewise linear manner, i.e. in a linear manner

(wherein

) Is previously scaled to B _c An 8-bit integer. Curve f is then plotted using T +1 pivot points _hw (. cndot.) is divided into T segments. Let the position of the pivot point be represented by the vector omega, so that

Wherein k is more than or equal to 0 and less than or equal to T.

For each two pivots on the curve, i.e., x1 ═ Ω (k) and x2 ═ Ω (k +1), the system calculates the connection point y1 ═ f _hw (x1) and y2 ═ f _hw Equation of the kth line segment of (x 2). The equation for the line has three parameters, slope, X-intercept (X1) and Y-intercept (Y1).

y-y1＝slope _k X (x-x1) equation 6

The slope, pivot and Y-intercept parameters for each segment are sent to the decoder. The curve is reconstructed at the decoder to calculate the modulation factor for each luminance intensity. For illustration, fig. 14 plots raw (software-based) and piecewise linear curves, where α is 0.25 and β is 2. If Y (C, R) ═ Y _CR Is the intensity of the luminance at position (C, R) in the Y-channel image, the noise value, i.e., n _CR Multiplied by f (y) _CR ) And added to the luminance channel to obtain the luminance value of the injected noise

Due to the modulation factor being B _c Within the bit code word, so that the product of the film grain value and the modulation factor can be shifted to the right by B _c One bit to get the correct luminance modulated film grain value.

Similar to film grain noise, the hardware implementation of digital camera noise also has several limitations.

1. Synthesizing gaussian random numbers in hardware is expensive due to the log function involved. As a temporary solution, these numbers may be generated in firmware and stored as digital camera noise blocks.

2. Storing digital camera noise images in on-chip memory is expensive due to memory limitations. Noise pixels may be calculated and added to Y, U and the V channel directly after modulation.

3. Only a few lines of Y, U and the V channel are read into memory as line buffers for processing. Since the image size can be huge, processing the entire image will be I/O and memory intensive. The line buffers will help to alleviate latency and memory requirements. The architecture can be significantly altered to achieve similar results as the software solution.

4. Floating point arithmetic is more complex for hardware implementations, so all values are converted to fixed point format. Some changes to the architecture are necessary when moving from floating point to fixed point arithmetic.

5. The block size of the digital camera noise pattern can be fixed to 1 × 1(B ═ 1), which basically means that the system adds only one gaussian random number at each position. This is consistent with a software solution.

One embodiment of digital camera noise synthesis in hardware limits the block size of the digital camera noise pattern to 1 × 1 and the probability threshold for row R is set to 2 ^D While the other rows have a threshold value of zero. Setting the probability threshold to 2 ^D Ensure that a block is added to each pixel location in the Y channel. Since the block size is limited to 1 × 1, there is no overlap between blocks. Thus, there is only one entry in the NMBT for row R.

A pseudo-random number generator is present for each row. When processing rows R and columns C, if the pseudo random number is less than 2 ^D The system adds a digital camera block to the pixel. Since these random numbers are always less than 2 ^D The system adds gaussian random numbers at each location. The blocks are randomly selected using the leading bits of the pseudo random number generator. The information of the block may be inserted in the NBMT of the row. NBMT to pixel value logic calculates digital camera noise values. This value is modulated with a modulation factor and added to column C of the line buffer. For a block of size 1 × 1, B ═ 1, and NBMT to pixel value logic is simple because ofAdjacent blocks do not overlap. Fig. 15 shows an example block diagram of a hardware implementation of digital camera noise synthesis, which may be described as a special case of fig. 10.

Digital camera noise is also added to the chrominance channel, as opposed to film grain, which is added only to the luminance channel. The architecture for injecting camera noise into the luminance and chrominance channels is essentially the same. The design of the chroma channel may be simplified by removing redundant components like additional pseudo-random number generators, threshold blocks, NBMT and NBMT to pixel value logic, etc., from the hardware architecture of fig. 15, as shown in the example of fig. 16.

For the example, assume that there are 1024 block patterns of digital camera noise. The first 10 bits of the PRN row R may be used to calculate the block index b _i . The 1 x1 block is chosen from the data set, modulated with luminance intensity and added to the chroma channel line buffer at the current position.

For the YUV 444 sequence, the modulation factor can be calculated from the collocated intensity in the luminance channel. The noise value is added to the chrominance channel after modulation. In other words, the noise value is first modulated with the luminance intensity and then added to the chrominance channel.

For a YUV 422 sequence, the Y-channel may be sub-sampled to half the width and then the modulation factor is calculated from the collocated pixels in the Y-channel. The noise value is modulated with the down-sampled luminance intensity and then added to the chrominance channel. Down-sampling using conventional filtering methods can be expensive to hardware. The modulation factor may work with coarse and approximate down-sampling. To reduce hardware complexity, the luminance channel is sub-sampled by a simple averaging of adjacent elements in the luminance channel row. The process is set forth in more detail in the pseudo-code below. Symbol Y _hh Representing a horizontally down-sampled version of the luminance channel Y obtained by taking the average of adjacent elements in the same row.

Fig. 17 shows an example unified noise injection architecture (in this example, film grain and camera) for a hardware implementation that allows for the synthesis of multiple types of noise. The pseudo-random number generator (1710) may generate a sequence of random numbers. If these random numbers meet a threshold condition, an aggregate noise value is calculated (1720) using the film grain (1721) or digital camera noise (1722) block data set, depending on what type of noise is determined to be needed (1715). The values are modulated (1730) with luminance intensity and added to the original luminance or chrominance channel. One line (1740) is processed at a time, so the line buffer only needs to be large enough to accommodate a line of YUV content. The noise-injected YUV content is converted (1750) to RGB for display.

There are two different scenarios where a noise injection model will be used to inject noise into the content. In a first scenario, a content provider may control the appearance of noise to be added to a video. The parameters will be set at the encoder end and transmitted to the decoder as part of the metadata. For the second scenario, the decoder may synthesize noise based on viewing conditions or display specifications. Settings of the content provider may have a higher priority than local adjustments and vice versa.

The noise synthesis parameters can be set at different stages of the workflow, i.e. either the settings that are preferred by the content provider at the encoder side or the settings that are locally adaptive at the decoder side. The decoder parameters may be configured into firmware or hardware.

For artistic reasons, content providers may want to add film grain or camera noise to a video sequence to achieve the effect of a movie. The noise is compressed at the time of encoding, and therefore, the movie effect is attenuated at the decoder side. Furthermore, a higher bit rate is required to compress noisy sequences, so transmitting a noise-free sequence along with parameters to generate noise at the decoder side reduces the bit rate requirement, but still preserves the film effect. For this there is an option to set parameters at the encoder side and send them to the decoder via metadata. The parameters are listed in table 2.

TABLE 2

Parameters may be read from the metadata to initialize the noise generation module at the decoder side. Likewise, parameters may be adjusted at the decoder side based on the viewing environment or display specifications. Either way, the decoder must synthesize the noise given a set of parameters. The decoder is comprised of firmware or a processor and a hardware system. Firmware can compute complex mathematical functions at the expense of speed, but hardware can perform low-level logical operations at a faster rate.

Based on speed requirements and computational complexity, the task of noise image generation must be shared among firmware and hardware. Synthetic film grain or camera noise block is computationally intensive for the hardware so it can be handed over to the firmware. There are only a few block patterns to build, so these firmware can be easily built during system startup without much processing overhead.

Example embodiments

Ee1. a computer-based method for creating a composite noisy image for a digital image, the method comprising: providing a plurality of noise block patterns; initializing a noise image; generating a probability image comprising randomly generated probability values at each pixel of the probability image, comparing each probability value in the probability image to a threshold criterion; selecting a noise block pattern from the plurality of noise block patterns if the probability value satisfies the threshold criterion, the noise block pattern having an anchor point; and if the probability value satisfies the threshold criterion, creating a composite noise image by placing the noise block pattern in the noise image such that the anchor point corresponds to pixels in the probability image that meet the threshold criterion.

Ee2. the computer-based method according to EE1, wherein the noise particle block pattern is randomly selected.

Ee3. the computer-based method according to EE1 or EE2, wherein the probability values of the probability image are evenly distributed probability values.

Ee4. the computer-based method of any one of EE 1-EE 3, wherein the noise block pattern comprises a film grain block pattern.

EE5. the computer-based method of any one of EE1 to EE4, further comprising aggregating values in regions of the noise image where the noise block patterns overlap.

EE6. the computer-based method of any one of EE1 to EE5, further comprising generating the noise block pattern prior to generating the noise image, wherein the noise block pattern is generated by an inverse DCT technique.

Ee7. the computer-based method according to EE6, wherein the generating further comprises removing low frequency DCT coefficients.

Ee8. the computer-based method according to EE6 or EE7, wherein the generating further comprises removing high frequency DCT coefficients.

Ee9. the computer-based method according to any one of EE1 to EE8, wherein the size of the probability image and the noise image are set to match the video frame.

EE10. the computer-based method of any one of EE1 to EE9, further comprising: editing a content image by adding the noise image to a channel of the content image.

Ee11. the computer-based method of EE10, further comprising: multiplying values in the noise image by a modulation factor before adding the noise image to the content image.

Ee12. the computer-based method according to EE11, wherein the value is a luminance intensity value of the noise image.

The computer-based method of any one of claims EE 10-EE 12, further comprising: generating a YUV content image based on the edited content image.

Ee14. the computer-based method of EE13, further comprising: generating an RGB content image based on the YUV content image.

Ee15. an apparatus comprising: a decoder that decodes the encoded image, creating a decoded clean sequence; a noise synthesis module that performs the method of any one of EE 1-EE 14, thereby creating a synthesized noise image; wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to create a rendered image for display.

Ee16. the apparatus of EE15, further comprising a modulation profile module for modulating the synthesized noise image prior to combination with the decoded cleaning sequence.

Ee17. a computer program product comprising data encoded to perform a method according to any one of EE 1-EE 14 when run on a processor.

Ee18. an apparatus for creating synthesized noise in an image, the apparatus comprising: a line buffer comprising rows from the image; firmware including a noise block pattern; a pseudo-random number generator that provides random values for pixels in the row; logic for comparing the random value to a threshold criterion; a data storage area including a noise block management table; logic that populates the noise block management table based on a random noise block pattern from the firmware being associated with pixels in the row, wherein the random values associated with the pixels meet the threshold criteria; logic that calculates a noise value for each pixel location in a row of the image based on the noise block management table; and logic to modify the row based on the noise value.

The apparatus of EE18, further comprising: logic for determining a modulation curve from the image, calculating a modulation factor, and applying the modulation factor to the noise value prior to modifying the rows.

Ee20. the apparatus as described in EE18 or EE19, wherein the pseudo random noise generator generates random numbers based on row indices and random offset values.

The apparatus as in any one of EE 18-EE 20, wherein the image is a luminance channel image.

The apparatus as in any one of EE 18-EE 20, wherein the image is a chroma channel image.

Ee23. the apparatus of any one of EE18 to EE22, further comprising a plurality of pseudo random number generators and a plurality of line buffers to process the image in parallel.

Ee24. a decoder system, comprising: a decoder that decodes the encoded image, creating a decoded clean sequence; a noise synthesis module comprising a device as described in any one of EE 18-EE 23, thereby creating a synthesized noise image; wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence, thereby creating a rendered image for display.

Ee25. a method for generating metadata for a decoder, the method comprising: setting a noise category parameter; setting a block size parameter; setting a threshold parameter; and setting a modulation factor curve parameter.

Ee26. the method as described in EE25, further comprising setting a noise variance parameter.

The method as in any one of EE 25-EE 26, further comprising setting a discrete cosine transform frequency range parameter.

The method as in any one of EE 25-EE 27, further comprising setting a plurality of block pattern parameters.

Ee29. the method of any one of EE 25-EE 28, further comprising setting a noise block management table entry parameter.

Various embodiments of the present disclosure have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims. Any significant features of this embodiment may be common to all embodiments where appropriate.

The examples set forth above are provided to those of ordinary skill in the art as a complete disclosure and description of how to make and use embodiments of the present disclosure, and are not intended to limit the scope of what the inventor/inventors regard as their disclosure.

Modifications of the above-described patterns for performing the methods and systems disclosed herein that are obvious to those of skill in the art are intended to fall within the scope of the appended claims. All patents and publications mentioned in the specification are indicative of the levels of skill of those skilled in the art to which the disclosure pertains. All references cited in this disclosure are incorporated by reference to the same extent as if each reference had been individually incorporated by reference in its entirety.

It is to be understood that this disclosure is not limited to a particular method or system, which can, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an" and "the" include plural referents unless the content clearly dictates otherwise. The term "plurality" includes two or more of the referents unless the content clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

Claims (29)

1. A computer-based method for creating a composite noisy image for a digital image, the method comprising:

providing a plurality of noise block patterns;

initializing a noise image;

generating a probability image comprising randomly generated probability values at each pixel of the probability image,

comparing each probability value in the probability image to a threshold criterion;

selecting a noise block pattern from the plurality of noise block patterns if the probability value satisfies the threshold criterion, the noise block pattern having an anchor point; and

if the probability value satisfies the threshold criterion, a composite noise image is created by placing the noise block pattern in the noise image such that the anchor point corresponds to pixels in the probability image that meet the threshold criterion.

2. The method of claim 1, wherein the noise particle block pattern is randomly selected.

3. The method of claim 1 or 2, wherein the probability values of the probability image are evenly distributed probability values.

4. The method of any of claims 1 to 3, wherein the noise block pattern comprises a film grain block pattern.

5. The method of any of claims 1 to 4, further comprising aggregating values in regions of the noise image where the noise block patterns overlap.

6. The method of any of claims 1 to 5, further comprising generating the noise block pattern prior to generating the noise image, wherein the noise block pattern is generated by an inverse DCT technique.

7. The method of claim 6, wherein the generating further comprises removing low frequency DCT coefficients.

8. The method of claim 6 or 7, wherein the generating further comprises removing high frequency DCT coefficients.

9. The method of any of claims 1 to 8, wherein the probability image and the noise image are sized to match a video frame.

10. The method of any of claims 1 to 9, further comprising: editing the content image by adding the noise image to a channel of the content image.

11. The method of claim 10, further comprising: multiplying values in the noise image by a modulation factor before adding the noise image to the content image.

12. The method of claim 11, wherein the value is a luminance intensity value of the noise image.

13. The method of any of claims 10 to 12, further comprising: generating a YUV content image based on the edited content image.

14. The method of claim 13, further comprising: generating an RGB content image based on the YUV content image.

15. An apparatus, comprising:

a decoder for decoding an encoded image to create a decoded clean sequence;

a noise synthesis module for performing the method of any one of claims 1 to 14 to create a synthesized noise image;

wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to create a rendered image for display.

16. The apparatus of claim 15, further comprising a modulation profile module to modulate the synthesized noise image prior to combining with the decoded cleaning sequence.

17. A computer program product comprising data encoded to perform the method of any of claims 1 to 14 when run on a processor.

18. An apparatus for creating synthesized noise in an image, the apparatus comprising:

a line buffer comprising rows from the image;

firmware including a noise block pattern;

a pseudo-random number generator for providing random values for pixels in the row;

logic for comparing the random value to a threshold criterion;

a data store comprising a noise block management table;

logic to populate the noise-block management table based on a random noise-block pattern from the firmware associated with pixels in the row, wherein the random values associated with the pixels meet the threshold criteria;

logic to calculate a noise value for each pixel location in the row of the image based on the noise block management table; and

logic to modify the row based on the noise value.

19. The apparatus of claim 18, further comprising: logic for determining a modulation curve from the image, calculating a modulation factor, and applying the modulation factor to the noise value prior to modifying the rows.

20. The apparatus of claim 18 or 19, wherein the pseudo random noise generator generates a random number based on a row index and a random offset value.

21. The apparatus of any one of claims 18 to 20, wherein the image is a luminance channel image.

22. The apparatus of any of claims 18 to 20, wherein the image is a chroma channel image.

23. The apparatus of any of claims 18 to 22, further comprising a plurality of pseudo-random number generators and a plurality of line buffers for processing the image in parallel.

24. A decoder system, comprising:

a decoder for decoding an encoded image to create a decoded clean sequence;

a noise synthesis module comprising the apparatus of any of claims 18 to 23 to create a synthesized noise image;

wherein the decoder is configured to combine the synthesized noise image with the decoded clean sequence to create a rendered image for display.

25. A method for generating metadata for a decoder, the method comprising:

setting a noise category parameter;

setting a block size parameter;

setting a threshold parameter; and

setting the parameters of the modulation factor curve.

26. The method of claim 25, further comprising setting a noise variance parameter.

27. The method of any one of claims 25 or 26, further comprising setting a discrete cosine transform frequency range parameter.

28. The method of any of claims 25 to 27, further comprising setting a plurality of block pattern parameters.

29. The method of any one of claims 25 to 28, further comprising setting a noise block management table entry parameter.

Images