GB2507577A

GB2507577A - Detecting noise using edge detection and filtering

Info

Publication number: GB2507577A
Application number: GB1219923.8A
Authority: GB
Inventors: Jonathan Mckinnell; Mark Glanville; Phil Tudor
Original assignee: British Broadcasting Corp
Current assignee: British Broadcasting Corp
Priority date: 2012-11-05
Filing date: 2012-11-05
Publication date: 2014-05-07
Also published as: GB201219923D0

Abstract

A method of analyzing an image to determine the presence or absence of noise uses a first mask image, using edge detection, and a second mask image by applying a filter, and combines the first and second masks to produce a noise image. The noise image is then processed to emphasize small regions of similar luminance and to de-emphasize larger regions of similar luminance. An average of the processed noise image is determined and compared with a threshold to determine whether or not to assert that an image is noisy. The filter could be a 2D second derivative filter or a Laplace filter. The process of emphasizing regions might comprise assigning a greater probability to speckles of one pixel, and lower probabilities to speckles of more than one pixel. The method is aimed at automatically distinguishing between good and bad content.

Description

Method and Apparatus for Detection of Noise

BACKGROUND OF THE INVENTION

The present invention relates to systems and methods for monitoring the quality of audio video content from each of multiple sources. In particular, the invention relates to checking for noise.

Programme makers are being required to produce greater volumes of high quality content for a larger variety of delivery platforms and audiences with ever decreasing budgets. Alerting production staff of potential problems with audio or video content could help save time and money both on the production and in the editing suite by expediting the process whilst potentially avoiding the failure of a technical review. There are a variety of possible technical errors or potential problems which can occur in a production, such as noise. Noise can manifest itself as a speckle-like, random noise across the entire frame (which could be caused by incorrect camera gain settings or low light levels) or as noise which occurs only for one or two frames, potentially only for a few pixels.

SUMMARY OF THE INVENTION

We have appreciated the need for improved systems and methods for alerting users to the fact that a source of audio video content may not be providing that content at a required quality level.

In broad terms, the invention resides in alerting potential problems with footage to production staff could help save time and money both on the production and in the editing suite by expediting the process whilst potentially avoiding the failure of a technical review.

An embodiment of the invention uses a combination of a combined mask image that is further refined using object detection, for example using a contiguous regions of pixel method.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described in more detail by way of example with reference to the accompanying drawings, in which: Figure 1: is a block diagram showing the main functional components of a system embodying the invention; Figure 2: is an image showing noise; -Figure 3: is a possible noise detection algorithm; Figure 4: shows an Output of a Sobel filter; Figure 5: shows an Inverse of Sobel image with a threshold applied: Figure 6: shows an Image Convolved from Laplace Filter: Figure 7: shows a Masked Sobel Image from the Convolved Laplace Image; and Figure 8: shows a detected noise map following object size probability scaling.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

Automatic shot classification is a general term for the tagging of images with metadata that can improve the overall workflow within a production. This metadata could provide cost or time saving benefits, better archiving retrieval or be useful for other non-production purposes.

An example of time and cost saving would be if images with noise could be logged. With the continuing uptake of digital media and digital workflows, traditional methods of checking the technical quality of content are becoming increasingly limited, in part due to the sheer volume of content now being produced. Both academic and broadcasting standardisation work has been carried out into a perception based quality measurement. In many cases it is becoming increasingly impossible for human eyes to view all content for errors even at the delivery stage of the programme making chain The UK public service broadcasters, through the Digital Production Partnership initiative, have agreed a set of technical standards that programmes must achieve.

The technical quality of a programme is assessed before it is broadcast. Failure to meet the standards required in this technical review can result in significant extra cost5. In some cases it may not be possible to correct mistakes, as content cannot always be re-shot. The ability to detect poor quality footage during production could enable a system to warn production teams that something may be wrong, potentially prompting immediate action to be taken to correct the error.

Equally useful would be a system that can review footage before the edit and flag unusable footage so that the editor does not waste time and money viewing it.

A system embodying the invention provides automated quality checking, particularly for noise. A variety of possible checks can be performed in order to ascertain the quality of footage. Each of these checks can be carried out separately, and subsequently combined and presented to the production staff via a Graphical User Interface (GUI).

A system embodying the invention will now be described in relation to Figure 1 which shows the functional components of an entire system embodying the invention. Multiple sources of audio video content provide that audio video content to one or more displays 12 via a connection path 7. Typically, the sources will be cameras either in a studio or at a remote location connected via a wired or wireless communication link 7 direct to the displays 12 of a production team. In addition, the cameras will connect to a variety of other studio equipment including recorders, controllers, and so on that are not shown here for simplicity of explanation. In addition to connecting to the displays 12, the cameras 2 have an input 3 to a monitoring system 4. The input 3 may comprise multiple separate hardware inputs, one for each camera, or more typically will be a communication network over which audio video content from each of the cameras will be streamed and received at a single network input port to the monitoring system 4.

The monitoring system 4 has an output 5 that can be asserted to the displays 12 to provide a visual indication of the existence of a quality problem with one or more of the cameras 2. This visual indication can be separate from the display of image on the display, but is preferably an overlay on the relevant display or part of the display. In this regard, the output signal on the output 5 may include the audio video content itself combined with the visual indicator, In such an arrangement the monitoring system 4 would be included as part of a larger system such as a production suite.

The monitoring system 4 also has an output B providing data indicating the nature of any quality issues regarding the audio video sources coupled to a metadata store 13 arranged receive the data and to as store metadata describing any quality problems with the corresponding feeds. The metadata may be periodically stared, such as at a certain interval of a number of frames, or may be stored whenever there is a change indicated by the signal. 1o*

The monitoring system 4 comprises an analysis module 6, a settings database 8 and an output control 10. The settings database 8 stores user definable values to indicate the sources 2 for which monitoring is required and the types of analysis that should be performed to determine if those sources are correctly functioning. The types of analysis will be described later.

The analysis module 6 undertakes the analysis of each audio video signal in accordance with the settings from the settings database 8 and provides the results to output control 10 which asserts the output signal either separately or as part of the audio video signal at the output 5 as already mentioned.

The settings database B may be implemented by any known database -technology and functionally provides a table with a row for each source 2 and columns for each of the types of analysis that are to be performed. The settings database B may be updated by an input at input line 9, which would typically be by a graphical user interface software controller. In this way, the user can select exactly which of the analysis techniques should be applied to which of the camera sources 2. In addition, the settings database stores for each source analysis combination the nature of the alert that should be given, for example green, amber or red. The user thus has control over the nature of the analysis for each source as well as the nature of the indication given.

The type of analysis that will be performed will now be described in greater detail with reference to the remaining figures. In digital images, image noise can occur at the point of capture due to properties of the sensors used to detect the photons

S

falling upon them. In this note we will be principally concerned with detecting digital image noise, however, similar methods could be used to detect certain types of film grain noise, for example.

Digital image noise can occur in a variety of ways due to a variety of underlying physical mechanisms. Sensor noise occurs due to the electron excitation mechanism from an incident photon, and it can be modelled using a Poisson distribution. Similarly, thermal noise can occur on the sensor due to the effect thermal temperature can have on the electrons, and this noise tends to be Gaussian distributed. Heavy tailed distribution noise can also occur for a variety of reasons, which is often termed "salt and pepper" noise. As well as the underlying mechanisms associated with the sensor, quantisation noise (associating the sensor response with discreet pixels of finite bit depth, which can be uniform distributed) and amplification noise (related to signal processing techniques) can occur. In low light levels for example, the camera operator may decide to add gain', which amplifies the signal, in turn adding more noise and further amplifying the existing noise. If the gain is too high, the noise can become visually distracting to the viewer. Therefore, digital image noise in general can have a variety of different attributes, and it would be desirable for an image noise detection algorithm to detect as many different types of image noise as possible.

The algorithm and system of this disclosure identifies image noise which would could be a stochastic mix of Poisson, Gauss or uniform distributed random "speckle-like" noise.

In broad terms, the process of this disclosure aims to isolate the unwanted image noise from the image detail. Figure 2 is an image showing an example of a high degree of noise, caused by too high a gain setting on the camera, which should fail a broadcaster's technical review requirements. Note that to the eye it is easier to discern the noise from the rain in the dark un-detailed regions of the image (their jackets) rather than the highly detailed sections (for example the trees).

This already highlights the difficulty a noise detection algorithm can face (e.g. discerning noise from rain) and exemplifies a general feature of any noise

S

detection algorithm, namely the ability to distinguish noise (due to the method of capture) from genuine image detail (which was present in the actual scene).

A first aspect of a noise detection algorithm is to process the image to attempt to remove genuine detail from the scene. This can be done in practice by creating two masks of the image; one from edge detection (for example using a Sobel filter) and a second mask from applying a Laplace filter. In intuitive terms this can be thought of as the local variance between the first spatial derivative and the second spatial derivative (or local divergence). If a local region has a high spatial divergence this could be due to true image detail or due to noise. lithe inverse of the edge detector is combined with the Laplace image then an image representing the noise in the image can be formed. An outline of the algorithm is shown in Figure 3.

Figure 4 shows the output of Sobel filter (edge detector) and Figure 5 shows the inverse of Sobel image with a threshold applied. Figure 6 shows the image convolved from Laplace Filter and Figure 7 is the masked Sobel Image from the Convolved Laplace Image.

A drawback of the Laplace to Sobel comparison approach is that the overall level for the threshold is still in general dependent upon the level of image detail. A possibility is to use an adaptive threshold.

Instead, a different approach is taken in this disclosure, in which the combined mask image is further refined using object detection (for example using contiguous regions of pixel method).

The object detection algorithm applies a probability curve to each speckle-like noise pixel in Figure 7. Those pixels which are true speckles -i.e. objects of size just one pixel, are assigned the greatest probability whilst those pixels which are in larger speckles are assigned lower probabilities (are grey rather than white) with the largest speckle-like objects being assigned the lowest probability. This can serve to further reduce the effect of image detail, as remaining features such as edges are likely to form the largest objects and hence are assigned the lowest probabilities as can be seen in Figure 8.

Figure 8 above can be thought of a representation of the "noisiness" of the original image. A noisier original image should result in more speckles in Figure 5, and a less noisy image in fewer speckles. Some image structure is still visible, mainly in what were bright parts of the image. The pixel values in these brighter regions were saturated and thus those regions could not contain any noise. The average value of Figure 8 can now be calculated to give a representation of the image noise value, a threshold applied and a warning flagged to the production staff if the image noise value is above the threshold. The threshold may be determined with a variety of test footage and camera gain settings, but it is also possible to classify this parameter using a large set of known "noisy" and known good" footage.

Claims

SCLAIMS1. A method of analysing an image to determine the presence or absence of noise, comprising: -creating a first mask of the image using edge detection; -creating second a second mask by applying a filter; -combining the first and second masks to produce a noise image; -processing the noise image to emphasise small regions of similar luminance and to de-emphasise larger regions of similar luminance; -determining an average of the processed noise image; and -comparing the average to a threshold to determine whether to assert that the image is noisy.
2. A method according to claim 1, wherein the filter is a 2D second derivative filter.
3. A method according to claim 1, wherein the filter is a Laplace filter.
4 A method according to claim 1, wherein the step of processing to emphasise comprises assigning a greater probability to speckles of one pixel and assigning successively lower probabilities to speckles of greater than one pixel.
5. A method according to claim 1 wherein the step of processing to emphasis comprises an object recognition algorithm.
6 Apparatus far analysing an image to determine the presence or absence of noise, comprising: -means for creating a first mask of the image using edge detection; -means for creating second a second mask by applying a filter; -means for combining the first and second masks to produce a noise image; -means for processing the noise image to emphasise small regions of similar luminance and to de-emphasise larger regions of similar luminance; -means for determining an average of the processed noise image; and -means for comparing the average to a threshold to determine whether to assert that the image is noisy.
7. Apparatus according to claim 6, wherein the filter is a 20 second derivative filter.
8. Apparatus according to claim 6. wherein the filter is a Laplace filter.
9. Apparatus according to claim 6, wherein the step of processing to emphasise comprises assigning a greater probability to speckles of one pixel and assigning successively lower probabilities to speckles of greater than one pixel.
10. Apparatus according to claim 6, wherein the step of processing to emphasis comprises an object recognition algorithm.