WO2008059292A2

WO2008059292A2 - Holographic data processing apparatus

Info

Publication number: WO2008059292A2
Application number: PCT/GB2007/050693
Authority: WO
Inventors: Edward Buckley
Original assignee: Light Blue Optics Ltd
Priority date: 2006-11-15
Filing date: 2007-11-15
Publication date: 2008-05-22
Also published as: GB2433371A; GB0622777D0; GB2434706B; WO2008059292A3; GB2434706A; GB2433371B; GB0708790D0

Abstract

We describe a hardware error diffusion calculation system (1004) for performing an error diffusion quantisation of a hologram. The system comprises address and data buses (622, 624, 626) for a first, hologram memory block storing data representing pixels of said hologram,- for a second, error memory block storing an error matrix representing errors in quantising said hologram; for a third, diffusion weights memory block storing a set of diffusion weights for said error diffusion calculation; and a plurality of error diffusion processor (602) blocks each coupled to said address and data buses, each being configured for sequential processing of a spatial region of said hologram. A said processor block includes a pixel address generator to generate a sequence of pixel addresses for a said spatial region for processing by the processor block and an error diffusion calculation block (616) coupled to said pixel address generator and to said address and data buses to perform an error diffusion calculation for a pixel of said hologram ident ified by a said pixel address. Memory corresponding the spatial regions processed by said processor blocks overlap and each of said processor blocks has a different start pixel address such that said processor blocks are able to perform error diffusion calculations with respective error diffusion calculation blocks on different pixels of the hologram in parallel.

Description

Data Processing Apparatus

FIELD OF THE INVENTION

This invention relates to data processing hardware for generating holograms, in particular for image display applications.

BACKGROUND TO THE INVENTION

There are a number of techniques which may be employed to generate data for creating a hologram from image data, hi general, however, such techniques introduce noise into the hologram data, for example resulting from quantisation into binary phase for display on a spatial light modulator (SLM), or quantisation into a larger number of binary bits.

Some examples of techniques for generating hologram data from image data are described in the following:

M. A. Seldowitz, J.P. Allebach, and D. W. Sweeney, "Synthesis of digital holograms by direct binary search," Applied Optics, vol. 26, pp. 2788-, 1987; M.P. Dames, RJ. Dowling, P. McKee, and D. Wood "Efficient optical elements to generate intensity weighted spot arrays: design and fabrication," applied Optics, vol. 30, pp. 2688-2691, 1991; and J.R. Fienup, "Iterative method applied to image reconstruction and to computer-generated holograms," Optical Engineering, vol. 19, pp. 297-305, 1979.

One particularly preferred procedure for calculating hologram data is the OSPR procedure which is described in more detail later.

One technique for noise reduction is called error diffusion (ED). Broadly speaking in such a technique a window is defined and the signal-to-noise (SNR) ratio within the window is improved at the expense of the SNR outside the window by, in effect, moving errors from within the window to outside the window. The error diffusion technique is described in more detail later.

One problem with implementing the error diffusion technique in practice is that it is computationally very intensive. Background prior art can be found in EP0606987A; Kang H.R., "Parallel Error Diffusion", Proc SPIE, 2001, vol 4663, pp360-369; and JP2000-122514A. One technique to address this problem using software has been described by P. T. Metaxas in "Optimal parallel error-diffusion dithering," in Proceedings of the 1999 Electronic Imaging. SPIE, 1999; and "Parallel digital halftoning by error-diffusion," in Proc. of the FCRC 2003, Paris, 2003; see also US6307978. However this is still computationally intensive and requires a mainframe computer system for its implementation. This effectively precludes the use of the error- diffusion technique in practical, real-life applications.

Accordingly there is a need for improved approaches.

SUMMARY OF THE INVENTION

In a first aspect of the invention there is therefore provided a hardware error diffusion calculation system for performing an error diffusion quantisation of a hologram, the system comprising: an address bus and a data bus for a first, hologram memory block, said hologram memory block for storing data representing pixels of said hologram; an address bus and a data bus for a second, error memory block, said error memory block for storing an error matrix representing errors in quantising said hologram; an address bus and a data bus for a third, diffusion weights memory block, said diffusion weights memory block for storing a set of diffusion weights for said error diffusion calculation; and a plurality of error diffusion processor blocks each coupled to said address and data buses for each of said first, second and third memory blocks, each of said processor blocks being configured for sequential processing of a spatial region of said hologram, a said processor block including a pixel address generator to generate a sequence of pixel addresses for a said spatial region for processing by the processor block and an error diffusion calculation block coupled to said pixel address generator and to said address and data buses to perform an error diffusion calculation for a pixel of said hologram identified by a said pixel address; wherein spatial regions processed by said processor blocks overlap, and wherein each of said processor blocks has a different start pixel address, such that said processor blocks are able to perform error diffusion calculations with respective said error diffusion calculation blocks on different pixels of said hologram in parallel.

In embodiments the multiple error diffusion processor blocks are all in communication with global, shared memory storing the hologram data for error diffusion processing/binarisation, the error matrix, and the diffusion weights (kernel). These data may stored in separately addressable parts of the same memory or in different memories and, as the skilled person will understand, the address and data buses may be shared and/or combined. Preferably a single memory block stores both the continuous hologram data and the quantised hologram data since the quantised hologram data may overwrite the continuous hologram data.

The pixel address generator enables the blocks of hologram data processed by each processor block to overlap but nonetheless determines a sequence for the calculations which, in embodiments, preserves the order of the error diffusion calculations so that valid data are available when required. More particularly the sequence of pixel addresses defines a generally diagonal path through a spatial region processed by a processor block. The start pixel or point for the different processor blocks is also staggered over the spatial region of the hologram, more particularly in embodiments along one dimension of the hologram.

Preferred embodiments of the system also include a scheduler to generate a control signal for each processor block to control each processor block to start processing in turn. This facilitates an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) implementation in which the architecture may impose a memory access delay between one memory access and the next.

In preferred embodiments the processor block includes a controller to provide the start pixel address for a processor block. In this way the allocation of processing blocks to overlapping spatial regions of the hologram data is performed locally by a processor block rather than by overall global control. The start pixel address may be determined by a hardwired or other non- volatile stored value, for example in embodiments where the hologram size is fixed. (In such embodiments the "controller" need only have the function of providing such a fixed address, although the scheduler will nonetheless start up each processor block in order). However in other embodiments the hardware is flexible and allows for a range of different hologram sizes, in which case the controller may determine a start pixel address for a processor block dependent upon a value dependent upon the size of the hologram, in particular in one dimension (u or v). The start pixel addresses are preferably staggered along the other dimension of the hologram (v or u). In embodiments each processor block has a processor block identifier and the controller is configured to determine a start pixel address using this identifier to stagger the start pixel addresses in this way.

In preferred embodiments the error diffusion calculation block comprises a multiply- accumulate block coupled to the data buses for the error matrix and diffusion weights memory blocks, the multiply - accumulate block having an output which is used for determining a quantised hologram pixel value, in particular by thresholding the multiply- accumulate block output, for example about zero. Preferably the error diffusion calculation block also includes a subtractor to subtract the thresholded data from the (continuous) hologram data to provide updated error data for writing to the error memory block. This subtractor may either use the continuous hologram data in the hologram memory block or may employ data derived from an output of the multiply-accumulate block, that is changed hologram data, this having been modified by the diffused errors. In this way either "standard" ED or MAE may be implemented. Optionally the error diffusion calculation block may be configurable to perform either type of error diffusion calculation by selecting the input to the subtractor in response to a value stored in a register associated with the error diffusion processor block.

As previously implied, preferably the quantisation comprises binarisation; the error diffusion system may be employed with amplitude and/or phase data.

In preferred embodiments an error diffusion processor block includes a memory controller coupled to the pixel address generator to provide addresses for the error memory and diffusion weights memory blocks to address an error diffusion window over which the error diffusion calculation is performed. Thus the memory controller generates a sequence of addresses [r, s] to cover the error diffusion window and, in preferred embodiments, also translates [u, v] to memory addresses for the hologram memory block. Optionally but preferably the memory controller may also perform validity checking on the generated addresses to ensure that only valid pixel addresses are generated, that is pixel addresses that are within the spatial region defined by the hologram data.

Preferably the hardware error diffusion calculation system is embodied in either an FPGA or an ASIC. Such an implementation may optionally include memory for the hologram and/or error and/or diffusion weights memory blocks.

The invention also provides a holographic image display system including a hardware error diffusion calculation system as described above, in particular employing an OSPR-type holographic subframe calculation procedure. Where a hardware OSPR engine is employed, preferably the OSPR engine and the error diffusion calculation system are pipelined so that an error diffusion calculation for one subframe is performed concurrently with a holographic subframe calculation for the next subframe. However, this pipelined approach may also be employed in non-OSPR based systems.

Thus in a further aspect the invention provides a holographic image display processing system, the system being configured to provide data to display an image holographically, said image being defined by displayed image data, the system comprising: a first system to generate hologram data from said image data; a second system to quantise said hologram data for display; and wherein said first and second systems are pipelined such that said hologram data quantisation is performed concurrently with said hologram data generation.

In some preferred embodiments the second system to quantise said hologram data for display comprises an error diffusion system. However in other embodiments the second system may be configured to implement other forms of quantisation. For example the pipelines systems may be configured to implement the noise compensation process illustrated in Figure 11 of WO 2007/031797, which specific figure, and the accompanying description of which, is hereby incorporated by reference. Thus in this case the second system may be configured to implement a quantization process which performs a noise compensation by compensating for (subtracting) noise from one or more previous sub-frames.

In preferred embodiments the processing system is configured to provide data to display the image using a plurality of holographically generated temporal sub frames, the temporal subframes being displayed sequentially in time such that they are perceived as a single noise-reduced image, the first system being configured to generate holographic subframe data, and the first and second systems being pipelined such that the quantisation of this data for one subframe is performed substantially concurrently with the generation of holographic subframe data for a subsequent subframe, in particular the next subframe. The invention also provides a holographic image display system incorporating such a processing system.

Thus, broadly speaking in embodiments a first hardware hologram data generation system is coupled to a hardware error diffusion calculation system, in particular as described above, these two systems being pipelined to enable a hologram to be generated and an error diffusion calculation to be performed, in particular with one (sub)frame latency. In some preferred embodiments to economise on memory the two systems, that is the hologram calculation system and the hardware error diffusion calculation system share common memory for storing the hologram data.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will now be further described, by way of example only, with reference to the accompanying figures in which:

Figure 1 shows an outline block diagram of an embodiment of a hologram calculation hardware accelerator for a holographic image display system;

Figure 2 shows an example holographic projection system; Figure 3 shows a block diagram of a system in which an image frame is used to produce one or more holographic sub-frames;

Figure 4 shows example energy distributions for an image before and after phase- modulation;

Figure 5 shows the variation of replay field noise energy and SNR as a function of error diffusion kernel size;

Figure 6a to 6c show a hardware error diffusion calculation system according to an embodiment of an aspect of the invention;

Figures 7a and 7b show, respectively, a schematic illustration of calculations performed by the system of Figure 6, and time scheduling of error diffusion processing cores for the hardware of Figure 6;

Figure 8 shows a graph of processable OSPR sub frames against kernel size for a range of numbers of processor cores for the hardware of Figure 6;

Figures 9a and 9b show a holographically generated image respectively without and with error diffusion processing; and

Figures 10a to 10c show, respectively, a pipelined hologram data generation system with error diffusion according to an embodiment of an aspect of the invention, a timing diagram for the pipeline, and a further example of a pipelined hologram data generation system.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

To assist in understanding embodiments of the invention, and to provide context, we first describe one technique, OSPR, for calculating hologram data with which embodiments of the invention may be employed (although the skilled person will understand that applications of embodiments of the invention are not limited to hologram data calculated using the OSPR technique).

OSPR

Our preferred procedure for calculating hologram data, for example for display on an SLM, is what we refer to in broad terms as One Step Phase Retrieval (OSPR). However strictly speaking in some implementations it could be considered that more than one step is employed (as described for example in GB0518912.1 and GB0601481.5, incorporated by reference, where "noise" in one sub-frame is compensated in a subsequent sub-frame).

Thus we have previously described, in WO2005/059660, a method of displaying a holographically generated video image comprising plural video frames, the method comprising providing for each frame period a respective sequential plurality of holograms and displaying the holograms of the plural video frames for viewing the replay field thereof, whereby the noise variance of each frame is perceived as attenuated by averaging across the plurality of holograms.

Broadly speaking in our preferred method the SLM is modulated with holographic data approximating a hologram of the image to be displayed. However this holographic data is chosen in a special way, the displayed image being made up of a plurality of temporal sub-frames, each generated by modulating the SLM with a respective sub-frame hologram. These sub-frames are displayed successively and sufficiently fast that in the eye of a (human) observer the sub-frames (each of which have the spatial extent of the displayed image) are integrated together to create the desired image for display.

Each of the sub-frame holograms may itself be relatively noisy, for example as a result of quantising the holographic data into two (binary) or more phases, but temporal averaging amongst the sub-frames reduces the perceived level of noise. Embodiments of such a system can provide visually high quality displays even though each sub-frame, were it to be viewed separately, would appear relatively noisy. A scheme such as this has the advantage of reduced computational requirements compared with schemes which attempt to accurately reproduce a displayed image using a single hologram, and also facilitate the use of a relatively inexpensive SLM. (Here it will be understood that an SLM will, in general, provide phase rather than amplitude modulation, for example a binary device providing relative phase shifts of zero and τ, +1 and -1 for a normalised amplitude of unity).

We have also described, in PCT/GB2006/050152 (incorporated by reference), a hardware accelerator for a holographic image display system, the image display system being configured to generate a displayed image using a plurality of holographically generated temporal sub-frames, said temporal sub-frames being displayed sequentially in time such that they are perceived as a single reduced-noise image, each said sub- frame being generated holographically by modulation of a spatial light modulator with holographic data such that replay of a hologram defined by said holographic data defines a said sub-frame, the hardware accelerator comprising: an input buffer to store image data defining said displayed image; an output buffer to store holographic data for a said sub-frame; at least one hardware data processing module coupled to said input data buffer and to said output data buffer to process said image data to generate said holographic data for a said sub-frame; and a controller coupled to said at least one hardware data processing module to control said at least one data processing module to provide holographic data for a plurality of said sub-frames corresponding to image data for a single said displayed image to said output data buffer.

In this preferably a plurality of the hardware data processing modules is included for processing data for a plurality of the sub-frames in parallel. In preferred embodiments the hardware data processing module comprises a phase modulator coupled to the input data buffer and having a phase modulation data input to modulate phases of pixels of the image in response to an input which preferably comprises at least partially random phase data. This data may be generated on the fly or provided from a non-volatile data store. The phase modulator preferably includes at least one multiplier to multiply pixel data from the input data buffer by input phase modulation data, hi a simple embodiment the multiplier simply changes a sign of the input data. An output of the phase modulator is provided to a space-frequency transformation module such as a Fourier transform or inverse Fourier transform module. In the context of the holographic sub-frame generation procedure described later these two operations are substantially equivalent, effectively differing only by a scale factor. In other embodiments other space-frequency transformations may be employed (generally frequency referring to spatial frequency data derived from spatial position or pixel image data). In some preferred embodiments the space-frequency transformation module comprises a one-dimensional Fourier transformation module with feedback to perform a two-dimensional Fourier transform of the (spatial distribution of the) phase modulated image data to output holographic sub-frame data. This simplifies the hardware and enables processing of, for example, first rows then columns (or vice versa).

In preferred embodiments the hardware also includes a quantiser coupled to the output of the transformation module to quantise the holographic sub-frame data to provide holographic data for a sub-frame for the output buffer. The quantiser may quantise into two, four or more (phase) levels. In preferred embodiments the quantiser is configured to quantise real and imaginary components of the holographic sub-frame data to generate a pair of sub-frames for the output buffer. Thus in general the output of the space-frequency transformation module comprises a plurality of data points over the complex plane and this may be thresholded (quantised) at a point on the real axis (say zero) to split the complex plane into two halves and hence generate a first set of binary quantised data, and then quantised at a point on the imaginary axis, say Qj, to divide the complex plane into a further two regions (complex component greater than 0, complex component less than 0). Since the greater the number of sub-frames the less the overall noise this provides further benefits.

Preferably one or both of the input and output buffers comprise dual-ported memory. In some particularly preferred embodiments the holographic image display system comprises a video image display system and the displayed image comprises a video frame. Figure 1, which is modified from PCT/GB2006/050152, shows a block diagram of an embodiment of such a hardware accelerator. Each buffer preferably comprises dual- port memory such that data is written into the buffer and read out from the buffer simultaneously. The hardware block performs a series of operations on each of the image frames, I, and for each one produces one or more holographic subframes, h, under control of the controller unit.

The holographic subframe data is sent to the output buffer and may be supplied to a display device, such as a SLM, optionally via a driver chip. However in embodiments of the system we describe the holographic subframe data provides an input to error diffusion calculation hardware, described later. (The skilled person will understand, however, that the error diffusion calculation hardware we describe may also receive input data from a hologram calculation system comprising a combination of software and hardware, or just software; it also need not receive hologram data from an OSPR- based hologram calculation system).

Such subframes are outputted from the aforementioned output buffer and. The control signals by which this process is controlled are supplied from one or more. The control signals preferably ensure that one or more holographic subframes are produced and sent to the SLM per video frame period. In an embodiment, the control signals transmitted from the controller to both the input and output buffers are read / write select signals, whilst the signals between the controller and the hardware block comprise various timing, initialisation and flow-control information.

The hardware (and/or software) may implement a version or variant of the algorithm given below. The algorithm is a method of generating, for each still or video frame I = I_xy, sets of N binary-phase holograms h^... h^. Statistical analysis of the algorithm has shown that such sets of holograms form replay fields that exhibit mutually independent additive noise. 1 , Let G__xy = /_vyβxp f jφl'_y j where ψ_Λf k uniformly distributed between 0 and 2π for 1 < n < N /2 and 1 < x_t y < m

2, Let gil_v = F^{~~ ]} [G_W ] where F^{~~ ]} represents the two-dimensional inverse Fourier transform operator, for 1 < n < N /2

3. Let n& = Sl{g^} for 1 < ι, < N /2

4. Let n& ' ^N/2) = 3 {^tf } for 1 < n < N /2

and 1 < H < Λ^r

Step 1 forms N targets G^⁰ equal to the amplitude of the supplied intensity target I_xy, but with independent identically-distributed (i.i.t.)) uniformly-random phase. Step 2 computes the N corresponding full complex Fourier transform holograms g^ . Steps 3 and 4 compute the real part and imaginary part of the holograms, respectively. Binarisation of each of the real and imaginary parts of the holograms is then performed in step 5: thresholding around the median of m^ ensures equal numbers of -1 and 1 points are present in the holograms, achieving DC balance (by definition) and also minimal reconstruction error. The quantisation may be performed in a number of ways; in an embodiment, the median value of rn^ is assumed to be zero. This assumption can be shown to be valid and the effects of making this assumption are minimal with regard to perceived image quality. Further details can be found in the applicant's earlier application (ibid), to which reference may be made.

Figure 2 shows an example holographic projection system, further details of which may be found in PCT/GB2006/050158 to which reference may be made. A laser diode 20 (for example, at 532nm), provides substantially collimated light 22 to a spatial light modulator 24 such as a pixellated liquid crystal modulator. The SLM 24 phase modulates light 22 with a hologram and the phase modulated light is provided to a demagnifying optical system 26 which projects a 2D (in this example) image onto screen 14. In the illustrated embodiment, optical system 26 comprises a pair of lenses 28, 30 with respective focal lengths f₂, fi<f₂, spaced apart at distance fi+f₂. Optical system 26 (which is not essential) increases the size of the projected holographic image by diverging the light forming the displayed image, as shown. One or more of the lenses may be encoded in the hologram, as described in UK patent application GB 0606123.8 filed on 28 March 2006. A filter may be included to filter out unwanted parts of the displayed image, for example a zero order undiffracted spot. In a measurement device, the demagnifying optics may be omitted; alternatively lens 30 (L4) and screen 14 may be replaced by, say, a digital camera.

A hologram calculation system 100 has an input 102 to receive image data from the consumer electronic device defining the image to be displayed. The hologram calculation system 100 implements a procedure, for example along the lines described above, to generate phase hologram data, in an OSPR-based display device data for a plurality of holographic sub-frames. This data is provided from an output 104 of the hologram calculation system 100 to the SLM 24, optionally via a driver integrated circuit if needed. The hologram calculation system 100 drives SLM 24 to project a plurality of phase hologram sub-frames which combine to give the impression of displayed image 14 in the replay field (RPF). The hologram calculation system 100 may comprise dedicated hardware or Flash or other read-only memory storing processor control code to implement an OSPR-type hologram generation procedure in conjunction with a DSP (digital signal processor), or a combination of hardware and software.

In embodiments of the above-described system the SLM may comprise a ferroelectric liquid crystal-based SLM. However in general any type of pixellated microdisplay which is able to phase modulate light may be employed for the SLM, optionally in association with an appropriate driver chip if needed. Preferred embodiments use an electrically addressable SLM. Suitable SLMs include, but are not limited to, liquid crystal SLMs including LCOS (liquid crystal on silicon) and DLP (registered TM) (digital light processing) SLMs. A suitable SLM is available from CRL Opto (Forth Dimension Displays Ltd of Scotland, UK), with part number SXGA-R2-H1 (pixel pitch ^_x = ^_y = U.β2μ m)

Figure 3 shows a block diagram of a system in which an image frame, I_xy, is used to produce one or more holographic sub-frames by means of a set of operations comprising one or more of: a phase modulation stage, a space-frequency transfoπnation stage and a quantisation stage. The purpose of the phase-modulation block is to redistribute the energy of the input frame in the spatial-frequency domain, such that improvements in final image quality are obtained after performing later operations. The phase-modulation data may comprise a pseudo-random sequence.

The quantisation block of Figure 3 has the purpose of taking complex hologram data, which is produced as the output of the preceding space- frequency transform block, and mapping it to a restricted set of values, which correspond to actual phase modulation levels that can be achieved on a target SLM. The number of quantisation levels may be set at two, for example for an SLM producing phase retardations of 0 or π at each pixel. Figure 3 shows use of the real part of the holographic sub-frame data but alternatively real and imaginary components of the holographic sub-frame data may be quantised to generate a pair of sub-frames, each with two phase-retardation levels (for discretely pixellated fields these are uncorrelated).

Figure 4 shows an example of how the energy of a sample image is distributed before and after a phase-modulation stage in which a random phase distribution is used. It can be seen that modulating an image by such a phase distribution has the effect of redistributing the energy more evenly throughout the spatial-frequency domain.

It is next helpful in understanding embodiments of the invention to describe error diffusion.

Error Diffusion

Error diffusion techniques applied to the binarisation of continuous hologram patterns have been described in the following background material: M.P. Chang and O.K. Ersoy, "Iterative interlacing error diffusion for synthesis of computer-generated holograms," Applied Optics, vol. 32, pp. 3122-, 1993; R. Eschbach, "Comparison of error diffusion methods for computer- generated holograms," Applied Optics, vol. 30, pp. 4361-, 1991; R. Eschbach and Z. Fan, "Complex-valued error diffusion for off-axis computer generated holograms," Applied Optics, vol. 32, pp. 3130-1993; A. A. Falou, M. Elbouz, and H. Hamam, "Segmented phase-only filter binarised with a new error diffusion approach," Journal of Optics A: Pure and Applied Optics, vol. 7, 2005; O. B. Frank Fetthauer, "On the error diffusion algorithm: object dependence of the quantization noise," Optics Communications, vol. 120, 1995; F. Fetthauer and O. Bryngdahl, "Use of error diffusion with space-variant optimized weights to obtain high- quality quantized images and holograms," Optics Letters, vol. 23, pp. 739-741, 1998; L. Ge, M. Duelli, and R. W. Cohn, "Improved- fidelity error diffusion through blending with pseudorandom encoding," J. Opt. Soc. Am. A, vol. 17, pp. 1606-1616, 2000.

An error diffusion procedure covering two variants of the procedure, ED and MAE, is given below. The example procedure relates to a PxP pixel hologram (although there is no need for u and v both to have ranges [1;P]) with pixel dimensions [u,v] within which a window with pixel dimensions [r,s] is defined.

111 HV h KV MΛE algorit hm

3. Le e,

///J_n, - Ji_1n, ED algorii Inn

4. L(M, Ji_ιιv whoiv Q = median (//4,)

In this procedure m_uv represents continuous hologram data (real and/or imaginary component), e_uv diffused errors, d_rs diffusion weights, and h_uv a binary phase representation of the hologram, hi the techniques we describe later m_uv may comprise a real or imaginary component of hologram data from an OSPR procedure. Conveniently Q is a median pixel value but may also be a constant, for example zero.

To obtain local SNR improvement in a window W defined in the RPF (replay field) RPF noise is optimised in the region CFby diffusing hologram pixels according to a diffusion kernel of size K where d_rs is the appropriately bandlimited Fourier transform of the window function W. More particularly the diffusion kernel is calculated by calculating the Fourier transform (in 2 or more dimensions) of the window function, and then truncating the potentially infinite Fourier series, for example taking a set of components around zero-spatial frequency. The window function may conveniently comprise a function defined over the area of the replay field, with a value of "1" over the window and a value of "0" elsewhere.

In step 2 e_uv comprises a matrix which represents errors introduced by the quantisation (binarisation) process, more particularly the error in binarising one or more previous pixels [u,v]. The diffusion kernel d_rs represents a weighting of these errors over a window of dimension [r,s], preferably centred on the currently processed pixel [u,v].

As shown, the error matrix e_wvmay initially be set to zero and will gradually accumulate error data as more pixels are processed. An error for a currently processed pixel is calculated at step 3. There are two main ways in which this error may be determined, although applications of embodiments of the invention are not limited to these. A Minimum Average Error (MAE) calculation may be employed to determine the difference between a binarised pixel value h_uv and a real (and/or imaginary) part of the complex hologram data; or in a standard error diffusion (ED) procedure the difference may be between the binarised pixel value and a changed (c) value determined in error diffusion step 2. These two approaches merely differ in the quantisation of the diffused errors e_uv and can be made to behave equivalently by appropriate choice of the diffusion weights d_rs. Thus either an ED or an MAE procedure can be implemented by changing the kernel (i.e. the data stored in the diffusion kernel memory) and embodiments of the invention we describe are able to implement both ED and MAE (but not limited to these). An example of an error diffusion technique with iterative weight calculation is described in: A. Kirk, K. Powell, and T. Hall, "A generalisation of the error diffusion method for binary computer generated hologram design," Optics Communications, vol. 92, 1992.

The binarisation step 4 of the procedure may then performed: thresholding around the median provides substantially equal numbers of -1 and 1 points in the hologram, giving DC balance and also low reconstruction error. However the median value may be assumed to be zero with minimal impact on perceived image quality.

Broadly speaking the error diffusion step 2 diffuses errors over a window of size [r,s] determining a changed or adjusted value for the real and/or imaginary component of the complex hologram data taking into account these diffused errors, that is taking into account the binarisation which is employed (at a later step) for displaying the hologram on an. The error broadly speaking comprises a difference between a quantised (binarised) pixel and the unquantised, continuous value of the pixel.

The region over which the error diffusion is applied depends upon the size of the window, a larger window using a larger diffusion kernel. Broadly speaking the size of the diffusion kernel determines the "quality" of the diffusion process but a larger kernel, even with hardware, requires greater computation. Similarly a greater improvement in signal-to-noise ratio (SNR) can be achieved by using a larger diffusion kernel (or a less truncated Fourier series): The window becomes increasingly well-defined, the SNR increases and the signal and noise histograms become increasingly better separated.

Thus there is a trade off between the window size, desired SNR improvement, and the number of hardware error diffusion processors (as described below). In theory the size of the window can approach the size of the replay field but it then becomes harder to remove noise from the window; in practice a smaller window can nonetheless provide useful benefits because the points in the (replay field) window are still effectively at higher resolution, albeit the image area is reduced. Figure 5 shows the variation of RPF noise energy and SNR as a function of kernel size, showing that the noise energy falls rapidly as K increases, leading to a similarly rapid SNR rise. The figure can be used to determine a point at which increased computation outweighs the benefit of increased SNR; here at approximately K = 15. Therefore this was set as the maximum kernel size for the hardware implementation described below.

Hardware error diffusion calculation system

We will describe an error diffusion processor which is designed to act as a co-processor to augment an OSPR computation engine. However, assuming that the ED and OSPR processors are used in tandem such that one OSPR subframe is calculated and the ED binarisation procedure is performed immediately afterwards, then in order to remove the need for buffering many hologram sub frames the ED binarisation of the first subframe should preferably take place in parallel with the OSPR calculation of the second subframe. This implies that the ED co-processor should preferably take no longer to complete its operation than the time it takes to calculate one OSPR subframe. At, for example, a 60Hz video frame rate a system employing eight OSPR subframes per video frame would preferably need to calculate each subframe within 2 ms.

Referring to the error diffusion procedure described above, each pixel of a P x P hologram should be traversed and Kx K multiplications and additions performed at each pixel. We will describe an FPGA-based design which employed a Xilinx^® Virtex- 4 SX 35 FPGA (for which the design was written in Verilog^® using Xilinx^® ISE tool). The FPGA logic was configured to replicate multiple cores, each of which was capable of simultaneously calculating part of the error diffusion procedure. The resulting parallelisation resulted in an approximate 7V_c-fold speed increase, where N_c is the number of replicated cores.

Referring to Figure 6a, this shows an embodiment of a hardware error diffusion calculation system 600 according to an aspect of the invention. The system 600 comprises a plurality of error diffusion processor blocks or cores 602a - h, each of which is coupled to a scheduler 604. The system also includes an error diffusion weight (d) memory 606, an error (e) memory 608, and a hologram data (m) memory 610 for storing values d_rs, e_uv, and m_uv respectively as described in the ED procedure and preferably also h_uv. Each of these memories has an associated data bus and an associated address bus coupled to each of the processor blocks 602 (these connections have been omitted in Figure 6a, for clarity). The memories 606, 608 and 610 may either be internal or external to the FPGA.

Preferably the hologram data memory 610 is dual ported to allow continuous hologram data to be written to this memory and quantised (binarised) hologram data to be read from this memory for output to a display; the input and output lines are not explicitly shown in Figure 6a.

In operation the scheduler 604 provides a value, /c, to each of the processor blocks, each of which also has an associated processor block identifier (PROC ID). The timing of the sending of the value of /cby the scheduler to a processor block controls the timing of the initiation of the processor block in performing its calculations. This timing is shown in Figure 7b, where it can be seen that each processor block is started in succession with a small delay, for example a few memory access cycles, in between each. (The delay is implementation-memory specific and is chosen to be sufficient to allow stored data to become valid after a memory write).

Referring now to Figure 6b, this shows details of a processor block or core 602 of Figure 6a. The processor block 602 comprises a controller 612 with a control bus 614 coupled to scheduler 604 of Figure 6a, to receive a value of k from the scheduler. As described in more detail below, the value of /c depends upon the size of the hologram to be processed and determines how the error diffusion calculation is allocated amongst the available processor blocks. The controller waits for the value of /c to be sent by the scheduler and then provides an RDY (ready) output 614a to a process_pixel block 616, described below, the RDY signal initiating the process_pixel core. The controller also provides values of u and v to a valid pixel block 618 which determines whether or not these values of u and v are valid, if so providing the values on respective buses 620a, b to the process_pixel block 616, in effect to provide x-y addressing of a hologram pixel to be processed. The valid_pixel block 618 comprises logic to ensure that the indices u, v represent a valid memory location and hence that the boundary conditions of the ED process are met - in other words it determines whether or not the values u and v lie outside the boundary of the hologram.

The controller 612 provides u and v pixel address data which is controlled to select pixels of the hologram data to be processed along a generally diagonal line (in the [u, v] space of the hologram). The general direction of the calculation is shown in Figure 7a; in more detail the pixels are chosen according to a sawtooth pattern along the diagonal as indicated by the open circles in Figure 7a. This is described in more detail below. The u and v values depend upon the value of A: from the scheduler (which depends upon the hologram size) and the processor block identifier "and whether values are provided depends upon whether or not the processor block has been "awakened".

The process_pixel block 616 has data and address bus connections 622a, b, 624a, b and 626a, b for the d, e and m memories, 606, 608 and 610 respectively. Details of the process_pixel block 616 are shown in Figure 6c.

Referring to Figure 6c, the v and u value buses 620a, b provide an input to a memory controller 630 which provides address outputs 622b, 624b, and 626b for the diffusion weights, error, and hologram memories 606, 608, 610 respectively. More particularly the memory controller provides addresses which sequence over the diffusion kernel, in effect the indices r, s of step 2 of the example ED procedure described above. The diffusion weights data is received on data bus 622a and the error data is received on data bus 624a, both of these data buses providing an input to a multiply-accumulate block 632 which performs the summation over r and s in step 2 of the above-described procedure. A summer 634 is coupled to the output of multiply-accumulate block 632 and to an input data bus 626aa from the hologram data memory to form the changed hologram data value m^c of step 2 of the procedure. This value is provided to a thresholder 636 to threshold about 0, a constant value or, for example, a median value of the hologram data which may be determined in advance, say by the hologram data calculation system. This thresholder performs step 4 of the above-described procedure to binarise the hologram data, providing an output h to output hologram data memory bus 626ab to be written back into the hologram data memory. A subtractor is also provided which, in the illustrated embodiment, has a first input from the output of the thresholder 636 (K) and a second input from the hologram data memory bus 626aa (m), providing an error value output (e) to error memory data bus 624a. This implements the MAE procedure described above (see step 3). Alternatively the second input to subtractor 638 may be taken from the output of the summer 634, that is m^c, to implement a standard ED procedure, hi a further alternative the second input to subtractor 638 may user selectable or programmable, for example by means of a register (not shown) to enable user selection of programming of MAE or standard ED, optionally whilst running.

Thus, broadly speaking, the process_pixel block 616 operates over a region of the hologram, receiving continuous hologram data on input bus 626aa and outputting binarised hologram data on output bus 626ab (at the same location), also providing error data on bus 624a to update the error memory 608.

The skilled person will understand that where reference is made to "continuous" hologram data this is continuous to a degree permitted by the width of the hologram data bus, and is thereafter quantised thus reducing the number of bits which are needed to describe this data, in general, binarised so that the data for a single pixel of the hologram may be represented by a single bit. The skilled person will also understand that in the arrangement of Figure 6c clock and reset lines will normally be present but in the figure these have been omitted for clarity.

Referring again to Figure 6a and 6b, the operation of the scheduler 604 and controller 612 will now be described in more detail, in particular with reference to the pseudocode shown below. k = ceil((P-l)/3); for (procID = k : k+Nc)

%

% Everything below represents the operation

% of one processor with identity "procID"

7. alpha = procID - k; u = 2; v = 3*alpha + 1;

for (t = i : 2*P+P) if valid__pixel(u, v) process__pixel(u, v++) ; if valid_pixel(Ui, v) process_pixel(u, v++) ; if valid_pixel(u, v) process_pixel(u++, v—); end end

The first line of this pseudo code illustrates the operation of the scheduler 604. The hologram has dimensions P x Q and a value for k is calculated based upon the value of P. The processor blocks are then initiated in turn using the calculated value of k, with a short delay between the initiation of each processor block. Note that one processor block does not have to complete its calculations before the next processor block is initiated. In the pseudo-code, as previously mentioned Nc represents the total number of processor blocks.

The rest of the pseudo-code describes the operation of the controller 612 of Figure 6b. More particularly the controller determines a block or spatial region of the hologram data on which to work by choosing initial value for u and v based upon the processor block identifier and the value of A: received from the scheduler. It will be seen that in this example pseudo-code a processor block always begins at u = 2 but that each processor block has a different starting value of v so that the processor blocks are allocated to successive, overlapping regions of the hologram data. In the example pseudo-code the value of /c is determined by dividing by the P axis into three and a processor block is allocated to start at every third value of v in the Q direction. The controller also, in this example, defines a group of three hologram pixels to be processed (u, v++; u++; v— ) in a group, the incremental effect of processing these groups of pixels being to carry the calculation in a generally diagonal direction. This is shown schematically in Figure 7a, and the skilled person will appreciate that variants on the above-described pseudo-code are possible which also have the effect of generally diagonal processing.

The scheduler 604 of Figure 6a and the controller 612 of Figure 6b are, in embodiments, configured according to the above-described pseudo-code, for example using a hardware design language such as Verilog^®. The valid_pixel function has previously been described; in embodiments it is configured to determine whether a value of [u, v] is valid, and should therefore be processed, by determining whether or not the value is within the boundary of the hologram.

As previously mentioned, preferably the hardware is implemented on an FPGA or ASIC. The skilled person will recognise that although it is often convenient to implement a system which employs the scheduler to determine a value for /c, in a system where the hologram size is fixed (in at least one dimension) a calculation of A: is unnecessary. Similarly the starting values of u and v for a particular processor block could be hardwired, dispensing with this aspect of the function of the controller.

As mentioned above, in preferred applications of the hardware ED calculation system, the system is used as a co-processor for an OSPR engine, and it is therefore useful to consider the potential maximum speed of operation of embodiments of the hardware, since this relates to (potentially limits) the maximum number of OSPR subframes that may be calculated. The speed of the ED processor is a function of the operation speed (of the FPGA), kernel size and number of replicated cores. Four designs were evaluated using TVc = 4, 16, 64, and 128. Using the aforementioned FPGA the maximum clock frequency for each of these was determined by the Xilinx^® tools to be approximately 128MHz, and this was used to determine the execution time as a function of kernel size, which it was postulated dictates the maximum number of OSPR subframes that is able to processed by embodiments of the hardware ED system. Figure 8 shows the results of this procedure, plotting the maximum number of OSPR sub frames against the kernel size for different numbers of replicated processor cores, from which it can be seen that with a kernel size of K =15 providing 64 or 128 processor cores enables the computation of 8 or 16 OSPR sub frames respectively. Each processor core is implemented on the FPGA in a DSP 48 block, and the available number of these DSP 48 blocks in the example Xilinx^® FPGA ultimately determine the maximum speed/OSPR subframes.

Three different 128-processor designs were implemented, each having a different kernel size, and these were tested by binarising a 512 x 512 pixel continuous hologram resulting from an OSPR algorithm. The results were as follows:

It can be seen that with K — 15 an execution time of less than 2ms is readily achieved. The resulting binary holograms corresponding to OPSR (K= I) and ED (K = IS) were displayed on an SLM and imaged onto a projection surface using a demagnification lens assembly, as illustrated in Figure 2.

Figure 9a illustrates the K = 1 replay field, and Figure 9b the K = 15 replay field; these were determined to have measured contrast ratios of 25:1 and 100:1 respectively, showing an SNR increase by approximately a factor of 4.

Referring now to Figure 10a, this shows an example of a pipelined holographic data calculation and error diffusion system 1000 comprising a hologram data calculation system 1002, for example similar for that described above with reference to Figure 1, closely coupled to a hardware error diffusion calculation system 1004, also as described above, both under control of a pipeline controller 1006. The hologram data calculation system 1002 receives image data into an input buffer 1002a from which the data is provided to a hologram data calculation module 1002b which outputs the calculated hologram data to a hologram data memory 1002c. Preferably the hologram data memory 1002c is shared with the error diffusion calculation system 1004 as schematically shown in Figure 10a.

The error diffusion calculation system 1004 comprises a set of error diffusion processors 602 coupled to error diffusion kernel memory 606, diffused error memory 608, and under control of a scheduler 604, as well as being a coupled to the hologram data memory 1002c (which corresponds to memory 610 described above). Hologram data may be read out from the hologram data memory 1002c and provided, for example, to an SLM as illustrated in Figure 2.

The operation of the system of Figure 10a is illustrated in Figure 10b in which the successive integers refer to successive OSPR subframes. As can be seen once the pipeline has started (under control of pipeline controller 1006) one sub frame is output for each pipeline state, with a latency of one sub frame. More particularly the error diffusion engine 1004 performs an error diffusion calculation one OSPR sub frame whilst the OSPR engine 1002 calculates continuous hologram data for a subsequent subframe.

Referring next to Figure 10c, this shows a further example of a pipelined holographic data calculation and data quantization system 1020, in which like elements to those of Figure 10a are indicated by like reference numerals. In the pipelined system of Figure 10c, a quantization engine 1022 performs pipelined quantization of the hologram data in hologram data memory 1002c. The quantization engine 1022 uses one or more quantization data processors 1024 which interact with local quantization data memory 1026 and perform quantization calculations under control of a controller 1028. The quantization data memory 1026 may store, for example, intermediate results of quantization calculations, such as a median value of quantised data. In still further embodiments the pipelined quantization comprises a noise compensation process such as that illustrated in Figure 11 of WO 2007/031797 (hereby incorporated by reference). The skilled person will understand that the above described techniques may be employed with either monochrome or colour holographic image display systems, in the latter case applying the technique to each colour plane or the input/holographic image.

Applications for the above described system include, but are not limited to, the following: mobile phone; PDA; laptop; digital camera; digital video camera; games console; in-car cinema; personal navigation systems (in-car or wristwatch GPS); head- up/helmet-mounted displays for automobiles or aviation; watch; personal media player (e.g. MP3 player, personal video player); dashboard mounted display; laser light show box; personal video projector (a "video iPod (RTM)"); advertising and signage systems; computer (including desktop); and a remote control unit.

We have described embodiments of the invention in the particular context of holographic image display systems. However the skilled person will understand that embodiments of the invention are not limited in this way and have application to hologram data used for other purposes, for example in optical metrology systems and optical sensors as described in our co-pending UIC patent application GB0612882.1 filed 29 June 2006.

No doubt many other effective alternatives will occur to the skilled person. It will be understood that the invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.

Claims

CLAIMS:

1. A hardware error diffusion calculation system for performing an error diffusion quantisation of a hologram, the system comprising: an address bus and a data bus for a first, hologram memory block, said hologram memory block for storing data representing pixels of said hologram; an address bus and a data bus for a second, error memory block, said error memory block for storing an error matrix representing errors in quantising said hologram; an address bus and a data bus for a third, diffusion weights memory block, said diffusion weights memory block for storing a set of diffusion weights for said error diffusion calculation; and a plurality of error diffusion processor blocks each coupled to said address and data buses for each of said first, second and third memory blocks, each of said processor blocks being configured for sequential processing of a spatial region of said hologram, a said processor block including a pixel address generator to generate a sequence of pixel addresses for a said spatial region for processing by the processor block and an error diffusion calculation block coupled to said pixel address generator and to said address and data buses to perform an error diffusion calculation for a pixel of said hologram identified by a said pixel address; wherein spatial regions processed by said processor blocks overlap, and wherein each of said processor blocks has a different start pixel address, such that said processor blocks are able to perform error diffusion calculations with respective said error diffusion calculation blocks on different pixels of said hologram in parallel.

2. A hardware error diffusion calculation system as claimed in claim 1 further comprising a scheduler to generate a control signal for each processor block to control each said processor block to start processing in turn.

3. A hardware error diffusion calculation system as claimed in claim 1 or 2 wherein a said processor block further comprises a controller to provide said start pixel address.

4. A hardware error diffusion calculation system as claimed in claim 3 when dependent on claim 2 wherein said scheduler is configured to generate a value (k) dependent on a size of said hologram in pixels in one dimension (P;u), and wherein said controller is configured to determine said start pixel address from said value (k).

5. A hardware error diffusion calculation system as claimed in claim 4 wherein each said processor block has a processor block identifier, and wherein a said controller is configured to determine said start pixel addresses using said processor block identifier such that said spatial regions are distributed along a direction (v) orthogonal to a direction of said first dimension.

6. A hardware error diffusion calculation system as claimed in any preceding claim wherein said sequence of pixel addresses defines a generally diagonal path through a said spatial region processed by a said processor block.

7. A hardware error diffusion calculation system as claimed in any preceding claim wherein a said error diffusion calculation performed by a said error diffusion calculation block is performed over an error diffusion window, and wherein a said error diffusion calculation block comprises a memory controller coupled to said pixel address generator and to said address buses for at least said second to third memory blocks to provide a set of addresses for said second and third memory blocks to cover said error diffusion window.

8. A hardware error diffusion calculation system as claimed in claim 7 wherein a said error diffusion calculation block further comprises a multiply accumulate block coupled to said data buses for said second and third memory blocks and having an output for determining a quantised hologram pixel value for writing to said hologram memory block.

9. A hardware error diffusion calculation system as claimed in claim 8 wherein said quantisation comprises binarisation, and wherein said error diffusion calculation block further comprises a threshold block to threshold data derived from said output of said multiply accumulate block.

10. A hardware error diffusion calculation system as claimed in any claim 8 or 9 wherein said error diffusion calculation block further comprises circuitry to update said error memory block with a quantisation error value determined using said quantised hologram pixel value.

11. A hardware error diffusion calculation system as claimed in any preceding claim including said first, second and third memory blocks.

12. A hardware error diffusion calculation system as claimed in any preceding claim embodied in an FPGA or ASIC.

13. A holographic image display system including the hardware error diffusion calculation system of any preceding claim.

14. A holographic image display system as claimed in claim 13 wherein a displayed image is formed from a sequence of holographically generated spatially overlapping temporal subframes.

15. A holographic image display system as claimed in claim 14 further comprising a holographic subframe calculation system coupled to said error diffusion calculation system such that said error diffusion quantisation for one subframe is performed concurrently with hologram calculation for a subsequent subframe.

16. A holographic image display processing system, the system being configured to provide data to display an image holographically, said image being defined by displayed image data, the system comprising: a first system to generate hologram data from said image data; a second system to quantise said hologram data for display; and wherein said first and second systems are pipelined such that said hologram data quantisation is performed concurrently with said hologram data generation.

17. A holographic image display processing system as claimed in claim 16 configured to provide data to display said image using a plurality of holo graphically generated temporal subframes, said temporal subframes being displayed sequentially in time such that they are perceived as a single noise-reduced image, wherein said first system is configured to generate hologram data for a said subframe from said image data, and wherein said first and second systems are pipelined such that said hologram data quantisation for one subframe is performed concurrently with said hologram data generation for a subsequent subframe.

18. A holographic image display system as claimed in claim 16 or 17 wherein said second system comprises a hardware error diffusion calculation system.

19. A holographic image display system as claimed in any one of claims 16 to 18 wherein said first system comprises a hardware system to generate said hologram data.

20. A holographic image display system as claimed in any one of claims 16 to 19 wherein said first system comprises an OSPR engine.

21. A holographic image display system as claimed in any one of claims 16 to 20 wherein said first and second systems share common memory for storing the hologram data.

22. A holographic image display system including a holographic image display processing system as claimed in any one of claims 16 to 21.

23. A mobile phone; PDA; laptop; digital camera; digital video camera; games console; in-car cinema; personal navigation system; head-up or helmet-mounted displays for an automobile or aviation; watch; personal media player; dashboard mounted display; laser light show box; personal video projector; advertising or signage system; computer (including desktop); or remote control unit; including a holographic image display system as claimed in claim 22.