GB2307133A - Video camera image stabilisation system - Google Patents

Video camera image stabilisation system Download PDF

Info

Publication number
GB2307133A
GB2307133A GB9523190A GB9523190A GB2307133A GB 2307133 A GB2307133 A GB 2307133A GB 9523190 A GB9523190 A GB 9523190A GB 9523190 A GB9523190 A GB 9523190A GB 2307133 A GB2307133 A GB 2307133A
Authority
GB
United Kingdom
Prior art keywords
gt
lt
sep
image
means
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB9523190A
Other versions
GB9523190D0 (en
Inventor
Robert William Maclaughl Smith
Philip John Kent
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
UK Secretary of State for Defence
Original Assignee
UK Secretary of State for Defence
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by UK Secretary of State for Defence filed Critical UK Secretary of State for Defence
Priority to GB9523190A priority Critical patent/GB2307133A/en
Publication of GB9523190D0 publication Critical patent/GB9523190D0/en
Priority claimed from GB9622261A external-priority patent/GB2307371B/en
Publication of GB2307133A publication Critical patent/GB2307133A/en
Application status is Withdrawn legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/60Rotation of a whole image or part thereof
    • G06T3/606Rotation by memory addressing or mapping
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/12Video games, i.e. games using an electronically generated display having two or more dimensions involving interaction between a plurality of game devices, e.g. transmisison or distribution systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/232Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor
    • H04N5/23248Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor for stable pick-up of the scene in spite of camera body vibration
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/232Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor
    • H04N5/23248Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor for stable pick-up of the scene in spite of camera body vibration
    • H04N5/23251Motion detection
    • H04N5/23254Motion detection based on the image signal
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/232Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor
    • H04N5/23248Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor for stable pick-up of the scene in spite of camera body vibration
    • H04N5/23264Vibration or motion blur correction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N2005/2255Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscope, borescope

Abstract

An image stabilisation system provides means for counteracting a rotation of an image caused by a camera rotating about its optical axis. An image is digitised by a unit 12 and convolved with two filter kernels by two convolution units 14, 16 to enhance the vertical and horizontal components of edges in the image. A histogram of the orientations of the edges is then correlated with an edge orientation histogram of a previous image to determine a relative degree of rotation which is then used to correct the rotated image for display on a display unit 24.

Description

IMAGE STABILISATION SYSTEM This invention relates to systems for stabilising images.

There are a number of applications in which an image is conveyed remotely from a moving camera to a viewing screen, such as in pipelinelborehole inspections or in remotely controlled undersea vehicles. However, movement of the vehicle or camera also causes the image on the viewing screen to move, and if this movement is rapid or rotational then information from the scene under observation could be lost.

Furthermore, if the viewing screen image is being used to control the vehicle or camera, then such movement may cause an operator to lose control with the possible result that the vehicle and/or camera may be damaged.

It is an object of the invention to provide an image stabilisation system.

The invention provides an image stabilisation system comprising means for determining an angular distribution of scene edges in an acquired image and means for correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image and for generating an alignment correction in response thereto.

The invention has the advantage that it enables images to be stabilised to correct for rotational movements of an image acquisition device, such as a rotating camera.

A further advantage of the invention is that the image is stabilised using only information contained within the image, without the need for additional sensor data, for example data from gyroscopes or other inertial systems.

The stabilisation may be performed without assuming the presence of any specific features in the image, for example the horizon or walls, which could be used to orientate the image. All that is required is that there are features in the scene which have non-circular edges in a grey-scale representation of the image.

In an alternative embodiment, the invention provides an image stabilisation system comprising means for determining an angular distribution of scene edges in an acquired image, means for correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image, means for predicting an alignment correction for the acquired image from an analysis of previous images, and means for determining an error term from a comparison of the predicted alignment correction and an actual alignment correction required to maximise the correlation between the angular distribution of the acquired image and the angular edge distribution of a previous image. ) The invention further provides an image stabilisation system incorporating means for generating pixel image edge angles, wherein the rotational angle of this image is obtained from a comparison of the pixel image edges in a current image with the pixel image edges in a reference image.

The invention also provides a method of stabilising an image comprising the steps of (i) determining an angular distribution of scene edges in an acquired image; (ii) correlating the angular distribution of the edges in the acquired image with the edge angular distribution of a previous image; and (iii) generating an alignment correction in response to the correlation.

The invention further provides a method of stabilising an image comprising the steps of (i) determining an angular distribution of scene edges in an acquired image; (ii) correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image; (iii) predicting an alignment correction for the acquired image from an analysis of previous images; and (iv) determining an error term from a comparison of the predicted alignment correction and an actual alignment correction required to maximise the correlation between the angular distribution of the acquired image and the angular edge distribution of a previous image.

Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings in which: Figure 1 shows a schematic diagram of an image stabilisation system; Figure 2 shows a schematic diagram of an alternative image stabilisation system; Figure 3 shows a functional-level diagram of an image processing system; Figure 4 shows a diagram of a transputer module of the Figure 3 system; Figure 5 shows a diagram of a transputer array of the Figure 3 system; and Figure 6 shows an example of a video output of the Figure 5 transputer array.

Referring to Figure 1, there is shown an image stabilisation system, indicated generally by 10. The system 10 enables an image to be stabilised in situations where the image is captured by a camera (not shown) which is rotating about its optical axis.

The system 10 incorporates a digitisation unit 12. The digitisation unit 12 receives an input from the camera and generates a digitised 256 x 256 pixel output image of an acquired image. The digitisation unit 12 is an electrical circuit which incorporates a KSV3110 analogue to digital converter and a SAA1043 video synchronisation circuit.

The digitised image is input simultaneously to two convolver units 14 and 16. The convolver units 14 and 16 each comprises two 2D convolver devices manufactured by SGS Thomson and having a product code IMSA110, cascaded together. The convolver units 14 and 16 are used to carry out vertical and horizontal edgeenhancement convolutions respectively. Such convolutions are described by L.Spacek in two documents - "Thinning Image Boundaries", CSM 187, Department of Computer Science, University of Essex, Colchester 1993; and "The Vision Library Manual", OSM 199, Department of Computer Science, University of Essex, Colchester 1993.

The convolver units 14 and 16 function as follows. Any edges in the image are found where the image has spatial intensity changes, and these are enhanced by convolving with two separate filter kernels which enhance the vertical and horizontal components of the edges of the images. The vertical component kernel is shown below: <img class="EMIRef" id="026738286-00040001" />

<tb> <SEP> 5 <SEP> " <SEP> 10 <SEP> 8 <SEP> 5 <tb> <SEP> 2 <SEP> 5 <SEP> 10 <SEP> 5 <SEP> 2 <tb> <SEP> o <SEP> 0 <SEP> 0 <SEP> 0 <SEP> <SEP> O <SEP> <tb> -2 <SEP> -5 <SEP> -10 <SEP> -5 <SEP> -2 <tb> -5 <SEP> -8 <SEP> -10 <SEP> -8 <SEP> -8 <tb> and the horizontal component kernel is:: <img class="EMIRef" id="026738286-00040002" />

<tb> -5 <SEP> -2 <SEP> 0 <SEP> 2 <SEP> 5 <tb> -8 <SEP> -5 <SEP> 0 <SEP> 5 <SEP> 8 <tb> -10 <SEP> -10 <SEP> 0 <SEP> t <SEP> 10 <SEP> 10 <tb> -8 <SEP> -5 <SEP> 0 <SEP> 5 <SEP> 8 <tb> -5 <SEP> -2 <SEP> 0 <SEP> 2 <SEP> 5 <tb> The exact nature of the kemels is not relevant to the overall system. Other edge enhancement kemels may be used in place of the above kemels.

The application of the vertical and horizontal component filters by convolver units 14 and 16 respectively results in a vertical gradient image output from convolver unit 14 and a horizontal gradient image output from convolver unit 16; the gradient image outputs are normalised in the respective convolver units and are in the form of an 8-bit value. A gradient image is an image in which the pixel values are the vertical or horizontal component of the gradient of the pixel intensity values of the original input image. The original greyscale image is differentiated in the horizontal direction to obtain the horizontal gradient image and differentiated in the vertical direction to obtain the vertical gradient image. The gradient images are then input to a polar conversion unit 18. The conversion unit 18 incorporates a look-up table (LUT) which is pre-programmed with the necessary transforms to convert from Cartesian coordinates to polar co-ordinates as given in Equations 1 and 2. The LUT comprises four HM62256 RAM devices. The two component images are then combined by the conversion unit 18 on a pixel-by-pixel basis to generate two images which contain the magnitude and angle of the edges in the image. The magnitude, M, is given by: <img class="EMIRef" id="026738286-00050001" />

where x and y are the horizontal and vertical components of the edge intensities at a particular pixel.

The angle of the edges in the image (A) is given by: <img class="EMIRef" id="026738286-00050002" />

<tb> A <SEP> = <SEP> tan-1 <SEP> (Y) <SEP> <SEP> (2) <tb> where x and y have the same meaning as for equation (1), and tan~ is the arctangent function resolved over a range of -7t to +s.

The two images which contain the magnitude (M) and angle (A) of the edges in the image are output from unit 18 and input to a transputer array 20. The transputer array 20 then generates a histogram over the whole image, to provide a magnitude weighted frequency of occurrence of each possible angle value. The magnitude (M) generated by the unit 18 is normalised and output as an 8-bit binary number; the edge angle (A) generated by unit 18 is also an 8-bit binary number, giving 256 numbers to represent either the magnitude or the angle.

Rather than adding 1 to the histogram for each pixel with a particular angle, the pixel's edge magnitude is added to the histogram. This causes strong edges to make a greater contribution to the histogram than weaker ones. Very weak edges do not make a contribution, because the transputers in the array 20 impose a lower threshold on the magnitude values. This lower threshold is at a pixel edge magnitude of twenty. Pixel edge magnitudes which are less than twenty do not contribute to the histogram.

The histogram forms a 256 element vector, which describes the edge angle content of the image. The histogram wraps around" from the angle code 255 to the angle code 0.

The transputer array 20 also smoothes the histogram by convolving it with a 25element Gaussian kernel. This removes noise in the histogram, and compensates for the fact that the angular resolution of the histogram is limited. The values in the Gaussian kernel are as follows: <img class="EMIRef" id="026738286-00060001" />

<tb> 3 <SEP> 6 <SEP> 11 <SEP> 20 <SEP> 35 <SEP> 55 <SEP> 83 <SEP> 117 <SEP> 155 <SEP> 192 <SEP> 225 <SEP> 247 <SEP> 255 <SEP> 247 <SEP> 225 <SEP> 192 <SEP> 155 <SEP> 117 <SEP> 83 <SEP> 55 <SEP> 35 <SEP> 20 <SEP> 11 <SEP> 6 <SEP> 3 <tb> The 25-element Gaussian kernel is aligned with the first 25 elements of the 256element histogram. Corresponding elements of the histogram and the kernel are multiplied together and the products summed. The resulting sum is then the smoothed histogram entry which is aligned with the centre of the kemel.The kernel is then moved along one element and the process repeated to give a second convolved histogram entry adjacent the first. This process is repeated until all 256 elements of the smoothed histogram are obtained, wrapping round when the end of the original 256 element histogram is reached, i.e. treating it as a circular structure with no start or finish.

Because the enhancement convolution kernels do not have sufficient resolution to generate all possible angle codes once the orthogonal gradient components have been converted to magnitude and angle, zeros are present in the angle histogram.

These zeros introduce unwanted high angular frequency information into the histogram. The histogram is also inherently noisy due to noise in the input image; the noise being assumed to have a Gaussian distribution. The convolution with a Gaussian kernel reduces the noise and at the same time acts as a low pass filter, attenuating the high angular frequency information.

The smoothed histogram generated by the array 20 is then correlated with a reference histogram also stored in the array 20. The reference histogram is a histogram which was obtained from a previous reference image. The current histogram is then shifted over a range of angles, centred on zero, and at each shift it is correlated with the reference histogram. The shift angle which produces the maximum correlation value is taken as the angular displacement between the current image and the reference image.

If the shift angle is not zero then there is sufficient angular deviation between the current image and the reference image to update the reference histogram. When this occurs the current histogram becomes the new reference histogram. If the shift angle is zero (all arithmetic is integer, so the shift angle is quantified into steps such that 256 steps represent 360 degrees of rotation)0 then the current and reference frames do not differ sufficiently to update the reference histogram. When this occurs the current reference frame remains as the reference frame. This prevents a very slow rotation in the image, i.e. a rotation where the inter-frame difference is less that 1/256 of a revolution, being missed.

The range of angles over which the correlation is performed limits the rotation speed which can be accommodated by the system 10. A larger range requires more computational effort, but results in a higher maximum rotational speed.

The angular shift with respect to the reference image is accumulated with angular shifts from previous correlations, back to some arbitrary reference time. The accumulation is performed modulo 256, to give the fraction of a complete revolution by which the current image must be rotated to align it with the original reference from the beginning of the accumulation process.

This angle is then used to rotate the current image on a pixel by pixel basis to produce an output image which is rotationally aligned with the original reference image. This is achieved by a rotator unit 22 incorporating two TMC2302 image manipulation sequencer devices using the following transform equations: x = x cosO + y sinO and y' = -x sinO + y cosO where (x, y) are the co-ordinates of a pixel in the output image and (x', y') are calculated pixel co-ordinates from where the pixel value of the input image will be taken for the pixel in the output image and e is the rotational angle that is required to bring the image into alignment. The co-ordinates (x, y) and (x', y') are offsets from the centre of the image.Where the calculated (x', y') co-ordinates do not fall exactly on a pixel in the input image, the nearest neighbouring pixel to (x', y') in the input image is used to give the value for the output pixel at (x, y).

For square input and output images some pixels in the output image will map to pixel locations outside the input image. To produce a more aesthetically pleasing output a circular image could be produced by applying a circular mask to the output image.

The diameter of the mask would equal the width or height of the image, and would only generate the pixels that fall within the mask. The image output from the rotator unit 22 is displayed on a display 24.

Figure 2 shows a further image stabilisation system 110. The system 110 incorporates components similar to the Figure 1 having reference numerals increased by 100 compared with the reference numerals of the similar components in the system 10. The system 110 includes a transputer array 130 which as well as performing the operations of the transputer array 20 of Figure 1 also performs a Kalman filter operation. A Kalman filter is an algorithm for predicting the next value of a data sequence based on analysis of the past history of the sequence and knowledge of the characteristics of a system producing the sequence. The Kalman filter is used to predict the angular displacement of the next image. This predicted angular displacement is passed to a rotator unit 122. The rotator unit 122 uses the predicted angular displacement to correct data of a new image received directly from a digitisation unit 112. The actual alignment of the new image would then be compared with the predicted alignment. The actual angular displacement would be measured and by the array 130 and used to produce an error term to improve the prediction for a subsequent image. This arrangement reduces the latency with which the system is able to display the corrected images compared with the system 10.

In an alternative system (not shown), a video signal from a camera may be routed prior to digitisation to a display device capable of displaying an image with a variable rotational displacement. Such a device might be constructed using an oscilloscope with an x-y input faciiity together with an intensity modulation input. The x and y inputs are driven by a digital to analogue converter controlled by address generators, constructed from programmable logic devices, which provide the necessary address sequences to give the desired rotational offset. The intensity modulation input is driven by the video signal. The desired rotational offset is determined as previously described in relation to the Figure 2 system. This system would display the corrected images with zero latency, which is of significant benefit for a control-loop system.

It is possible that a system of the invention might be subject to periodic drift in the angular rotations. This may be overcome by providing a system with a means for manually inputting an angular offset, for example by means of a thumbwheel. This manually input offset is then used to provide an initial offset to the rotation.

Whiist the previously described embodiments provide a means of stabilising an image in situations where a system camera is rotating about its optical axis, the principles behind the described image stabilisation systems may be extended to situations where an axis of rotation is parallel to but not coincident with a camera's optical axis. The algorithms required for such a stabilisation system would have to take into account the separation between the axis of rotation and the optical axis.

A component level block diagram of an image processing system 200 is shown in Figure 3. The system 200 may implement the operation of the system 10. The system 200 contains two distinct types of processor. To the left of the figure are a number of low-level pixel processing modules, referred to as digital signal processing (DSP) modules. The DSP modules are an acquisition circuit module 210, colour and monochrome display modules 212 and 214 respectively, a histogramming module 218, an FFT module 220, a robust edge detector module 222, a dual lookup table module 224 and a pipeline 226 of 2-dimensional convolvers 228. The DSP modules each have relatively limited functionality but are designed to perform common image processing tasks which are computationally very intensive such as convolution, morphology and histogramming.They are each designed to perform their specific function with a minimum of latency from input to output, to minimise the overall latency of processor pipelines. This is important where the image processing system is part of a larger closed-loop application.

To the right of Figure 3 is a parallel array 230 of microprocessor-based image processing modules 232. High level tasks of an application are implemented in the array 230. The processing modules 232 each contain a transputer, or a combination of a transputer and a PowerPC microprocessor. The processing modules may be programmed in a number of high-level languages, such as C and OCCAM.

The processing modules 232 are interconnected by a software reconfigurable, digital video crossbar switching matrix 234. New modules may be added to the crossbar, which can accommodate up to 64 inputs and 64 outputs.

All processing modules, including the parallel processing modules, use a common video input and output protocol. This is a synchronous protocol, in which the unit of transfer is an entire image of 512 x 512 pixels, with a gap of 8 pixels between rows, to accommodate the write transfer operations in video RAM (VRAM) devices (not shown) which are used as framestores in the parallel processor array 230. A 7.5 MHz pixel clock is centrally generated, and distributed to all processors. This gives a frame transfer time of 35.5 ms, well within a nominal 40 ms frame period of the system 200. The spare time within the frame period is used to absorb the latency of DSP modules. A latency of up to 64 rows of video may be accommodated within the spare 4.5 ms, allowing very complex DSP pipelines to be constructed.

Video input is from a 25 Hz CCIR-624 monochrome source (for example, a video camera or a VCR). It is digitised to a resolution of 512 rows and 512 columns per frame, using 8 bits per pixel. There are two video displays; one is a three-colour component (RGB) display module 212, which uses a Brooktree RAMDAC to provide pseudo colour, 4-bit graphical overlay and cursor; the other display is a monochrome display module 214.

The pipeline 226 contains four convolvers 228, each capable of convolving a 3 row by 7 column kernel with an image at pixel rate, each convolver 228 being based on a SGS Thomson A110 module. The four convolvers 228 may be operated separately, giving four different convolution operations, or may be cascaded together to give larger kernel sizes. Multiple modules may be cascaded together to increase the kernel size further.

The dual lookup table module 224 is based on a bank of SRAM, 64K by 16 bits. Two eight bit video inputs are concatenated to form the 16 bit address for the memory, and the 16 bit data produces two eight bit video outputs. By initialising the SRAM appropriately, any two functions of two input variables can be calculated in real-time.

This has numerous applications such as rectangular to polar vector conversion, polar to rectangular vector conversion, arithmetic and logic operations, and thresholding operations.

The histogramming module 218 computes a grey scale histogram within up to 1024 separate regions of the input image in real-time. Histogram regions may be defined either by simple geometric partitioning of the image, or by means of a second video input (not shown) (allowing up to 256 arbitrarily shaped regions to be defined). A third video input (not shown) provides the value to accumulate to the histogram for each image pixel. This input passes through a transformation LUT which allows arbitrary weighting functions to be applied to the histograms. A microprocessor (not shown) on the module can read and analyse the histograms. For simple global histograms, the image can then be transformed using a LUT. This allows such operations as adaptive thresholding and contrast enhancement to be carried out in real-time.

The FFT module 220, based on a Sharp LH9124 FFT DSP is capable of performing a 2 dimensional FFT or IFFT on a 512 x 512 image in under 40 ms. The input and output framestores are double buffered to allow a throughput of 25 frames per second. In addition, the module contains two EPROM LUTs to enable the real time conversion between real / imaginary and magnitude / angle representation of the complex results of the transform.

The robust edge detector module 222 enhances the edges within a grey-scale image using two programmable 3 by 3 convolutions (e.g. a Sobel), then it thins the edges to single pixel lines using non-maximal suppression. This operation is performed at pixei-rate using programmable logic hardware. The image is then hysteresis thresholded.

The processing modules are interconnected by the video crossbar switching matrix 234. This is a fully non-blocking crossbar switch, in which any input may be connected to any one or more outputs. The video crossbar switching matrix 234 may be reconfigured in software at the start of each video frame. This allows the DSP modules to be interconnected in pipelines and parallel pipelines, and allows the interconnections to be dynamically changed during algorithm execution. The video crossbar switching matrix 234 contains 64 input and 64 output ports, each of which is nine bits wide, carrying eight bits of video data and a video synchronisation signal.

The video crossbar switching matrix 234 is implemented using nine 64 by 64 single bit crossbar integrated circuits from LSI Logic, operating in a bit-sliced manner. They are configured using a transputer 236, using a set of memory mapped registers. The configuration registers are double buffered, and are clocked by the video frame sync signal. This means that the transputer has an entire frame period in which to initialise a new configuration, which then takes effect instantaneously at the next frame sync.

This simplifies software, since the transputer does not have to be precise about when it changes the configuration registers.

The parallel processing array 230 consists of twenty identical image processing modules 232. Two variants of the image processing modules 232 are currently in use. The original version of the module contains a floating point (T805) transputer.

The new version of the module is a hybrid of a T805 transputer with a Motorola PowerPC (either a PPC604 or a PPC603e). This is discussed in more detail below.

A block diagram of an image processing module 232 containing a transputer 250 is shown in Figure 4. The module 232 contains 4 MByte of DRAM 252, and 2 MByte of Video RAM (VRAM) 254. The VRAM 254 is used as framestore memory. It is split into two 1 MByte banks 256, each containing four 512 x 512 byte framestores. Each framestore bank 256 has a video input port, a video output port, and a map output port. Each output port has an output enable signal. The map port is a copy of the video output port, but connects to a different bus on an array backplane. and is used when the module is operating as the array controller.

The video input/output operations are controlled by a VRAM controller 258 which is a programmable logic device. In order to perform an input or output operation, the transputer 250 writes to a control register. It is then free to continue processing with virtually continuous access to memory. The VRAM controller 258 steals a single memory cycle every four rows of video in order to perform a read or write transfer cycle within the VRAM 254. The VRAM banks 256 are independent of each other, so both banks may be transferring video data simultaneously.

The modules 232 are integrated together into the array 230 as shown in Figure 5.

There are twenty identical processing modules 232 each as shown in Figure 4. The input buffer broadcasts video input to all modules using two buses IN A and IN B.

The MAP OE A and B buses are output enables that determine which module will act as the controller for bus A and B. The MAP A and B buses are the controller's data which is decoded in the output decoder to enable the video output of the appropriate module using the OE A and B buses. The enabled module then outputs its data onto the OUT A or B buses, which the output buffer outputs onto the Video Out A and B buses. All transputer links go to the link crossbar switch module, which, in addition to interconnecting the links, provides eight connections into the DSP section of the system 200 architecture.

The video ports of all modules are connected to common video buses, an input and an output bus for each bank. Incoming video frames from the video crossbar switching matrix 234 are broadcast to all modules simultaneously. The software running on the array 230 determines which modules will acquire the frame. Frames are acquired in their entirety by the modules 232. The software determines which patch of the frame is to be processed by each module 232.

This approach has several benefits: (1) it means that the algorithm may partition the processing of the frame over the array in an arbitrary manner, since all of the data is present on all processors; (2) in non-deterministic applications which require a work farm approach to provide dynamic load balancing, the farm controller may simply pass tokens to the workers to identify patches within the frame, which minimises the communications overheads inherent in such an application; (3) although a processor is mainly concerned with a defined patch of the image, it may occasionally require information about pixels in other parts of the image (for example at the borders of its patch), this information is available locally, so no interprocessor communication is required.

When processing of the image is completed, partial results will be contained in the framestores of various modules 232 within the array 230. If it is required to transfer the results back out of the array, for example, to display them, then it is necessary to stitch the patches together into a single coherent image. This is done as follows.

One of the modules 232 is selected to be the controller for the array. This module 232 does not take part in the image processing operation; instead, it constructs within a framestore, a module address map which defines for each pixel in the result image the source module 232 for that pixel. All modules 232, including the controller, output their results simultaneously. The controller's map goes via a separate output bus, called the Map bus, to a decoder, which then generates an output enable signal which enables the output of the appropriate module. This process repeats at pixel rate, so the Output bus contains a coherent frame of data. A simplified example of this process is shown in Figure 6. The example assumes four worker modules, and shows how the map reflects the partitioning of the results across the workers.

The transputer links are connected to a crosspoint switch array. This may be configured in software, to produce any topology that can be produced with a 4-valent network. The links are generally used for high-level messages, which typically have low data rates, well below the 20 Mbit sr that the links can support. There is generally no need to pass video data between processors using the links, as the data is already broadcast to all processors.

The computational power of the T805 is modest by comparison with modern microprocessors (a 30 MHz T805 is capable of 30 MIPs and 4.3 MFLOPs). To improve the performance of the parallel processing array, the transputer modules may be replaced by hybrid modules consisting of a T805 transputer and a PowerPC processor (either a 604 or a 603e). The modules are commercial boards from Transtech Parallel Systems Ltd., with a special interface based on VRAMs to give real-time access to the video data. The transputer is used as a communications processor, interfacing the PowerPC processor to the rest of the system in such a way that the whole module looks like a transputer, but with considerably improved performance. Code written in C for the transputer array may be ported to the PowerPC processor modules with only minor changes.A factor of between 10 and 20 speed up has been observed on a range of image processing applications comparing a 66 MHz PowerPC 604 against a 30 MHz T805.

The system 200 is housed in a 19 inch double Eurocard (6U) rack. The DSP modules have either a single or double Eurocard (220 mm by 100 or 233 mm) form factor, and connect to the video crossbar switching matrix 234 and other subsystems by means of ribbon cables with single-in-line header sockets plugging onto the back of the 96way backplane connectors. This makes it easy to customise a system for a particular application, or to add a new module to an existing system. The parallel processing modules 232 are interconnected by a backplane, which broadcasts the video data to all modules, and implements the pixel switching technique described earlier for recombining the results from the modules.

Control of the system is via a single transputer link and transputer control signals.

This may be directly connected to interface cards within a variety of hosts, including PC or SUN workstations, or by means of a transputer ethemet gateway to an ethernet network. The link has a bandwidth of 20 Mbit s~', which is sufficient for code booting, debugging and for user interaction with the system.

Algorithm development is carried out on a host computer, using cross-compilers to generate the executable code, which is then downloaded to the system 200 and run.

Standard toolsets from SGS-Thomson, or from 3L support ANSI C, C++, OCCAM and FORTRAN. The DSP modules within the system 200 are viewed by the programmer as transputers which have special memory mapped peripheral devices.

Framestores are viewed as two-dimensional arrays which are mapped into fixed locations in memory. Libraries of drivers for the DSP hardware have been created and may be used by the application programmer. Code may be ported from other sources to the system 200 fairly easily, especially if the code is written in ANSI C.

Code debugging is carried out using standard debugging tools. The Inquest toolset from SGS Thomson provides full symbolic debugging facilities on all processors within the network simultaneously, and also provides network profiling tools to evaluate how efficiently the algorithm has been partitioned across the transputer array. Tools for the PowerPC offer similar faciiities. An ANSI C compiler is used on the host workstation to compile object files, and a library module is called by the user's program to boot this onto the PowerPC processor at runtime. Symbolic debugging of the PowerPCs is also available, concurrently with debugging of the transputer processors.

A common problem with real-time parallel processing systems is processor synchronisation. This is handled in the system 200 as follows. Synchronisation to the video data is carried out by connecting the event (interrupt) input of all transputers in the system to the video framesync. An event handler process on each transputer can wait for the next framesync, or keep a count of framesync pulses, to synchronise the user's code. Synchronisation between processors may be carried out explicitly, by message passing between the processors, or implicitly, by waiting for framesync events.

The system has been designed as a generic architecture, for developing a wide range of real-time. It may be used for real-time correction of imagery from a rotationally unstable camera.

Claims (22)

1. An image stabilisation system comprising means for determining an angular distribution of scene edges in an acquired image and means for correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image and for generating an alignment correction in response thereto.
2. An image stabilisation system according to' Claim 1 wherein the means for determining the angular distribution of scene edges comprises means for performing edge enhancement convolutions and means for generating a magnitude weighted frequency of occurrence histogram over a plurality of angles.
3. An image stabilisation system according to Claim 2 wherein the means for performing the edge enhancement convolutions comprises means for generating horizontal and vertical gradient images from the acquired image.
4. An image stabilisation means according to Claim 3 wherein the means for generating horizontal and vertical gradient images is arranged to convolve the acquired image with vertical and horizontal filter kernels.
5. An image stabilisation system according to Claim 4 wherein the horizontal and vertical filter kernels are: <img class="EMIRef" id="026738287-00170001" />
<tb> <SEP> 5 <SEP> 8 <SEP> 10 <SEP> 8 <SEP> 5 <tb> <SEP> 2 <SEP> 5 <SEP> 10 <SEP> 5 <SEP> 2 <tb> <SEP> O <SEP> O <SEP> O <SEP> O <SEP> O <SEP> <tb> <SEP> -2 <SEP> -5 <SEP> -10 <SEP> -5 <SEP> -2 <tb> <SEP> -5 <SEP> -8 <SEP> <SEP> -10 <SEP> -8 <SEP> -8 <SEP> <tb> and <tb> <img class="EMIRef" id="026738287-00180001" />
<tb> 5 <SEP> -2 <SEP> 1 <SEP> 0 <SEP> 2 <SEP> 5 <tb> -8 <SEP> -5 <SEP> I <SEP> 5 <SEP> 8 <SEP> <tb> -10 <SEP> -10 <SEP> 0 <SEP> 10 <SEP> 10 <tb> <SEP> -8 <SEP> -5 <SEP> 0 <SEP> 5 <SEP> 8 <tb> -5 <SEP> -2 <SEP> 0 <SEP> 2 <SEP> 5 <tb>
6.An image stabilisation system according to any one of Claims 2 to 5 wherein the system includes means for performing a smoothing convolution on the histogram.
7. An image stabilisation system according to Claim 6 wherein the means for performing the smoothing convolution is arranged to convolve the histogram with a Gaussian kernel.
8. An image stabilisation system comprising means for determining an angular distribution of scene edges in an acquired image, means for correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image, means for predicting an alignment correction for the acquired image from an analysis of previous images, and means for determining an error term from a comparison of the predicted alignment correction and an actual alignment correction required to maximise the correlation between the angular distribution of the acquired image and the angular edge distribution of a previous image.
9. An image stabilisation system according to Claim 8 wherein the means for predicting the alignment correction comprises means for operating a Kalman filter algorithm.
10. An image stabilisation system according to any previous claim wherein the system includes means for enabling a system operator to correct a periodic drift in a stabilised image.
11. An image stabilisation system incorporating means for generating pixel image edge angles, wherein the rotational angle of this image is obtained from a comparison of the pixel image edges in a current image with the pixel image edges in a reference image.
12. An image stabilisation system according to Claim 11 wherein the current image is arranged to be used as the reference image after the rotational angle is obtained.
13. An image stabilisation system according to Claim 11 or Claim 12 wherein the system includes means for enhancing spatial intensity changes in the image.
14. An image stabilisation system according to Claim 13 wherein the enhancing means incorporates separate vertical and horizontal component filters.
15. An image stabilisation system according to any one of Claims 11 to 14, wherein the representation of each detached edge angle in an image is determined by the magnitude of the pixels contributing to that edge angle.
16. An image stabilisation system according to any one of Claims 11 to 15 wherein the system incorporates a circular mask to generate a circular image.
17. An image stabilisation system according to any one of Claims 11 to 16 wherein the system includes means for predicting the angular rotation of an image, the prediction being based on the rotation of previous images.
18. An image stabilisation system according to Claim 17 wherein the predicting means includes a Kalman filter.
19. An image stabilisation system according to any one of Claims 11 to 18 wherein the system incorporates means for correcting drift in image orientation.
20. An image stabilisation system.
21. A method of stabilising an image comprising the steps of (i) determining an angular distribution of scene edges in an acquired image; (ii) correlating the angular distribution of the edges in the acquired image with the edge angular distribution of a previous image; and (iii) generating an alignment correction in response to the correlation.
22. A method of stabilising an image comprising.the steps of (i) determining an angular distribution of scene edges in an acquired image; (ii) correlating the edge angular distribution of the acquired image with the edge angular distribution of a previous image; (iii) predicting an alignment correction for the acquired image from an analysis of previous images; and (iv) determining an error term from a comparison of the predicted alignment correction and an actual alignment correction required to maximise the correlation between the angular distribution of the acquired image and the angular edge distribution of a previous image.
GB9523190A 1995-11-13 1995-11-13 Video camera image stabilisation system Withdrawn GB2307133A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
GB9523190A GB2307133A (en) 1995-11-13 1995-11-13 Video camera image stabilisation system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB9523190A GB2307133A (en) 1995-11-13 1995-11-13 Video camera image stabilisation system
GB9622261A GB2307371B (en) 1995-11-13 1996-10-25 Image stabilisation system

Publications (2)

Publication Number Publication Date
GB9523190D0 GB9523190D0 (en) 1996-01-17
GB2307133A true GB2307133A (en) 1997-05-14

Family

ID=10783805

Family Applications (1)

Application Number Title Priority Date Filing Date
GB9523190A Withdrawn GB2307133A (en) 1995-11-13 1995-11-13 Video camera image stabilisation system

Country Status (1)

Country Link
GB (1) GB2307133A (en)

Cited By (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1071285A1 (en) * 1999-07-19 2001-01-24 Texas Instruments France Vertical compensation in a moving camera
WO2003088147A1 (en) * 2002-04-16 2003-10-23 Koninklijke Philips Electronics N.V. Image rotation correction for video or photographic equipment
EP1465428A1 (en) * 2003-04-02 2004-10-06 Toyota Jidosha Kabushiki Kaisha Vehicular image display apparatus and vehicular image display method
US20090122866A1 (en) * 2004-10-22 2009-05-14 Greenparrotpictures Limited Dominant motion estimation for image sequence processing
US7716008B2 (en) 2007-01-19 2010-05-11 Nintendo Co., Ltd. Acceleration data processing program, and storage medium, and acceleration data processing apparatus for use with the same
US7774155B2 (en) 2006-03-10 2010-08-10 Nintendo Co., Ltd. Accelerometer-based controller
US7786976B2 (en) 2006-03-09 2010-08-31 Nintendo Co., Ltd. Coordinate calculating apparatus and coordinate calculating program
US7877224B2 (en) 2006-03-28 2011-01-25 Nintendo Co, Ltd. Inclination calculation apparatus and inclination calculation program, and game apparatus and game program
US7927216B2 (en) 2005-09-15 2011-04-19 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
US7931535B2 (en) 2005-08-22 2011-04-26 Nintendo Co., Ltd. Game operating device
US7942745B2 (en) 2005-08-22 2011-05-17 Nintendo Co., Ltd. Game operating device
US8072424B2 (en) 2004-04-30 2011-12-06 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US8089458B2 (en) 2000-02-22 2012-01-03 Creative Kingdoms, Llc Toy devices and methods for providing an interactive play experience
US8157651B2 (en) 2005-09-12 2012-04-17 Nintendo Co., Ltd. Information processing program
US8267786B2 (en) 2005-08-24 2012-09-18 Nintendo Co., Ltd. Game controller and game system
US8308563B2 (en) 2005-08-30 2012-11-13 Nintendo Co., Ltd. Game system and storage medium having game program stored thereon
US8313379B2 (en) 2005-08-22 2012-11-20 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
US8409003B2 (en) 2005-08-24 2013-04-02 Nintendo Co., Ltd. Game controller and game system
US8475275B2 (en) 2000-02-22 2013-07-02 Creative Kingdoms, Llc Interactive toys and games connecting physical and virtual play environments
US8608535B2 (en) 2002-04-05 2013-12-17 Mq Gaming, Llc Systems and methods for providing an interactive game
US8629836B2 (en) 2004-04-30 2014-01-14 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US8702515B2 (en) 2002-04-05 2014-04-22 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US8708821B2 (en) 2000-02-22 2014-04-29 Creative Kingdoms, Llc Systems and methods for providing interactive game play
US8753165B2 (en) 2000-10-20 2014-06-17 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US8907889B2 (en) 2005-01-12 2014-12-09 Thinkoptics, Inc. Handheld vision based absolute pointing system
US8913003B2 (en) 2006-07-17 2014-12-16 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer using a projection marker system
US9176598B2 (en) 2007-05-08 2015-11-03 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer with improved performance
US9186585B2 (en) 1999-02-26 2015-11-17 Mq Gaming, Llc Multi-platform gaming systems and methods
US9261978B2 (en) 2004-04-30 2016-02-16 Hillcrest Laboratories, Inc. 3D pointing devices and methods
US9446319B2 (en) 2003-03-25 2016-09-20 Mq Gaming, Llc Interactive gaming toy
US10159897B2 (en) 2004-11-23 2018-12-25 Idhl Holdings, Inc. Semantic gaming and application transformation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2074806A (en) * 1980-04-10 1981-11-04 Erlebach Eng Ltd Video lock-on system
EP0079195A2 (en) * 1981-11-10 1983-05-18 Cbs Inc System for stabilizing television picture
WO1990000334A1 (en) * 1988-07-01 1990-01-11 Plessey Overseas Limited Improvements in or relating to image stabilisation
GB2275842A (en) * 1992-12-21 1994-09-07 Gold Star Co Automatic image stabilizing system for a camcorder

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2074806A (en) * 1980-04-10 1981-11-04 Erlebach Eng Ltd Video lock-on system
EP0079195A2 (en) * 1981-11-10 1983-05-18 Cbs Inc System for stabilizing television picture
WO1990000334A1 (en) * 1988-07-01 1990-01-11 Plessey Overseas Limited Improvements in or relating to image stabilisation
GB2275842A (en) * 1992-12-21 1994-09-07 Gold Star Co Automatic image stabilizing system for a camcorder

Cited By (97)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9861887B1 (en) 1999-02-26 2018-01-09 Mq Gaming, Llc Multi-platform gaming systems and methods
US9731194B2 (en) 1999-02-26 2017-08-15 Mq Gaming, Llc Multi-platform gaming systems and methods
US9186585B2 (en) 1999-02-26 2015-11-17 Mq Gaming, Llc Multi-platform gaming systems and methods
US9468854B2 (en) 1999-02-26 2016-10-18 Mq Gaming, Llc Multi-platform gaming systems and methods
US6781623B1 (en) 1999-07-19 2004-08-24 Texas Instruments Incorporated Vertical compensation in a moving camera
EP1071285A1 (en) * 1999-07-19 2001-01-24 Texas Instruments France Vertical compensation in a moving camera
US8491389B2 (en) 2000-02-22 2013-07-23 Creative Kingdoms, Llc. Motion-sensitive input device and interactive gaming system
US8790180B2 (en) 2000-02-22 2014-07-29 Creative Kingdoms, Llc Interactive game and associated wireless toy
US8814688B2 (en) 2000-02-22 2014-08-26 Creative Kingdoms, Llc Customizable toy for playing a wireless interactive game having both physical and virtual elements
US8708821B2 (en) 2000-02-22 2014-04-29 Creative Kingdoms, Llc Systems and methods for providing interactive game play
US8368648B2 (en) 2000-02-22 2013-02-05 Creative Kingdoms, Llc Portable interactive toy with radio frequency tracking device
US9579568B2 (en) 2000-02-22 2017-02-28 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US10188953B2 (en) 2000-02-22 2019-01-29 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US9814973B2 (en) 2000-02-22 2017-11-14 Mq Gaming, Llc Interactive entertainment system
US9149717B2 (en) 2000-02-22 2015-10-06 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US8089458B2 (en) 2000-02-22 2012-01-03 Creative Kingdoms, Llc Toy devices and methods for providing an interactive play experience
US8475275B2 (en) 2000-02-22 2013-07-02 Creative Kingdoms, Llc Interactive toys and games connecting physical and virtual play environments
US8164567B1 (en) 2000-02-22 2012-04-24 Creative Kingdoms, Llc Motion-sensitive game controller with optional display screen
US8169406B2 (en) 2000-02-22 2012-05-01 Creative Kingdoms, Llc Motion-sensitive wand controller for a game
US8184097B1 (en) 2000-02-22 2012-05-22 Creative Kingdoms, Llc Interactive gaming system and method using motion-sensitive input device
US9713766B2 (en) 2000-02-22 2017-07-25 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US9474962B2 (en) 2000-02-22 2016-10-25 Mq Gaming, Llc Interactive entertainment system
US8686579B2 (en) 2000-02-22 2014-04-01 Creative Kingdoms, Llc Dual-range wireless controller
US9931578B2 (en) 2000-10-20 2018-04-03 Mq Gaming, Llc Toy incorporating RFID tag
US8961260B2 (en) 2000-10-20 2015-02-24 Mq Gaming, Llc Toy incorporating RFID tracking device
US9480929B2 (en) 2000-10-20 2016-11-01 Mq Gaming, Llc Toy incorporating RFID tag
US8753165B2 (en) 2000-10-20 2014-06-17 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US9320976B2 (en) 2000-10-20 2016-04-26 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US9737797B2 (en) 2001-02-22 2017-08-22 Mq Gaming, Llc Wireless entertainment device, system, and method
US9393491B2 (en) 2001-02-22 2016-07-19 Mq Gaming, Llc Wireless entertainment device, system, and method
US8711094B2 (en) 2001-02-22 2014-04-29 Creative Kingdoms, Llc Portable gaming device and gaming system combining both physical and virtual play elements
US8913011B2 (en) 2001-02-22 2014-12-16 Creative Kingdoms, Llc Wireless entertainment device, system, and method
US9162148B2 (en) 2001-02-22 2015-10-20 Mq Gaming, Llc Wireless entertainment device, system, and method
US8384668B2 (en) 2001-02-22 2013-02-26 Creative Kingdoms, Llc Portable gaming device and gaming system combining both physical and virtual play elements
US10179283B2 (en) 2001-02-22 2019-01-15 Mq Gaming, Llc Wireless entertainment device, system, and method
US8702515B2 (en) 2002-04-05 2014-04-22 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US9616334B2 (en) 2002-04-05 2017-04-11 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US9463380B2 (en) 2002-04-05 2016-10-11 Mq Gaming, Llc System and method for playing an interactive game
US8827810B2 (en) 2002-04-05 2014-09-09 Mq Gaming, Llc Methods for providing interactive entertainment
US9162149B2 (en) 2002-04-05 2015-10-20 Mq Gaming, Llc Interactive entertainment systems and methods
US10010790B2 (en) 2002-04-05 2018-07-03 Mq Gaming, Llc System and method for playing an interactive game
US8608535B2 (en) 2002-04-05 2013-12-17 Mq Gaming, Llc Systems and methods for providing an interactive game
US9272206B2 (en) 2002-04-05 2016-03-01 Mq Gaming, Llc System and method for playing an interactive game
WO2003088147A1 (en) * 2002-04-16 2003-10-23 Koninklijke Philips Electronics N.V. Image rotation correction for video or photographic equipment
US9039533B2 (en) 2003-03-25 2015-05-26 Creative Kingdoms, Llc Wireless interactive game having both physical and virtual elements
US10022624B2 (en) 2003-03-25 2018-07-17 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US9993724B2 (en) 2003-03-25 2018-06-12 Mq Gaming, Llc Interactive gaming toy
US9707478B2 (en) 2003-03-25 2017-07-18 Mq Gaming, Llc Motion-sensitive controller and associated gaming applications
US8373659B2 (en) 2003-03-25 2013-02-12 Creative Kingdoms, Llc Wirelessly-powered toy for gaming
US9770652B2 (en) 2003-03-25 2017-09-26 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US8961312B2 (en) 2003-03-25 2015-02-24 Creative Kingdoms, Llc Motion-sensitive controller and associated gaming applications
US9446319B2 (en) 2003-03-25 2016-09-20 Mq Gaming, Llc Interactive gaming toy
US9393500B2 (en) 2003-03-25 2016-07-19 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US7948517B2 (en) 2003-04-02 2011-05-24 Toyota Jidosha Kabushiki Kaisha Vehicular image display apparatus and vehicular image display method
EP1465428A1 (en) * 2003-04-02 2004-10-06 Toyota Jidosha Kabushiki Kaisha Vehicular image display apparatus and vehicular image display method
US8072424B2 (en) 2004-04-30 2011-12-06 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US9946356B2 (en) 2004-04-30 2018-04-17 Interdigital Patent Holdings, Inc. 3D pointing devices with orientation compensation and improved usability
US9298282B2 (en) 2004-04-30 2016-03-29 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US9575570B2 (en) 2004-04-30 2017-02-21 Hillcrest Laboratories, Inc. 3D pointing devices and methods
US8629836B2 (en) 2004-04-30 2014-01-14 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US8937594B2 (en) 2004-04-30 2015-01-20 Hillcrest Laboratories, Inc. 3D pointing devices with orientation compensation and improved usability
US9261978B2 (en) 2004-04-30 2016-02-16 Hillcrest Laboratories, Inc. 3D pointing devices and methods
US9675878B2 (en) 2004-09-29 2017-06-13 Mq Gaming, Llc System and method for playing a virtual game by sensing physical movements
US20090122866A1 (en) * 2004-10-22 2009-05-14 Greenparrotpictures Limited Dominant motion estimation for image sequence processing
US8385418B2 (en) * 2004-10-22 2013-02-26 Google Inc. Dominant motion estimation for image sequence processing
US10159897B2 (en) 2004-11-23 2018-12-25 Idhl Holdings, Inc. Semantic gaming and application transformation
US8907889B2 (en) 2005-01-12 2014-12-09 Thinkoptics, Inc. Handheld vision based absolute pointing system
US7931535B2 (en) 2005-08-22 2011-04-26 Nintendo Co., Ltd. Game operating device
US9011248B2 (en) 2005-08-22 2015-04-21 Nintendo Co., Ltd. Game operating device
US7942745B2 (en) 2005-08-22 2011-05-17 Nintendo Co., Ltd. Game operating device
US8313379B2 (en) 2005-08-22 2012-11-20 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
US10155170B2 (en) 2005-08-22 2018-12-18 Nintendo Co., Ltd. Game operating device with holding portion detachably holding an electronic device
US9700806B2 (en) 2005-08-22 2017-07-11 Nintendo Co., Ltd. Game operating device
US9498728B2 (en) 2005-08-22 2016-11-22 Nintendo Co., Ltd. Game operating device
US10238978B2 (en) 2005-08-22 2019-03-26 Nintendo Co., Ltd. Game operating device
US8409003B2 (en) 2005-08-24 2013-04-02 Nintendo Co., Ltd. Game controller and game system
US9498709B2 (en) 2005-08-24 2016-11-22 Nintendo Co., Ltd. Game controller and game system
US8834271B2 (en) 2005-08-24 2014-09-16 Nintendo Co., Ltd. Game controller and game system
US8870655B2 (en) 2005-08-24 2014-10-28 Nintendo Co., Ltd. Wireless game controllers
US8267786B2 (en) 2005-08-24 2012-09-18 Nintendo Co., Ltd. Game controller and game system
US9227138B2 (en) 2005-08-24 2016-01-05 Nintendo Co., Ltd. Game controller and game system
US10137365B2 (en) 2005-08-24 2018-11-27 Nintendo Co., Ltd. Game controller and game system
US9044671B2 (en) 2005-08-24 2015-06-02 Nintendo Co., Ltd. Game controller and game system
US8308563B2 (en) 2005-08-30 2012-11-13 Nintendo Co., Ltd. Game system and storage medium having game program stored thereon
US8708824B2 (en) 2005-09-12 2014-04-29 Nintendo Co., Ltd. Information processing program
US8157651B2 (en) 2005-09-12 2012-04-17 Nintendo Co., Ltd. Information processing program
US7927216B2 (en) 2005-09-15 2011-04-19 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
US8430753B2 (en) 2005-09-15 2013-04-30 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
USRE45905E1 (en) 2005-09-15 2016-03-01 Nintendo Co., Ltd. Video game system with wireless modular handheld controller
US7786976B2 (en) 2006-03-09 2010-08-31 Nintendo Co., Ltd. Coordinate calculating apparatus and coordinate calculating program
US7774155B2 (en) 2006-03-10 2010-08-10 Nintendo Co., Ltd. Accelerometer-based controller
US7877224B2 (en) 2006-03-28 2011-01-25 Nintendo Co, Ltd. Inclination calculation apparatus and inclination calculation program, and game apparatus and game program
US8041536B2 (en) 2006-03-28 2011-10-18 Nintendo Co., Ltd. Inclination calculation apparatus and inclination calculation program, and game apparatus and game program
US8473245B2 (en) 2006-03-28 2013-06-25 Nintendo Co., Ltd. Inclination calculation apparatus and inclination calculation program, and game apparatus and game program
US8913003B2 (en) 2006-07-17 2014-12-16 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer using a projection marker system
US7716008B2 (en) 2007-01-19 2010-05-11 Nintendo Co., Ltd. Acceleration data processing program, and storage medium, and acceleration data processing apparatus for use with the same
US9176598B2 (en) 2007-05-08 2015-11-03 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer with improved performance

Also Published As

Publication number Publication date
GB9523190D0 (en) 1996-01-17

Similar Documents

Publication Publication Date Title
Elad et al. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images
Gribbon et al. A novel approach to real-time bilinear interpolation
KR100888095B1 (en) Image stabilization using color matching
US5524223A (en) Instruction accelerator for processing loop instructions with address generator using multiple stored increment values
US7050068B1 (en) Generation of jittered sub-pixel samples using programmable sub-pixel offsets
US4985848A (en) High speed image processing system using separate data processor and address generator
US6825843B2 (en) Method and apparatus for loop and branch instructions in a programmable graphics pipeline
US20050195188A1 (en) Method and apparatus for dual pass adaptive tessellation
US20060005178A1 (en) Optimized chaining of vertex and fragment programs
EP1569128A2 (en) System and method for accelerating and optimizing the processing of machine learning techniques using a graphics processing unit
EP0150060A2 (en) Multifunctional image processor
EP0447227B1 (en) Method for Generating Addresses to Textured Graphics Primitives Stored in RIP Maps
Downton et al. Parallel architectures for image processing
US6151034A (en) Graphics hardware acceleration method, computer program, and system
US5210705A (en) Digital filtering with single-instruction, multiple-data processor
EP0591412B1 (en) Real-time recognition of pointing information from video
EP1789927B1 (en) Increased scalability in the fragment shading pipeline
US5185599A (en) Local display bus architecture and communications method for Raster display
US6329996B1 (en) Method and apparatus for synchronizing graphics pipelines
US5615322A (en) Image synthesizing system for producing three-dimensional images
US6535216B1 (en) Multi-processor graphics accelerator
US5218674A (en) Hardware bit block transfer operator in a graphics rendering processor
US4864629A (en) Image correlation system
US5109348A (en) High speed image processing computer
EP0314341B1 (en) Computer graphic apparatus for processing lighting model information

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)