WO1996008791A1 - Manipulation d'images - Google Patents

Manipulation d'images Download PDF

Info

Publication number
WO1996008791A1
WO1996008791A1 PCT/GB1995/002189 GB9502189W WO9608791A1 WO 1996008791 A1 WO1996008791 A1 WO 1996008791A1 GB 9502189 W GB9502189 W GB 9502189W WO 9608791 A1 WO9608791 A1 WO 9608791A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
scene
depth
image
distance
Prior art date
Application number
PCT/GB1995/002189
Other languages
English (en)
Inventor
Paul Grace
Sean Broughton
Original Assignee
Paul Grace
Sean Broughton
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Paul Grace, Sean Broughton filed Critical Paul Grace
Priority to EP95931338A priority Critical patent/EP0729623A1/fr
Priority to GB9612554A priority patent/GB2298990B/en
Publication of WO1996008791A1 publication Critical patent/WO1996008791A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering

Definitions

  • This invention relates to image manipulation.
  • the invention is concerned with a system which permits the manipulation of two-dimensional images in such a way as to take into account three-dimensional effects.
  • the animated object is required to appear to pass in front of or behind objects in the real scene these or real objects should be obscured and revealed in accordance with their position in the implied third dimension of the recorded image.
  • the animated image should also be accordingly obscured or revealed.
  • the real scene captured for example on film, or video tape has no measure of depth, other than implied perspective, it takes a skilled artist to cut out some of the foreground objects, and insert the animated object between the background plane and the foreground object.
  • the next stage will be to combine this computer generated, animated sequence in with real film footage, i.e. video or cinematographic shoots of real objects, people and so forth. If the animated toothpaste tube is to be the top layer of the combined sequence, this can be achieved easily. Matting instructions are easily defined to replace film image by computer animated image across a frame wherever the computer animated image has a non-zero value. Of course, the whole advertisement will comprise a series of frames with the tube moving between a number of positions in all three dimensions.
  • the matting technique is limited in the degree of realism that it can produce.
  • the flying toothpaste tube may need to appear to be on such a trajectory as to go behind some objects, but in front of others.
  • Real situations can be far more complex than this simplistic example, as there could be many tens (if not hundreds) of levels of 'foreground' and 'background' .
  • the positioning of the animated image in the implied third dimension of a recorded image is problematic with traditional techniques, because the real scene has been recorded onto a two-dimensional medium for example film or video, and the depth information has been lost.
  • US-A-5,283,640 there is disclosed a system for producing and receiving three-dimensional images.
  • a laser projector and a pair of sensors are used to triangulate the distance from the camera to objects in a scene.
  • the depth information is used to instantaneously produce two images of the scene on respective television screens which correspond to the view of the human left and right eye. That system thus produces a binocular view of the scene, but is not concerned with the storing and manipulation of two- dimensional data, taking depth into account.
  • the laser projector and sensors might be usable to create signals which are converted into distance data, although that is not suggested.
  • a method of recording and manipulating image data in which there are recorded (a) two-dimensional image data for a scene and (b) distance data representing the respective distances from a reference of a plurality of objects at different positions in the scene; and in which the image data is manipulated taking into account the distance data.
  • the scene will be a real scene and the distance data will be calculated by suitable means, although the scene could be created artificially and distance data assigned to objects.
  • the image data will be stored as a series of frames and there will be a corresponding serious of 'frames' of data from distance scans across the scene for each frame, containing the distance readings at various points in the scene.
  • the distance may be measured from a reference point, line or surface. It may be most convenient for there to be measurement relative to the focal plane of the camera recording the image of the scene.
  • an instrument to provide the distance data in the case of a real scene. This can be achieved by a separate recording instrument, for example a two-dimensional infra red or ultrasonic sound scanner.
  • This scanner during one frame time of the film or video recording medium will send out a raster scan of infra red or ultrasonic sound. By timing the delay of return for each effective position of the scan, a depth measurement can be obtained.
  • any 'signal' outside the visible spectrum of the film or video and the audible spectrum of the associated sound is acceptable.
  • this may include ultraviolet, radar wavelengths or sub-sonic sound.
  • the radar wavelength a particularly preferred range is the microwave range.
  • One method of constructing the 'ultrasound' scanner can be from a rotating sound source, mounted on a vertical axis. As the directional pulsed sound emitter rotates past the scene to be recorded, it 'scans' a horizontal line, by 'firing' pulses at each pixel to be scanned. By sampling the time delay of the returning of each of these pulses, a 'distance' measure is derived. This can be coded as a digital value, and stored at the relevant pixel address for that line. By 'nodding' the top of the vertical axis towards the scene, different (but lower) scan lines can be described. This can be repeated until a complete frame has been described. Similar methods can be incorporated with infra red light, instead of ultrasound.
  • Edge detection techniques such as Laplacian operators, Sobel Operators, and other techniques disclosed in Pratt's textbook 'Digital Image Processing' can be used. This will allow the analysis of the recorded scene into 'objects'. As long as each object has one 'depth' point, the system will work. If there is more than one depth measurement point per object, these can be averaged to form an 'average object depth'.
  • these can ranked to find a maximum object depth and a minimum object depth.
  • This 'depth plane' signal can be recorded as a low resolution image, synchronous with the main image recording.
  • Such an 'image' can be synchronised with the main image using techniques that would be used to synchronise multiple camera shoots, or to synchronise sound. These techniques include the recording of a 'timecode' on all of the material shot. This is .a simple technique where a unique timecode value is supplied, in parallel, to all recording media in a shoot. This timecode is incremented by one count on each successive frame. Thus all of the recording media capture the same unique code for each given frame.
  • the depth recording medium will probably (but not necessarily) be digital, in which case the digital number recorded will be a measure of the 'depth' or 'distance' from the camera.
  • the allocation of depth or distance per digital level can be on a linear, logarithmic, or non-linear (look up) basis, depending on the resolution of depth recording required at particular distance.
  • the calibration of depth versus digital recorded level can be via a 'depth slate' . This consists of a 'calibration' frame, shot at the start of a scene. In this calibration frame, a series of numbered 'marker' targets are recorded, and their distances from the camera measured accurately with a tape measure. When the digital level recorded on the depth recording medium for the target is read off, it can be compared to the distance displayed on the target and thus a correlation between the digital level on the recording medium and the actual distance can be derived.
  • the 'depth' recording can be edited with the picture, just as when sound is recorded separately from pictures, but using timecode, the resultant edited picture has corresponding sound.
  • the 'depth' information can be used for the process of artificially 'relighting' the scene. This can be done because we have not only the 'order' of depth of objects in a scene (e.g. cat is in front of car which is in front of brick wall) , but the actual depth dimensions
  • the invention may take many forms, including for example the provision of a camera and an ultrasonic or infra red scanner. There may be changes to the systems particularly described.
  • Figure la is a schematic representation of an animated image of a toothpaste tube
  • Figure lb is a schematic representation of a recorded image of a real scene
  • Figure lc is a schematic representation of a composite image of the images of Figures la and lb showing simple matting of the toothpaste tube 'on top' of the recorded image
  • Figure Id is a schematic representation of a composite image of the images of Figures la and lb showing complex matting of the toothpaste tube in accordance with the invention
  • Figure 2a is a schematic representation of the fine grid of video or film pixels
  • Figure 2b is a schematic representation of the coarse grid of the scanned depth pixels
  • Figure 3 is a schematic representation of a system suitable for carrying out the invention.
  • a system suitable for carrying out the invention comprises a film or video camera 6 which supplies image data to an image recording medium 5 such as video tape or cinematographic film.
  • the image recording medium may be stored within the video or film camera or may be in a separate device such as a video recorder.
  • a timecode generator 9 supplies a timecode to both the image recording medium 5 and the depth recording medium 8 for subsequent synchronisation of the data.
  • Other synchronisation codes which could be used include Filmcode, Xeycode, Aaton-code, and Arri-code.
  • Reference numeral 4 of Figure 3 indicates the real scene which is to be recorded by the system.
  • the camera 6 films' the scene in video or film resolution.
  • the depth scanner 7 measures the distance of the objects comprising the scene from some arbitrary reference plane.
  • the image resolution will be significantly higher than the depth scan resolution as shown in Figure 2b.
  • a video scan according to the European Recommendation 601 Standard, consists of 720 picture elements scanned along each of 576 active lines.
  • a typical 'sonic' or 'infra red' depth scan will probably only need a resolution of typically 70 x 50.
  • the depth scan will be calibrated using a 'depth slate'.
  • the depth slate is a frame of image data in which are recorded a series of marker targets positioned in the scene at known distances from the depth scanner.
  • the 'marker' targets can have on them a 'blackboard' region, where the distance can be written in chalk.
  • a more sophisticated version could consist of LEDs (light emitting diodes) which could display the distance as typed in from an operator's console.
  • the depth data is in the form of a low resolution 'image' of the scene with pixel values corresponding to the distance of an object, or part of an object, from the depth scanner 7.
  • suitable media for the recording of such an image include the Sony Betacam tape recording medium. Whilst this has the capability to record full bandwidth image signals, low bandwidth can be accomplished by repeating picture elements and lines to "make up" the right number of pixels and lines required by such a recording medium.
  • DAT digital audio tape
  • the important criteria in the choice of medium is that it supports timecode.
  • Fig. la shows the image of the toothpaste tube 3
  • Fig. lb shows a real filmed background comprising object 1 and 2
  • Fig. lc shows the toothpaste tube superimposed on the background.
  • object 1 is behind object 2.
  • Fig. Id shows what may be required.
  • the toothpaste tube appears to be positioned in front of object 1 but behind object 2.
  • the system shown in Figure 3 will be used to record an image of the background scene, as represented by Figure lb, together with the associated depth data.
  • Edge detection techniques will be used to define each object of the scene in the image and to associate one or more depth values, as calibrated using the depth slate, to each object. It is then possible to compare the magnitude of the 'depth' of the trajectory of the toothpaste tube with the 'depth' of each object (e.g. distance from a reference plane) .

Abstract

L'invention concerne un procédé pour enregistrer et manipuler des données d'images, dans lequel des données d'images bidimensionnelles concernant une scène réelle (4) sont enregistrées sur un support d'enregistrement (5) d'images en utilisant une caméra à film ou une caméra vidéo (6). On enregistre également des données concernant les distances des objets présents sur la scène, sur un support (8) pour enregistrer la profondeur, en utilisant un scanneur (7) de profondeur. Les données relatives aux distances et les données d'images sont synchronisées grâce à une indexation temporelle réalisée sur les deux supports par un dispositif d'indexation temporelle (9). L'image est ensuite manipulée en tenant compte des données relatives aux distances.
PCT/GB1995/002189 1994-09-15 1995-09-15 Manipulation d'images WO1996008791A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP95931338A EP0729623A1 (fr) 1994-09-15 1995-09-15 Manipulation d'images
GB9612554A GB2298990B (en) 1994-09-15 1995-09-15 Image manipulation

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB9418610.3 1994-09-15
GB9418610A GB9418610D0 (en) 1994-09-15 1994-09-15 Image manipulation

Publications (1)

Publication Number Publication Date
WO1996008791A1 true WO1996008791A1 (fr) 1996-03-21

Family

ID=10761378

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB1995/002189 WO1996008791A1 (fr) 1994-09-15 1995-09-15 Manipulation d'images

Country Status (3)

Country Link
EP (1) EP0729623A1 (fr)
GB (2) GB9418610D0 (fr)
WO (1) WO1996008791A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1156298A2 (fr) * 2000-05-15 2001-11-21 Weimatic GmbH Procédé et appareil pour la détection d' objets tridimensionels
WO2003100703A3 (fr) * 2002-05-28 2004-05-27 Casio Computer Co Ltd Appareil de sortie d'images composites et appareil de distribution d'images composites
DE102008042333B4 (de) * 2007-09-28 2012-01-26 Omron Corp. Gerät zur dreidimensionalen Messung und Verfahren zum Einstellen einer Bildaufnahmevorrichtung

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LEE AND MYERS: "3-D PERSPECTIVE VIEW OF SONAR IMAGES", ASILOMAR CONFERENCE ON SIGNALS, SYSTEMS & COMPUTERS, 31 October 1988 (1988-10-31) - 2 November 1988 (1988-11-02), SAN JOSE CA US, pages 73 - 77, XP000130237 *
SUENAGA AND WATANABE: "A METHOD FOR THE SYCHRONIZED ACQUISITION OF CYLINDRICAL RANGE AND COLOR DATA", IEICE TRANSACTIONS, vol. E74, no. 10, TOKIO JP, pages 3407 - 3416, XP000279320 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1156298A2 (fr) * 2000-05-15 2001-11-21 Weimatic GmbH Procédé et appareil pour la détection d' objets tridimensionels
EP1156298A3 (fr) * 2000-05-15 2002-06-19 Weimatic GmbH Procédé et appareil pour la détection d' objets tridimensionels
WO2003100703A3 (fr) * 2002-05-28 2004-05-27 Casio Computer Co Ltd Appareil de sortie d'images composites et appareil de distribution d'images composites
KR100710512B1 (ko) * 2002-05-28 2007-04-24 가시오게산키 가부시키가이샤 합성화상배신장치, 합성화상배신방법 및 기록매체
KR100710508B1 (ko) * 2002-05-28 2007-04-24 가시오게산키 가부시키가이샤 합성화상출력장치, 합성화상출력방법 및 기록매체
KR100752870B1 (ko) * 2002-05-28 2007-08-29 가시오게산키 가부시키가이샤 합성화상출력장치, 합성화상출력방법 및 기록매체
US7787028B2 (en) 2002-05-28 2010-08-31 Casio Computer Co., Ltd. Composite image output apparatus and composite image delivery apparatus
EP2357795A3 (fr) * 2002-05-28 2013-05-15 Casio Computer Co., Ltd. Appareil de sortie d'images composites et appareil de distribution d'images composites
DE102008042333B4 (de) * 2007-09-28 2012-01-26 Omron Corp. Gerät zur dreidimensionalen Messung und Verfahren zum Einstellen einer Bildaufnahmevorrichtung
US8340355B2 (en) 2007-09-28 2012-12-25 Omron Corporation Three-dimensional measurement instrument, image pick-up apparatus and adjusting method for such an image pickup apparatus

Also Published As

Publication number Publication date
EP0729623A1 (fr) 1996-09-04
GB9612554D0 (en) 1996-08-14
GB2298990A (en) 1996-09-18
GB9418610D0 (en) 1994-11-02
GB2298990B (en) 1998-12-09

Similar Documents

Publication Publication Date Title
US5963247A (en) Visual display systems and a system for producing recordings for visualization thereon and methods therefor
US6335765B1 (en) Virtual presentation system and method
US6724386B2 (en) System and process for geometry replacement
US9041899B2 (en) Digital, virtual director apparatus and method
US9160938B2 (en) System and method for generating three dimensional presentations
Matsuyama et al. 3D video and its applications
US20060165310A1 (en) Method and apparatus for a virtual scene previewing system
US4738522A (en) Method and apparatus for coordinated super imposition of images in a visual display
US20030174285A1 (en) Method and apparatus for producing dynamic imagery in a visual medium
US11514654B1 (en) Calibrating focus/defocus operations of a virtual display based on camera settings
JP2000503177A (ja) ２次元映像（ｍｏｔｉｏｎ ｐｉｃｔｕｒｅ）を３次元映像に変換する方法及び装置
EP4111677B1 (fr) Synchronisation de données d'images à sources multiples
US11615755B1 (en) Increasing resolution and luminance of a display
US11328436B2 (en) Using camera effect in the generation of custom synthetic data for use in training an artificial intelligence model to produce an image depth map
WO1996008791A1 (fr) Manipulation d'images
GB2399248A (en) Projection of supplementary image data onto a studio set
CA2191711A1 (fr) Systemes et procedes d'affichage visuel et de production d'enregistrements destines a etre visualises sur ledit affichage
Braham The digital backlot [cinema special effects]
KR102654323B1 (ko) 버추얼 프로덕션에서 2차원 이미지의 입체화 처리를 위한 방법, 장치 및 시스템
GB2586831A (en) Three-Dimensional Content Generation using real-time video compositing
CN114641980A (zh) 使用任意捕获的输入对捕获图像的遮挡视图的重构
Mirpour A Comparison of 3D Camera Tracking Software
Brown-Simmons Computer animation within videospace

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): GB JP US

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH DE DK ES FR GB GR IE IT LU MC NL PT SE

WWE Wipo information: entry into national phase

Ref document number: 1995931338

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWP Wipo information: published in national office

Ref document number: 1995931338

Country of ref document: EP

WWR Wipo information: refused in national office

Ref document number: 1995931338

Country of ref document: EP

WWW Wipo information: withdrawn in national office

Ref document number: 1995931338

Country of ref document: EP