EP1507235A1 - Method and apparatus for producing composite images which contain virtual objects - Google Patents

Method and apparatus for producing composite images which contain virtual objects Download PDF

Info

Publication number
EP1507235A1
EP1507235A1 EP04018582A EP04018582A EP1507235A1 EP 1507235 A1 EP1507235 A1 EP 1507235A1 EP 04018582 A EP04018582 A EP 04018582A EP 04018582 A EP04018582 A EP 04018582A EP 1507235 A1 EP1507235 A1 EP 1507235A1
Authority
EP
European Patent Office
Prior art keywords
image
virtual object
marker
marker object
virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP04018582A
Other languages
German (de)
French (fr)
Inventor
Werner G. Lonsing
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LONSING, WERNER G.
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of EP1507235A1 publication Critical patent/EP1507235A1/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/006Mixed reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00Indexing scheme for image rendering
    • G06T2215/16Using real world measurements to influence rendering

Definitions

  • This invention relates to the production of composite images, videos, or films which exhibit virtual objects. More particularly, this invention relates to methods and apparatus for rendering, scaling, and/or locating virtual objects within composite images, videos, or films employing marker objects as reference objects.
  • CGI's computer generated images
  • this invention fulfills the above-enumerated problems in the art by providing:
  • computer generated three-dimensional objects are registered (displayed) in real scenes and can be modified (e.g. in appearance, size, shape, color, texture) on location.
  • lighting and/or weather conditions are detected by a computer from a recorded image and the appearance of the computer generated three-dimensional object is modified to reflect such conditions.
  • the image generation apparatus generally includes a camera 1 (e.g. digital or film) capable of capturing either still or moving images and a position determining means 3 (e.g. a GPS, alternative forms of triangulation, radar, sonar, a laser range finder, surveying equipment, etc.) located on camera 1.
  • a camera 1 e.g. digital or film
  • a position determining means 3 e.g. a GPS, alternative forms of triangulation, radar, sonar, a laser range finder, surveying equipment, etc.
  • a marker object 5 is provided which, as will be described in detail below, is employed for establishing a reference position from which a location of a virtual object will be assigned.
  • a marker object 5 it is preferred that the precise shape and dimensions of marker object 5 be known and stored in a memory, for example.
  • the position of marker object 5 is determined.
  • the position of marker object 5 can be determined manually, but is preferably determined via position determining means 7 (e.g. GPS device) located on the marker object.
  • position determining means 7 e.g. GPS device
  • at least one computer 9 having a memory device is provided to execute various functions (which will be described below) as well as to store various fixed and collected data.
  • a wireless local area network (WLAN) (not shown) is provided by which the various parts of the invention can communicate by radio waves.
  • WLAN wireless local area network
  • other forms of wireless communication or hard-wiring for data transmission can, of course, be used.
  • computer 9 contains program code in its memory for executing various tasks including methods of image processing.
  • the program code (software) stored in memory is capable of detecting marker object 5 in an image captured by camera 1 (e.g. via thresholding, color detection, or any other detection method).
  • resident software is capable of determining size and shape values of objects in an image captured by camera 1, and is capable of comparing the size and shape values of the objects to the known size and shape of marker object 5 so that marker object 5 can be detected in the captured image.
  • the canvas size (image size i.e. width and height in number of pixels) of the camera should be known, so that the scale and thus the size of the marker object in the image can be determined.
  • computer 9 contains software for generating a virtual object (for generating it on site).
  • software such as CAD-type software may be used.
  • the virtual object can be generated by a separate source (e.g. a second computer) as required.
  • the virtual object can simply be resident in computer 9's memory (having been created independently elsewhere) or be received wirelessly (e.g. by satellite or antenna).
  • the computer should contain interfaces for at least the input of video signals and the input of position determination means data (e.g. IE 1394 or USB ports).
  • position determination means data e.g. IE 1394 or USB ports.
  • computer 9 employs a virtual camera 11 (e.g. generated by a software or program code component such as OpenGl TM or Renderman TM) which is capable of rendering two-dimensional images from three-dimensional virtual objects for use in composing composite images (comprising a combination of a virtual object and a real scene image captured by camera 1). Details regarding the operation of the virtual camera will be discussed at length below. Generally speaking, however, once a composite image (or images) is generated, the composite image(s) can be displayed on display device 17 (e.g. a closed circuit monitor, LCD screen, etc.) so that the image can be reviewed (e.g. such as so a director can view a filmed scene on location and re-shoot if necessary).
  • display device 17 e.g. a closed circuit monitor, LCD screen, etc.
  • the image can be stored in storage device 15 (a conventional memory device) or simply stored in the memory of computer 9.
  • composite image(s) can be transmitted via conventional transmission device 19 to a remote location (e.g. to a television studio for broadcasting via antenna or via satellite or to a remote storage location).
  • the geographical site for image capture must first be selected.
  • the site which is selected would likely be the site where a real building is desired to be constructed.
  • the site which is chosen would be a site at which an event or scene from a script is desired to be filmed.
  • various prospective locations or various views of a single location can be filmed so that the impact of the selection of a particular highway or railroad route can be viewed with some degree of realism (e.g. a highway can be shown "virtually" at a hearing to traverse, alternately, both the north and south side of a mountain for comparative purposes).
  • a precise "target" position at the geographical location) at which a virtual object is desired to be displayed must be chosen. Furthermore, characteristics of the chosen geographical site should be observed, and a marker object appropriate for such location selected.
  • the object for the purpose of facilitating detection of the marker object, it is preferred that the object have a uniform geometric shape (e.g. a sphere, square, or triangle) and that the object have a coloration and/or texture which is as distinct as possible from the surroundings/scene that it will be placed in. For example, if the predominant coloration of a location is green (e.g.
  • Figs.4a-4d illustrate these general principals with regards at least to the contrast of marker object 5 with respect to it surroundings in a given scene. As will be noted from Figs. 4a and 4b, a darkly colored marker object is difficult to detect against a dark background (Fig. 4a), but is easy to detect against a light background (Fig. 4b). The converse effect may seen in Figs. 4c and 4d.
  • the appearances of the different marker objects must be distinct, either in shape or surface, so that they can be detected individually.
  • marker object selection guideline it is noted that it is typically desirable for the marker object to be as large as is practically possible. In this regard, employing a large-type marker object makes the detection and data collection steps easier and less error prone (i.e. extrapolation errors are minimized as the marker object gets larger). It is contemplated, however, that marker objects as small as ping pong balls (or smaller) can be used with success and the use of objects of that size may be desirable in some circumstances (e.g. in the foreground if the location of the target position is inaccessible or remote).
  • position determining means 7 is preferably a GPS-type unit (i.e. because they are inexpensive, sufficiently accurate, usable universally geographically), various other mechanisms may be employed such as alternative forms of triangulation, radar, sonar, a laser range finder, surveying equipment, etc. It is noted, that current GPS systems are subject to random data errors or data aberrations (conventionally +/- 3 meters).
  • the position of marker object 5 can be automatically determined at the instant at which camera 1 captures an image (i.e. position determining means 7 can be programmed to record position information e.g. with a timestamp when camera 1 is triggered).
  • camera 1 After marker object 5 has been placed at the target location, and the position of marker object 5 has been determined, camera 1 (see Fig. 3) is used to compose and capture an image or series of images (e.g. still or moving, digital or film) of a scene containing marker object 5 (see Fig. 5, which illustrates marker object 5 placed in a natural scene). Images are, of course, composed according to the commercial or artistic purpose of the desired final composite image(s). If it is desired that the virtual object be animated or that the virtual object appear at multiple target positions within a scene, object 5 is simply moved or repositioned and subsequent position determinations (with position determination means 7) made as each image or series of images are taken.
  • image file format such as TIFF, JPEG TM or specific information in individual movie frames. This information is determined through camera 1 (e.g. logical size in width and height defined in pixel, the amount of information in bits each pixel has e.g. color depth, transparency, 24-bit RGB or 48-bit RGBA, etc.) and is transferred to computer 9.
  • digital image file formats provide information about the order of pixels (most importantly the starting point and scanning direction), which is used during subsequent image processing calculations. It is noted that if a film-type image (or images) is taken, the image must be scanned with a conventional scanner (not shown) in order to produce a digitized image and entered into computer 9.
  • position of the camera is measured and/or calculated and then stored in association with its respective image(s).
  • position determination means 3 e.g. GPS, alternative triangulation means, radar, sonar, a laser range finder, surveying equipment, etc.
  • a particularly useful device for this purpose is a BluetoothTM-GPS device like the Fortuna Bluetooth GPSmart TM (hard wiring the localization device can result in reduced costs and higher speed and accuracy, however). If camera 1 is communicably connected to computer 9, images as well as positional information can be transmitted to and stored in computer 9's memory as each image and/or position determination is recorded i.e. in real-time.
  • virtual object 10 may be generated prior to image capture (e.g. via computer 9, or any other computer with conventional 3-D generation software such as CAD-type software, etc.), it is also possible to generate virtual object 10 on site, or to "receive" the virtual object by transmission (e.g. from a television studio transmission by antenna or by satellite). It is further noted that this step can be performed in an order different than the order described herein and that such order can be selected according to various needs which will be determined by the user as desired.
  • program code (software) is operated on computer 9 to "detect" marker object 5 on the captured two-dimensional image using methods of image processing.
  • program code software
  • certain known qualities of the marker object previously recorded and stored in computer 9) can be "searched” for by the software employed. For example, in order to detect these qualities, color thresholding and contour-detection can be used. Because the marker objects and scenes are highly variant, all or only a few qualities of marker object 5 may be recorded and used (e.g. size, shape, and skin texture, etc.).
  • marker point "MP” may be any point on marker object 5 which is desired, it is typically convenient and efficient to select a easily identifiable marker point such as the center point of the marker object (particularly if such object is a sphere). Alternatively, marker point "MP" may be the corner of a cube for example. Using the center point of a sphere, however (see Fig. 5), because the position and size (i.e. diameter) of marker object 5 is known (and has been pre-input into computer 9), the precise position of the marker point can be determined with relative ease (within certain degrees of error depending on the integrity of the acquired positional data).
  • marker point "MP” is the value which is used as a reference position to determine the position in the image at which virtual object 10 will be oriented. It is noted in this regard, that virtual object 10 can be displayed at this reference position "MP", or at a position distant from it. However, marker point "MP" is the reference position from which the final position of virtual object 10 should be calculated/determined.
  • appearance qualities of marker object 5 can be detected and compared to reference values (color, texture, or shading) related to the appearance of the marker object stored in computer 9. If any differences are detected (e.g. due to lighting conditions, etc.), data regarding these differences is stored and can be employed during the rendering of the virtual object to provide an additional degree of realism to the final composite image.
  • virtual object 10 begins as data representing a three-dimensional object created according to pre-selected criteria. For example, if an architect is desirous of displaying a digital rendering of a particular building design, data representing virtual object 10 will be generated according to specifications for the envisioned building. Similarly, if a movie director wishes to render a fictitious character, a specialist programmer will generate data representing the appearance of the character according to the director's specifications (e.g. from a storyboard or other drawing). Data related to the virtual object may stored on computer 9 (e.g.
  • virtual object 10 is editable and can be modified in all dimensions and qualities (e.g. size, color, etc.). In still further preferred embodiments, virtual object 10 is editable on location and/or in real-time (e.g. employing accelerated graphics).
  • computer 9 In order to render virtual object 10, computer 9 (see Fig. 3) employs a virtual camera 11 for projecting an image of the virtual object in two dimensions. In order to do this meaningfully, however, this computational virtual camera must be synchronized with camera 1 (see Figs. 2 and 5). Specifically, virtual camera 11 should be positioned virtually so that at least its viewing direction (or orientation) and canvas size are synchronized with that of camera 1. In addition, camera calibration occurs and object attributes are rendered. Synchronization ensures that when virtual object 10 is rendered as a two-dimensional image (e.g. so that it can be displayed on a flat screen or monitor), the image that is produced is displayed from the perspective of camera 1 at the time of image capture.
  • synchronizing coordinates implies the synchronization of different coordinate-systems (e.g. coordinates from the GPS are transformed to coordinates in the virtual model-space).
  • the position of marker point "MP' is then synchronized in the same manner.
  • the canvas size which is to be rendered by the virtual camera should be matched to the "real" canvas size of the image captured by camera 1.
  • computer 9 should be supplied with data regarding camera 1's canvas size.
  • Canvas size in this sense can be defined as the logical size in width and height, defined in pixels, of the captured image. This can be easily determined using image information in the digital image (i.e. the width and height in pixels) or is known by referring to commercial standards (e.g. VGA, 640x480).
  • the viewing direction of camera 1 must be determined in order to apply it to virtual camera 11 for synchronization.
  • viewing direction can be calculated from the position of camera 1 relative to the position of marker object 5 i.e. at the instant at which camera 1 was used to capture the relevant image of marker object 5.
  • the viewing direction is the imaginary axis extending between the viewpoint and the eyepoint.
  • the virtual camera After determining the canvas size (or image pane) according to the above guidelines (in the first step of synchronizing the virtual camera to real camera 1), the virtual camera should be virtually positioned at the position of camera 1.
  • the position of camera 1 and camera 11 are aligned at identical coordinates in the same three-dimensional coordinate system (e.g. a coordinate system having values such as x, y, and z).
  • synchronizing coordinates implies the synchronization of different coordinate system (e.g. coordinates from the GPS are transformed to coordinates in the model-space).
  • the position of marker point "MP" is then synchronized in the same manner.
  • camera 11 should be aligned so that it is aimed along camera 1's viewing direction.
  • marker point "MP" as viewpoint (without correction) centers the marker point virtually in the middle of the image.
  • the correct viewing direction is calculated using the deviation of the marker point detected in the image of the real scene according to the center of that image as deviation of the viewing direction of the real scene and the viewing direction as if camera 1 is directly aimed at the marker point "MP".
  • the calculation is based on triangulation using the deviation in pixels in the captured image as well as the known (calculated and/or measured) distance between position of camera 1 and marker 5, and the field of view.
  • marker objects e.g. marker object 5
  • marker objects provide the advantage of implicit error correction.
  • using a marker object (in an image) to calculate camera orientation typically always results in usable data (even if errors were otherwise introduced).
  • using marker objects is self-correcting, in effect, for certain types of measurement errors and is therefore the preferred mode of orientation/angle determination.
  • the field of view of camera 1 is known (from knowledge of camera 1's optics, etc.), this value can be assigned to the virtual camera to achieve synchronization of canvas sizes (it is noted that field of view is typically more useful than focal length primarily because focal length is normally defined in relation to a specific image pane i.e. and image pane values are often not known). However, it is important to note that it is not mandatory that the optical system of camera 1 (and thus its field of view) be known and that alternative methods for determining field of view can be employed. If field of view is not known, then, the field of view (or virtual focal length of camera 1 according to canvas size) can be determined from the size of detected marker object 5 and the known (i.e. measured or calculated) distance between camera 1 and marker 5.
  • field of view can be calculated from the detected size of detected marker object 5 (in the image of the real scene), the distance between the marker object and the camera, the known real size of the marker object, and the size of the image (i.e. the size of camera 1's canvas) via simple triangulation.
  • field of view is calculated from marker object values, it is helpful if a relatively large marker object is employed to reduce extrapolation errors (e.g. a marker object should be approximately 1/20 th of the size of the object being simulated).
  • a final composite image (containing virtual object 10 overlayed in a real/natural scene image) can be composed.
  • computer 9 simply merges the original image capture (i.e. the natural or real scene image) with the created digital image of virtual object 10.
  • Figs. 2 and 5 illustrate these final steps which are performed by software resident on computer 9.
  • the image can be stored on computer 9, displayed on a monitor (e.g. display device 17), saved to a separate storage device 15, or transmitted to a remote location (e.g. a television studio) via transmission device 19 (e.g. by satellite, radiowaves, etc.) (see Fig. 1 or 2).
  • the appearance of the image of virtual object 10 can be edited (i.e. its appearance is tailorable to specific conditions). More specifically, data collected relating to the appearance of marker object 5 in live conditions can be employed to tailor the appearance of virtual object 10 as it will appear in the composite image.
  • virtual object 10 may be employed to "fine tune" the virtual objects appearance so that it can be assimilated into a composite image as seamlessly as possible (e.g. for the greatest degree of realism or other desired factor).
  • This tailorability can be performed automatically with computer 9 software (prior to, during, or after virtual object rendering), or manually as desired.
  • each of the cameras 1 must be provided with a device for determination of its current position, e. g. GPS. It is essential to see at least one of the markers at any time, so that position and eyepoint of each of the cameras 1 can be calculated by the computer 9 in each of the captured pictures.
  • a plurality of markers it ist also possible to film fastly moved film sequences, e. g. races with cars, ships, aircrafts and so on.
  • the position of a marker can be determined fastly and exactly enough, to move a marker simultaneously with the movement of a camera. Therefore the marker can have its own motor or can be driven by an external device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Methods and apparatus for production of composite images, videos, or films which exhibit virtual objects. More particularly, methods and apparatus for rendering, scaling, and/or locating, virtual objects within composite images, videos, or films employing marker objects as reference objects.

Description

    FIELD OF THE INVENTION
  • This invention relates to the production of composite images, videos, or films which exhibit virtual objects. More particularly, this invention relates to methods and apparatus for rendering, scaling, and/or locating virtual objects within composite images, videos, or films employing marker objects as reference objects.
  • BACKGROUND OF THE INVENTION
  • The use of representative models (typically less than full-scale), both real and computer generated, has heretofore been important to various commercial industries. For example, small-scale physical models (e.g. constructed from wood) have been prolifically employed in the construction/architecture industry in order so that proposed building designs can be viewed and evaluated for aesthetics, etc. prior to incurring the expense of commencing or completing construction. Also, computer generated images (hereinafter CGI's) have been employed for similar purposes and provide certain advantages in that CGI's are easily duplicated and can be transmitted electronically to distant locations virtually instantly (as compared to the cumbersome process of packing a fragile wooden model and shipping it via ground or air freight).
  • Similarly, the use of special effects in both the movie and television industries is becoming increasingly common. In a typical movie or television program which employs special effects, a real scene might be augmented with CGI's. For example, a computer generated character or fictitious animal might be inserted into a previously filmed location shot. Conversely, in science fiction movies for example, entire scenes can be generated by computer with either real or computer generated characters existing in the scene.
  • Certain drawbacks exist with respect to the above technologies, however, and improvements which address these drawbacks are desired. For example, the detail engineered or crafted into a physical model is time demanding and labor intensive. Similarly, changes to the design or appearance of a physical model often require expenditures of time and resources nearly equal to that of building the original model itself. Furthermore, it is difficult to envision the full aesthetic nature of a project if built only on a small scale and with non-realistic building materials (e.g. with wood, plaster, or paper) and not the actual building materials which are desired (e.g. glass, concrete, steel, etc. including their associated colors, textures, reflectivity, etc.). Still furthermore, construction of models approaching 1:1 scale is expensive and typically impractical except for in some industries (e.g. the movie industry).
  • In addition to problems with physical models, the use of computer technologies related to CGI's presents various additional drawbacks. For example, in some applications, it is desired that a CGI be precisely sized and located relative to the scene (or object within the scene) in which it appears. Existing technologies do not adequately address these problems of scale and positioning. Furthermore, in outdoor environments, weather/lighting conditions change continuously and therefore affect the appearance of imaged/filmed scenes. In order to maintain an adequate degree of realism, the appearance of a CGI ("virtual object") inserted into a scene should be modified accordingly (to reflect the lighting and/or appearance of its "live" surroundings). Current technologies do not sufficiently resolve this problem.
  • In view of the above, it is apparent that there exists a need in the art for methods and apparatus which overcome one or more of the above drawbacks. It is a purpose of this invention to fulfill these needs in the art, as well as other needs which will become apparent to the skilled artisan once given the following disclosure.
  • SUMMARY OF THE INVENTION
  • Generally speaking, this invention fulfills the above-enumerated problems in the art by providing:
  • a method of rendering a composite image including at least a virtual object displayed in a real scene image, the method comprising:
  • determining size and shape qualities of a marker object;
  • locating the marker object at a location at or near a location at which a virtual object is desired to be virtually placed;
  • determining a position of the marker object;
  • employing an image capturing means to capture at least one image of a real scene containing the marker object;
  • determining a position of the image capturing means when the at least one image of a real scene is captured;
  • analyzing the image of a real scene to detect the marker object;
  • rendering a virtual object by synchronizing a virtual camera with the image capturing means, synthesizing a canvas having a size and shape corresponding to a size and shape of the image of a real scene, and projecting the virtual object onto the canvas so that the virtual object is displayed on the canvas at an orientation determined with reference to the position of the marker object in the image of a real scene;
  • composing a composite image comprising the image of a real scene and the virtual object.
  • In alternative embodiments, therein is provided:
  • apparatus for rendering a composite image including at least a virtual object displayed in a real scene image, the apparatus comprising:
  • a marker object of known size and shape and located at known coordinates;
  • an image capturing means located at known coordinates;
  • a computer in communication with the image capturing means, and tangibly embodying a program of instructions executable by the computer to perform steps comprising:
  • detecting the marker object when displayed in an image captured by the image capturing means;
  • rendering a virtual object by synchronizing a virtual camera with the image capturing means, synthesizing a canvas having a size and shape corresponding to a size and shape of the image of a real scene, and projecting the virtual object onto the canvas so that the virtual object is displayed on the canvas at an orientation determined with reference to the position of the marker object in the image of a real scene;
  • composing a composite image comprising a combination of the virtual object and a real scene image captured by the image capturing means.
  • It is an object of this invention to allow combination of real images with computer generated three-dimensional objects, preferably on-site and in real-time, so that the appearance of reality at a specific geographical location (as a captured image or images) can be augmented for various commercial and/or artistic purposes.
  • In some embodiments, computer generated three-dimensional objects are registered (displayed) in real scenes and can be modified (e.g. in appearance, size, shape, color, texture) on location.
  • In further embodiments of this invention, lighting and/or weather conditions are detected by a computer from a recorded image and the appearance of the computer generated three-dimensional object is modified to reflect such conditions.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a graphical representation of one embodiment of the composite image generation apparatus and method according to the subject invention.
  • FIG. 2 illustrates a simplified diagram of the functionality of the apparatus and method according to FIG. 1.
  • FIG. 3 illustrates the physical elements of the embodiment of the invention according to FIG. 1.
  • FIGS. 4a-4d demonstrate marker object selection factors according to one embodiment of the subject invention.
  • FIG. 5 is a graphical representation of the image composition stage according to the embodiment of the invention illustrated in FIGS. 1 and/or 2.
  • DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS
  • For a more complete understanding of the present invention and advantages thereof, reference is now made to the following description of various illustrative and non-limiting embodiments thereof, taken in conjunction with the accompanying drawings in which like reference numbers indicate like features.
  • Referring initially to Figs. 1-3, a graphical representation of one embodiment of the subject invention is illustrated therein. As such, Figs. 1-3 illustrate apparatus as well as method steps for producing a composite image comprised of a combination of a virtual object and a real scene image. In this regard, the image generation apparatus according to the subject invention, generally includes a camera 1 (e.g. digital or film) capable of capturing either still or moving images and a position determining means 3 (e.g. a GPS, alternative forms of triangulation, radar, sonar, a laser range finder, surveying equipment, etc.) located on camera 1. In addition, a marker object 5 is provided which, as will be described in detail below, is employed for establishing a reference position from which a location of a virtual object will be assigned. For this purpose, it is preferred that the precise shape and dimensions of marker object 5 be known and stored in a memory, for example.
  • In the steps of producing a composite image according to the subject invention, the position of marker object 5, once it is placed at or near the location at which a virtual object is desired to be displayed, is determined. In this regard, the position of marker object 5 can be determined manually, but is preferably determined via position determining means 7 (e.g. GPS device) located on the marker object. In addition, at least one computer 9 (having a memory device) is provided to execute various functions (which will be described below) as well as to store various fixed and collected data.
  • In order so that position determining means 3 and 7 and camera 1 can communicate with computer 9, a wireless local area network (WLAN) (not shown) is provided by which the various parts of the invention can communicate by radio waves. In less preferred embodiments, other forms of wireless communication or hard-wiring for data transmission can, of course, be used.
  • In order to perform various tasks related to the invention, computer 9 contains program code in its memory for executing various tasks including methods of image processing. By way of non-limiting example, in one embodiment, the program code (software) stored in memory is capable of detecting marker object 5 in an image captured by camera 1 (e.g. via thresholding, color detection, or any other detection method). In alternative embodiments, resident software is capable of determining size and shape values of objects in an image captured by camera 1, and is capable of comparing the size and shape values of the objects to the known size and shape of marker object 5 so that marker object 5 can be detected in the captured image. For this purpose, the canvas size (image size i.e. width and height in number of pixels) of the camera should be known, so that the scale and thus the size of the marker object in the image can be determined.
  • Furthermore, in at least one embodiment, computer 9 contains software for generating a virtual object (for generating it on site). In such an embodiment, commercially available software such as CAD-type software may be used. In alternative embodiments, however, the virtual object can be generated by a separate source (e.g. a second computer) as required. Or, the virtual object can simply be resident in computer 9's memory (having been created independently elsewhere) or be received wirelessly (e.g. by satellite or antenna).
  • The computer, of course, should contain interfaces for at least the input of video signals and the input of position determination means data (e.g. IE 1394 or USB ports).
  • In further preferred embodiments, computer 9 employs a virtual camera 11 (e.g. generated by a software or program code component such as OpenGl ™ or Renderman ™) which is capable of rendering two-dimensional images from three-dimensional virtual objects for use in composing composite images (comprising a combination of a virtual object and a real scene image captured by camera 1). Details regarding the operation of the virtual camera will be discussed at length below. Generally speaking, however, once a composite image (or images) is generated, the composite image(s) can be displayed on display device 17 (e.g. a closed circuit monitor, LCD screen, etc.) so that the image can be reviewed (e.g. such as so a director can view a filmed scene on location and re-shoot if necessary). Alternatively, the image can be stored in storage device 15 (a conventional memory device) or simply stored in the memory of computer 9. Further alternatively, composite image(s) can be transmitted via conventional transmission device 19 to a remote location (e.g. to a television studio for broadcasting via antenna or via satellite or to a remote storage location).
  • In order to render two-dimensional images from three-dimensional objects as well as produce composite images using the rendered two-dimensional images according to the methods/apparatus of the present invention, various initial steps must first be taken. In this regard, as an initial matter, the geographical site for image capture must first be selected. In an architectural or construction type example, the site which is selected would likely be the site where a real building is desired to be constructed. Conversely, it may be desired to view a virtual building at several geographical sites, so that the most desirable land can be selected for a particular construction project (e.g. prior to purchase and/or groundbreaking). Similarly, if the instant invention is being employed in the production of a television program or movie, the site which is chosen would be a site at which an event or scene from a script is desired to be filmed. In yet another example, it may be desirable to display alternate highway or railroad routes at public hearings. In such case, various prospective locations or various views of a single location can be filmed so that the impact of the selection of a particular highway or railroad route can be viewed with some degree of realism (e.g. a highway can be shown "virtually" at a hearing to traverse, alternately, both the north and south side of a mountain for comparative purposes).
  • Once the geographical site is selected, a precise "target" position (at the geographical location) at which a virtual object is desired to be displayed must be chosen. Furthermore, characteristics of the chosen geographical site should be observed, and a marker object appropriate for such location selected. In this regard, for the purpose of facilitating detection of the marker object, it is preferred that the object have a uniform geometric shape (e.g. a sphere, square, or triangle) and that the object have a coloration and/or texture which is as distinct as possible from the surroundings/scene that it will be placed in. For example, if the predominant coloration of a location is green (e.g. a grassy field), it would be less desirable to select a green marker object than it would be to select a red one (the objective being, of course, for marker object 5 to contrast well with its surroundings for ease of detection). Similarly, if the marker object is placed in a scene overlooking a city exhibiting various square or rectangular-type shapes (buildings, windows, etc.), it would be more advantageous to select a spherical marker object than it would be to select a cubic or rectangular one. The skin or texture of the marker object should be selected with similar factors taken into account. Figs.4a-4d illustrate these general principals with regards at least to the contrast of marker object 5 with respect to it surroundings in a given scene. As will be noted from Figs. 4a and 4b, a darkly colored marker object is difficult to detect against a dark background (Fig. 4a), but is easy to detect against a light background (Fig. 4b). The converse effect may seen in Figs. 4c and 4d.
  • Similarly, if more than one marker object is used (e.g. such as if the position of the camera has to be computed directly from known marker positions), the appearances of the different marker objects must be distinct, either in shape or surface, so that they can be detected individually.
  • As an additional marker object selection guideline, it is noted that it is typically desirable for the marker object to be as large as is practically possible. In this regard, employing a large-type marker object makes the detection and data collection steps easier and less error prone (i.e. extrapolation errors are minimized as the marker object gets larger). It is contemplated, however, that marker objects as small as ping pong balls (or smaller) can be used with success and the use of objects of that size may be desirable in some circumstances (e.g. in the foreground if the location of the target position is inaccessible or remote).
  • Although the above guidelines for choosing a marker object have been provided to satisfy the "best mode" requirements of the patent laws, it is noted that many other appearances, sizes, and shapes of marker objects can be successfully employed. The above guidelines have simply been provided so that computer 9, employing its software, can most readily detect the marker object.
  • Referring now to Figs. 3 and 5, once the marker object appearance is chosen and the target position for the marker object is selected, marker object 5 is placed at the target location and the precise position of marker object 5 is determined. As aforesaid, this position determination may be made manually or with position determining means 7. Although position determining means 7 is preferably a GPS-type unit (i.e. because they are inexpensive, sufficiently accurate, usable universally geographically), various other mechanisms may be employed such as alternative forms of triangulation, radar, sonar, a laser range finder, surveying equipment, etc. It is noted, that current GPS systems are subject to random data errors or data aberrations (conventionally +/- 3 meters). However, employing apparatus and methods according to the subject invention described herein, such errors are overcome and are not of sufficient magnitude to prevent successful image composition (i.e. a possible position error of 3 meters at a distance of 100 meters allows an error tolerance of 3%). Furthermore, more precise location determination means are available, and superior devices will likely be created in the future.
  • In alternative embodiments, the position of marker object 5 can be automatically determined at the instant at which camera 1 captures an image (i.e. position determining means 7 can be programmed to record position information e.g. with a timestamp when camera 1 is triggered).
  • After marker object 5 has been placed at the target location, and the position of marker object 5 has been determined, camera 1 (see Fig. 3) is used to compose and capture an image or series of images (e.g. still or moving, digital or film) of a scene containing marker object 5 (see Fig. 5, which illustrates marker object 5 placed in a natural scene). Images are, of course, composed according to the commercial or artistic purpose of the desired final composite image(s). If it is desired that the virtual object be animated or that the virtual object appear at multiple target positions within a scene, object 5 is simply moved or repositioned and subsequent position determinations (with position determination means 7) made as each image or series of images are taken.
  • As an image is taken, it is digitized or scanned in order to allow computational calculations. Therefore, information about the digital image is produced by the image capturing device and transmitted or stored in an image file format (such as TIFF, JPEG ™ or specific information in individual movie frames). This information is determined through camera 1 (e.g. logical size in width and height defined in pixel, the amount of information in bits each pixel has e.g. color depth, transparency, 24-bit RGB or 48-bit RGBA, etc.) and is transferred to computer 9. In addition, digital image file formats provide information about the order of pixels (most importantly the starting point and scanning direction), which is used during subsequent image processing calculations. It is noted that if a film-type image (or images) is taken, the image must be scanned with a conventional scanner (not shown) in order to produce a digitized image and entered into computer 9.
  • As each image is taken, the position of the camera is measured and/or calculated and then stored in association with its respective image(s). Although the position of camera 1 can be determined manually (e.g. with sophisticated maps or by triangulation from three, known position marker objects), it is preferred that position determination means 3 be used (e.g. GPS, alternative triangulation means, radar, sonar, a laser range finder, surveying equipment, etc.). A particularly useful device for this purpose is a Bluetooth™-GPS device like the Fortuna Bluetooth GPSmart ™ (hard wiring the localization device can result in reduced costs and higher speed and accuracy, however). If camera 1 is communicably connected to computer 9, images as well as positional information can be transmitted to and stored in computer 9's memory as each image and/or position determination is recorded i.e. in real-time.
  • Once at least one image containing marker object 5 (as well as other desired natural or man made features) has been captured and the image and target position information has been transferred to computer 9 (e.g. via WLAN or IE 1394), data processing and the rendering of virtual object 10 can occur. Although virtual object 10 may be generated prior to image capture (e.g. via computer 9, or any other computer with conventional 3-D generation software such as CAD-type software, etc.), it is also possible to generate virtual object 10 on site, or to "receive" the virtual object by transmission (e.g. from a television studio transmission by antenna or by satellite). It is further noted that this step can be performed in an order different than the order described herein and that such order can be selected according to various needs which will be determined by the user as desired.
  • In the next step, program code (software) is operated on computer 9 to "detect" marker object 5 on the captured two-dimensional image using methods of image processing. Although no precise method or software for detecting the image is required, certain known qualities of the marker object (previously recorded and stored in computer 9) can be "searched" for by the software employed. For example, in order to detect these qualities, color thresholding and contour-detection can be used. Because the marker objects and scenes are highly variant, all or only a few qualities of marker object 5 may be recorded and used (e.g. size, shape, and skin texture, etc.).
  • Once marker object 5 has been detected, computer 9 (via its software) determines the position of marker point "MP" on marker object 5. Although marker point "MP" may be any point on marker object 5 which is desired, it is typically convenient and efficient to select a easily identifiable marker point such as the center point of the marker object (particularly if such object is a sphere). Alternatively, marker point "MP" may be the corner of a cube for example. Using the center point of a sphere, however (see Fig. 5), because the position and size (i.e. diameter) of marker object 5 is known (and has been pre-input into computer 9), the precise position of the marker point can be determined with relative ease (within certain degrees of error depending on the integrity of the acquired positional data). In image rendering, marker point "MP" is the value which is used as a reference position to determine the position in the image at which virtual object 10 will be oriented. It is noted in this regard, that virtual object 10 can be displayed at this reference position "MP", or at a position distant from it. However, marker point "MP" is the reference position from which the final position of virtual object 10 should be calculated/determined.
  • Furthermore, if desired, appearance qualities of marker object 5 (as it appears in the captured image) can be detected and compared to reference values (color, texture, or shading) related to the appearance of the marker object stored in computer 9. If any differences are detected (e.g. due to lighting conditions, etc.), data regarding these differences is stored and can be employed during the rendering of the virtual object to provide an additional degree of realism to the final composite image.
  • Referring still to Fig. 5, once scale and marker point "MP" position information have been determined, if virtual object 10 has been defined, in the next step, an image can be created by viewing this object. By way of non-limiting example, then, virtual object 10 begins as data representing a three-dimensional object created according to pre-selected criteria. For example, if an architect is desirous of displaying a digital rendering of a particular building design, data representing virtual object 10 will be generated according to specifications for the envisioned building. Similarly, if a movie director wishes to render a fictitious character, a specialist programmer will generate data representing the appearance of the character according to the director's specifications (e.g. from a storyboard or other drawing). Data related to the virtual object may stored on computer 9 (e.g. having been created thereon with CAD software) or may be imported from another source (e.g. from another computer or storage media on site) and input therein. It is further noted that in preferred embodiments, virtual object 10 is editable and can be modified in all dimensions and qualities (e.g. size, color, etc.). In still further preferred embodiments, virtual object 10 is editable on location and/or in real-time (e.g. employing accelerated graphics).
  • In order to render virtual object 10, computer 9 (see Fig. 3) employs a virtual camera 11 for projecting an image of the virtual object in two dimensions. In order to do this meaningfully, however, this computational virtual camera must be synchronized with camera 1 (see Figs. 2 and 5). Specifically, virtual camera 11 should be positioned virtually so that at least its viewing direction (or orientation) and canvas size are synchronized with that of camera 1. In addition, camera calibration occurs and object attributes are rendered. Synchronization ensures that when virtual object 10 is rendered as a two-dimensional image (e.g. so that it can be displayed on a flat screen or monitor), the image that is produced is displayed from the perspective of camera 1 at the time of image capture.
  • Therefore, when synchronized, the position of camera 1 (the real camera) and virtual camera 11 are aligned at identical coordinates in the same three-dimensional coordinate system (e.g. having values such as x, y, and z). It is noted, that synchronizing coordinates implies the synchronization of different coordinate-systems (e.g. coordinates from the GPS are transformed to coordinates in the virtual model-space). The position of marker point "MP' is then synchronized in the same manner.
  • As a first step of camera synchronization, the canvas size which is to be rendered by the virtual camera should be matched to the "real" canvas size of the image captured by camera 1. In order to accomplish this step, computer 9 should be supplied with data regarding camera 1's canvas size. Canvas size in this sense can be defined as the logical size in width and height, defined in pixels, of the captured image. This can be easily determined using image information in the digital image (i.e. the width and height in pixels) or is known by referring to commercial standards (e.g. VGA, 640x480).
  • Then, as one of the most important issues regarding this invention, the viewing direction of camera 1 must be determined in order to apply it to virtual camera 11 for synchronization. In particular, viewing direction can be calculated from the position of camera 1 relative to the position of marker object 5 i.e. at the instant at which camera 1 was used to capture the relevant image of marker object 5. In this regard, the viewing direction is the imaginary axis extending between the viewpoint and the eyepoint.
  • After determining the canvas size (or image pane) according to the above guidelines (in the first step of synchronizing the virtual camera to real camera 1), the virtual camera should be virtually positioned at the position of camera 1. When positioned as such, and thus synchronized, the position of camera 1 and camera 11 are aligned at identical coordinates in the same three-dimensional coordinate system (e.g. a coordinate system having values such as x, y, and z). It is noted that synchronizing coordinates implies the synchronization of different coordinate system (e.g. coordinates from the GPS are transformed to coordinates in the model-space). The position of marker point "MP" is then synchronized in the same manner.
  • Next, an additional step is necessary to synchronize the viewing direction of virtual camera 11. In this step, camera 11 should be aligned so that it is aimed along camera 1's viewing direction. Using the marker point "MP" as viewpoint (without correction) centers the marker point virtually in the middle of the image.
  • In order to determine the exact viewing direction, then, additional information about the orientation of camera 1 is used. This information is provided by the captured image and the location devices. Therefore, to determine the viewing direction relative to the marker point, the correct viewing direction is calculated using the deviation of the marker point detected in the image of the real scene according to the center of that image as deviation of the viewing direction of the real scene and the viewing direction as if camera 1 is directly aimed at the marker point "MP". The calculation is based on triangulation using the deviation in pixels in the captured image as well as the known (calculated and/or measured) distance between position of camera 1 and marker 5, and the field of view.
  • With this information, the orientation of the cameras in both scenes, real and virtual, is synchronized.
  • As a side note, it is recognized that it is important that the orientation or viewing direction of camera 1 be determined using marker objects (e.g. marker object 5). Although it is technically possible to use an alternative device or devices to calculate/determine camera 1's orientation/viewing direction, using marker objects provides the advantage of implicit error correction. Thus, using a marker object (in an image) to calculate camera orientation typically always results in usable data (even if errors were otherwise introduced). In other words, using marker objects is self-correcting, in effect, for certain types of measurement errors and is therefore the preferred mode of orientation/angle determination.
  • If the field of view of camera 1 is known (from knowledge of camera 1's optics, etc.), this value can be assigned to the virtual camera to achieve synchronization of canvas sizes (it is noted that field of view is typically more useful than focal length primarily because focal length is normally defined in relation to a specific image pane i.e. and image pane values are often not known). However, it is important to note that it is not mandatory that the optical system of camera 1 (and thus its field of view) be known and that alternative methods for determining field of view can be employed. If field of view is not known, then, the field of view (or virtual focal length of camera 1 according to canvas size) can be determined from the size of detected marker object 5 and the known (i.e. measured or calculated) distance between camera 1 and marker 5. This is particularly important when using a camera 1 which has a continuous zoom lens. In such a case, it is difficult to know what field of view is being employed for a particular degree of "zoom". Therefore, in such cases, field of view can be calculated from the detected size of detected marker object 5 (in the image of the real scene), the distance between the marker object and the camera, the known real size of the marker object, and the size of the image (i.e. the size of camera 1's canvas) via simple triangulation. However, if field of view is calculated from marker object values, it is helpful if a relatively large marker object is employed to reduce extrapolation errors (e.g. a marker object should be approximately 1/20th of the size of the object being simulated).
  • Referring still to Fig. 5, after virtual object 10 has been rendered (i.e. on an intermediate virtual canvas in computer 9's memory), a final composite image (containing virtual object 10 overlayed in a real/natural scene image) can be composed. In this regard, once virtual object rendering has occurred, computer 9 simply merges the original image capture (i.e. the natural or real scene image) with the created digital image of virtual object 10. With proper synchronization and image combining, virtual object 10 will be displayed in the captured image at the appropriate scale, location, and orientation desired. Figs. 2 and 5 illustrate these final steps which are performed by software resident on computer 9. Once the composite image is generated, the image can be stored on computer 9, displayed on a monitor (e.g. display device 17), saved to a separate storage device 15, or transmitted to a remote location (e.g. a television studio) via transmission device 19 (e.g. by satellite, radiowaves, etc.) (see Fig. 1 or 2).
  • In preferred embodiments, after virtual object 10 has been rendered (or during it's rendering), the appearance of the image of virtual object 10 can be edited (i.e. its appearance is tailorable to specific conditions). More specifically, data collected relating to the appearance of marker object 5 in live conditions can be employed to tailor the appearance of virtual object 10 as it will appear in the composite image.
  • For example, although some information about virtual object 10 is known prior to its rendering (e.g. shape, general color, etc.), data acquired during previous steps (relating to marker object 5's depth of color saturation, shading, brightness, etc. as detected in the captured image) may be employed to "fine tune" the virtual objects appearance so that it can be assimilated into a composite image as seamlessly as possible (e.g. for the greatest degree of realism or other desired factor). This tailorability can be performed automatically with computer 9 software (prior to, during, or after virtual object rendering), or manually as desired.
  • If two or more cameras are provided, ist is possible to get information of extended sceneries, e. g. a location that is extended with dimensions of hundreds of meters or even kilometers. That might be helpful e. g. for creating long virtual constructions like bridges crossing a landscape that is filmed by a plurality of cameras 1. Each of the cameras 1 must be provided with a device for determination of its current position, e. g. GPS. It is essential to see at least one of the markers at any time, so that position and eyepoint of each of the cameras 1 can be calculated by the computer 9 in each of the captured pictures. By using a plurality of markers it ist also possible to film fastly moved film sequences, e. g. races with cars, ships, aircrafts and so on.
  • It might also be possible, if the position of a marker can be determined fastly and exactly enough, to move a marker simultaneously with the movement of a camera. Therefore the marker can have its own motor or can be driven by an external device.
  • In each case the exact determination of the marker's and the camera's current position is essential for good results that make sure to get a good adaption of the virtual object.
  • Once given the above disclosure, many other features, modifications, and improvements will become apparent to the skilled artisan. Such other features, modifications, and improvements are therefore considered to be part of this invention, the scope of which is to be determined by the following claims:

Claims (25)

  1. A method of rendering a composite image including at least a virtual object displayed in a real scene image, said method comprising:
    determining size and shape qualities of a marker object;
    locating said marker object at a location at or near a location at which a virtual object is desired to be virtually placed;
    determining a position of said marker object;
    employing an image capturing means to capture at least one image of a real scene containing said marker object;
    determining a position of said image capturing means when said at least one image of a real scene is captured;
    analyzing said image of a real scene to detect said marker object;
    rendering a virtual object by synchronizing a virtual camera with said image capturing means, synthesizing a canvas having a size and shape corresponding to a size and shape of said image of a real scene, and projecting said virtual object onto said canvas so that said virtual object is displayed on said canvas at an orientation determined with reference to said position of said marker object in said image of a real scene;
    composing a composite image comprising said image of a real scene and said virtual object.
  2. A method according to claim 1 further comprising determining at least one reference value of said marker object relating to color, texture, or shading prior to said image capturing step;
       detecting a color, texture, or shading value of said marker object after said marker object is captured in said at least one image;
       comparing said detected value to said reference value; and
       rendering an appearance of said virtual object in said composite image in accordance with a difference between said detected value and said reference value thereby to tailor an appearance of said virtual object according to live conditions present at said location of said marker object at a time of said at least one image capture.
  3. A method according to claim 2 wherein said appearance of said virtual object is modified in color, color saturation, texture, or shading according to said difference between said detected value and said reference value.
  4. A method according to claim 2 wherein said appearance of said virtual object is modified in real time.
  5. A method according to claim 1 wherein said virtual object is three-dimensional.
  6. A method according to claim 5 further comprising orienting said virtual object according to a relationship of said position of said image capturing means to said position of said marker object.
  7. A method according to claim 6 further comprising determining a viewing direction of said image capturing means at a time of said at least one image capture and virtually viewing said virtual object according to said viewing direction.
  8. A method according to claim 6 wherein said image capturing means is capable of capturing moving images, and said marker object is portable and moveable during image capture; and wherein said virtual object is animatable in said composite image in a manner corresponding to movements of said marker object.
  9. A method according to claim 6 wherein said virtual object is renderable in real time.
  10. A method according to claim 9 wherein said virtual object is animatable in real time.
  11. A method according to claim 6 wherein said virtual object is editable.
  12. A method according to claim 2 further comprising determining aesthetic qualities of at least a second marker object;
       locating said at least second marker object at or near at least a second location corresponding to a location at which at least a second virtual object is desired to be displayed;
       determining a position of said at least second marker object;
       differentiating said at least second marker object from said first marker object according to said aesthetic qualities; and
       analyzing said image of a real scene to detect said at least second marker object, and detecting a scale of said at least second marker object in said image of a real scene;
       rendering at least a second virtual object including sizing said at least second virtual object according to said scale of said at least second marker object, and virtually locating said at least second virtual object on a canvas at a position determined with reference to said position of said at least second marker object in said image of a real scene;
       composing a composite image comprising said image of a real scene and said at least second virtual object.
  13. Apparatus for rendering a composite image including at least a virtual object displayed in a real scene image, said apparatus comprising:
    a marker object of known size and shape and located at known coordinates;
    an image capturing means located at known coordinates;
    a computer in communication with said image capturing means, and tangibly embodying a program of instructions executable by said computer to perform steps comprising:
    detecting said marker object when displayed in an image captured by said image capturing means;
    rendering a virtual object by synchronizing a virtual camera with said image capturing means, synthesizing a canvas having a size and shape corresponding to a size and shape of said image of a real scene, and projecting said virtual object onto said canvas so that said virtual object is displayed on said canvas at an orientation determined with reference to said position of said marker object in said image of a real scene;
    composing a composite image comprising a combination of said virtual object and a real scene image captured by said image capturing means.
  14. Apparatus according to claim 13 wherein said steps further comprise:
    detecting a color, texture, or shading value of said marker object after said marker object is captured in at least one image by said image capturing means;
    comparing said detected value to a color, texture, or shading reference value of said marker object; and
    rendering an appearance of said virtual object in said composite image in accordance with a difference between said detected value and said reference value thereby to tailor an appearance of said virtual object according to live conditions present at a location of said marker object at a time of said at least one image capture.
  15. Apparatus according to claim 14 wherein said steps further comprise modifying an appearance of said virtual object with regard to color, color saturation, texture, or shading according to said difference between said detected value and said reference value.
  16. Apparatus according to claim 14 wherein said image capturing means is a video camera capable of capturing moving images and said marker object is portable.
  17. Apparatus according to claim 14 further including at least one position determining means for determining a position of said image capturing means and for determining a position of said marker object, wherein said at least one position determining means and said image capturing means are communicably connected to said computer.
  18. Apparatus according to claim 14 further including viewing direction determining means for determining a viewing direction of said image capturing means at a time of said at least one image capture and including means for orienting said virtual object according to said viewing direction.
  19. Apparatus according to claim 17 wherein said first and second position determining means comprise global positioning devices.
  20. Apparatus according to claim 14 wherein said virtual object is three-dimensional and manually editable.
  21. Apparatus according to claim 13 wherein said second position determining means comprises at least three objects having known positions, and wherein a position of said image capturing means is determinable by triangulation from said at least three objects.
  22. Apparatus according to one of the claims 13 to 21 wherein two or more image capturing means are using the same said one or more mareker object(s).
  23. A method according to one of the claims 1 to 11 wherein two or more image capturing means are using the same said one or more marker object(s).
  24. A method according to claim 11 wherein said virtual object is editable on location.
  25. A method according to claim 24 wherein said virtual object is editable on location and in real time.
EP04018582A 2003-08-15 2004-08-05 Method and apparatus for producing composite images which contain virtual objects Withdrawn EP1507235A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/641,156 US7391424B2 (en) 2003-08-15 2003-08-15 Method and apparatus for producing composite images which contain virtual objects
US641156 2003-08-15

Publications (1)

Publication Number Publication Date
EP1507235A1 true EP1507235A1 (en) 2005-02-16

Family

ID=33565274

Family Applications (1)

Application Number Title Priority Date Filing Date
EP04018582A Withdrawn EP1507235A1 (en) 2003-08-15 2004-08-05 Method and apparatus for producing composite images which contain virtual objects

Country Status (2)

Country Link
US (2) US7391424B2 (en)
EP (1) EP1507235A1 (en)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1887526A1 (en) * 2006-08-11 2008-02-13 Seac02 S.r.l. A digitally-augmented reality video system
EP2113881A1 (en) * 2008-04-29 2009-11-04 Holiton Limited Image producing method and device
DE102004061841B4 (en) * 2003-12-22 2010-09-16 Metaio Gmbh Markerless tracking system for augmented reality applications
DE102009056013A1 (en) 2009-09-09 2011-03-10 Volkswagen Ag Method for generating augmented-reality image, involves receiving actual image of actual environment by camera
DE102009040848A1 (en) 2009-10-05 2011-04-07 Volkswagen Ag Augmented-reality-images generating method for determining relative position between sensor and surface of object e.g. component of land vehicle, involves providing optical time variable marker with display for displaying information
DE102011122206A1 (en) 2011-12-23 2013-06-27 Volkswagen Aktiengesellschaft Method for representation of virtual image component i.e. augmented reality image, on transparent display of augmented reality system, involves determining position of component, and representing virtual image component by display
WO2013170871A1 (en) * 2012-05-14 2013-11-21 Abb Research Ltd Method and industrial control system for industrial process equipment maintenance
DE102012014995A1 (en) 2012-07-28 2014-01-30 Volkswagen Aktiengesellschaft Method for generating augmented reality image and/or representing virtual image component on transparent display, involves producing synthetic image of object from surrounding area to be recorded using CAD data
EP2779102A1 (en) * 2013-03-12 2014-09-17 E.sigma Systems GmbH Method of generating an animated video sequence
GB2516691A (en) * 2013-07-30 2015-02-04 Bifold Fluidpower Ltd Visualisation method
EP2546806A3 (en) * 2011-07-11 2017-06-28 Deutsche Telekom AG Image based rendering for AR - enabling user generation of 3D content
EP2397994A3 (en) * 2010-06-15 2017-07-26 Nintendo Co., Ltd. Information processing system for superimposing a virtual object on a real space correcting deviations caused by error in detection of marker in a photographed image.
CN110517355A (en) * 2018-05-22 2019-11-29 苹果公司 Environment for illuminating mixed reality object synthesizes

Families Citing this family (89)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8458028B2 (en) * 2002-10-16 2013-06-04 Barbaro Technologies System and method for integrating business-related content into an electronic game
SE0203908D0 (en) * 2002-12-30 2002-12-30 Abb Research Ltd An augmented reality system and method
US7019742B2 (en) * 2003-11-20 2006-03-28 Microsoft Corporation Dynamic 2D imposters of 3D graphic objects
JP4356983B2 (en) * 2004-02-10 2009-11-04 キヤノン株式会社 Image processing method and image processing apparatus
US7868904B2 (en) * 2005-04-01 2011-01-11 Canon Kabushiki Kaisha Image processing method and image processing apparatus
JP4914123B2 (en) * 2005-07-15 2012-04-11 キヤノン株式会社 Image processing apparatus and method
US8797352B2 (en) * 2005-08-09 2014-08-05 Total Immersion Method and devices for visualising a digital model in a real environment
JP4847203B2 (en) * 2006-04-27 2011-12-28 キヤノン株式会社 Information processing method and information processing apparatus
JP2010519860A (en) * 2007-02-21 2010-06-03 ピクセル ベロシティー,インク. Scalable system for wide area monitoring
JP4857196B2 (en) * 2007-05-31 2012-01-18 キヤノン株式会社 Head-mounted display device and control method thereof
US9013505B1 (en) * 2007-11-27 2015-04-21 Sprint Communications Company L.P. Mobile system representing virtual objects on live camera image
US9058764B1 (en) * 2007-11-30 2015-06-16 Sprint Communications Company L.P. Markers to implement augmented reality
US20090153550A1 (en) * 2007-12-18 2009-06-18 Disney Enterprises, Inc. Virtual object rendering system and method
KR100912264B1 (en) * 2008-02-12 2009-08-17 광주과학기술원 Method and system for generating user-responsive augmented image
US8542906B1 (en) 2008-05-21 2013-09-24 Sprint Communications Company L.P. Augmented reality image offset and overlay
JP4579316B2 (en) * 2008-06-30 2010-11-10 任天堂株式会社 IMAGING DEVICE, IMAGING SYSTEM, AND GAME DEVICE
US8379056B2 (en) * 2009-02-27 2013-02-19 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Device and method for providing a video signal of a virtual image
US8314832B2 (en) 2009-04-01 2012-11-20 Microsoft Corporation Systems and methods for generating stereoscopic images
US8509479B2 (en) 2009-05-29 2013-08-13 Microsoft Corporation Virtual object
US9129644B2 (en) * 2009-06-23 2015-09-08 Disney Enterprises, Inc. System and method for rendering in accordance with location of virtual objects in real-time
US9286720B2 (en) * 2009-08-20 2016-03-15 Northrop Grumman Systems Corporation Locative video for situation awareness
KR101184876B1 (en) * 2010-02-11 2012-09-20 삼성전자주식회사 Apparatus and method for creating character's dynamic effect related to image contents
JP5570284B2 (en) * 2010-04-16 2014-08-13 キヤノン株式会社 Image processing apparatus and method
US8411092B2 (en) 2010-06-14 2013-04-02 Nintendo Co., Ltd. 2D imposters for simplifying processing of plural animation objects in computer graphics generation
KR101330811B1 (en) * 2010-08-25 2013-11-18 주식회사 팬택 Apparatus and Method for augmented reality using instant marker
JP5250598B2 (en) * 2010-10-05 2013-07-31 株式会社ソニー・コンピュータエンタテインメント Image display device and image display method
JP5216834B2 (en) * 2010-11-08 2013-06-19 株式会社エヌ・ティ・ティ・ドコモ Object display device and object display method
JP5674441B2 (en) * 2010-12-02 2015-02-25 新日鉄住金ソリューションズ株式会社 Information processing system, control method thereof, and program
KR101669119B1 (en) * 2010-12-14 2016-10-25 삼성전자주식회사 System and method for multi-layered augmented reality
JP5671349B2 (en) * 2011-01-06 2015-02-18 任天堂株式会社 Image processing program, image processing apparatus, image processing system, and image processing method
US8866898B2 (en) 2011-01-31 2014-10-21 Microsoft Corporation Living room movie creation
JP5799521B2 (en) * 2011-02-15 2015-10-28 ソニー株式会社 Information processing apparatus, authoring method, and program
JP5691632B2 (en) * 2011-02-24 2015-04-01 株式会社大林組 Image composition method
US20120259744A1 (en) * 2011-04-07 2012-10-11 Infosys Technologies, Ltd. System and method for augmented reality and social networking enhanced retail shopping
US8817046B2 (en) * 2011-04-21 2014-08-26 Microsoft Corporation Color channels and optical markers
JP2012243147A (en) * 2011-05-20 2012-12-10 Nintendo Co Ltd Information processing program, information processing device, information processing system, and information processing method
US9155964B2 (en) * 2011-09-14 2015-10-13 Steelseries Aps Apparatus for adapting virtual gaming with real world information
JP2013127774A (en) * 2011-11-16 2013-06-27 Canon Inc Image processing device, image processing method, and program
WO2013075316A1 (en) * 2011-11-24 2013-05-30 Microsoft Corporation Interactive multi-modal image search
US8606645B1 (en) 2012-02-02 2013-12-10 SeeMore Interactive, Inc. Method, medium, and system for an augmented reality retail application
US9384711B2 (en) 2012-02-15 2016-07-05 Microsoft Technology Licensing, Llc Speculative render ahead and caching in multiple passes
US20130215229A1 (en) * 2012-02-16 2013-08-22 Crytek Gmbh Real-time compositing of live recording-based and computer graphics-based media streams
US9578226B2 (en) 2012-04-12 2017-02-21 Qualcomm Incorporated Photometric registration from arbitrary geometry for augmented reality
GB201208088D0 (en) 2012-05-09 2012-06-20 Ncam Sollutions Ltd Ncam
US9286122B2 (en) 2012-05-31 2016-03-15 Microsoft Technology Licensing, Llc Display techniques using virtual surface allocation
US9230517B2 (en) 2012-05-31 2016-01-05 Microsoft Technology Licensing, Llc Virtual surface gutters
US9235925B2 (en) * 2012-05-31 2016-01-12 Microsoft Technology Licensing, Llc Virtual surface rendering
US9177533B2 (en) 2012-05-31 2015-11-03 Microsoft Technology Licensing, Llc Virtual surface compaction
US10380469B2 (en) * 2012-07-18 2019-08-13 The Boeing Company Method for tracking a device in a landmark-based reference system
US9429912B2 (en) * 2012-08-17 2016-08-30 Microsoft Technology Licensing, Llc Mixed reality holographic object development
US9215368B2 (en) * 2012-12-02 2015-12-15 Bachir Babale Virtual decals for precision alignment and stabilization of motion graphics on mobile video
WO2014107681A1 (en) * 2013-01-04 2014-07-10 Awyse, Inc. System and method for providing augmented reality on mobile devices
US9336629B2 (en) 2013-01-30 2016-05-10 F3 & Associates, Inc. Coordinate geometry augmented reality process
US9104298B1 (en) * 2013-05-10 2015-08-11 Trade Only Limited Systems, methods, and devices for integrated product and electronic image fulfillment
US9307007B2 (en) 2013-06-14 2016-04-05 Microsoft Technology Licensing, Llc Content pre-render and pre-fetch techniques
US10139623B2 (en) * 2013-06-18 2018-11-27 Microsoft Technology Licensing, Llc Virtual object orientation and visualization
JP2015088096A (en) * 2013-11-01 2015-05-07 株式会社ソニー・コンピュータエンタテインメント Information processor and information processing method
JP6353214B2 (en) 2013-11-11 2018-07-04 株式会社ソニー・インタラクティブエンタテインメント Image generating apparatus and image generating method
US9530250B2 (en) * 2013-12-10 2016-12-27 Dassault Systemes Augmented reality updating of 3D CAD models
US9191620B1 (en) 2013-12-20 2015-11-17 Sprint Communications Company L.P. Voice call using augmented reality
US8976191B1 (en) 2014-03-13 2015-03-10 Qualcomm Incorporated Creating a realistic color for a virtual object in an augmented reality environment
US20160093000A1 (en) * 2014-09-25 2016-03-31 Collectrium Inc. Novel cloud-based art inventory management system, incorporating techniques, methods and devices
JP6317854B2 (en) * 2015-03-30 2018-04-25 株式会社カプコン Virtual three-dimensional space generation method, video system, control method thereof, and recording medium readable by computer device
US10810797B2 (en) * 2015-05-22 2020-10-20 Otoy, Inc Augmenting AR/VR displays with image projections
WO2017110071A1 (en) * 2015-12-25 2017-06-29 パナソニックIpマネジメント株式会社 Size measurement device, delivery box apparatus, and size measurement method
US10845188B2 (en) 2016-01-05 2020-11-24 Microsoft Technology Licensing, Llc Motion capture from a mobile self-tracking device
MX2018010282A (en) * 2016-02-25 2018-12-19 Dainippon Printing Co Ltd Three-dimensional shape data and texture information generation system, photographing control program, and three-dimensional shape data and texture information generation method.
US10685490B2 (en) * 2016-03-10 2020-06-16 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and storage medium
US11062383B2 (en) 2016-05-10 2021-07-13 Lowe's Companies, Inc. Systems and methods for displaying a simulated room and portions thereof
US10157408B2 (en) 2016-07-29 2018-12-18 Customer Focus Software Limited Method, systems, and devices for integrated product and electronic image fulfillment from database
JP2018077644A (en) * 2016-11-08 2018-05-17 富士ゼロックス株式会社 Information processing system and program
US9754397B1 (en) * 2017-04-07 2017-09-05 Mirage Worlds, Inc. Systems and methods for contextual augmented reality sharing and performance
WO2018231207A1 (en) * 2017-06-13 2018-12-20 Mario Iobbi Visibility enhancing eyewear
US10248971B2 (en) 2017-09-07 2019-04-02 Customer Focus Software Limited Methods, systems, and devices for dynamically generating a personalized advertisement on a website for manufacturing customizable products
EP3721418A1 (en) * 2017-12-05 2020-10-14 Diakse Method of construction of a computer-generated image and a virtual environment
US10192115B1 (en) 2017-12-13 2019-01-29 Lowe's Companies, Inc. Virtualizing objects using object models and object position data
US10657721B2 (en) 2018-02-09 2020-05-19 Paccar Inc Systems and methods for providing augmented reality support for vehicle service operations
WO2019156682A1 (en) * 2018-02-09 2019-08-15 Yin David Wei Cheng Systems and methods for providing augmented reality support for vehicle service operations
CN108510597A (en) * 2018-03-09 2018-09-07 北京小米移动软件有限公司 Edit methods, device and the non-transitorycomputer readable storage medium of virtual scene
US10504288B2 (en) 2018-04-17 2019-12-10 Patrick Piemonte & Ryan Staake Systems and methods for shared creation of augmented reality
US10818093B2 (en) 2018-05-25 2020-10-27 Tiff's Treats Holdings, Inc. Apparatus, method, and system for presentation of multimedia content including augmented reality content
US10984600B2 (en) 2018-05-25 2021-04-20 Tiff's Treats Holdings, Inc. Apparatus, method, and system for presentation of multimedia content including augmented reality content
KR102173724B1 (en) * 2018-06-14 2020-11-03 주식회사 넥슨코리아 Apparatus, method and computer program for game service
US10929670B1 (en) 2019-10-21 2021-02-23 The Boeing Company Marker-to-model location pairing and registration for augmented reality applications
US11087525B2 (en) 2020-01-08 2021-08-10 International Business Machines Corporation Unsupervised learning of three dimensional visual alphabet
JP7474338B2 (en) * 2020-01-23 2024-04-24 クロー・バーチャル・ファッション・インク Method and apparatus for automatically sewing garment pattern pieces
US11900521B2 (en) 2020-08-17 2024-02-13 LiquidView Corp Virtual window apparatus and system
US11948257B2 (en) * 2022-05-09 2024-04-02 Rovi Guides, Inc. Systems and methods for augmented reality video generation
CN116320364B (en) * 2023-05-25 2023-08-01 四川中绳矩阵技术发展有限公司 Virtual reality shooting method and display method based on multi-layer display

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0838787A2 (en) * 1996-10-16 1998-04-29 HE HOLDINGS, INC. dba HUGHES ELECTRONICS A multi-user real-time virtual reality system and method
US6064749A (en) * 1996-08-02 2000-05-16 Hirota; Gentaro Hybrid tracking for augmented reality using both camera motion detection and landmark tracking
US20020094189A1 (en) * 2000-07-26 2002-07-18 Nassir Navab Method and system for E-commerce video editing
WO2003034705A2 (en) * 2001-10-19 2003-04-24 University Of North Carolina At Chapel Hill Methods and systems for dynamic virtual convergence and head mountable display

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2661061B1 (en) 1990-04-11 1992-08-07 Multi Media Tech METHOD AND DEVICE FOR MODIFYING IMAGE AREA.
US5815411A (en) 1993-09-10 1998-09-29 Criticom Corporation Electro-optic vision system which exploits position and attitude
US5625765A (en) 1993-09-03 1997-04-29 Criticom Corp. Vision systems including devices and methods for combining images for extended magnification schemes
US6064398A (en) 1993-09-10 2000-05-16 Geovector Corporation Electro-optic vision systems
JP2001510642A (en) * 1996-04-19 2001-07-31 スポットズーム アクチーセルスカペット Method and apparatus for processing objects in television images
GB9702636D0 (en) * 1997-02-01 1997-04-02 Orad Hi Tech Systems Limited Virtual studio position sensing system
US6166744A (en) 1997-11-26 2000-12-26 Pathfinder Systems, Inc. System for combining virtual images with real-world scenes
US6625299B1 (en) * 1998-04-08 2003-09-23 Jeffrey Meisner Augmented reality technology
US6628298B1 (en) * 1998-07-17 2003-09-30 The Regents Of The University Of California Apparatus and method for rendering synthetic objects into real scenes using measurements of scene illumination
US6330356B1 (en) 1999-09-29 2001-12-11 Rockwell Science Center Llc Dynamic visual registration of a 3-D object with a graphical model
US6570581B1 (en) 1999-10-25 2003-05-27 Microsoft Corporation On-location video assistance system with computer generated imagery overlay
US6430416B1 (en) * 1999-11-15 2002-08-06 Trimble Navigation Limited Hybrid radio location system using a combination of satellite pseudoranges and radio pseudoranges
US6903707B2 (en) * 2000-08-09 2005-06-07 Information Decision Technologies, Llc Method for using a motorized camera mount for tracking in augmented reality
US6850250B2 (en) * 2000-08-29 2005-02-01 Sony Corporation Method and apparatus for a declarative representation of distortion correction for add-on graphics in broadcast video
GB0022447D0 (en) * 2000-09-13 2000-11-01 Bae Systems Plc Measurement method
KR20020025301A (en) * 2000-09-28 2002-04-04 오길록 Apparatus and Method for Furnishing Augmented-Reality Graphic using Panoramic Image with Supporting Multiuser
WO2002073955A1 (en) * 2001-03-13 2002-09-19 Canon Kabushiki Kaisha Image processing apparatus, image processing method, studio apparatus, storage medium, and program
US7176936B2 (en) * 2001-03-27 2007-02-13 Siemens Corporate Research, Inc. Augmented reality guided instrument positioning with modulated guiding graphics
US7605826B2 (en) * 2001-03-27 2009-10-20 Siemens Corporate Research, Inc. Augmented reality guided instrument positioning with depth determining graphics
JP2004534963A (en) * 2001-03-30 2004-11-18 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Methods, systems and devices for augmented reality
US7215322B2 (en) * 2001-05-31 2007-05-08 Siemens Corporate Research, Inc. Input devices for augmented reality applications
US7274380B2 (en) * 2001-10-04 2007-09-25 Siemens Corporate Research, Inc. Augmented reality system
US7095424B2 (en) * 2001-10-26 2006-08-22 Canon Kabushiki Kaisha Image display apparatus and method, and storage medium
US20030103651A1 (en) * 2001-12-03 2003-06-05 Kurt Novak Photogrammetric apparatus
US20030227542A1 (en) * 2001-12-20 2003-12-11 Xiang Zhang Single-computer real-time stereo augmented reality system
JP4047622B2 (en) * 2002-04-26 2008-02-13 日本放送協会 Display device, lighting device, and photographing system
JP2004287699A (en) * 2003-03-20 2004-10-14 Tama Tlo Kk Image composition device and method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6064749A (en) * 1996-08-02 2000-05-16 Hirota; Gentaro Hybrid tracking for augmented reality using both camera motion detection and landmark tracking
EP0838787A2 (en) * 1996-10-16 1998-04-29 HE HOLDINGS, INC. dba HUGHES ELECTRONICS A multi-user real-time virtual reality system and method
US20020094189A1 (en) * 2000-07-26 2002-07-18 Nassir Navab Method and system for E-commerce video editing
WO2003034705A2 (en) * 2001-10-19 2003-04-24 University Of North Carolina At Chapel Hill Methods and systems for dynamic virtual convergence and head mountable display

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ABAD F ET AL: "Integrating synthetic objects into real scenes", COMPUTERS AND GRAPHICS, PERGAMON PRESS LTD. OXFORD, GB, vol. 27, no. 1, February 2003 (2003-02-01), pages 5 - 17, XP004405346, ISSN: 0097-8493 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102004061841B4 (en) * 2003-12-22 2010-09-16 Metaio Gmbh Markerless tracking system for augmented reality applications
EP1887526A1 (en) * 2006-08-11 2008-02-13 Seac02 S.r.l. A digitally-augmented reality video system
EP2113881A1 (en) * 2008-04-29 2009-11-04 Holiton Limited Image producing method and device
DE102009056013A1 (en) 2009-09-09 2011-03-10 Volkswagen Ag Method for generating augmented-reality image, involves receiving actual image of actual environment by camera
DE102009040848A1 (en) 2009-10-05 2011-04-07 Volkswagen Ag Augmented-reality-images generating method for determining relative position between sensor and surface of object e.g. component of land vehicle, involves providing optical time variable marker with display for displaying information
DE102009040848B4 (en) 2009-10-05 2023-07-20 Volkswagen Ag Procedure for operating an augmented reality system
EP2397994A3 (en) * 2010-06-15 2017-07-26 Nintendo Co., Ltd. Information processing system for superimposing a virtual object on a real space correcting deviations caused by error in detection of marker in a photographed image.
EP2546806A3 (en) * 2011-07-11 2017-06-28 Deutsche Telekom AG Image based rendering for AR - enabling user generation of 3D content
DE102011122206A1 (en) 2011-12-23 2013-06-27 Volkswagen Aktiengesellschaft Method for representation of virtual image component i.e. augmented reality image, on transparent display of augmented reality system, involves determining position of component, and representing virtual image component by display
WO2013170871A1 (en) * 2012-05-14 2013-11-21 Abb Research Ltd Method and industrial control system for industrial process equipment maintenance
DE102012014995A1 (en) 2012-07-28 2014-01-30 Volkswagen Aktiengesellschaft Method for generating augmented reality image and/or representing virtual image component on transparent display, involves producing synthetic image of object from surrounding area to be recorded using CAD data
EP2779102A1 (en) * 2013-03-12 2014-09-17 E.sigma Systems GmbH Method of generating an animated video sequence
GB2516691B (en) * 2013-07-30 2020-10-07 Bifold Fluidpower Ltd Visualisation method
GB2516691A (en) * 2013-07-30 2015-02-04 Bifold Fluidpower Ltd Visualisation method
CN110517355A (en) * 2018-05-22 2019-11-29 苹果公司 Environment for illuminating mixed reality object synthesizes
US11694392B2 (en) 2018-05-22 2023-07-04 Apple Inc. Environment synthesis for lighting an object
CN110517355B (en) * 2018-05-22 2024-03-15 苹果公司 Ambient composition for illuminating mixed reality objects

Also Published As

Publication number Publication date
US20050035980A1 (en) 2005-02-17
US7750926B2 (en) 2010-07-06
US7391424B2 (en) 2008-06-24
US20090051682A1 (en) 2009-02-26

Similar Documents

Publication Publication Date Title
US7750926B2 (en) Method and apparatus for producing composite images which contain virtual objects
US8963943B2 (en) Three-dimensional urban modeling apparatus and method
CN106327573B (en) A kind of outdoor scene three-dimensional modeling method for urban architecture
US9384277B2 (en) Three dimensional image data models
Woodward et al. Mixed reality for mobile construction site visualization and communication
Neumann et al. Augmented virtual environments (ave): Dynamic fusion of imagery and 3d models
CN107067447B (en) Integrated video monitoring method for large spatial region
EP2913796B1 (en) Method of generating panorama views on a mobile mapping system
US20120155744A1 (en) Image generation method
US20110211040A1 (en) System and method for creating interactive panoramic walk-through applications
JP4980617B2 (en) 2D drawing and video composition display device
WO2001078014A1 (en) Real world/virtual world correlation system using 3d graphics pipeline
JP2003099799A (en) Method for simulating motion of three-dimensional physical object stationary in changeless scene
Honkamaa et al. Interactive outdoor mobile augmentation using markerless tracking and GPS
WO2004008744A1 (en) Road and other flat object video plan-view developing image processing method, reverse developing image conversion processing method, plan-view developing image processing device, and reverse developing image conversion processing device
CN115641401A (en) Construction method and related device of three-dimensional live-action model
JP2023546739A (en) Methods, apparatus, and systems for generating three-dimensional models of scenes
CN114299236A (en) Oblique photogrammetry space-ground fusion live-action modeling method, device, product and medium
Gubo et al. A multi-range approach for cultural heritage survey: a case study of a medieval church in Slovakia
Ortin et al. Occlusion-free image generation for realistic texture mapping
CA2710746A1 (en) Method for displaying a virtual image
Karras et al. Generation of orthoimages and perspective views with automatic visibility checking and texture blending
Frueh Automated 3D model generation for urban environments
JP4530214B2 (en) Simulated field of view generator
Hirzinger et al. Photo-realistic 3D modelling-From robotics perception to-wards cultural heritage

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL HR LT LV MK

17P Request for examination filed

Effective date: 20050805

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

17Q First examination report despatched

Effective date: 20050909

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: LONSING, WERNER G.

RIN1 Information on inventor provided before grant (corrected)

Inventor name: LONSING, WERNER G.

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: LONSING, WERNER G.

RIN1 Information on inventor provided before grant (corrected)

Inventor name: LONSING, WERNER G.

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20131118