EP1504597A1 - Method for displaying an output image on an object - Google Patents

Method for displaying an output image on an object

Info

Publication number
EP1504597A1
EP1504597A1 EP04726647A EP04726647A EP1504597A1 EP 1504597 A1 EP1504597 A1 EP 1504597A1 EP 04726647 A EP04726647 A EP 04726647A EP 04726647 A EP04726647 A EP 04726647A EP 1504597 A1 EP1504597 A1 EP 1504597A1
Authority
EP
European Patent Office
Prior art keywords
projector
output image
markers
pose
predetermined
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP04726647A
Other languages
German (de)
French (fr)
Other versions
EP1504597B1 (en
Inventor
Ramesh Raskar
Clifton L. Forlines
Paul A. Beardsley
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Publication of EP1504597A1 publication Critical patent/EP1504597A1/en
Application granted granted Critical
Publication of EP1504597B1 publication Critical patent/EP1504597B1/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3141—Constructional details thereof
    • H04N9/3173—Constructional details thereof wherein the projection device is specially adapted for enhanced portability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3191—Testing thereof
    • H04N9/3194—Testing thereof including sensor feedback
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • This invention relates generally to digital projectors, and more particularly to a projectors used for augmented reality.
  • the field of augmented reality is concerned with enhancing what we see and hear around us in the physical world.
  • researchers in this field superimpose computer graphics onto our view of real world objects. This enhances both our perceptions and our interactions with the real world. It enables improved performance of real world tasks.
  • PDA digital personal assistant
  • the physical size of a PDA limits the information that can be presented to the user, regardless of future improvements in pixel resolution.
  • the information displayed on a PDA is in the coordinate frame of the camera, not the user's own viewpoint.
  • Use of a PDA requires the user to context-switch between the physical environment and the display.
  • a PDA must be kept either on the user's person or nearby but easily visible.
  • a PDA only provides augmentation to an individual, or at best a small group of people clustered around a veiy small device.
  • Another method of providing an augmented view of the world is through the use of 'see-through' video micro-display eyeglasses.
  • a drawback is that the tight coupling between the user's eyes coordinate frame and the displayed image coordinate frame requires very accurate processing of user head motion, e.g., a user head rotation must be accurately matched by a complementary rotation in the displayed image.
  • video micro-display eyeglasses are for individual use only, not shared use. Because the display is only a centimeter or so from the eye, working with an augmented object is awkward because other objects, such as the user's hands, can easily occlude the augmentation.
  • fatigue factors of eye- worn displays are well known.
  • Projector-based augmented reality projects augmentation directly onto the physical scene. It decouples the user coordinate frame from the relationship between the projector and the physical object.
  • One advantage of this is that there is no need for the difficult task of determining user head motion, when showing projected information.
  • a rotating movie camera has been used to acquire a film of a living room, replete with furniture, and people. The room and furniture were then painted a neutral white, and the film was projected back onto the walls and furniture using a rotating projector that was precisely registered with the original camera, see Naimark, "Displacements," Exhibit at the San Francisco Museum of Modern Art, San Francisco, CA 1984. This crucial co-location of the acquiring camera and displaying projector is common to most systems that use pre-recorded images, or image sequences to illuminate physical objects.
  • a projector and fiber-optic bundle have been used to animate the head of a fictional fortuneteller inside a real crystal ball, see U.S. Patent No.
  • the "Luminous Room” project treats a co-located camera -projector pair as an I/O bulb to sense and project imagery onto flat surfaces in the real physical surroundings of a room or a designated workspace, see Underkoffler et al. "Emancipated pixels: Real-world graphics in the luminous room," SIGGRAPH '99, pp. 385-392, 1999. Their main focus is interaction with the information via luminous and tangible interfaces. They recognized co-planar 2D physical objects, tracked the 2D positions and orientations in the plane, and projected light from overhead to reproduce the appropriate sunlight shadows.
  • the invention provides a method for uniquely identifying a target object.
  • the invention provides a method for determine the physical relationship between a projector and the object. Knowing the unique identification of the object and the relative position of the projector and the object, enables the projection of calibrated images that are related to the object.
  • the invention can be applied to a variety of systems where it is desirable to augment an object with projected images that are specific to that object.
  • Examples include information displays on objects, e.g., info ⁇ nation displays for a printer include the print queue or the expected time for jobs to complete. Info ⁇ nation displays for books could include comments that appear in the margin.
  • Training applications in which instructions are displayed on a known object.
  • Physical indexing in which a user is guided towards a requested object, e.g., by projected arrows on the ground and by highlighting of specific areas in a storage area or workspace.
  • a 'Magic Lens' to simulate transparency of an opaque surface. Projection of language specific information about an object of interest to a tourist.
  • the invention further provides means for adapting the projected content according to several factors, including the history of interaction, the position and motion of the projector, and the presence of other similar devices.
  • Figure 1 is a block diagram of a projector according to the invention.
  • Figure 2 is a block diagram of an object augmented by an image displayed by the projector of Figure 1;
  • Figure 3 is a block diagram of a method for displaying images on an object according to the invention.
  • Figure 1 shows a context aware projector 100 according to our invention. It should be noted that the projector 100 can be used in alone or in a cooperative combination with other similar projectors.
  • Our projector 100 includes a microprocessor 110, a memory 120, and an I/O interface 130 comiected by buses 140, generally a processing unit 103.
  • the processing unit is conventional in its electronic stmcture, but unconventional in its operation when performing the method steps described herein.
  • the projector also includes a projector sub-system 150 for displaying output images 102, and a camera sub-system 160 for acquiring input images 101 of a scene including uniquely identified objects.
  • a network sub-system 170 allows the projector to communicate with other similar devices, or other computing devices, local or remote. Therefore, the network system can be connected to an antenna or infrared transceiver 180, depending on the communications medium.
  • a user interface 190 can provide input and output data to facilitate operation of the projector.
  • the interface can include left, center, and right buttons as are normally found on an input mouse.
  • the interface can also include a "scroll" wheel, as are known in the art.
  • Internal sensors 195 e.g., tilt sensors or accelerometers can determine an orientation of the projector. It should be noted that the sensors are internal only, and do not sense the environment external to the projector. It should be noted that the sensors are optimal.
  • the projector 100 can perfo ⁇ n a number of functions, including smart keystone coi ⁇ ection, orientation compensated image intensities, auto brightness, zoom and focus, 3D scanning for geometry and texture capture, and smart flash for the camera sub-system 160.
  • the projector function is secondaiy to the camera function.
  • the projector merely illuminates specific areas in a scene observed by the camera.
  • additional adaptive functions are described in greater detail below.
  • a relative position of the object with respect to the projector 100 is determined from unique visual markers 211-214 attached to the object 201.
  • pose we mean the (x,y,z) position of the projector in the same coordinate frame of the object, and the orientation of the projector.
  • orientation we mean the amount of rotation about any of the three major axes.
  • the markers are circular, see enlargement, with multiple colored segments, e.g. four to six.
  • the pattern of the colors define different unique identifications, see U.S. Patent Application Sn. 09/628,461,
  • the color-coding encodes a unique identity. It should be noted that other similar markers can be used. One or more markers can be used as described below in detail.
  • the points on the object 201 are used to obtain a unique solution of camera pose with the calibrated camera sub-system 160.
  • the points are obtained from images acquired at a long range of tlie multiple markers, or from the shape of a single marker at closer-range.
  • the projector pose is known via the system calibration.
  • augmentation information retrieved from the memory 120 or downloaded via the network 210, can then be displayed.
  • the shape, size, and location where tlie image is displayed can be predetennined.
  • Important applications where context-based augmentation can be used include: information displays on objects, either passive display, or training applications in which instructions 220 are displayed as part of a sequence of steps; physical indexing where a user is guided towards a requested object; identifying the presence of, and projecting, electronic data items which have been attached to the environment.
  • Context-based augmentation does not have rigorous requirements for accuracy because small offsets in positioning of augmented info ⁇ nation are usually not of consequence. It does however put more emphasis on modes of use. For example, distance from an object can be used to adjust level-of-detail in tl e projected augmentation 220.
  • the system is calibrated so that the following information is known - the intrinsic parameters of tlie projector, the intrinsic parameters of the camera, the relative position and orientation between tlie projector and camera.
  • This calibration can be done as a pre-calibration step before the system is in operation, or can be done as start-up processing when the system begins operation. Any of tlie well known camera calibration techniques can be used.
  • tlie camera The purpose of tlie camera is to acquire the pose of the system relative to a surface of interest in the scene.
  • 'pose' is used to mean the (x, y, z) location, and orientation. This is achieved by processing an input image of the object to determine the pose of the camera, then using the calibration infonnation to infer die pose of the projector. After the pose of the projector is known, the augmentation information 220 can be projected to a desired place on the surface of interest.
  • our method 300 first acquires an input image (Ij) 301 of a scene.
  • the scene includes at least four visual markers. These visual markers are color-coded, and the color-coding specifies identity.
  • the markers can be thought of as analogous to barcodes, but based on color rather than striping.
  • the markers are detected 315 in the input image 301.
  • the markers are used to index 320 a database record (R) 321, stored perhaps in the memory, 120, or some other memory accessible via the network 120.
  • R database record
  • the record includes positional information for the markers in an appropriate coordinate frame e.g. the coordinate frame of the surface of interest to which it is attached.
  • the positional information includes the (x, y, z) position of the markers in the real world as well as their appearance and geometry, e.g., color, shape and size.
  • the positional info ⁇ nation of the record 321 is used to determine 330 a pose
  • an output image (I 0 ) 341 is retrieved 140 from the memory.
  • the output image can be stored in the record 321, or elsewhere. The specific image retrieved can depend on the unique identification, history of interaction, the position and motion of the projector, and the presence of other similar devices.
  • the output image is then warped 345 according to the projector position and orientation.
  • the method 300 then projects the warped image 346 to display the information 220.
  • the warping adjusts the pixels in the output image so that they appear vertically and horizontally aligned with the coordinate frame of the scene, for an arbitrary pose of the projector 100.
  • the result is an undistorted appearance of the displayed info ⁇ nation on surface of the physical object.
  • the means of warping the content are described in detail in U.S. Patent Application WSn., 08/930,426 "Simulating Motion of Static Objects in Scenes, filed August 14, 2001, incorporated herein by reference.
  • the database can record that a particular record was accessed. Then, in step 340, the database can return alternative content based not only on the identity of tlie markers being viewed but also on the number of times the markers were been viewed in the past.
  • step 315 of the basic mode of operation the method detects four markers because the homography used to determine the pose of the camera rely on four known points. This step uses the center of each marker as a known point. Because the record for each marker stores not only the (x, y, z) position of its center, but also its radius, a single marker can provide multiple known points. Using this extra geometric information, the method needs to only correctly detect and identify at least marker in step 315.
  • the visual markers can be made to be invisible to the human eye by negating them in the output image according to the description in the record 321, or their appearance in the input image.
  • the pose of the projector is used to warp the content being projected.
  • step 335 in which the position of the projector is detennined, we can record this position in a database.
  • step 340 when we retrieve content from the database, we can retrieve content based on the motion of the projector by examining the history of previous positions of the projector.
  • One example use of this variant projects low detailed images when the user moves quickly to move the projected image across an object, and high-detailed images when the user is moving slowly, or standing still.
  • step 340 when we retrieve content from the database, the database can record diat the content is being accessed and can record the identification of die device accessing the record. If multiple devices are accessing a record, then the content can be altered based on this knowledge.
  • tiiis variant repositions the content on the surface of an object so that the images projected from multiple devices do not overlap and obscure one another. If multiple people are projecting a list of their queued print jobs on the surface of a printer, die devices are able to reposition these lists so that they do not overlap one another.
  • the content retrieved from the database in step can also be affected by the time of retrieval.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geometry (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Projection Apparatus (AREA)
  • Transforming Electric Information Into Light Information (AREA)
  • Control Of Indicators Other Than Cathode Ray Tubes (AREA)

Abstract

A method displays an output image on an object. A set of unique markers are fixed to the object at predetermined locations. An input image of the object and the markers is acquired with a camera in a fixed physical relationship to a projector. A pose of the projector with respect to the markers is determined from the image. Then, one or more output images related to the object can be projected onto the object, at predetermine locations, according to the pose of the projector and the unique markers.

Description

DESCRIPTION
Method for Displaying an Output Image on an Object
Technical Field
This invention relates generally to digital projectors, and more particularly to a projectors used for augmented reality.
Background Art
The field of augmented reality is concerned with enhancing what we see and hear around us in the physical world. Researchers in this field superimpose computer graphics onto our view of real world objects. This enhances both our perceptions and our interactions with the real world. It enables improved performance of real world tasks.
There are multiple methods for achieving augmented reality. One method of providing an augmented view of the world is through the use of an LCD-screen based digital personal assistant. (PDA). That technique has some drawbacks. The physical size of a PDA limits the information that can be presented to the user, regardless of future improvements in pixel resolution. The information displayed on a PDA is in the coordinate frame of the camera, not the user's own viewpoint. Use of a PDA requires the user to context-switch between the physical environment and the display. A PDA must be kept either on the user's person or nearby but easily visible. A PDA only provides augmentation to an individual, or at best a small group of people clustered around a veiy small device. Another method of providing an augmented view of the world is through the use of 'see-through' video micro-display eyeglasses. A drawback is that the tight coupling between the user's eyes coordinate frame and the displayed image coordinate frame requires very accurate processing of user head motion, e.g., a user head rotation must be accurately matched by a complementary rotation in the displayed image. Also video micro-display eyeglasses are for individual use only, not shared use. Because the display is only a centimeter or so from the eye, working with an augmented object is awkward because other objects, such as the user's hands, can easily occlude the augmentation. Finally, fatigue factors of eye- worn displays are well known.
Projector-based augmented reality projects augmentation directly onto the physical scene. It decouples the user coordinate frame from the relationship between the projector and the physical object. One advantage of this is that there is no need for the difficult task of determining user head motion, when showing projected information.
A rotating movie camera has been used to acquire a film of a living room, replete with furniture, and people. The room and furniture were then painted a neutral white, and the film was projected back onto the walls and furniture using a rotating projector that was precisely registered with the original camera, see Naimark, "Displacements," Exhibit at the San Francisco Museum of Modern Art, San Francisco, CA 1984. This crucial co-location of the acquiring camera and displaying projector is common to most systems that use pre-recorded images, or image sequences to illuminate physical objects. A projector and fiber-optic bundle have been used to animate the head of a fictional fortuneteller inside a real crystal ball, see U.S. Patent No. 4,978,216 "Figure with back projected image using fiber optics" issued to Liljegren, et al, December 18, 1990. Slides of modified photographs augmented with fine details have been used with very bright projectors to render imageiy on a very large architectural scale. A well-known modern realization of this idea is Le Son et Lumiere on Chateau de Blois in the Loire Valley of France. In addition, this medium is now being used elsewhere around the world to illuminate large-scale structures such as bridges.
The "Luminous Room" project treats a co-located camera -projector pair as an I/O bulb to sense and project imagery onto flat surfaces in the real physical surroundings of a room or a designated workspace, see Underkoffler et al. "Emancipated pixels: Real-world graphics in the luminous room," SIGGRAPH '99, pp. 385-392, 1999. Their main focus is interaction with the information via luminous and tangible interfaces. They recognized co-planar 2D physical objects, tracked the 2D positions and orientations in the plane, and projected light from overhead to reproduce the appropriate sunlight shadows.
All those systems render compelling visualizations. However, cumbersome alignment processes can take several hours, even for a single projector. Many similar systems are based on the notion of one or more environmental sensors assisting a processor, which can compute the Euclidean or affϊne relationships between projectors and displays. Without the centralized sensors, the projectors are unable to augment the target objects. Those systems require a fixed special relationship between the projectors and the augmented objects. Any movement of either part requires the system to be recalibrated. Furthermore, the fixed nature of the projectors in those systems dictates that only certain parts of the objects can be augmented.
Therefore, it is desired to provide a context aware projector that does not have the limitations and problems of the prior art systems.
Disclosure of Invention
The invention provides a method for uniquely identifying a target object. Furtheπnore, the invention provides a method for determine the physical relationship between a projector and the object. Knowing the unique identification of the object and the relative position of the projector and the object, enables the projection of calibrated images that are related to the object.
The invention can be applied to a variety of systems where it is desirable to augment an object with projected images that are specific to that object. Examples include information displays on objects, e.g., infoπnation displays for a printer include the print queue or the expected time for jobs to complete. Infoπnation displays for books could include comments that appear in the margin. Training applications in which instructions are displayed on a known object. Physical indexing in which a user is guided towards a requested object, e.g., by projected arrows on the ground and by highlighting of specific areas in a storage area or workspace. Recovery and projection of electronic data items which have been attached to the environment. A 'Magic Lens' to simulate transparency of an opaque surface. Projection of language specific information about an object of interest to a tourist. The invention further provides means for adapting the projected content according to several factors, including the history of interaction, the position and motion of the projector, and the presence of other similar devices.
Brief Description of the Drawings
Figure 1 is a block diagram of a projector according to the invention;
Figure 2 is a block diagram of an object augmented by an image displayed by the projector of Figure 1; and
Figure 3 is a block diagram of a method for displaying images on an object according to the invention.
Best Mode for Carrying Out the Invention
Geometrically Aware Projector
Figure 1 shows a context aware projector 100 according to our invention. It should be noted that the projector 100 can be used in alone or in a cooperative combination with other similar projectors.
Our projector 100 includes a microprocessor 110, a memory 120, and an I/O interface 130 comiected by buses 140, generally a processing unit 103. The processing unit is conventional in its electronic stmcture, but unconventional in its operation when performing the method steps described herein. The projector also includes a projector sub-system 150 for displaying output images 102, and a camera sub-system 160 for acquiring input images 101 of a scene including uniquely identified objects.
A network sub-system 170 allows the projector to communicate with other similar devices, or other computing devices, local or remote. Therefore, the network system can be connected to an antenna or infrared transceiver 180, depending on the communications medium.
A user interface 190 can provide input and output data to facilitate operation of the projector. For instance, the interface can include left, center, and right buttons as are normally found on an input mouse. The interface can also include a "scroll" wheel, as are known in the art.
Internal sensors 195, e.g., tilt sensors or accelerometers can determine an orientation of the projector. It should be noted that the sensors are internal only, and do not sense the environment external to the projector. It should be noted that the sensors are optimal.
Projector Operation
The projector 100 can perfoπn a number of functions, including smart keystone coiτection, orientation compensated image intensities, auto brightness, zoom and focus, 3D scanning for geometry and texture capture, and smart flash for the camera sub-system 160. For this last function, the projector function is secondaiy to the camera function. Here, the projector merely illuminates specific areas in a scene observed by the camera. A number of additional adaptive functions are described in greater detail below.
System Structure and Method
As shown in Figure 2, we use our calibrated projector-camera system 100 for uniquely identifying an object 201 , and projecting object specific information 220. A relative position of the object with respect to the projector 100 is determined from unique visual markers 211-214 attached to the object 201. By pose, we mean the (x,y,z) position of the projector in the same coordinate frame of the object, and the orientation of the projector. By orientation we mean the amount of rotation about any of the three major axes.
In the preferred embodiment, the markers are circular, see enlargement, with multiple colored segments, e.g. four to six. The pattern of the colors define different unique identifications, see U.S. Patent Application Sn. 09/628,461,
"Badge Identification System," filed July 31, 2000, incorporated herein by reference.
The color-coding encodes a unique identity. It should be noted that other similar markers can be used. One or more markers can be used as described below in detail.
Four or more points on the object 201 are used to obtain a unique solution of camera pose with the calibrated camera sub-system 160. The points are obtained from images acquired at a long range of tlie multiple markers, or from the shape of a single marker at closer-range. After the camera pose is determined, the projector pose is known via the system calibration. After the position and orientation of the projector are know, augmentation information, retrieved from the memory 120 or downloaded via the network 210, can then be displayed. The shape, size, and location where tlie image is displayed can be predetennined.
Important applications where context-based augmentation can be used include: information displays on objects, either passive display, or training applications in which instructions 220 are displayed as part of a sequence of steps; physical indexing where a user is guided towards a requested object; identifying the presence of, and projecting, electronic data items which have been attached to the environment.
Context-based augmentation does not have rigorous requirements for accuracy because small offsets in positioning of augmented infoπnation are usually not of consequence. It does however put more emphasis on modes of use. For example, distance from an object can be used to adjust level-of-detail in tl e projected augmentation 220.
The system is calibrated so that the following information is known - the intrinsic parameters of tlie projector, the intrinsic parameters of the camera, the relative position and orientation between tlie projector and camera. This calibration can be done as a pre-calibration step before the system is in operation, or can be done as start-up processing when the system begins operation. Any of tlie well known camera calibration techniques can be used.
The purpose of tlie camera is to acquire the pose of the system relative to a surface of interest in the scene. Here 'pose' is used to mean the (x, y, z) location, and orientation. This is achieved by processing an input image of the object to determine the pose of the camera, then using the calibration infonnation to infer die pose of the projector. After the pose of the projector is known, the augmentation information 220 can be projected to a desired place on the surface of interest.
As shown in Figure 3, our method 300 first acquires an input image (Ij) 301 of a scene. The scene includes at least four visual markers. These visual markers are color-coded, and the color-coding specifies identity. The markers can be thought of as analogous to barcodes, but based on color rather than striping.
Next, the markers are detected 315 in the input image 301. The markers are used to index 320 a database record (R) 321, stored perhaps in the memory, 120, or some other memory accessible via the network 120.
The record includes positional information for the markers in an appropriate coordinate frame e.g. the coordinate frame of the surface of interest to which it is attached. The positional information includes the (x, y, z) position of the markers in the real world as well as their appearance and geometry, e.g., color, shape and size.
The positional infoπnation of the record 321 is used to determine 330 a pose
(CP) 331 of the camera with respect to tlie markers, and hence with respect to the surface of interest. The camera pose is used to infer 335 a pose (PP) 336 of die projector. This calculation is perforated using known homographies, a well known image processing technique. Next, an output image (I0) 341 is retrieved 140 from the memory. The output image can be stored in the record 321, or elsewhere. The specific image retrieved can depend on the unique identification, history of interaction, the position and motion of the projector, and the presence of other similar devices.
The output image is then warped 345 according to the projector position and orientation. The method 300 then projects the warped image 346 to display the information 220. The warping adjusts the pixels in the output image so that they appear vertically and horizontally aligned with the coordinate frame of the scene, for an arbitrary pose of the projector 100. The result is an undistorted appearance of the displayed infoπnation on surface of the physical object. The means of warping the content are described in detail in U.S. Patent Application WSn., 08/930,426 "Simulating Motion of Static Objects in Scenes, filed August 14, 2001, incorporated herein by reference.
History of Augmentation
We provide a variant of the basic mode that takes advantage of the history of augmentation. In step 320, the database can record that a particular record was accessed. Then, in step 340, the database can return alternative content based not only on the identity of tlie markers being viewed but also on the number of times the markers were been viewed in the past.
Alternative to Using Four Markers
We provide an alternative to the basic mode of operation tiiat relies on fewer visual markers being visible on the target. In step 315 of the basic mode of operation, the method detects four markers because the homography used to determine the pose of the camera rely on four known points. This step uses the center of each marker as a known point. Because the record for each marker stores not only the (x, y, z) position of its center, but also its radius, a single marker can provide multiple known points. Using this extra geometric information, the method needs to only correctly detect and identify at least marker in step 315.
Alternative to Visible Markers
We provide an alternative to the basic mode of operation that removes the appearance of the visual markers. The visual markers can be made to be invisible to the human eye by negating them in the output image according to the description in the record 321, or their appearance in the input image.
Alternative Content Based on Projector Position
We provide a variant of the basic mode of operation that takes further advantage of the position of the projector relative to tlie object being augmented. In the basic mode of operation, in step 345, the pose of the projector is used to warp the content being projected. We can also use the pose to retrieve pose specific content from the database.
One example use of tiiis variant projects high-level semantic information when die user is far away from an augmented object, and then to project low-level detailed infoπnation when die user is close to the same object. Alternative Content Based on Projector Motion
We provide a variant of the basic mode of operation that takes advantage of the motion of the projector. During step 335, in which the position of the projector is detennined, we can record this position in a database. Then, in step 340, when we retrieve content from the database, we can retrieve content based on the motion of the projector by examining the history of previous positions of the projector.
One example use of this variant projects low detailed images when the user moves quickly to move the projected image across an object, and high-detailed images when the user is moving slowly, or standing still.
Alternative Content Based on the Presence of Similar Devices
We provide a variant of the basic mode of operation that takes advantage of the presence of similar devices. In step 340, when we retrieve content from the database, the database can record diat the content is being accessed and can record the identification of die device accessing the record. If multiple devices are accessing a record, then the content can be altered based on this knowledge.
One example use of tiiis variant repositions the content on the surface of an object so that the images projected from multiple devices do not overlap and obscure one another. If multiple people are projecting a list of their queued print jobs on the surface of a printer, die devices are able to reposition these lists so that they do not overlap one another.
Alternative Content Based on the Time of Projection The content retrieved from the database in step can also be affected by the time of retrieval.
Alternative to Image Projection
We also provide a variant of the basic mode of operation that warps and projects video content instead of still images.
Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. A method for displaying an output image on an object, comprising: fixing a set of unique markers to the object at predetermined locations; acquiring an input image of the object and the plurality of markers with a camera in a fixed physical relationship to a projector; determining a pose of the projector with respect to the set of markers; projecting an output image onto the object according to the pose of the projector and the set of unique markers.
2. The method of claim 1 wherein each marker in the set is circular with a plurality of different colored segments.
3. The method of claim 1 wherein there are four markers in the set.
4. The method of claim 1 wherein a geometry of the markers is predetermined, and the set includes one marker.
5. The method of claim 1 wherein a geometry of each markers is predetermined, and a location of the marker on the object is predetermined.
6. The method of claim 5 further comprising: storing the geometry and the location in a memory accessible by the projector.
7. The method of claim 1 wherein the pose includes a location and orientation of dien projector.
8. The method of claim of claim 1 further comprising: warping the output image according to the pose of the projector prior to projecting the output image.
9. The method of claim 1 wherein the output image is dependent history of interaction between the projector and the object.
10. The method of claim 1 wherein the output image is dependent on the pose of the projector.
11. The method of claim 8 wherem the warping horizontally and vertically aligns the output image to a coordinate frame of the object.
12. The method of claim 1 further comprising: negating an appearance of the set of marker in the output image to make the set of markers invisible.
13. The method of claim 1 further comprising: detennining a motion of the projector from the pose; and selecting the output image dependent on the motion.
14. The method of claim 1 further comprising: selecting the output image based on a time of projection.
15. The mediod of claim 1 further comprising: projecting a sequence of images.
16. The method of claim 1 wherein the output image is dependent on the co- presence of similar devices.
17. The method of claim 1 wherein the output image has a predetermined shape and is projected at a predetermined location on the object.
EP04726647.3A 2003-04-11 2004-04-08 Method for displaying an output image on an object Expired - Lifetime EP1504597B1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US412178 2003-04-11
US10/412,178 US7292269B2 (en) 2003-04-11 2003-04-11 Context aware projector
PCT/JP2004/005069 WO2004093444A1 (en) 2003-04-11 2004-04-08 Method for displaying an output image on an object

Publications (2)

Publication Number Publication Date
EP1504597A1 true EP1504597A1 (en) 2005-02-09
EP1504597B1 EP1504597B1 (en) 2015-02-11

Family

ID=33131159

Family Applications (1)

Application Number Title Priority Date Filing Date
EP04726647.3A Expired - Lifetime EP1504597B1 (en) 2003-04-11 2004-04-08 Method for displaying an output image on an object

Country Status (5)

Country Link
US (1) US7292269B2 (en)
EP (1) EP1504597B1 (en)
JP (1) JP4584246B2 (en)
CN (1) CN1698357B (en)
WO (1) WO2004093444A1 (en)

Families Citing this family (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100449440C (en) * 2003-02-03 2009-01-07 西门子公司 Projection of Synthetic Information
JP2007206618A (en) * 2006-02-06 2007-08-16 Olympus Corp Information presentation apparatus
JP2007206619A (en) * 2006-02-06 2007-08-16 Olympus Corp Information presentation apparatus and method
US7717569B2 (en) * 2006-04-13 2010-05-18 Nokia Corporation Projector screen with one or more markers
EP2108135A4 (en) * 2007-01-31 2013-02-20 Micromentis Inc Head-mounted display apparatus for profiling system
US8994757B2 (en) 2007-03-15 2015-03-31 Scalable Display Technologies, Inc. System and method for providing improved display quality by display adjustment and image processing using optical feedback
JP2009008905A (en) * 2007-06-28 2009-01-15 Ricoh Co Ltd Information display device and information display system
US7874681B2 (en) 2007-10-05 2011-01-25 Huebner Kenneth J Interactive projector system and method
JP5286772B2 (en) * 2007-12-18 2013-09-11 セイコーエプソン株式会社 Image display device
DE102008001629A1 (en) * 2008-05-07 2009-11-12 Robert Bosch Gmbh Projection means comprehensive device
US9323422B2 (en) * 2008-10-27 2016-04-26 Autodesk, Inc. Spatially-aware projection pen display
JP2010166360A (en) * 2009-01-16 2010-07-29 Seiko Epson Corp Projector, control method thereof, and control program thereof
US9538167B2 (en) * 2009-03-06 2017-01-03 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for shader-lamps based physical avatars of real and virtual people
US8393740B2 (en) * 2009-03-12 2013-03-12 Canon Kabushiki Kaisha Image projection system with keystone correction
US8231233B2 (en) * 2009-06-17 2012-07-31 Motorola Mobility, Inc. Portable electronic device and method of power management for same to accommodate projector operation
US20100321382A1 (en) 2009-06-18 2010-12-23 Scalable Display Technologies, Inc. System and method for injection of mapping functions
JP5573238B2 (en) * 2010-03-04 2014-08-20 ソニー株式会社 Information processing apparatus, information processing method and program
JP5740822B2 (en) * 2010-03-04 2015-07-01 ソニー株式会社 Information processing apparatus, information processing method, and program
JP5598033B2 (en) * 2010-03-15 2014-10-01 ソニー株式会社 Information processing apparatus, information processing method, and program
WO2012047340A1 (en) 2010-10-07 2012-04-12 Sony Computer Entertainment Inc. Tracking head position and orientation
WO2012047222A1 (en) 2010-10-07 2012-04-12 Sony Computer Entertainment Inc. 3-d glasses with illuminated light guide
CN102652322A (en) * 2010-10-12 2012-08-29 松下电器产业株式会社 Display processing device, display method and program
CN103329540B (en) 2010-11-15 2016-08-24 斯加勒宝展示技术有限公司 Systems and methods for calibrating display systems using manual and semi-automatic techniques
US9111326B1 (en) 2010-12-21 2015-08-18 Rawles Llc Designation of zones of interest within an augmented reality environment
US8845110B1 (en) 2010-12-23 2014-09-30 Rawles Llc Powered augmented reality projection accessory display device
US8905551B1 (en) 2010-12-23 2014-12-09 Rawles Llc Unpowered augmented reality projection accessory display device
US8845107B1 (en) 2010-12-23 2014-09-30 Rawles Llc Characterization of a scene with structured light
US9134593B1 (en) 2010-12-23 2015-09-15 Amazon Technologies, Inc. Generation and modulation of non-visible structured light for augmented reality projection system
US9721386B1 (en) 2010-12-27 2017-08-01 Amazon Technologies, Inc. Integrated augmented reality environment
US9508194B1 (en) 2010-12-30 2016-11-29 Amazon Technologies, Inc. Utilizing content output devices in an augmented reality environment
US9607315B1 (en) 2010-12-30 2017-03-28 Amazon Technologies, Inc. Complementing operation of display devices in an augmented reality environment
MX346859B (en) 2011-02-11 2017-04-04 Ops Solutions Llc Light guided assembly system and method.
DE102011015987A1 (en) * 2011-04-04 2012-10-04 EXTEND3D GmbH System and method for visual presentation of information on real objects
US9497447B2 (en) 2011-06-15 2016-11-15 Scalable Display Technologies, Inc. System and method for color and intensity calibrating of a display system for practical usage
US9118782B1 (en) 2011-09-19 2015-08-25 Amazon Technologies, Inc. Optical interference mitigation
JP2014532225A (en) * 2011-09-30 2014-12-04 インテル コーポレイション A mechanism for facilitating context-aware model-based image composition and rendering in computing devices
CN103106665A (en) * 2011-11-11 2013-05-15 周建龙 Method capable of automatically tracking moving object in space-augmented reality system
US9132346B2 (en) 2012-04-04 2015-09-15 Kenneth J. Huebner Connecting video objects and physical objects for handheld projectors
KR20200105547A (en) 2012-04-19 2020-09-07 스케일러블 디스플레이 테크놀로지스, 인크. System and method of calibrating a display system free of variation in system input resolution
WO2013173724A1 (en) 2012-05-17 2013-11-21 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for utilizing synthetic animatronics
CN103869961A (en) * 2012-12-18 2014-06-18 联想(北京)有限公司 Method and system for interacting projector and video camera
US9881383B2 (en) 2013-01-28 2018-01-30 Virtek Vision International Ulc Laser projection system with motion compensation and method
CN104995573B (en) * 2013-02-25 2018-04-13 Abb 技术有限公司 Method and device for monitoring and controlling industrial processes
EP2972585A4 (en) 2013-03-15 2016-10-19 Scalable Display Technologies Inc System and method for calibrating a display system using a short throw camera
KR101465497B1 (en) 2013-07-09 2014-11-26 성균관대학교산학협력단 Apparatus and system for dynamic projection mapping and method thereof
WO2015070258A1 (en) 2013-11-11 2015-05-14 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for improved illumination of spatial augmented reality objects
US9473913B2 (en) 2013-11-12 2016-10-18 At&T Intellectual Property I, L.P. System and method for small cell based augmented reality
WO2015086036A1 (en) * 2013-12-12 2015-06-18 Testo Ag Method for the positionally accurate projection of a mark onto an object, and projection apparatus
WO2016103204A1 (en) 2014-12-24 2016-06-30 Verity Studios Ag Flexibly supported movable platform
US11062383B2 (en) * 2016-05-10 2021-07-13 Lowe's Companies, Inc. Systems and methods for displaying a simulated room and portions thereof
CN110023832A (en) * 2016-06-23 2019-07-16 奥特涅茨公司 Interactive content management
US10009586B2 (en) * 2016-11-11 2018-06-26 Christie Digital Systems Usa, Inc. System and method for projecting images on a marked surface
JP6773609B2 (en) * 2017-06-23 2020-10-21 ウエストユニティス株式会社 Remote support system, information presentation system, display system and surgery support system
US20190052851A1 (en) 2017-08-11 2019-02-14 Hilti Aktiengesellschaft System and Method for Recalibrating a Projector System
KR102014095B1 (en) * 2017-09-29 2019-08-27 대한민국 Recognition Support System and Method
CN109788262A (en) * 2017-11-15 2019-05-21 中强光电股份有限公司 Optical projection system and its image projecting method
DE102017010683B4 (en) * 2017-11-17 2019-08-14 domeprojection.com GmbH Method for automatic restoration of a measured state of a projection system
US10192115B1 (en) 2017-12-13 2019-01-29 Lowe's Companies, Inc. Virtualizing objects using object models and object position data
US11188154B2 (en) * 2018-05-30 2021-11-30 International Business Machines Corporation Context dependent projection of holographic objects
FR3086383B1 (en) 2018-09-21 2020-08-28 Diotasoft PROCESS, MODULE AND SYSTEM FOR PROJECTING ON A PART OF AN IMAGE CALCULATED FROM A DIGITAL MODEL
WO2020210841A1 (en) * 2019-04-12 2020-10-15 Daniel Seidel Projection system with interactive exclusion zones and topological adjustment
JP7099497B2 (en) 2020-07-28 2022-07-12 セイコーエプソン株式会社 Image generation method, image generation system, and program
JP2022147145A (en) * 2021-03-23 2022-10-06 カシオ計算機株式会社 Projector, projection system, method for correcting correction value, and program
CN112991467B (en) * 2021-05-08 2021-07-20 光量信息科技(宁波)有限公司 Camera-based laser projection identification automatic guiding positioning and real-time correction method
JP7703907B2 (en) * 2021-06-02 2025-07-08 セイコーエプソン株式会社 Projector and method for controlling the projector

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4978216A (en) * 1989-10-30 1990-12-18 Walt Disney Company Figure with back projected image using fiber optics
JP3382045B2 (en) * 1995-01-24 2003-03-04 富士通株式会社 Image projection system
DE19502459A1 (en) * 1995-01-28 1996-08-01 Wolf Henning Three dimensional optical measurement of surface of objects
SE510810C2 (en) * 1997-01-13 1999-06-28 Qualisys Ab Motion Analysis System
US6191812B1 (en) * 1997-04-01 2001-02-20 Rt-Set Ltd. Method of providing background patterns for camera tracking
JP2000241874A (en) * 1999-02-19 2000-09-08 Nec Corp Method and device for automatically adjusting screen position for projector
US6554431B1 (en) * 1999-06-10 2003-04-29 Sony Corporation Method and apparatus for image projection, and apparatus controlling image projection
JP3509652B2 (en) * 1999-08-23 2004-03-22 日本電気株式会社 Projector device
US6520647B2 (en) * 2000-08-17 2003-02-18 Mitsubishi Electric Research Laboratories Inc. Automatic keystone correction for projectors with arbitrary orientation
JP3497805B2 (en) * 2000-08-29 2004-02-16 オリンパス株式会社 Image projection display device
US6697761B2 (en) * 2000-09-19 2004-02-24 Olympus Optical Co., Ltd. Three-dimensional position/orientation sensing apparatus, information presenting system, and model error detecting system
JP2002247614A (en) * 2001-02-15 2002-08-30 Ricoh Co Ltd Projector
JP2003015218A (en) * 2001-07-03 2003-01-15 Ricoh Co Ltd Projection display device
US7075587B2 (en) * 2002-01-04 2006-07-11 Industry-Academic Cooperation Foundation Yonsei University Video display apparatus with separate display means for textual information
JP2004274283A (en) * 2003-03-07 2004-09-30 Casio Comput Co Ltd Projection system, projection device, image supply device, and program

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2004093444A1 *

Also Published As

Publication number Publication date
JP2006523067A (en) 2006-10-05
US7292269B2 (en) 2007-11-06
JP4584246B2 (en) 2010-11-17
US20040201823A1 (en) 2004-10-14
CN1698357B (en) 2010-04-28
WO2004093444A1 (en) 2004-10-28
EP1504597B1 (en) 2015-02-11
CN1698357A (en) 2005-11-16

Similar Documents

Publication Publication Date Title
EP1504597B1 (en) Method for displaying an output image on an object
Azuma A survey of augmented reality
Klein Visual tracking for augmented reality
CN108062776B (en) Camera Attitude Tracking method and apparatus
Pinhanez The everywhere displays projector: A device to create ubiquitous graphical interfaces
KR20230025909A (en) Augmented Reality Eyewear 3D Painting
JP7548835B2 (en) Augmented reality system using visual object recognition and stored geometry to create and render virtual objects - Patents.com
KR20230029885A (en) Augmented reality eyewear with speech bubbles and translation
KR20180101496A (en) Head-mounted display for virtual and mixed reality with inside-out location, user body and environment tracking
US12216387B2 (en) External recording indicators
CN110377148B (en) Computer readable medium, method of training object detection algorithm and training device
KR20150093831A (en) Direct interaction system for mixed reality environments
US20220172319A1 (en) Camera-based Transparent Display
Kijima et al. Distributed display approach using PHMD with infrared camera
JP2013218535A (en) Method and device for displaying finger integrated into cg image in three-dimensionally modeled cg image and wide viewing angle head mount display device for displaying three-dimensionally modeled cg image
US10992928B1 (en) Calibration system for concurrent calibration of device sensors
KR20230025913A (en) Augmented Reality Eyewear with Mood Sharing
KR102183692B1 (en) An augmented reality service apparatus for a mirror display by recognizing the reflected images on the mirror and method thereof
Chan et al. Enabling beyond-surface interactions for interactive surface with an invisible projection
Zheng Spatio-temporal registration in augmented reality
Livingston Vision-based tracking with dynamic structured light for video see-through augmented reality
McIlroy et al. Kinectrack: 3d pose estimation using a projected dense dot pattern
CN109688400A (en) Electronic equipment and mobile platform
CN107403406B (en) Method and system for converting between solid image and virtual image
US11982814B2 (en) Segmented illumination display

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20041210

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL HR LT LV MK

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: MITSUBISHI DENKI KABUSHIKI KAISHA

DAX Request for extension of the european patent (deleted)
RBV Designated contracting states (corrected)

Designated state(s): DE FR GB

RIN1 Information on inventor provided before grant (corrected)

Inventor name: BEARDSLEY, PAUL, A.

Inventor name: FORLINES, CLIFTON, L.

Inventor name: RASKAR, RAMESH

17Q First examination report despatched

Effective date: 20090428

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20140905

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602004046605

Country of ref document: DE

Effective date: 20150326

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602004046605

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20151112

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 13

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

REG Reference to a national code

Ref country code: DE

Ref legal event code: R084

Ref document number: 602004046605

Country of ref document: DE

REG Reference to a national code

Ref country code: GB

Ref legal event code: 746

Effective date: 20170424

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230309

Year of fee payment: 20

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230302

Year of fee payment: 20

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230512

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230228

Year of fee payment: 20

REG Reference to a national code

Ref country code: DE

Ref legal event code: R071

Ref document number: 602004046605

Country of ref document: DE

REG Reference to a national code

Ref country code: GB

Ref legal event code: PE20

Expiry date: 20240407

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF EXPIRATION OF PROTECTION

Effective date: 20240407

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF EXPIRATION OF PROTECTION

Effective date: 20240407