GB2602356A - Optical markers - Google Patents

Abstract

An optical marker for use marking a location in an optical tracking system comprises a substrate 16 having a reflective surface and a series of regularly positioned through-holes 17 extending through the reflective surface. These apertures 17 are positioned to be located around display elements of a display panel. Engagement structures may be provided to fix the marker to the display panel. The display panel may be a ceiling panel arranged to illuminate a subject being videoed by a camera. An applicator (Fig. 8) for applying optical markers is also disclosed.

Description

OPTICAL MARKERS

This invention relates to optical markers.

One way to make a video is to use a camera to image a subject against a background that can be manipulated to show an arbitrary scene. To do this, the subject may be imaged against a screen of a known colour, typically green. Then in post-processing the green background in the captured video is replaced by the desired arbitrary background scene. Another approach is to image the subject against a display wall, with the display wall showing the arbitrary background scene. This latter approach requires less post-processing.

The camera can be moved during the shooting of the scene, for cinematographic reasons. The background scene will typically contain objects at a range of depths. As the camera moves, it is desirable to manipulate the background to avoid parallax effects that can otherwise manifest as an unnatural motion of the background objects. To achieve this, a computer can be provided with data defining a three-dimensional map of the objects in the background. Then the computer can generate the background scene in dependence on the relative position of the camera and the background. To do this, that relative position must be known.

One way of detecting the relative position is to mount the camera on wheels which are provided with distance and direction sensors The sensors can sense the motion of the camera as it translates across the floor.

Another way of detecting the relative position is to use an optical tracking system. Figure 1 shows an example of such a system. Figure 1 shows a camera 1 in a studio 2. The camera is movable on wheels 3. The studio contains walls 4. These could be green (or other known colour) screen walls or display walls such as LED or back-projection walls. Reflective markers 5 are positioned in the studio. A positioning assembly comprising an illuminating light 6 and a camera 7 are attached to the camera. The reflective markers are positioned in an irregular pattern so that the constellation of the markers appears differently from different locations in the studio. In operation, the light 6 illuminates the markers and the positioning camera 7 captures the light reflected from the markers. A processor, which could be in the positioning assembly or elsewhere, processes the images captured by the camera to estimate the camera's location. This can be done with reference to learned locations of the markers and by using a suitable positioning algorithm such as a SLAM (simultaneous localisation and mapping) algorithm. It is convenient for the markers 5 to be positioned on the ceiling since this typically allows an unimpeded optical path to and from the positioning assembly. Markers could be placed on the wall, but this would normally interfere with the formation of the artificial background scene.

To achieve certain lighting effects in the video, such as highlights or reflections or to avoid certain shadows, it is sometimes desirable to illuminate it the subject in ways that can interfere with the operation of the system shown in figure 1. For example, it may be desired to provide an illuminated ceiling formed of LED panels or to use individual high-level LED panels. The illumination from these may be controlled during the videoing of a scene to generate the desired effects. Such panels may block the optical path to the markers 5, making it difficult to use the system of figure 1.

There is a need for an improved way of locating an object, e.g. a camera in a studio.

According to the present invention there is provided apparatus as set out in the accompanying claims.

The present invention will be described by way of example with reference to the drawings.

In the drawings: Figure 1 shows a camera in a studio.

Figure 2 shows a studio with display panels.

Figure 3 shows part of a display panel.

Figure 4 shows a first form of optical marker.

Figure 5 shows a side view or cross-section of the marker of figure 4.

Figure 6 shows a second form of optical marker.

Figure 7 shows optical markers on a movable display panel.

Figure 8 shows an applicator for optical markers.

Figure 2 shows a studio configured with display walls 10 and a display ceiling 11. Each wall and the ceiling comprise a multitude of individual display elements. The display elements are shown schematically in figure 2. In practice the display elements could be multicolour LEDs. The LEDs are controlled by a display controller 12 to display a desired scene, including on the ceiling. The ceiling could, for example, dynamically become brighter or darker in selected locations to simulate the lighting effects of moving through a forest or entering a tunnel.

The discussion below will refer to mechanisms for estimating the position and orientation of objects. For conciseness only position will be referred to below, but it should be understood that orientation can be obtained at the same time.

A camera equipped with a positioning apparatus as described with reference to figure 1 may be used in the studio shown in figure 2. The positioning apparatus may, for example, be a StarTracker apparatus as is commercially available from Mo-Sys Engineering Ltd. Figure 3 is a closer view of a part of the display ceiling 11. The ceiling is oriented to face upwards as depicted in figure 3. The ceiling is a generally planar structure. In the plane of the ceiling there are many display elements. Each display element has a housing 13 which is mounted to a support sheet 15. The housing extends out of the support sheet. Typically, the support sheet is perforated with holes at a suitable pitch, and each display element is clipped or otherwise attached through a respective hole. This allows the display elements to be readily disposed at a desired pitch. Inside each housing 13 the display element has a lens 14. Typically the lens has the form of a convex dome. Behind the lens each display element has one or more light-emitting devices, typically LEDs. Each display element may have a single light-emitting device, which might emit red, blue, green or white light under the control of the controller 12. Alternatively, each display element may have multiple independently controllable light-emitting devices, for example red, green and blue emitters which can be controlled so the device as a whole emits light appearing to be of a desired colour. Other display panels such as display walls and mobile display panels can have the same structure as the panel shown in figure 3.

Figure 4 shows a first optical marker suitable for use with a display panel of the type shown in figure 3. The marker 16 is in the form of a plate having holes 17 through it. The holes are sized and positioned so as to match the size and position of the housings 13 of the display elements of the panel of figure 3. This means that the marker can be attached to the panel of figure 3 with the through-holes 17 registered with that set of the housings 13 that is within the perimeter of the marker 16. This allows light from the display elements within the perimeter of the marker to shine unimpeded through the holes 17.

Figure 5 shows a side view of the marker of figure 4. The marker comprises a rigid or pliable substrate 19 and a retroreflective layer 18 which is applied over the substrate. The retroreflective layer may contain dispersed reflective particles or other reflectors. The effect of the retroreflective layer is to reflect incident light over a wide arc, allowing the marker to return light to and from a range of incident angles with respect to the positioning apparatus. When the marker is attached to the display panel it is attached with the retroreflective layer facing away from the panel.

The marker may be provided with a mechanism to adhere it to the display panel. For example, the sides of the holes17 may be sized so as to form a tight friction fit around the housings 13, or the sides may be profiled so as to snap fit to the housings 13 or recesses or projections in the structure 15 carrying them, or the rear of the substrate 19 may be provided with an adhesive.

Figure 6 shows an alternative design of marker. In the marker of figure 6, each hole accommodates multiple display elements.

Typically the display elements are provided on a regular grid. The grid may comprise linear rows of display elements. Display elements of successive rows of the grid may be offset along the rows. The through-holes in the markers may be regularly spaced to match the spacing of the display elements.

Other shapes of marker are possible. For example, the markers could be linear and could fit snugly in the gaps between adjacent rows of display elements. Or the markers could be cruciform and fit snugly in the gaps between orthogonal pairs of rows of display elements. In each case, it is preferred that the outer surface of the marker is retroreflective.

In use, markers of the types described above are applied to a display panel, for example to the ceiling 11 of a studio as shown in figure 2. The markers can be applied at irregular locations. Then a positioning apparatus of the type shown in figure 1 can be used in the studio to locate a camera or other apparatus to which it is attached. To avoid interference between the illuminator 6 of the positioning apparatus and the lighting provided by the display panel, the illuminator 6 may operate outside the visible spectrum, for example in the infra-red. The camera 7 of the positioning apparatus may have a spectral filter or otherwise be configured to selectively capture light of the frequency emitted by the illuminator 6.

Figure 7 shows a further arrangement. In the arrangement of figure 7, camera 1 is again located in a studio. The studio comprises a movable display panel 30. This can be positioned and controlled to provide desired lighting effects, for example highlights or glows from a desired direction on to a subject. The display panel can be moved to simulate motion of an illuminating object. Markers 16 of the type described above are attached to the display side of the display panel 30. The display side faces the studio space and camera 1. On the rear of the display panel 30 is mounted a second positioning apparatus 31 of the type mounted to camera 1. The second positioning apparatus can image markers 32 on the ceiling, walls or other structure of the studio. This allows the position of the movable panel 30 in the studio to be estimated. The positioning apparatus of the camera 1 can image markers 16 and optionally other markers, and thereby estimate the position of the camera 1 relative to the movable panel. The positioning apparatus 31 of the panel can communicate directly with the positioning apparatus 6, 7 of the camera, or both can communicate with a third computing device. The position of the panel relative to the studio and the position of the camera relative to the panel are then both available so that the position of the camera relative to the studio can be estimated. A panel of this type may be hung at any suitable height. It could be a non-illuminating panel. It could be used in settings other than studios or other video capture settings. For example, in an industrial setting where a device is to determine its position using a positioning apparatus such as 6, 7 an illuminated or non-illuminated panel could be hung between the device and the ceiling of the space where the device is operating, or above the device if the device is outdoors. This can assist the device to establish its position if its view of markers on the ceiling is obscured or if there is no ceiling above the device on which to place markers.

Figure 8 shows an applicator for markers of the type described above. The applicator comprises a head 40 and a handle 41. The head comprises a perimeter wall 42. Inboard of the perimeter wall are multiple locating structures 43. The locating structures are configured to mate with respective ones of the display elements of a display panel. To that end, the locating structures are spaced at the same spacing as the display elements. Also, the distal face of each locating structure is shaped so as to mate with the shape of a respective display element. For example, if each display element has a convex lens, each locating structure can have a matching concave recess 44 which fits to the lens. The locating structures are in the form of pillars. This allows a marker that is intended to be fitted to a display panel to be loaded into the head, with the pillars passing through the through-holes 17,20 of the marker. The side of the marker that is to face the display panel is positioned outwards. In the base of the head is a movable platen 45. The platen is attached to a plunger 46 which passes through then handle 41 and terminates in a push-button 47. The platen is biased to be recessed in the head by a spring (not shown). When the button is pressed, the platen moves distally for pressing a marker loaded in the head into engagement with a display panel. In operation, a marker is loaded into the applicator. The applicator is offered to the display panel. Because of the shaping and positioning of the locating structures the applicator will naturally guide or find engagement with the panel in the intended location: that is with each locating structure mated to a respective display element. This engagement can be felt by the user. Then the user can press pushbutton 47 to move the platen and push the pre-loaded marker into engagement with the display panel. The marker will friction fit, snap fit or adhesive fit to the panel under the action of the platen. Then the applicator can be removed. This can simplify the attachment of the markers to the panel.

The applicator of figure 8 employs the shape of the LED panel to guide the applicator into the correct position. Other approaches could be used. For example, the LED panel could be caused to display a predetermined pattern. All the display elements could be turned on, or LED elements in groups could be turned on to indicate suitable irregularly spaced locations for markers. Those locations could be determined automatically by a suitably programmed computer. They could be determined so as to be spaced apart by a predetermined minimum spacing. They could be determined so as to have a predetermined density across the area of the panel. With LED elements turned on it may be easier to position the markers so that they do not obscure the LED elements. A user may see the light from the LED elements passing through holes in the marker, and this may guide the user to position the marker correctly. An automated applicator may detect light from the LED elements using an image capturing device incorporated into the applicator. The applicator may then inform a user when the applicator is in the correct position, or automatically apply the marker when it is in the correct position.

Thus, the applicator comprises a structure or device for enabling correct registration of the applicator with the LED elements of the panel to be assessed, and a mechanism for holding a marker in a position and/or orientation whereby its registration with the LED elements can be assessed by the applicator. The device for assessing registration may have a surface configuration that mates with a surface configuration of the LED elements to allow registration to be assessed and/or established. Or it may have a light detecting device. That light detecting device could detect light passing through holes in a marker when the marker is being held by the holding mechanism.

Figure 9 shows a further example of a marker. The marker of figure 9 has holes 60 for fitting around display elements, a reflective surface 61 and tangs 62 for snap fitting to the display elements or to a structure carrying them. Each tang is a projection carried by a resilient leg which extends from the body comprising the reflective surface 61.

In the examples above, the markers are reflective, preferably retroreflective. The markers could emit light of a suitable frequency, in which case the positioning apparatus need not be provided with an illuminator.

The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims (15)

1. CLAIMS1. An optical marker comprising a substrate having a reflective surface and a series of regularly positioned through-holes extending through the reflective surface for location around display elements of a display panel.
2. 2. An optical marker as claimed in claim 1, wherein the reflective surface is planar.
3. 3. An optical marker as claimed in any preceding claim, wherein the reflective surface is retroreflective.
4. 4. An optical marker as claimed in any preceding claim, wherein the marker is provided with an engagement structure extending from a face of the substrate opposite to the reflective surface, the engagement structure comprising a resiliently movable projection for snap-fit engagement with a display panel.
5. 5. An optical marker as claimed in any preceding claim, wherein each through-hole is square or rectangular.
6. 6. An optical marker as claimed in any preceding claim, wherein at least some of the through-holes are disposed in a linear row.
7. 7. An optical marker as claimed in claim 6, wherein at least some of the through-holes are disposed in multiple linear rows each of multiple through-holes.
8. 8. A display panel comprising a plurality of display elements and an optical marker attached to the panel, the marker having a reflective surface and a series of through-holes extending through the reflective surface, each such through-hole being located around one or more display elements of the panel so as to permit light from those display elements to shine through the hole.
9. 9. A display panel as claimed in claim 8, wherein the panel is a ceiling panel oriented to shine light downwards.
10. 10. A display panel as claimed in claim 8 01 9, the panel being suspended below a ceiling.
11. 11. A video studio comprising a display panel as claimed in any of claims 8 to 10, arranged to illuminate a video subject.
12. 12. A locating apparatus comprising a display panel as claimed in any of claims 8 to 10 and a camera, there being a locating apparatus attached to the camera, the locating apparatus comprising an illuminator and a locating camera, the illuminator and the locating camera being directed towards the display panel.
13. 13. An applicator for optical markers, the applicator comprising a locator for locating the applicator relative to a periodic light emitter structure and a template for holding an optical marker relative to the locator, whereby an optical marker held by the template can be located in registration with the light emitter structure.
14. 14. An applicator as claimed in claim 13, wherein the locator has a surface relief for physically mating with surface relief of the light emitter structure.
15. 15. An applicator as claimed in claim 13, wherein the locator has an optical detector for detecting positioning of the locator with the light emitter structure such that an optical marker held by the template is in registration with the light emitter structure.