Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
  • H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B15/00—Special procedures for taking photographs; Apparatus therefor
  • G03B15/08—Trick photography
  • G03B15/10—Trick photography using back-projection, i.e. blending artificial background with real foreground
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
  • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
  • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
  • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
  • H04N5/275—Generation of keying signals

Definitions

• This invention relates to controlling one or more of (i) the focussing operations of a camera and (ii) the operation of a controllable backdrop.
• An alternative approach is to use a screen as the background, to display a still image or a video on the screen, and to video the subject against the screen with a stationary or moving camera.
• the screen can be a plain screen onto which the still image or video is projected, or it can be display device or an array of display devices. This approach can reduce the amount of post processing that is needed, but it requires a display system to present the background on the display - typically a graphics engine - at the location where the video is shot.
• Figure 1 shows apparatus for capturing video of a subject against a dynamic background.
• a camera 1 is located so as to capture video of a subject 2 against a display wall 3.
• the system is configured so that focus of the camera is controlled in dependence on the relative locations of the camera and the display wall.
• the system is also configured so that the image on the display can be blurred depending on a focus setting for the camera.
• the image may be a still image or a frame of a video.
• Camera 1 is capable of capturing video. It has a lens 4 whose focal point can be adjusted by motor 5. It has an image sensor (not shown) for sensing an image. An image sensed by the sensor can be stored by the camera or transmitted by the camera for storage or display elsewhere.
• the camera may have another motor for adjusting the lens’s depth of field.
• the field of view and/or aperture of the lens may be independently adjustable, optionally using other motors.
• the focus of the camera may be controlled by an operator at the camera. This option will be discussed in more detail below.
• the operator of the camera may control the direction of the camera and a focus puller may control the camera’s focus using follow-focus 6 which is remote from the camera.
• the follow focus comprises a knob 7 or other input device by which the focus puller can provide inputs to control the focus of the camera.
• a display controller 8 controls the still image or video shown on screen 3.
• the display controller is linked to a data store 9 which stores still and/or video data (which may in each case be photographic or computer-generated) to be displayed.
• the display wall 3 is composed of an array of display screens 3a.
• it could be a screen on to which images are projected.
• the focus controller 6 is connected directly to the camera 1 by a wired or wireless link.
• the follow-focus communicates with the camera via a link device 10.
• the link device receives focus commands from the follow- focus over a link 11 and forwards at least some of those commands to both the camera 1 and the display controller 8 over links 12 and 13 respectively.
• Each of links 11 to 13 may independently be a wired or a wireless link.
• the link device presents the same logical and/or physical interface to the follow focus as the camera 1 presents.
• the link device presents the same logical and/or physical interface to the camera as the follow-focus presents.
• One or both of these features can mean that the link device can readily be interposed between a pre-existing camera and follow-focus.
• the follow-focus may be adapted to work with the link device.
• Link device 10 and the display controller 8 together provide a mechanism for controlling aspects of the camera 1 and image(s) that are displayed on the display wall 3.
• Link device 10 may comprise a processor 14 and a memory 15 storing in a non transient way code executable by the processor to cause the link device to perform a set of the operations described herein.
• Display controller 8 may comprise a processor 16 and a memory 17 storing in a non-transient way code executable by the processor 16 to cause the display controller to perform a set of the operations described herein.
• the split of functions between the link device and the display controller can be chosen in any convenient way, and may be shared with other devices not shown in figure 1.
• Subject 2 performs between the display wall 3 and the camera 1.
• the camera is operated so that the subject falls within the camera’s field of view.
• the camera then captures video of the subject with the appearance of the display wall as a backdrop.
• focus control it may be desired for the captured video to be focussed on the subject.
• the focus puller adjusts the follow-focus 6.
• the follow-focus sends control signals to cause the motor 5 of the camera to adjust the lens 4 of the camera so the camera maintains focus on the subject. These signals are forwarded unchanged to the camera by the link device 10.
• the focus puller adjusts the follow-focus to cause it to send signals that are such as to command the focus of the camera to shift from the subject to the object.
• the subject is positioned closer to the display wall 3 than the camera.
• the object is displayed on the display wall and appears to be further from the camera than the display wall.
• the focus command signals from the follow-focus are processed by the link device 10. If the focus command signal from the follow-focus commands the camera’s focus to be in the range from the camera to the display wall, then the link device passes that signal to the camera.
• the motor 5 causes the focus of the camera to change in dependence on the signal received by the camera. If the focus command signal from the follow-focus commands the camera’s focus to be more distant from the camera than the display wall then the link device does not pass that signal to the camera, and signals the display controller 16 with the desired focus position.
• the display controller then performs an image processing operation on the image(s) it is displaying on the display wall 3 to adapt its appearance in dependence on the desired focus position. For example, it may apply a blur to portions of the image(s) other than those that are intended to appear at the distance from the camera that corresponds to the desired focus position. Since the focus of the camera is kept on the display wall, the camera can accurately capture the appearance of the display wall. The opposite can be done when a series of commands is generated to move the focus of the camera from a location beyond the display wall to a location between the display wall and the camera.
• the camera 1 may be adapted to implement a focus limit or cap on the distance at which the camera lens focusses.
• the link device 10 may determine this limit or cap depending on the distance between the camera and the display wall.
• the link device may then send this limit to the camera 1 which acts as an upper bound on the camera motor 5 so that the focus of the camera lens can only be adjusted up to and including that cap or limit. For example, if the camera were positioned 10m away from the display wall, the limit on the focus distance of the camera lens would be capped at 10m.
• the camera motor would adjust the focus of the lens according to the distance required by any command from the follow-focus below the cap or limit.
• the link device may send the same command from the follow-focus to the display controller, which would adjust the screen accordingly so that the focus was now on the virtual object. In this set up, the link device transmits the follow-focus command to the camera, along with the determined limit. If the camera 1 does not move, and assuming the display wall is stationary, the distance between the camera and the display wall remains the same, so the limit does not need to be re-sent with every new command from the follow-focus.
• the limit sent to the camera by the link device is dependent on the position of the camera, which may be calculated in any appropriate way. If the position of the camera 1 changes, the limit may need to be accordingly adjusted.
• the display controller 8 may select a range of virtual distances for displayed objects that are not to be blurred. For example, if focus is set at 20m, objects at a virtual distance between 18m and 22m might not be blurred and objects at distances outside that range may be blurred. The depth of the range that is not blurred may be dependent on the current depth of field of the lens 4, which may be signalled to the link device 10.
• the display controller 8 can use information indicating the distances from the camera of subject matter in regions of the display. Information indicating the distances may be pre-stored with the data in store 9.
• the distance information may be inferred by the display controller 8, e.g. using a trained distance estimation algorithm.
• the blurring of the display wall by the display controller 8 may be dependent on information indicating the distance of the camera 1 from the display wall.
• the position of the camera 1 may be sensed by a location sensor 18 on the camera and signalled to the display controller.
• the display controller may have pre-stored the location of the display wall 3 so it can determine the distance of the camera from the display wall.
• the distance of the camera from the wall may be small relative to the virtual distance from the camera to an object shown on the display wall so satisfactory results may be obtained by assuming a distance (e.g. 10m) from the camera to the display wall.
• a camera position sensed by sensor 18 may also be signalled to link device 10 and used together with knowledge of the location of the display wall 3 to determine at what depth to limit the camera’s focus.
• the position of the camera may be estimated in any convenient way.
• a sensor on the camera that senses fiducial markers, beacons or other features in the environment of the camera, and processing circuitry on the camera or remote from it may use the data from that sensor to estimate the camera’s position.
• the camera could carry a marker or beacon that may be sensed by one or more sensors in the environment, and processing circuitry could then use the output of such sensor(s) to estimate the location of the camera.
• the absolute location of the camera may be estimated.
• the location of the camera relative to the screen may be estimated.
• the link device 10 may receive from the camera information indicating the camera’s depth of field.
• the link device may receive from location sensor 18 the location and orientation of the camera 1 .
• the link device may receive from the camera a zoom setting of lens 4.
• the link device may perform a geometric calculation to estimate the range of distances from the camera of those parts of the display wall as are within the field of view of the camera.
• the link device may then, when the camera’s focus is intended to be on the display wall, signal the camera so as to adjust its depth of field (e.g. by extending its depth of field and/or adjusting the aperture of the lens) to maintain the whole of the display wall within its field of view in focus.
• the link device may be configured to mitigate such effects.
• the link device may be configured to estimate focal configurations of the camera at which such effects are likely to occur. This may be estimated in dependence on information about the relative location of the camera and the display wall and the lens setting of the camera, all of which can be obtained as described above, and information about the pixel spacings of the camera and the display wall, which may be provided in advance to the link device. The link device may then avoid the camera dwelling in focus configurations for which Moire is likely to occur.
• the follow-focus commands the camera to adopt such a configuration, commanding the camera to instead adopt a configuration in which the focus is long or short of that configuration.
• the follow-focus is remote from the camera.
• focus commands could be provided by a focus controller integral with the camera.
• the link device 10 could be integrated with the camera or part of it, for example the lens.
• the devices 1 , 10, 6, 8 may be coupled by dedicated or by shared interfaces on the respective devices.
• an output from link device 10 to display controller 8 may be constituted by a dedicated physical interface on link device 10 or by the link device transmitting signals addressed to display controller 8.
• the link device 10 may forward to the display controller all focus commands received from the follow-focus 6, or may forward such commands only when the focus of the camera is commanded to be at or beyond the screen 3.
• the embodiment discussed above may be convenient to implement because there may be no need to adapt the focus controller and/or the camera.
• the device 10 can simply be interposed between the two.
• the control of the motor of the camera can be capped by this limit.
• the device 10 may control the focus of the lens in a number of ways. Once a focus limit has been determined, commands from the focus controller 6 to focus the camera at a distance beyond that limit may simply not be forwarded to the camera. Alternatively, such commands may be adapted by device 10 to be commands to focus the camera at the limit and then forwarded to the camera. Alternatively, the device 10 may signal the limit to the camera and the camera itself may then block focussing beyond that limit. The device 10 may forward both the limit and the focus puller command distance, and the camera 1 may focus using the lower of the two distances. In each case, the functions of the camera may be performed by any relevant parts of the camera, including the lens.
• the focus of the camera lens should ideally be adjustable in response to both a new focus controller command and in response to movement of the camera itself.
• the position of the camera in front of the display wall is monitored with a camera position sensor 18.
• the focus command from the focus controller remains fixed on a virtual object on the display wall whilst the camera moves around in front of the display wall.
• the camera is positioned 20m away from the display wall, and the focus puller is focussed on an object at 30m away, so the focus command has surpassed the display wall by 10m and the camera has been set to focus at 20m.
• the camera were to move forward by 5m, it may be the case that the camera does not receive any indication to adjust the lens focus, because the focus puller’s command is still to focus on the virtual image at 10m from the display wall which has remained stationary. This would leave the camera no longer focused on the display wall because it moved forward by 5m.
• the location of the camera In order for the focus of the camera to remain on the display wall, the location of the camera must be known so that the focus of camera lens can be adjusted to remain on the wall if it is detected that the camera has moved position.
• the screen is one that can adaptively display information, and that information can be captured by the camera during filming.
• the information may be displayed by being emitted from light emitting devices on the screen (as in the case of an LED wall) or by being projected on to and reflected from the wall.
• the wall may be a screen suitable for chroma keying to replace the parts of an image that include the screen with other information.
• a chroma key screen may be of a colour that is separable from the subject 2, for example a so- called green screen or a so-called blue screen.
• the screen may be of a single hue.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Studio Devices (AREA)
• Focusing (AREA)
• Projection Apparatus (AREA)
• Indication In Cameras, And Counting Of Exposures (AREA)
• Studio Circuits (AREA)
• Automatic Focus Adjustment (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=72339419

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (2)

Citations (2)

Family Cites Families (15)

• 2020
  • 2020-07-24 GB GBGB2011524.2A patent/GB202011524D0/en not_active Ceased
• 2021
  • 2021-07-23 WO PCT/GB2021/051907 patent/WO2022018455A1/en not_active Ceased
  • 2021-07-23 EP EP21751852.1A patent/EP4186224A1/en active Pending
  • 2021-07-23 US US18/005,894 patent/US12425731B2/en active Active
  • 2021-07-23 GB GB2110618.2A patent/GB2596235B/en active Active
  • 2021-07-23 CN CN202180060345.4A patent/CN116195243A/zh active Pending
  • 2021-07-23 JP JP2023503131A patent/JP2023534505A/ja active Pending

Patent Citations (2)

Also Published As

Similar Documents

Legal Events