GB2324429A

GB2324429A - Electronic zoom control in a virtual studio

Info

Publication number: GB2324429A
Application number: GB9706706A
Authority: GB
Inventors: Avi Sharir; Michael Tamir
Original assignee: Orad Hi Tec Systems Ltd
Current assignee: Orad Hi Tec Systems Ltd
Priority date: 1997-04-02
Filing date: 1997-04-02
Publication date: 1998-10-21
Also published as: GB9706706D0; AU6509998A; WO1998044723A1

Abstract

A camera in a virtual studio uses a background panel 18 as a reference to allow accurate location of the captured image in the virtual studio image. When the camera manually zooms in on, for example a face the amount of the reference panel 208 seen in the field of view is reduced until at the limit of the area indicated by dotted line 240 the panel is no longer visible and further manual zooming is not possible without the system losing lock. At this point the invention allows the cameraman to select an area of the image and electronically zoom in on it. To increase magnification and/or image quality texture mapping may be used to implement the electronic zooming.

Description

VIRTUAL CAMERA CONTROL SYSTEM The present invention relates to a video camera control system and more particularly to a system providing dual control of a video output from a camera.

The system is particularly useful for a chroma-key ("blue box") and for a virtual set apparatus.

In a virtual set apparatus a TV camera is used to capture a foreground object such as a presenter. In a preferred virtual set solution the foreground object is situated in front of a coded pattern composed of two shades of key colour (for example, two shades of blue).

In a chroma-key apparatus the foreground object is located in a plain blue (or another key colour) studio and the shooting camera is completely stationary and cannot pan, tilt, zoom or dolly. In a first virtual set system as described in copending PCT Application No. WO95/30312 to the present applicant a unique pattern is described enabling the camera to be freely moved.

The system depends on the camera field of view being wide enough to include at least some of the lines in the pattern. In co-pending PCT Application No. GB96/02227 to the same applicant a method is described for ensuring that only a minimum number of lines of the pattern are required to calculate the whole camera viewpoint data from the pattern.

There may however be instances where no lines are visible because the camera has zoomed in too far. In such cases the system will not be able to calculate the viewpoint information and to properly render the background image. To ensure that this does not occur the cameraman has to ensure that the field of view always includes sufficient lines of the background panel to maintain lock.

This prevents very close views of either presenters or real objects held by presenters - such as a newspaper or book and therefore detracts from the system performance.

It is an objective of the present invention to enable extremely close views to be obtained of foreground objects in virtual set systems without losing lock even if the camera continues to zoom in. It is another objective of the present invention to enable close ups and pan/tilt motions in a chroma-key production where the camera is absolutely stationary.

The present invention provides a video control system suitable for a virtual studio apparatus including a video camera, said control system comprising electronic image control means for selecting a portion of a video image captured by said camera, said electronic image control means being operative to perform zoom/pan and or tilt operations on said selected portion of said video image.

In the virtual studio apparatus the video image contains a foreground object and at least a portion of a patterned chroma-key panel.

The present invention also provides a virtual studio apparatus comprising a patterned chroma-key panel, a TV camera situated in a foreground area in front of the patterned chroma-key panel to be able to view the panel the TV camera providing an output to a video processor connected to the camera said video processor calculating the camera viewpoint information, a graphics computer using the calculated viewpoint to render a pre-designed 3D scene, a chroma-keyer used to generate the composited foreground/background image, the apparatus further comprising an electronic image control means, said electronic image control means being connected to the video processor to enable a further pan, tilt or zoom operation to be actuated in addition to any pan/tilt or optical zoom operation already performed by the camera.

Preferably the electronic image controller comprises joy stick control means for positioning a cursor on the video image provided by said TV camera said joy stick control means being operative to provide a pan/tilt/zoom action on the image to expand the region of the image proximate to said cursor position to enlarge a selected area of the image and to move inside" the image provided by camera 12 by virtually panning, tilting and zooming in/out.

The same can be done on an image produced by the fixed camera in a chroma-key apparatus.

The present invention also comprises a method of creating a television image comprising the steps of obtaining in real time a first video image of a defined area by operation of a video camera, displaying in real time said video image of said defined area, selecting a portion of the defined area expanding the portion of the defined area to fill the complete television image, displaying the expanded portion simultaneously with the display of said video image of said defined area and moving the expanded portion "inside" the said video image.

Preferably the step of enhancing said video image of said portion is by texture mapping.

Preferably the method is used in a virtual studio system, the defined area comprising at least a substantial portion of a patterned chroma-key background panel said substantial portion being sufficient to enable perspective transformation calculation of a virtual background to be included in said television picture, and in which said selected portion of the defined area does not include enough of said background panel to enable perspective transformation calculation of the virtual background.

Embodiments of the present invention will now be described, by way of example with reference to the accompanying drawings, in which: Figure 1 shows a known virtual studio system incorporating a patterned chroma-key panel as described in copending PCT Application No. WO95/30312 and is incorporated for explanation; Figure 2 shows a portion of the patterned chroma key panel of Figure 1 illustrating the present invention; Figure 3 shows diagrammatically apparatus according to the present invention; Figure 4 shows a video picture illustrating the technique of texture mapping as applied to the system of the present invention; Figure 5 shows schematically a remote controlled camera system incorporating the apparatus of the present invention; Figure 6 shows a first video picture illustrating an area to be enlarged according to the present invention; and Figure 7 shows the selected area of Figure 6 as enlarged in accordance with the present invention.

With reference now to the drawings, Figure 1 shows a virtual studio 10 with a camera 12. A foreground object, in this case a presenter 14, is positioned in the foreground area 16 in front of the camera 12.

In the preferred embodiment now described a background chromakey panel 18 is provided which has a plurality of vertical and horizontal stripes 20,22 forming a unique pattern. The operation of this pattern in relation to a virtual studio is described in the above mentioned copending PCT applications.

The camera 12 in this example is assumed to be operated manually by a cameraman (not shown). The camera can move (it can be shoulder mounted) in all directions within the area 16 and can also pan, tilt and zoom under manual control.

As illustrated in Figure 2, when the camera zooms in to the face 140 of the presenter 14 the number of lines 20,22 seen in the field of view of the camera becomes smaller. In this case the picture or image 24 as shown in Figure 2 contains only two horizontal lines 202,204 and two vertical lines 222,224. As described in the second copending PCT Application No. GB96/02227 the system is still able to work even though these lines are not continuous due to their disappearance behind face 140.

However, if we now consider zooming in further to the area indicated by dotted line 240, it can be seen that no lines are visible because the face obscures them. This will also be true of other objects onto which a close zoom is required.

The system will start to lose lock if it cannot see the lines 22,24 on att least portions of the lines and therefore the cameraman is constrained in close ups to ensure that the lines are visible in the picture 24.

In the present invention as now described with reference to Figures 3 to 7, this disadvantage is overcome by providing an electronic zoom control separate from the camera optical zoom control. This zoom control may be operable within the control room of the studio or could be operable by the cameraman with a separate unit. The system operates as follows.

The cameraman zooms in to the picture as shown in Figure 2 and then ceases to zoom. The image to be transmitted is then shown in picture frame 24.

For a closer zoom an electronic zoom controller is used to identify a smaller area, as illustrated with reference also to Figures 6 and 7. The smaller area may be identified by using a cross hair cursor 242 to centre onto the subject to be shown in greater detail. The zooming is then accomplished by use of a joystick or other suitable control to continue zooming onto the area 240 the image (of rocks and trees) being magnified as shown in Figure 7 to fill the complete image to be transmitted as shown in figure 7.

The system does not ever lose lock because the image information input into the virtual studio circuitry still contains the image 24 as shown in Figure 2 or Figure 6 which still contains the image of the lines 202,204 and 222,224. Thus the viewer sees a close up of the object (either rocks 62,64 and tree 66 of Figure 6 or the face 140 of presenter 14 in Figure 1) and the camera 12 can continue to pan and tilt and move within the area 16 to provide differing close up views of the objects.

Because of the electronic zooming there will be a slight degradation in the quality of the picture because the image 240 has to be expanded to fill the whole picture screen 24. There is obviously a practical limit of acceptability in relation to the quality of the picture able to be transmitted but theoretically there is no real limit.

In practice, with a standard TV camera 12 a 50% magnification does not degrade the transmitted picture substantially. With a high definition camera and using high resolution background greater electronic zoom magnification may be practically acceptable.

Techniques such as texture mapping as illustrated diagrammatically in Figure 4 may assist in allowing greater magnification or better quality pictures with the same magnification. Figure 4A illustrates within the rectangle 241 a target (original image) and Figure 4B the destination enlarged image. In Figure 4A area 244 is to be expanded to become area 246 in Figure 4B. The average value of pixels in area 244 is used for area 246 to provide the magnified area. Thus the magnified image comprises the same pixel density as the previous image but the value of the pixels in a specified area is averaged. This is often not noticeable in any way to the viewer and therefore the electronic zoom does not practically degrade the picture.

The apparatus and control circuitry 30 is shown diagrammatically in Figure 3.

The camera 12 provides a foreground video image on line 122 to a perspective transformation unit 32. the perspective transformation unit 32 extracts from the video image the perspective information derived from the background panel 18. The foreground video is then output on line 322 to chroma-keyer 34 which may be, for example, an ULTIMATE chroma-keyer in which foreground and background are combined to be output on line 342 and subsequently transmitted by transmitter 36.

The background is provided by, for example, a graphics computer 38 on line 382.

The background is generated by the graphics computer via rendering of a predesigned graphical model at the calculated viewpoint.

An electronic zoom controller 40 is connected to the graphics computer 38 and to the perspective transformation unit 32 via respective outputs 402,404, the controller 40 having suitable controls such as a joy stick 42.

Many monitors will normally be provided in the control booth to show the video router or studio director the image being transmitted or recorded. In this example three monitors are shown. The first monitor 44 will show the scene as viewed by camera 12 with the background panel 18. This enables the studio director to ensure that this image has sufficient of the panel 18 in the picture frame 24 to ensure adequate locking. It is possible to provide an electronic indication to the studio director when this is occurring to assist in the control function. If such loss of lock occurs then the camera must zoom back or move back.

The joy stick 42 is then used by the studio director if he requires a closer zoom and the zooming is monitored on monitor 46 which shows the selected foreground image zoomed in as selected by the control 42 and electronic zoom controller 40. The studio director can also electronically alter the pan or tilt as well as and/or in addition to the zoom.

The background is also modified by the controller 40 which controls graphics computer 38 to magnify the background or a virtual foreground object in the correct perspective and size to match the electronically zoomed image.

It is noted here that if the foreground object 140 completely fills the picture frame then there will be no requirement for a background image but virtual objects may be present which pass in front of the face. Also if only a minute amount of background is shown such as the area 242 in Figure 2, this will require to be added.

The combined optically and electronically zoomed (and also panned/tilted) picture will be shown on the monitor 48 to enable the producer to see the effect of the electronic zooming/panning/tilting and in particular the picture quality.

In Figure 5 a system is shown in which no cameraman is required.

The camera 12' is in this case assumed to be static but has a mount with pan, tilt motors and a lens with zoom and focus motors controlled by a controller 500 which is operable by the video router or producer in the studio control room.

The virtual studio is provided with 2D high resolution background 502 which may comprise, for example, weather or other maps.

The camera 12' can be zoomed into the background until only a small area is shown in the picture. To zoom in further would lose lock as explained above but by using the electronic zoom controller within the processing apparatus 30 further magnification of the image can be obtained to produce a more detailed picture. In this case, due to the use of a 2D high resolution background, no picture quality will be lost for any practical system unless of course the electronic zoom is carried to absurdity.

The camera 12 can therefore be zoomed into a very small area which can then be further magnified. It can then be zoomed back smoothly without any loss of lock.

The system therefore enables the studio controller to select extremely close up views in a virtual studio apparatus without cutting to a different camera or having to insert close ups later in an edited version.

Claims

1. A video control system suitable for a virtual studio apparatus including a video camera, said control system comprising electronic image control means for selecting a portion of a video image captured by said camera, said electronic image control means being operative to perform zoom/pan and or tilt operations on said selected portion of said video image.

2. A video control system as claimed in claim 1 wherein said video camera is a static camera.

3. A video control system as claimed in claim 1 wherein said video camera is capable of movement in three planes x,y and z.

4. A video control apparatus for a chroma-key production (blue box) system, said control system comprising electronic image control means for selecting a portion of a video image, said electronic control means being operative to perform zoom/pan and/or tilt operations on said selected portion of said video image.

5. A virtual studio apparatus comprising a patterned chroma-key panel, a TV camera situated in a foreground area in front of the patterned chroma-key panel to be able to view the panel; the TV camera providing an output to a video processor connected to the camera said video processor comprising a graphics computer to incorporate a real foreground object into a virtual background scene generated within the ~ video processor, the apparatus further comprising an electronic image control means, said electronic image control means being connected to the video processor to enable a further pan, tilt or zoom operation to be actuated in addition to any pan, tilt or optical zoom operation already performed by the camera.

6. A virtual studio apparatus as claimed in claim 5 in which the electronic zoom controller comprises joy stick control means for positioning a cursor on the video picture provided by said TV camera said joy stick control means being operative to provide a zoom action on the picture to expand the region of the picture proximate to said cursor position to enlarge a selected area of the picture.

7. A method of creating a television picture comprising the steps of obtaining in real time a first video image of a defined area by operation of a video camera, displaying in real time said video image of said defined area, selecting a portion of the defined area, expanding the portion of the defined area to fill the complete television picture, displaying the expanded portion simultaneously with the display of said video image of said defined area.

8. A method of creating a television picture as claimed in claim 7 further comprising the step of enhancing said video image of said portion by texture mapping.

9. A method of creating a television picture as claimed in claim 7 in a virtual studio system in which the defined area comprises at least a substantial portion of a patterned chroma-key background panel said substantial portion being sufficient to enable perspective locking of a virtual background to be included in said television picture, and in which said portion of the defined area does not include enough of said background panel to enable perspective locking of the virtual background.