JPH1032755A - Large-screen image composing processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate the image of ultrahigh resolution and a wide angle in short time by radioscopy-converting an image face and generating a large screen through the use of camera data obtained by means of the operation of a television camera. SOLUTION: The image-picked up image from the television camera 1, a swing angle (pan and tilt angle) and the angle of view on the image-picked up image outputted from a camera data detection part 2 at the same timing are inputted to the image conversion circuit 7 of a large-screen image generation part 3. An image conversion circuit 7 calculates a parameter necessary for the geometric conversion (radioscopy conversion) from the swing angle and the angle of view, converts the image-picked up image by using the obtained parameter and outputs a display position address and the converted image in the address. A large-screen background image written into a large screen background memory 9 in the large-screen image generation part 3 and the large-screen foreground image written into a large-screen foreground memory 10 are composed in an image composer 11 to obtain a large-screen composed image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image signal processing apparatus for processing a remote sensing image, and more particularly, to a method for successively dividing a plurality of television image signals obtained by spatially dividing and photographing the same scene. The present invention relates to a large-screen image synthesizing apparatus for reconstructing a reconstructed large-screen image signal, and a large-screen image synthesizing apparatus for fitting a moving subject such as a car into the reconstructed large-screen image as it is or as a strobe locus.

[0002]

Generally, in order to obtain an image with a wide field of view, a method of attaching a wide-angle lens or a fish-eye lens to a camera and taking an image is considered. However, this method has a drawback that only low-resolution and distorted images can be captured.

In order to obtain a high-resolution image, a method of capturing a still image such as a photograph by a scanner is conceivable. However, this method has drawbacks that a moving image cannot be handled and it takes time to capture the moving image.

On the other hand, as a conventional technique for reconstructing a continuous large screen from a plurality of divided and captured images, for example, a continuous sensing image of the earth by a satellite or a remote sensing image such as photogrammetry using an aircraft is known. There is a joining technique. However, in all of these, photographing (vertical photographing) is basically performed under the condition that the photographing target surface and the projection surface face each other, and the image joining technique optically corrects a deviation at a pass point on a plotter. Things are central.

Further, Japanese Patent Publication No. Hei 6-3 related to the present applicant's application
The “video synthesizing method” disclosed in Japanese Patent No.
A plurality of television image signals obtained by spatially dividing and imaging the same scene are reconstructed into a continuous large-screen image signal. It only makes discrepancies at the boundaries inconspicuous.

An object of the present invention is to provide a novel large-screen image synthesizing processing apparatus which solves the above-mentioned problems.

[0007]

SUMMARY OF THE INVENTION As described below, the present invention detects camera data consisting of a swing angle of a television camera and an angle of view of a picked-up image, and converts the picked-up image from the detected camera data. By calculating the parameters required for the geometric transformation (here, perspective transformation), the images are perspective-transformed based on this, and connected to each other,
A large-screen image with a high resolution and a wide angle of view is created.

[0008] More specifically, in the present invention, a fixed-point camera (television camera) is imaged while continuously panning and tilting, and the image is obtained from individual images obtained in an arbitrary video field or video frame. Based on the camera data at the time, each of the perspective-transformed images (plural) is electrically generated on a virtual plane placed in real space, and the generated images are connected adjacent to each other to create a large screen image I am trying to do it.

That is, the large-screen image synthesizing processing apparatus of the present invention continuously pans each partial image constituting the large-screen image,
A television camera that captures images while tilting, a camera data detection unit that detects a swing angle of the television camera and an angle of view of an image captured by the camera for each arbitrary video field or video frame; Based on the swing angle and the angle of view detected in the section, an image conversion circuit that perspective-transforms a plurality of captured images captured for each of the arbitrary video fields or video frames, and a plurality of perspective-transformed images in the image conversion circuit. And a large-screen memory for storing the image at a predetermined address determined by the perspective transformation.

[0010] The large-screen image synthesizing apparatus according to the present invention comprises:
The large-screen memory is provided separately for the background and foreground, and in each of those large-screen memories, stores a plurality of perspective-transformed images of the background image and the foreground image,
The stored images are read from the respective large-screen memories by matching the addresses of both memories, and the foreground image is overwritten on the background image within a range not exceeding the area where the foreground image data exists. In which an image in which a foreground image is fitted is obtained.

The large-screen image synthesizing apparatus according to the present invention comprises:
An image is read from each of the large screen memories after the background image is stored at all addresses of the background large screen memory.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on embodiments with reference to the accompanying drawings. FIG. 1 is a block diagram showing an embodiment of a large-screen image synthesis processing device according to the present invention. As shown in the figure, the apparatus of the present invention is roughly divided into a television camera 1 and a camera data detection unit 2 for detecting a swing angle such as pan and tilt and an image angle of view of the television camera and the captured image, respectively.
A large-screen image generating unit 3 for storing, at a predetermined address in a memory, a converted image obtained by perspectively converting a captured image based on the swing angle and the angle of view; and a plurality of memories for an output image of the large-screen image generating unit 3. In this case, the output image is overwritten in a predetermined area and in a predetermined order, and then a geometric conversion circuit 4 for performing necessary geometric conversion such as enlargement, reduction, and movement. Note that the last geometric transformation circuit 4 in the above is not always necessary for configuring the present invention.

Referring to FIG. 1, the operation of the apparatus will be described along the flow of images. First, the swing angle detector 5 included in the camera data detection unit 2 receives pan and tilt information from, for example, the camera platform of the television camera 1, and generates a large signal centering on a predetermined direction with respect to the received signal. Considering the generation of a screen image, the swing angle when the camera is pointed at the center of the screen image is set to zero degree, and the swing angle of the television camera from there is calculated to obtain a field unit (1/6).
(Every 0 seconds). Similarly, the angle-of-view detector 6 of the camera data detection unit 2 receives zoom information from, for example, a zoom lens of the television camera 1, calculates the horizontal and vertical angle of view of the image based on the zoom information, and (Every 1/60 second).

In the above description, the pan / tilt information and the angle-of-view information can be obtained by attaching a high-resolution rotary encoder or linear encoder to the pan axis and tilt axis of the camera platform and the zoom lens of the television camera, respectively.
A signal having a pulse number proportional to the swing angle and the angle of view can be obtained from each of the rotary encoders and the like.

The captured image from the television camera 1 and the swing angles (pan and tilt angles) and the angle of view of the captured image output from the camera data detection unit 2 at the same timing are converted by the image conversion unit 3 of the large screen image generation unit. Input to the circuit 7. The image conversion circuit 7 calculates parameters necessary for geometric transformation (perspective transformation) of the image based on the swing angle and the angle of view, performs transformation of the captured image using the obtained parameters, Correspondingly, a display position address indicating a position on the large screen to be displayed and a converted image at the address are output.

Here, the principle of perspective transformation will be described.
FIG. 2 shows an image plane (camera image plane) obtained by capturing an image of a subject such as a background at a certain time on a large screen memory (corresponding to a large screen display screen). It is a conceptual diagram showing the principle of what is called perspective transformation.

In FIG. 2, it is assumed that the pan and tilt angles are measured as P and T (degrees) from the angle detector 5 (see FIG. 1) in the direction in which the camera is facing. It is also assumed that the horizontal angle of view is measured as Zh (degrees) from the angle of view detector.

Here, the viewpoint is placed at the same position as the camera, and the distance F from the viewpoint to the large screen (projection plane) to be projected is set.
Is set in advance. Further, with the viewpoint set at the origin (0, 0, 0) in the (x, y, z) orthogonal coordinate system, the axis perpendicular to the projection plane is the z axis, the horizontal axis parallel to the projection plane is the x axis, and the projection plane is the x axis. The parallel vertical axis is defined as the y-axis.

As shown in FIG. 2, if the screen size of an image plane captured by the camera (hereinafter simply referred to as a camera image plane) is H and V (pixels) in each of the horizontal and vertical directions, the horizontal angle of view Zh From the horizontal screen size H, the distance L from the viewpoint to the camera image plane is represented by L = H / (2 × tan (Zh / 2)).

A pixel point (X2, Y
2, Z2) is a point (X1, X2) at which a line connecting the point (X2, Y2, Z2) and the viewpoint (0, 0, 0) intersects the projection plane.
Y1, Z1) is a point projected while maintaining the same pixel information as that point, so that points (X1, Y1, Z1) and (X2,
Y2, Z2). That is,

(Equation 1)

Further, since the camera image plane is a sampling image, if the pixels constituting the camera image plane are coordinate-transformed on the projection plane in a one-to-one relationship, the pixels on the projection plane are, for example, square grids. Will not line up regularly. For this reason, the value of a pixel serving as a sampling point on the projection plane is generated by interpolation using an interpolation filter using, for example, the value of a pixel coordinate-converted from a camera image plane located near the pixel. As the simplest interpolation filter, a filter that performs weighted addition in inverse proportion to the distance to the four neighboring pixels can be used.

Next, when synthesizing a large screen image according to the present invention, it is possible to select either to perform this for a background image (background capture mode) or to perform on a foreground image such as a moving subject (foreground capture mode). By combining the two images obtained separately, it is possible to display, for example, a running car (foreground) in real time in a background configured in advance on a large screen according to the present invention. . This will be described below.

When the background capture mode is designated on the operation unit 8 shown in FIG. 1, the large screen background memory 9 is selected by the switch SW, and the output of the image conversion circuit 7 is the point (X2, Y2, Z2) on the camera screen. Is written to the address (X1, Y1, Z1) of the large screen background memory 9. Also,
When the foreground capture mode is specified on the operation unit 8,
The large screen foreground memory 10 is selected by the switch SW, and the output of the image conversion circuit 7 is similarly output to the large screen foreground memory 1.
0 is written to the address (X1, Y1, Z1).

FIG. 3 is an explanatory diagram showing a processing procedure in the background capture mode. In FIG. 3, an operation (tilt, pan,
(Zoom) to calculate the address corresponding to each pixel by instantaneous calculation of the camera image plane at a certain pan angle P ', tilt angle T', and angle of view Z '(the calculation method has already been described with reference to FIG. 2). While performing perspective transformation, the transformed image is sequentially written to the display position address of the large screen background memory 9 (see FIG. 1).

As described above, unless the camera is operated extremely quickly, continuous field images include overlapping portions, and as a result, pixel data can be written to the large-screen background memory 9 without gaps. According to this processing procedure, a large screen background image can be created.

FIG. 4 is an explanatory diagram showing a processing procedure in the foreground fetch mode. In FIG. 4, a camera image plane of a certain pan angle P ″, tilt angle T ″, and angle of view Z ″ corresponds to each pixel by instantaneous calculation according to a camera operation following a moving subject (in this example, a car). The perspective conversion is performed while calculating the address to be converted, and the converted image is sequentially written to the display position address of the large-screen foreground memory 10 (see FIG. 1). Images can be created.

However, in this case, unlike the case of the large-screen background image, the image has data only in the range captured by the camera operation following the subject.

For example, by taking timing control from the operation unit as shown in FIG. 1, the capture timing of the captured image is performed in video frame units, and the area written in the previous frame on the large screen foreground memory 10 during the vertical blanking period. Is cleared to zero, and the converted image of the current frame is sequentially written to the display position address of the large screen foreground memory 10. This operation is performed for each video frame, and an image is synthesized with a large-screen background image created in advance as described later, so that a moving subject with an arbitrary camera angle can be displayed.

If a moving subject is not cleared to zero during the blanking period, the foreground image on the large screen is displayed as a strobe locus. In this strobe trajectory display, for example, the interval of the strobe display is determined by thinning out the input timing of the camera image plane at predetermined time intervals by timing control from the operation unit 8 (see FIG. 1). Can be used for special effects.

The large screen background image and the large screen foreground image (both output from the large screen image generation unit 3 shown in FIG. 1) obtained as described above do not exceed the area where the foreground image data exists. The foreground image is overwritten on the background image by the image synthesizer 11 in the range, and the output is supplied to the geometric conversion circuit 4, and the final output image is obtained from the geometric conversion circuit 4. Note that the overwriting can be performed on all the areas where the foreground image data exists, or a subject key signal may be generated as described later and may be performed only on the subject portion.

FIG. 5 is an explanatory diagram showing a processing procedure when a large screen background image and a large screen foreground image are combined and a part of the combined image is extracted and reduced. In FIG. 5, a large screen background memory 9 in the large screen image generation unit 3 (FIG. 1)
(The same applies hereinafter.) The large-screen background image written in the large-screen foreground memory 10 and the large-screen foreground image written in the large-screen foreground memory 10 are combined by the image combiner 11 to form a large-screen combined image.
This large screen composite image is cut out at a site (position) to be output in the composite image by the operator operating the operation unit 8, and a reduced output image is output as an output of the geometric transformation circuit 4. Here, the reduction can be changed to the enlargement, and the reduction ratio and the enlargement ratio can be arbitrarily controlled via the operation unit 8.

As can be seen from FIG. 5, the large-screen background image is read repeatedly after the image is completed, that is, after the background image is stored in all addresses of the large-screen background memory. On the other hand, the large-screen foreground image is desirably written to and read from the large-screen foreground memory in real time. This is because the background is often a still image.

In the above-described embodiment of the present invention, the large-screen memory has two surfaces, that is, a large-screen background memory and a large-screen foreground memory. Compositing can also be achieved.
In addition to the method of attaching a measuring device to the camera platform described in the present embodiment, the camera data
al, “Estimation of camera Parameters from image se
quence for model-based video coding ”, IEEE Trans.
As disclosed in CAV-VT, vol. 4, no. 3, June, 1994, the method may be changed to a method of detecting a captured image of a camera by image processing.

Further, in the present invention, since perspective transformation is used as a technique for generating a large screen image, depending on conditions such as outdoors, the luminance and the color tone may change with time, causing a sense of discomfort at the boundary between the foreground and the background. Is also conceivable. To match the lightness and color tone of the foreground and background, see, for example, Okui et al.
The use of the brightness color tone adjustment circuit described in Proceedings of the 13th Annual Meeting of the Institute of Image Electronics Engineers of Japan enables synthesis without uncomfortable feeling.

Further, for example, a difference between the foreground and the background is obtained for each of the RGB signals, and a key signal is generated by setting a threshold value to "1" and a threshold value or less to "0" at a certain threshold value. Thereby, it is possible to extract the subject. Therefore, when the strobe track image is generated, even if the converted image overlaps the previous frame and the current frame, the RG
By overwriting only the image of the subject where any of the key signals B is "1" in the memory 10, it is also possible to display the strobe locus image of the subject when the strobe display interval is short.

[0036]

According to the present invention, an image plane captured by a television camera is perspective-transformed using camera data (swing angle, angle of view) obtained by operation of the television camera to create a large screen. Therefore, an image having an ultra-high resolution and a wide angle of view can be created in a short time as an image finally obtained.

Further, according to the present invention, by preparing a plurality of large screen memories, it is possible to display moving images and strobe trajectories.

Further, the geometric transformation circuit according to the embodiment of the present invention can output an image of an arbitrary part (position) and size from the created large screen image, and realizes a free camera work by post-processing. be able to.

Also, in contrast to the prior art, the present invention does not require geometrical matching between images that are in contact with each other when creating a large screen image. High-speed processing is possible because the images are connected only by calculating the parameters of the perspective transformation only from the angle of view of the images and performing image transformation. Further, by providing two or more large-screen memories, it is possible to input a moving subject by converting and writing the input image separately for the foreground and the background.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a large-screen image synthesis processing device according to the present invention.

FIG. 2 is a conceptual diagram illustrating the principle of perspective transformation.

FIG. 3 is an explanatory diagram showing a processing procedure in a background capture mode.

FIG. 4 is an explanatory diagram showing a processing procedure in a foreground capturing mode.

FIG. 5 is an explanatory diagram showing a processing procedure when a large screen background image and a large screen foreground image are combined and a part of the combined image is extracted and reduced.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Television camera 2 Camera data detection part 3 Large screen image generation part 4 Geometric conversion circuit 5 Shake angle detector 6 Angle of view detector 7 Image conversion circuit 8 Operation part 9 Large screen background memory 10 Large screen foreground memory 11 Image synthesis Switch SW P, P ', P "Pan angle T, T', T" Tilt angle Z, Z ', Z "View angle

Claims (3)

[Claims] 1. A television camera which continuously captures each partial image constituting a large screen image while panning and tilting, and a swing angle of the television camera and an angle of view of an image captured by the camera are respectively arbitrary. A camera data detection unit that detects each video field or video frame, and a plurality of imaging images that are captured every arbitrary video field or video frame based on the swing angle and the angle of view detected by the camera data detection unit. A large screen comprising at least an image conversion circuit for perspective-transforming images, and a large-screen memory for storing a plurality of images subjected to perspective transformation in the image conversion circuit at predetermined addresses determined by the perspective transformation. Image synthesis processing device. 2. The large-screen image synthesizing apparatus according to claim 1, wherein said large-screen memory is provided separately for a background and a foreground; A plurality of perspective-transformed images of the image and the foreground image are stored, and the stored images are read out from the respective large-screen memories by matching the addresses of both memories, and the background is stored in a range not exceeding the area where the foreground image data exists. A large-screen image synthesizing apparatus, wherein an image is configured to overwrite a foreground image on an image, and an image in which the foreground image is embedded in a background image is obtained from the apparatus. 3. The large-screen image synthesizing apparatus according to claim 2, wherein the reading of the image from each of the large-screen memories is performed after the background image is stored in all addresses of the large-screen memory for the background. A large-screen image synthesizing apparatus characterized by being configured to perform the processing.