JP3520318B2 - Video composition arithmetic processing apparatus, method and system thereof - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video synthesizing apparatus for synthesizing one image in real time from a plurality of received video data, and more particularly to a general-purpose circuit, a method and a system thereof.

[0002]

2. Description of the Related Art The conventional video synthesizing method is a "video signal system" in which a video signal is input at the same speed timing as the speed of displaying on the display, and a drawing speed of the data written in the image memory (frame memory) on the display. It can be classified into a “bitmap display method” which is read according to.

[0003]

[Problem to try solving to the invention] In the former case, it is necessary to equalize the synchronization and the drawing speed in the input image, the former has required labor. The latter way, the video data (synchronization, asynchronous both) of the plurality of video signal format or synthesized, in the case of the synthesis of video data in the bit map type of a video signal system, the images data digitized 1
The main method was to create a bitmap image data in an image memory and display it on the display.

[0004] The latter way from capture one screen in memory, for the synthesis of the screen, it takes time until the screen synthesis can be obtained. As long as the processing in the vertical retrace blanking time is not completed, it is not possible to reflect the update contents of the input image to the synthesis image. Even if the processing of the composite screen is completed within the vertical blanking time, it is delayed by at least one frame, and is 1 for the NTSC video signal.
There will be a time delay of / 60 seconds.

[0005] For example, in a video conference in which a conference is held based on images of a plurality of other parties, one's own image, and a composite image on a bulletin board, a time lag occurs in the composite screen of the TV screen images, especially when the other party's location is remote. There was a problem that the meeting did not proceed smoothly. Further, in order to reliably calculate the composite image for one frame within the vertical flyback time, a large computer that performs high-speed calculation processing is required.

[0006]

In order to solve the above problems, the present invention provides a plurality of writing line buffers, frame memories, reading line buffers corresponding to respective video signals to be video-synthesized, and for each pixel. A control memory for storing synthetic control instruction data for designating an arithmetic method,
The output side buffer, the arithmetic unit for independently performing the arithmetic operation of video composition for each pixel, and the controller write the video signal data into the frame memory for each horizontal scanning line,
Video compositing with versatility to obtain composited video signals in real time for any video signal by reading out the frame memory, performing pixel-by-pixel video compositing calculations, and reading the output side buffer at the drawing speed. An arithmetic processing device and a method thereof are provided.

[0007] Also, among the final synthesized multiple source video of the received signal assumed to produce an image, any source video signal, to the user's request by switching the source video signal to provide the user Provided is a video synthesizing calculation processing system for creating a corresponding synthetic video signal.

Further, an address processing unit is provided on the input side of the video composition arithmetic processing unit so that the position of the viewer is detected by a sensor, and the address unit of the video signal and the corresponding compositing command control are followed in accordance with the sensor output. A video composition arithmetic processing system that processes data to make the composite video realistic, and an address processing unit for binocular composite video and two sets of the video composition arithmetic processing device are provided to obtain a stereo composite video. An image synthesis arithmetic processing system is provided. The latter address processing unit can be omitted.

[0009] The present invention by the lever, the composite image instead of creating a single bitmap type of video data, by combining one row to be displayed at the moment to display only the (lines) on the fly Highly versatile video composition display
It is Ru can. The arithmetic unit of the video composition arithmetic processing unit is a standard process, and parallel processing can increase the speed and reduce the size. Further, with the system of the present invention, it is possible to obtain in real time a composite image and a stereo configuration image that follow the position of the viewer for any image signal.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an embodiment of a video synthesizing arithmetic processing unit of the present invention. In the figure, 1 represents the whole apparatus of the present invention, 1A, 1B and 1C are video signal input terminals, 2
A, 2B, 2C are write line buffers, 3A, 3
B and 3C represent frame memories and 4A, 4B and 4C represent read line buffers. Plural sets of these are provided corresponding to the video signals necessary for creating the composite video. In the example of FIG. 1, the video signals to be video-synthesized are A, B,
The case of three Cs is shown.

1K is an input terminal for combining control command data, 5
Is an arithmetic unit for performing a composite operation by a method of designating a plurality of video data by the composite control command data, 6 is a line buffer for the operation result, 7 is a line buffer for the composite video output, 8
Is a control memory, 9 is a control memory line buffer, 10
Represents a controller that generates write and read timing signals (C2 to C6) for each buffer and memory.

Write line buffers 2A, 2B, 2
C, the read line buffers 4A, 4B and 4C, the calculation result line buffer 6, the composite video output line buffer 7, and the control memory line buffer 9 are F respectively.
It has an IFO configuration.

Before describing the invention, an example system using the invention will be described. FIG. 2 schematically shows an example of a system which receives a plurality of video signals, synthesizes the video using the apparatus of the present invention, outputs the synthesized video signal in real time, and displays the synthesized video on the display device. It is a thing. Figure 2
Inside, 20A, 20B, 20C represents a camera, 21
A, 21B and 21C are analog / digital converters, 25
Is a digital / analog converter, 26 is a synchronizing signal adding circuit, 27 is a display device, and 27a is a display screen for displaying a composite image.

22 B and 22 C are key signal generators, 23
B 2 and 23 C are analog / digital converters, and 24 is a composite control command data generation device that outputs composite control command data K that specifies a calculation method for combining the images of the video signals A, B, and C. In addition, in this specification, the same reference numeral represents the same thing.

When the composite video signal is an analog signal, the analog / digital converters 21A, 21B shown in FIG.
The composite signal is converted into a digital signal by 21C. FIG. 3 is a diagram for explaining this operation. FIG. 3A shows an example of the configuration of the analog / digital converters 21A, 21B and 21C, and FIG. 3B shows an output signal input to the device of the present invention.

In FIG. 3A, 31 is a sync signal separation circuit for extracting the horizontal sync signal H and the vertical sync signal V, and 32 is the pixel data (D1 to D) at the time of sampling the analog signal.
8) is an analog-digital converter, 33 is a pulse signal generator that outputs a write signal W and a command / data switching signal CS, and 34 is a start point of drawing a video signal on one horizontal scanning line (hereinafter referred to as a line). An address generator for creating a video signal address part (XX, YY, LL) is shown.

The command / data D of the output signal is composed of a packet unit including a video signal address portion (XX, YY, LL) which is a command and video data (DATA). XX (8 bits) represents the pixel number sample number (pixel number) in which the video data appears, YY (8 bits) represents the line number, and LL (8 bits) represents the sample number (pixel number). The write signal W is the command XX, YY, L
L is a timing signal for reading or writing corresponding to sample data (pixel data, 8 bits) of video data (DATA). CS is a command / data switching signal indicating that the H level is in the command signal period.

In the case of the composite signal of FIG. 3B, the horizontal position of the drawing start is fixed at the left end (XX
= “0”), and since the number of pixels LL is also constant, the address generator 34 can be realized by a combination of a counter circuit that generates the line number YY and a memory circuit that holds the values of XX and LL.

Digital video data (DATA) of the present invention
The operation up to writing one line of the above into the frame memory 3 will be described with reference to FIGS. 4, 5A to 5C, and 6. FIG. 4 is a schematic diagram for explaining the operation of the write line buffer 2 and the frame memory 3, and is an example when the signals of FIG. 3 are taken in. 41 and 43 are memories, 4
Reference numerals 2 and 44 represent write address control means, and 45 represents read address control means.

The data (D1 to D8) written in the write line buffer 2 is the command / data switching signal C.
During the H level period of S, the command is written in the write address control means 42 in synchronization with the write signal W. During the L level period of the command / data switching signal CS, the pixel data (DATA) of the video signal is sequentially written in the memory 41 of the writing line buffer 2 in synchronization with the writing signal W.

"LL" stored in the counter in the write address control means 42 is subtracted every time one pixel data is written. When the end of writing in the write line buffer 2 is detected by the counter value “0”, all the line data are transferred to the frame memory 3 within a predetermined period corresponding to the blanking period that follows. At the time of the above-mentioned predetermined period, the write address control means 44, based on the address parts XX, YY, and LL of the transferred packet data, transfers the video data of LL pixels from the memory 41 of the write line buffer 2 to the frame memory 3. The line address #YY is transferred to the XXth and subsequent lines.

FIG. 5 shows that the composite video signal A is non-
6 is a time chart showing an interlace scanning operation, a buffer writing operation, and a frame writing operation when the composite video signal F is an interlaced scanning operation. In the drawing, (a) shows line Y of the input video signal A, line data A _Y of Y + 1,
It shows a state in which A _{Y + 1} is input. The line data A _Y written in the write line buffer 2A is written in the storage area of the line address #Y of the frame memory 3A at time t2 in the horizontal blanking period (see (b) and (c)). It should be noted that in FIG. 5, what is hatched indicates that the recorded video data is collectively transferred. In (c), the next video data is transferred and the recorded contents are updated (e), ( In f) and (g), the video data is stored and held for one line period.

FIG. 6 shows image data A, B, and
It is a figure which illustrates typically the recording update state of the video data in the frame memory in the state which C inputs asynchronously. The video signal A is a non-interlaced video signal, and B and C are interlaced scanning video signals. Incidentally, FIGS.
Line data with a dash (') added is t
It is indicated that the stored data is the video data of the previous frame at time (or elapsed time in FIG. 5). For example, A _Y
Indicates the latest data, and A _{Y + 2} ′ indicates the data of the subsequent frame.

The video data A, B, C are asynchronously and sequentially written in the writing line buffers 2A, 2B, 2C (FIG. 6 (a)). Each write line buffer 2A, 2
The line data of the video signals A, B and C written in B and 2C are stored in the frame memories 3A and 3B during the next horizontal blanking period.
The line data is written in the memory area of the 3C line address to update the line data. (See FIGS. 6 (b) and 6 (c))

Next, a method of synthesizing video based on the video data stored in the frame memories 3A, 3B and 3C will be described. The key signal generators 22B and 22C generate a key signal indicating whether the foreground image signal is used or the background image signal is used, and the analog / digital converters 23B and 23C select the foreground image signal by the key signal. The key control signal is digitized on a pixel-by-pixel basis so that the combination control data corresponding to is set to "1" and the combination control data corresponding to the other pixels is set to "0". Transfer the signal to the device 24.

The key signal can be created by a known key signal generator which performs chroma key signal processing, luminance key signal processing, Z key signal processing using distance information, and signal processing using the difference from the background image. However, in the example of FIG. 2, since the synthesis control command data is determined only by the key signals of the foreground video signals B and C, the key signal of the background video signal A is unnecessary and is omitted. (Details will be described later.)

The synthesizing control command data creating device 24 is composed of a computer, generates synthesizing control command data K from the key signal of the foreground video signal and the distance relation information, and synthesizes the image as a command for designating the calculation method of the pixels of the synthetic screen. Processor 1
The data is written in a predetermined area corresponding to the pixel of the composite screen of the control memory 8 therein. The synthesis control command data K is data having a function corresponding to a machine language of a computer when performing a video synthesis operation.

The control memory 8 is a frame memory 3A, 3
The memory area is determined corresponding to the pixel storage areas of B and 3C. The row and column addresses of the control memory 8 correspond to the line address of the video signal and the sample data number (pixel data number). That is, the combined control command data stored in the control memory is the frame memories 3A, 3B,
It is recorded and read in correspondence with 3C pixels.

By reading the composition control command data corresponding to the pixels of the frame memories 3A, 3B, 3C, the arithmetic processing procedure of the image composition is stored in the control memory 8. When the memory cell of the control memory 8 is 8 bits, the control memory 8 can store 256 kinds of synthetic control command data (calculation command) K in pixel units. The composite control command data creation device 24 can confirm and correct the composite control command data stored in the control memory 8 on the monitor screen.

Readout timing C of the controller 10
The line data of the same line address of the frame memories 3A, 3B, 3C are transferred from the respective frame memories to the reading line buffers 4A, 4B, 4C in synchronization with the timing 2 (see FIG. 5D) ( Figure 5,
See time t3).

The video data A of FIG. 5E represents the video data of each video signal as a representative. Frame memory 3A
Line address ..., # Y-2, #Y, # Y + 2,
Line data of _AY-2 , _AY , A
_{Y + 2} ′, A _{Y + 4} ′, ... Are read for each line data and sequentially written in the reading line buffer 4A in synchronization with the timing C2. Since the line data of the line Y + 4 has not been updated at this time, the line data A _{Y + 4} ′ of the previous scan is read. In the example of FIG. 6, at time t, the line data of the line address # 3 in the frame memories 3A, 3B, 3C are read in the line buffers 3A, 3B,
In the case of the timing of reading in 3C, the line data A3 ', B3, C3 are transferred to the reading buffers.

On the other hand, in the composite control command data K, one line of the row address corresponding to the read line address of each frame memory is read from the control memory 8 in response to the read signal C2, and the control memory line buffer is read. 9
Written in. Read line buffers 4A, 4
B, 4C, and the control memory line buffer 9 are respectively
Computer data cache memory, to have a function of the instruction cache memory.

Video data corresponding to the pixels of the frame memories 3A, 3B and 3C and synthesis control command data are read from the read line buffers 4A, 4B and 4C and the control memory line buffer 9 in synchronization with the timing pulse of the controller. It is then taken into the calculation unit 5. In the calculation unit 5, the composite video is calculated for each pixel by the calculation method specified by the combination control command data, and is sequentially transferred to the line buffer 6 for calculation results in synchronization with the timing of the controller 10 for each pixel.

In this way, when the composite video for one line is calculated in pixel units and written in the calculation result line buffer 6, the calculation result line buffers 7 to 1
The combined video data for the lines is transferred to the line buffer 7 for outputting the composite video signal. At the same speed as the drawing speed of the display device 27, the controller 10 outputs the composite video signal in pixel units from the line buffer 7 for outputting the composite video signal to the external D / A converter 25. The composite video signal is converted into an analog amount by an external D / A converter 25, and signals such as a horizontal sync signal and a vertical sync signal are added by a sync signal adding circuit 26 and input to the display device 27 as a display video signal.
It is drawn on the display screen 27a.

The calculation process may be completed between time t4 and time t5, and since this calculation can be independently processed for each pixel for one line, the calculation speed can be substantially increased by using a plurality of calculation devices. Is easily possible.

FIG. 7 shows an embodiment in which pipeline parallel arithmetic processing is performed in three arithmetic units in order to speed up the arithmetic processing in pixel units. Reference numerals 51 and 55 are registers for storing data for one pixel, and 52 and 54 are data memories for storing data for one pixel. Reference numeral 56 denotes a control memory, which is a register for storing pixel-by-pixel synthesis control command data K;
Represents a memory for storing the combined control command data K.

The register 51 converts serial 3-pixel data in parallel and inputs it to each operation unit 53, and the register 55 serially converts the operation result parallel pixel data in the input order and outputs it to the operation result line buffer 6. To do. In this example, three times faster processing can be performed. If n arithmetic units are used, n times faster arithmetic processing can be performed.

In the example system using the present invention of FIG.
The input video signal is interlaced (video signal A), non-interlaced scanning signals (video signals B and C), and the composite video output signal is non-interlaced scanning signal (FIGS. 5 and 6). These are the write address control means 44 and the read address control means 45 of the frame memory 43.
Address to the frame memory 43 by the address control of
Writing, it is clear that can be dealt with <br/> Rukoto read address from the frame memory 43.

That is, the value of the packet for each line is
For interlaced scan signals, [1, 1, ...
[1,3, ...], [1,5, ...]
Frame memory line addresses # 1, # 3, # 5, ...
.. for non-interlaced scan signals [1, 1, ...
3, ...,], so that they are written in the storage areas of the line addresses # 1, # 2, # 3 ,. If the composite video output signal is a non-interlaced scanning signal, the read address control means 4
5, the frame memory line addresses # 1 and #
The line data stored in Nos. 2, # 3, ... May be read.

In FIG. 2, the input video signal is described as an analog composite video signal by a camera shooting signal. However, even if it is a TV shooting signal, a satellite broadcast video signal, or a digital composite video signal, a computer graphic The image synthesis of the present invention is possible even with (CG) signal input. These signals can be dealt with by adjusting the packet data creation in the analog / digital converters 21A, 21B and 21C.

FIG. 8 shows an example of creating a composite image according to the present invention. In this example, a background image is given to the video signal A, a car image is given to the video signal B, and a person's image is given to the video signal C, and the synthetic control command data of the synthetic video is the video signal A,
Command data KA, KB, which selects only B and C, respectively
This is an example in the case of KC. In this example, the video signals A and B are video signals having the same number of pixels as the composite video signal F, and the video signal C is a video signal having a smaller number of pixels. The video signal C is written in a predetermined storage area of the frame buffer corresponding to the pixel position of the composite screen. Since the portion not written (black part of FIG video signal C) may not be ultimately used, it may be any value.

The synthesizing control command data creating device 24, based on the synthesizing control data As, Bs, Cs indicating which of the images of these input video signals A, B, C is to be used, the video signals A, B. , C which selects only C
A, KB, and KC are created and written in a predetermined area corresponding to the pixel of the composite screen in the control memory 8.

For example, an example of each video signal A, B, C in FIG. 2 will be described. In this example, as a reference for selecting each video signal on the composite image, a video signal having a standard distance close to the viewer is prioritized, that is, is selected in reverse order of the standard distance. The standard distance on the composite image is as follows: background image signal A> foreground image signal B
> Because it is foreground video signal C, the instruction K for composite screen pixels of the synthesis control instruction data K (x, y) (x , y) is combined control data B s, the following values for Cs (instruction code,
8 bits) are allocated. Command KC to select video signal C when Cs (x, y) = "1", select video signal B when Cs (x, y) = "0" and Bs (x, y) = "1" Command KB, command KA for selecting video signal A when Cs (x, y) = "0" and Bs (x, y) = "0". At this time, selection of video signal A is unnecessary, so video signal A
The key signal of can be omitted.

The composite control command data KA, KB, KC can be displayed on the monitor screen of the composite control command data creating device 24 in association with the video signal as shown in FIG. At the point of background video signal A, the specified 1 of the car of video signal B
A composite image F in which the table and one designated person among the persons of the image signal C are present is obtained and drawn on the display device 27.

The plurality of pieces of image data are not lost at the time of producing the composite video. For example, an image of a part hidden behind an object that is not used in the composite image is also stored in the frame memory. By sending and receiving image information by utilizing this characteristic positively, it is possible to significantly reduce the amount of information required when transmitting. For example, if there is a car in a landscape and there is an image of a person moving in front of it, this is transmitted as an independent image as the background of the image, the image of the car, and the image of the person, and the receiving side By using the device of the present invention, it is possible to reproduce them by combining them.

A fast-moving vehicle can be sent at 60 frames per second and a background at 5 frames per second. According to the display system of the image synthesis screen of the present invention, it is not necessary to retransmit the background screen which is once behind the automobile due to the movement of the automobile when the background screen is re-displayed when the automobile passes. This is because the background image that has not been displayed in the composite video is not lost, but is not displayed in the composite and remains in the image memory (frame memory).

Similarly, in the present invention, composite video data can be created and displayed using a plurality of video data at that moment regardless of the respective image memories that are rewritten every moment. For example, it is possible to display a composite image of the above-mentioned background of 5 frames per second, a person 's 15 frames per second, and a car 's 60 frames per second at a readout speed corresponding to the display performance, for example, 70 frames per second. it can.

FIG. 9 shows images of these plural materials,
Instead of processing those video signals and creating a composite video before broadcasting, create the video signal of the material and the composite control command data or the composite control data, broadcast these data, and the receiving side data based on using an image synthesis processing apparatus 1 of the present invention to obtain a combined image, luciferyl stem example be output in real time at a drawing speed of the receiver-side display device is shown schematically. The system of FIG. 9 (a), transmits source video signal created by the broadcast station, the synthesis control instruction data,
9 is an example of a system for creating a composite video on the receiving side.
System (b) is source video signal produced in the broadcasting station side, it sends a combined control data, received in the receiving side
Synthetic control data to generate a composite control command data on the basis of an example system that creates a combined image.

In the figure, 60 is a synthesis control command data creating device, 61 is a transmission packet creating unit, 62 is a sending device, and 63.
Represents a receiving device. Reference numeral 60S represents a synthesis control data creation device, and 60R represents a synthesis control command data creation device. In the examples of FIGS. 9A and 9B, the key signal generator and the device corresponding to the analog / digital converter are incorporated in the synthesis control command data creating device 60 and the total control data creating device 60S. ing.

In the example of FIG. 9A, the material video signals A, B and C are video signals of the background, car and person photographed by the cameras 20A, 20B and 20C, respectively. The synthesizing control command data creating device 60 creates synthesizing control command data K for each pixel based on each imaged image screen. The video signals A, B, C and the composite control command data K are subjected to optimum data compression and processing, and then the transmission packet creation unit 61.
Is formed into a packet signal and transmitted.

The synthesis control data creating device 60 of FIG. 9B.
S is a device that creates combined control data from the video signals A, B, and C by various key processes, for example. 2 and 9
Strictly speaking, the key signal does not directly correspond to the composition control data as it is, and the standard arrangement position (viewer corrects the position on the composite image from the distance relationship information (LA, LB, LC or LA: LB: LC)). The position to see without) has been decided. This is because the screen position (x, y) is set to the standard position so that the image is binarized and the composition intended by the cameraman (photographer) is obtained.

Device 6 for synthesizing distance relation information
Examples of a method of inputting into 0S include a method in which a cameraman (photographer) directly inputs the value, and a method in which a distance measuring device is installed for each camera and the measurement result is automatically input. For example, the cameraman can set the image signals A, B, so that LA = 10 km from the observer (viewer) to the mountain, LB = 30 m to the car, and LC = 10 m to the person.
Set the standard position of C.

In FIG. 9B, the combined control command data creating device 60R receives the combined control commands As to Cs and LA to L.
It is a device that creates the composition control command data K based on C and transmits the composition control command data K to the control memory of the video composition control device 1. When the receiving side corrects the display position of the video signal of the composite screen, the distance relation information is transmitted from the transmitting side together with the composite control data.

In the example of FIG. 9, for example, a high-precision image can be transmitted as a substitute for a background signal having little movement, and a person and a car can take advantage of the feature that the image range is narrow in spite of the movement, so that only a small area can be transmitted. You can send the image. For example, this is possible if all the images are transmitted in the data format of one line, one packet (XX, YY, LL, pixel data).

FIG. 10 shows a time chart example of the transmission packet data (A, B, C, K) and the video synthesis calculation result (F) in the system of FIG. 9A, and a one-frame video screen corresponding to the time chart. Example (A *, B *, C *), Example of composition control command data (K *), Computation video composition screen (F *)
It is a diagram schematically showing an example. In this example, each transmission packet data is a frame image unit, and the subscript represents the transmission order. Combined control command data K1 *, K2
Symbols * and K3 * display the calculation instruction data KA, KB and KC for each pixel stored in the control memory in correspondence with frame images. The transmission packet is transmitted asynchronously according to the transmission information amount of the video data. The black portions of the video signals B and C represent pixels that are not transmitted.

In FIG. 11A, for example, for a material video group that is supposed to produce a final composite video from three material videos, the user's video is used instead of the second material video. it allows shows an image synthesizing processing system example behalf of the user to obtain the movies image that plays <br/> specific actor schematically.

In FIG. 11A, reference numerals 65 and 66 represent switching means for switching the video signal and the composition control data in accordance with the user's material video signal switching setting (B to D in the figure). Reference numeral 20D represents a camera by which the user shoots the material of the composite video. Reference numeral 67 denotes a plurality of material video signals (A, B, C) and composite control data (A
s, Bs, Cs), and 68 is a recording / reproducing device for reproducing when used as a material of a composite image.
Reference numeral 69 denotes a material display position detecting circuit, and 69 denotes a material display position detecting circuit for switching between the output of No. 3 and the output of the recording / reproducing device 67. Also, source video signal and the control data is more received by the receiving device 63 via the switching circuit 68, it may be recorded in the recording and reproducing apparatus 67. Reference numerals 21D and 23D denote analog / digital conversion units, and 22D denotes a key signal generator, which perform the same operations as those in FIG.

The standard position data of the material video signal D is created by the analog / digital converters 21D and 23D.
Assuming that the material video signal D is a substitute video signal of the actor video of the material video signal B, if the actor moves around, it is necessary to follow the movement of the acting actor. The address parts of the output signals D and Ds of the analog / digital converters 21D and 23D are adjusted and output so that the material video signal B and the material video signal D are at the same position.

The material display position detection circuit 69 is specially provided for accurately performing this adjustment. The material display position detection circuit 69 takes in the composition control data Bs or the material video signal B, calculates the position on the composition screen of the actor, and outputs it to the analog / digital converters 21D and 23D.
Output to. Analog / digital converters 21D and 23D
Is the output signal D. D
s are output to the video composition arithmetic processing unit 1 and the composition control command creation unit 60R, respectively.

FIG. 11B exemplifies material video data and synthesis control command data of the video synthesis arithmetic processing system. In the figure, A, B, and C are the material video data, and the black portions are the pixels that are not broadcast in the material video data.
s and Cs are synthesis control data indicating the effective range of each material video, which are represented by screen displays. Similarly,
D represents video data obtained by photographing the user, and Ds represents synthesis control data indicating an effective range as a material. F is a composite video data which is a composite image of the received material video data A, B and C, and FD is a composite video data which is a composite of the received material video data A and C and the material video data D taken by the user. It is a representation.

The video signal photographed by the user with the camera 20D and the key signal obtained by the key signal generator 22D are displayed at the display position of the material image B detected by the material display position detection circuit 69. The image data, D, and the combined control data Ds are processed.

When a user creates a composite video by using the video taken by the user, the switching means 65, 66 switches B of the material video signals A, B, C to the material video signal D, and the composite control command data. The combination control data Bs of the combination control data As, Bs, Cs of the material video signal input to the creating device 60R is switched to the combination control data Ds of the material video signal. The synthesizing control command data creating device 60R creates the synthesizing control command data K from the synthesizing control data As, Bs, Ds, and the video synthesizing processing device 1 uses the material video signals A, B, D and the synthesizing control command K data for the user. The composite video FD corresponding to the request is obtained. The data of the recording / reproducing device 67 can be used by the switching circuit 68.

For example, assuming that the first, second, and third material video signals A, B, and C are videos of the background, actors, and tools, respectively, the second material is switched by switching the switching means 65 and 66. User's video signal D instead of video signal B
Use to get a composite video of the user playing instead of the corresponding actor.

In FIG. 12, the same data as in FIG. 9B is received, and on the receiving side, the composite video screen is moved in real time by following the moving position of the viewer to give the composite video screen a stereoscopic effect. 2 schematically shows an example of a video composition processing system using the device of the present invention in the above device. The functional system of FIG. 12 is connected at the receiving side portion corresponding to the * mark in FIG. 9 (b).
Reference numeral 70 denotes a video signal address section (XX, YY, LL) at the head of the received image signal packet and an address processing section for processing the corresponding synthesis control command data at the same time, 71 is a display device, 72 is a display device 71 and a viewer. A position sensor 73 for detecting the positional relationship with the is a feedback signal for feeding back the value to the address processing unit 70.

The positional relationship between the display device 71 and the viewer is detected by the position sensor 72, and the value is fed back to the address processing unit 70. Based on the distance data extracted from the received packet data and the relative positional relationship, the address processing unit 70 individually increases or decreases the processing amount of XX, YY, LL of each video signal address unit. The processed command of the video signal address portion and the video data are input to the device 1 of the present invention to draw a composite video on the display device 71.

In the address processing unit 70, the XX of each video signal address unit (XX, YY, LL) is calculated based on the positional relationship of the depths of the video signals photographed by the cameras 20A, 20B and 20C on the composite video screen. The increase / decrease value is calculated based on the feedback signal 73, and as a result, the portion protruding from the combined screen is deleted and the pixel number LL is adjusted. At the same time, the layout of the control command data for selecting each video signal of the composite control command data created by the composite control command data creating device 60R is also corrected based on the increase / decrease value of XX calculated from the viewing position.

Referring to FIG. 13, video signals A, B, C and synthesis control data A by address processing unit 70.
The address increase / decrease control of s, Bs, and Cs will be briefly described. (A) shows an example of position variation calculation when the material image A is immovable and (b) is when the material image C is immovable. Δx represents the position variation amount from the position P1 to the position P2 by the viewer. This position variation amount Δx is measured by the position sensor 72, proportionally distributed by the distance relation information LA, LB, LC, and the position variation amount ΔxA, Δ of the image data A, B, C.
xB and ΔxC are calculated. Video signal by the number of pixels corresponding to the position variation A, B, C in the address XX, synthesis control data As, Bs, is Gana address increment adjustment Cs,
From the address processing unit 70, video signals A ′, B ′,
C ', the combined control data As', Bs ', Cs' are output.

With this, the composite video according to the movement of the viewer's viewpoint can be displayed in real time, and the viewer can appreciate the stereoscopic video. On the right side of FIG. 12, the image at the viewing position P 1 and the image at the viewing position P 2 are schematically illustrated. Even at the same moment, the viewing position P 2
, The person on the screen is on the right side of the mountain in the background of the screen, but at the viewing position P 1, the mountain and the person overlap each other and are displayed.

FIG. 14 is connected at the receiving side portion corresponding to the mark * in FIG. 9B, receives a plurality of video signals, and the receiving side uses two video synthesis arithmetic processing devices of the present invention, 1 schematically shows a stereo image synthesizing calculation processing system which processes an address portion of a video signal and outputs two synthetic images having binocular parallax. In FIG. 14A, 1R and 1L are image combining arithmetic processing units for the right and left eyes, 70R and 70L are address processing units for the right and left eyes, 74R and 74L are address processing units 70R, The adjustment input for increasing / decreasing the 70L video signal address portion (XX, YY, LL) is shown.

The address processing units 70R and 70L receive the packets so that the combined image output signals FR and FL of the image combining arithmetic processing units for the right and left eyes have a sufficient amount of perspective due to the binocular parallax. Based on the distance data extracted from the data, the adjustment inputs 74R and 74L are used to adjust each address portion. Composite video signal FR, R
L is supplied to a stereo display device (not shown). With the stereo display device on the receiving side, a stereo image with a stereoscopic effect obtained by using binocular parallax,
The viewer can see in real time.

The video composition calculation system of FIG.
This is a system in which address processing of one of the received video data in the video data of both eyes in FIG. 14A is omitted.
In FIG. 14B, the left-eye video data is not subjected to address processing, and the address processing unit 70 adjusts and inputs the address portion by the binocular parallax amount of the right-eye and left-eye composite video signals.
Adjust to increase or decrease by.

[0072]

According to the video synthesizing calculation process of the present invention, a frame memory is prepared for each video signal to be video-synthesized, one line of video data is written in the frame memory by the writing line buffer, and one line is written from the frame memory. Since the data is read and the arithmetic processing of the image combination is performed, there is less time delay in creating a combined image as compared with the conventional case where images are combined one frame later, and an image combination screen can be obtained in real time. By providing a write line buffer and a read line buffer, access competition of the frame memory can be prevented. Furthermore, the processing speed is increased by incorporating the synthesis control command data of the video signal corresponding to the machine language of the computer into the calculation unit for each pixel.

In the present invention, it is said that instead of creating one bitmap type video data of a composite image, only one line (line) displayed at the moment of display is composited and displayed on the spot. , Highly versatile video composition display
It is Ru can. The combining operation unit is a standard process, and parallel processing can increase the speed, and it can be created with an LSI chip and can be downsized.

In the apparatus of the present invention, all of these processes proceed in parallel at the same time, so that parallel processing can be performed, and high speed is not required for individual processing in the apparatus, and it takes time until a composite image is obtained. The effect that it can be shortened. Moreover,
Video input, video output, and control input are asynchronous. Furthermore, there is also an effect that it can operate even if the drawing speed (the number of screen updates per second) is different.

Multiple video inputs, access to control memory,
The video output may be asynchronous, and it is not necessary for the respective video update cycles to be the same. Furthermore, a video input with an indefinite update interval or a video signal of only a part of the screen may be used. Thus, each video input signal can be given independently of the other video input signals, and the complexity of the entire system using this circuit can be reduced.

In the video composition processing system of the present invention, a composite image requested by the user can be obtained by using the material video signal created by the user. Especially not only for practicing or watching a specific cast in the field of drama,
By acting as a substitute, a new way of enjoying the theater is realized.

Further, in the video synthesizing processing system in which the address portion of the received video signal is increased / decreased on the input side of the video synthesizing arithmetic processing apparatus of the present invention, the synthetic image which follows the movement of the viewer and the address processing on the input side are performed. By addressing at least one of the two composite video signals, a stereo composite image can be obtained in real time for any video data.

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example system that uses the present invention.

FIG. 3 is a diagram showing input data of the present invention.

FIG. 4 is a diagram showing an outline of an input side line buffer and a frame memory of the present invention.

FIG. 5 is a diagram showing a time chart of video data.

FIG. 6 is a diagram for explaining the input and recording update timing of each video data.

FIG. 7 is a diagram illustrating parallel pipeline arithmetic processing.

FIG. 8 is a diagram illustrating an example of video composition according to the present invention.

FIG. 9 is a diagram illustrating an example of a video composition calculation processing system when receiving data used in the present invention to create a composite video.

FIG. 10 is a diagram illustrating the data of FIG.

FIG. 11 is a diagram illustrating an example of a video composition calculation processing system that creates a composite video using a user's video that is another invention.

FIG. 12 is a diagram illustrating an example of a video composition arithmetic processing system that is another invention and creates a composite video by following the position of a viewer.

FIG. 13 is a diagram for explaining address increase / decrease control in address processing.

FIG. 14 is a diagram illustrating an example of a video composition arithmetic processing system that creates a stereo composite image that is another invention.

[Explanation of symbols]

1 Video synthesis processor 2,2A to 2C write line buffer 3,3A-3C frame memory 4,4A-4C line buffer for reading 5 computing section 6 Line buffer for calculation results 7 Line buffer for composite video output 8 control memory 9 Line buffer for control memory 10 controller 24,60 Synthesis control command data creation device 60S Synthesis control data creation device 60R synthesis control command data creation device Image capture device for capturing 20D users 21D, 23D analog / digital converter 22D key signal generator 63 Receiver 65, 66 switching means 67 Recording / reproducing apparatus 68 Switching circuit 70, 70R, 70L Address processing unit 72 Position sensor 74, 74R, 74L adjustment input

Claims (9)

(57) [Claims] 1. A plurality of first line buffers provided corresponding to a plurality of video signals input asynchronously, for reading video data of one horizontal scanning line of the video signals, and a first line buffer. A plurality of frame memories provided corresponding to the first line buffer to which the video data of one horizontal scanning line is transferred when reading of data for one horizontal scanning line is completed, and composition for designating an operation method for each pixel A control memory that stores control command data, a corresponding pixel of each frame memory, a calculation unit that fetches the composite control command data corresponding to the corresponding pixel, and performs a video composition operation, and composite video data for one horizontal scanning line. Minutes of the output line buffer that stores the minutes and outputs the composite video data at the drawing speed, the reading of the frame memory and the control memory, the operation unit, and the output line buffer. Video synthesis processing apparatus characterized by comprising a controller to take. 2. The video composition arithmetic processing apparatus according to claim 1, wherein arithmetic processing for each pixel is performed by pipeline parallel processing. 3. One horizontal scanning line data of a plurality of video data received asynchronously is read into each of the first line buffers, and immediately after the reading is completed, the corresponding one of the frame memories is read into the first line buffer. The data is read, the corresponding pixel data from each frame memory and control memory, and the data for one horizontal scanning line of the synthesis control command data is read into the respective second line buffers, and the image synthesis operation is performed for each pixel. A video composition arithmetic processing method characterized in that one horizontal scanning line segment of the composite video data is transferred to a third line buffer and is output at a drawing speed of a display device. 4. A video composition arithmetic processing device according to claim 1, a photographing device for photographing each material image constituting a screen of a background, a tool, and an actor, and a final processing using each material image photographed by the photographing device. A composite image processing unit for producing a composite processing instruction data for creating a composite processing instruction data for producing a composite processing instruction data, and a video signal of each material image and a composition control instruction data creating apparatus. A video composition arithmetic processing system, wherein the output composition control instruction data is input to the video composition arithmetic processing device. 5. A video composition arithmetic processing device according to claim 1, a photography device for obtaining a photographed image of a user, and a material image obtained by photographing each material constituting a screen of a background, a tool, and an actor is recorded and reproduced. Recording / reproducing device, and a composite control command data creating device for creating, as composite control command data, a calculation processing procedure for creating a final composite video using the material video obtained from the recording / playback device and the user's captured video And a video signal of each material video and the composite control command data output from the composite control command data creation device are input to the video composite operational processing device. Video composition processing system. 6. A video composition arithmetic processing device according to claim 1, an imaging device for obtaining a video image of a user, and a composite control of whether or not each pixel of the material video of the video image of the user is validated. A circuit device that creates data, and a broadcast of composition control data that represents whether or not the video signals of the material videos that have captured each material that constitutes the screen of the background, tool, and actor, and whether or not each pixel of these material videos is valid Combining control that creates a processing procedure for creating a final composite video using the receiving device that receives the image, the material video obtained from the receiving device, the captured video of the user, and the composite control data as composite control command data. And a video signal of each material video and the composite control command data output from the composite control command data creating device are input to the video composite arithmetic processing unit. Video compositing processing system. 7. A video synthesizing processor according to claim 1, and whether the video signals of a plurality of input material videos and respective pixels of the video signals are provided on the input side of the video synthesizing processor. The address processing unit that processes the address part of the composite control data that indicates whether or not it is input, and the calculation processing procedure for creating the final composite video by inputting the composite control data whose address part has been processed as composite control command data. It is equipped with a composite control command data creation device to create, a position sensor that detects the position of the viewer, and a feedback signal line that feeds back the detection signal of the position sensor to the address processing unit, and composite control with the video signal of each material video A video compositing arithmetic processing system, characterized in that the compositing control command data output from the command data creating device is input to the video compositing arithmetic processing device. . 8. A video composition arithmetic processing device according to claim 1, which inputs the same plurality of video signals, respectively, and a plurality of input devices provided on the input sides of the two video composition arithmetic processing devices. Input processing processing section for processing the video signal of the material video and the address section of the composite control data, and the processing procedure for inputting the composite control data with the processed address section to produce the final composite video. was and a two synthetic control command data generating apparatus to produce a synthesis control instruction data, so that the difference in the address portion is the two address processing section two synthetic image becomes binocular disparity of the viewer , And inputs the video signal of each material video and the composite control command data output from the composite control command data creating device to the video composite arithmetic processing device. Processing system. 9. Two video composition arithmetic processing devices according to claim 1, each of which inputs the same plurality of video signals, and a plurality of input material images provided on the input side of one video composition arithmetic processing device. One address processing unit that processes the address part of the video signal and the composite control data and the composite control data with the processed address part are input, and the composite processing procedure for creating the final composite video is controlled. Input the synthetic control command data creation device to generate as command data and the synthetic control data whose address part is not processed
The calculation processing procedure to create the final composite video.
Combined control command data created as combined control command data
; And a creation device, the difference in the address portion is set so that two synthetic images is binocular disparity of the viewer, the output synthesis control image signals of the respective source video from the synthesis control instruction data generating device A video composition arithmetic processing system, characterized in that command data is input to the video composition arithmetic processing device.