JP2006222942A - Image synthesizer, and method and program for synthesizing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a plurality of synthesized images by synthesizing a plurality of input images respectively with different patterns while preventing increase of the circuit scale to the highest possible extent. <P>SOLUTION: Images are synthesized to scale each of I pieces of input images with a plurality of scaling factors to generate a plurality of scaled input images for each of the input images, to store the plurality of the scaled input images generated for each of the input images into respective prescribed areas in a image storing means, to read out the input images from the image storing means respectively based on J pieces of image arrangement information, and to generate synthesized images from the read out input images based on respective image arrangement information. The image arrangement information determines the arrangement of each of the input images on the respective synthesized images. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video synthesizing apparatus, a video synthesizing method, and a video synthesizing program for inputting a plurality of video signals and outputting a plurality of video signals obtained by synthesizing them with different patterns.

  In a video conference, in the case of a so-called multipoint video conference in which a plurality of terminals communicate with each other by means of microphone sound, camera video, etc., simply connecting each other's terminals with a full mesh places a burden on the communication and terminal processing. large. Therefore, a method of providing an MCU (Multipoint Conference Unit) is usually used. The MCU is a kind of server and has a function of connecting to each terminal, receiving audio and video from each terminal, synthesizing them, and transmitting the synthesized one to each terminal. With this MCU, each terminal communicates only with the MCU, and it is possible to obtain audio and video from all participants simply by receiving the synthesized voice and synthesized video from the MCU, in terms of communication and terminal processing. Efficiency is good. As described above, the MCU plays an important role in the multipoint video conference, and the video composition technology is used here.

Another application of video composition technology is a screen splitting unit for surveillance cameras. Since a plurality of surveillance cameras are usually installed in a building or the like, if they are observed and recorded on different monitors, the equipment becomes large and inconvenient. Therefore, a method is usually used that uses a screen division unit to combine multiple camera images into a single video signal that can be confirmed on a single monitor or recorded on a single recorder. .
Japanese Patent Laid-Open No. 11-88854 JP-A-5-103324

  However, most of the conventional video composition techniques output a single composite video, and there have been few proposals for outputting a plurality of composite videos.

  For example, in the case of a video conference, a video conference can also be realized in a form in which all the participants see one type of composite video as in the conventional case. However, in pursuit of higher convenience, there is a demand for zooming and displaying a specific video among a plurality of synthesized videos. Japanese Patent Application Laid-Open No. 11-88854 shows one solution. However, with the technique disclosed in Japanese Patent Laid-Open No. 11-88854, when a certain image is zoomed, it is not possible to view images other than the zoomed image. Change the way of compositing, for example, embed another video in a zoomed video, or zoom the video to some extent instead of zooming the video to the full screen, and other participants around it. If it is possible to present a composite image such as a small display, it is easier to use. In this case, it is considered that each participant wants to zoom in on different videos, so it is required to synthesize videos with different patterns for each terminal and output a plurality of different synthesized videos. The

  The form of outputting a plurality of synthesized videos is only described in Japanese Patent Laid-Open No. 5-103324 as far as we know. In Japanese Patent Application Laid-Open No. 5-103324, an input video stored in a video memory is reduced by a reduction circuit and synthesized by a synthesis circuit and output. However, there is a problem that the circuit becomes complicated in consideration of actually mounting this form. Specifically, the following problems occur.

  First, a synthesis circuit that outputs a synthesized video signal usually has to output a signal at a timing according to the standard of the output signal. Therefore, in order to do so, an input signal must be input to itself at a timing where it can be taken into consideration (within an allowable timing if a buffer is provided in between). On the other hand, a reduction circuit that generates an input signal to the synthesis circuit must read data from the video memory at the same timing in consideration of its own processing delay in order for the synthesis circuit to output a signal at the above timing. I must. In this way, the output timing of the reduction circuit is determined in consideration of the processing delay of the synthesis circuit in accordance with the output timing of the signal from the final synthesis circuit, and further from the video memory in consideration of the processing delay of the reduction circuit. You must decide when to read the data.

  However, in the above configuration, when the synthesis circuit can dynamically change the synthesis pattern and the reduction circuit can dynamically change the reduction rate, the delay timing also dynamically changes. As a result, processing for dealing with delays is not simple, leading to circuit complexity and circuit scale increase.

  The present invention provides a video synthesizing apparatus, a video synthesizing method, and a video synthesizing program capable of generating a plurality of synthesized videos by synthesizing a plurality of input videos with different patterns while preventing an increase in circuit scale as much as possible.

  An image composition apparatus according to an aspect of the present invention is an I image scaling unit to which an input image is input, and each of the input images input to the image synthesizing unit is scaled by a plurality of magnifications. I video scaling means for generating a scaled input video, and receiving the write request for the scaled input video, temporarily storing the scaled input video in the area specified by the write request. Each of the plurality of scaled input videos generated by the video scaling means corresponding to the video scaling means and each of the plurality of scaled input videos generated by the video scaling means. Requesting the video storage means to request reading of the input video based on each of the plurality of write request means and the J pieces of video arrangement information. Each of the video arrangement information is each input on the composite video. Projection Read request means for determining the arrangement of the image, read control means for receiving the read request and reading the input video from the video storage means, each corresponding to the video arrangement information and read by the read control means. J composite video generation means for generating a composite video from the input video.

  According to another aspect of the present invention, there is provided a video composition method, wherein each of the I input videos is scaled by a plurality of magnifications to generate a plurality of scaled input videos for each of the input videos. Writing each of the plurality of scaled input videos generated in a predetermined area in the video storage means, reading the input video from the video storage means based on each of the J video arrangement information, The video layout information defines the layout of each input video on the composite video, and the composite video is generated from the input video read for each video layout information.

  The video composition program as one aspect of the present invention includes a step of scaling each of the I input videos by a plurality of magnifications to generate a plurality of scaled input videos for each input video, and the input Writing each of the plurality of scaled input videos generated for each video into a predetermined area in the video storage unit, and reading out the input video from the video storage unit based on each of the J video arrangement information The video layout information defines a layout of each input video on the composite video, and a step of generating a composite video from the input video read for each video layout information. Let the computer run.

  According to the present invention, it is possible to generate a plurality of synthesized videos by synthesizing a plurality of input videos with different patterns while preventing an increase in circuit scale as much as possible.

(First embodiment)
FIG. 1 is a block diagram schematically showing a configuration of a video composition device according to the first embodiment of the present invention. FIG. 2 is a block diagram schematically showing the configuration of a video conference system to which the video composition apparatus of FIG. 1 is applied.

  In FIG. 2, terminals 12 (1) to 12 (4) are connected to the video composition device 11 via a network (not shown). The terminals 12 (1) to 12 (4) generate video signals (input video) 13 (1) to 13 (4) and transmit them to the video synthesizer 11. Further, the terminals 12 (1) to 12 (4) transmit the video arrangement information (layout information) 14 (1) to 14 (4) used for video synthesis in the video synthesis device 11 to the video synthesis device 11. . Each time the layout is changed, the terminals 12 (1) to 12 (4) newly generate video arrangement information and transmit it to the video composition device 11. The video synthesizing device 11 synthesizes the video signals 13 (1) to 13 (4) based on the video arrangement information 14 (1) to 14 (4) and synthesizes them for the terminals 12 (1) to 12 (4). The video is generated, and the composite video signals (output video) 15 (1) to 15 (4) are transmitted to the terminals 12 (1) to 12 (4). In other words, the video synthesizing apparatus 11 receives four video signals, and generates and outputs four synthesized video signals obtained by synthesizing these video signals. For example, the video signal is transmitted as a digital signal of ITU-R (International Telecommunication Union-Radiocommunication) BT.656 specification, and the video signal includes pixel value information. In order to synchronize the video frame, a synchronization code is inserted into the video signal at a predetermined timing. The synchronization code is a set of FF, 00, 00, and XX, and XX contains a code indicating a position on the frame.

  Note that when transmitting a video signal, it may actually be compressed and transmitted using an MPEG decoder / encoder (not shown). In other words, the video signal is first encoded into MPEG streaming on the device that transmits the video, and the video signal is restored by decoding on the device that has received the MPEG streaming. In this case, according to the above, it is assumed that the input video signal from the terminal decoded by the video synthesizer and the output video signal to the terminal to be encoded are BT.656 specifications, for example.

  In FIG. 1, the video composition device 11 includes an analysis unit 21 (1) to 21 (4), a variable reduction unit 22 (1-1) to 22 (1-4), and 22 (2-1) to 22 (2- 4), 22 (3-1) to 22 (3-4), 22 (4-1) to 22 (4-4), RAM control unit 23, RAM 24, request units 25 (1) to 25 (4), Generation units 26 (1) to 26 (4) and a video arrangement information management unit 27 are provided. The variable reduction unit includes write request means. The video arrangement information management unit 27 includes a magnification designation unit.

  The analysis units 21 (1) to 21 (4) analyze the synchronization codes of the input video signals 13 (1) to 13 (4) and obtain the coordinates of the current pixel data on the input video.

  The video layout information management unit 27 manages the layout of the composite video for the terminals 12 (1) to 12 (4) based on the video layout information 14 (1) to 14 (4). The video arrangement information management unit 27 calculates the magnification (reduction rate in this example) of each input video 13 (1) to 13 (4) based on the video arrangement information 14 (1) to 14 (4) and calculates The reduction rate is changed to the variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 ( 4-1) to 22 (4-4). When the video layout information is updated, the reduction rate of each input video 13 (1) to 13 (4) is calculated based on the updated video layout information and notified to the variable reduction unit.

  Variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 (4-1) ) To 22 (4-4) are configured such that the input video can be reduced at a plurality of reduction ratios. Variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 (4-1) ) To 22 (4-4) reduce the input videos 13 (1) to 13 (4) based on the reduction rate specified by the video arrangement information management unit 27.

  For example, the variable reduction units 22 (1-1), 22 (1-2), 22 (1-3), and 22 (1-4) output video 15 according to the reduction rate designated by the video arrangement information management unit 27. (1) -15 (4) The input video 13 (1) arranged on the screen is reduced. In addition, the variable reduction units 22 (1-1), 22 (2-1), 22 (3-1), and 22 (4-1) output video 15 according to the reduction rate specified by the video arrangement information management unit 27. (1) The input images 13 (1) to 13 (4) arranged on the top are reduced.

  Variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 (4-1) ) To 22 (4-4) respectively reduce the input images 13 (1) to 13 (4) based on the coordinate information analyzed by the analysis units 21 (1) to 21 (4) in the previous stage and The RAM control unit 23 is requested to write the reduced video (pixel value information) to the area (frame buffer for each variable reduction unit). The variable reduction unit corresponds to, for example, a video scaling unit.

  The RAM control unit 23 controls writing to the RAM 24 that stores the video signal and reading from the RAM 24. The RAM control unit 23 includes variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), and 22 (3-1) to 22 (3-4). , 22 (4-1) to 22 (4-4), the pixel value information received from the variable reduction units in the RAM 24 is written to the designated area, or read from the request units 25 (1) to 25 (4). Based on the request, the pixel value information is read from the RAM 24 and output to the generation units 26 (1) to 26 (4). The RAM control unit corresponds to, for example, a write request unit and a read control unit.

  The generation units 26 (1) to 26 (4) generate and generate synthesized video signals by combining the video signals 13 (1) to 13 (4) based on the pixel value information received from the RAM control unit 23. The composite video signal is output while inserting a synchronization signal and a blank as appropriate. At the same time, the generators 26 (1) to 26 (4) manage the coordinates in the video frame of the currently output composite video signal, and coordinate information to be read out next based on the currently output composite video signal. To the requesting units 25 (1) to 25 (4).

  The request units 25 (1) to 25 (4) refer to the video arrangement information in the video arrangement information management unit 27 based on the coordinate information from the generation units 26 (1) to 26 (4), and A request for reading pixel value information to be read next is issued to the control unit 23. The RAM control unit 23 reads the requested pixel value information from the RAM 24 and passes it to the generation units 26 (1) to 26 (4).

  The RAM 24 includes variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 ( 4-1) to 22 (4-4) each have a frame buffer. That is, the RAM 24 has a total of 4 × 4 = 16 frame buffers corresponding to these variable reduction units. For example, pixel value information is sequentially written into each frame buffer from the head address, and is overwritten again from the head address every time the frame of the input video is changed. If a so-called double buffer structure is adopted, the capacity of each frame buffer is doubled.

  In addition, since the above-mentioned RAM control unit may receive a plurality of requests at the same time, there may actually be a time lag from when it is requested until it is executed. The processing may be performed sequentially, or the processing may be performed in the order of predetermined priority). Therefore, a buffer that temporarily holds data may be provided between each variable reduction unit and the RAM control unit, between the RAM control unit and each generation unit, or both.

  The detailed flow of processing by the video composition apparatus as described above will be described below.

  In the input system on the left side of FIG. 1, 8-bit parallel pixel value information is sequentially generated from the upper left pixel of the screen to the lower right pixel of the screen as a video signal by a 27 MHz clock. Input to (4). Further, a predetermined synchronization code is inserted into the video signal at the start position and end position of each line in the frame, and a synchronization code indicating it is further inserted into the video signal at the start position of the frame.

  The analysis units 21 (1) to 21 (4) manage X coordinate values (for example, 0 to 1715 (including horizontal blanks)) and Y coordinate values (for example, 0 to 524 (including vertical blanks)) inside. . The analysis units 21 (1) to 21 (4) increment the X coordinate value by 1 every time pixel data arrives, and when the X coordinate value reaches the maximum value, count again from 0 and increase the Y coordinate value by 1. To do. When the analysis unit 21 (1) to 21 (4) detects the above-described synchronization code, if the synchronization code indicates the start position of the frame, the X coordinate value and the Y coordinate value managed internally are analyzed. Frame synchronization is achieved by resetting.

  Variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4), 22 (3-1) to 22 (3-4), 22 (4-1) ) To 22 (4-4), for example, in the case of 1/1 reduction, if the current coordinate position is a blank area, the pixel value information is invalid and is discarded. If the current coordinate position is not a blank area, the pixel value information is directly passed to the RAM control unit 23, and a 1/1 reduction frame corresponding to the coordinates of the pixel value information is used as an address to which the pixel value information is to be written. Specifies an address in the buffer. In the case of 1/2 reduction, if the current coordinate position is not a blank area, the pixel value information is appropriately averaged to perform RAM control on information obtained by reducing the input video by 1/2 in the vertical and horizontal directions. The address to be written is also transferred to the RAM control unit 23 in the same manner as described above. Similarly, in the case of other reduction ratios, the input video is appropriately reduced and transferred to the RAM control unit 23.

  For example, when the video layout as shown in FIG. 3 is requested for the output video 15 (1), the input video 13 (1) is reduced by 1/2, so that the variable reduction unit 22 (1-1) Performs 1/2 reduction. The RAM control unit 23 writes pixel value information appropriately received from the variable reduction unit 22 (1-1) to an address in the RAM 24 designated by the variable reduction unit 22 (1-1). The same applies to the other input images 13 (2) to 13 (4).

  On the other hand, in the output system on the right side of FIG. 1, in order to generate the BT.656 signal as in the input system, the generators 26 (1) to 26 (4) manage and analyze the X coordinate value and the Y coordinate value. Similar to the units 21 (1) to 21 (4), the value is updated one by one using a 27 MHz clock.

  Here, when the video layout of FIG. 3 described above is requested, the request unit 25 (1), based on the coordinate value specified by the generation unit 26 (1), the coordinate value is at the upper left position of the video layout. If there is an input video 13 (1) reduced by 1/2, if the upper right, input video 13 (2) reduced by 1/2, if lower left, input video 13 (3) reduced by 1/2, right If it is below, the address of the coordinate in the input video 13 (4) reduced by 1/2 is designated to the RAM control unit 23. The RAM control unit 23 reads the pixel value information from the address in the RAM 24 designated by the request unit 25 (1), and passes the read pixel value information to the generation unit 26 (1).

  The generation units 26 (1) to 26 (4) output the received pixel value information as a ITU-R BT.656 signal with a 27 MHz clock while appropriately inserting blanks or synchronization codes.

  As described above, according to the present embodiment, it is possible to generate and output four synthesized videos by synthesizing four input videos in four patterns.

  In addition, each variable reduction unit may perform processing in synchronization with the corresponding input video, and each request unit may perform processing in synchronization with the corresponding output video. That is, the variable reduction unit and the request unit are mutually connected. Since these are asynchronous and do not need to be linked, a simple algorithm is realized as compared with the technique described in Japanese Patent Laid-Open No. 5-103324, and thus a simpler circuit with a smaller circuit scale can be realized.

  The request unit generates a pixel value information read request for a plurality of output videos (output videos 15 (1) to 15 (4)) with one (request unit 25) as shown in FIG. It may be in the form required by The point is that a request corresponding to a plurality of output videos is made.

  In the above-described embodiment, an example in which four video signals are input to generate four composite videos has been described. However, the number of input video signals is not necessarily four, and the generated composite video The number is not limited to four. The number of video signals and the number of synthesized videos may be more or less as long as they are plural, and these numbers may be different from each other. The standard of the input video signal and the output video signal may not be ITU-R BT.656. In this embodiment, a method of simply averaging the pixel values is used as the reduction process. However, since the reduction algorithm itself is not related to the essence of this embodiment, a method using some kind of filter is used. Also good. Further, the RAM and the RAM control unit are each a single device as in this embodiment, and in addition to the number of variable reduction units and the number of outputs (the number of video arrangement information), Each may be divided into a plurality of elements, and the present invention includes this. Also, depending on the arrangement pattern, only a part of the input video may be arranged, and the input video may be arranged by cutting out a part of the input video. The input images may be arranged so as to overlap each other. A background color may be displayed in an area where no video is arranged. A frame may be displayed on the video. Further, it may include reversing the video, or combining a plurality of videos transparently. The processing by the analysis unit, variable reduction unit, video arrangement information management unit, request unit, and generation unit in FIG. 1 may be realized by causing a computer to execute a program generated based on a normal programming technique. It may be realized in hardware. What has been described in this paragraph also applies to the embodiments described later.

(Second Embodiment)
FIG. 4 is a block diagram schematically showing a configuration of a video composition device according to the second embodiment of the present invention.

  This embodiment is different from the first embodiment in that the variable reduction units 22 (1-1) to 22 (1-4), 22 (2-1) to 22 (2-4) in FIG. 22 (3-1) to 22 (3-4) and 22 (4-1) to 22 (4-4) are fixed reduction units 28 (1-1) to 28 (1-4) that reduce the fixed magnification. ), 28 (2-1) to 28 (2-4), 28 (3-1) to 28 (3-4), and 28 (4-1) to 28 (4-4).

  Fixed reduction units 28 (1-1) to 28 (1-4), 28 (2-1) to 28 (2-4), 28 (3-1) to 28 (3-4), 28 (4-1) ) To 28 (4-4) are reduced only by a fixed reduction rate, and the reduction rate is not designated from the video arrangement information management unit 27. Fixed reduction units 28 (1-1) to 28 (1-4), 28 (2-1) to 28 (2-4), 28 (3-1) to 28 (3-4), 28 (4-1) ) To 28 (4-4), regardless of the dynamically changing image arrangement information, the input image is always reduced at the same reduction rate, and the pixel value information is written into the RAM 24 via the RAM control unit 23. In this example, four fixed rates of 1/1, 1/2, 1/3, and 1/4 are given, and there are a total of 4 × 4 = 16 frame buffers on the RAM 24 (so-called double buffer). In the case of, it is twice that).

  Other configurations and operations are the same as those in the first embodiment.

  That is, the request units 25 (1) to 25 (4) are based on the video arrangement information in the video arrangement information management unit 27 and the coordinate information specified by the generation units 26 (1) to 26 (4). A request for reading pixel value information from the frame buffer in the RAM 24 is sent to the RAM control unit 23. The RAM control unit 23 reads out the pixel value information from the RAM 24 and outputs it to the generation units 26 (1) to 26 (4). For example, in the case of the video layout as shown in FIG. 3 described above, the request units 25 (1) to 25 (4) have the coordinate values designated by the generation units 26 (1) to 26 (4) at the upper left of the video layout. The input image 13 (1) reduced by 1/2 if it is a position, the input image 13 (2) reduced by 1/2 if it is the upper right, and the input image 13 (3) reduced by 1/2 if it is the lower left. If it is the lower right, the address of the coordinate of the input video 13 (4) reduced by 1/2 is designated to the RAM control unit 23. The RAM control unit 23 reads pixel value information from an address in the RAM 24 designated by the request units 25 (1) to 25 (4), and passes the read pixel value information to the generation units 26 (1) to 26 (4). .

  Here, the effect of this embodiment will be described in comparison with the first embodiment and the prior art.

  First, in the first and second embodiments, the number of input images is four, but the number of input images is N. In the second embodiment, four fixed reduction units are used, ie, 1/1, 1/2, 1/3, and 1/4. However, if there are a plurality of fixed reduction units, the number is larger than this. The number may be less, and this number is M. However, since the 1/1 reduction unit does not substantially perform reduction processing, the total number of fixed reduction units is substantially (M−1) × N. On the other hand, in the above-mentioned Japanese Patent Laid-Open No. 5-103324 or the first embodiment, N × N variable reduction units are required to realize the same N input images and N output images. Taking these ratios, it is (M-1) / N.

  Here, in consideration of a realistic video conference MCU and an image dividing unit of a surveillance camera, the value of N is usually 4 or more in many cases. On the other hand, with regard to the value of M, the number of input images arranged on the composite screen is about 4 in one direction (that is, about 4 × 4 = 16 on the screen at most), so the reduction rate is 1/1. , 1/2, 1/3, 1/4, 2/3, 3/4 (total of 6) are often used, and the number of them is at most 5 except 1/1. Therefore, when N ≧ 4 and M = 6, (M−1) / N ≦ 5 / 4≈1, and the number of fixed reduction parts is at most equal to or less than the number of variable reduction parts. .

  However, in general, a variable reduction unit with a variable reduction rate must cope with any reduction rate as compared with a fixed reduction unit with a fixed reduction rate, so that the circuit scale is much larger than that with a fixed reduction unit. Therefore, when (M−1) is about the same as N, the circuit scale of the entire reduction unit can be made much smaller than that of the first embodiment and the prior art by this embodiment.

(Third embodiment)
In the first and second embodiments, it has been described that the request unit of FIGS. 1 and 4 makes a read request to the RAM control unit using the video arrangement information. In practice, the request unit of FIGS. The request unit makes a read request to the RAM control unit 23 using request information generated from the video arrangement information. Therefore, in order for the request unit to make a read request, it is necessary to obtain the request information in advance. This request information is generated by the video arrangement information management unit 27. Therefore, for example, as shown in FIG. 5, the video arrangement information management unit 27 generates request information based on the video arrangement information 14 (1) to 14 (4), and requests information generation units 31 (1) to 31 ( 4).

  Here, the video arrangement information and the request information will be briefly described.

  The video layout information is information that defines the layout of the input video, and is generally natural in the form of “place this input video at this position on the output video”. Specifically, assuming that the coordinates of the effective part of the video (the part excluding the blank) are X = 0 to 720 and Y = 0 to 480, in the case of the composite video in FIG. 3, 1 / of the input video 13 (1). 2 Reduced video to (0, 0), 1/2 reduced video of input video 13 (2) to (360, 0), 1/2 reduced video of input video 13 (3) to (0, 240) Then, the information is a half-reduced video of the input video 13 (4) to (360, 240). When this is expressed as a set of subset information (input video number, x coordinate, y coordinate, magnification), (1, 0, 0, 1/2) (2, 360, 0, 1/2). Become. Input video number 1 corresponds to input video 13 (1), input video number 2 corresponds to input video 13 (2),.

  On the other hand, the request information is for determining a request content to the RAM control unit 23 based on the current coordinates received from the generation unit by the request unit. “The input video is arranged at this position on the output video. It is desirable that the format is “Yes”. That is, the coordinate (0, 0) is a 1/2 reduced image of the input image 13 (1), the coordinate (360, 0) is a 1/2 reduced image of the input image 13 (2), and the coordinates (0, 0). 240) is a 1/2 reduced image of the input image 13 (3) and coordinates (360, 240) are 1/2 reduced images of the input image 13 (4). The request unit can be obtained by designating an input video to be requested from the current coordinates. More specifically, an address on the video RAM to be requested is designated. The correspondence to the addresses on the RAM and the conversion process are omitted.

  Note that this arrangement example is a simple example, but there may be cases where, for example, the input images are arranged so as to overlap each other. However, in that case, the request information generation unit analyzes the overlap, and the overlapped area is such that the highest-order input video only becomes the arrangement video, and the other processes are the same. In addition, there may be a region where no video is arranged. In this case, only a predetermined background color or background video is arranged in the region, and the other processes are the same.

  Here, the request unit previously stores the request information (or at least part of the request information) in advance before starting the generation of the video frame by the generation unit, that is, before making a read request to the RAM control unit 23. It is necessary to set. Therefore, the request information generation units 31 (1) to 31 (4) in FIG. 5 perform the video arrangement information 15 during a period in which the request units 25 (1) to 25 (4) are not operating, for example, during the vertical blank period. Request information is generated from (1) to 15 (4). Then, the request units 25 (1) to 25 (4) receive the request information necessary for the current video frame during the vertical blank period from the request information generation units 31 (1) to 31 (4) set. The request information is generated in a blank period in which the requesting units 25 (1) to 25 (4) do not operate, and this request information is repeatedly used in the effective period (non-blank period) of the video frame.

  Here, in FIG. 5, a plurality of request information generation units 31 (1) to 31 (4) are provided in the video arrangement information management unit 27, but as shown in FIG. One request information generation unit 51 is provided, and the request information generation unit 51 generates request information for each of the request units 25 (1) to 25 (4) in a time-sharing manner, whereby the circuit scale can be further reduced. This will be described in detail below.

  FIG. 6 is a diagram illustrating a state in which video frames are output along the time axis from the generation units 26 (1) to 26 (4) illustrated in FIG. 1 or 4.

  The hatched portion indicates the vertical blank 41 and the horizontal blank 42. A portion surrounded by the vertical blank 41 and the horizontal blank 42 is an effective portion of the video frame. Each of the generating units 26 (1) to 26 (4) is configured to generate Y coordinates that are shifted from each other by a time corresponding to a vertical blank. That is, each video frame is generated and output at a timing such that the vertical blank periods do not overlap each other.

  By shifting the generation and output timing of the video frame in this way, the time for generating the request information for each request unit 25 (1) to 25 (4) can be shifted, so as shown in FIG. Request information can be generated in a time-sharing manner using one request information generation unit. However, it is assumed that the time for generating one piece of request information is shorter than the vertical blank period.

  In the present embodiment described above, each generation unit generates a video frame so that the vertical blank periods do not overlap at all. However, this is an example, and part of the video frames may overlap. Is not limited to the above. Further, the request information generation unit generates the request information within the vertical blank period, but this is only an example, and the request information generation unit may generate the request information during the effective period of the video frame. In short, it is only necessary to be able to shift the timing of generating and outputting video frames so that the request information generating unit can generate request information used by a plurality of requesting units in a time-sharing manner.

  As described above, according to the present embodiment, since the request information generation unit can be shared by a plurality of request units, the circuit scale can be further reduced.

(Fourth embodiment)
In the first to third embodiments, the reduction rate is discrete in the case of the configuration of the fixed reduction unit shown in FIG. 4, and the mounting cost is taken into consideration even in the case of the variable reduction unit shown in FIG. May become discrete. In the present embodiment, a mode for compensating for these discrete reduction rates will be described.

  FIG. 8 is a block diagram schematically showing a configuration of a video composition device according to the fourth embodiment of the present invention.

  A complementary expansion / contraction rate generation unit 61 is arranged between the video arrangement information management unit 27 and the requesting units 25 (1) to 25 (4). The complementary enlargement / reduction ratio generation unit corresponds to, for example, a magnification dividing unit. In this example, the fixed reduction unit shown in FIG. 4 is used as the reduction unit of the input video, but the variable reduction unit shown in FIG. 1 may be used. The complementary enlargement / reduction rate generation unit may be prepared for each output video (for each generation unit), but in this example, a single complementary enlargement / reduction rate generation unit 61 performs processing for a plurality of output videos by time division. I do.

  The complementary enlargement / reduction rate generation unit 61 performs reduction rate division processing that represents the reduction rate calculated by the video layout information management unit 27 as a product of the reduction rate that can be executed by the reduction unit and the complementary enlargement / reduction rate.

  For example, assuming that information indicating 1 / 2.5 reduction of the input video 13 (1) is given from the video arrangement information management unit 27, the complementary enlargement / reduction ratio generation unit 61 represents 1 / 2.5 = 1/2 × 4/5. Therefore, 4/5 is generated as the complementary scaling factor. Then, the complementary enlargement / reduction ratio generation unit 61 gives request information requesting that the 1/2 video of the input video 13 (1) is thinned to 4/5 and used, for example, to the request unit 25 (1). By doing so, as a result, a 1 / 2.5 reduced video is output to the generation unit 26 (1). Alternatively, since it can be expressed as 1 / 2.5 = 1 / 3.times.6 / 5, the complementary enlargement / reduction rate generation unit 61 generates the complementary enlargement / reduction rate 6/5, and the input video is supplied to the request unit 25 (1). Request information requesting to use a part of the 1/3 video in an overlapping manner may be provided so that the 1/3 video of 13 (1) is 6/5 times.

  As described above, according to the present embodiment, it is possible to realize a continuous reduction rate for the input video while keeping the reduction unit small.

(Fifth embodiment)
In the present embodiment, another mode for compensating for a discrete reduction rate will be described.

  FIG. 9 is a block diagram schematically showing a configuration of a video composition device according to the fifth embodiment of the present invention. The difference from FIG. 8 is that enlargement / reduction units 71 (1) to 71 (4) that perform complementary enlargement / reduction of video are arranged between the RAM control unit 23 and the generation units 26 (1) to 26 (4). The point is. The enlargement / reduction unit corresponds to, for example, a video supplement scaling unit, a video supplement reduction unit, or a video supplement enlargement unit.

  For example, assuming that information indicating 1 / 2.5 reduction of the input video 13 (1) is given from the video arrangement information management unit 27, the complementary enlargement / reduction ratio generation unit 61 represents 1 / 2.5 = 1/2 × 4/5. Therefore, 4/5 is generated as the complementary reduction ratio. Then, the complementary enlargement / reduction rate generation unit 61 gives request information for requesting 1/2 reduction of the input video 13 (1) to the request unit 25 (1), for example, and the enlargement / reduction unit 71 (1). By instructing to perform 4/5 reduction, as a result, a 1 / 2.5 reduction video is output to the generation unit 26 (1). Alternatively, since the complementary expansion / contraction rate generation unit 61 can be expressed as 1 / 2.5 = 1/3 × 6/5, 6/5 is generated as the complementary expansion / contraction rate, and the expansion / contraction units 71 (1) to 71 (4). May be instructed to perform 6/5 magnification.

  Here, as can be understood from the first to fourth embodiments described above, the embodiment of the present invention arranges a reduction unit for reducing the input video in the previous stage of the RAM control unit, and performs synthesis processing. One of the major features is that the reduction processing and the synthesis processing are separated to facilitate the processing by arranging the requesting unit and the generation unit for performing the processing in the subsequent stage of the RAM control unit. From this point of view, the fifth embodiment at first glance seems to be contrary to this feature. However, since the enlargement / reduction unit in the present embodiment only supplements the discrete reduction rate of the reduction unit, the enlargement / reduction rate performed by the enlargement / reduction unit is close to 1, and the fluctuation range is small. Therefore, since the fluctuation of the processing delay due to the fluctuation of the enlargement / reduction ratio by the enlargement / reduction unit is small, and the estimation of the delay is easy, one of the effects according to the embodiment of the present invention is to simplify the processing. It does not impair reducing the circuit scale.

  The fifth embodiment described above can be combined with the fourth embodiment.

  That is, first, the complementary enlargement / reduction rate generation unit represents the reduction rate of the input video as a product of the reduction rate executable by the reduction unit and the complementary enlargement / reduction rate.

  Thereafter, for example, when the complementary scaling ratio is larger than 1, the input video read from the RAM control unit is enlarged by the complementary scaling ratio, and when the complementary scaling ratio is less than 1, the input video is The request unit requests the RAM control unit to draw and use the data according to the complementary expansion / contraction rate.

  Alternatively, when the complementary scaling ratio is less than 1, the input video read out from the RAM control unit is reduced by the complementary scaling ratio, and when the complementary scaling ratio is larger than 1, the input video is complemented by the scaling ratio. The request unit requests the RAM control unit to use the same amount in accordance with

  As described above, the video conference server is shown as an embodiment. However, the present invention relates to general synthesis processing for inputting a plurality of videos and outputting a plurality of synthesized videos, and is limited to the video conference server. It is not something. For example, you can combine multiple surveillance camera images for surveillance purposes, or synthesize PC screens, TV screens, game screens, etc. in the office or home. The signal format does not matter (may be different for each channel).

1 is a block diagram schematically showing a configuration of a video composition device according to a first embodiment of the present invention. It is a block diagram which shows roughly the structure of the video conference system to which the video synthesizing | combining apparatus of FIG. 1 is applied. It is a figure which shows an example of video arrangement | positioning. It is a block diagram which shows roughly the structure of the video synthesizing | combining apparatus according to the 2nd Embodiment of this invention. It is a figure which shows the structure by which the video arrangement | positioning information management part was provided with the several request information generation part. It is a figure which shows a mode that a some video frame is output along time. It is a figure which shows the structure by which the video arrangement | positioning information management part was provided with the single request information generation part. It is a block diagram which shows roughly the structure of the video synthesizing | combining apparatus according to the 4th Embodiment of this invention. It is a block diagram which shows roughly the structure of the video synthesizing | combining apparatus according to the 5th Embodiment of this invention. It is a block diagram which shows schematically the other structure of the video synthesizing | combining apparatus according to the 1st Embodiment of this invention.

Explanation of symbols

11 Video composition device 12 (1) -12 (4) Terminal 13 (1) -13 (4) Input video 14 (1) -14 (4) Output video 15 (1) -15 (4) Video arrangement information 21 ( 1) to 21 (4) Analysis units 22 (1) to 22 (4) Variable reduction unit 23 RAM control unit 24 RAM
25 (1) to 25 (4), 25 Request unit 26 (1) to 26 (4) Generation unit 27 Video layout information management unit 28 (1) to 28 (4) Fixed reduction unit 51 Request information generation unit 61 Complementary reduction Rate generator 71 (1) -71 (4) Enlargement / reduction unit

Claims (18)

I pieces of image scaling means each for receiving an input image, wherein each of the input images inputted to itself is scaled by a plurality of magnifications to generate a plurality of scaled input images. The image scaling means of
Video storage means for receiving a write request for the scaled input video and temporarily storing the scaled input video in an area designated by the write request;
Each of the plurality of scaled input videos generated by the video scaling means corresponding to the video scaling means is requested to write to a predetermined area in the video storage means. A plurality of write request means;
Read request means for making a read request for an input video based on each of the J video layout information, each of the video layout information defining a layout of each input video on the composite video,
Read control means for receiving the read request and reading the input video from the video storage means;
J composite video generation means, each corresponding to the video arrangement information, for generating a composite video from the input video read by the read control means;
A video composition device. A magnification designating unit for designating a plurality of magnifications for each of the video scaling units based on the J video layout information;
The image scaling unit generates the plurality of scaled input images by scaling the input image according to the plurality of magnifications designated by the magnification designation unit.
The video synthesizing apparatus according to claim 1.   3. The video synthesizing apparatus according to claim 2, wherein the video scaling unit includes J variable scaling units.   2. The video synthesizing apparatus according to claim 1, wherein the video scaling unit includes a plurality of fixed scaling units each having a different fixed magnification.   2. The video composition apparatus according to claim 1, wherein the video scaling unit includes fixed scaling units equal to the number of magnifications that can be specified by the video arrangement information with respect to an input video input to the video scaling unit. Request information for sequentially generating information on the input video to be arranged in each coordinate area on the composite video as request information to be referred to when the read request unit makes the read request from each of the video arrangement information. A generation unit;
The read request unit starts the read request to the read control unit so that the request information generation unit shifts at least time necessary for generating the request information in accordance with the generation order of the request information.
The video synthesizing apparatus according to claim 1.   The video composition apparatus according to claim 6, wherein the request information generation unit finishes generation processing of request information corresponding to the video layout information within a vertical blank period of the composite video corresponding to the video layout information.   2. The video composition apparatus according to claim 1, wherein the video scaling unit generates a plurality of reduced input videos by reducing the input video input thereto by a plurality of reduction ratios. The magnification of the input video arranged according to the video arrangement information is expressed as a product of a magnification that can be scaled by the video scaling means and a complementary scaling factor. When the complementary scaling factor is less than 1, the scaling factor is changed. When the multiplied input image is used by being drawn for a time corresponding to the complementary scaling factor, and the complementary scaling factor is greater than 1, the scaled input video is overlapped by the amount corresponding to the complementary scaling factor. Further comprising a magnification division means for requesting the read request means to use
The video synthesizing apparatus according to claim 1.   The magnification of the input video arranged according to the video arrangement information is represented as a product of a magnification that can be scaled by the video scaling means and a complementary reduction ratio having a value less than 1, and the scaled input video The video composition apparatus according to claim 1, further comprising a magnification division unit that requests the read request unit to draw and use for a period corresponding to the complementary reduction ratio. The magnification of the input video arranged according to the video arrangement information is expressed as a product of a magnification that can be scaled by the video scaling means and a complementary enlargement factor having a value larger than 1, and the scaled input video Further comprising a magnification division means for requesting the read request means to use the same in accordance with the complementary enlargement ratio,
The video synthesizing apparatus according to claim 1. Magnification dividing means for expressing the magnification of the input video arranged according to the video arrangement information as a product of a magnification that can be scaled by the video scaling means and a complementary scaling ratio;
Each scaled input video read from the readout control unit is scaled by the complementary scaling factor, and the scaled input video is converted to the synthesized video generation unit. Video complementary scaling means to output to
The video composition apparatus according to claim 1, further comprising: A magnification dividing means for expressing the magnification of the input video arranged according to the video arrangement information as a product of a magnification that the video scaling means can scale and a complementary reduction ratio having a value less than 1;
Each scaled input video read from the readout control means is arranged corresponding to the synthesized video generation means, and reduced by the complementary reduction ratio, and the reduced input video is output to the synthesized video generation means Video complement reduction means to
The video composition apparatus according to claim 1, further comprising: A magnification dividing means for expressing the magnification of the input video arranged according to the video arrangement information as a product of a magnification that can be changed by the video scaling means and a complementary enlargement ratio having a value larger than 1.
Each of the scaled input videos that are arranged corresponding to the synthesized video generation means and read from the readout control means are enlarged by the complementary enlargement ratio, and the enlarged input video is output to the synthesized video generation means Video supplement enlargement means,
The video composition apparatus according to claim 1, further comprising: Magnification dividing means for expressing the magnification of the input video arranged according to the video arrangement information as a product of a magnification that can be scaled by the video scaling means and a complementary scaling ratio;
When each of the composite video generation means is arranged correspondingly and the complementary scaling factor is larger than 1, the scaled input video read from the readout control means is enlarged by the complementary scaling factor, Video supplement expansion means for outputting the input video to the composite video generation means,
The magnification dividing unit requests the read request unit to draw and use the scaled input video for a time corresponding to the complementary scaling factor when the complementary scaling factor is less than 1.
The video synthesizing apparatus according to claim 1. Magnification dividing means for expressing the magnification of the input video arranged according to the video arrangement information as a product of a magnification that can be scaled by the video scaling means and a complementary scaling ratio;
When each of the composite video generation means is arranged correspondingly and the complementary scaling ratio is less than 1, the scaled input video read from the readout control means is reduced by the complementary scaling ratio and reduced. Video supplement reduction means for outputting a later input video to the synthesized video generation means, and
The magnification dividing unit requests the read control unit to use the scaled input video in an overlapping manner according to the complementary magnification when the complementary scaling factor is greater than 1.
The video synthesizing apparatus according to claim 1. Each of the I input videos is scaled by a plurality of magnifications to generate a plurality of scaled input videos for each of the input videos,
Writing each of the plurality of scaled input videos generated for each of the input videos into a predetermined area in the video storage means;
Based on each of the J video arrangement information, the input video is read from the video storage means, and the video arrangement information defines the arrangement of each input video on the composite video,
Generating a composite video from the input video read for each video layout information;
A video composition method characterized by the above. Scaling each of the I input videos by a plurality of magnifications to generate a plurality of scaled input videos for each of the input videos;
Writing each of the plurality of scaled input videos generated for each of the input videos into a predetermined area in the video storage means;
Reading the input video from the video storage means based on each of the J video layout information, wherein the video layout information defines the layout of each input video on the composite video; and
Generating a composite video from the input video read for each video layout information;
Is a video composition program that causes a computer to execute.

Images