JP6513305B2 - Video combining apparatus and video combining method - Google Patents

Description

  The present invention relates to a video combining apparatus and a video combining method for generating one composite video from a plurality of videos (that is, a plurality of video data) acquired by a plurality of imaging devices.

  In order to widen the shooting angle of view of a video, video combining processing is known which combines a plurality of videos acquired by shooting with a plurality of imaging devices (that is, a plurality of cameras) to generate one composite video. (See, for example, Patent Documents 1, 2 and 3). Usually, in video composition processing for generating one composite video, video processing such as lens distortion correction processing, viewpoint conversion processing, and projection conversion processing is performed on each of a plurality of videos output from a plurality of imaging devices. To be done. Since the processing load of these video processing is very large, it is difficult to perform these video processing in real time by a normal processing unit (CPU: Central Processing Unit). Therefore, in the conventional device, the video composition processing is performed by a GPU (Graphics Processing Unit) which is a parallel processing device capable of operating in parallel with a normal processing device.

Patent No. 4744823 JP, 2015-207802, A JP, 2016-066842, A

  However, even in the case of using a parallel computing device such as a GPU, the load of the video synthesizing process increases as the number of imaging devices increases (that is, the number of images to be synthesized increases). In particular, when performing blending processing on an image in a superimposed region which is a boundary between imaging ranges by the imaging device in the projection conversion processing, processing waiting occurs until a plurality of images in the superimposed region are input. There is a problem that the processing time required for the combining process becomes long.

  The present invention has been made to solve the above-described conventional problems, and an object thereof is to provide one image from a plurality of videos acquired by a plurality of imaging devices even when the number of imaging devices is increased. An object of the present invention is to provide a video combining apparatus and a video combining method capable of performing video combining processing for generating a combined video in a short time.

The video compositing apparatus according to an aspect of the present invention is a video compositing apparatus that generates one composite video from a plurality of videos acquired by a plurality of imaging devices, and a video receiving unit that receives the plurality of videos. The image processing apparatus further includes: a parameter input unit to which camera parameters of the plurality of imaging devices are input; and a video processing unit that generates the composite video from the plurality of videos, and the video processing unit In the imaging device specified by the first imaging device specifying information for specifying the corresponding imaging device among the plurality of imaging devices for each pixel of the composite video using the first imaging device specifying information Creating a reference table including a corresponding first pixel position and a first weighted coefficient at the corresponding first pixel position, and referring to the reference table to draw the image of the composite image Each time, the first value obtained by multiplying the first weighted coefficient by the pixel value of the corresponding first pixel position in the imaging device specified by the first imaging device specifying information is substituted. By doing this, the composite video is generated, and the video processing unit uses the camera parameters to set each of the pixels of the composite video among the plurality of imaging ranges captured by the plurality of imaging devices. A second imaging device identification information identifying one of the imaging devices having a superimposed region which is an imaging range to be superimposed, and a second corresponding to the superimposed region in the imaging device identified by the second imaging device identification information Creating another reference table including the pixel position of the pixel and the second weighted coefficient at the second pixel position corresponding to the overlapping area, and referring to the other The pixel value of the corresponding pixel position in the imaging device specified by the second imaging device specifying information is multiplied by the second weighted coefficient for each pixel corresponding to the second pixel position in the overlapping area. By substituting the second value obtained by performing the blending process and performing the blending process, a portion corresponding to the superimposed region in the composite image is generated .

A video combining method according to another aspect of the present invention is a video combining method for generating one combined video from a plurality of videos acquired by a plurality of imaging devices, and a camera input in advance for the plurality of imaging devices. First imaging device identification information for identifying a corresponding imaging device among the plurality of imaging devices for each pixel of the composite video using a parameter, and imaging identified by the first imaging device identification information Creating a first lookup table including corresponding first pixel locations in the device and first weighted coefficients at the corresponding first pixel locations, with reference to the first lookup table The pixel value of the corresponding first pixel position in the imaging device specified by the first imaging device specification information is multiplied by the first weighted coefficient for each pixel of the composite video. By substituting the first value that is, the step of generating the synthesized image, using the camera parameters, for each pixel of the composite image, the plurality of imaging range to be imaged by the plurality of imaging devices The second imaging device identification information for identifying one of the imaging devices having an overlapping region which is an imaging range to overlap with each other, and the overlapping region in the imaging device identified by the second imaging device identification information Creating a second reference table including a corresponding second pixel position and a second weighted coefficient at a second pixel position corresponding to the overlapping area; and referring to the second reference table Of the pixels of the composite image, the corresponding pixels in the imaging device specified by the second imaging device specifying information for each pixel corresponding to the second pixel position of the overlapping area By substituting the second value obtained by multiplying the second weighted coefficient by the pixel value of the pixel and performing the blending process, a portion corresponding to the superimposed area in the composite video is generated And the step of

  According to the present invention, even when the number of imaging devices increases, it is possible to perform, in a short time, video combining processing for generating one composite video from a plurality of videos acquired by a plurality of imaging devices.

FIG. 1 is a functional block diagram schematically showing a configuration of a video synthesis apparatus according to Embodiment 1 of the present invention. FIG. 1 is a hardware configuration diagram schematically showing a video synthesis apparatus according to Embodiment 1. FIG. 7 is a diagram showing an example of the correspondence between pixels of a synthesized video and pixels of a plurality of imaging devices in the video synthesizing apparatus according to Embodiment 1; FIG. 6 is a diagram showing an example of a superimposed region of imaging ranges of a plurality of imaging devices in the video compositing device according to Embodiment 1. FIG. 7 is a diagram showing the range of pixels of the imaging device included in the first reference table in the video compositing device according to the first embodiment. FIG. 7 is a diagram showing the range of pixels of the imaging device included in the second reference table in the video compositing device according to the first embodiment. 7 is a flowchart showing an operation of the video combining apparatus according to Embodiment 1 (that is, a video combining method according to Embodiment 1). FIG. 16 is a diagram illustrating an example of a superimposed region of trapezoidal imaging ranges of a plurality of imaging devices in the video compositing device according to the second embodiment. FIG. 14 is a diagram showing an example in which the imaging ranges of a plurality of imaging devices in the video compositing device according to the second embodiment are simplified. FIG. 16 is a diagram showing the range of pixels of the imaging device included in the first reference table in the video compositing device according to the second embodiment. FIG. 16 is a diagram showing a range (superimposed region) of pixels of an imaging device included in a second reference table in the video compositing device according to the second embodiment. FIG. 16 is a diagram showing a range (superimposed area) of pixels of the imaging device included in the third reference table in the video compositing device according to the second embodiment. FIG. 16 is a diagram showing a range (superimposed region) of pixels of the imaging device included in the fourth reference table in the video compositing device according to the second embodiment.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration FIG. 1 is a functional block diagram schematically showing a configuration of a video combining device 1 according to Embodiment 1 of the present invention. The video compositing apparatus 1 is an apparatus capable of implementing the video compositing method according to the first embodiment. The video compositing apparatus 1 includes one composite video (i.e., a plurality of video data) output from a plurality of imaging devices (i.e., a plurality of cameras) Cam1, ..., Cami, ..., CamN. Generate one composite video data). N is an integer of 2 or more, and i is an arbitrary integer of 1 or more and N or less. When the video is a moving image, the video compositing device 1 performs processing for creating one composite video frame from the N video frames output from the N imaging devices Cam1, ..., CamN. Repeatedly every time a video frame is input from Cam1, ..., CamN, moving image data as composite video data is generated. The generated composite video data is output to the display device 2. The display device 2 displays a video based on the received composite video data.

  Examples of the composite video are a panoramic video which is a horizontally long video having a wide field of view and a overhead video which is a video viewed down from a high position. In the first embodiment, a case will be described where a composite video generated by combining a plurality of videos aligned in the left-right direction (one-dimensional direction) acquired by a plurality of imaging devices is a panoramic video. Further, in the second embodiment described later, the case where the composite video generated by combining the plurality of videos arranged in the vertical and horizontal directions (two-dimensional direction) acquired by the plurality of imaging devices is a bird's eye view Do. The image synthesizing device 1 creates in advance a reference table having information on the pixels of the imaging devices Cam1, ..., CamN corresponding to the pixels of the synthesized image, and sets the pixel values of the pixels of the synthesized image using this reference table ( substitute.

  As shown in FIG. 1, the video combining apparatus 1 according to the first embodiment includes a video receiving unit 4, a parameter input unit 5, a video processing unit 6 having a storage unit 6a, and a display processing unit 7. doing. The storage unit 6 a may be provided outside the video processing unit 6. The video synthesizing apparatus 1 shown in FIG. 1 uses a memory as a storage unit 6a for storing a program as software and a processor as an information processing unit for executing a program stored in the memory (for example, by a computer) ) Can be realized. Note that a part of the video synthesizing apparatus 1 shown in FIG. 1 may be realized by a memory for storing a program and a processor for executing the program.

  The video receiving unit 4 receives a plurality of video data output from the plurality of imaging devices Cam 1,..., Cam N, and outputs the received video data to the video processing unit 6. The video data may be decoded by the video reception unit 4 and the decoded video data may be output to the video processing unit 6.

  The parameter input unit 5 receives information indicating camera parameters for the plurality of imaging devices Cam1, ..., CamN obtained by calibration performed in advance, that is, estimation of parameters of the lens and the imaging device of the image sensor, and image processing Output to section 6. The camera parameters are, for example, internal parameters that are camera parameters unique to the imaging devices Cam1, ..., CamN, external parameters that are camera parameters indicating the position and orientation of the imaging devices Cam1, ..., CamN in the world coordinate system, the imaging devices Cam1, ... ..., including lens distortion correction coefficients (eg, lens distortion correction map) and the like used to correct distortion specific to the lens of CamN (eg, distortion in the radial direction of the lens and distortion in the circumferential direction of the lens) .

  In the first embodiment, the video processing unit 6 creates a reference table for video composition at the time of initialization using the camera parameters calculated by calibration performed in advance, and stores the reference table in the storage unit 6a. Store. The video processing unit 6 generates composite video data from a plurality of video data (video frames) output from the video reception unit 4 with reference to the reference table.

  The display processing unit 7 outputs the composite video data generated by the video processing unit 6 to the display device 2.

  FIG. 2 is a hardware configuration diagram schematically showing the video combining apparatus 1 according to the first embodiment. The video synthesizing device 1 includes a main processor 10, a main memory 11, an auxiliary memory 12, a video processing processor 13 which is a parallel computing device such as a GPU, a video processing memory 14, an input interface 15, and a file interface 16. , Display interface 17 and video input interface 18.

  The video processing unit 6 of FIG. 1 includes the main processor 10, the main memory 11, the auxiliary memory 12, the video processing processor 13, and the video processing memory 14 shown in FIG. Further, the storage unit 6a of FIG. 1 includes the main memory 11, the auxiliary memory 12, and the video processing memory 14 shown in FIG. Further, the parameter input unit 5 of FIG. 1 includes the file interface 16 shown in FIG. Further, the video receiving unit 4 of FIG. 1 includes the video input interface 18 shown in FIG. Further, the display processing unit 7 of FIG. 1 includes the display interface 17 shown in FIG. However, FIG. 2 only shows an example of the hardware configuration of the video combining apparatus 1 shown in FIG. 1, and the hardware configuration can be variously changed. Further, the correspondence between the functional blocks 4 to 7 shown in FIG. 1 and the hardware configurations 10 to 18 shown in FIG. 2 is not limited to the above example.

  The parameter input unit 5 of FIG. 1 acquires camera parameter information calculated by calibration performed in advance from the auxiliary memory 12 and writes the file interface 16 in the main memory 11.

  The auxiliary memory 12 may store camera parameters calculated by calibration performed in advance. Also, the main processor 10 may store camera parameters in the main memory 11 through the file interface 16. Further, the main processor 10 may store the still image file in the auxiliary memory 12 when creating a composite image from the still image.

  The input interface 15 receives device input such as mouse input, keyboard input, touch panel input, etc., and sends input information to the main processor 10.

  The video processing memory 14 stores input video data transferred from the main memory 11 and composite video data created by the video processing processor 13.

  The display interface 17 and the display device 2 are connected by an HDMI (registered trademark) (High-Definition Multimedia Interface) cable or the like. The composite video is output to the display device 2 via the display interface 17 as the display processing unit 7.

  The video input interface 18 as the video receiving unit 4 receives video input of the imaging devices Cam1, ..., CamN connected to the video combining device 1, and stores the input video in the main memory 11. The imaging devices Cam1, ..., CamN are, for example, a network camera, an analog camera, a USB (Universal Serial Bus) camera, an HD-SDI (High Definition Serial Digital Interface) camera, and the like. Note that as the video input interface 18, a standard conforming to the device to be connected is used.

<< 1-2 >> Operation [initialization process]
First, the video processing unit 6 in FIG. 1 determines the resolution W_synth × H_synth of the composite video to be created, and secures a memory area for storing the composite video in the storage unit 6 a in FIG. 1. Here, W_synth indicates the number of pixels in the horizontal direction of the rectangular composite video, and H_synth indicates the number of pixels in the vertical direction of the composite video. In other words, the video processing processor 13 determines the resolution W_synth × H_synth of the composite video to be created, and secures a memory area for storing the composite video in the video processing memory 14.

  Next, the image processing unit 6 in FIG. 1 receives the camera parameters (internal parameters, external parameters, lens distortion correction data, projection plane, etc.) of the imaging devices Cam1, ..., CamN input from the parameter input unit 5 in FIG. Reference tables for the imaging devices Cam1, ..., CamN are created and stored in the storage unit 6a. In other words, the video processing processor 13 creates a reference table for the imaging devices Cam1, ..., CamN from the camera parameters of the imaging devices Cam1, ..., CamN input from the file interface 16, in other words, using FIG. And store in the video processing memory 14.

FIG. 3 is a diagram showing an example of the correspondence between the pixels of the composite video and the pixels of the plurality of imaging devices Cam1, ..., CamN in the video combining apparatus 1 according to the first embodiment.
The reference tables for the imaging devices Cam1, ..., CamN are, as shown in Fig. 3, the pixels (x_cam1, y_cam1), ..., (x_camN, y_camN) of the imaging devices Cam1, ..., CamN corresponding to the pixels of the composite image. And the α values (alpha values) in the pixels (x_cam1, y_cam1),..., (X_camN, y_camN) of the corresponding imaging devices Cam1,. x_cam1 indicates the x-coordinate of the pixel of the image sensor of the imaging device Cam1 in which the camera number i = 1, and y_cam1 indicates the y-coordinate of the pixel of the image sensor of the imaging device Cam1 in which the camera number i = 1. The α value is a weighting coefficient used for blending processing of superimposed regions of imaging ranges of the imaging devices Cam1, ..., CamN. The α value is a camera parameter indicating the opacity of pixel data, and is a value within the range of 0 to 1, where α value = 0 represents complete transparency and α value = 1 represents complete opacity.

When there is no pixel of the imaging devices Cam1, ..., CamN corresponding to the pixel of the composite video, a value indicating that the corresponding pixel is not present is set in the reference table. The correspondence between the pixels (x_cam1, y_cam1),..., (X_camN, y_camN) of the imaging devices Cam1, ..., CamN and the pixels (x_synth, y_synth) of the composite image is calculated from the defined coordinates on the projection plane before the projection conversion process. It can be calculated by back-calculating coordinates, coordinates before viewpoint conversion processing, and coordinates before lens distortion correction processing. x_synth indicates the x-coordinate of the pixel of the composite video, and y_synth indicates the y-coordinate of the pixel of the composite video. When the reference table is used, the video synthesis processing is performed using the pixels (x_cam1, y_cam1) of the imaging devices Cam1, ..., CamN corresponding to the pixels (x_synth, y_synth) of the composite video as the pixel values of the pixels (x_synth, y_synth) of the composite video. ,..., (X_camN, y_camN) is a process of substituting pixel values.
However, when the video synthesis process is performed using the reference tables of the imaging devices Cam1, ..., CamN in order, the processing time increases in proportion to the increase in the number of imaging devices Cam1, ..., CamN. In addition, even when using a parallel computing device such as a GPU, processing delay occurs when performing blending processing in the overlap region, so processing time is increased as the number of imaging devices Cam1, ..., CamN increases. Will increase.

  FIG. 4 is a diagram showing an example of the overlapping area of the imaging ranges of the plurality of imaging devices Cam1, ..., Cam4 in the video synthesizing apparatus 1 according to the first embodiment. Usually, when creating a panoramic composite image as shown in FIG. 4, a superimposed region exists in the imaging range of the adjacent imaging devices Cam1, ..., Cam4. Blending is applied to the superimposed region, but the pixel values of different imaging devices Cam1, ..., Cam4 are referred to, weighting is performed by multiplying the pixel value by the weighting coefficient α, and the weighted pixel values are combined. Since substitution (blending) is performed as the pixel value of the corresponding pixel of the video, processing waiting time occurs for the video data output from the imaging devices Cam1, ..., Cam4. Note that, since blending processing is performed in the overlapping region, the overlapping region is also referred to as a blending region.

  FIG. 5 is a diagram showing the range of pixels of the imaging devices Cam1, ..., Cam4 included in the reference table (first reference table) in the video compositing device 1 according to the first embodiment. Further, FIG. 6 is a diagram showing the range of the pixels of the imaging devices Cam1, ..., Cam4 included in another reference table (second reference table) in the video combining apparatus 1 according to the first embodiment. In the video combining apparatus 1 according to the first embodiment, the video processing unit 6 creates a reference table for video combining from the reference tables of the imaging devices Cam1, ..., CamN. The reference table for image composition is a two reference table holding information on the upper side of the blend area (the image pickup device on the left side) and the lower side of the blend region (the image pickup device on the right side). It consists of one reference table and the second reference table shown in FIG.

  The first reference table for image composition includes the camera number i as the first imaging device specifying information, the pixel (x_cami, y_cami) of the corresponding imaging device Cami, and the α value of the pixel of the corresponding imaging device Cami Hold. The α value of the pixels other than the overlapping region is 1. The example of FIG. 5 shows the case where there are four imaging devices Cam1, ..., Cam4. FIG. 5 shows an example, and the range of the pixels of the imaging devices Cam1, ..., Cam4 included in the first reference table is not limited to the example of FIG.

  The second reference table for image composition is used as second imaging device identification information for identifying an imaging device having a superimposed region among a plurality of imaging ranges imaged by a plurality of imaging devices Cam1, ..., CamN. The camera number i, the corresponding pixel (x_cami, y_cami) of the imaging device Cami, and the α value of the pixel of the overlapping region of the corresponding imaging device Cami are held. FIG. 6 shows an example, and the range of the pixels of the imaging devices Cam1, ..., Cam4 included in the second reference table is not limited to the example of FIG.

  The pixels other than the overlapping region of the imaging range of the imaging devices Cam1, ..., CamN and the pixels corresponding to the imaging device on the left side of the overlapping region (or the imaging device on the right side) can be simultaneously substituted into the pixels of the composite image. . What holds the information of these pixels is the upper lookup table (first lookup table) shown in FIG. The lower side reference table shown in FIG. 6 holds information of pixels corresponding to the imaging device (or the imaging device on the left side) on the right side of the overlapping region of the imaging ranges of the imaging devices Cam1, ..., CamN. 2 reference table).

  For example, the pixel values of the pixels in the imaging ranges of the imaging devices Cam1, Cam2, Cam3 and Cam4 shown in FIG. 5 can be simultaneously substituted into the pixels of the composite image using the first reference table. Further, the pixel values of the pixels in the overlap region of the imaging ranges of the imaging devices Cam2, Cam3 and Cam4 shown in FIG. 6 can be simultaneously substituted into the pixels of the composite image using the second reference table. In the first embodiment, by using the video processing processor, which is a parallel computing device such as a GPU, and using the first and second reference tables, no processing waiting occurs in the video combining process, and the imaging devices Cam1, Cam1, The combined video can be generated in two steps, that is, substitution processing using the first reference table and substitution processing using the second reference table, regardless of the number of ..., CamN.

[Video input processing]
The video input interface 18 in the video receiving unit 4 acquires video data of one frame of the imaging devices Cam1, ..., CamN, and stores the video data in the main memory 11. The acquired video data is transferred from the main memory 11 to the video processing memory 14.

[Video composition processing]
The video processing processor 13 in the video processing unit 6 combines the pixel values of the input video transferred to the video processing memory 14 with the first reference table and the second reference table to correspond to the pixels of the input video. It substitutes as the pixel value of the pixel of the image. This processing procedure is described below.

The following video synthesis processing is executed by the video processing processor 13 in parallel with the processing of the main processor 10.
<1> First, as the first process, the video processing processor 13 determines from the first reference table the camera number i corresponding to each pixel (x_synth, y_synth) in the composite video and the camera number i in the imaging device Cami The pixel position (x_cami, y_cami) and the weighting coefficient α are extracted.
<2> Next, as the second process, the video processing processor 13 refers to the pixel value of the input video (x_cami, y_cami) of the camera number i on the video processing memory 14, and assigns a weighting factor α to this pixel value. And is substituted for the pixel of the composite video (x_synth, y_synth) on the video processing memory 14.

Next, the video processing processor 13 executes the following video synthesis processing in parallel with the processing of the main processor 10.
<3> First, as the third process, the video processing processor 13 determines from the second reference table that the imaging device Cami corresponds to the camera number i and the camera number i corresponding to each pixel (x_synth, y_synth) in the composite video. The pixel position (x_cami, y_cami) and the weighting coefficient α are extracted.
<4> Next, as the fourth process, the video processing processor 13 refers to the pixel value of the input video (x_cami, y_cami) of the camera number i on the video processing memory 14, and assigns a weighting factor α to this pixel value. And is substituted for the pixel of the composite video (x_synth, y_synth) on the video processing memory 14. As a result, blending processing is performed on the pixels in the superimposed area of the composite image.

  FIG. 7 is a flowchart showing the operation of the video compositing apparatus according to the first embodiment (that is, the video compositing method according to the first embodiment). After creating the reference table in the initialization process (step S1), the video processing unit 6 continues the video input process (step S2) and the video combining process (step S4) until the video input is completed (step S4) , Repeat. When the positions of the imaging devices Cam1, ..., CamN shift, the video processing unit 6 corrects the positional shift using the feature points on the video and the like, and creates a new reference table in the background, By replacing the currently used reference table with a new reference table, it is possible to create an aligned composite image.

  The display processing unit 7 transmits panoramic composite video data as composite video data created by the video processing unit 6 to the display device 2. The display device 2 displays a video based on the received panoramic composite video data. The display device 2 may display the panoramic composite video on one display screen, or may display the panoramic composite video over a plurality of display screens. Further, the display device 2 may cut out and display only a partial area of the panoramic composite video.

<< 1-3 >> Effects As described above, according to the video compositing device 1 and the video compositing method according to the first embodiment, the decoding load of input video data according to the number of imaging devices Cam1, ..., CamN. However, the load on the video synthesis processing of the video acquired by the imaging devices Cam1, ..., CamN hardly increases.

Furthermore, when lens distortion correction processing, viewpoint conversion processing, and projection conversion processing are applied to input images for each of the imaging devices Cam1, ..., CamN, the processing time increases as the number of imaging devices Cam1, ..., CamN increases. . Also, even in the case where lens distortion correction processing, viewpoint conversion processing, and projection conversion processing are collectively performed by preparing a reference table for each of the imaging devices Cam1, ..., CamN, processing in the overlapping region of the imaging devices Cam1, ..., CamN boundary Since waiting occurs, the processing time increases as the number of imaging devices Cam1, ..., CamN increases.
In the video combining apparatus 1 and the video combining method according to the first embodiment, the first reference table is configured with only data that can be substituted into pixels at the same time, paying attention to processing waiting between videos of imaging devices having overlapping regions adjacent to each other. As a result, it is possible to realize the video composition processing with the number of steps of the maximum number of imaging devices Cam1, ..., CamN related to the superimposed region. That is, in the first embodiment, since the maximum number of imaging devices Cam1, ..., CamN relating to the superimposed area is a panoramic composite image having two units, the step using the first reference table and the second reference table The combining process can be performed in two steps consisting of steps using

<< 2 >> Second Embodiment
<< 2-1 >> Configuration In the first embodiment, the video combining apparatus and the video combining method for generating one composite video (panoramic video) from a plurality of videos arranged in the left and right direction have been described. On the other hand, in a second embodiment of the present invention, a video combining apparatus and a video combining method for generating one composite video (overview video) from a plurality of videos arranged in the vertical and horizontal directions will be described.

  In the second embodiment, the arrangement of the plurality of imaging devices Cam1, ..., CamN, and the video processing unit 6 (or the video processing processor of FIG. 2) in FIG. 1 are the reference table (first reference table) and others. The second embodiment differs from the first embodiment in that video synthesis processing is performed using reference tables (second to fourth reference tables). Except for these points, the second embodiment is the same as the first embodiment. Therefore, in the description of the second embodiment, reference is also made to FIGS. 1, 2 and 7 used in the description of the first embodiment.

<< 2-2 >> Operation FIG. 8 is a diagram showing an example of a superimposed region of trapezoidal imaging ranges of a plurality of imaging devices Cam1, ..., Cam9 in the video combining apparatus according to the second embodiment. As shown in FIG. 8, when creating a bird's-eye view composite video, it is possible to create one large bird's-eye view video by performing viewpoint conversion and projection conversion on the images acquired by the plurality of imaging devices Cam1, ..., Cam9. it can. FIG. 8 shows an example of the arrangement of the plurality of imaging devices Cam1, ..., Cam9, and the arrangement method of the plurality of imaging devices is not limited.

  As shown in FIG. 8, when a plurality of imaging devices Cam 1,..., Cam 9 are arranged, the number of imaging devices corresponding to the same pixel (41 in FIG. It becomes a stand. Similar to the generation of a panoramic composite image, the imaging device is configured by creating four types of reference tables, that is, first to fourth reference tables, as reference tables configured with only pixels that can be substituted simultaneously into pixels of the composite video. The video composition processing can be executed in four steps regardless of the number of the video signal.

  FIG. 9 is a diagram showing an example of the overlapping area of the imaging ranges of the plurality of imaging devices Cam1, ..., Cam9 in the video synthesizing apparatus according to the second embodiment. In FIG. 9, in order to simplify the example of FIG. 8, the imaging range is drawn in a rectangle. FIG. 10 is a diagram showing a first reference table for an imaging range in the video compositing apparatus according to the second embodiment. 11 to 13 are a second reference table, a third reference table, and a fourth reference table, which are other reference tables for the imaging range (superimposed area) in the video compositing apparatus according to the second embodiment. FIG. FIGS. 9 to 13 illustrate rectangular images after projection conversion of the imaging devices Cam 1,..., Cam N in order to simplify the description, but the process is the same even if it has a trapezoidal shape or another shape. The first to fourth reference tables shown in FIGS. 9 to 13 are merely examples, and the shape and number of reference tables are not limited to the examples of FIGS. 9 to 13.

[Video input processing]
When the video reception unit 4 completes acquisition of one frame of input video of the imaging devices Cam1, ..., CamN, the input video data is transferred from the main memory 11 to the video processing memory 14.

[Video composition processing]
The pixel values of the input video transferred to the video processing memory 14 are substituted into the composite video using the first to fourth reference tables. The procedure is shown below.

The video processing processor 13 of the video processing unit 6 executes the following operations in parallel.
<11> In the first process, the video processing processor 13 detects the camera number i corresponding to each pixel (x_synth, y_synth) in the composite video from the first reference table shown in FIG. The corresponding pixel position (x_cami, y_cami) in, and the weighting coefficient α are extracted.
<12> In the second process, the video processing processor 13 refers to the pixel value of the pixel (x_cami, y_cami) of the input video of the imaging device Cami of the camera number i on the video processing memory 14 and weights this pixel value. The coefficient is multiplied and attached to the pixel of the composite video (x_synth, y_synth) on the video processing memory 14.
<13> In the third process, the video processing processor 13 executes the same process as the eleventh process and the twelfth process on the second reference table.
<14> In the fourth process, the video processing processor 13 executes the same process as the eleventh process and the twelfth process on the third reference table.
<15> In the fifth process, the video processing processor 13 executes the same process as the eleventh process and the twelfth process on the fourth reference table.
Further, the processing procedure of the entire video processing unit is the same as that of FIG. Further, the operation of the display processing unit is also the same as that of the first embodiment.

<2-3> Effects According to the video compositing apparatus and the video compositing method according to the second embodiment, the decoding load of input video data increases according to the number of imaging devices Cam1, ..., CamN. ,... The load of the video synthesis processing of the video acquired by CamN hardly increases.

  Further, in the video combining apparatus and the video combining method according to the second embodiment, attention is paid to processing waiting between videos of imaging devices having overlapping regions adjacent to each other, and only data that can simultaneously substitute the first reference table is used. By doing this, it is possible to realize the video composition processing only with the maximum number of steps of the imaging devices Cam1, ..., CamN related to the overlapping area. As a result, even if the number of imaging devices Cam1, ..., CamN increases, it becomes possible to create a composite image in the same processing time. For example, in the case of a bird's eye synthetic video, the process can be performed in four steps.

  In the case of the bird's-eye synthetic image, when the individual cameras are arranged as shown in FIG. 7, the processing can be performed in up to four steps.

  DESCRIPTION OF SYMBOLS 1 video synthesizing device, 2 display device, 4 video receiving unit, 5 parameter input unit, 6 video processing unit, 6a storage unit, 7 display processing unit, 10 main processor, 11 main memory, 12 auxiliary memory, 13 video processing processor, 14 image processing memory, 15 input interfaces, 16 file interfaces, 17 display interfaces, 18 image input interfaces, Cam1, ..., Cami, ..., CamN Imaging devices (cameras).

Claims (4)

A video combining apparatus that generates one composite video from a plurality of videos acquired by a plurality of imaging devices, the video combining apparatus comprising:
A video receiving unit that receives the plurality of videos;
A parameter input unit to which camera parameters of the plurality of imaging devices are input;
A video processing unit that generates the composite video from the plurality of videos;
Equipped with
The video processing unit
First imaging device identification information for identifying a corresponding imaging device among the plurality of imaging devices for each pixel of the composite video using the camera parameters input in advance, and the first imaging device identification Creating a lookup table including corresponding first pixel locations in the imaging device identified by the information and first weighted coefficients at the corresponding first pixel locations;
With reference to the reference table, for each pixel of the composite video, the first weighted coefficient is applied to the pixel value of the corresponding first pixel position in the imaging device specified by the first imaging device specification information. The composite image is generated by substituting the first value obtained by multiplication .
The video processing unit
Among the plurality of imaging ranges imaged by the plurality of imaging devices, one of the imaging devices having a superimposed region which is an overlapping imaging range is specified for each pixel of the composite video using the camera parameter Second imaging device identification information, a second pixel position corresponding to the overlapping region in the imaging device identified by the second imaging device identification information, and a second pixel position corresponding to the overlapping region Create another lookup table that contains the second weighting factor,
In the image pickup apparatus specified by the second image pickup apparatus specifying information for each pixel corresponding to the second pixel position of the superimposed area among the pixels of the composite video with reference to the other reference table The second value obtained by multiplying the second weighted coefficient by the pixel value of the corresponding pixel position is substituted to perform blending processing, thereby corresponding to the superimposed region of the composite video An image synthesizing apparatus characterized by generating a part . Of the plurality of imaging ranges imaged by the plurality of imaging devices, imaging ranges to be superimposed on each other are superimposed in the left-right direction, and the composite video is a panoramic video,
The video compositing apparatus according to claim 1 , wherein the other reference table is one table. Of the plurality of imaging ranges imaged by the plurality of imaging devices, imaging ranges to be superimposed on each other are superimposed in the vertical and horizontal directions, and the composite video is a bird's-eye view video,
The video compositing apparatus according to claim 1 , wherein the other reference table is three tables. A video combining method for generating one composite video from a plurality of videos acquired by a plurality of imaging devices, comprising:
First imaging device identification information for identifying a corresponding imaging device of the plurality of imaging devices for each pixel of the composite image using camera parameters input in advance for the plurality of imaging devices; Creating a first reference table including a corresponding first pixel position in the image pickup device specified by the image pickup device identification information of 1 and a first weighted coefficient at the corresponding first pixel position; ,
With reference to the first reference table, the first weight is assigned to the pixel value of the corresponding first pixel position in the imaging device identified by the first imaging device identification information for each pixel of the composite video Generating the composite image by substituting a first value obtained by multiplying an attached coefficient ;
Among the plurality of imaging ranges imaged by the plurality of imaging devices, one of the imaging devices having a superimposed region which is an overlapping imaging range is specified for each pixel of the composite video using the camera parameter Second imaging device identification information, a second pixel position corresponding to the overlapping region in the imaging device identified by the second imaging device identification information, and a second pixel position corresponding to the overlapping region Creating a second lookup table including a second weighting factor;
An imaging device specified by the second imaging device specifying information for each pixel corresponding to a second pixel position of the overlapping area among the pixels of the composite video with reference to the second reference table The second value obtained by multiplying the pixel value of the corresponding pixel position in the second by the second weighting factor is substituted for the blending processing, thereby corresponding to the superimposed region in the composite video. And V. generating an image portion.

Images