WO2022264418A1

WO2022264418A1 - Video compositing system, video compositing method, and video compositing program

Info

Publication number: WO2022264418A1
Application number: PCT/JP2021/023247
Authority: WO
Inventors: 広夢宮下; 真二深津; 英一郎松本; 麻衣子井元
Original assignee: 日本電信電話株式会社
Priority date: 2021-06-18
Filing date: 2021-06-18
Publication date: 2022-12-22

Abstract

A video compositing system according to one aspect of the present invention composites a first video and a second video having partially overlapping imaging regions, the system comprising: an acquisition unit that acquires a first image included in the first video and a second image included in the second video; a compositing unit that composites the first image and the second image to form a composite image; an analysis unit that analyzes the first image and the second image to generate correction information for correcting the composite image; a correction unit that corrects the composite image by using the correction information and an overlap section image, which is one section of the first image overlapping with one section of the second image; and an output unit that outputs the corrected composite image.

Description

Video Synthesis System, Video Synthesis Method, and Video Synthesis Program

The present invention relates to technology for synthesizing multiple videos.

As an image synthesis technology, a method is known in which images captured by multiple camera devices with partial overlap are input and combined to output a wide-angle panoramic image. On the other hand, methods have been proposed for analyzing video and correcting or processing the video based on the analysis results.

Patent document 1 analyzes input images in which part of the shooting area overlaps, detects an object in each input image, and integrates the detection results of the object in the overlapping area between the input images, so that object tracking information is displayed on the panoramic image. is disclosed.

Japanese Patent Application Laid-Open No. 2020-182100

Patent Document 1 proposes a video effect in which a synthesized image and object tracking information are output separately, and the object tracking information is overlaid on the synthesized image in subsequent processing. With the technique disclosed in Patent Document 1, it is possible to change the entirety of the composite video as a video effect, but it is impossible to apply the video effect in units of the original video (video that is input to the combining process). Can not. For example, when synthesizing a first input image and a second input image to generate a panoramic image, the first input image and the second input image are blended in the process. It is not possible to implement change processing such as correcting the luminance of only the pixels corresponding to the first input image.

In other words, when performing analysis processing and synthesis processing of input video in parallel, and performing processing for changing to a synthesized image based on information obtained by the analysis processing, the content of the change processing is controlled for each input video. can't

On the other hand, in the sequential procedure of analysis processing, modification processing, and subsequent synthesis processing, the amount of delay increases because parallel processing is not possible, and the real-time performance of video viewing is significantly reduced.

The purpose of the present invention is to provide a technology that ensures real-time video viewing and controls the process of changing to a synthesized video for each input video.

A video compositing system according to an aspect of the present invention is a video compositing system that synthesizes a first video and a second video in which photographing areas partially overlap, wherein a first image included in the first video is and an acquisition unit that acquires a second image included in the second video, a combining unit that combines the first image and the second image to generate a combined image, the first image and an analysis unit that analyzes the second image and generates correction information for correcting the synthesized image; an overlapping partial image that is a portion of the first image that overlaps a portion of the second image; and the correction. and an output unit for outputting the corrected synthetic image.

According to the present invention, a technology is provided that ensures real-time video viewing and controls the process of changing to a synthesized video for each input video.

FIG. 1 is a block diagram showing a video synthesizing system according to one embodiment of the present invention. FIG. 2 is a diagram for explaining processing in the image synthesizing unit shown in FIG. FIG. 3 is a diagram for explaining processing in the image correction unit shown in FIG. 1; FIG. 4 is a block diagram showing the hardware configuration of a computer according to one embodiment of the invention. FIG. 5 is a flow chart illustrating a video composition method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

In the embodiment described here, an example of panorama video synthesis in which a panorama video is generated by synthesizing two videos (moving images) whose shooting areas partially overlap will be taken. In panorama image synthesis, two images with partially overlapping photographing areas are input, and the images are combined so that the overlapping portions overlap. Note that it is also possible to generate a panorama video by synthesizing three or more videos.

For panorama video synthesis, a deformation matrix for adjusting the position of the images so that there is no subject shift between images, and an alpha value for seamlessly combining the images are required. Here, it is assumed that these synthetic parameters are determined in advance by general extraction and comparison of local feature amounts or by manual setting by an operator.

[composition]
FIG. 1 schematically shows a configuration example of a video synthesizing system 100 according to one embodiment of the present invention. As shown in FIG. 1, the video synthesizing system 100 includes a transmission server 110 and a reception server 120 as information processing devices. Sending server 110 is communicatively connected to receiving server 120 . For example, the sending server 110 is connected to the receiving server 120 by a video transmission cable or an IP transmission network.

The transmission server 110 is connected to the

imaging devices

101 and 102 that capture images, and receives images from the

imaging devices

101 and 102 . As the

imaging devices

101 and 102, for example, video cameras that output video signals in real time can be used. A video player or the like may be used instead of the

imaging devices

101 and 102 . The

imaging devices

101 and 102 are arranged so that images with partially overlapping imaging areas can be obtained. The receiving server 120 is connected to the display device 103 . Display device 103 may be a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

The transmission server 110 includes an image acquisition unit 111, an image synthesis unit 112, an image encoding unit (also referred to as a compression unit) 113, an image transmission unit 114, an image analysis unit 115, and a correction information transmission unit 116.

The image acquisition unit 111 acquires two images whose imaging regions partially overlap. These images are referred to as input image A and input image B hereinafter. The input image A is a frame included in the video obtained by the imaging device 101 , and the input image B is the frame included in the video obtained by the imaging device 102 . Each of the

imaging devices

101 and 102 generates frames at a predetermined frame rate and sequentially transmits the frames to the transmission server 110 . The image acquisition unit 111 sequentially receives pairs of the input image A and the input image B from the

imaging devices

101 and 102 .

The image synthesizing unit 112 synthesizes the input image A and the input image B acquired by the image acquiring unit 111 to generate a synthesized image. Image composition processing will be described later.

The image encoding unit 113 compresses the synthesized image generated by the image synthesizing unit 112 in order to reduce the data amount of the synthesized image. Specifically, the image encoding unit 113 encodes the synthesized image to obtain encoded data. Furthermore, the image encoding unit 113 compresses the overlapping partial image in order to reduce the data amount of the overlapping partial image, which is the overlapping portion in one of the input image A and the input image B. FIG. In this embodiment, a portion of the input image A that overlaps a portion of the input image B is used as an overlapping partial image, and is referred to as an overlapping partial image A. FIG. Specifically, the image encoding unit 113 encodes the overlapping partial image to generate encoded data in the same manner as the synthesized image.

The image transmission unit 114 transmits the encoded data of the composite image and the overlapping partial image A obtained by the image encoding unit 113 to the receiving server 120 . In order to reduce the amount of communication between the transmission server 110 and the reception server 120 , the composite image and the overlapping partial image A are encoded and transmitted from the transmission server 110 to the reception server 120 .

The image analysis unit 115 analyzes the input image A and the input image B acquired by the image acquisition unit 111 and generates correction information for correcting the synthesized image generated by the image synthesis unit 112 . Image analysis processing and correction information will be described later. The correction information transmission section 116 transmits the correction information generated by the image analysis section 115 to the reception server 120 .

The reception server 120 includes an image reception unit 121 , an image decoding unit (also referred to as a restoration unit) 122 , a correction information reception unit 123 , an image correction unit 124 and an output unit 125 .

The image receiving unit 121 receives encoded data of the composite image and the overlapping partial image A from the transmission server 110 . The image decoding unit 122 restores the combined image and the overlapping partial image A. Specifically, the image decoding unit 122 decodes the encoded data received by the image receiving unit 121 to obtain the composite image and the overlapping partial image A. The correction information receiving unit 123 receives correction information from the transmission server 110 . The image correcting unit 124 corrects the composite image obtained by the image decoding unit 122 using the overlapping partial image A obtained by the image decoding unit 122 and the correction information received by the correction information receiving unit 123. do. The correction processing will be described later. The output unit 125 outputs the synthesized image corrected by the image correction unit 124. FIG. For example, the output unit 125 displays the corrected composite image on the display device 103 .

Next, the transmission server 110 will be described in detail.

The image synthesis unit 112 receives the input image A and the input image B from the image acquisition unit 111 . The image synthesizing unit 112 deforms the input image A and the input image B according to the deformation matrix, which is the synthesizing parameter. In the input image A and the input image B illustrated in FIG. 2, the right portion of the input image A overlaps the left portion of the input image B. As shown in FIG. It is necessary to move the input image A and the input image B so that there is no deviation when the input image A and the input image B are superimposed. Input image A is subjected to a global left translation and input image B is subjected to a global right translation.

The image synthesizing unit 112 seamlessly combines the deformed input image A and the input image B by alpha blending to generate one synthesized image. In this embodiment, the alpha value referred to in alpha blending is determined for each coordinate of the synthesized image, and indicates the ratio at which the pixel values of the input image A and the pixel values of the input image B are mixed. Alpha values range from 0 to 1.

Let A _{(m, n)} be the pixel value of input image A at coordinates (m, n), B _{(m, n)} be the pixel value of input image B at coordinates (m, n), and alpha at coordinates (m, n). When the value is α _{(m, n)} , the pixel value C _{(m, n) of the composite image at coordinates (m, n)} is expressed as follows.

In the example shown in FIG. 2, the composite image has the same size as input image A and input image B. In this case, the left portion of input image A and the right portion of input image B are deleted during image synthesis. Alternatively, the composite image may be a different size than the input images A,B. For example, if the size of the input image A and the input image B is 1920×1080 and the size of the overlapping portion is 320×1080, a composite image with a size of 3520×1080 can be obtained. Overlaps are areas where pixels overlap in alpha blending, as shown in FIG.

The image encoding unit 113 encodes the composite image and the overlapping partial image A, and the image transmitting unit 114 transmits the coded data of the composite image and the overlapping partial image A to the reception server 120 . The image encoding unit 113 may make adjustments such as lowering the resolution and increasing the compression rate in order to further reduce the amount of data of the overlapping partial image A. FIG. For example, the image encoding unit 113 may lower the resolution of the overlapping partial image A and then encode the overlapping partial image A. The image encoding unit 113 may reduce the size of the overlapping partial image A to, for example, 1/2 or 1/4. Further, when the color space of the image is YCbCr, which is often used in video processing, only the Y signal, which is a luminance signal indicating luminance, is overlapped in order to reduce the amount of communication between the transmission server 110 and the reception server 120. The image A may be transmitted from the transmission server 110 to the reception server 120 . In this case, the image encoding unit 113 extracts the Y signal from the overlapping partial image A. FIG.

Note that if a sufficient transmission band is secured, an uncompressed transmission method that transmits the data of the composite image and the overlapping partial image A as they are may be used. In embodiments utilizing an uncompressed transmission scheme, the image encoder 113 is eliminated.

In this embodiment, it is assumed that the

imaging devices

101 and 102 have different image sensor sensitivity settings. Generally, in panorama video synthesis, it is recommended to unify camera parameters such as gain, shutter speed, and color temperature between imaging devices in order to eliminate luminance differences between input images. However, when using correction functions such as auto gain control and auto shutter, it is not possible to unify the sensitivity settings of image sensors in a plurality of imaging apparatuses.

Input image A and input image B shown in FIG. 2 were captured with different camera parameters, and input image A is relatively brighter than input image B. Therefore, in the synthesized image, the left side portion corresponding to the input image A is relatively brighter than the right side portion corresponding to the input image B. FIG.

In order to eliminate the brightness shading that occurs in the composite image as described above, the receiving server 120 corrects the composite image. The image analysis unit 115 generates information used for image correction performed in the receiving server 120 .

The image analysis unit 115 receives the input image A and the input image B from the image acquisition unit 111 and calculates the luminance difference between the input image A and the input image B. FIG. When the reference luminance is lum and the luminance average of the input image A is ave _A , the luminance difference dA of the input image _A is calculated as follows.

As one method of correcting the input image A so as to conform to the reference luminance, there is a method of multiplying all pixels of the input image A by a uniform correction value. Let this correction value be f _A. The image analysis unit 115 calculates the correction value _fA as follows.

The image analysis unit 115 performs the same calculation as described above on the input image _B to obtain the correction value fB. The correction value fA is a correction value for adjusting the input image _A to the reference luminance, and the correction value fB is a correction value for adjusting the input image _B to the reference luminance.

When input image A is corrected based on input image B, image analysis unit 115 sets the ratio of the average luminance of the overlapping portion of input image A to the average luminance of the overlapping portion of input image B as correction value f _A Ask as In this case, the image analysis unit 115 calculates only the correction value f _A and sets f _B =1.

The correction information transmission unit 116 transmits correction information including the correction values f _A and f _B obtained by the image analysis unit 115 to the receiving server 120 . Correction information is very small compared to image data. Therefore, the time required for transmission processing in the correction information transmission unit 116 is much shorter than the sum of the time required for encoding processing in the image encoding unit 113 and the time required for transmission processing in the image transmission unit 114. is assumed.

Next, the receiving server 120 will be described in detail.

The image receiving unit 121 receives encoded data of the composite image and the overlapping partial image A from the transmission server 110 . The image decoding unit 122 decodes the encoded data of the synthesized image to obtain the synthesized image, and decodes the encoded data of the overlapping partial image A to obtain the overlapping partial image A. If the overlapping partial image A has been reduced in the transmission server 110, the image decoding unit 122 enlarges the overlapping partial image A to its original size. The correction information receiving unit 123 receives correction information including the correction values f _A and f _B from the transmission server 110 .

The image correction unit 124 uses the overlapping partial image A and the correction values f _A and f _B to perform luminance correction on the composite image. Due to the luminance difference between the input image A and the input image B, luminance shading occurs on the synthesized image. Brightness correction is performed to eliminate brightness gradation on the composite image.

As shown in FIG. 3, the image correction unit 124 first divides the composite image into an area corresponding to the input image A, an overlapping area, and an area corresponding to the input image B. The overlapping area is an area between the area corresponding to the input image A and the area corresponding to the input image B, and corresponds to the portion where the input image A and the input image B are overlapped.

The image correction unit 124 performs luminance correction on the area corresponding to the input image _A based on the correction value fA. Specifically, the image correction unit 124 multiplies each pixel value in the region corresponding to the input image _A by the correction value fA. For each coordinate (m, n) in the region corresponding to the input image A, when the pixel value before correction is CA _{(m, n)} , the pixel value after correction DA _{(m, n)} is calculated as follows. be.

Further, the image correction unit 124 performs luminance correction on the area corresponding to the input image _B based on the correction value fB. Specifically, the image correction unit 124 multiplies each pixel value in the region corresponding to the input image _B by the correction value fB.

Further, the image correction unit 124 performs luminance correction on the overlapping area based on the overlapping partial image A and the correction values f _A and f _B . Specifically, for each coordinate (m, n) in the overlap region, Ap _{(m, n)} is the pixel value of the overlapping partial image A, CW (m, n) is the pixel value before correction, and CW _{(m, n)} is the pixel value after correction. When the value is DW _{(m, n)} , the image correction unit 124 calculates the post-correction pixel value DW _{(m, n)} as follows.

　Brightness correction for each region may be performed by parallel processing.

The image correction unit 124 integrates the luminance-corrected areas (specifically, the area corresponding to the input image A, the overlapping area, and the area corresponding to the input image B) to obtain an output image.

In the example described above, the image correction unit 124 divides the composite image into three regions, performs luminance correction on each region, and integrates the three luminance-corrected regions. Alternatively, the image correction unit 124 may virtually divide the composite image by limiting the range on the composite image. In this case, the integration process is omitted.

FIG. 4 schematically shows a hardware configuration example of a computer 400 according to one embodiment of the invention. A computer 400 shown in FIG. 4 corresponds to the transmission server 110 or the reception server 120 shown in FIG.

In the example shown in FIG. 4, the computer 400 comprises a processing circuit 401, a memory 402, an input/output interface 403, and a communication interface 404. Processing circuitry 401 is communicatively coupled to memory 402 , input/output interface 403 , and communication interface 404 .

When computer 400 is transmission server 110 , processing circuitry 401 is configured to perform the sequence of operations described with respect to transmission server 110 . When computer 400 is receiving server 120 , processing circuitry 401 is configured to perform the sequence of operations described with respect to receiving server 120 .

In one embodiment, processing circuitry 401 may include a general-purpose processor such as a CPU (central processing unit). Memory 402 may include random access memory (RAM) and storage devices. RAM includes volatile memory such as SDRAM. RAM is used by general-purpose processors as working memory. Storage devices include non-volatile memory such as flash memory. The storage device stores various data including a video synthesizing program. The video compositing program includes computer-executable instructions.

The general-purpose processor expands the video composition program stored in the storage device to RAM, and interprets and executes the video composition program. When the computer 400 is the transmission server 110 , the video synthesizing program, when executed by the general-purpose processor, causes the general-purpose processor to perform the series of processes described with respect to the transmission server 110 . When computer 400 is receiving server 120 , the video synthesizing program, when executed by a general-purpose processor, causes the general-purpose processor to perform a series of processes described with respect to receiving server 120 .

The program may be provided to the computer 400 while being stored in a computer-readable recording medium. In this case, the computer 400 has a drive for reading data from the recording medium and obtains the program from the recording medium. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. Also, the program may be distributed through a network. Specifically, the program may be stored in a server on a network, and computer 400 may download the program from the server.

In other embodiments, the processing circuit 401 may include a dedicated processor such as an ASIC (application specific integrated circuit) or FPGA (field programmable gate array). Memory 402 may store configuration data that define the operation of the dedicated processor. Memory 402 may be internal to a dedicated processor.

The input/output interface 403 is an interface for connecting peripheral devices. A communication interface 404 is an interface for communicating with an external device. When the computer 400 is the transmission server 110 , the processing circuit 401 includes a video capture card, receives images from the

imaging devices

101 and 102 via the input/output interface 403 , and synthesizes them to the reception server 120 via the communication interface 404 . The coded data of the image, the coded data of the overlapping partial image, and the correction information are transmitted. When the computer 400 is the receiving server 120, the processing circuit 401 receives the encoded data of the composite image, the encoded data of the overlapping partial image, and the correction information from the transmission server 110 via the communication interface 404, and receives the input/output interface. 403 to the display device 103 .

[motion]
The operation of the image synthesizing system 100 will be described.

FIG. 5 schematically shows an example of a video compositing method executed by the video compositing system 100. FIG. The flow shown in FIG. 5 is executed each time the video synthesizing system 100 acquires a video frame. The processing shown in steps S501 to S505 in FIG. 5 is executed by transmission server 110. FIG. The processes shown in steps S502 and S503 and the processes shown in steps S504 and S505 may be executed in parallel. The processing shown in steps S506 to S509 is executed by the reception server 120. FIG. The processing shown in step S506 and the processing shown in step S507 may be executed in parallel.

In step S<b>501 , the image acquisition unit 111 acquires the input image A from the imaging device 101 and acquires the input image B from the imaging device 102 .

In step S502, the image synthesizing unit 112 synthesizes the input image A and the input image B to generate a synthetic image. For example, the image synthesizing unit 112 transforms the input image A and the input image B according to the transformation matrix, and combines the transformed input image A and the input image B by alpha blending.

In step S<b>503 , the image encoding unit 113 encodes the synthesized image to obtain encoded data, and the image transmission unit 114 transmits the encoded data of the synthesized image to the receiving server 120 . Further, the image coding unit 113 reduces the resolution of the overlapping partial image A, specifically, the image coding unit 113 reduces the size of the overlapping partial image A. Subsequently, the image encoding unit 113 encodes the overlapping partial image A to obtain encoded data, and the image transmitting unit 114 transmits the encoded data of the overlapping partial image A to the receiving server 120 .

In step S504, the image analysis unit 115 compares the pixels included in the overlapping portion of the input image A with the pixels included in the overlapping portion of the input image B to determine the luminance difference of the input image A with respect to the input image B. calculate. For example, the image analysis unit 115 obtains the luminance difference of the input image A by dividing the average luminance of the overlapping portion of the input image A by the average luminance of the overlapping portion of the input image B. FIG.

In step S<b>505 , the image analysis unit 115 generates correction information from the luminance difference of the input image A, and the correction information transmission unit 116 transmits the correction information to the receiving server 120 . For example, the image analysis unit 115 obtains the reciprocal of the luminance difference of the input image _A as the correction value fA. The correction information transmission unit 116 transmits the correction value f _A to the reception server 120 as correction information.

In step S506, the image receiving unit 121 receives the encoded data of the composite image and the overlapping partial image A from the transmission server 110, and the image decoding unit 122 decodes the encoded data to generate the composite image and the overlapping partial image A. obtain. The image decoding unit 122 restores the overlapping partial image A to its original size. In step S<b>507 , the correction information receiving unit 123 receives correction information including the correction value f _A from the transmission server 110 .

In step S508, the image correction unit 124 divides the composite image into an area corresponding to the input image A, an area corresponding to the input image B, and an overlapping area, and performs luminance correction on these areas based on the correction information. . In the example described here, input image B is used as a reference (f _B =1), and luminance correction for the region corresponding to input image B need not be performed. For example, the image correction unit 124 performs luminance correction on the region corresponding to the input image A using the correction value f _A according to the above equation (4), and corrects the correction value f _A and the overlapping partial image A according to the above equation (5). is used to perform luminance correction for overlapping regions.

In step S509, the image correction unit 124 integrates the area corresponding to the input image A, the area corresponding to the input image B, and the overlapping area to generate an output image. to display.

The video synthesizing system 100 executes the above-described flow for each frame, thereby obtaining a synthesized video including multiple synthesized images. As a result, the composite image is displayed on the display device 103 in real time.

[effect]
The image synthesizing system 100 acquires an input image A that is a frame included in the image obtained by the imaging device 101 and an input image B that is a frame included in the image obtained by the imaging device 102, and generates the input image A and the input image B. to generate a composite image, analyze the input image A and the input image B to generate correction information, and correct the composite image using the overlapping portion image that is the overlapping portion of the input image A and the correction information and output the corrected composite image.

By using the overlapping partial image for the correction processing of the composite image, it is possible to control the correction processing of the composite image for each input image. In overlapping regions in the composite image, correction is performed using overlapping subimage A according to equation (5) above. Equation (5) above can be modified as follows.

Here, Bp _{(m, n)} represents the pixel value of the overlapping partial image B. In this manner, correction processing is performed for each input image in the overlap region in the composite image.

In general panoramic video synthesis, it is assumed that the sensitivity settings of the image sensors in the imaging device are manually unified, and it is almost impossible to correct or process each input image individually. No. Even if correction or processing is required, it is basically performed as preprocessing for image synthesis. Specifically, panorama video synthesis is performed in a sequence of analysis processing, correction processing, and synthesis processing.

In the above configuration, correction processing is performed on the composite image. This enables the synthesizing process and the analyzing process to be performed in parallel, thereby shortening the processing delay.

Therefore, the video synthesizing system 100 having the above configuration can control the process of changing to a synthetic video for each input video while ensuring real-time video viewing.

As mentioned above, in general panoramic video synthesis, camera parameters are fixed manually, and automatic gain control is not set. However, it may not be possible to shoot with appropriate brightness due to changes in the subject or lighting due to camerawork or changes in time. Alternatively, if auto-correction is applied, equipment is required to distribute settings to match brightness across all cameras, increasing cost.

The video synthesis system 100 generates correction information including a correction value f _A for adjusting the input image A to the reference luminance and a correction value f _B for adjusting the input image B to the reference luminance, and the correction values f _A , Using fB and the overlapping subimage _A , perform brightness correction on the composite image. Specifically, the video synthesizing system 100 uses the correction value f _A to perform luminance correction on the region corresponding to the input image A in the synthesized image, and corrects the correction values f _A and f _B and the overlapping partial image A to is used to perform luminance correction on the region corresponding to the portion where the first image and the second image are superimposed in the composite image, and the correction value f _B is used to correct the input image B in the composite image. Perform brightness correction for the corresponding region. Thereby, it is possible to perform luminance correction for each of the input images A and B on the composite image. As a result, even when automatic correction is applied, luminance unevenness that occurs in the composite image can be eliminated without a device for distributing settings.

The video synthesizing system 100 includes a transmission server 110 and a reception server 120 connected in series, and a series of processes are executed by the transmission server 110 and the reception server 120 . Basically, the transmission server 110 is installed at the shooting base, and the reception server 120 is installed at the projection base. There is no guarantee that the network bandwidth from the shooting base to the projection base will be abundant, and it is desirable that the transmission capacity be as small as possible.

The transmission server 110 may compress the overlapping partial image A and transmit it to the reception server 120 . Transmission server 110 may reduce the resolution of overlapping partial image A in order to compress overlapping partial image A. FIG. For example, the transmission server 110 reduces the size of the overlapping partial image A to 1/2 or 1/4. By compressing the overlapping partial image A, the transmission capacity can be reduced. As a result, the transmission band can be saved.

The overlapping partial image is only referenced in the correction process, and is only indirectly involved in the quality of the final output image. In an example where the size (number of pixels) of the input image is 1920×1080 and the size of the overlapping partial image is 320×1080, the size of the overlapping partial image is reduced to 1/4 (80 A test was conducted to compare the output image with the case where the image was reduced to 270 × 270), but there was no noticeable deterioration in viewing.

[Modification]
In the embodiment described above, the video compositing system 100 includes two information processing devices, specifically the transmission server 110 and the reception server 120 . In other embodiments, video composition system 100 may be implemented by a single information processing device. In this case, the image encoding unit 113, the image transmission unit 114, the correction information transmission unit 116, the image reception unit 121, the image decoding unit 122, and the correction information reception unit 123 may be deleted.

A processing unit may be provided between the image correction unit 124 and the output unit 125 to perform additional correction or processing on the synthesized image. Alternatively, such a processing unit may be provided in a further server which is connected to the receiving server 120 via the video transmission network. In that case, the output unit 125 transmits the composite image to another server.

It should be noted that the present invention is not limited to the above-described embodiments, and can be variously modified in the implementation stage without departing from the gist of the present invention. Further, each embodiment may be implemented in combination as appropriate, in which case the combined effect can be obtained. Furthermore, various inventions are included in the above embodiments, and various inventions can be extracted by combinations selected from the disclosed plurality of components. For example, even if some components are deleted from all the components shown in the embodiment, if the problem can be solved and effects can be obtained, the configuration in which these components are deleted can be extracted as an invention.

DESCRIPTION OF SYMBOLS 100... Video synthesis system 101... Imaging device 102... Imaging device 103... Display device 110... Transmission server 111... Image acquisition part 112... Image synthesis part 113... Image encoding part 114... Image transmission part 115... Image analysis part 116... Correction Information transmission unit 120 Reception server 121 Image reception unit 122 Image decoding unit 123 Correction information reception unit 124 Image correction unit 125 Output unit 400 Computer 401 Processing circuit 402 Memory 403 Input/output interface 404 Communication interface

Claims

A video synthesizing system for synthesizing a first video and a second video in which shooting areas partially overlap,
an acquisition unit that acquires a first image included in the first image and a second image included in the second image;
a synthesizing unit that synthesizes the first image and the second image to generate a synthesized image;
an analysis unit that analyzes the first image and the second image to generate correction information for correcting the composite image;
a correction unit that corrects the composite image using an overlapping partial image that is a portion of the first image that overlaps a portion of the second image and the correction information;
an output unit that outputs the corrected composite image;
A video compositing system with
A process of synthesizing the first image and the second image in the synthesizing unit and a process of analyzing the first image and the second image in the analyzing unit are performed in parallel;
The image synthesizing system according to claim 1.
The analysis unit generates the correction information including a correction value for adjusting the first image to a reference luminance,
The correction unit uses the correction value and the overlapping partial image to perform luminance correction on the synthesized image.
3. The video synthesizing system according to claim 1 or 2.
The correction unit uses the correction value to perform luminance correction on a region corresponding to the first image in the synthesized image, and uses the correction value and the overlapping partial image to perform luminance correction in the synthesized image. performing luminance correction on a region corresponding to a portion where the first image and the second image are superimposed;
4. The video synthesizing system according to claim 3.
Equipped with a first information processing device and a second information processing device,
The first information processing device is
the acquisition unit;
the analysis unit;
the correction unit;
a compression unit that compresses the overlapping partial image;
a transmitting unit configured to transmit the combined image, the compressed overlapping partial image, and the correction information to the second information processing device;
including
The second information processing device is
a receiving unit that receives the composite image, the compressed overlapping partial image, and the correction information from the first information processing device;
a restoring unit that restores the overlapping partial image from the compressed overlapping partial image;
the correction unit;
including,
5. The video synthesizing system according to any one of claims 1 to 4.
compressing the overlapping partial image includes reducing the resolution of the overlapping partial image;
6. The video synthesizing system according to claim 5.
A video synthesis method for synthesizing a first video and a second video in which shooting areas partially overlap,
obtaining a first image included in the first image and a second image included in the second image;
synthesizing the first image and the second image to generate a synthesized image;
analyzing the first image and the second image to generate correction information for correcting the composite image;
correcting the composite image using an overlapping partial image that is a portion of the first image that overlaps a portion of the second image and the correction information;
outputting the corrected composite image;
A video composition method comprising:
A video synthesizing program for causing a computer to function as each unit included in the video synthesizing system according to any one of claims 1 to 5.