JP2024051665A - Moving image synthesis system, moving image synthesis method, and program - Google Patents

Abstract

To generate a natural synthetic moving image including a plurality of subjects existing in different spaces.SOLUTION: A moving image synthesis system 30 includes a moving image acquisition part 41 for acquiring a first moving image captured by a first imaging apparatus in a first real space and a second moving image captured by a second imaging apparatus in a second real space and an image processing part 42 for executing synthesis processing for generating a synthetic moving image V including the image of a first subject in the first moving image and the image of a second subject in the second moving image. The synthesis processing includes adjustment processing for adjusting the image of the first subject and the image of the second subject according to the first imaging distance of the first imaging apparatus and the second imaging distance of the second imaging apparatus.SELECTED DRAWING: Figure 4

Description

This disclosure relates to technology for combining multiple videos.

Technology for combining multiple videos recorded separately has been proposed in the past. For example, Patent Document 1 discloses a technology for capturing multiple videos using different cameras, and combining a video of a user cut out from each of the multiple videos with a specified background video.

Japanese Patent No. 6627861

In the technology of Patent Document 1, the imaging distance between the subject, the user, and the camera may differ for each camera. Therefore, Patent Document 1 has a problem in that, for example, an unnatural video is generated that does not properly reflect the difference in imaging distance for each subject. In consideration of the above circumstances, one aspect of the present disclosure aims to generate a natural composite video that includes multiple subjects located in different spaces.

In order to solve the above problems, a video compositing system according to one aspect of the present disclosure includes a video acquisition unit that acquires a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space, and an image processing unit that executes a compositing process to generate a composite video including an image of a first subject in the first video and an image of a second subject in the second video, the compositing process including an adjustment process that adjusts the image of the first subject and the image of the second subject according to a first imaging distance of the first imaging device and a second imaging distance of the second imaging device.

A video compositing method according to one aspect of the present disclosure acquires a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space, and performs a compositing process to generate a composite video including an image of a first subject in the first video and an image of a second subject in the second video, the compositing process including an adjustment process to adjust the image of the first subject and the image of the second subject according to a first imaging distance of the first imaging device and a second imaging distance of the second imaging device.

A program according to one aspect of the present disclosure causes a computer system to function as a video acquisition unit that acquires a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space, and an image processing unit that executes a synthesis process to generate a synthetic video including an image of a first subject in the first video and an image of a second subject in the second video, the synthesis process including an adjustment process that adjusts the image of the first subject and the image of the second subject according to a first imaging distance of the first imaging device and a second imaging distance of the second imaging device.

1 is a block diagram illustrating a configuration of a moving image recording system according to a first embodiment. 3A to 3C are schematic diagrams of moving images captured by the imaging devices. FIG. 1 is a block diagram illustrating a configuration of a moving image compositing system. FIG. 1 is a block diagram illustrating a functional configuration of a moving image compositing system. 4 is a block diagram illustrating a specific configuration of an image processing unit. FIG. FIG. 13 is an explanatory diagram of blurring processing. 11 is an explanatory diagram showing the relationship between an imaging distance and a blur amount. FIG. 1 is a schematic diagram of a composite video. FIG. 1 is a schematic diagram of a composite video. FIG. 1 is a schematic diagram of a composite video. 1 is a flowchart of the operation of the moving image synthesis system. 11 is a flowchart illustrating a specific procedure of a synthesis process. FIG. 13 is a block diagram of an image processing unit in the third embodiment. FIG. 13 is an explanatory diagram of a virtual moving image in the third embodiment. 13 is a flowchart of a synthesis process according to a third embodiment. 13A to 13C are schematic diagrams of a composite moving image in the third embodiment. FIG. 13 is a block diagram illustrating a functional configuration of a moving image compositing system according to a fourth embodiment. 13 is an explanatory diagram showing the relationship between the imaging distance and the blur amount in a modified example. 13 is an explanatory diagram showing the relationship between the imaging distance and the blur amount in a modified example. 13 is an explanatory diagram showing the relationship between the imaging distance and the blur amount in a modified example. 13 is an explanatory diagram showing the relationship between the imaging distance and the blur amount in a modified example.

The embodiments of the present disclosure will be described with reference to the drawings. The embodiments described below include various technically suitable limitations. The scope of the present disclosure is not limited to the embodiments exemplified below.

[First embodiment]
1 is a block diagram illustrating the configuration of a video recording system 100 in the first embodiment. The video recording system 100 is a computer system for producing distribution content C. The distribution content C is information distributed to the terminal device 200 for viewing by a user of the terminal device 200. The distribution content C is composed of, for example, video and audio of a competitive event (esports) in which multiple competitors compete against each other in a video game.

The terminal device 200 is an information device such as a smartphone, a tablet terminal, or a personal computer. The distribution content C is distributed from the video recording system 100 to the terminal device 200 via a communication network such as the Internet. Note that, for convenience, only one terminal device 200 is shown in FIG. 1, but in reality, the distribution content C is distributed to multiple terminal devices 200.

The video recording system 100 comprises multiple recording systems 20-1 to 20-3 and a video compositing system 30. Each recording system 20-n (n = 1 to 3) communicates with the video compositing system 30 via a communication network. Each of the multiple recording systems 20-1 to 20-3 is installed in a different recording studio Rn. Each recording studio Rn is a different real space. Each recording studio Rn is located, for example, at a remote location from each other.

In each recording studio Rn, there is a subject Qn to be recorded. The subject Qn is a performer in the distribution content C, such as a contestant or commentator in a competitive event. The background of the subject Qn in each recording studio Rn is a specific color, such as a green back or a blue back.

As illustrated in FIG. 1, each recording system 20-n includes an imaging device 21-n, a sound collecting device 22-n, and a communication device 23-n. Note that one or both of the sound collecting device 22-n and the communication device 23-n may be mounted on the imaging device 21-n.

The imaging device 21-n is a camera that generates a video Vn in a recording studio Rn. Each imaging device 21-n includes, for example, an optical system such as a photographing lens, an imaging element that receives incident light from the optical system, and a processing circuit that generates data for the video Vn according to the amount of light received by the imaging element. The format of the data representing the video Vn is arbitrary.

Figure 2 is a schematic diagram of the video Vn generated by each imaging device 21-n. The imaging device 21-1 records the video V1 by imaging the subject Q1 in the recording studio R1. Similarly, the imaging device 21-2 records the video V2 by imaging the subject Q2 in the recording studio R2, and the imaging device 21-3 records the video V3 by imaging the subject Q3 in the recording studio R3. Each imaging device 21-n captures the subject Qn in a pan-focus state in which the subject is substantially in focus over a wide range in the optical axis direction. Therefore, each subject image Gn does not substantially suffer from optical blurring due to being away from the focal plane of the imaging device 21. In other words, the image of the subject Qn in each video Vn (hereinafter referred to as the "subject image Gn") has a clear contour or boundary.

As illustrated in FIG. 1, the imaging distance Dn differs for each imaging device 21-n. The imaging distance Dn is the distance between the imaging device 21-n and the subject Qn. The imaging distance Dn is the distance between the surface of the photographing lens or the imaging surface of the imaging element and the subject Qn. In the following explanation, it is assumed that the imaging distance D2 is greater than the imaging distance D1 and the imaging distance D3 is greater than the imaging distance D2 (D1<D2<D3). On the other hand, imaging conditions other than the imaging distance Dn, such as the focal length or the aperture value, are common to the multiple imaging devices 21-1 to 21-3. Therefore, even if the actual heights of the multiple subjects Q1 to Q3 are the same, the size of the subject image Gn in each video Vn differs for each video Vn according to the imaging distance Dn, as illustrated in FIG. 2.

The sound collection device 22-n in FIG. 1 is a microphone that collects sound An in a recording studio Rn. Sound An is, for example, sound produced by a subject Qn in the recording studio Rn. Specifically, data representing the waveform of sound An is generated by the sound collection device 22-n. The format of the data representing sound An is arbitrary.

The communication device 23-n communicates with the video compositing system 30 via a communication network such as the Internet. The communication path between the communication device 23-n and the video compositing system 30 is composed of a wired section or a wireless section. Specifically, the communication device 23-n transmits material data Mn to the video compositing system 30. The material data Mn is data representing the video Vn captured by the imaging device 21-n and the sound An collected by the sound collection device 22-n.

Figure 3 is a block diagram illustrating the configuration of a video compositing system 30. The video compositing system 30 is a computer system that generates and distributes distribution content C. The video compositing system 30 is realized by an information device such as a smartphone, a tablet terminal, or a personal computer. Note that the video compositing system 30 may be realized by a general-purpose information device such as those exemplified above, or by a video device dedicated to generating distribution content C.

The video compositing system 30 includes a control device 31, a storage device 32, a communication device 33, an operation device 34, and a playback device 35. Note that the video compositing system 30 may be realized as a single device, or may be realized as multiple devices configured separately from each other.

The control device 31 is a single or multiple processors that control each element of the video synthesis system 30. Specifically, the control device 31 is configured with one or more types of processors, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

The storage device 32 is a single or multiple memories that store the programs executed by the control device 31 and various data used by the control device 31. For example, a well-known recording medium such as a semiconductor recording medium or a magnetic recording medium, or a combination of multiple types of recording media, is used as the storage device 32. Note that, for example, a portable recording medium that is detachable from the video synthesis system 30, or a recording medium that the control device 31 can access via a communication network (e.g., cloud storage) may be used as the storage device 32.

The communication device 33 communicates with the terminal device 200 and each recording system 20-n via a communication network. For example, the communication device 33 receives material data Mn transmitted by each recording system 20-n. The communication device 33 also transmits distribution content C to the terminal device 200. Note that the distribution of the distribution content C to the terminal device 200 may be performed by a distribution system separate from the video recording system 100. For example, the distribution content C transmitted from the video synthesis system 30 may be held in the distribution system, and the distribution content C may be distributed from the distribution system to the terminal device 200. Furthermore, the distribution content C may be recorded in a recording medium such as the storage device 32, in addition to being distributed to the terminal device 200. In other words, distribution to the terminal device 200 may be omitted.

The operation device 34 is an input device that accepts instructions from a user of the video recording system 100. The user of the video recording system 100 is, for example, the creator of the distribution content C. For example, a control operated by the user or a touch panel that detects contact by the user is used as the operation device 34. Note that the operation device 34, which is separate from the video compositing system 30, may be connected to the video compositing system 30 by wire or wirelessly.

The playback device 35 plays the distribution content C under the control of the control device 31. In parallel with the recording in each recording studio Rn, the playback device 35 plays the distribution content C. Users of the video recording system 100 can check the distribution content C. Specifically, the playback device 35 includes a display device and a sound emitting device. The display device displays the video of the distribution content C (synthetic video V described below). For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device. The sound emitting device emits the sound of the distribution content C (synthetic sound A described below). For example, a speaker or a headphone is used as the sound emitting device. Note that the playback device 35, which is separate from the video compositing system 30, may be connected to the video compositing system 30 by wire or wirelessly.

Figure 4 is a block diagram illustrating an example of the functional configuration of the video synthesis system 30. The control device 31 executes a program stored in the storage device 32 to realize multiple functions (video acquisition unit 41, image processing unit 42, audio processing unit 43, and output processing unit 44) for generating the distribution content C. Note that the functions of the control device 31 may be realized by multiple devices configured separately from each other. Some or all of the functions of the control device 31 may be realized by a dedicated electronic circuit.

The video acquisition unit 41 acquires multiple pieces of material data M1 to M3. Specifically, the video acquisition unit 41 receives material data Mn, including video Vn and audio An recorded by each recording system 20-n, from the recording system 20-n via the communication device 33. That is, the video acquisition unit 41 acquires multiple videos V1 to V3 and multiple audios A1 to A3. Each piece of material data Mn is data that will become the material for the distribution content C.

The image processing unit 42 generates a composite video V by executing a compositing process. The compositing process is a process of compositing multiple videos V1 to V3 acquired by the video acquisition unit 41. That is, as illustrated in Figures 8 to 10, the composite video V is a video that includes subject image G1 of video V1, subject image G2 of video V2, and subject image G3 of video V3. That is, the compositing process is an image process that combines multiple subject images G1 to G3.

The audio processing unit 43 in FIG. 4 generates synthetic audio A by mixing multiple audios A1 to A3. The mixing ratio of each audio A1 to A3 is set, for example, according to an instruction from the user via the operation device 34. The output processing unit 44 generates distribution content C including the synthetic video V and the synthetic audio A. The output processing unit 44 in the first embodiment distributes the distribution content C from the communication device 33 to the terminal device 200.

Figure 5 is a block diagram illustrating a specific configuration of the image processing unit 42. As illustrated in Figure 5, the image processing unit 42 in the first embodiment includes a distance identification unit 421, a subject extraction unit 422, a subject selection unit 423, and an image adjustment unit 424.

The distance determination unit 421 determines the imaging distance Dn for each of the multiple videos V1 to V3. As described above, the imaging distance Dn is the distance between the imaging device 21-n and the subject Qn.

The distance determination unit 421 in the first embodiment determines the imaging distance Dn by detecting a distance index in the video Vn. The distance index is a marker that is added to each subject Qn in advance to determine the imaging distance Dn. A distance index of a common size is added to multiple subjects Q1 to Q3. Therefore, there is a correlation in which the larger the imaging distance Dn, the smaller the size of the distance index in the video Vn. Using the above correlation, the distance determination unit 421 estimates the imaging distance Dn according to the size of the distance index in each video Vn. For example, the distance determination unit 421 estimates the imaging distance Dn by analyzing the video Vn so that the imaging distance Dn becomes a smaller value as the size of the distance index in the video Vn increases. Note that it is not necessarily necessary to directly assign the distance index to the subject Qn. For example, a distance index may be installed at a point in the recording studio Rn at a distance equivalent to the imaging distance Dn of the subject Qn.

The subject extraction unit 422 extracts a subject image Gn from each of the multiple videos V1 to V3. Specifically, the subject extraction unit 422 extracts the subject image Gn by removing an area corresponding to a background of a specific color (e.g., a green back or a blue back) from each video Vn. Note that in a form in which the video acquisition unit 41 acquires a video Vn from which the subject image Gn has been extracted, the subject extraction unit 422 may be omitted.

The subject selection unit 423 selects one of the multiple subjects Q1 to Q3 as the reference subject Qref. The reference subject Qref is a subject Qn among the multiple subjects Q1 to Q3 that the viewer of the distribution content C should pay attention to. The user specifies one of the multiple subjects Q1 to Q3 by operating the operation device 34 while watching the distribution content C played by the playback device 35. The subject selection unit 423 selects the subject Qn specified by the user by operating the operation device 34 among the multiple subjects Q1 to Q3 as the reference subject Qref. The selection of the reference subject Qref by the subject selection unit 423 can also be said as the selection of one of the multiple videos V1 to V3, or the selection of one of the multiple imaging devices 21-1 to 21-3.

The image adjustment unit 424 executes an adjustment process. The adjustment process is image processing that adjusts each of the multiple subject images G1 to G3 according to the imaging distance Dn. The adjustment process in the first embodiment includes blurring and superimposition.

[Blurring]
The blurring process is a processing process that blurs each object image Gn. That is, the blurring process changes the contour or boundary of the object image Gn to a vague state. Specifically, the blurring process is a filter process that replaces the pixel value of each pixel that constitutes the object image Gn with the average value (e.g., simple average or weighted average) of multiple pixel values within a predetermined range (hereinafter referred to as the "processing range") that includes the pixel. The processing range is, for example, a rectangular range centered on one pixel to be replaced. The larger the processing range, the greater the degree to which the object image Gn is blurred. The blurring process is an image process that simulates the blur that occurs optically in an object that is separated from the focal plane of the optical system in the imaging device.

Figure 6 is an explanatory diagram of the blurring process. In the blurring process, the blur amount Bn (B1 to B3) is controlled. The blur amount Bn is an image parameter that indicates the degree to which the subject image Gn is blurred. Specifically, the blur amount Bn is a non-negative value that specifies the size of the processing range in the blurring process. As illustrated in Figure 6, the larger the blur amount Bn, the larger the processing range becomes, and as a result, the degree to which the subject image Gn is blurred increases. A blur amount Bn of zero means that the subject image Gn is not blurred.

The image adjustment unit 424 in the first embodiment individually controls the blur amount Bn for each subject image Gn. Specifically, the image adjustment unit 424 adjusts the blur amount Bn of the subject image Gn according to the imaging distance Dn of each subject Qn. As can be understood from the above explanation, the blurring process is an example of a processing process that adjusts the image parameters (blur amount Bn) of each subject Qn according to each imaging distance Dn.

Figure 7 is an explanatory diagram showing the relationship between the imaging distance Dn and the blur amount Bn. The horizontal axis of Figure 7 is the imaging distance Dn, and the vertical axis is the blur amount Bn. The reference value Dref in Figure 7 is a numerical value that serves as a reference for the imaging distance Dn. Specifically, the image adjustment unit 424 sets the imaging distance Dn corresponding to the reference subject Qref selected by the subject selection unit 423 as the reference value Dref. As described above, the reference value Dref is the imaging distance Dn that corresponds to one of the multiple subjects Q1 to Q3 (reference subject Qref). The reference value Dref corresponds to the focal plane of the optical system in the imaging device.

As can be seen from FIG. 7, the image adjustment unit 424 sets the blur amount Bn to zero when the imaging distance Dn is a value within the focusing range P0. The focusing range P0 is a range that includes the reference value Dref. For example, a range of a certain width centered on the reference value Dref is set as the focusing range P0. The focusing range P0 corresponds to the depth of field that can be considered to be substantially in focus in a real imaging device.

As can be seen from FIG. 7, outside the focusing range P0, the image adjustment unit 424 sets the blur amount Bn of the subject image Gn to a larger value as the difference |Dref-Dn| between the reference value Dref and the imaging distance Dn of each subject Qn increases. Specifically, the blur amount Bn changes linearly with the imaging distance Dn.

Assuming distances Da1 and Da2 as the numerical values of the imaging distance Dn. Distances Da1 and Da2 are numerical values within a range Pa below the lower limit value ra of the focusing range P0. The difference between distance Da2 and the reference value Dref exceeds the difference between distance Da1 and the reference value Dref (|Dref-Da2|>|Dref-Da1|). When imaging distance Dn is distance Da1, image adjustment unit 424 sets blur amount Bn of subject image Gn to set value Ba1. On the other hand, when imaging distance Dn is distance Da2, image adjustment unit 424 sets blur amount Bn of subject image Gn to set value Ba2 that exceeds set value Ba1. Note that distance Da1 is an example of the "first distance" and distance Da2 is an example of the "second distance". Also, set value Ba1 is an example of the "first set value" and set value Ba2 is an example of the "second set value".

Similarly, distances Db1 and Db2 are assumed as the numerical values of the imaging distance Dn. Distances Db1 and Db2 are numerical values within a range Pb that exceeds the upper limit value rb of the focusing range P0. The difference between distance Db2 and the reference value Dref exceeds the difference between distance Db1 and the reference value Dref (|Dref-Db2|>|Dref-Db1|). When the imaging distance Dn is distance Db1, the image adjustment unit 424 sets the blur amount Bn of the subject image Gn to a set value Bb1. On the other hand, when the imaging distance Dn is distance Db2, the image adjustment unit 424 sets the blur amount Bn of the subject image Gn to a set value Bb2 that exceeds the set value Bb1. Note that distance Db1 is an example of the "first distance" and distance Db2 is an example of the "second distance". Also, set value Bb1 is an example of the "first set value" and set value Bb2 is an example of the "second set value".

Figures 8 to 10 are schematic diagrams of a composite video V that focuses on the blur amount Bn of each subject image Gn. Figures 8 to 10 show an example of a composite video V in which multiple subject images G1 to G3 are composited.

In FIG. 8, it is assumed that the subject Q2 is selected as the reference subject Qref. The imaging distance D2 of the subject Q2 is set to the reference value Dref, and as a result, the blur amount B2 of the subject image G2 is set to zero. That is, the contour or boundary of the subject image G2 is maintained in a clear state. On the other hand, the imaging distance D1 of the subject Q1 is below the lower limit ra of the focusing range P0. As a result of the blur amount B1 corresponding to the imaging distance D1 being applied to the blurring process of the subject image G1, the subject image G1 in the composite video V becomes a blurred image compared to the subject image G2. Similarly, the imaging distance D3 of the subject Q3 exceeds the upper limit rb of the focusing range P0. As a result of the blur amount B3 corresponding to the imaging distance D3 being applied to the blurring process of the subject image G3, the subject image G3 in the composite video V becomes a blurred image compared to the subject image G2. As described above, the composite video V in FIG. 8 is perceived as a video captured in a state where the subject Q2 is in focus. That is, subject Q1 is given a foreground blur, and subject Q3 is given a background blur. Therefore, viewers of distributed content C are likely to focus on subject Q2.

In FIG. 9, it is assumed that subject Q1 is selected as the reference subject Qref. Therefore, subject image G1 maintains a clear outline or boundary, while subject images G2 and G3 are blurred. Because imaging distance D3 exceeds imaging distance D2 within range Pb, the blur amount B3 of subject image G3 exceeds the blur amount B2 of subject image G2. In other words, subject image G3 is a blurred image compared to subject image G2. As described above, subject image G1 is displayed clearly compared to the other subject images Gn (G2, G3), so viewers of distributed content C are likely to focus on subject Q1.

In FIG. 10, it is assumed that subject Q3 is selected as the reference subject Qref. Therefore, the contour or boundary of subject image G3 is maintained in a clear state, and subject images G1 and G2 are blurred images. Because imaging distance D1 is lower than imaging distance D2 within range Pa, the blur amount B1 of subject image G1 exceeds the blur amount B2 of subject image G2. In other words, subject image G1 is a blurred image compared to subject image G2. As described above, subject image G3 is displayed clearly compared to the other subject images Gn (G1, G2), so viewers of distributed content C are likely to focus on subject Q3.

[Superimposition processing]
The superimposition process is an image process in which multiple subject images G1 to G3 are superimposed on each other. In the superimposition process, the image adjustment unit 424 controls the front and rear of the multiple subject images G1 to G3 according to each imaging distance Dn. Specifically, the front and rear of each subject image Gn are adjusted so that the subject image Gn is positioned further back as the imaging distance Dn increases.

For example, in Figures 8 to 10, it is assumed that subject image G1 and subject image G2 partially overlap, and subject image G2 and subject image G3 partially overlap. As described above, imaging distance D1 is less than imaging distance D2. Therefore, in the superimposition process, image adjustment unit 424 superimposes each subject image Gn so that subject image G1 is located in front of subject image G2. In other words, the portion of subject image G2 that overlaps with subject image G1 is hidden behind subject image G1.

In addition, the imaging distance D2 is less than the imaging distance D3. Therefore, in the superimposition process, the image adjustment unit 424 superimposes each subject image Gn so that the subject image G2 is located in front of the subject image G3. That is, the portion of the subject image G3 that overlaps with the subject image G2 is hidden behind the subject image G2. As illustrated in the examples of Figures 8 to 10, in the first embodiment, the front and back of each subject image Gn are controlled according to the imaging distance Dn. Therefore, a natural composite video V can be generated in which the actual position of each subject Qn is reflected in the front and back of each subject image Gn.

As described above, the blurring and superimposing processes performed by the image adjustment unit 424 are examples of adjustment processes that adjust each subject image Gn according to each imaging distance Dn. Now, focus on subjects Qn1 and Qn2 (n1 = 1 to 3, n2 = 1 to 3, n1 ≠ n2). The adjustment processes are comprehensively expressed as processes that adjust subject images Gn1 and Gn2 according to the imaging distances Dn1 and Dn2 of the imaging devices 21-n1 and 21-n2.

The imaging device 21-n1 is an example of a "first imaging device," and the imaging distance Dn1 is an example of a "first imaging distance." The imaging device 21-n2 is an example of a "second imaging device," and the imaging distance Dn2 is an example of a "second imaging distance." Furthermore, the subject image Gn1 is an example of an "image of a first subject," and the subject image Gn2 is an example of an "image of a second subject." The video Vn1 is an example of a "first video," and the video Vn2 is an example of a "second video." The recording studio Rn1 is an example of a "first real space," and the recording studio Rn2 is an example of a "second real space."

Figure 11 is a flowchart of the operation of the video synthesis system 30. The operation of Figure 11 is initiated, for example, in response to an instruction from a user via the operation device 34, and is thereafter repeated at a predetermined interval.

The control device 31 (video acquisition unit 41) acquires multiple pieces of material data M1 to M3 (S1). The control device 31 (image processing unit 42) generates a composite video V by executing a synthesis process S2. The control device 31 (audio processing unit 43) also generates a composite voice A by mixing multiple voices A1 to A3 (S3). Note that the order of generating the composite video V (S2) and generating the composite voice A (S3) may be reversed. The control device 31 (output processing unit 44) generates a distribution content C including the composite video V and the synthetic voice A (S4), and distributes the distribution content C to the terminal device 200 (S5).

Figure 12 is a flowchart of the synthesis process S2 in Figure 11. When the synthesis process S2 is started, the control device 31 (distance determination unit 421) determines the imaging distance Dn for each of the multiple videos V1 to V3 (S21). The control device 31 (subject extraction unit 422) extracts a subject image Gn from each of the multiple videos V1 to V3 (S22). Note that the order of determining the imaging distance Dn (S21) and extracting the subject image Gn (S22) may be reversed.

The control device 31 (subject selection unit 423) selects one of the multiple subjects Q1 to Q3 as the reference subject Qref (S23). Specifically, the subject Qn specified by operating the operation device 34 is selected as the reference subject Qref. Note that the user can specify the desired subject Qn as the reference subject Qref at any point during the synthesis process S2. Therefore, the reference subject Qref can be changed at any point during the playback of the distribution content C.

The control device 31 (image adjustment unit 424) executes adjustment processing (S24, S25). Specifically, the control device 31 executes blurring processing S24 and superimposition processing S25 to generate a composite video V. That is, the composite processing S2 includes adjustment processing (S24, S25) that adjusts each subject image Gn according to each imaging distance Dn.

As described above, in the first embodiment, in the synthesis process S2 that synthesizes multiple videos V1 to V3, an adjustment process is performed to adjust the subject image Gn according to each imaging distance Dn. That is, as shown in the examples of Figures 8 to 10, the relationship between each imaging distance Dn is reflected in the relationship between each subject image Gn in the synthetic video V. Therefore, a natural synthetic video V including multiple subjects Q1 to Q3 located in different recording studios Rn can be generated. In particular, in the first embodiment, the blur amount Bn, which is an image parameter of each subject Qn, is adjusted according to the imaging distance Dn. Therefore, the effect of being able to generate a natural synthetic video V including multiple subjects Q1 to Q3 is particularly remarkable.

In particular, in the first embodiment, the blurring process S24 is performed so that the blurring amount Bn in each subject image Gn differs depending on the imaging distance Dn. Therefore, a natural composite video V can be generated that mimics the tendency of real imaging, in which the degree of optical blur changes depending on the imaging distance Dn.

In addition, the greater the difference between the imaging distance Dn and the reference value Dref, the greater the blur amount Bn applied to the blurring process S24 of the subject image Gn. Therefore, a natural composite video V can be generated that faithfully mimics the tendency of real imaging, in which the optical blur of the subject increases the further away in the depth direction (front-back direction) it is from the focal plane located at the point corresponding to the reference value Dref.

In the first embodiment, the blur amount Bn of each subject image Gn is set with the imaging distance Dn corresponding to one of the multiple subjects Q1 to Q3 (reference subject Qref) as the reference value Dref. Therefore, a natural composite video V can be generated in which the blur amount Bn of each subject image Gn is set with reference to the reference subject Qref. Also, a composite video V can be generated in which the reference subject Qref is particularly likely to attract attention among the multiple subjects Q1 to Q3.

[Second embodiment]
A second embodiment will be described. Note that, for elements having the same functions as those in the first embodiment in each aspect exemplified below, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

In the first embodiment, the subject selection unit 423 selects the reference subject Qref in response to an instruction from a user. In the second embodiment, the subject selection unit 423 selects the reference subject Qref from the multiple subjects Q1 to Q3 in response to the results of analyzing the multiple material data M1 to M3 (S23). As a method for the subject selection unit 423 to select the reference subject Qref, for example, the following aspect 1 or aspect 2 is adopted.

[Aspect 1]
The subject selection unit 423 in aspect 1 selects the reference subject Qref according to the results of analyzing the multiple moving images V1 to V3. For example, there is a general tendency that viewers pay particular attention to a subject Qn that is particularly moving among the multiple subjects Q1 to Q3. Taking the above tendency into consideration, the subject selection unit 423 selects the subject Qn that corresponds to the moving image Vn that changes greatly over time among the multiple subjects Q1 to Q3 as the reference subject Qref.

Specifically, the subject selection unit 423 calculates the amount of change over time of the image for each of the multiple videos V1 to V3, and selects the subject Qn corresponding to the video Vn with the largest amount of change as the reference subject Qref. According to the above embodiment, the subject Qn with the most prominent movement among the multiple subjects Q1 to Q3 is selected as the reference subject Qref. Note that the amount of change may be calculated by analyzing the entire video Vn, or may be calculated by analyzing the subject image Gn in the video Vn.

In a scene in which each of the multiple subjects Q1 to Q3 moves in sequence, the subject Qn whose movement is most prominent among the multiple subjects Q1 to Q3 changes over time. Therefore, the reference subject Qref changes at any point in the composite video V. For example, when a transition occurs from a state in which subject Qn1 moves to a state in which subject Qn2 moves, the reference subject Qref changes from subject Qn1 to subject Qn2. In other words, the subject image Gn that is clearly displayed in the composite video V switches from time to time.

[Aspect 2]
The subject selection unit 423 in aspect 2 selects the reference subject Qref according to the results of analyzing the multiple sounds A1 to A3. For example, there is a general tendency that viewers pay particular attention to the subject Qn who is speaking among the multiple subjects Q1 to Q3. Taking the above tendency into consideration, the subject selection unit 423 selects the subject Qn corresponding to the sound An with the loudest volume among the multiple subjects Q1 to Q3 as the reference subject Qref.

Specifically, the subject selection unit 423 calculates the volume of each of the multiple sounds A1 to A3, and selects the subject Qn corresponding to the sound An with the loudest volume as the reference subject Qref. According to the above embodiment, the subject Qn that is speaking among the multiple subjects Q1 to Q3 is selected as the reference subject Qref.

In a scene in which multiple subjects Q1 to Q3 each speak in sequence, the subject Qn with the loudest sound An changes over time. Therefore, the reference subject Qref changes at any point in the composite video V. For example, when a state transition occurs from one in which subject Qn1 speaks to one in which subject Qn2 speaks, the reference subject Qref changes from subject Qn1 to subject Qn2. In other words, the subject image Gn that is clearly displayed in the composite video V switches from time to time.

As can be understood from the explanations of aspects 1 and 2, the subject selection unit 423 in the second embodiment is expressed as an element that selects a reference subject Qref from a plurality of subjects Q1 to Q3 according to the results of analyzing a plurality of pieces of material data M1 to M3 (videos V1 to V3 or audio A1 to A3).

The second embodiment also achieves the same effects as the first embodiment. Furthermore, in the second embodiment, the reference subject Qref is selected from the multiple subjects Q1 to Q3 according to the results of analyzing the multiple material data M1 to M3. Therefore, an appropriate subject Qn that the viewer should pay particular attention to in the distributed content C can be selected as the reference subject Qref without the need for instructions from the user. Note that the selection of the reference subject Qref according to the results of the analysis of each material data Mn (second embodiment) and the selection of the reference subject Qref according to instructions from the user (first embodiment) may be used together.

[Third embodiment]
13 is a block diagram of the image processing unit 42 in the third embodiment. The image processing unit 42 in the third embodiment includes an image generating unit 425 in addition to the same elements as those in the first embodiment (a distance identifying unit 421, a subject extracting unit 422, a subject selecting unit 423, and an image adjusting unit 424). The image generating unit 425 generates a virtual moving image Vz.

Figure 14 is an explanatory diagram of a virtual video Vz. The virtual video Vz is a video of a virtual space Z. The virtual space Z is a virtual space in which multiple virtual objects Om (m = 1, 2) are arranged. In other words, the virtual space Z is a space separate from a real space such as a recording studio Rn, and is a space generated by information processing by a computer. Each virtual object Om is a virtual display element arranged in the virtual space Z for the purpose of, for example, performance or decoration. Note that in Figure 14, a virtual living thing active in the virtual space Z is exemplified as the virtual object Om. However, inanimate elements such as buildings and natural objects in the virtual space Z may also be arranged in the virtual space Z as the virtual object Om.

A virtual imaging device (hereinafter referred to as the "virtual imaging device") is installed within the virtual space Z. The virtual imaging device is a virtual camera that captures the virtual space Z. The virtual video Vz generated by the image generation unit 425 is a video captured of the virtual space Z by the virtual imaging device. Various types of image processing, such as 3D rendering, are used to generate the virtual video Vz. Note that the format of the data representing the virtual video Vz is arbitrary.

A virtual imaging distance Em is set for each virtual object Om. The virtual imaging distance Em is the distance between the virtual imaging device and the virtual object Om in the virtual space Z. In FIG. 14, it is assumed that the virtual imaging distance E2 of the virtual object O2 exceeds the virtual imaging distance E1 of the virtual object O1 (E2>E1).

Figure 15 is a flowchart of the synthesis process S2 in the third embodiment. In the synthesis process S2 in the third embodiment, the control device 31 (image generation unit 425) generates a virtual video Vz (S26). Note that the generation of the virtual video Vz (S26) may be performed at any stage before the start of the adjustment processes (S24, S25).

The synthesis process S2 in the third embodiment is an image process that generates a synthetic video V by synthesizing multiple subject images G1 to G3 with a virtual video Vz. FIG. 16 is a schematic diagram of the synthetic video V in the third embodiment. As illustrated in FIG. 16, the synthetic video V includes multiple subject images G1 to G3 and multiple virtual objects Om (O1, O2).

In the adjustment process (S24, S25) of the third embodiment, each subject image Gn is adjusted according to the imaging distance Dn, as in the first embodiment, and each virtual object Om is adjusted according to the virtual imaging distance Em.

Specifically, in the blurring process S24, the image adjustment unit 424 blurs each subject image Gn with a blur amount Bn according to the imaging distance Dn, and blurs each virtual object Om with a blur amount Bm according to the virtual imaging distance Em. The virtual imaging distance Em is used to control the blur amount Bm, just like the imaging distance Dn. For example, outside the focusing range P0, the image adjustment unit 424 sets the blur amount Bm of the virtual object Om to a larger value the greater the difference |Dref-Em| between the reference value Dref and the virtual imaging distance Em of each virtual object Om. As a result of the control exemplified above, each virtual object Om becomes a blurred image according to the virtual imaging distance Em, as exemplified in FIG. 16.

In addition, in the superimposition process S25, the image adjustment unit 424 controls the front and rear of each subject image Gn in the virtual space Z according to the imaging distance Dn, and also controls the front and rear of each virtual object Om in the virtual space Z according to the virtual imaging distance Em. Specifically, the front and rear of each virtual object Om is adjusted so that the virtual object Om is positioned further back as the virtual imaging distance Em increases.

For example, in FIG. 16, it is assumed that the virtual imaging distance E1 of the virtual object O1 is a numerical value between the imaging distance D1 of the subject image G1 and the imaging distance D2 of the subject image G2 (D1<E1<D2). Therefore, the image adjustment unit 424 places the subject image G1 in front of the virtual object O1, and places the virtual object O1 in front of the subject image G2. That is, the part of the virtual object O1 that overlaps with the subject image G1 is hidden behind the subject image G1, and the part of the subject image G2 that overlaps with the virtual object O1 is hidden behind the virtual object O1. Similarly, the subject image G2 is located in front of the virtual object O2, and the virtual object O2 is located in front of the subject image G3.

The third embodiment achieves the same effect as the first embodiment. Furthermore, in the third embodiment, a subject image Gn captured in a recording studio Rn is superimposed on a virtual object Om in a virtual space Z. Therefore, a variety of composite videos V including not only real subjects Qn but also virtual objects Om can be generated. Moreover, the virtual object Om is adjusted according to the virtual imaging distance Em. Therefore, compared to a form in which the virtual imaging distance Em is not reflected, a natural composite video V can be generated in which the virtual object Om appears to be located in the same space as each subject Qn.

The candidates for the reference subject Qref selected by the subject selection unit 423 may include the virtual object Om. For example, the subject selection unit 423 selects the reference subject Qref from a plurality of candidates (hereinafter referred to as "candidate subjects") including a plurality of subjects Q1 to Q3 and a plurality of virtual objects O1 and O2. Specifically, the subject selection unit 423 selects, as the reference subject Qref, a candidate subject designated by the user through an operation on the operation device 34 from among the plurality of candidate subjects. Therefore, the virtual object Om may be selected as the reference subject Qref, and the virtual imaging distance Em of the virtual object Om may be set as the reference value Dref.

In addition, aspect 1 of the second embodiment may be applied to the form in which the reference subject Qref is selected from a plurality of candidate subjects including the virtual object Om. Specifically, the subject selection unit 423 selects the reference subject Qref according to the results of analyzing the plurality of videos V1 to V3 and the virtual video Vz. For example, the subject selection unit 423 selects the reference subject Qref from a video with a large temporal change among a plurality of videos including the plurality of videos V1 to V3 and the videos of each virtual object Om. Therefore, when the amount of change in the video of the virtual object Om exceeds the amount of change in each video Vn, the subject selection unit 423 selects the virtual object Om as the reference subject Qref.

In addition, in a form in which the virtual object Om produces sound, the aspect 2 of the second embodiment may be applied. Specifically, the subject selection unit 423 selects the reference subject Qref according to the result of analyzing the multiple sounds A1 to A3 and the sound corresponding to the virtual video Vz. For example, the subject selection unit 423 selects the subject Qn or virtual object Om corresponding to the sound with the loudest volume among the multiple sounds including the multiple sounds A1 to A3 and the sound of each virtual object Om as the reference subject Qref. Therefore, when the virtual object Om produces sound at a certain volume, the subject selection unit 423 selects the virtual object Om as the reference subject Qref.

[Fourth embodiment]
17 is a block diagram illustrating a functional configuration of a moving image synthesizing system 30 according to the fourth embodiment. The control device 31 according to the fourth embodiment functions as an imaging control unit 45 in addition to the same elements as those in the first embodiment (a moving image acquisition unit 41, an image processing unit 42, an audio processing unit 43, and an output processing unit 44). The imaging control unit 45 controls a plurality of imaging devices 21-1 to 21-3.

Each imaging device 21-n in the fourth embodiment is a PTZ (Panoramac-Tilt-Zoom) camera that can change the conditions for capturing video Vn (hereinafter referred to as "imaging conditions"). The imaging conditions are conditions that define the range captured by the imaging device 21-n. For example, the imaging conditions include the imaging direction and imaging magnification. The imaging direction is the direction of the optical axis of the shooting lens, and changes, for example, to the horizontal direction (pan) and vertical direction (tilt). The imaging magnification is, for example, a magnification (zoom) that corresponds to the focal length.

The imaging control unit 45 controls each imaging device 21-n in response to a user's operation on the operation device 34. The user operates the operation device 34 while watching the distribution content C played by the playback device 35 to specify the imaging conditions for each imaging device 21-n. The imaging control unit 45 generates control data X that specifies the imaging conditions specified by the user. The control data X is data that specifies the imaging direction and imaging magnification. For example, the control data X specifies the imaging direction and imaging magnification as the amount of change (relative value) with respect to the current numerical value, or as an absolute value based on a predetermined value. Note that the time series of the control data X may be stored in advance in the storage device 32.

The imaging control unit 45 transmits control data X from the communication device 33 to the multiple recording systems 20-1 to 20-3. That is, common control data X is supplied to the multiple imaging devices 21-1 to 21-3. Each imaging device 21-n is of the same model and has common specifications such as operating characteristics. Therefore, the multiple imaging devices 21-1 to 21-3 operate in the same way in response to the control data X. That is, the supply of control data X by the imaging control unit 45 controls the multiple imaging devices 21-1 to 21-3 to common imaging conditions.

The imaging control unit 45 transmits the control data X to the multiple recording systems 20-1 to 20-3 in parallel over time. That is, the control data X is supplied to the multiple imaging devices 21-1 to 21-3 in parallel over time. Therefore, the imaging conditions of each imaging device 21-n change in parallel over time according to the control data X.

As can be understood from the above explanation, the imaging control unit 45 operates the multiple imaging devices 21-1 to 21-3 in parallel with each other in time under common imaging conditions. For example, when the imaging direction of the imaging device 21-1 changes by a specific angle, the imaging directions of the imaging devices 21-2 and 21-3 also change by the same angle in parallel with the change in the imaging direction of the imaging device 21-1. Furthermore, when the imaging magnification of the imaging device 21-1 changes to a predetermined magnification, the imaging magnifications of the imaging devices 21-2 and 21-3 also change to the same magnification in parallel with the change in the imaging magnification of the imaging device 21-1. In other words, the imaging conditions of the multiple imaging devices 21-1 to 21-3 are linked to each other and change to common conditions.

The fourth embodiment achieves the same effects as the first embodiment. Furthermore, each imaging device 21-n captures a video Vn under common imaging conditions. Therefore, a natural composite video V can be generated that is perceived by the viewer as if multiple subjects Q1 to Q3, which are actually located in different recording studios Rn, are located in a common space.

The second and third embodiments are also similarly applied to the fourth embodiment. For example, in the third embodiment in which the virtual video Vz is synthesized into multiple videos V1 to V3, the image generation unit 425 may link the imaging conditions for the virtual imaging device to capture the virtual video Vz to the imaging conditions of each imaging device 21-n. In other words, the multiple imaging devices 21-1 to 21-3 and the virtual imaging device may operate in parallel with each other in time under common imaging conditions.

[Modification]
Each of the above-mentioned exemplary embodiments may be modified in various ways. Specific modified embodiments that may be applied to each of the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined to the extent that they are not mutually contradictory.

(1) In each of the above-described embodiments, the imaging distance Dn is determined using a distance index, but the method of determining the imaging distance Dn of each subject Qn is not limited to the above examples. For example, the imaging distance Dn may be determined by the following example 1 or 2.

[Aspect 1]
The distance specification unit 421 may specify the imaging distance Dn according to the result of face detection for each subject Qn. For example, the distance specification unit 421 uses the result of face detection for the video Vn to calculate a size index (hereinafter referred to as an "evaluation index") for each element of the face of the subject Qn. Face detection is a process for detecting the face of the subject Qn. For example, a numerical value such as the size of the face or the distance between the eyes is calculated as the evaluation index.

There is a correlation in which the evaluation index decreases as the imaging distance Dn increases. Taking the above correlation into consideration, the distance determination unit 421 determines the imaging distance Dn according to the evaluation index. For example, the distance determination unit 421 sets the imaging distance Dn to a smaller value as the evaluation index increases compared to a reference value, and sets the imaging distance Dn to a larger value as the evaluation index decreases compared to the reference value. Note that, since there are individual differences in the size of each facial element, it is desirable to prepare a reference value for the evaluation index individually for each subject Qn. For example, the evaluation index calculated when the imaging distance Dn is a predetermined value is stored in advance in the storage device 32 as a reference value.

In the above description, face detection of subject Qn has been exemplified, but the distance determination unit 421 may use the results of skeletal estimation for subject Qn to calculate an evaluation index for determining the imaging distance Dn. Skeletal estimation is a process of estimating the skeleton of subject Qn, such as joints. For example, the distance between specific joints (e.g., arm length) or the ratio is calculated as the evaluation index. The distance determination unit 421 determines the imaging distance Dn according to the evaluation index related to the skeleton. Since skeletons vary from person to person, the reference value of the evaluation index may be prepared individually for each subject Qn.

[Aspect 2]
In a configuration in which a distance measuring device measures the imaging distance Dn when the imaging device 21-n captures an image, the distance determination unit 421 acquires the imaging distance Dn measured by the distance measuring device. The distance measuring device is an optical sensor that uses distance measuring light such as infrared light or ultraviolet light. The distance measuring device has, for example, a LiDAR (Light Detection and Ranging) function. The imaging distance Dn may also be determined by an autofocus function of the imaging device 21-n. The autofocus function is a function that automatically focuses on the subject Qn. The imaging distance Dn is determined according to the result of controlling the photographing lens.

Any method other than the method 1 and the method 2 may be used to determine the imaging distance Dn. For example, the distance determination unit 421 may determine the imaging distance Dn by using the results of imaging a distance indicator (marker) of the subject Qn by a plurality of imaging devices 21-n.

It should be noted that the imaging distance Dn of each subject Qn does not need to be determined by calculation processing by the control device 31 (distance determination unit 421). For example, the imaging distance Dn actually measured before recording in each recording studio Rn using a measuring device such as a tape measure may be stored in advance in the storage device 32 for each subject Qn. In the synthesis process S2, the control device 31 (distance determination unit 421) acquires the imaging distance Dn of each subject Qn from the storage device 32 (S21). As can be understood from the above explanation, the determination of the imaging distance Dn by the distance determination unit 421 (S21) also includes the reading of the imaging distance Dn that has been stored in advance.

(2) In each of the above-described embodiments, the blur amount Bn is set according to the imaging distance Dn, but the blur amount Bn may depend on a control parameter other than the imaging distance Dn. For example, the image adjustment unit 424 may control the correlation between the imaging distance Dn and the blur amount Bn according to a virtual aperture value (hereinafter referred to as "virtual aperture value F"). The image adjustment unit 424 sets the virtual aperture value F according to, for example, an instruction from the user to the operation device 34.

Figure 18 is a graph showing the relationship between the imaging distance Dn and the blur amount Bn in this modified example. Figure 18 also shows a graph in which the virtual aperture value F is set to two different values (F1, F2). Value F1 is lower than value F2.

The image adjustment unit 424 controls the focus range P0 according to the virtual aperture value F. Specifically, as illustrated in FIG. 18, the focus range P0 when the virtual aperture value F is set to a value F1 is set to a narrower range than the focus range P0 when the virtual aperture value F is set to a value F2 (>F1). Through the above control, the real-world tendency for the depth of field to shrink as the aperture value of the shooting lens becomes smaller is simulated.

The image adjustment unit 424 also controls the blur amount Bn for the imaging distance Dn according to the virtual aperture value F. Specifically, even when the imaging distance Dn is set to the same value, the blur amount Bn when the virtual aperture value F is set to the value F1 exceeds the blur amount Bn when the virtual aperture value F is set to the value F2 (>F1). Through the above control, the actual tendency that the smaller the aperture value of the shooting lens, the greater the degree of optical blur is is simulated.

In the above description, the focus has been on the virtual aperture value F, but the control parameter that affects the blur amount Bn is not limited to the virtual aperture value F. For example, the image adjustment unit 424 may control the correlation between the imaging distance Dn and the blur amount Bn according to a virtual focal length (hereinafter referred to as "virtual focal length"). The virtual focal length is set according to an instruction from the user to the operation device 34, for example. Specifically, the image adjustment unit 424 reduces the focusing range P0 as the virtual focal length increases, and sets the blur amount Bn for the imaging distance Dn to a larger value. According to the above embodiment, the actual tendency that the depth of field is more likely to be reduced and the optical blur is more likely to be increased as the focal length of the photographing lens increases is simulated.

As can be understood from the above examples, the imaging distance Dn exemplified in each of the above-mentioned embodiments and the virtual aperture value F and virtual focal length exemplified in this modified example are collectively expressed as control parameters for controlling the blur amount Bn. The control parameters are not limited to the types of variables exemplified above.

(3) The relationship between the imaging distance Dn and the blur amount Bn is not limited to the relationship illustrated in FIG. 7. For example, as illustrated in FIG. 19, a form in which the blur amount Bn changes in a curve with respect to the imaging distance Dn is also possible. Also, as illustrated in FIG. 20, the focus range P0 in which the blur amount Bn does not change with respect to the imaging distance Dn may be omitted.

As shown in FIG. 21, it is also possible to assume that the relationship between the imaging distance Dn and the blur amount Bn differs between ranges Pa and Pb. FIG. 21 illustrates a case in which the gradient of the blur amount Bn with respect to the imaging distance Dn differs between ranges Pa and Pb. As can be seen from FIG. 21, the numerical range of the blur amount Bn also differs between ranges Pa and Pb. According to the embodiment in FIG. 21, it is possible to make the blur characteristics of the subject Qn different between the front side and the back side of the reference subject Qref.

(4) In each of the above embodiments, it is assumed that the imaging conditions are common to the multiple imaging devices 21-1 to 21-3, but it is also assumed that the imaging conditions, such as the imaging magnification, are different for each imaging device 21-n. Also, in an embodiment in which the shooting lens of each imaging device 21-n is a zoom lens, it is also assumed that the imaging magnification (focal length) is set individually for each imaging device 21-n.

The size of the subject image Gn in each video Vn depends not only on the imaging distance Dn but also on imaging conditions such as imaging magnification. For example, even if the size of the subject Qn itself and the imaging distance Dn are the same, the size of the subject image Gn increases as the imaging magnification of the imaging device 21-n increases. In order to create a natural relationship between the sizes of the multiple subject images G1 to G3, the image adjustment unit 424 compensates for the differences in imaging magnification for each imaging device 21-n in the synthesis process S2. In other words, the differences in size of the subject image Gn caused by differences in imaging magnification are reduced.

Specifically, the image adjustment unit 424 enlarges or reduces each subject image Gn in inverse proportion to the imaging magnification. For example, if the imaging magnification of the imaging device 21-n1 is twice that of the imaging device 21-n2, the image adjustment unit 424 adjusts the size of the subject image Gn2 to 1/2 without changing the size of the subject image Gn1. Alternatively, the image adjustment unit 424 may adjust the size of the subject image Gn1 to 2 without changing the size of the subject image Gn2. According to the above embodiment, the difference in imaging conditions for each imaging device 21-n is compensated for, and as a result, a natural composite video V can be generated.

In a configuration in which the image adjustment unit 424 cannot acquire the imaging magnification of each imaging device 21-n, the image adjustment unit 424 may adjust the size of each subject image Gn according to the imaging distance Dn. Specifically, the image adjustment unit 424 enlarges or reduces the size of each subject image Gn so that the size of the distance index in each subject image Gn is inversely proportional to the imaging distance Dn. For example, when the imaging distance Dn2 is twice the imaging distance Dn1, the size of one or both of the subject images Gn1 and Gn2 is adjusted so that the size of the distance index in the subject image Gn2 is 1/2 the size of the distance index in the subject image Gn1.

(5) In each of the above embodiments, the video compositing system 30 is installed in a location separate from each recording studio Rn. However, the video compositing system 30 may be installed in the recording studio Rn. Also, the video compositing system 30 may be mounted on the imaging device 21-n.

(6) In each of the above embodiments, the subject image Gn is extracted by removing the area of the video Vn that corresponds to the background of a specific color, but the method of extracting the subject image Gn from the video Vn is not limited to the above examples. For example, the subject extraction unit 422 may extract the subject image Gn from the video Vn by a known object detection process. Examples of the object detection process include object detection using an estimation model such as a deep neural network, or object detection using image processing such as a background difference method. As can be understood from the above explanation, the background of the recording studio Rn does not need to be a specific color.

(7) In each of the above-mentioned embodiments, the imaging distance Dn corresponding to the reference subject Qref is set as the reference value Dref, but the method of setting the reference value Dref is not limited to the above examples. For example, a numerical value arbitrarily specified by the user through operation of the operation device 34 may be set as the reference value Dref. The reference value Dref is changed as needed in response to instructions from the user. The image adjustment unit 424 applies the reference value Dref specified by the user to the blurring process S24. In addition, assuming a case where distribution content C is produced for an event whose progress is planned in advance, the time series of the reference value Dref may be stored in the storage device 32 in advance. The image adjustment unit 424 sequentially applies the reference values Dref acquired in chronological order from the storage device 32 to the blurring process S24.

As can be understood from the above examples, the reference value Dref is not limited to the imaging distance Dn of the reference subject Qref. In other words, the reference value Dref may be set regardless of the imaging distance Dn of the reference subject Qref. Therefore, the selection of the reference subject Qref (subject selection unit 423, S23) may be omitted in this disclosure.

(8) In each of the above embodiments, the multiple subject images G1 to G3 are synthesized in the superimposition process S25, but the stage at which the adjustment process is performed during the synthesis process of synthesizing the multiple subject images G1 to G3 is arbitrary. For example, the multiple subject images G1 to G3 may be synthesized after the blurring process S24 or adjustments such as front and rear adjustments are performed on each subject image Gn, or the multiple subject images G1 to G3 may be synthesized and then the adjustment process is performed on each subject image Gn in the synthetic video V.

(9) In each of the above embodiments, blurring processing S24 for the subject image Gn is exemplified, but the processing for adjusting the image parameters of the subject image Gn is not limited to the blurring processing S24 exemplified above. For example, image processing for adjusting any image parameter such as brightness (exposure), saturation, hue, contrast, clarity, etc. is collectively expressed as "processing." In the processing, the above image parameters for the subject image Gn are adjusted according to the imaging distance Dn.

For example, the image adjustment unit 424 sets image parameters such as brightness, saturation, contrast, or clarity to smaller values as the difference |Dref-Dn| between the reference value Dref and the imaging distance Dn of each subject Qn increases. Even in the above embodiment, a composite video V can be generated in which the reference subject Qref is particularly likely to attract attention. In addition, a form in which the hue of the subject image Gn is adjusted according to the difference |Dref-Dn| between the reference value Dref and the imaging distance Dn is also envisioned. For example, the image adjustment unit 424 adjusts the subject image Gn to a specific hue when the difference |Dref-Dn| is within a predetermined range.

(10) In each of the above embodiments, the recording studio Rn is exemplified as a real space, but the real space is not limited to an indoor space such as the recording studio Rn. For example, the recording system 20-n may be installed in a real space such as an outdoor space. As can be understood from the above explanation, the real space is defined as an actual space in the real world, regardless of whether it is indoors or outdoors.

(11) In each of the above-described embodiments, the composite video V is generated by combining multiple videos V1 to V3, but other images may be further combined. For example, the screen of a game played by subject Qn may be combined with the composite video V. Also, a video captured by an imaging device other than imaging device 21-n installed in recording studio Rn may be combined with the composite video V. Also, in each of the above-described embodiments, the composite sound A is generated by combining multiple sounds A1 to A3, but other sounds may be further combined. For example, the sound of a game played by subject Qn may be combined with the composite sound A.

(12) When the communication delay between the video compositing system 30 and the recording system 20-1 differs from the communication delay between the video compositing system 30 and the recording system 20-2, the videos V1 and V2 acquired by the video compositing system 30 may not be synchronized in time with each other. For example, a situation may be assumed in which one of the videos V1 and V2 is delayed relative to the other. In the above situation, the image processing unit 42 (e.g., the image adjustment unit 424) may synchronize the multiple videos V1 to V3 with each other in time. According to the above embodiment, a natural composite video V can be generated in which the time lag between the multiple videos V1 to V3 is reduced.

(13) In the fourth embodiment, the imaging direction and imaging magnification are exemplified as imaging conditions, but the imaging conditions controlled by the imaging control unit 45 are not limited to the above examples. For example, the "imaging conditions" also include conditions that do not affect the imaging range itself, such as the focus position, aperture value (iris), exposure time (shutter speed), exposure value, or white balance.

(14) In each of the above embodiments, three recording systems 20-1 to 20-3 are illustrated, but the number of recording systems 20-n is arbitrary. For example, this disclosure is similarly applicable to a configuration in which two recording systems 20-n are installed, or a configuration in which four or more recording systems 20-n are installed.

For example, assuming that the video recording system 100 is configured to include N (N is a natural number equal to or greater than 2) imaging devices 21-1 to 21-N, one imaging device 21-n1 (n1 = 1 to N) selected from the N imaging devices 21-1 to 21-N is an example of a "first imaging device" in this disclosure, and the other imaging device 21-n2 (n2 = 1 to N, n2 ≠ n1) is an example of a "second imaging device" in this disclosure.

(15) As described above, the functions of the video synthesis system 30 according to each of the above embodiments are realized by the cooperation of one or more processors constituting the control device 31 and the program stored in the storage device 32. The above-mentioned programs can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the recording medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

(16) In this disclosure, the term "nth" (n is a natural number) is used only as a formal or convenient label to distinguish each element in notation and does not have any substantive meaning. Therefore, there is no room for restrictive interpretation of the position of each element or the order of processing, etc., based on the term "nth".

[Additional Notes]
From the above description, for example, preferred aspects of the present disclosure can be understood as follows. In addition, in order to facilitate understanding of each aspect, reference numerals in the drawings are conveniently written in parentheses below, but the present disclosure is not limited to the illustrated aspects.

[Appendix 1]
A moving image compositing system (30) according to one aspect (Supplementary Note 1) of the present disclosure includes a moving image acquisition unit (41) that acquires a first moving image (Vn1) captured by a first imaging device (21-n1) in a first real space (Rn1) and a second moving image (Vn2) captured by a second imaging device (21-n2) in a second real space (Rn2), and an image (Gn1) of a first subject (Qn1) in the first moving image (Vn1) and an image (Gn2) of a second subject (Qn2) in the second moving image (Vn2). and an image processing unit (42) that executes a synthesis process (S2) for generating a synthetic moving image (V) including an image (Gn1) of the first subject (Qn1) and an image (Gn2) of the second subject (Qn2), the synthesis process (S2) including adjustment processes (S24, S25) for adjusting an image (Gn1) of the first subject (Qn1) and an image (Gn2) of the second subject (Qn2) in accordance with a first imaging distance (Dn1) of the first imaging device (21-n1) and a second imaging distance (Dn2) of the second imaging device (21-n2).

According to the above aspect, in the synthesis process (S2) for synthesizing the first video (Vn1) captured by the first imaging device (21-n1) and the second video (Vn2) captured by the second imaging device (21-n2), an adjustment process (S24, S25) is executed for adjusting the image (Gn1) of the first subject (Qn1) in the first video (Vn1) and the image (Gn2) of the second subject (Qn2) in the second video (Vn2) according to the first imaging distance (Dn1) and the second imaging distance (Dn2). That is, the relationship between the first imaging distance (Dn1) and the second imaging distance (Dn2) is reflected in the relationship between the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) in the synthetic video (V). Therefore, a natural synthetic video (V) including multiple subjects (Qn) located in different real spaces (Rn) can be generated.

"(First/Second) Real Space (Rn)" is a space that actually exists in the real world, and is a concept that contrasts with virtual space (Z).

The "(first/second) video" is a dynamic image composed of a time series of multiple images. The first video (Vn1) and the second video (Vn2) are captured, for example, in parallel with each other in time. That is, the period during which the first video (Vn1) is captured and the period during which the second video (Vn2) is captured overlap with each other on the time axis. However, the first video (Vn1) and the second video (Vn2) do not have to be captured in parallel with each other. That is, the period during which the first video (Vn1) is captured and the period during which the second video (Vn2) is captured do not have to overlap with each other on the time axis.

The "composite process (S2)" is an arbitrary image process that generates a composite video (V) including an image (Gn1) of a first subject (Qn1) in a first video (Vn1) and an image (Gn2) of a second subject (Qn2) in a second video (Vn2). The stage at which the adjustment processes (S24, S25) are performed in the composite process (S2) is arbitrary. For example, the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) may be combined after the adjustment process (S24, S25) is performed on at least one of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2), or the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) may be combined before the adjustment process (S24, S25) is performed on at least one of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2). In addition, the adjustment process (S24, S25) (for example, the superimposition process (S25) described later) may be performed in the process of combining the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2).

The "imaging distance (Dn)" is the distance between the (first/second) imaging device and the (first/second) subject. Any method can be used to determine the imaging distance (Dn). For example, if the imaging distance (Dn) is a preset value determined in advance, the imaging distance (Dn) may be stored in advance in a storage device.

The imaging distance (Dn) may be estimated by analyzing a video captured by an imaging device. For example, the imaging distance (Dn) is estimated by analyzing the size of a predetermined distance indicator (marker) added to a subject in the video. Also, the imaging distance (Dn) is estimated for a subject of known size (e.g., a performer) by analyzing the size of the subject in the video using recognition technology such as face recognition or skeletal recognition.

In an environment where a ranging device measures the imaging distance (Dn) when an image is captured by an imaging device, the imaging distance (Dn) measured by the ranging device is acquired. The ranging device is an optical sensor that uses ranging light such as infrared light or ultraviolet light. The ranging device has, for example, a LiDAR (Light Detection and Ranging) function or an autofocus function. The ranging device may be mounted on the imaging device or installed separately from the imaging device.

The "adjustment process (S24, S25)" is an image process that adjusts the image of the subject according to the imaging distance (Dn). "Depending on the imaging distance (Dn)" means that the image of the subject in the composite image depends on the imaging distance (Dn). That is, assuming that the imaging distance (Dn) can be set to a first value and a second value that are different from each other, for example, this means that the image after the adjustment process (S24, S25) when the imaging distance (Dn) is the first value will be different from the image after the adjustment process (S24, S25) when the imaging distance (Dn) is the second value. However, it is possible that the image will not change even if the imaging distance (Dn) changes.

The "multiple imaging devices" may include one or more imaging devices other than the first imaging device (21-n1) and the second imaging device (21-n2). That is, the image processing unit (42) may generate a composite video (V) by combining three or more videos captured by different imaging devices. The "first video (Vn1)" is one video selected from the multiple videos, and the "second video (Vn2)" is one video among the multiple videos other than the first video (Vn1). That is, the composite video (V) may be generated by combining only the first video (Vn1) and the second video (Vn2), or may be generated by combining the first video (Vn1) and the second video (Vn2) with one or more other videos.

[Appendix 2]
In a specific example (Supplementary Note 2) of Supplementary Note 1, the adjustment process (S24, S25) includes a processing process for adjusting the image parameters of the first subject (Qn1) and the image parameters of the second subject (Qn2) according to the first imaging distance (Dn1) and the second imaging distance (Dn2). In the above aspect, the image parameters of the first subject (Qn1) and the image parameters of the second subject (Qn2) are adjusted according to the first imaging distance (Dn1) and the second imaging distance (Dn2). Therefore, a natural composite video (V) including multiple subjects (Qn) located in different spaces can be generated.

The "processing process" is any image processing that adjusts the image parameters of the first subject (Qn1) and the second subject (Qn2) according to the first imaging distance (Dn1) and the second imaging distance (Dn2). For example, the "adjustment process (S24, S25)" is an example of an image processing that increases the difference between the image parameters of the first subject (Qn1) and the image parameters of the second subject (Qn2) the greater the difference between the first imaging distance (Dn1) and the second imaging distance (Dn2). However, for example, when the first imaging distance (Dn1) and the second imaging distance (Dn2) are close to or coincident with each other, it is possible that the image parameters of the first subject (Qn1) and the image parameters of the second subject (Qn2) coincide with each other.

An "image parameter" is any parameter for controlling the visual characteristics of an image. For example, any characteristic value such as the amount of blur (Bn), brightness (exposure), saturation, hue, contrast, and clarity are examples of "image parameters." A combination of multiple different parameters may also be interpreted as an "image parameter."

[Appendix 3]
In a specific example (Supplementary Note 3) of Supplementary Note 2, the processing includes a blurring process (S24) for blurring an image, and the image parameter is a blurring amount (Bn). According to the above aspect, the blurring process (S24) is executed so that the blurring amount (Bn) in the image (Gn1) of the first object (Qn1) and the blurring amount (Bn) in the image (Gn2) of the second object (Qn2) differ depending on the first imaging distance (Dn1) and the second imaging distance (Dn2). It is possible to generate a natural composite moving image (V) that simulates the tendency of real imaging in which the degree of optical blur changes depending on the imaging distance (Dn).

The "blurring process (S24)" is an image process that blurs an image. For example, a smoothing process that replaces the pixel value of each pixel in a video with the average value within a predetermined range that includes the pixel is an example of the blurring process (S24). The blur amount (Bn) is an image parameter that controls the degree to which the image is blurred by the blurring process (S24). For example, the image parameter that specifies the size of the range in which the average pixel value is calculated is exemplified as the "blur amount (Bn)".

When the first imaging distance (Dn1) and the second imaging distance (Dn2) are different, the blur amount (Bn) of the image (Gn1) of the first subject (Qn1) and the blur amount (Bn) of the image (Gn2) of the second subject (Qn2) are set to different values. The blurring process (S24) may be performed on both the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2), or the blurring process (S24) may be performed on only one of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2).

[Appendix 4]
In a specific example (Supplementary Note 4) of Supplementary Note 3, the first imaging distance (Dn1) can be set to a first distance (Da1, Db1) and a second distance (Da2, Db2) that are different from each other, a difference between the second distance (Da2, Db2) and a reference value (Dref) exceeds a difference between the first distance (Da1, Db1) and the reference value (Dref), and the image processing unit (42) determines that the first imaging distance (Dn1) is equal to or greater than the first distance (Da When the first imaging distance (Dn1) is the second distance (Da2, Db2), the blur amount (Bn) for the image (Gn1) of the first object (Qn1) is set to a first set value (Ba1, Bb1), and when the first imaging distance (Dn1) is the second distance (Da2, Db2), the blur amount (Bn) for the image (Gn1) of the first object (Qn1) is set to a second set value (Ba2, Bb2) that exceeds the first set value (Ba1, Bb1). According to the above aspect, the blur amount (Bn) applied to the blurring process (S24) of the image (Gn1) of the first object (Qn1) increases as the difference between the first imaging distance (Dn1) and the reference value (Dref) increases. Therefore, it is possible to generate a natural composite moving image (V) that faithfully simulates the tendency of real imaging, in which the optical blur of the object increases as the object moves away in the depth direction (front-back direction) from the focal plane located at the point corresponding to the reference value (Dref).

The "reference value (Dref)" is a specific numerical value that the imaging distance (Dn) can take. For example, the second imaging distance (Dn2) is set as the "reference value (Dref)". According to the above embodiment, a composite video (V) can be generated in which the second subject (Qn2) is in focus and the first subject (Qn1) is blurred. If the imaging distance (Dn) is a numerical value within a predetermined range including the reference value (Dref), the subject blur amount (Bn) may be set to zero. According to the above embodiment, the optical tendency of the elements within the depth of field to be entirely in focus can be simulated.

[Appendix 5]
In a specific example (Supplementary Note 5) of Supplementary Note 4, the reference value (Dref) is an imaging distance (Dn) corresponding to a reference subject (Qref) which is one of a plurality of subjects (Qn) corresponding to a plurality of moving images including the first moving image (Vn1) and the second moving image (Vn2). According to the above aspect, the imaging distance (Dn) corresponding to one of the plurality of subjects (Qn) (reference subject (Qref)) is set as the reference value (Dref) to set the blur amount (Bn) for the image (Gn1) of the first subject (Qn1). Therefore, a natural composite moving image (V) can be generated in which the blur amount (Bn) of the image of each subject is set based on the reference subject (Qref). In addition, a composite moving image (V) can be generated in which the reference subject (Qref) among the plurality of subjects (Qn) is particularly likely to attract attention.

The "reference subject (Qref)" is one or more subjects selected from a plurality of subjects (Qn) corresponding to different videos. The conditions for selecting the reference subject (Qref) are arbitrary. For example, a subject selected by a user from a plurality of subjects (Qn) is set as the reference subject (Qref). Note that the "plurality of videos" may include not only videos actually captured in the real space (Rn) but also virtual videos representing the virtual space (Z). The virtual video is a video captured by a virtual imaging device of a virtual object (Om) in the virtual space (Z). The reference subject (Qref) is selected from a plurality of subjects (Qn) including a subject in the real space (Rn) and a virtual object (Om) in the virtual space (Z).

[Appendix 6]
In a specific example (Supplementary Note 6) of Supplementary Note 5, the image processing unit (42) selects the reference subject (Qref) from the multiple subjects (Qn) according to a result of analyzing the multiple videos or the multiple sounds corresponding to the multiple videos. According to the above aspect, the reference subject (Qref) is selected from the multiple subjects (Qn) according to a result of analyzing the video or sound of each subject. Therefore, an appropriate subject that the viewer should pay particular attention to in the composite video (V) can be selected as the reference subject (Qref) without the need for instructions from the user.

The specific content of the process for analyzing each video and the conditions for applying the results of the analysis to the selection of the subject are arbitrary. For example, in one embodiment, a subject that changes significantly over time among multiple subjects (Qn) (i.e., a subject in motion) is selected as the reference subject (Qref). In another embodiment, in which each video includes audio, a subject that corresponds to a video with a louder audio volume among multiple subjects (Qn) (i.e., a subject speaking) is selected as the reference subject (Qref).

[Appendix 7]
In any specific example (Supplementary Note 7) of Supplementary Note 1 to Supplementary Note 6, the adjustment process (S24, S25) includes a superimposition process (S25) for superimposing an image (Gn1) of the first subject (Qn1) and an image (Gn2) of the second subject (Qn2), and in the superimposition process (S25), the front and rear of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) is controlled according to the first imaging distance (Dn1) and the second imaging distance (Dn2). According to the above aspect, the front and rear of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) is controlled according to the first imaging distance (Dn1) and the second imaging distance (Dn2). Therefore, the actual positions of the first subject (Qn1) and the second subject (Qn2) are reflected in front of and behind the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2), thereby generating a natural composite moving image (V).

"Superimposition process (S25)" is image processing that superimposes multiple subjects (Qn). Superimposition process (S25) causes the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) to partially overlap. The "front and back" between the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) refers to the positional relationship of whether the image (Gn2) of the second subject (Qn2) is placed behind the image (Gn1) of the first subject (Qn1), or whether the image (Gn1) of the first subject (Qn1) is placed behind the image (Gn2) of the second subject (Qn2).

"The foreground and background between the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) is controlled according to the first imaging distance (Dn1) and the second imaging distance (Dn2)" means that the foreground and background between the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) in the composite video (V) depends on the first imaging distance (Dn1) and the second imaging distance (Dn2). For example, when the first imaging distance (Dn1) is less than the second imaging distance (Dn2), the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) are superimposed so that the image (Gn1) of the first subject (Qn1) is located in front of the image (Gn2) of the second subject (Qn2), and when the second imaging distance (Dn2) is less than the first imaging distance (Dn1), the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) are superimposed so that the image (Gn2) of the second subject (Qn2) is located in front of the image (Gn1) of the first subject (Qn1).

As can be understood from the above examples, the "front and back" of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) can also be thought of as the priority in displaying the images (Gn1) and (Gn2). For example, a state in which the image (Gn1) is displayed in front of the image (Gn2) can also be expressed as a state in which the image (Gn1) is displayed in priority over the image (Gn2), and a state in which the image (Gn2) is displayed in front of the image (Gn1) can also be expressed as a state in which the image (Gn2) is displayed in priority over the image (Gn1).

In addition, the "front and back" of the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) can also be thought of as a distinction between display/non-display of the image (Gn1) and the image (Gn2). For example, a state in which the image (Gn1) is displayed in front of the image (Gn2) can also be expressed as a state in which the part of the image (Gn1) that overlaps with the image (Gn2) is displayed, and the part of the image (Gn2) that overlaps with the image (Gn1) is not displayed. Similarly, for example, a state in which the image (Gn2) is displayed in front of the image (Gn1) can also be expressed as a state in which the part of the image (Gn2) that overlaps with the image (Gn1) is displayed, and the part of the image (Gn1) that overlaps with the image (Gn2) is not displayed.

[Appendix 8]
In any of the specific examples (Supplementary Note 8) of Supplementary Note 1 to Supplementary Note 7, the image processing unit (42) generates the composite video (V) including the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) and a virtual object (Om) captured by a virtual imaging device in a virtual space (Z) in the synthesis process (S2), and adjusts the virtual object (Om) according to the virtual imaging distance (Em) of the virtual imaging device in the adjustment process (S24, S25). According to the above aspect, the images of the subjects (first subject (Qn1) and second subject (Qn2)) captured in the real space (Rn) are superimposed on the virtual object (Om) in the virtual space (Z). Therefore, a variety of composite videos (V) including not only real subjects but also virtual objects (Om) can be generated. Moreover, since the virtual object (Om) is adjusted according to the virtual imaging distance (Em), a natural composite moving image (V) can be generated in which the virtual object (Om) appears to exist in the same space as each subject.

"Virtual space (Z)" is a virtual space that is set by various types of information processing such as image processing, and is a concept that is contrasted with real space (Rn). A virtual object (Om) is an object that exists in the virtual space (Z) and is imaged by a virtual imaging device in the virtual space (Z). A "virtual imaging device" is a virtual imaging device installed in the virtual space (Z). The imaging conditions by the virtual imaging device may be either fixed or variable.

The content of the "adjustment process (S24, S25)" for the image of the virtual object (Om) is arbitrary. For example, the "adjustment process (S24, S25)" including the processing process or superimposition process (S25) exemplified in each of the above-mentioned embodiments is performed not only on the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) but also on the image of the virtual object (Om).

[Appendix 9]
The moving image compositing system (30) according to any one of the specific examples (Supplementary Note 9) of Supplementary Note 1 to Supplementary Note 8 further includes an imaging control unit (45) that operates the first imaging device (21-n1) and the second imaging device (21-n2) in parallel with each other in time under common imaging conditions. According to the above aspect, the first imaging device (21-n1) captures the first moving image (Vn1) and the second imaging device (21-n2) captures the second moving image (Vn2) under common imaging conditions. Therefore, it is possible to generate a natural composite moving image (V) that can be perceived by a viewer as if the subject in the first real space (Rn1) and the subject in the second real space (Rn2) were located in a common space.

"Imaging conditions" are conditions related to the imaging of video by the (first/second) imaging device. Specifically, conditions that define the imaging range, such as the imaging direction or imaging magnification, are exemplified as "imaging conditions". The imaging direction is, for example, the pan direction (left/right direction) or the tilt direction (up/down direction). The imaging magnification is, for example, the zoom magnification. However, "imaging conditions" also include conditions that do not affect the imaging range itself, such as the focus position, aperture value (iris), exposure value, and white balance. "Imaging conditions" are also expressed as operating conditions that are subject to control by, for example, the imaging control unit (45).

"Operating under common imaging conditions" includes cases where the imaging conditions by the first imaging device (21-n1) and the imaging conditions by the second imaging device (21-n2) are completely the same, as well as cases where the imaging conditions by the first imaging device (21-n1) and the imaging conditions by the second imaging device (21-n2) are substantially the same. "When the imaging conditions are substantially the same" refers to cases where the imaging conditions between the first imaging device (21-n1) and the second imaging device (21-n2) are similar to each other to the extent that the difference in the imaging conditions is not perceived by the viewer, or cases where the difference between the imaging conditions by the first imaging device (21-n1) and the imaging conditions by the second imaging device (21-n2) is so small that it is caused by differences in characteristics between the two (for example, differences due to manufacturing errors).

For example, assuming that the first imaging device (21-n1) and the second imaging device (21-n2) are of the same model, common control data instructing the imaging conditions is transmitted to both the first imaging device (21-n1) and the second imaging device (21-n2), and as a result, the first imaging device (21-n1) and the second imaging device (21-n2) operate under common imaging conditions. Also, assuming that the first imaging device (21-n1) and the second imaging device (21-n2) are of different models, separate control data is transmitted to the first imaging device (21-n1) and the second imaging device (21-n2) so that the difference in imaging operation between the two models is eliminated, and as a result, the first imaging device (21-n1) and the second imaging device (21-n2) operate under common imaging conditions.

[Appendix 10]
A moving image compositing method according to one aspect (Supplementary Note 10) of the present disclosure includes acquiring a first moving image (Vn1) captured by a first imaging device (21-n1) in a first real space (Rn1) and a second moving image (Vn2) captured by a second imaging device (21-n2) in a second real space (Rn2), and performing a compositing process (S2) to generate a composite moving image (V) including an image (Gn1) of a first subject (Qn1) in the first moving image (Vn1) and an image (Gn2) of a second subject (Qn2) in the second moving image (Vn2), the compositing process (S2) including an adjustment process (S24, S25) to adjust the image (Gn1) of the first subject (Qn1) and the image (Gn2) of the second subject (Qn2) in accordance with a first imaging distance (Dn1) of the first imaging device (21-n1) and a second imaging distance (Dn2) of the second imaging device (21-n2).

[Appendix 11]
A program according to one aspect (Supplementary Note 11) of the present disclosure includes a video acquisition unit (41) that acquires a first video (Vn1) captured by a first imaging device (21-n1) in a first real space (Rn1) and a second video (Vn2) captured by a second imaging device (21-n2) in a second real space (Rn2), and an image (Gn1) of a first subject (Qn1) in the first video (Vn1) and an image (Gn2) of a second subject (Qn2) in the second video (Vn2). A program that causes a computer system to function as an image processing unit (42) that executes a synthesis process (S2) to generate a composite moving image (V), the synthesis process (S2) including an adjustment process (S24, S25) that adjusts an image (Gn1) of the first subject (Qn1) and an image (Gn2) of the second subject (Qn2) in accordance with a first imaging distance (Dn1) of the first imaging device (21-n1) and a second imaging distance (Dn2) of the second imaging device (21-n2).

100...video recording system, 200...terminal device, 20-n (20-1 to 20-3)...recording system, 21-n (21-1 to 21-3)...imaging device, 22-n (22-1 to 22-3)...sound collection device, 23-n (23-1 to 23-3)...communication device, 30...video synthesis system, 31...control device, 32...storage device, 33...communication device, 34...operation device, 35...playback device, 41...video acquisition unit, 42...image processing unit, 43...audio processing unit, 44...output processing unit, 45...imaging control unit, 421...distance determination unit, 422...subject extraction unit, 423...subject selection unit, 424...image adjustment unit, 425...image generation unit.

Claims (11)

a video acquisition unit that acquires a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space;
an image processing unit that executes a synthesis process to generate a synthetic moving image including an image of a first subject in the first moving image and an image of a second subject in the second moving image,
The video compositing system, wherein the compositing process includes an adjustment process of adjusting the image of the first subject and the image of the second subject in accordance with a first imaging distance of the first imaging device and a second imaging distance of the second imaging device. The moving image compositing system according to claim 1 , wherein the adjustment process includes a processing process for adjusting image parameters of the first object and image parameters of the second object in accordance with the first imaging distance and the second imaging distance. The processing includes a blurring process for blurring an image,
The moving image synthesis system according to claim 2 , wherein the image parameter is an amount of blurring. the first imaging distance is settable to a first distance and a second distance that are different from each other;
a difference between the second distance and a reference value that is greater than a difference between the first distance and a reference value;
4. The video compositing system of claim 3, wherein the image processing unit sets an amount of blur for the image of the first subject to a first setting value when the first imaging distance is the first distance, and sets an amount of blur for the image of the first subject to a second setting value that is greater than the first setting value when the first imaging distance is the second distance. The moving image compositing system according to claim 4 , wherein the reference value is an imaging distance corresponding to a reference subject which is any one of a plurality of subjects corresponding to each of a plurality of moving images including the first moving image and the second moving image. The moving image compositing system according to claim 5 , wherein the image processing unit selects the reference subject from the plurality of subjects according to a result of analyzing the plurality of moving images or a plurality of sounds respectively corresponding to the plurality of moving images. the adjustment process includes a superimposition process of superimposing the image of the first subject and the image of the second subject,
The moving image synthesis system according to claim 1 , wherein in the superimposing process, a front-back position between the image of the first subject and the image of the second subject is controlled in accordance with the first imaging distance and the second imaging distance. The image processing unit includes:
In the synthesis process, the synthetic video is generated including the image of the first subject and the image of the second subject, and a virtual object captured by a virtual imaging device in a virtual space;
The video compositing system according to claim 1 , wherein the adjustment process adjusts the virtual object in accordance with a virtual imaging distance of the virtual imaging device. 2. The moving image synthesis system according to claim 1, further comprising an imaging control unit that causes the first imaging device and the second imaging device to operate in parallel with each other in time under common imaging conditions. acquiring a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space;
executing a synthesis process for generating a synthetic moving image including an image of a first subject in the first moving image and an image of a second subject in the second moving image;
The moving image compositing method is realized by a computer system, wherein the compositing process includes an adjustment process of adjusting the image of the first subject and the image of the second subject according to a first imaging distance of the first imaging device and a second imaging distance of the second imaging device. a video acquisition unit that acquires a first video captured by a first imaging device in a first real space and a second video captured by a second imaging device in a second real space; and
an image processing unit that executes a synthesis process to generate a synthetic moving image including an image of a first subject in the first moving image and an image of a second subject in the second moving image;
A program for causing a computer system to function as
The synthesis process includes an adjustment process of adjusting the image of the first subject and the image of the second subject in accordance with a first imaging distance of the first imaging device and a second imaging distance of the second imaging device.

Images