JP7150501B2 - Control device, control method, and program - Google Patents

Description

The present invention relates to a system for the generation of virtual viewpoint content.

2. Description of the Related Art Conventionally, there are known a system in which the same scene is photographed by a plurality of cameras and the images are distributed while switching the cameras, and a system in which an object (subject) can be observed from a virtual viewpoint specified by a user. In such a system, if there is a difference in the color tone of the captured images captured by each camera, the distributed video or the generated video will be unnatural.

Patent Literature 1 discloses a method of optimizing setting parameters for correcting each captured image so that each captured image captured by three or more cameras has a uniform color tone.

JP 2009-122842 A

However, in the prior art, when individual differences between adjacent cameras are large, the individual differences cannot be absorbed by color correction alone. In other words, the conventional color correction has limitations in color matching between cameras.

SUMMARY OF THE INVENTION The present invention has been made in view of such problems, and an object of the present invention is to reduce the color difference between images captured by a plurality of cameras.

A control device according to an embodiment of the present invention comprises: determining means for determining an arrangement of a plurality of cameras in a multi-camera system for generating virtual viewpoint content based on images captured by the plurality of cameras; and an output means for outputting according to the determination by the determination means, wherein the determination means determines the arrangement of the plurality of cameras so that the color difference between the photographed images of the adjacent cameras is reduced .

ADVANTAGE OF THE INVENTION According to this invention, the color difference of each picked-up image image|photographed with several cameras can be made small.

1 is a block diagram showing a configuration example of an image processing apparatus according to a first embodiment; FIG. 1 is a block diagram showing a configuration example of an image processing system according to a first embodiment; FIG. 4 is a conceptual diagram showing an example of multi-camera arrangement in the first embodiment; FIG. FIG. 4 is a conceptual diagram showing an example of color chart imaging according to the first embodiment; FIG. 5 is a diagram showing an example of an inter-camera color difference evaluation table in the first embodiment; 4 is a flowchart of multi-camera selection processing in the first embodiment. 4 is a flowchart of inter-camera color difference evaluation processing in the first embodiment. FIG. 4 is a conceptual diagram showing an example of an important line-of-sight segment in the first embodiment; FIG. 10 is a diagram showing an example of a reference color and a color difference evaluation table of a camera in the second embodiment; FIG. 11 is a conceptual diagram showing an example of relative positional relationship of cameras in the second embodiment; FIG. 11 is a conceptual diagram showing an example of color differences between a reference color and a camera in the second embodiment; 10 is a flowchart of multi-camera selection processing according to the second embodiment. 10 is a flowchart of multi-camera selection processing according to the second embodiment. FIG. 13 is a diagram showing an example of a user interface for weighting designation in the third embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential for the solution of the present invention. In addition, the same configuration will be described by attaching the same reference numerals.

(First embodiment)
[Configuration of Image Processing Apparatus and Image Processing System]
FIG. 1 is a block diagram showing a configuration example of an image processing apparatus according to this embodiment.

A plurality of cameras (multi-cameras) selected by the image processing apparatus 100 are arranged as a plurality of cameras 212 of an image processing system (multi-camera system) 200, which will be described later. That is, the image processing device 100 functions as a multi-camera selection device in this embodiment.

The CPU 101 executes an operating system (OS) and various programs stored in the ROM of the main memory 102 and a hard disk drive (HDD) 105 using the RAM of the main memory 102 as a work memory. Each component is controlled via a system bus 114 such as a PCI (Peripheral Component Interconnect) bus. Furthermore, various programs including image processing, which will be described later, are executed.

CPU 101 accesses HDD 105 via system bus 114 and serial ATA interface (SATA I/F) 103 . It also accesses a network 115 such as a local area network (LAN) via the network I/F 104 .

The CPU 101 also communicates with devices connected to a serial bus interface (I/F) 108 and a serial bus 110 such as USB and IEEE1394. Then, image data to be subjected to image processing, which will be described later, is acquired from the image input device 113 including a card reader, and the processing result is output to the printer 109, for example, the processing result instructed by the user is printed. Image data to be subjected to image processing may be read from the HHD 105 or a server on the network 115 .

The CPU 101 also displays a user interface and processing results of processing, which will be described later, on the monitor 107 via the graphic accelerator 106 , and inputs user instructions via a keyboard 111 and a mouse 112 connected to the serial bus 110 .

FIG. 2 is a block diagram showing a configuration example of an image processing system according to this embodiment.

The image processing system 200 is a system (multi-camera system) that takes pictures and collects sounds using a plurality of cameras and microphones installed in facilities such as stadiums and concert halls. A plurality of cameras selected by the image processing apparatus 100 described above are arranged as a plurality of cameras 212 of the image processing system 200 .

Image processing system 200 includes sensor systems 210 a - 210 z, image computing server 300 , controller 400 , switching hub 280 and end user terminal 290 .

The controller 400 has a control station 410 and a virtual camera operation UI (user interface) 430 .

The control station 410 manages the operation state and controls parameter settings for each block that configures the image processing system 200 through the networks 410a to 410c, 280a, 280b, and 270a to 270y. Here, the network may be GbE (Gigabit Ethernet) or 10GbE conforming to the IEEE standard, which is Ethernet (registered trademark), or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. Also, the network is not limited to these, and may be another type of network.

First, the operation of transmitting the 26 sets of images and sounds of sensor systems 210a-210z from sensor system 210z to image computing server 300 will be described. In image processing system 200, sensor systems 210a-210z are connected by a daisy chain.

In this embodiment, the 26 sets of systems from the sensor system 210a to the sensor system 210z are referred to as the sensor system 210 without distinction unless otherwise specified. Similarly, the devices in each sensor system 210 will be referred to as a microphone 211, a camera 212, a pan head 213, and a camera adapter 220 without distinction unless otherwise specified. Although the number of sensor systems is described as 26 sets, this is merely an example, and the number is not limited to this. Also, the plurality of sensor systems 210 may not have the same configuration, and may be configured with different models of devices, for example. In this embodiment, unless otherwise specified, the term "image" will be described as including the concepts of moving images and still images. That is, the image processing system 200 of this embodiment can process both still images and moving images. Also, in the present embodiment, an example in which the virtual viewpoint content provided by the image processing system 200 includes a virtual viewpoint image and a virtual viewpoint sound will be mainly described, but the present invention is not limited to this. For example, virtual viewpoint content does not have to include audio. Further, for example, the sound included in the virtual viewpoint content may be sound collected by a microphone closest to the virtual viewpoint. In addition, in this embodiment, for the sake of simplification of explanation, the description of audio is partially omitted, but it is assumed that basically both images and audio are processed.

Sensor systems 210 each have one camera 212 . That is, the image processing system 200 has multiple cameras 212 for photographing the object from multiple directions. Note that the plurality of cameras 212 will be described using the same reference numerals, but may differ in performance and model. A plurality of sensor systems 210 are connected by a daisy chain. With this connection mode, it is possible to reduce the number of connection cables and labor for wiring work when the resolution of captured images is increased to 4K or 8K and the capacity of image data is increased due to increased frame rates.

Note that the connection configuration is not limited to this, and a star network configuration in which each sensor system 210 is connected to the switching hub 280 and data is transmitted and received between the sensor systems 210 via the switching hub 280 may be employed.

Moreover, although FIG. 2 shows a configuration in which all of the sensor systems 210 are cascade-connected so as to form a daisy chain, the present invention is not limited to this. For example, the plurality of sensor systems 210 may be divided into several groups, and the sensor systems 210 may be daisy-chained for each divided group. Then, the camera adapter 220 that is the end of the division unit may be connected to the switching hub and input the image to the image computing server 300 . Such a configuration is particularly effective in stadiums. For example, a stadium may have multiple floors and the sensor system 210 may be deployed on each floor. In this case, it is possible to input to the image computing server 300 for each floor or for each half circumference of the stadium. System flexibility can be achieved.

In addition, control of image processing in the image computing server 300 is switched depending on whether there is one camera adapter 220 or two or more terminal camera adapters 220 that input images to the image computing server 300 . That is, control is switched depending on whether the sensor system 210 is divided into multiple groups. If there is only one camera adapter 220 for image input, the image of the entire circumference of the stadium is generated while transmitting the image through the daisy chain connection. is taken. That is, if the sensor system 210 is not divided into groups, it is synchronized. However, if there are a plurality of camera adapters 220 that input images to the image computing server 300, the delay from the time an image is captured until it is input to the image computing server 300 is different for each lane (route) of the daisy chain. Different cases are conceivable. That is, when the sensor system 210 is divided into a plurality of groups, the timing at which the image data for the entire circumference is input to the image computing server 300 may not be synchronized. Therefore, in the image computing server 300, it is necessary to perform subsequent image processing while checking the collection of image data by synchronous control that waits until the image data of all circumferences are ready and synchronizes them.

In this embodiment, the sensor system 210a has a microphone 211a, a camera 212a, a platform 213a, and a camera adapter 220a. Note that the sensor system 210a is not limited to this configuration, and may have at least one camera adapter 220a and one camera 212a or one microphone 211a. Further, for example, the sensor system 210a may be configured with one camera adapter 220a and multiple cameras 212a, or may be configured with one camera 212a and multiple camera adapters 220a. Also, the sensor system 210 may include devices other than the microphone 211a, the camera 212a, the platform 213a, and the camera adapter 220a. Alternatively, the camera 212 and the camera adapter 220 may be integrated. Furthermore, at least part of the functionality of camera adapter 220 may be provided by front-end server 330 . In this embodiment, the sensor systems 210b to 210z have the same configuration as the sensor system 210a, so description thereof will be omitted. The configuration of each sensor system 210 is not limited to the same as that of the sensor system 210a, and the configuration of each sensor system 210 may be different.

The sound collected by the microphone 211a and the image captured by the camera 212a are subjected to image processing in the camera adapter 220a and then transmitted to the next sensor system 210b through the daisy chain 270a. Similarly, the sensor system 210b transmits the collected sound and the captured image together with the image and sound acquired from the sensor system 210a to the next sensor system 210c. By continuing such operations, images and sounds acquired by sensor systems 210 a - 210 z are transmitted from sensor system 210 z through network 280 b to switching hub 280 and then to image computing server 300 .

Although the cameras 212a to 212z and the camera adapters 220a to 220z are separated in this embodiment, they may be integrated in the same housing. In that case, the microphones 211 a to 211 z may be built into the integrated camera 212 or may be connected to the outside of the camera 212 .

Next, the configuration and operation of image computing server 300 will be described. The image computing server 300 of this embodiment processes image and audio data obtained from the terminal sensor system 210z. The image computing server 300 has a front end server 330 , a database 350 (hereinafter also referred to as DB), a back end server 370 and a time server 390 .

The time server 390 has a function of distributing time and synchronization signals, and distributes the time and synchronization signals to the sensor systems 210a to 210z via the switching hub 280. FIG. The camera adapters 220a to 220z that have received the time and synchronization signal Genlock the cameras 212a to 212z based on the time and the synchronization signal, and perform image frame synchronization. That is, the time server 390 synchronizes the photographing timings of the multiple cameras 212 . Accordingly, since the image processing system 200 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, it is possible to suppress deterioration in the quality of the virtual viewpoint image due to the deviation of the capturing timing. In this embodiment, the time server 390 manages the time synchronization of a plurality of cameras 212, but this is not limitative, and each camera 212 or each camera adapter 220 independently performs time synchronization processing. you can go

The front-end server 330 reconstructs the segmented transmission packets and converts the data format for images and sounds acquired from the terminal sensor system 210z. Then, it writes to the database 350 according to the camera identifier, data type, and frame number.

The back-end server 370 accepts the designation of the viewpoint input by the user from the virtual camera operation UI 430, reads the corresponding image and audio data from the database 350 based on the accepted viewpoint, and renders the virtual viewpoint image. to generate

Note that the configuration of the image computing server 300 is not limited to this. For example, at least two of the front-end server 330, database 350, and back-end server 370 may be integrated. Also, at least one of the front-end server 330, the database 350, and the back-end server 370 may be included in multiple numbers. Also, devices other than those described above may be included at any location within image computing server 300 . Furthermore, at least part of the functions of the image computing server 300 may be included in the end user terminal 290 or the virtual camera operation UI 430 .

The backend server 370 transmits the generated virtual viewpoint image to the end user terminal 290 . A user operating the end-user terminal 290 can view images and listen to audio according to the specified viewpoint. That is, the backend server 370 generates virtual viewpoint content based on captured images (multi-viewpoint images) captured by the plurality of cameras 212 and viewpoint information. More specifically, the back-end server 370 generates a virtual viewpoint based on image data of a predetermined region extracted from images captured by the cameras 212 by the camera adapters 220 and a viewpoint specified by user operation. Generate content. The backend server 370 then provides the generated virtual viewpoint content to the end user terminal 290 . Note that the virtual viewpoint content may be generated by a device other than the backend server 370 included in the image computing server 300 , or may be generated by the controller 400 or the end user terminal 290 .

The virtual viewpoint content in this embodiment is content that includes a virtual viewpoint image as an image obtained when an object is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing a view at a specified viewpoint. A virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily designated by the user. The virtual viewpoint image also includes an image corresponding to a viewpoint specified by the user from among a plurality of candidates and an image corresponding to a viewpoint automatically specified by the device. Note that, in the present embodiment, an example in which voice data (audio data) is included in the virtual viewpoint content is mainly described, but voice data does not necessarily have to be included. Also, the back-end server 370 may store virtual viewpoint images in H.264, for example. 264 or HEVC, and then transmitted to the end user terminal 290 using the MPEG-DASH protocol. Alternatively, the virtual viewpoint image may be transmitted to the end user terminal 290 uncompressed. In particular, the former, which performs compression encoding, assumes that the end user terminal 290 is a smart phone or a tablet. On the other hand, the latter assumes a display capable of displaying uncompressed images. That is, the image format can be switched according to the type of the end user terminal 290. FIG. Also, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission methods may be used.

Thus, the image processing system 200 has three functional domains: an image collection domain, a data storage domain, and an image generation domain. The image collection domain includes sensor systems 210-210z. The data storage domain includes database 350 , front end server 330 and back end server 370 . The video production domain includes virtual camera manipulation UI 430 and end user terminal 290 . Note that the configuration is not limited to this configuration, and for example, the virtual camera operation UI 430 can directly acquire images from the sensor systems 210a to 210z. However, in this embodiment, rather than acquiring images directly from the sensor systems 210a-210z, a data storage function is placed in the middle. Specifically, the front-end server 330 converts the image data and audio data generated by the sensor systems 210a to 210z and the meta information of these data into the common schema and data type of the database 350. FIG. As a result, even if the cameras 212 of the sensor systems 210a to 210z are changed to cameras of other models, the front-end server 330 can absorb the changed differences and register them in the database 350. FIG. With such a configuration, it is possible to reduce the risk that the virtual camera operation UI 430 will not operate properly when the camera 212 is changed to a camera of another model.

Also, the virtual camera operation UI 430 does not access the database 350 directly, but accesses it via the backend server 370 . The back-end server 370 performs common processing related to image generation processing, and the virtual camera operation UI 430 absorbs the differences of the application related to the operation UI. With such a configuration, in the development of the virtual camera operation UI 430, it is possible to focus on the development of the functional requirements of the UI operation device and the UI for operating the virtual viewpoint image to be generated. Also, the backend server 370 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 430 . With such a configuration, the backend server 370 can flexibly respond to requests from the virtual camera operation UI 430 .

As described above, in the image processing system 200, the backend server 370 generates a virtual viewpoint image based on the images captured by the multiple cameras 212 for capturing images of the object from multiple directions. Note that the image processing system 200 according to the present embodiment is not limited to the physical configuration described above, and may be configured logically. Further, this embodiment can also be applied to a case in which captured images are transmitted to the end-user terminal 290 by switching between the plurality of cameras 212 without generating a virtual viewpoint image, for example.

[Multi camera selection]
As described above, a virtual viewpoint image is generated based on a plurality of captured images captured at the same timing by the image processing system 200 . Therefore, if there is a difference in the characteristics (brightness and color) of the images captured by the plurality of cameras 212, the virtual viewpoint image generated when the virtual viewpoint is moved becomes unnatural.

Therefore, in the present embodiment, in the image processing system 200, a plurality of cameras (hereinafter also referred to as multi-cameras) 212 are selected so that the color difference between the cameras is small, and the relative positional relationship between the selected cameras is determined. do.

When the image processing system 200 has more than the number (M) of cameras (L) required to generate a virtual viewpoint image, the M cameras are used so that the color difference between the cameras becomes small. must be selected.

The number of combinations of M (M≦L) cameras selected from L cameras is given by equation (1).

FIG. 3 is a conceptual diagram showing an example of a multi-camera arrangement in this embodiment, showing an example in which M cameras are arranged in a circle. Note that the M cameras are not limited to the arrangement shown in the figure, as long as they capture the object from different directions. For example, the M cameras may be arranged in a circle, an ellipse, or a line. In this embodiment, for the sake of simplification, a circle will be described as an example.

The total number of circular permutations for arranging M cameras in a circle is given by Equation (2).

Therefore, the total number of circular permutations for selecting M (M≦L) multi-cameras from L cameras and arranging the selected M cameras in a circle is given by Equation (3).

That is, since there are as many candidates for a multi-camera set in which M cameras are arranged in a circle as the number indicated by equation (3), the color difference between cameras is evaluated for each candidate, and the multi-camera set with the smallest color difference is evaluated. Decide on a set of cameras. Note that the multi-camera set represents the relative positional relationship of the M cameras. Thereby, the relative positional relationship between the plurality of cameras 212 arranged in the image processing system 200 and each camera can be obtained.

[Capturing a color chart and extracting color patches]
In order to obtain the color difference between cameras, the same color chart is photographed by different cameras.

FIG. 4 is a conceptual diagram showing an example of photographing a color chart according to this embodiment, and shows an example of photographing a color chart 500 with L cameras.

A color chart 500 is composed of a plurality of color patches. For each color patch, device-independent color values such as CIE XYZ and CIE Lab are measured in advance by a colorimeter or the like. The color chart 500 as an object may be placed in an observation booth in a darkroom, or may be placed at the position of the object in the shooting scene. Further, the color chart 500 may be photographed simultaneously by each camera from different directions, or may be photographed by directing the cameras one by one.

The image processing apparatus 100 obtains, from the image input device 113, captured images of the L cameras 501, 502, 503, 504, . Extract the color value of . Here, the extraction of the color value of each color patch may be automatic or manual.

Next, the image processing apparatus 100 converts the color value of each color patch from the color space of the captured image to a device-independent color space such as CIE XYZ or CIE Lab. If the color space of the captured image is a "defined color space" such as ITU-R BT.709, ITU-R BT.2020, DCI-P3, ACES, sRGB, Adobe RGB, device-independent color space can be converted to Also, if the captured image is a RAW image, it is converted into a "defined color space" by RAW development processing. In addition, the method of acquiring images captured by the L cameras 501, 502, 503, 504, . may be acquired via the network 115.

[Evaluation of color difference between cameras]
FIG. 5 shows an example of an inter-camera color difference evaluation table in this embodiment, showing an example of inter-camera color difference evaluation for a multi-camera set in which M cameras are arranged in a circle.

Each row of the inter-camera color difference evaluation table 600 indicates the color of each color patch of the color chart 500 . The inter-camera color difference evaluation table 600 includes 24 rows of data when the color chart is ColorChecker Classic (24 colors), for example. The color chart 500 may be a self-made color chart or a commercially available color chart such as ColorChecker Digital SG (140 colors).

The Lab value 610 of each color patch of the color chart 500 stores the Lab value obtained by measuring each color patch of the color chart 500 with a colorimeter.

The Lab value 620 of each camera for each color patch of the color chart 500 is obtained by extracting and converting the color of each color patch from the captured image of each camera as a candidate for a multi-camera set in which M cameras are arranged in a circle. Lab value is stored.

The image processing apparatus 100 evaluates the inter-camera color difference in the multi-camera set and obtains the multi-camera set with the smallest color difference.

The total number of combinations in which two cameras are selected from M cameras and the color difference is obtained is given by equation (4).

If the color difference is evaluated with respect to the total number of combinations of formula (4), the color difference of the entire multi-camera can be reduced. In this embodiment, for the sake of simplification, an example of evaluating the color difference between adjacent cameras will be described. do.

The color difference 630 between adjacent cameras for the i-th color patch in the circularly arranged multi-cameras can be obtained by Equation (5). Here, the Lab value of the camera 212a is (L ^* ₁ (i),a ^* ₁ (i),b ^* ₁ (i)), and the Lab value of the camera 212b is (L ^* ₂ (i),a ^* ₂ ( i),b ^* ₂ (i)).

Similarly, ΔE ₂₃ ( _i ), _{ΔE 34} (i), . Ask.

Here, in this embodiment, ΔE76 (CIE76) is used as the color difference formula for simplification, but ΔE94 (CIE94), ΔE00 (CIEDE2000), or the like may be used.

Furthermore, the simple average color difference 640 between adjacent cameras for the i-th color patch in the candidates of the multi-camera set in which M cameras are arranged in a circle can be obtained by Equation (6).

On the other hand, since the color difference 630 between adjacent cameras for each color patch is obtained by equation (5), the color difference 630 between adjacent cameras for adjacent cameras (j, j+1) in candidates for a multi-camera set in which M cameras are arranged in a circle is An average color difference 650 can also be determined. The average color difference between adjacent cameras 650 can selectively store a simple average color difference (ΔE _j(j+1) ) _a or a weighted average color difference (ΔE _j(j+1) ) _w depending on the purpose of use.

In photography, if the color of an object cannot be predicted in advance, it is possible to evaluate the color difference between cameras by simple averaging. Simple average color difference between adjacent cameras (ΔE _j(j+1) ) _a for adjacent camera (j, j+1) is the color difference between adjacent cameras (j, j+1) for i-th color patch as ΔE _j(j+1) ( i), where N is the total number of color patches, it can be obtained by equation (7).

On the other hand, when filming takes place in facilities such as stadiums and concert halls, important colors (lawn, uniforms, advertisements, skin colors, costumes, musical instrument colors, etc.) can be known in advance at the rehearsal stage. many. Therefore, it is possible to perform inter-camera color difference evaluation in which a specific color is weighted. For example, a color patch close to the important color is specified from all color patches of the color chart 500, and the color difference of the adjacent camera (j, j+1) with respect to the specified color patch is given greater weight than other color patches. The weighted average color difference between adjacent cameras (ΔE _j(j+1) ) _w for the adjacent camera ( _j , j+1) can be obtained by Equation (8), where wi is the weight for the i-th color patch. Let ΔE _j(j+1) (i) be the color difference of the adjacent camera (j, j+1) with respect to the i-th color patch, and N be the total number of color patches.

In addition, depending on the calculation method of the average color difference between adjacent cameras 650, a simple average color difference or a weighted average color difference is also used for the color difference evaluation result 660 between cameras. That is, when the simple average of the simple average color difference (ΔE _j(j+1) ) _a is stored (equation (9)), and when the simple average of the weighted average color difference (ΔE _j(j+1) ) _w is stored There is a case (equation (10)) where

[Determination of multi-camera placement]
In order to determine the optimum multi-camera set, the inter-camera color difference evaluation result 660 calculated by the formula (9) or the formula (10) among all candidates for the multi-camera set shown in the formula (3) It is necessary to find a set of multi-cameras that minimizes .

If a multi-camera set that minimizes the inter-camera color difference evaluation result 660 can be determined, the relative positional relationship between the cameras 212 arranged in the image processing system 200 and each camera can be uniquely determined. However, since the multi-camera set is determined by circular permutation, there remains a degree of freedom to rotate the overall arrangement of the cameras while maintaining the relative positional relationship of each camera.

Therefore, the optimal multi-camera arrangement is determined so that the section where the line of sight that is important for shooting overlaps with the section where the average color difference 650 between adjacent cameras is small on the circumference of the multi-camera. Sections where important lines of sight gather for shooting include, for example, the main stand side of a stadium, the stage front side of a concert hall, and the like.

The image processing apparatus 100 uses the distribution of the average color difference 650 between adjacent cameras to identify a section in which the average color difference of the adjacent camera (j, j+1) is small, and overlaps the section where the line of sight that is important for shooting is specified. to determine the placement of the multi-camera.

[Process flow of multi-camera selection]
FIG. 6 is a flowchart of multi-camera selection processing in this embodiment.

A series of processes shown in the flowchart is performed by the CPU 101 of the image processing apparatus 100 loading a control program stored in a storage device such as the main memory 102 or the HDD 105 into a RAM and executing the control program. Alternatively, some or all of the functions of the steps in the flowcharts may be realized by hardware such as ASICs and electronic circuits. The symbol "S" in the description of each process means a step in the flowchart. The same applies to other flowcharts.

First, in S1001, captured images of the color chart 500 captured by each of the L cameras are acquired.

Next, in S1002, the color value of each color patch of the color chart 500 is extracted from each photographed image.

Next, in S1003, the color values of each extracted color patch are converted into Lab values.

Next, in S1004, M (M≦L) multi-cameras are selected from the L cameras, and candidates for a multi-camera set in which the M cameras are arranged in a circle are determined. That is, the CPU 101 of the image processing apparatus 100 functions as candidate determination means for determining candidates for a multi-camera set. Specifically, multi-camera set candidates for the total number of circular permutations in which M cameras represented by Equation (3) are arranged in a circle are sequentially determined.

Here, an example of generating four multi-camera set candidates (L=5, M=4) from five cameras will be described.

Combinations for selecting four multi-cameras from five cameras (1, 2, 3, 4, 5) are the following five (= ₅ C ₄ ) from equation (1).
(1,2,3,4)
(1,2,3,5)
(1,2,4,5)
(1,3,4,5)
(2,3,4,5)

Further, the candidates (circular permutation) of the multi-camera set in which the four multi-cameras (1, 2, 3, 4) are arranged in a circle are the following six (=3!) from equation (2).
(1,2,3,4)
(1,2,4,3)
(1,3,2,4)
(1,3,4,2)
(1,4,2,3)
(1,4,3,2)

Similarly, for other multi-camera combinations (1,2,3,5), (1,2,4,5), (1,3,4,5), (2,3,4,5) However, candidates for multi-camera pairs are obtained. Then, the total number of candidates for a set of four multi-cameras generated from five cameras (1, 2, 3, 4, 5) is 30 as shown in Equation (3).

Next, in S<b>1005 , inter-camera color difference evaluation results 660 are calculated for the M multi-camera set candidates determined in S<b>1004 and stored in the inter-camera color difference evaluation table 600 . That is, the CPU 101 of the image processing apparatus 100 functions as color difference evaluation means for evaluating inter-camera color differences. Specifically, the average color difference of color patches between adjacent cameras arranged in a circle is calculated, and the simple average of the calculated average color differences is calculated as the inter-camera color difference evaluation result 660 for the candidates for the multi-camera set. do. It should be noted that there are a plurality of methods for obtaining the inter-camera color difference evaluation result 660 depending on the purpose of use, such as Equations (9) and (10) described above.

Next, in S1006, it is determined whether or not the processing from S1004 to S1005 has been performed for all candidates of the multi-camera set. If not completed, the process returns to S1004. On the other hand, in the case of end, the process proceeds to S1007.

Next, in S1007, a multi-camera set with the smallest inter-camera color difference evaluation result 660 is determined among all candidates for the multi-camera set. In other words, the CPU 101 of the image processing apparatus 100 functions as a set determination unit that determines a set of multi-cameras. Here, the relative positional relationship of each camera in the multi-camera is determined.

Next, in S1008, the information of the important line-of-sight section in which the lines of sight that are important for shooting are gathered is acquired, and the arrangement of the multi-cameras is determined based on the important line-of-sight section. That is, the CPU 101 of the image processing apparatus 100 functions as an arrangement determination unit that determines the arrangement of the set of multi-cameras. Also, the important line-of-sight segment means a specific segment captured by a multi-camera.

Here, an example is shown in which the information of the important line-of-sight section is information that "the important camera is located between the p-th and the q-th among the M multi-cameras that have already been determined".

FIG. 8 is a conceptual diagram showing an example of important line-of-sight segments in this embodiment. Assuming that the important cameras are between the p-th and the q-th, the number of cameras in that section is (q−p+1). Therefore, in the average color difference between adjacent cameras 650, a simple average of the average color differences between adjacent cameras at (qp) locations for (qp+1) adjacent cameras is sequentially obtained. Then, ave(ΔE _r→r+qp ) _a or ave(ΔE _r→r+qp ) Identify the interval where _w is minimum.

In addition, if the importance distribution (or weight coefficient) for the average color difference between adjacent cameras is known in advance for the p to q-th important cameras, the weight for the average color difference between each adjacent camera is given by v _j (j ∈[r, r+qp-1]). That is, equations (13) and (14) for obtaining the weighted average of the average color difference between adjacent cameras can be used instead of the equations (11) and (12) for obtaining the simple average of the average color difference between adjacent cameras. .

Then, the r-th camera is found in the section where ave(ΔE _r→r+qp ) _a and ave(ΔE _r→r+qp ) _w , which are simple averages of average color differences between adjacent cameras, are minimum. Alternatively, the r-th camera is found in the section where weighted(ΔE _r→r+qp ) _a and weighted(ΔE _r→r+qp ) _w , which are weighted averages of average color differences between adjacent cameras, are minimum. The multi-camera arrangement can be determined by rotating the circular array of multi-cameras so that the r-th camera coincides with the p-th camera.

Note that the importance distribution for the important line-of-sight segment and the average color difference between adjacent cameras can be specified by a setting file, or can be specified by the user via a user interface.

FIG. 7 is a flowchart of inter-camera color difference evaluation (S1005) in this embodiment.

FIG. 7A shows an example of processing for evaluating the simple average color difference between all cameras.

In S1009, the inter-camera color difference of each color patch is obtained for all combinations of two cameras out of the M cameras, and the inter-camera simple average color difference for all color patches is obtained. Then, the inter-camera color difference evaluation result 660 is calculated by simply averaging the total number of inter-camera simple average color differences shown in Equation (4).

FIG. 7B shows an example of processing for evaluating the simple average color difference between adjacent cameras.

In S1010, the color difference between adjacent cameras of each color patch is obtained by Equation (5) instead of obtaining the color difference between cameras for all combinations. Then, the simple average color difference between adjacent cameras is obtained by equation (7), and the color difference evaluation result 660 between cameras is calculated according to equation (9).

FIG. 7(c) shows an example of processing for evaluating the weighted average color difference between all cameras.

In S1011, the inter-camera color difference of each color patch is obtained for all combinations of two cameras out of the M cameras, the weighting factor for each color patch is obtained, and the weighted average inter-camera color difference for all color patches is obtained. Ask for Then, an inter-camera color difference evaluation result 660 is calculated by simply averaging the total number of weighted average inter-camera color differences shown in Equation (4).

FIG. 7D shows an example of processing for evaluating the weighted average color difference between adjacent cameras.

In S1012, the color difference between adjacent cameras of each color patch is obtained by Equation (5) instead of obtaining the color difference between cameras for all combinations. Then, a weighting factor for each color patch is obtained, a weighted average color difference between adjacent cameras is obtained by equation (8), and an inter-camera color difference evaluation result 660 is calculated according to equation (10).

Note that the weighting coefficient for each color patch can be specified by a setting file or by a user interface.

In this embodiment, an example of sequentially evaluating multi-camera set candidates has been shown, but other solutions may be applied as a combinatorial optimization problem.

[effect]
As described above, according to the present embodiment, multi-cameras are selected so that the inter-camera color difference is small, and a set of multi-cameras is determined. can be made smaller. Also, the arrangement of the multi-cameras can be determined according to the important line-of-sight segment.

(Second embodiment)
In the first embodiment, M (M≦L) multi-cameras are selected from L cameras, and the color difference between cameras is calculated for candidates of multi-camera sets (circular permutation) in which the M cameras are arranged in a circle. A method for determining the set of multi-cameras with the smallest evaluation result 660 has been described. The method of the first embodiment is effective as a method of determining the optimal multi-camera arrangement, but as the number of cameras (L and M) increases, the total number of multi-camera set candidates increases rapidly. It takes a lot of processing time.

In this embodiment, a method for determining an optimal multi-camera arrangement within a practical time even if the number of cameras is large will be described. Note that the first embodiment and the present embodiment may be switched or combined according to the number of cameras and the processing capability of the image processing apparatus 100 .

[Multi camera selection]
In the first embodiment, the inter-camera color difference evaluation is performed after selecting M (M≦L) multi-camera candidates from L cameras. On the other hand, in the present embodiment, the L cameras are ranked according to the color difference from the reference color, and the M multi-cameras are selected in descending order of color difference.

There are a method based on the color chart and a method based on the master camera for ranking according to the color difference from the reference color.

A method based on a color chart is effective when emphasis is placed on faithful color reproduction. If the color adjustment of the camera is insufficient, or if the image is created in the camera, the color difference may become large.

On the other hand, the method based on the master camera is effective when importance is attached to color matching between cameras. When cameras of the same model are used as multiple cameras, the characteristics of each camera are similar, so even if an image is created in the camera, the color difference will be small. In cases where a master camera has been determined by convention, such as when using a combination of cameras of different models as a multi-camera, or when compatibility with conventional image creation is emphasized, the master camera that is the standard Just use the camera. If a master camera has not been determined, a master camera may be selected based on a color chart or an average value of a plurality of cameras, and multi-cameras may be determined based on that master camera. That is, based on the color difference between the captured images of the color chart captured by L cameras and the measured values of the color chart, a specific camera with the smallest color difference may be determined as the master camera. Alternatively, a specific camera that captures a captured image that is closest to the average value of the color difference between each captured image of the color chart captured by the L cameras and the measured value of the color chart may be determined as the master camera. Alternatively, instead of selecting the master camera based on the color chart, the master camera may be selected based on the average color difference or weighted average color difference of a plurality of cameras.

FIG. 9 shows an example of a reference color and a color difference evaluation table of cameras in this embodiment, and shows an example of color difference evaluation of L cameras.

Each row of the color difference evaluation table 700 between the reference color and the camera indicates the color of each color patch of the color chart 500. For example, if the color chart is ColorChecker Classic (24 colors), there are 24 rows. The color chart 500 may be a self-made color chart or a commercially available color chart such as ColorChecker Digital SG (140 colors).

The Lab value of the reference color 710 stores the Lab value of the reference color of the color chart or the reference color of the master camera. In the case of the reference color of the color chart, the Lab value obtained by measuring each color patch of the color chart 500 with a colorimeter is stored. On the other hand, in the case of the reference color of the master camera, the Lab value obtained by extracting and converting the color of each color patch from the captured image of the master camera that captured the color chart 500 is stored.

The Lab value 720 of each camera for each color patch of the color chart 500 stores the Lab value obtained by extracting and converting the color of each color patch from the images captured by each of the L cameras.

A camera color difference 730 for the i-th color patch in each camera can be obtained by equation (15). Here, the Lab value of the reference color is (L ^* ₀ (i), a ^* ₀ (i), b ^* ₀ (i)), and the Lab value of camera 1 is (L ^* ₁ (i), a ^* ₁ ( i), b ^* ₁ (i)).

Similarly, camera color differences 730 for the _L cameras are obtained as ΔE ₂ ( _i ), ΔE ₃ (i), .

Here, in this embodiment, ΔE76 (CIE76) is used as the color difference formula for simplification, but ΔE94 (CIE94), ΔE00 (CIEDE2000), or the like may be used.

Since the color difference for each camera for each color patch is determined by equation (15), it is also possible to determine the average camera color difference 740 for the j-th camera among the L cameras. The camera average color difference 740 can selectively store the simple average color difference (ΔE _j ) _a or the weighted average color difference (ΔE _j ) _w depending on the purpose of use.

Before photographing, if the color of the object cannot be predicted in advance, simple average camera color difference evaluation can be performed. Camera simple average color difference (ΔE _j ) _a for the j-th camera among the L cameras is ΔE _j (i) for the camera j's camera color difference for the i-th color patch, and N for the total number of color patches in the color chart 500. Then, it can be obtained by equation (16):

On the other hand, when filming takes place in facilities such as stadiums and concert halls, important colors (lawn, uniforms, advertisements, skin colors, costumes, musical instrument colors, etc.) can be known in advance at the rehearsal stage. many. Therefore, it is possible to perform camera color difference evaluation in which a specific color is weighted. For example, a color patch close to the important color is specified from all the color patches of the color chart 500, and the camera color difference for the specified color patch is given greater weight than other color patches. The camera-weighted average color difference (ΔE _j ) _w for the j-th camera among the L cameras can be obtained by Equation (17). Let ΔE _j (i) and w _i denote the camera color difference of camera j for the i-th color patch and its weight, and N be the total number of color patches in the color chart 500 .

Next, since the camera average color difference 740 is obtained by the equation (16) or (17), the camera average color difference 740 is used for ranking. If M cameras are selected in descending order of camera average color difference 740, a multi-camera capable of reproducing colors close to the reference color (the reference color of the color chart or the reference color of the master camera) can be constructed.

[Determination of multi-camera placement]
Up to this point, M multi-cameras have been selected and the cameras have been ranked (in descending order of camera average color difference 740 with respect to the reference color), but the relative positional relationship of each camera has not been determined. do not have. A method for reducing the color difference between cameras in a multi-camera arrangement in which M cameras are arranged in a circle will be described below.

FIG. 10 is a conceptual diagram showing an example of relative positional relationships of multi-cameras in this embodiment. For the sake of simplicity, the six cameras 800-805 arranged in a circle indicate color values 1-6, respectively.

FIG. 10A shows an example in which six cameras 800 to 805 are arranged clockwise in ascending order of color values.

In FIG. 10A, the color value difference between cameras 800 and 801, 801 and 802, 802 and 803, 803 and 804, and 804 and 805 is 1, but the difference between cameras 805 and 800 is 5. is. Here, the sum of the color value differences is 10, and the simple average of the color value differences is 10/6 (≈1.667).

On the other hand, FIG. 10B shows an example in which the six cameras 800 to 805 are arranged so that the distance from the camera 800 decreases in ascending order of color values.

In FIG. 10B, the color value difference between cameras 800 and 801 and 805 and 804 is 1, but the color value difference between cameras 801 and 803, 803 and 805, 804 and 802, and 802 and 800 is 2. Here, the sum of the color value differences is 10, and the simple average of the color value differences is 10/6 (≈1.667).

10(a) and 10(b), the simple average of the difference in color values is the same value. FIG. 10(b), in which changes to , is preferable.

FIG. 11 is a conceptual diagram showing an example of the color difference between the reference color and the camera in this embodiment.

Although FIG. 10 shows an example in which the color values are one-dimensional, the actual color values are three-dimensional as shown in FIG. FIG. 11 shows an example in which the camera 800 is used as the master camera. As shown in the figure, the color difference between the reference color (master camera) and each camera is defined by the Euclidean distance, so even if each camera is arranged in order of decreasing color difference with respect to the reference color, it is guaranteed that the color difference between cameras will be small. no. For example, in FIG. 11, the color difference between the cameras 803 and 804 is greater than the color difference between the cameras 800 and 803 and the color difference between the cameras 800 and 804 .

In order to ensure that the color difference between cameras is small, use the master camera as a reference to find the camera with the smallest color difference, then use that camera as the reference to find the camera with the smallest color difference, and so on. All you have to do is search for a camera that has

Therefore, when the ranking of all the M cameras (in descending order of inter-camera color difference) is completed, as shown in FIG. Arrange the cameras in order.

Moreover, when the number of cameras is large, M cameras may be divided into P (P<M) groups and arranged. Here, an example will be described in which 30 multi-cameras are divided into 6 groups (M=30, P=6) and arranged. It is also assumed that each group is assigned 5 (=M/P) cameras.

The first group can be obtained by performing color difference evaluation of 30 multi-cameras in the reference color and camera color difference evaluation table 700 . After the reference color of the master camera is stored as the Lab value 710 of the reference color and the camera average color difference 740 is obtained, the camera average color difference 740 is used for ranking. Five cameras (including the master camera) with the smallest camera average color difference 740 are selected as the first group. For the camera average color difference 740, the simple average color difference (ΔE _j ) _a or the weighted average color difference (ΔE _j ) _w can be selected according to the purpose of use.

Next, in the camera average color difference 740 based on the master camera, the camera with the sixth smallest camera average color difference is set as the representative camera of the second group. The second group can be obtained by evaluating the color difference of 25 cameras, excluding the 5 cameras of the first group, from the 30 multi-cameras in the reference color and camera color difference evaluation table 700 . After the reference color of the representative camera of the second group is stored in the Lab value 710 of the reference color and the camera average color difference 740 is obtained, the cameras are ranked according to the camera average color difference 740 . Five cameras (including the representative camera of the second group) are selected as the second group from the smallest camera average color difference 740 .

Similarly, the representative cameras of the third to sixth groups and the five cameras of each group are sequentially selected.

Next, the cameras in each of the first to sixth groups are assigned to the positions of the cameras 800 to 805 in FIG. 10(b). The five cameras in each group are arranged on the same circle so that the distances from the camera 800 become shorter in order of decreasing average color difference with the representative camera of each group. Note that once the group to which the camera belongs is determined, for example, the position of the camera 800 is assigned to the first group, the assigned position of the camera is automatically determined. Therefore, each camera may be arranged when the group to which each camera belongs is decided, instead of arranging each camera after all the cameras of the group are selected.

By the above processing, it was possible to uniquely determine the relative positional relationship of each camera in the multi-camera. Placement should be confirmed. In other words, the section where the line of sight that is important for shooting (the main stand side of a stadium, the front side of the stage in a concert hall, etc.) overlaps with the section where the average color difference 650 between adjacent cameras on the circumference of the multi-camera is small. Determine multi-camera placement.

[Process flow of multi-camera selection]
12 and 13 are flowcharts of multi-camera selection processing in the second embodiment. The same reference numerals are assigned to the same processing as in the first embodiment, and the description thereof is omitted, and the points different from the first embodiment will be briefly described.

FIG. 12 shows an example of processing for sequentially searching for cameras with small inter-camera color differences using the master camera as a reference and selecting a multi-camera.

Through the processing from S1001 to S1003, similarly to the first embodiment, captured images of the color chart 500 captured by each of the L cameras are acquired, and the color values of each color patch are converted into Lab values. Then, each color patch of the color chart 500 is stored in the Lab value 720 of each camera.

Next, in S2001, the Lab value obtained by measuring each color patch of the color chart 500 is stored in the Lab value 710 of the reference color, and the camera color difference 730 for the L cameras is obtained.

Next, in S2002, a weighting factor for each color patch is obtained, the average camera color difference 740 of the L cameras is obtained, and the camera with the smallest average camera color difference is selected as the master camera.

Next, in S2003, the reference color of the master camera is stored in the Lab value 710 of the reference color, and M cameras with the smallest camera average color difference 740 are selected as multi-cameras. That is, the CPU 101 of the image processing apparatus 100 functions as multi-camera selection means for selecting M cameras as multi-cameras.

Next, in S2004, with the reference color of the master camera stored in the Lab value 710 of the reference color, the camera with the smallest camera average color difference 740 is selected as the neighboring camera.

Next, in S2005, the master camera is placed at the position of the camera 800 in FIG. 10B, and the neighboring cameras are placed at the position of the camera 801.

Next, in S2006, it is determined whether or not the processing from S2004 to S2005 has been performed for all the cameras (M cameras) configuring the multi-camera. If not completed, the process returns to S2004. On the other hand, in the case of end, the process proceeds to S1005.

When the process returns to S2004, the color of each color patch of the neighboring camera selected immediately before is stored as the next reference color in the reference color Lab value 710, and the camera average color difference 740 at that time is the minimum. is selected as the "next nearby camera". However, cameras that have already been selected as nearby cameras are not selected. The “next nearby camera” is placed at the position of the camera 802 in FIG. 10B in S2005.

Next, in S1005, inter-camera color differences of the M multi-cameras are calculated, and an average inter-adjacent camera color difference 650 is obtained.

Next, in S1008, the multi-camera arrangement is determined using the information of the important line-of-sight section in which the lines of sight important for photographing gather and the distribution of the average color difference 650 between adjacent cameras.

On the other hand, in FIG. 13, instead of a single neighboring camera, M multi-cameras are divided into P (P<M) groups, and the representative camera of each group and the cameras belonging to each group are sequentially searched. An example of processing for selecting a camera is shown.

By the processing from S1001 to S2003, similarly to FIG. 12, captured images of the color chart 500 captured by each of L cameras are obtained, and M cameras are selected as multi-cameras in descending order of camera average color difference 740. do.

Next, in S2007, the master camera is selected as the representative camera of the first group.

Next, in S2008, with the reference color of the representative camera of the first group stored in the Lab value 710 of the reference color, (M/P) units (including the representative camera of the group) from the smallest camera average color difference 740 are calculated. are selected as cameras belonging to the first group.

Next, in S2009, the camera selected as the camera belonging to the first group is placed at the position of the camera 800 in FIG. 10B. The cameras in the first group are arranged counterclockwise on the circumference in ascending order of camera average color difference with respect to the representative camera.

Next, in S2010, it is determined whether or not the processing from S2007 to S2009 has been performed for all P groups. If not completed, the process returns to S2007. On the other hand, in the case of end, the process proceeds to S1005.

Note that when the process returns to S2007, the ((M/P)+1)th camera with the smallest average camera color difference 740 with respect to the representative camera of the previous group is set as the "representative camera of the next group". Select as Next, in S2008, the reference color of the "representative camera of the next group" is stored in the Lab value 710 of the reference color, and (M/P) units (representative camera of the group) are selected from the smallest camera average color difference 740. ) are selected as “cameras belonging to the following group”. However, cameras that have already been selected as belonging to a group are not selected. Next, in S2009, the cameras belonging to the next group are arranged as shown in FIG. 10(b). The second group is arranged at the position of the camera 801, and the cameras in the second group are arranged clockwise on the same circumference following the first group in order of decreasing camera average color difference with respect to the representative camera. Furthermore, the third group is arranged at the position of the camera 802, and the cameras in the third group are arranged counterclockwise on the circumference following the first group in order of decreasing camera average color difference with respect to the representative camera. . In this way, cameras are arranged for all P groups.

Next, in S1005, inter-camera color differences of the M multi-cameras are calculated, and an average inter-adjacent camera color difference 650 is obtained.

Next, in S1008, the multi-camera arrangement is determined using the information of the important line-of-sight section in which the lines of sight important for photographing gather and the distribution of the average color difference 650 between adjacent cameras.

[effect]
As described above, in the present embodiment, cameras are selected so that the color difference becomes small with reference to a color chart or a master camera in a multi-camera. By doing so, it is possible to determine, within a practical time, a camera arrangement that reduces the color difference between images captured by multiple cameras.

(Third Embodiment)
In the first embodiment and the second embodiment, a method of selecting M (M≦L) multi-cameras from L cameras and determining the multi-camera arrangement so as to reduce the color difference between the cameras is described. explained.

In the present embodiment, in the first embodiment and the second embodiment, weighting is designated for a specific color, and weighting is designated for a section in which the line of sight that is important for shooting is gathered. A user interface in this case will be described.

FIG. 14 shows an example of a user interface for weighting designation in this embodiment.

FIG. 14(a) shows an example of a user interface for weighting a specific color. As described above, the user identifies a color patch close to the important color (for example, the color of the lawn, uniform, advertisement, skin color, costume, musical instrument, etc.) from all the color patches in the color chart 500, and selects the identified color patch. give more weight to the color patch than other color patches.

A color patch color name 900 represents the color name of each color patch of the color chart 500 . The color name can be set by a setting file, or can be directly edited and set by the user on the monitor 107 . If the color names are unknown, the expressions "Color1" and "Color2" may be used.

A color patch color 901 displays the color of each color patch in the color chart 500 . If the monitor 107 is an sRGB monitor, the Lab value 610 of each color patch of the color chart 500 and the Lab value 710 of the reference color may be converted into the sRGB color space and displayed.

Color patch weight 902 displays the weighting factor for each color patch entered by the user. The input weighting factors are obtained in the weighted average color difference evaluation between cameras (S1011) in FIG. 7C, the weighted average color difference evaluation between adjacent cameras (S1012), and the master camera selection (S2002) in FIGS. Used as a weighting factor for each color patch. The default value for the weighting factor is 1, and values greater than 1 are entered for important colors. The obtained weight coefficient is substituted as the weight wi for the _i -th color patch in equations (8) and (17), and is used to calculate the weighted average of the average color difference 650 between adjacent cameras and the average color difference 740 between cameras.

FIG. 14(b) shows an example of a user interface for weighting and designating important line-of-sight segments. As described above, as the information of the important line-of-sight section, the user can set the camera in the section where the line of sight that is important for shooting (the side of the main stand of the stadium, the front side of the stage in the concert hall, etc.) to the camera in the other section. give more weight.

A camera display 903 displays the arrangement of the multiple cameras 212 of the image processing system 200 on the monitor 107 . A camera display 903 allows a bird's eye view of where the cameras are arranged in the stadium or concert hall.

The camera weights for key line-of-sight segments 904 display the weighting factors for each camera entered by the user. The input weighting coefficients are used as the information of the important line-of-sight section acquired in the multi-camera arrangement determination (S1008) in FIGS. The default value of the weighting factor is 1, and a value greater than 1 is input for cameras in the important line-of-sight section. Since the obtained weighting coefficients are weighting coefficients for each camera, it is necessary to obtain the weighting coefficients between the adjacent cameras (two cameras) by linear interpolation of the weights of the two adjacent cameras or curve interpolation of the importance distribution of the important line-of-sight section. be. The calculated weighting factor is substituted as the weight v _j for the average color difference between adjacent cameras in equations (13) and (14), and is used for the weighted average calculation for the average color difference 650 between adjacent cameras.

[effect]
As described above, the present embodiment is provided with a user interface for weighting designation of a specific color or important line-of-sight segment. By doing so, it is possible to optimize the camera arrangement for the shooting site so as to reduce the color difference of each shot image shot by the multi-camera.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

It should be noted that not all the configurations of the above-described embodiments are essential configurations to obtain the effects. For example, even if only M cameras are selected from L cameras, the effect can be obtained. For example, if one of the L cameras has an individual difference that is clearly different from the other L−1 cameras, the camera can be excluded and the M cameras can be selected. This alone can also reduce the color difference between the images captured by the multi-camera. Also, as another example, if L=M, the step of selecting a camera can be omitted. In this case, the positional relationship of the L (=M) cameras is determined based on the images captured by each camera. This also makes it possible to reduce the color difference between the images captured by the multi-camera.

100 image processing device 101 CPU
102 Main memory 103 SATA I/F
104 Network I/F
105 HDDs
106 graphic accelerator 107 monitor 108 serial bus I/F
109 Printer 110 Serial Bus 111 Keyboard 112 Mouse 113 Image Input Device 114 System Bus 115 Network

Claims (20)

determining means for determining, based on images captured by the plurality of cameras, the arrangement of a plurality of cameras in a multi-camera system for generating virtual viewpoint content;
and output means for outputting according to the determination by the determination means ,
The control device, wherein the determining means determines the arrangement of the plurality of cameras so that the color difference between the images captured by adjacent cameras is small . selection means for selecting M (M<L) cameras to be used in a multi-camera system for generating virtual viewpoint content from L cameras, based on images captured by the L cameras;
further having
2. The control device according to claim 1, wherein said determining means determines the arrangement of the M cameras selected by said selecting means. The determining means is
Candidate determining means for determining a candidate for a multi-camera set representing the relative positional relationship of the M cameras selected by the selecting means;
color difference evaluation means for evaluating inter-camera color differences of each of the captured images captured by the M cameras for each of the candidates for the multi-camera set;
3. The control device according to claim 2 , further comprising pair determination means for determining the pair of the multi-cameras based on the evaluation result of the inter-camera color difference. 4. The control device according to claim 3, wherein said color difference evaluation means calculates said inter-camera color difference for all combinations of two cameras in said multi-camera set candidates. 4. The control device according to claim 3, wherein said color difference evaluation means calculates said inter-camera color difference for a combination of two adjacent cameras in said multi-camera set candidate. 6. The control device according to claim 4, wherein said color difference evaluation means calculates said inter-camera color difference using a weighting factor for a specific color. The determining means repeats selection of a camera positioned near the specific camera based on the inter-camera color difference with respect to the specific camera among the M cameras, thereby determining the relative positions of the M cameras. 3. A control device according to claim 2 , further comprising pair determination means for determining a pair of multi-cameras representing the relationship. The determination means divides the M cameras into P (P<M) groups, selects a specific camera in each of the P groups as a representative camera, and selects the P cameras based on the inter-camera color difference. 3. The control according to claim 2 , further comprising a set determining means for determining a set of multi-cameras representing the relative positional relationship of said M cameras by determining the arrangement of said groups of cameras. Device. 9. The control device according to claim 7, wherein the selecting means selects the M cameras in ascending order of color difference of each captured image captured by the L cameras with respect to a reference color. . 10. The control device according to claim 9, wherein the reference color is based on a color chart. 10. The control device according to claim 9, wherein the reference color is based on a master camera. 12. The control device according to claim 11, wherein the master camera is selected based on a color difference between each photographed image obtained by photographing a color chart by the L cameras and the color chart. 13. The control device according to claim 12, wherein said master camera is selected based on an average value of said color difference with said color chart. 14. A controller as claimed in claim 12 or 13, wherein the master camera is selected based on the color difference calculated using a weighting factor for a particular color. The determining means has an arrangement determining means for determining the arrangement of the multi- camera set in accordance with the inter-camera color difference of the images captured by the cameras in a specific section among the M cameras. 15. The control device according to any one of claims 2 to 14. 15. The control device according to claim 6, further comprising weight specifying means for specifying a weighting factor for said specific color. 16. The control device according to claim 15, further comprising weight designation means for designating a weight coefficient for the camera in the specific section. A program for causing a computer to function as the control device according to any one of claims 1 to 17. A control method implemented by a computer, comprising:
a determining step of determining an arrangement of a plurality of cameras in a multi-camera system for generating virtual viewpoint content based on images captured by the plurality of cameras;
and an output step of performing an output according to the determination by the determination step,
The control method , wherein the determining step determines the arrangement of the plurality of cameras so that a color difference between images taken by adjacent cameras is reduced . a filming process of filming with a plurality of cameras in a multi-camera system for generating virtual viewpoint content;
a determination step of determining the arrangement of the plurality of cameras based on the plurality of captured images obtained by the photographing step so that the color difference between the captured images of adjacent cameras is small;
A control method comprising:

Images