JP2020191618A - Arrangement determination device, system, arrangement determination method, and program - Google Patents

Abstract

To reduce a difference in the reproducibility of a subject between virtual viewpoints even in a wide shooting target area.SOLUTION: An arrangement determination device 108 that determines the arrangement of a plurality of cameras 102 used to generate a virtual viewpoint image acquires information representing a shooting target area 503, and determines the arrangement of the plurality of cameras 102 such that the entire shooting target area 503 is shot by two or more cameras 102 from among the plurality of cameras 102 on the basis of the information, and the entire shooting target area 503 is shot by other two or more cameras 102.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for determining the arrangement of a plurality of photographing devices.

In recent years, attention has been paid to a technique in which a plurality of photographing devices are arranged at different positions, photographed in synchronization, and a virtual viewpoint image is generated from a designated virtual viewpoint by using a plurality of images obtained by the photographing. ing.

Patent Document 1 describes a technique for generating a virtual viewpoint image, and describes arranging a plurality of photographing devices along the circumference so as to face the center of a circle which is a photographing target area.

JP-A-2010-79505

In Patent Document 1, there is a difference between the number of photographing devices for photographing a subject existing near the center of a circle, which is a photographing target area, and the number of photographing devices for photographing a subject existing at a position away from the subject. Further, when the imaging target area is large, the difference becomes large. Therefore, there may be a large difference between the virtual viewpoints in terms of the accuracy of reproducing the subject in the virtual viewpoint image.

Therefore, in view of the above problems, it is an object of the present invention to reduce the difference in the reproduction accuracy of the subject between the virtual viewpoints even in a wide range of shooting target areas.

One aspect of the present invention is as follows. That is, it is an arrangement determining device that determines the arrangement of a plurality of photographing devices used for generating a virtual viewpoint image, and is an area acquisition means for acquiring information representing an imaging target area to be photographed by the plurality of photographing devices. Based on the information acquired by the area acquisition means, the entire area of the imaging target area is photographed by two or more imaging devices among the plurality of imaging devices, and two or more of the plurality of imaging devices. It is characterized by having an arrangement determining means for determining the arrangement of the plurality of photographing devices so that the entire area to be photographed is photographed by two or more other photographing devices different from the photographing device.

According to the present invention, it is possible to reduce the difference in the reproduction accuracy of a subject between virtual viewpoints even in a wide range of shooting target areas.

Diagram showing a configuration example of an image processing system The figure which shows the hardware configuration example of the arrangement determination apparatus The figure which shows the structural example of the arrangement determination apparatus which concerns on Embodiment 1. A flowchart showing an example of a processing procedure of the arrangement determination device according to the first embodiment. The figure which shows the example of the imaging target area which concerns on Embodiment 1. Diagram to illustrate the resolution Diagram showing an example of the placement area The figure which shows the arrangement example of a plurality of cameras in an arrangement area Diagram showing an ideal arrangement of multiple cameras The figure which shows the arrangement example of a plurality of cameras Diagram showing other arrangement examples of multiple cameras Diagram showing other arrangement examples of multiple cameras Diagram showing the determined arrangement example A flowchart showing an example of a processing procedure of the arrangement determination device according to the second embodiment. The figure explaining the angle of the photographing target area as seen from the arrangement area The figure which shows the structural example of the arrangement determination apparatus which concerns on Embodiment 2. The figure which shows an example of the table which the arrangement determination apparatus which concerns on Embodiment 1 refers to. The figure which shows the structural example of the arrangement determination apparatus which concerns on Embodiment 3. The figure which shows an example of the table which the arrangement determination apparatus which concerns on Embodiment 3 refers to. The figure which shows the example of the imaging target area which concerns on Embodiment 3. A flowchart showing an example of a processing procedure of the arrangement determination device according to the third embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

<Embodiment 1>
FIG. 1 is a diagram showing an example of the configuration of the image processing system 100. The image processing system 100 includes a multi-camera system 101, a hub 104, a control device 105, and a virtual viewpoint image generation device 106. Further, the image processing system 100 may include a virtual viewpoint designation device 107 and an arrangement determination device 108.

The image processing system 100 is a system that generates a virtual viewpoint image representing a view from a designated viewpoint based on a plurality of images taken by a plurality of photographing devices and a designated virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The multi-camera system 101 has a plurality of cameras (shooting devices) 102aa to 102 lc that shoot the shooting target space from a plurality of directions. The plurality of cameras 102aa to 102 lc belong to any one of the plurality of camera groups 103a to 103l. In the following, when it is not necessary to distinguish between them, they are referred to as a camera 102 and a camera group 103.

Each camera group 103 has one or more cameras 102. Specifically, three cameras 102aa, 102ab, and 102ac belong to the camera group 103a. However, the number of cameras belonging to the camera group 103 is not limited to three, and may be two or less, or four or more. Further, the number of cameras 102 belonging to each of the camera groups 103 may be the same or different.

The plurality of images taken by the plurality of cameras 102 are sent to the virtual viewpoint image generator 106 through the hub 104. Further, the control device 105 controls the shooting of the multi-camera system 101 and manages the arrangement status of the plurality of cameras 102 of the multi-camera system 101.

The virtual viewpoint image generation device 106 generates a virtual viewpoint image viewed from a viewpoint designated by the virtual viewpoint designation device 107 based on a plurality of images taken by the multi-camera system 101. The viewpoint information designated by the virtual viewpoint designation device 107 is information indicating the position and orientation of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may be information that has a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image and indicates the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. .. The virtual viewpoint designation device 107 is connected to the virtual viewpoint image generation device 106 or the image processing system 100 by wire or wirelessly.

The virtual viewpoint image generation device 106 generates a virtual viewpoint image by, for example, the following method. First, a plurality of images are acquired by shooting from different directions with a plurality of photographing devices. Next, a foreground image in which a foreground area corresponding to a predetermined object (subject) such as a person or a ball is extracted and a background image in which a background area other than the foreground area is extracted are acquired from the plurality of images. Further, in order to generate a foreground model representing the three-dimensional shape of a predetermined subject and texture data for coloring the foreground model based on the foreground image, and to color the background model representing the three-dimensional shape of the background such as a stadium. Texture data is generated based on the background image. Then, the texture data is mapped to the foreground model and the background model, and the virtual viewpoint image is generated by rendering according to the virtual viewpoint indicated by the viewpoint information. The background is the field and goal post. However, the method of generating the virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of the captured image without using the three-dimensional model can be used.

The arrangement determination device 108 determines the arrangement of a plurality of cameras 102 included in the multi-camera system 101 as described later. The arrangement determination device 108 may or may not be included in the image processing system 100. The arrangement determination device 108 is connected to the control device 105 or the image processing system 100 by wire or wirelessly.

The foreground image and the background image described above may be extracted from the captured image by the virtual viewpoint image generation device 106, respectively, or may be extracted by another device, for example, the camera 102. When the foreground image and the background image are extracted by the cameras 102, both the foreground image and the background image may be extracted by each of the plurality of cameras 102, or some cameras of the plurality of cameras 102 may extract the foreground image. It may be extracted and the other part may extract the background image. Further, the plurality of cameras 102 may include a camera 102 that does not extract either a foreground image or a background image.

FIG. 2 is a diagram showing an example of the hardware configuration of the arrangement determination device 108. The CPU 201 realizes each function of the information processing device by controlling the entire information processing device using computer programs and data stored in the ROM 202 and the RAM 203. The information processing device may have one or more dedicated hardware different from the CPU 201, and the dedicated hardware may execute at least a part of the processing by the CPU 201. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 202 stores programs and the like that do not require changes.

The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, data supplied from the outside via the communication I / F 207, and the like. The auxiliary storage device 204 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data. The display unit 205 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the information processing device.

The operation unit 206 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 201 in response to an operation by the user. The communication I / F 207 is used for communication with an external device of the information processing device. For example, when the information processing device is connected to an external device by wire, a communication cable is connected to the communication I / F 207. When the information processing device has a function of wirelessly communicating with an external device, the communication I / F 207 includes an antenna. The bus 208 connects each part of the information processing device to transmit information.

In the present embodiment, it is assumed that the display unit 205 and the operation unit 206 exist inside the arrangement determination device 108, but at least one of the display unit 205 and the operation unit 206 exists as another device outside the arrangement determination device 108. You may be doing it. In this case, the CPU 201 may operate as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

FIG. 3 is a diagram showing an example of the functional configuration of the arrangement determination device 108. The arrangement determination device 108 includes an allowable value acquisition unit 303, a photographing target area acquisition unit 304, a camera group number acquisition unit 305, and an arrangement determination unit 306. Further, the arrangement determination device 108 may have an event information storage unit 301 and a venue information storage unit 302. The present embodiment will be described by exemplifying an example in which the arrangement determination device 108 has an event information storage unit 301 and a venue information storage unit 302. However, the event information storage unit 301 and the venue information storage unit 302 may be included in a device different from the arrangement determination device 108, and the arrangement determination device 108 may acquire necessary information from the device.

The event information storage unit 301 stores information about the event to be photographed. The event information is, for example, the following information. Information that can identify the type of event and other events, information on the size of the field, information on the moving range of the subject, information on the density of the subject, information on the use of the virtual viewpoint image, information on the scale of the event, and the like. The types of events are sports competitions such as rugby, soccer, bouldering, baseball, athletics, water competitions, and ice competitions, concerts, plays, and the like, and are not particularly limited. The field is a surface on which moving objects such as a person, a ball, and a tool used in the event can move. The moving range of the subject is set for each event. The density of subjects is a numerical value of the density of people who are subjects in an event. For example, when comparing the density of rugby, soccer, and baseball, rugby has the highest density and baseball has the lowest density. The reason why the density of rugby subjects is high is that play such as scrum and mall occurs. The virtual viewpoint image is used for live broadcasting, archive distribution, behavior analysis, tactical analysis, and the like.

The venue information storage unit 302 stores information about the venue where the plurality of cameras 102 are arranged. The venue information is, for example, the following information. Information on the name of the venue, information on the size of the venue, information on the structure in which the cameras 102 can be arranged and their positions in the venue, information on the number of cameras 102 that can be arranged, and the like. The venues are stadiums, which are venues for games such as rugby and soccer, and venues for plays and concerts. The venue may also be a practice field for sports teams. The structure in which the camera 102 can be arranged and the information indicating the position thereof may be represented by a two-dimensional image or may be represented as three-dimensional data.

When the camera 102 takes a picture of a predetermined reference subject, the permissible value acquisition unit 303 sets and acquires a lower limit value of the resolution which is the size in the image of the subject. Specifically, the lower limit resolution is set according to the use of the virtual viewpoint image. For example, when the virtual viewpoint image is used for a bird's-eye view image for live broadcasting, the lower limit resolution is set small, and when the virtual viewpoint image is used for a replay image, the lower limit resolution is set large. The lower limit resolution will be described later.

The permissible value acquisition unit 303 may store in advance the correspondence between the use of the virtual viewpoint image, which is one of the event information, and the lower limit resolution as a table, and set by referring to the table. That is, when the permissible value acquisition unit 303 receives the input of information that can identify the event from the operator via the operation unit 206, the tolerance value acquisition unit 303 refers to the event information stored in the event information storage unit 301 and its table, and sets the lower limit resolution. You may try to get it. An example of the referenced table is shown in FIG. 17 (a). The referenced table may be associated with information on the usage of the virtual viewpoint image and information on the lower limit resolution, but is not limited to this. Event information other than the above and information on the lower limit resolution may be associated with each other. In addition, the referenced table may be associated with information that can identify the event, such as the name of the event, and the lower limit resolution. Further, the referenced table may be stored in the event information storage unit 301, or may be stored in an external device.

Further, the permissible value acquisition unit 303 may acquire information such as a numerical value of the lower limit resolution directly from the operator via the operation unit 206.

The shooting target area acquisition unit 304 determines the shooting target area, which is the shooting target area of the multi-camera system 101, based on the event information. Specifically, the photographing target area acquisition unit 304 may have the following configuration. That is, first, the shooting target area acquisition unit 304 stores a table in which at least a part of the event information and the information representing the shooting target area are associated with each other. At least a part of the event information is information on the size of the field, information on the moving range of the subject, and the like. Then, when the shooting target area acquisition unit 304 receives the input of the information that can identify the event from the operator via the operation unit 206, the shooting target area acquisition unit 304 refers to the event information stored in the event information storage unit 301 and its table, and is the shooting target. Determine the area. An example of the referenced table is shown in FIG. 17 (b). In the table to be referred to, the information on the size of the field, the information on the moving range of the subject, and the information on the shooting target area may be associated with each other, but the reference is not limited to this. Event information other than the above may be associated with information on the shooting target area. Further, in the referenced table, information that can identify the event, such as the name of the event, may be associated with the shooting target area. Further, the referenced table may be stored in the event information storage unit 301, or may be stored in an external device.

Further, the shooting target area acquisition unit 304 may acquire information on the shooting target area directly from the operator via the operation unit 206.

Here, the shooting target area will be described. FIG. 5A is a diagram showing an example of a shooting target area, and this figure shows an example when the event is rugby. The imaging target area 503 is a two-dimensional surface, and constitutes a part of the space 504 in FIG. 5 (b). The shooting target area 503 is determined by the shooting target area acquisition unit 304 based on the size of the field 501 and the moving range of a subject such as a person (player) or a ball. Here, the movement range includes not only the movement range on the field 501 but also the periphery of the field 501 because the player stands outside the touch line when throwing in by lineout, for example. The size of the movement range may be set according to the event. For example, in the case of rugby, the moving distance may be defined by the distance or angle from the touch line, dead ball line, touch in goal line, goal post 502, etc., or is set to include the field 501. It may be specified in the range. Further, in the case of soccer, the distance or angle from the touch line, the goal line, the position of the corner flag, or the like may be similarly specified. Further, in the case of baseball, the movement range may be set to include the foul zone.

As described above, the photographing target area 503 is set to include the play area (field 501) and further include a predetermined range outside the play area. By setting the photographing target area 503 in this way, it is possible to generate a high-quality virtual viewpoint image in the entire area of the field 501.

FIG. 5B is a diagram showing an example in which a space 504 in consideration of the height direction is set as a shooting target area. That is, in the example of FIG. 5B, the imaging target area is a three-dimensional space. In FIG. 5B, the shooting target area is set so as to include the range including the space above the goal post 502, mainly considering the moving range of the ball. In addition, the imaging target area may be a space in which a predetermined height is taken into consideration from the field 501. For example, the predetermined height is the height of the person (player), the height when the person jumps, the reaching point of the player who is lifted at the time of lineout, and the like.

The camera group number acquisition unit 305 acquires the number of camera groups 103 constituting the multi-camera system 101 so as to shoot the shooting target area 503 from a plurality of directions. Specifically, the camera group number acquisition unit 305 sets the number of camera groups 103 based on the density of the subjects. The camera group number acquisition unit 305 sets the number of camera groups 103 more when the density of the subjects is large than when the density of the subjects is small. The camera group number acquisition unit 305 stores in advance the correspondence between the density of subjects, which is one of the event information, and the number of camera groups 103 as a table, and sets by referring to the table. You may. That is, when the camera group number acquisition unit 305 receives the input of the information that can identify the event via the operation unit 206, the camera group 103 refers to the event information stored in the event information storage unit 301 and its table. You may try to get the number of. An example of the referenced table is shown in FIG. 17 (c). The table to be referred to may be, but is not limited to, as long as the information on the density of the subject and the information on the number of camera groups 103 are associated with each other. Event information other than the above may be associated with information on the number of camera groups 103. Further, in the referenced table, information that can identify the event, such as the name of the event, may be associated with the number of camera groups 103. Further, the referenced table may be stored in the event information storage unit 301, or may be stored in an external device.

Further, the camera group number acquisition unit 305 may acquire information on the number of camera groups 103 directly from the operator via the operation unit 206.

The arrangement determination unit 306 determines the arrangement of the cameras 102 based on the lower limit resolution, the shooting target area 503, the number of camera groups, and the venue information. The lower limit resolution is acquired by the permissible value acquisition unit 303, the shooting target area 503 is acquired by the shooting target area acquisition unit 304, and the number of camera groups is acquired by the camera group number acquisition unit 305. The venue information is stored in the venue information storage unit 302. The arrangement determination unit 306 acquires necessary venue information from the venue information storage unit 302 based on the information for identifying the venue input via the operation unit 206. For example, the venue information to be acquired includes information on a structure in which the cameras 102 can be arranged and their positions in the venue, information on the number of cameras 102 that can be arranged, and the like.

Specifically, the arrangement determination unit 306 first determines the arrangement position of the camera group 103 based on the photographing target area 503, the number of camera groups, and the venue information. Next, the arrangement determination unit 306 determines the arrangement of the plurality of cameras 102 to which the camera group 103 belongs. At that time, the arrangement determination unit 306 determines the arrangement of the plurality of cameras 102 belonging to each camera group 103 so that the entire area of the imaging target area 503 is photographed by the plurality of cameras 102 belonging to the camera group 103. Further, the arrangement determination unit 306 determines the arrangement of the plurality of cameras 102 in the plurality of cameras 102 belonging to the camera group 103 so that the resolution exceeds the lower limit resolution in the entire area of the imaging target area 503. It should be noted that the arrangement of the plurality of cameras 102 is determined so as to satisfy the above-mentioned conditions regarding the shooting target area and the conditions regarding the resolution in all the camera groups 103, but the present invention is not limited to this. For example, the arrangement of the plurality of cameras 102 may be determined so that only the plurality of cameras 102 belonging to a predetermined number of camera groups 103 satisfy the above conditions. Further, the arrangement of the plurality of cameras 102 belonging to the camera group 103 may be determined so that the camera group 103 at a predetermined position and the camera group 103 having a predetermined relationship satisfy the above conditions. That is, the plurality of cameras 102 may be arranged so that a plurality of cameras 102 belonging to a part of the camera group 103 capture a part of the photographing target area 503 instead of the entire area. Further, in a plurality of cameras 102 belonging to a part of the camera group 103, the resolution does not have to exceed the lower limit resolution in a part of the photographing target area 503.

FIG. 4 is a flowchart showing an example of the processing procedure of the arrangement determination device 108. The process performed by the arrangement determination device 108 will be described with reference to FIG.

First, the arrangement determination device 108 receives an instruction for arrangement determination from the operator via the operation unit 206. According to the instruction, in step S401, the permissible value acquisition unit 303 acquires the lower limit resolution.

Here, the lower limit resolution will be described, but before that, the resolution will be described. The resolution referred to here is the size of a reference subject in the image captured by the camera 102, and is represented by the number of pixels. The reference subject may be a typical subject or a virtual subject, or may be a specific subject or an arbitrary subject in the event to be photographed. For example, an upright athlete with a height of 2 m is defined as a reference subject. In that case, in the image taken by the camera 102, the number of pixels from the tip of the head to the tip of the foot of an upright athlete with a height of 2 m is the resolution.

FIG. 6 is a diagram for explaining the resolution. The resolution 601a is defined as the number of pixels in the captured image representing the height of the reference subject (here, an upright athlete having a height of 2 m) at the position 602a in the field 501. Further, the number of pixels in the captured image representing the height of the reference subject at the position 602b is set to the resolution 601b. The resolution varies depending on the distance from the camera 102, the focal length of the camera 102, and the like. For example, the resolution 601a at the position 602a near the camera 102 is larger than the resolution 601b at the position 602b far away.

The lower limit resolution is an allowable value of the resolution in the image captured by the camera 102. The lower limit resolution affects the image quality of the subject in the virtual viewpoint image. That is, in the virtual viewpoint image, when the number of pixels of the reference subject becomes larger than the lower limit resolution, the image quality deteriorates due to the same effect as the digital zoom. Therefore, the lower limit resolution is determined according to how large the subject is to be captured in the virtual viewpoint image and what kind of composition is desired. For example, consider taking a reference subject as large as a quarter of the size of a virtual viewpoint image. The size of the virtual viewpoint image is 1920 pixels × 1080 pixels. Then, the lower limit resolution is determined to be 1080 pixels ÷ 4 = 270 pixels. The lower limit resolution may be determined so as to allow a certain degree of digital zoom. For example, if it is determined that a digital zoom of 1.2 times is acceptable, 270 ÷ 1.2 = 225 pixels is determined as the lower limit resolution.

This lower limit resolution indicates how many pixels or more the reference subject at an arbitrary position in the field 501 is represented in the captured image. However, it is not limited to this. For example, the lower limit resolution may indicate how many pixels or more the reference subject in a predetermined position or a predetermined range is represented in the captured image, not at an arbitrary position in the field 501. Good.

As described above, the permissible value acquisition unit 303 determines and acquires the lower limit resolution based on the information that can identify the event input from the operator via the operation unit 206 and the table stored in advance.

Next, in step S402, the shooting target area acquisition unit 304 determines and acquires the shooting target area 503. Specifically, as described above, the imaging target area 503 is determined based on the information that can identify the event input from the operator via the operation unit 206 and the table stored in advance. The imaging target area 503 is as described above.

In step S403, the camera group number acquisition unit 305 determines and acquires the number of camera groups 103. The number of camera groups 103 is determined according to the number of directions required for the virtual viewpoint image generation device 106 to accurately generate three-dimensional shape data of the subject. The number of directions required to generate the three-dimensional shape data of the subject varies depending on the density of the subject. The camera group number acquisition unit 305 associates the density of the subjects with the number of required directions (that is, the number of camera groups) and stores them in advance as a table. For example, the camera group number acquisition unit 305 generates a virtual viewpoint image using CG (computer graphics) that imitates the subject, and determines the number of directions required to see the image quality and the viewpoint position, and determines the density of the subject. It is stored in advance as a table in association with. For example, in a situation where a subject exists alone, such as a baseball pitcher, occlusion is unlikely to occur, so shooting from a small number of directions is acceptable, but in a situation where a large number of subjects are crowded, such as a rugby scrum, from many directions. I need to shoot. The number of directions determined here is determined as the number of camera groups 103. Hereinafter, as an example, it is assumed that the number of camera groups 103 is determined to be 12. Further, the arrangement area of the camera group 103 may also be determined in this step, but in the present embodiment, it is determined in the next step S404.

In step S404, the arrangement determination unit 306 determines the arrangement area of the camera group 103 based on the number of camera groups and the venue information. The arrangement determination process will be described in detail with reference to FIGS. 7 to 12. FIG. 7 is a diagram showing an example of the arrangement position of the camera group 103. First, from the venue information, for example, the design drawing of the stadium 702, the area where the camera 102 can be arranged is determined as a candidate for the arrangement areas 701a to 701l. Candidates for placement areas 701a-701l are pillars, passageways, handrails, ceilings, etc. of stadium 702. The determined candidate for the arrangement area may be displayed on the display unit 205. In the following, unless there are special circumstances, the arrangement areas 701a to 701l are referred to as arrangement areas 701.

Next, the placement area 701 of the camera group 103 is determined from the candidates of the placement area 701 according to the criteria described below. When the distance between two adjacent camera groups 103 for shooting a subject becomes wide, the image quality of the virtual viewpoint image generated from the virtual viewpoint designated between them deteriorates. Therefore, if the intervals between the camera groups 103 are not uniform, the image quality of the virtual viewpoint image varies depending on the position of the designated virtual viewpoint. Therefore, it is preferable that the arrangement areas 701 of the camera group 103 are determined to be as evenly spaced as possible when viewed from the subject in the image shooting target area 503. For example, the arrangement area 701 is determined so that the angles between the adjacent arrangement areas 701 are equal when viewed from the subject at the center 703 of the imaging target area 503. When the number of camera groups 103 is 12, the ideal angle between adjacent arrangement areas 701 is 360 degrees / 12 = 30 degrees. Therefore, for example, the arrangement area 701a and the arrangement area 701b are determined so that the angle 704ab between them is 30 degrees. However, due to structural reasons such as the position of the pillars of the stadium 702, it may not be possible to determine the arrangement areas 701 at equal intervals. Therefore, for example, the arrangement area 701 may be determined by setting a reference in advance, such as setting 1.5 times the ideal angle as the allowable range. That is, the arrangement area 701 is determined so that the angle between the adjacent arrangement areas 701 when viewed from the subject does not exceed a predetermined value.

In step S405, the arrangement determination unit 306 determines the number of two or more cameras 102 belonging to the camera group 103 for each camera group 103 based on the arrangement area 701 of the camera group 103, the lower limit resolution, and the shooting target area 503. And determine the placement. At that time, the number and arrangement of two or more cameras 102 belonging to the camera group 103 are determined so as to satisfy the following conditions. That is, for each camera group 103, the number and arrangement of two or more cameras 102 are determined so that the entire area of the shooting target area 503 is shot. Further, the number and arrangement of the cameras 102 are determined so that the resolution exceeds the lower limit resolution in the entire area of the imaging target area 503. Further, the arrangement is determined so that the number of cameras 102 is as small as possible. The arrangement of the camera 102 referred to here includes the position and orientation of the camera 102.

First, the positions where the plurality of cameras 102 are arranged in the arrangement area 701 will be described with reference to FIG. The plurality of cameras 102 belonging to the camera group 103 are arranged in the corresponding arrangement areas 701. For example, the plurality of cameras 102 belonging to the camera group 103a are arranged in the corresponding arrangement area 701a. Within the arrangement area 701, a plurality of cameras 102 belonging to the camera group 103 are arranged at positions as close as possible.

FIG. 9 shows an example in which three cameras 102aa, 102ab, and 102ac in the same camera group 103a are arranged so that their optical centers overlap each other in an ideal state. The shooting range of the camera 102aa is 901aa. The shooting range of the camera 102ab is 901ab. The shooting range of the camera 102ac is 901ac. The shooting range 901 of the camera 102 extends radially. Therefore, when the optical centers overlap, the cameras 102aa, 102ab, and 102ac can be arranged so that the shooting ranges 901aa, 901ab, and 901ac are in contact with each other. Since the shooting ranges 901 are in contact with each other without overlapping, the three cameras 102aa to 102ac can efficiently cover a wide area. However, in reality, due to the physical restrictions of the camera 102, the optical centers cannot be arranged so as to overlap each other. Therefore, by arranging them as close to each other as possible, the overlap of the shooting ranges 901 is reduced.

Returning to FIG. 8, for example, a plurality of cameras 102ha, 102hb, 102hc belonging to the camera group 103h are arranged within a range of a predetermined angle 801h when viewed from the center 703 of the photographing target area 503. For example, a plurality of cameras 102 belonging to the camera group 103 are arranged within a predetermined distance.

From another point of view, each camera 102 is arranged closer to the camera 102 of the camera group 103 to which it belongs than the camera 102 of the other camera group 103. For example, adjacent camera groups 103g and 103h will be described. It is assumed that the cameras 102ga, 102gb, and 102gc belong to the camera group 103g. Further, it is assumed that the cameras 102ha, 102hb, and 102hc belong to the camera group 103h. In this case, the camera 102gc is arranged closer to the cameras 102ga, 102gb than the cameras 102ha, 102hb, 102hc. The reason is as follows. It is assumed that the camera 102gc is arranged near the camera 102ha. Then, the distance between the camera 102gc and the camera 102 of the camera group 103f located on the opposite side of the camera group 103h becomes wide, and the image quality of the virtual viewpoint image tends to deteriorate.

Next, an example of photographing the entire area of the photographing target area 503 with a plurality of cameras 102 belonging to the camera group 103 will be described with reference to FIGS. 10 to 12.

FIG. 10 shows an example in which the entire area of the photographing target area 503 is photographed by three cameras 102ia, 102ib, and 102ic for the camera group 103i. The shooting range of the camera 102ia is 901ia. The shooting range of the camera 102ib is 901ib. The shooting range of the camera 102ic is 901ic. The shooting target range corresponds to a region where the resolution of the reference subject in the shot image exceeds the lower limit resolution. As shown in FIG. 10, when these shooting ranges 901ia, 901ib, and 901ic are combined, the entire area of the shooting target area 503 is shot. For example, the camera 102ib is arranged in the center, the camera 102ia is arranged on the right side, the camera 102ic is arranged on the left side, and the three cameras 102 are arranged in a fan shape. As a result, the respective shooting ranges 901ia, 901ib, and 901ic also expand in a fan shape.

FIG. 10 is a diagram illustrating an imaging range 901ia to 901ic of the cameras 102ia-102ic with respect to the in-plane direction of the imaging target area 503. However, the shooting target area may be a three-dimensional space including up to a predetermined height of the field 501. In that case, the camera 102 shoots the shooting target area including the predetermined height direction of the field 501. Need to be placed. Since the positions of the optical centers of the three cameras 102 do not match, for example, a part of the shooting range 901ic and 901ib overlap. By arranging the three cameras 102 as close as possible, the overlap of the shooting ranges 901 can be reduced, and more efficient arrangement becomes possible. That is, the number of cameras 102 required for the camera group 103 can be reduced.

Here, the intersection of the optical axis of the camera 102 and the field 501 of the stadium 702 is defined as the gazing point 1001 of the camera 102. In FIG. 10, the gazing point of the camera 102ia is 1001ia. The gazing point of the camera 102ib is 1001ib. The gazing point of the camera 102ic is 1001ic. As shown in FIG. 10, the positions of the gazing points 1001 are all different. That is, the gazing points of the plurality of cameras 102 belonging to the camera group 103 are all different. However, one camera group 103 may include cameras 102 having the same gazing point.

Further, in the arrangement example of the camera 102 in FIG. 10, the optical axes of the cameras 102ia-102ic are oriented in different directions. Further, the line segments projected onto the field 501 of the line segments connecting the cameras 102ia to 102ic and the respective gazing points do not intersect with each other.

FIG. 11 is a diagram showing an example of arrangement of cameras 102 belonging to another camera group 103. The camera group 103g includes six cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. The entire area of the imaging target area 503 is photographed by the cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf.

FIG. 11A is a view of the photographing target area 503 viewed from above, and is a diagram showing the arrangement of 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf in the in-plane direction of the photographing target area 503. For the camera 102ga, the photographing range is 901ga and the gazing point is 1101ga. The same applies to the cameras 102gb to 102gf. The six cameras 102 are further divided into two subgroups. The cameras 102ga, 102gb, and 102gc belong to the first subgroup, and are arranged so as to shoot the front side of the shooting target area 503 (the side close to the placement area 701g of the camera group 103g) when viewed from the placement area 701. To do. The cameras 102gd, 102ge, and 102gf belong to the second subgroup, and are arranged so as to shoot the back side of the shooting target area 503 (the side far from the placement area 701g of the camera group 103g) when viewed from the placement area 701. To do. Further, as shown in FIG. 11A, the cameras 102 in each subgroup are arranged so that the shooting range 901 expands in a fan shape.

FIG. 11B is a side view of the photographing target area 503, and is a diagram showing the arrangement of the cameras 102 in the height direction of the photographing target area 503. As shown in FIG. 11B, the cameras 102ga, 102gb, 102gc that shoot the front side of the shooting target area 503 and the cameras 102gd, 102ge, 102gf that shoot the back side of the shooting target area 503 have different heights. Have been placed. For example, the camera 102gb that shoots the front side of the shooting target area 503 has a larger depression angle than the camera 102g that shoots the back side. Therefore, by arranging the camera 102gb that captures the front side of the imaging target area 503 on the lower side, it can be arranged closer without physical interference. That is, a plurality of cameras 102 belonging to the camera group 103 may be divided into subgroups, and the height may be changed for each subgroup.

FIG. 12 is a diagram showing another example of the arrangement of the cameras 102 belonging to the camera group 103g. The camera group 103g includes six cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. The entire area of the imaging target area 503 is photographed by the cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. In this example, the positions are arranged differently for each subgroup. For example, the placement area 701g is set to include two pillars. Then, different subgroups are assigned to each of the two pillars. For example, the cameras 102ga, 102gb, and 102gc belonging to the first subgroup are arranged on the left pillar when viewed from the imaging target area 503. The cameras 102gd, 102ge, and 102gf belonging to the second subgroup are arranged on the pillar on the right side when viewed from the imaging target area 503. That is, a plurality of cameras 102 belonging to the camera group 103 may be divided into subgroups, and the positions may be changed for each subgroup.

If the lower limit resolution is different, the number of cameras 102 for each camera group 103 will be different. When the lower limit resolution is increased, it is necessary to increase the focal length of the camera 102. Then, when the focal length is lengthened, the shooting range 901 of the camera 102 becomes narrower. Therefore, the number of cameras 102 required to shoot the entire area of the shooting target area 503 increases. On the contrary, if the lower limit resolution is reduced, the focal length of the camera 102 can be shortened. When the focal length is shortened, the shooting range 901 of the camera 102 becomes wider. Therefore, the number of cameras 102 required to shoot the entire area of the shooting target area 503 is reduced.

In step S406, the arrangement determination unit 306 totals the number of cameras 102 determined for each camera group 103 determined in S405, and determines the number of cameras 102 required for the multi-camera system 101.

In step S407, the arrangement determination unit 306 determines whether there is a problem with the number of cameras calculated in step S406. Specifically, the arrangement determination unit 306 determines that there is a problem when the number of cameras is larger or smaller than the predetermined number. If it is determined that there is no problem (YES), the process ends. If it is determined that there is a problem (NO), the process proceeds to step S408. The predetermined number of units may be set according to the budget, or may be set based on the processing capacity of the virtual viewpoint image generation device 106, the transmission bandwidth capacity when transmitting the captured image, and the like.

In step S408, the arrangement determination unit 306 determines which arrangement condition is to be changed. The arrangement condition that can be changed is any one of the shooting target area 503, the lower limit resolution, and the number of camera groups 103. The arrangement condition to be changed may be automatically determined by the arrangement determination unit 306, or may be directly input by the operator via the operation unit 206. Even if it is automatically determined by arrangement determining unit 306, to set a priority to an arrangement condition in advance change operator may specify whether to change the advance which arrangement conditions.

In step S409, the arrangement determination unit 306 determines whether or not the arrangement condition to be changed is the imaging target area 503. In the case of the imaging target area 503 (YES), the process proceeds to step S410. If not (NO), the process proceeds to step S411.

In step S410, the shooting target area acquisition unit 304 changes the shooting target area 503. If the reason for determining that there is a problem in step S407 is "the number of cameras is large", the shooting target area 503 is changed to be narrowed. On the contrary, when "the number of cameras is small", the shooting target area 503 is changed to be wide. When the shooting target area 503 is changed by the shooting target area acquisition unit 304, the process proceeds to step S405, and the arrangement determination unit 306 redetermines the number and arrangement of the cameras 102 based on the changed shooting target area 503. The shooting target area acquisition unit 304 may acquire the shooting target area from the operator via the operation unit 206.

In step S411, the arrangement determination unit 306 determines whether or not the arrangement condition to be changed is the lower limit resolution. In the case of the lower limit resolution (YES), the process proceeds to step S412. If not (NO), the process proceeds to step S413.

In step S412, the allowable value acquisition unit 303 changes the lower limit resolution. If the reason for determining that there is a problem in step S407 is "the number of cameras is large", the lower limit resolution is changed to be smaller. On the contrary, if "the number of cameras is small", change the lower limit resolution to be larger. After the lower limit resolution is changed by the permissible value acquisition unit 303, the process proceeds to step S405, and the arrangement determination unit 306 redetermines the number and arrangement of the cameras 102 based on the changed lower limit resolution. The permissible value acquisition unit 303 may acquire the lower limit resolution from the operator via the operation unit 206.

In step S413, the camera group number acquisition unit 305 changes the number of camera groups 103. If the reason for determining that there is a problem in step S407 is "the number of cameras is large", the number of camera groups 103 is changed to be reduced. On the contrary, when "the number of cameras is small", the number of camera groups 103 is increased. After the number of the camera groups 103 is changed by the camera group number acquisition unit 305, the process proceeds to step S404, and the arrangement determination unit 306 determines the arrangement area 701 of the camera group 103 again. Then, the process proceeds to step S405, and the arrangement determination unit 306 redetermines the number and arrangement of the cameras 102. The camera group number acquisition unit 305 may acquire the number of camera groups from the operator via the operation unit 206.

The effect of the arrangement determined by the arrangement determination device 108 will be described with reference to FIG. In each camera group 103, a plurality of cameras 102 belonging to each camera group 103 are arranged so that the entire area of the shooting target area 503 is photographed. That is, a subject at an arbitrary position in the photographing target area 503 is photographed by at least one camera 102 in each camera group 103. Consider the subject at position 1301. In the camera group 103a, the camera 102aa photographs the subject at the position of 1301. In the camera group 103b, the camera 102ba photographs the subject at the position 1301. The same applies hereinafter. In other words, the same number of cameras 102 as the number of camera groups 103 will shoot a subject at an arbitrary position in the shooting target area 503. Further, the arrangement areas 701 of each camera group 103 are determined so as to be as evenly spaced as possible. That is, it is possible to generate a virtual viewpoint image with good image quality for a subject at an arbitrary position in the photographing target area 503.

<Embodiment 2>
In the present embodiment, an example of the arrangement determination device 1600 different from the first embodiment will be described. FIG. 16 shows an example of the functional configuration of the arrangement determination device 1600 in the present embodiment. As shown in FIG. 16, the arrangement determination device 1600 has a camera number acquisition unit 1601 instead of the permissible value acquisition unit 303 of FIG. Further, the arrangement determination device 1600 has an arrangement determination unit 1602 instead of the arrangement determination unit 306 of FIG. Since the hardware configuration of the arrangement determination device 108 is the same as that in FIG. 2, it is omitted.

The camera number acquisition unit 1601 acquires the number of cameras 102 of the multi-camera system 101 directly from the operator via the operation unit 206. For example, the number of cameras 102 may be set based on the processing performance of the virtual viewpoint image generation device 106. The higher the processing performance of the virtual viewpoint image generator 106, the more images can be processed. That is, the higher the processing performance of the virtual viewpoint image generation device 106, the larger the number of cameras 102 can be set. Further, the number of cameras 102 may be set based on the budget. Increasing the number of cameras 102 increases the cost required for placement. That is, the larger the budget, the larger the number of cameras 102 can be set.

Further, the camera number acquisition unit 1601 may have the following configuration. That is, the camera number acquisition unit 1601 stores a table showing the correspondence between the scale of the event and the density of the subjects, which is one of the event information, and the number of cameras 102. Then, when the camera number acquisition unit 1601 receives the input of the information specifying the event from the operator via the operation unit 206, the camera 102 obtains the event information stored in the event information storage unit 301 and the table thereof. You may try to acquire the number of units. For example, when the scale of the event is large, the number of cameras 102 is determined to be larger than when the event is small. Further, when the density of the subjects is large, the number of cameras 102 is determined to be larger than that when the density of the subjects is small.

The arrangement determination unit 1602 determines the arrangement of the cameras 102 based on the number of cameras 102, the shooting target area 503, the number of camera groups, and the venue information. The number of cameras 102 is acquired by the camera number acquisition unit 1601, the shooting target area 503 is acquired by the shooting target area acquisition unit 304, and the number of camera groups is acquired by the camera group number acquisition unit 305.

FIG. 14 is a flowchart showing an example of the processing procedure of the arrangement determination device 108. Since steps S402 to S404 are the same processes as in FIG. 4, description thereof will be omitted.

In step S1401, the camera number acquisition unit 1601 acquires the number of cameras 102. For example, the number of cameras 102 may be set based on the processing performance of the virtual viewpoint image generation device 106. The higher the processing performance of the virtual viewpoint image generator 106, the more images can be processed. That is, the higher the processing performance of the virtual viewpoint image generation device 106, the larger the number of cameras 102 can be. For example, the number of cameras 102 may be set based on the budget. Increasing the number of cameras 102 increases the cost required for placement. That is, the larger the budget, the larger the number of cameras 102 can be.

In step S1402, the arrangement determination unit 1602 allocates the number of cameras 102 acquired in step S1401 to the number of camera groups 103 acquired in step S403. The number of cameras 102 required for the camera group 103 varies depending on the relationship between the imaging target area 503 acquired in step S402 and the arrangement area 701. For example, as shown in FIG. 15, the image pickup target area 503 seen from the arrangement area 701 is represented by an angle 1501. The angle 1501 of the photographing target area 503 as seen from the arrangement area 701a is 1501a. Then, the camera 102 is assigned to the camera group 103 according to the angle 1501. That is, more cameras 102 are allocated to the camera group 103 having a large angle 1501 of the shooting target area 503 as seen from the arrangement area 701.

In step S1403, the arrangement determination unit 1602 determines the arrangement of the cameras 102 for each camera group 103 based on the number of cameras 102 for each camera group 103 assigned in step S1402 and the venue information. The specific arrangement of the camera 102 is determined so that the entire area of the photographing target area 503 is photographed. Further, the camera 102 is arranged so that the resolution is as high as possible in the entire area of the shooting target area 503.

In step S1404, the arrangement determination unit 1602 calculates the resolution in the entire area of the shooting target area 503 for each camera group 103. A subject is photographed by a plurality of cameras 102 in an area overlapping the photographing range of the camera 102. In that case, the value of the camera 102 having a large resolution is used as the resolution.

In step S1405, the arrangement determination unit 1602 determines whether there is a problem with the resolution calculated in step S1404. For example, it is determined that there is a problem when a region having a resolution smaller than a predetermined value is within the imaging target region 503. If it is determined that there is no problem (YES), the process ends. If it is determined that there is a problem (NO), the process proceeds to step S1406.

In step S1406, the placement determination unit 1602 determines which placement condition to change. The arrangement condition that can be changed is either the number of the shooting target area 503 or the number of camera groups 103.

In step S1407, the arrangement determination unit 1602 determines whether or not the condition to be changed is the imaging target area 503. If the condition to be changed is the imaging target area 503 (YES), the process proceeds to step S1408. If the condition to be changed is not the imaging target area 503 (NO), the process proceeds to step S1409.

In step S1408, the shooting target area acquisition unit 304 changes the shooting target area 503. Specifically, the shooting target area acquisition unit 304 is changed to narrow the shooting target area 503 so that the calculated resolution is higher. After the shooting target area 503 is changed by the shooting target area acquisition unit 304, the process proceeds to step S404, and the placement determination unit 306 determines the placement area 701 of the camera group 103 again. Then, the process proceeds to step 1402 and S1403, and the arrangement determination unit 306 redetermines the number and arrangement of the cameras 102. The shooting target area acquisition unit 304 may acquire the shooting target area from the operator via the operation unit 206.

In step S1409, the camera group number acquisition unit 305 changes the number of camera groups 103. Specifically, the camera group number acquisition unit 305 is changed to reduce the number of camera groups 103 so as to increase the calculated resolution. This is because if the number of camera groups 103 is reduced, the number of cameras 102 assigned to the camera group 103 increases, and the resolution becomes higher. After the number of the camera groups 103 is changed by the camera group number acquisition unit 305, the process proceeds to step S404, and the arrangement determination unit 306 determines the arrangement area 701 of the camera group 103 again. Then, the process proceeds to steps S1402 and S1403, and the arrangement determination unit 306 determines the number and arrangement of the cameras 102 again. The camera group number acquisition unit 305 may acquire the number of camera groups from the operator via the operation unit 206.

Also in this embodiment, the same effect as that of the first embodiment can be obtained. That is, subjects at arbitrary positions in the shooting target area 503 are shot from the same number of cameras 102 as the number of camera groups 103. Further, the arrangement areas 701 of each camera group 103 are determined so as to be as evenly spaced as possible. That is, it is possible to generate a virtual viewpoint image with good image quality for a subject at an arbitrary position in the photographing target area 503.

<Embodiment 3>
In the present embodiment, an example of the arrangement determination device 1800 different from the first embodiment will be described. FIG. 18 shows an example of the functional configuration of the arrangement determination device 1800 according to the present embodiment. As shown in FIG. 18, the arrangement determination device 1800 includes an event information storage unit 1801, a venue information storage unit 1802, a permissible value acquisition unit 1803, a shooting target area acquisition unit 1804, a camera group number acquisition unit 1805, and an arrangement determination unit 1806. Have. Further, the arrangement determination device 1800 has a type acquisition unit 1807, an area setting unit 1808, a level information setting unit 1809, and a group number setting unit 1810. However, the type acquisition unit 1807, the area setting unit 1808, the level information setting unit 1809, and the group number setting unit 1810 are functional units used for generating event information, and the arrangement determination device has these functional units. You don't have to. That is, if the event information generated by the type acquisition unit 1807, the area setting unit 1808, the level information setting unit 1809, and the group number setting unit 1810 is stored in the event information storage unit 1801, the camera is arranged based on the event information. Can be determined. Since the hardware configuration of the arrangement determination device 1800 is the same as that in FIG. 2, it is omitted. Hereinafter, the description will be focused on the differences from the first embodiment.

In the present embodiment, a plurality of spaces are defined with respect to a space to be photographed by a plurality of cameras (for example, space 504 in FIG. 5B), and each of the defined plurality of spaces is defined as an imaging target area. And. Then, the arrangement determination device 1800 in the present embodiment determines the number and arrangement of camera groups and the number and arrangement of cameras in each camera group for a plurality of shooting target areas. However, it is not necessary to apply the present embodiment to all the shooting target areas among the plurality of shooting target areas, and the present embodiment may be applied to one of them.

The plurality of defined spaces are all included in the space (space 504 in FIG. 5B) to be photographed by the plurality of cameras. Further, the plurality of defined spaces may be defined exclusively with each other. That is, a plurality of spaces may be defined so as to be composed of a plurality of spaces that divide the space 504 of FIG. 5 (b). Further, of the plurality of defined spaces, any two spaces may partially overlap, and one space may be further included in the other space.

The type acquisition unit 1807 acquires the type of the main foreground (subject) appearing in the event and the movement range thereof. The type acquisition unit 1807 acquires the information by inputting the information regarding the type of the subject and the movement range thereof by the operator via the operation unit 206. An example of this information is shown in FIG. 19 (a). Here, assuming rugby as an event, a person (player) and a ball are input as the main subject types, and the person is divided into a stationary person and a moving person. Further, the range in which each subject moves is input corresponding to the type of the subject. Here, the range of movement indicates the range in which a person or a ball may be present during a rugby game. For example, the moving range of a stationary person indicates the range in which a stationary person may exist. Therefore, the moving range of the stationary person is almost the same as the vertical and horizontal lengths of the field. The height direction is equivalent to, for example, the height of the athlete (for example, 2 m). On the other hand, the moving range of a moving person is wider than the moving range of a stationary person. For example, in the height direction, in rugby, a player may lift another player by throwing in due to a lineout, so the moving range of a moving person is wider than the moving range of a stationary person. Specifically, the height is about 5 m. The range of movement of the ball becomes even wider. For example, in rugby, the kicked ball rises high, so the height of the moving range of the ball is set to 20 m. Also, in the vertical and horizontal directions, that is, in the in-plane direction of the field, a moving person or a ball has a wider range of movement than a stationary person.

The information acquired by the type acquisition unit 1807 is output to the area setting unit 1808, is also output to the event information storage unit 1801, and is stored in the event information storage unit 1801. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19A. However, the event information storage unit 1801 may store the information output from the type acquisition unit 1807 in a format other than the table shown in FIG. 19A.

In inputting the information shown in FIG. 19A, the operator, as in the first embodiment, can identify the type of event and other events, information on the size of the field, and the density of subjects. Enter information about the degree and the purpose of the virtual viewpoint image. However, when the arrangement determination device 1800 is used only for a specific event, the input of the event type and other information that can identify the event can be omitted.

The area setting unit 1808 sets a plurality of shooting target areas based on the information indicating the type of the subject acquired by the type acquisition unit 1807 and the moving range thereof. However, a part of any two or more shooting target areas among the plurality of shooting target areas may partially overlap. FIGS. 20A and 20B show the relationship between the shooting target areas in the example in which the three shooting target areas are set. FIG. 20A is a schematic view of the field viewed from above, and shows the relationship between the vertical and horizontal sizes of the three imaging target areas. Further, FIG. 20B is a side view of the field, showing the relationship between the heights of the three imaging target areas. Here, the first shooting target area 2001 corresponds to the moving range of the stationary person, the second shooting target area 2002 corresponds to the moving range of the moving person, and the third shooting target area 2003 corresponds to the moving range of the ball. .. The range of the first imaging target area 2001, the second imaging target area 2002, and the third imaging target area 2003 becomes wider in this order. Further, the first imaging target area 2001 is included in the second imaging target area 2002, and the second imaging target area 2002 is included in the third imaging target area 2003. However, a part of the first photographing target area 2001, not all of it, may be included in the second photographing target area 2002. A part of the second imaging target area 2002, but not all, may be included in the third imaging target area 2003.

Information indicating a plurality of shooting target areas set by the area setting unit 1808 is output to the level information setting unit 1809, output to the event information storage unit 1801, and stored by the event information storage unit 1801.

The area setting unit 1808 may set the shooting target area so that the same shooting target area corresponds to the moving range of the stationary person and the moving person. Specifically, the area setting unit 1808 does not set the first shooting target area 2001 of FIGS. 20A and 20B, but sets the remaining second shooting target area 2002 and the third shooting target area 2003. You may.

The level information setting unit 1809 sets the required level of the quality of the three-dimensional shape data and the required level of the lower limit resolution for each of the plurality of shooting target areas set by the area setting unit 1808. The level information setting unit 1809 receives an input by the operator via the operation unit 206 and sets the request level. FIG. 19B shows an example of the required level of quality of the three-dimensional shape data and the required level of the lower limit resolution corresponding to each imaging target area. Here, the required level of quality of 3D shape data and the required level of lower limit resolution are defined in three stages of "high", "medium", and "low". The required level of the lower limit resolution may be defined by the number of pixels representing the lower limit resolution as shown in the first embodiment.

The main subject of the first photographing target area 2001 is a stationary person. For a stationary person, the image quality of the virtual viewpoint image can be improved by displaying the virtual viewpoint image with high image quality. Therefore, the level information setting unit 1809 sets the required level of the quality of the three-dimensional shape data and the required level of the lower limit resolution to "high" for the first shooting target area 2001 corresponding to the moving range of the stationary person. And set. The required level of the lower limit resolution is "high", which means at least a level at which the facial features and contours become clear. As a specific numerical example, the number of pixels for a height of 2 m in the captured image is set to 400 pixels. This corresponds to requiring a resolution of at least 0.5 cm / pixel.

The main subject of the second photographing target area 2002 is a moving person. Since a moving person is moving unlike a stationary person, its image quality can be tolerated even if it is lower than that of a stationary person. Therefore, the required level of quality of the three-dimensional shape data and the lower limit resolution level for the second imaging target area 2002 are set to "medium". As a specific numerical example of the lower limit resolution level, the number of pixels for a height of 2 m in a captured image is set to 200 pixels.

The main subject of the third imaging target area 2003 is a ball. In rugby, the ball can move faster than the person and is smaller than the person, so that the image quality can be acceptable even if it is lower than the image quality of the moving person.

In FIG. 19B, the required level of the quality of the three-dimensional shape data and the required level of the lower limit resolution for one imaging target area are the same level, but the required level of the quality of the three-dimensional shape data depends on the event. And the required level of the lower limit resolution do not have to be the same level. For example, the required level of quality of the three-dimensional shape data for one imaging target area may be "high", and the required level of the lower limit resolution for the imaging target region may be "low".

The information set by the level information setting unit 1809 is output to the group number setting unit 1810 and the event information storage unit 1801, and is stored by the event information storage unit 1801. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19B. However, the event information storage unit 1801 may store the information output from the type acquisition unit 1807 in a format other than the table shown in FIG. 19B.

The group number setting unit 1810 sets the number of camera groups 103 from the required level of quality of the three-dimensional shape data set by the level information setting unit 1809. The accuracy of the three-dimensional shape data improves as the number of camera groups increases. Therefore, the higher the request level, the larger the number of camera groups. Each shooting target area and the number of camera groups are shown in FIG. 19 (c). Here, if the required level of quality of 3D shape data is "high", the number of camera groups is set to "24", if it is "medium", it is set to "12", and if it is "low", it is set to "8". An example is shown. The information set by the group number setting unit 1810 is also output to the event information storage unit 1801 and stored by the event information storage unit 1801. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19C. However, the event information storage unit 1801 may store the information output from the type acquisition unit 1807 in a format other than the table shown in FIG. 19C.

The number of groups setting unit 1810 may uniquely set the number of camera groups based on the required level of quality of the three-dimensional shape data, but may also set the number of camera groups based on the density of subjects. In the example of rugby, in the first shooting target area 2001, the density is high because play such as scrum and mall occurs, but in the second shooting target area 2002 and the third shooting target area 2003, the density is low. In consideration of this point, the group number setting unit 1810 sets the number of camera groups. That is, the group number setting unit 1810 is set so that the number of camera groups increases as the density increases.

The event information storage unit 1801 stores information about the event to be photographed. In addition to the information shown in the first embodiment, the event information may include information indicating the type of subject, information indicating the shooting area, information on the density of subjects in each shooting area, and the like, which are set for each event. Good.

The venue information storage unit 1802 stores information about the venue where the plurality of cameras 102 are arranged. The venue information is the same information as the information shown in the first embodiment.

When the camera 102 takes a picture of a predetermined reference subject, the permissible value acquisition unit 1803 sets a lower limit value of the resolution which is the size in the image of the subject. Specifically, the permissible value acquisition unit 1803 is the shooting target based on the information indicating the correspondence between the plurality of shooting target areas set for each event and the required level of the lower limit resolution, which is one of the event information. Set a lower limit for each area. Information indicating the correspondence relationship between the plurality of shooting target areas set for each event and the required level of the lower limit resolution may be stored in the event information storage unit 1801 as a correspondence table as shown in FIG. 19B. Alternatively, it may be stored in an external storage device. As a result, when the permissible value acquisition unit 1803 receives the input of information that can identify the event from the operator via the operation unit 206, the permissible value acquisition unit 1803 sets the lower limit resolution for the plurality of shooting target areas set for the event.

Further, the permissible value acquisition unit 1803 may acquire information such as a numerical value of the lower limit resolution directly from the operator via the operation unit 206.

The shooting target area acquisition unit 1804 determines a plurality of shooting target areas corresponding to the event based on the event information. Specifically, the photographing target area acquisition unit 1804 may have the following configuration. That is, first, the shooting target area acquisition unit 1804 stores a table in which at least a part of the event information and information representing a plurality of shooting target areas are associated with each other. At least a part of the event information includes information indicating the type of the subject, information on the moving range of the subject, information on the size of the field, and the like. Then, when the photographing target area acquisition unit 1804 receives the input of the information that can identify the event from the operator via the operation unit 206, the photographing target area acquisition unit 1804 refers to the event information stored in the event information storage unit 1801 and its table, and a plurality of information. Determine the shooting target area. The table to be referred to may be limited to, but is not limited to, as long as the information on the type of the subject, the information on the moving range of the subject, and the information on the plurality of shooting target areas are associated with each other. Event information other than the above may be associated with information on the shooting target area. Further, in the referenced table, information that can identify the event, such as the name of the event, may be associated with a plurality of shooting target areas. Further, the referenced table may be stored in the event information storage unit 1801 or may be stored in an external device.

Further, the shooting target area acquisition unit 1804 may acquire information on the shooting target area directly from the operator via the operation unit 206.

The camera group number acquisition unit 1805 may set the camera group by referring to the table in which the event information and the number of the camera groups 103 are associated with each other. That is, when the camera group number acquisition unit 305 receives the input of information that can identify the event via the operation unit 206, the camera group number acquisition unit 305 refers to the event information stored in the event information storage unit 1801 and its table, and refers to the shooting target area. The number of camera groups 103 is acquired for each. An example of the referenced table is shown in FIG. 19 (c). The referenced table may be associated with information on the shooting target area and information on the number of camera groups 103, but is not limited to this. Further, the referenced table may be stored in the event information storage unit 1801 or may be stored in an external device.

Further, the camera group number acquisition unit 1805 may acquire information on the number of camera groups 103 directly from the operator via the operation unit 206.

The arrangement determination unit 1806 determines the arrangement of the cameras 102 based on the lower limit resolution, a plurality of shooting target areas, the number of camera groups for each shooting target area, and venue information. The lower limit resolution is acquired by the permissible value acquisition unit 1803, a plurality of shooting target areas are acquired by the shooting target area acquisition unit 1804, and the number of camera groups is acquired by the camera group number acquisition unit 1805. The venue information is stored in the venue information storage unit 302. The arrangement determination unit 306 acquires necessary venue information from the venue information storage unit 302 based on the information for identifying the venue input via the operation unit 206. For example, the venue information to be acquired includes information on a structure in which the cameras 102 can be arranged and their positions in the venue, information on the number of cameras 102 that can be arranged, and the like.

First, the arrangement determination unit 1806 determines the arrangement position of the camera group 103 for each of the plurality of shooting target areas based on the number of camera groups and the venue information, as in the first embodiment. Next, the arrangement determination unit 306 determines the arrangement of the plurality of cameras 102 to which the camera group 103 belongs. At that time, the arrangement determination unit 306 determines the arrangement of the plurality of cameras 102 belonging to each camera group 103 so that the entire area of the imaging target area is photographed by the plurality of cameras 102 belonging to the camera group 103. That is, the arrangement of the plurality of cameras 102 belonging to the camera group 103 is determined so that the plurality of cameras 102 belonging to the camera group 103 of the first shooting target area 2001 shoot the entire area of the first shooting target area 2001. To. The arrangement of the plurality of cameras 102 belonging to the camera group 103 of the second shooting target area 2002 and the plurality of cameras 102 belonging to the camera group 103 of the third shooting target area 2003 are also determined in the same manner.

Further, the arrangement determination unit 1806 determines the arrangement of the plurality of cameras 102 in the plurality of cameras 102 belonging to the camera group 103 so that the resolution exceeds the lower limit resolution in the entire area of the imaging target area. It should be noted that the arrangement of the plurality of cameras 102 is determined so as to satisfy the above-mentioned conditions regarding the shooting target area and the conditions regarding the resolution in all the camera groups 103 corresponding to each shooting target area, but the present invention is not limited to this. .. For example, the arrangement of the plurality of cameras 102 may be determined so that only the plurality of cameras 102 belonging to a predetermined number of camera groups 103 satisfy the above conditions. Further, the arrangement of the plurality of cameras 102 belonging to the camera group 103 may be determined so that the camera group 103 at a predetermined position and the camera group 103 having a predetermined relationship satisfy the above conditions. That is, the plurality of cameras 102 may be arranged so that a plurality of cameras 102 belonging to a part of the camera group 103 capture a part of each shooting target area instead of the entire area. Further, in the plurality of cameras 102 belonging to some camera groups 103, the resolution does not have to exceed the lower limit resolution in a part of each shooting target area.

FIG. 21 is a flowchart showing an example of the processing procedure of the arrangement determination device 1800. The process performed by the arrangement determination device 1800 will be described with reference to FIG.

First, the arrangement determination device 1800 receives an instruction for arrangement determination from the operator via the operation unit 206. Then, in step S2101, the shooting target area acquisition unit 1804 determines and acquires a plurality of shooting target areas. Specifically, a plurality of shooting target areas are determined based on the information that can identify the event input from the operator via the operation unit 206 and the table stored in advance. As a plurality of shooting target areas, as shown in FIG. 20, a first shooting target area 2001, a second shooting target area 2002, and a third shooting target area 2003 are set.

Next, in step S2102, the permissible value acquisition unit 1803 acquires the required level of quality of the three-dimensional shape data and the level of the lower limit resolution. The permissible value acquisition unit 1803 requests the quality of the three-dimensional shape data for each imaging target area based on the information that can identify the event input from the operator via the operation unit 206 and the table stored in advance. And the lower limit resolution level is determined and acquired.

Next, in step S2103, the camera group number acquisition unit 1805 determines and acquires the number of camera groups 103 for each shooting target area. The number of camera groups 103 is determined according to the required level of quality of the three-dimensional shape data generated by the virtual viewpoint image generator 106. The number of directions required to generate the 3D shape data varies depending on the required level of quality of the 3D shape data. That is, when the required level of quality of the three-dimensional shape data is high, that is, when the accuracy of the three-dimensional shape data is increased, it is necessary to shoot from many directions, so that the number of camera groups 103 increases. On the other hand, when the required level of quality of the three-dimensional shape data is low, the accuracy of the three-dimensional shape data may be low, so that the shooting direction may be reduced, and as a result, the number of camera groups 103 can be reduced. it can.

The camera group number acquisition unit 1805 refers to the table by associating the camera group number acquisition unit 1805 with the required level of quality of the three-dimensional shape data and the number of required directions (that is, the number of camera groups). , The number of camera groups can be determined. Since the required level of quality of 3D shape data and the shooting target area correspond to each other, here is a table in which the required level of quality of 3D shape data and the number of camera groups correspond to each other. By referring to 19 (c)), the number of camera groups is determined.

Hereinafter, as an example, it is assumed that the number of camera groups 103 in each of the first shooting target area 2001, the second shooting target area 2002, and the third shooting target area 2003 is determined to be 30, 24, 10. Further, the arrangement area of the camera group 103 may also be determined by this process, but in the present embodiment, an example determined in the next step S2104 is shown.

In step S2104, the arrangement determination unit 1806 determines the arrangement area of the camera group 103 for each shooting target area based on the number of camera groups and the venue information. Since this process corresponds to performing the same process as in S404 of FIG. 4 for each of the plurality of shooting target areas, the description thereof will be omitted.

In step S2105, the placement determination unit 1806 determines the placement of two or more cameras 102 belonging to the camera group 103 for each camera group 103 in each shooting target area based on the placement area of the camera group 103 and the lower limit resolution request level. Determine the number and placement. Since this process corresponds to performing the same process as in S405 of FIG. 4 for each of the plurality of shooting target areas, the description thereof will be omitted. Further, the arrangement of the camera 102 determined by the arrangement determination unit 1806 is the same as that of the first embodiment.

In step S2106, the arrangement determination unit 1806 totals the number of cameras 102 determined for each camera group 103 in each imaging target area in S2105, and determines the number of cameras 102 required for the multi-camera system 101.

In step S2107, the arrangement determination unit 1806 determines whether there is a problem with the number of cameras calculated in step S2106. Since step S2107 is the same process as S407 in FIG. 4, detailed description thereof will be omitted. If it is determined that there is no problem (YES), the process ends. If it is determined that there is a problem (NO), the process proceeds to step S2108.

In step S2108, the arrangement determination unit 1806 determines which arrangement condition is to be changed. The arrangement condition that can be changed is any of the shooting target area, the lower limit resolution, and the number of camera groups 103. Since step S2108 is the same process as S408 in FIG. 4, detailed description thereof will be omitted.

In step S2109, the arrangement determination unit 1806 determines whether or not the arrangement condition to be changed is the imaging target area. In the case of the imaging target area (YES), the process proceeds to step S2110. If not (NO), the process proceeds to step S2111.

In step S2110, the shooting target area acquisition unit 1804 changes the shooting target area. In addition, one or more shooting target areas out of a plurality of shooting target areas are changed. That is, it is not necessary to change the entire shooting target area. If the reason for determining that there is a problem in step S2107 is "the number of cameras is large", the shooting target area is changed to be narrower than one of the plurality of shooting target areas. On the contrary, when "the number of cameras is small", it is changed so that one or more shooting target areas out of a plurality of shooting target areas are widened. When the shooting target area is changed by the shooting target area acquisition unit 1804, the process proceeds to step S2105, and the arrangement determination unit 1806 sets the number and arrangement of the cameras 102 based on the plurality of shooting target areas including the changed shooting target area. To determine again. The shooting target area acquisition unit 1804 may acquire the shooting target area from the operator via the operation unit 206.

In step S2111, the arrangement determination unit 1806 determines whether or not the arrangement condition to be changed is the lower limit resolution. In the case of the lower limit resolution (YES), the process proceeds to step S2112. If not (NO), the process proceeds to step S2113.

In step S2112, the permissible value acquisition unit 1803 changes the lower limit resolution request level. If the reason for determining that there is a problem is "the number of cameras is large", the lower limit resolution requirement level of one or more shooting target areas is changed to be small. On the contrary, when the reason for determining that there is a problem is "the number of cameras is small", the lower limit resolution requirement level of one or more shooting target areas is changed to be large. After the lower limit resolution is changed by the permissible value acquisition unit 1803, the process proceeds to step S2105, and the arrangement determination unit 1806 redetermines the number and arrangement of the cameras 102. The permissible value acquisition unit 1803 may acquire the lower limit resolution request level from the operator via the operation unit 206.

In step S2113, the camera group number acquisition unit 1805 changes the number of camera groups 103. If the reason for determining that there is a problem in step S2107 is "the number of cameras is large", the number of camera groups 103 in one or more shooting target areas is reduced. On the contrary, when the reason for determining that there is a problem is "the number of cameras is small", the number of camera groups 103 in one or more shooting target areas is increased. After the number of camera groups 103 is changed by the camera group number acquisition unit 1805, the process proceeds to step S2104, and the arrangement determination unit 1806 determines the arrangement area of the camera group 103 again. Then, the process proceeds to step S2105, and the arrangement determination unit 1806 determines the number and arrangement of the cameras 102 again. The camera group number acquisition unit 1805 may acquire the number of camera groups from the operator via the operation unit 206.

According to the present embodiment, the same effect as that of the first embodiment is obtained. That is, in each of the camera groups 103 in the shooting target area, a plurality of cameras 102 belonging to the camera group 103 are arranged so that the entire area of the corresponding shooting target area is shot. Therefore, a subject at an arbitrary position in the imaging target area is photographed by at least one camera 102 in each camera group 103. Then, the same number of cameras 102 as the number of camera groups 103 will shoot subjects at arbitrary positions in the shooting target area. Further, the arrangement areas of the respective camera groups 103 are determined to be as evenly spaced as possible. That is, it is possible to generate a virtual viewpoint image with good image quality for a subject at an arbitrary position in the shooting target area.

Further, in the present embodiment, when there are a plurality of required resolution levels in the image depending on the type of the subject and the moving range thereof, the shooting target area is set for each of the plurality of required levels. Then, by determining the number and arrangement of the camera groups 103 and the number and arrangement of the cameras 102 in the camera group for each shooting target area, the appropriate number of cameras according to the type of the subject can be appropriately positioned. Can be placed in. For example, when the space 504 of FIG. 5 of the first embodiment and the third imaging target area 2003 of FIG. 20 are the same, the number of cameras 102 can be reduced as compared with the first embodiment. The reason is that in the first embodiment, the number of camera groups 103 and the number of cameras 102 are uniformly determined regardless of the type of subject and the moving range. For example, in the first embodiment, even in an area distant from the ground, specifically, an area of the third imaging target area 2003 that is not included in the second imaging target area 2002, all the cameras belonging to the camera group shoot. Therefore, the number of cameras 102 increases. However, since only the ball is the subject in that area, the required level of quality of the three-dimensional shape data may be low, and it is not necessary to shoot at a high resolution. Therefore, according to the present embodiment, in consideration of this point, the number of camera groups 103 for photographing the area is reduced, and the number of cameras 102 belonging to the camera group 103 is also reduced.

In the third embodiment, for example, one or more camera groups may be shared between the first shooting target area 2001 and the second shooting target area. In that case, the number of cameras 102 can be further reduced. In this case, the following processing is performed. That is, the processing of S404 is performed as follows. The arrangement determination unit 1806 determines the arrangement of the camera group 103 with respect to the first imaging target area 2001. Further, the arrangement determination unit 1806 determines the arrangement of the camera group 103 with respect to the second imaging target area 2002. Then, it is determined whether or not there is a camera group 103 whose position is close to that of the camera group 103 determined for the second shooting target area 2002 among the camera groups 103 determined for the first shooting target area 2001. If there is a camera group 103 whose position is close to the camera group 103 determined with respect to the second shooting target area 2002 among the camera groups 103 of the first shooting target area 2001, the camera group 103 is set to the first shooting target area 2001. Try to remove from. The point that the number and arrangement of the cameras 102 belonging to the camera group 103 are determined according to the lower limit resolution requirement level and the quality requirement level of the three-dimensional shape data based on the shooting target area corresponding to the camera group 103 is described above. Same as the explanation. For example, it is assumed that 12 camera groups 701a to 701l as shown in FIG. 7 are determined for the first imaging target area 2001. On the other hand, it is assumed that the positions close to the camera groups 701b, 701d, 701f, 701h, 701j, and 701k as shown in FIG. 7 are determined with respect to the second imaging target area 2002. In that case, the camera groups 701b, 701d, 701f, 701h, 701j, and 701k are excluded from the camera group of the first photographing target area 2001. That is, as the camera group of the first photographing target area 2001, the camera groups 701a, 701c, 701e, 701g, 701i, and 701l remain, and the subsequent processing is performed.

As described above, three-dimensional shape data can be appropriately generated even by using a multi-camera system in which one or more camera groups are shared by the first imaging target area 2001 and the second imaging target area 2002. Further, by alternately arranging the camera group of the first shooting target area 2001 and the camera group 103 of the second shooting target area 2002, the quality level can reduce the bias depending on the direction when generating the three-dimensional shape data. Can be done. In the second shooting target area 2002, the three-dimensional shape data of the subject is generated not only by the cameras in the camera group 103 in the second shooting target area 2002 but also by the cameras in the camera group 103 in the first shooting target area 2001. Captured images are also used. However, when determining the pixel value of the virtual viewpoint image, in order to improve the image quality, only the captured image of the camera of the camera group 103 in the first imaging target area 2001, which has a high requirement level of the lower limit resolution, is used. If the images taken by the cameras of the camera group 103 in the first shooting target area 2001 are preferentially used, the images taken by the cameras of the camera group 103 in the second shooting target area 2002 may be used. For example, when the pixel value of the pixel of interest in the virtual viewpoint image is determined by the averaging process using weights for the pixel value of the position corresponding to the pixel of interest in a plurality of captured images, the process is as follows. That is, the weight of the pixel value at the position corresponding to the pixel of interest of the camera of the camera group 103 in the first shooting target area 2001 is set to the pixel of interest of the camera of the camera group 103 of the second shooting target area 2002. Make it larger than the weight of the pixel value at the corresponding position.

Although the first shooting target area 2001 and the second shooting target area 2002 are taken as examples, even if a part of the camera groups of any two or more shooting target areas among the plurality of shooting target areas are shared, the camera groups may be shared. The number of cameras 102 can be reduced.

Further, the type acquisition unit 1807 may acquire information on a plurality of shooting target areas instead of the type of the subject. That is, for example, in FIG. 19A, the type of the shooting target area (first shooting target area 2001, second shooting target area 2002, third shooting target area 2003) and the vertical and horizontal directions for defining the shooting target area are defined. , Information indicating the length of the height may be acquired. In addition, although the explanation was given using rugby as an example of an event, this embodiment can also be applied to ball games such as soccer and athletics. For example, in the 100m dash, which is a track and field race, it is conceivable that a high-quality virtual viewpoint image is required on the track on which the athlete runs, in the vicinity of the goal point, from other places. In that case, various demand levels may be increased with respect to the region near the goal point, and various demand levels may be lowered in other regions.

<Other Embodiments>
The object of the present invention is also achieved by the following. A storage medium in which a program code of software that realizes the functions of the above-described embodiment is recorded is supplied to a system or an apparatus. The computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As the storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, or the like can be used.

In addition, the case where the function of the above-described embodiment is realized by the following processing is also included. Based on the instructions of the program code read by the computer, the OS (operating system) running on the computer performs a part or all of the actual processing.

Further, the case where the function of the above-described embodiment is realized by the following processing is also included. First, the program code read from the storage medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer. Next, based on the instruction of the program code, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing.

304 Imaging target area acquisition unit 306 Arrangement determination unit

Claims (22)

It is an arrangement determination device that determines the arrangement of a plurality of photographing devices used for generating a virtual viewpoint image.
An area acquisition means for acquiring information representing a shooting target area to be shot by the plurality of shooting devices, and
Based on the information acquired by the area acquisition means, the entire area to be photographed is photographed by two or more photographing devices among the plurality of photographing devices, and the two or more photographing devices among the plurality of photographing devices are photographed. An arrangement determining device comprising: an arrangement determining means for determining the arrangement of the plurality of photographing devices so that the entire area to be photographed is photographed by two or more imaging devices different from the apparatus. Allowable value for acquiring the allowable value of the number of pixels representing the size of the reference subject in a predetermined direction in the image captured and acquired by the photographing device when the reference subject exists in the photographing target area. Has more acquisition means,
In the arrangement determining means, the entire area of the imaging target area is photographed by two or more imaging devices among the plurality of imaging devices based on the tolerance acquired by the tolerance acquisition means, and the other The entire area to be photographed is photographed by two or more photographing devices, and the size of the reference subject in the predetermined direction in the image captured by each of the plurality of photographing devices exceeds the allowable value. The arrangement determining device according to claim 1, wherein the arrangement of the plurality of photographing devices is determined. The arrangement determining device according to claim 2, wherein the reference subject is a subject having a predetermined size in a predetermined direction with respect to the photographing target area. The arrangement determination device according to claim 2 or 3, wherein the reference subject is a person having a predetermined height and standing upright. The arrangement determination device according to any one of claims 2 to 4, wherein the allowable value acquisition means determines the allowable value based on the use of the virtual viewpoint image. The arrangement determining device according to claim 5, wherein when the virtual viewpoint image is used in live broadcasting, the permissible value is smaller than when the virtual viewpoint image is used in archive distribution. Further having a number acquisition means for acquiring the number of the photographing devices,
Further, based on the number of units acquired by the number acquisition means, the arrangement determining means captures the entire region of the imaging target area by two or more imaging devices among the plurality of imaging devices, and the other two or more imaging devices. The arrangement determining device according to claim 1, wherein the arrangement of the plurality of photographing devices is determined so that the entire area to be photographed is photographed by the photographing device. The arrangement determination device according to claim 7, wherein the number acquisition means determines the number of units based on the processing performance of the device that generates a virtual viewpoint image. Based on the information about the venue where the plurality of photographing devices are arranged, the arrangement determining means further photographs the entire area of the photographing target area by two or more photographing devices among the plurality of photographing devices, and the other two The arrangement determining device according to any one of claims 1 to 8, wherein the arrangement of the plurality of photographing devices is determined so that the entire area to be photographed is photographed by the photographing device. Further having a group number acquisition means for acquiring the number of groups to which two or more photographing devices belong,
The arrangement determining means is
Based on the number of groups acquired by the group number acquisition means, the arrangement area of the group is determined.
The arrangement determining apparatus according to any one of claims 1 to 9, wherein the arrangement of the photographing apparatus belonging to the group is determined for each group. The arrangement determination device according to claim 10, wherein the group number acquisition means acquires the number of the groups based on the type of the event. The arrangement determining means sets the arrangement area of the group so that the angle formed by the line segment connecting the predetermined position of the imaging target area and each arrangement area of the two adjacent groups does not exceed the predetermined value. The arrangement determining apparatus according to claim 10 or 11, wherein the determination is made. The arrangement determining means according to any one of claims 10 to 12, wherein the arrangement of the plurality of photographing devices is determined so that the distance between the photographing devices in the group does not exceed a predetermined distance. Placement determination device. In the arrangement determining means, among the two or more imaging devices, the imaging device closer to the other two or more imaging devices is the two or more imaging devices than any of the other two or more imaging devices. The arrangement determining device according to any one of claims 1 to 13, wherein the arrangement of the plurality of photographing devices is determined so as to be close to other photographing devices among the photographing devices. The arrangement determining device according to any one of claims 1 to 14, wherein the photographing target area is at least a part of a space to be photographed by the plurality of photographing devices. The space includes a first shooting target area and a second shooting target area different from the first shooting target area.
In the arrangement determining means, the entire area of the first photographing target area is photographed by two or more first photographing devices among the plurality of photographing devices, and the first photographing device among the plurality of photographing devices is used. The arrangement of the plurality of photographing devices is determined so that the entire area of the first photographing target area is photographed by two or more second photographing devices different from the above.
In the arrangement determining means, the entire area of the second photographing target area is photographed by two or more third photographing devices different from the first and second photographing devices among the plurality of photographing devices, and the above-mentioned Among the plurality of imaging devices, the plurality of imaging devices are arranged so that the entire area of the first imaging target area is photographed by two or more fourth imaging devices different from the first to third imaging devices. The arrangement determining apparatus according to claim 15, wherein the determination is made. The arrangement determination device according to claim 16, wherein the second imaging target area includes the first photographing target area. It has multiple imaging devices used to generate virtual viewpoint images,
Two or more of the plurality of photographing devices are arranged so as to photograph the entire area to be photographed by the plurality of photographing devices, and two or more of the plurality of photographing devices. A system characterized in that two or more photographing devices different from the photographing device of the above are arranged so as to photograph the entire area to be photographed. Of the two or more imaging devices, the imaging device closer to the other two or more imaging devices is the other of the two or more imaging devices than any of the other two or more imaging devices. The system according to claim 18, wherein the system is arranged close to the photographing apparatus of the above. The system according to claim 18 or 19, further comprising a generation means for generating the virtual viewpoint image based on an image acquired by photographing by the plurality of photographing devices. It is an arrangement determination method for determining the arrangement of a plurality of photographing devices used for generating a virtual viewpoint image.
An area acquisition step of acquiring information representing an imaging target area to be photographed by the plurality of imaging devices, and an area acquisition step.
Based on the information acquired in the area acquisition step, the entire area of the imaging target area is photographed by two or more imaging devices among the plurality of imaging devices, and the two or more imaging devices among the plurality of imaging devices are photographed. A method for determining an arrangement, which comprises an arrangement determining step of determining the arrangement of the plurality of photographing devices so that the entire area to be photographed is photographed by two or more photographing devices different from the devices. A program for causing a computer to function as the arrangement determination device according to any one of claims 1 to 17.

Images