JP2020005089A - Imaging system, image processing apparatus, image processing method, and program - Google Patents

Abstract

To construct a system that can generate a high-quality virtual viewpoint image with a limited number of cameras.SOLUTION: A system that acquires a multi-viewpoint image for generating a virtual viewpoint image includes a plurality of cameras installed so as to image a predetermined position from different directions, and the plurality of cameras includes a first camera and a second camera installed at a two-fold symmetric position of the first camera about an axis passing through the predetermined position and perpendicular to an imaging field and at the closest position among the cameras to the two-fold symmetric position, and between the first and second cameras, a parameter that affects at least one of the texture resolution of an object in a captured image and the shooting range of the camera from among the parameters defining the characteristics is different.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system for acquiring a plurality of viewpoint images used when generating a virtual viewpoint image and a technique for generating a virtual viewpoint image.

  2. Description of the Related Art A technique for generating an image (virtual viewpoint image) obtained when an object (object) is viewed from an arbitrary virtual viewpoint from an image (multiple viewpoint images) obtained by capturing an object with a plurality of cameras is known. ing. The quality of the virtual viewpoint image depends on the number of cameras that have captured the object and the quality of the multi-viewpoint image. For example, in sports games such as soccer and rugby, objects such as players move around a large imaging space such as a competition field. In such an imaging scene, when a plurality of cameras are installed facing the center of the field, a high-quality virtual viewpoint image can be generated for a player or the like at the center of the field. On the other hand, for a player or the like at the end of the field, since the camera that holds the player or the like within the angle of view is limited, a multi-view image required for generating the virtual viewpoint image cannot be obtained or the quality is low. There is a possibility that only the virtual viewpoint image is obtained. By setting the angle of view of all cameras to be set so that the entire field can be accommodated, sufficient multiple viewpoint images can be obtained regardless of the position of the players etc. on the field, and virtual viewpoint images of constant quality can be generated Yes. However, in such a case, the size of a player or the like appearing in each captured image is small, and the texture thereof has a low resolution. Therefore, the texture of the object in the obtained virtual viewpoint image also has a low resolution.

  In this regard, Patent Literature 1 additionally provides a non-fixed zoom camera that tracks a specific object, and generates a high-quality virtual viewpoint image by additionally using a high-resolution texture for the specific object. A method is described.

JP 2012-185772 A

  According to the technology described in Patent Document 1, the quality of the virtual viewpoint image can be improved only for a specific object tracked by a non-fixed zoom camera. Therefore, if the technique of Patent Document 1 is applied to obtain a sufficient multi-viewpoint image for a sports scene or the like in which many objects such as players exist on the field, an enormous number of cameras are required. Become. Actually, the number of cameras that can be installed in a stadium or the like is limited due to physical space, cost, etc., so a smaller number of cameras can be used for imaging scenes with many objects A possible technology is desired.

  According to the present invention, a system for acquiring a multi-viewpoint image for generating a virtual viewpoint image includes a plurality of cameras installed so as to capture a predetermined position from different directions, and the plurality of cameras include: A first camera, and a two-fold symmetric position of the first camera or the two-fold symmetric position of the plurality of cameras at an axis perpendicular to an imaging field passing through the predetermined position. And a second camera installed at a position closest to the camera, and at least one set of cameras including the first camera and the second camera. It is characterized in that parameters affecting at least one of the texture resolution of the object and the shooting range of the camera are different.

  According to the present invention, a high-quality virtual viewpoint image can be generated with a limited number of cameras.

FIGS. 3A and 3B are diagrams illustrating an example of a camera arrangement for imaging an object from different viewpoints; FIGS. FIG. 4A is a diagram for explaining a captured image obtained when cameras at opposite positions have the same angle of view, and FIG. 7B is an image obtained when cameras at opposite positions have different angles of view. Diagram explaining the image Diagram showing an example of the system configuration (A) is a diagram illustrating an example of a hardware configuration of the control device, and (b) is a diagram illustrating an example of a software configuration of the control device. 5 is a flowchart illustrating a control flow of the image processing system according to the second embodiment. Diagram showing an example of the arrangement of each camera FIGS. 7A and 7B are diagrams illustrating an example of an arrangement of cameras for explaining an outline of a third embodiment; FIGS. 9 is a flowchart illustrating a control flow of the image processing system according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in the present embodiments are necessarily essential to the solution of the present invention. The same components will be described with the same reference numerals.

Embodiment 1

  In the present embodiment, consideration is given to suppressing a reduction in the sense of resolution of the virtual viewpoint image resulting from the small object size in the captured image and reducing the area in which the quality of the virtual viewpoint image is reduced due to the small number of cameras. The method of determining the system configuration will be described. Hereinafter, suppression of a reduction in the resolution of the virtual viewpoint image may be referred to as “improvement of image quality”, and reduction of a region where the quality of the virtual viewpoint image is reduced may be referred to as “improvement of degree of freedom of viewpoint”. The virtual viewpoint image is an image generated by an end user and / or an appointed operator or the like freely operating the position and orientation of a camera (virtual camera) corresponding to the virtual viewpoint, and includes a free viewpoint image and Also called an arbitrary viewpoint image. Further, the virtual viewpoint image may be a moving image or a still image.

  1A and 1B are diagrams illustrating an example of a camera arrangement for imaging an object from different viewpoints according to the present embodiment, where FIG. 1A is a three-dimensional bird's-eye view, and FIG. FIG. In this camera arrangement, images are taken by the seven cameras 11 to 17 and a plurality of viewpoint images for generating a virtual viewpoint image are acquired. The number of seven is an example, and it goes without saying that more cameras are installed when imaging is performed on a vast space such as soccer and rugby. As shown in FIGS. 1A and 1B, the cameras 11 to 17 are arranged so as to surround the entire circumference of the imaging space. The cameras 11 to 17 are installed toward an arbitrary point of gaze 18 set on the imaging field (the bottom surface of the imaging space) 10. Here, the gazing point indicates an arbitrary point (the line of sight of the camera) in the imaging space that the camera gazes at, and the cameras 11 to 17 illustrated in FIG. . Now, the gazing point 18 is located at the intersection between the optical axis of each camera and the bottom surface 18. The position of the gazing point is arbitrary, and may be set, for example, in the air in the imaging space.

  In the related art, in order to improve the degree of freedom of the viewpoint, a wide angle of view is set for all of the cameras 11 to 17 so that approximately the entire imaging space is within the field of view. Therefore, the texture of an object such as a person in a multi-viewpoint image obtained by a plurality of cameras has a low resolution, and as a result, only a low-quality virtual viewpoint image is generated. FIG. 2A is a diagram illustrating a captured image obtained when the camera 11 and the camera 15 located at the opposite position have the same angle of view. When both cameras have the same angle of view, the camera 11 obtains a captured image 24 and the camera 15 obtains a captured image 25. In this case, the numbers and types of the human objects 21 to 23 appearing in both the captured images 24 and 25 are approximately the same (completely coincident in the example of FIG. 2A). Furthermore, when generating the three-dimensional shape data of an object by a method such as the visual volume tolerance method, the silhouette image 26 based on the captured image 24 and the silhouette image 27 based on the captured image 25 are only inverted left and right. There is no significant difference in the information obtained therefrom. Therefore, even if the captured image obtained by the camera 11 (or the camera 15) is excluded and the three-dimensional shape data is generated, the accuracy of the generated three-dimensional shape data of the generated object is higher than the case where the three-dimensional shape data is not excluded. Not much different.

  In the present embodiment, different angles of view are set for the cameras installed facing each other with the point of gaze in between, and one camera has a wide angle of view, and the other camera has a narrow angle of view. Set. Specifically, an angle of view is set for the cameras 11, 12, 13, 14, 16, and 17 such that approximately the entire imaging space including the imaging field 10 is within the field of view. Then, a narrower angle of view is set for the camera 15 located at an opposite position on a line (hereinafter, referred to as a “base line”) 19 connecting the camera 11 and the gazing point 18. FIG. 2B is a diagram illustrating a captured image obtained when the camera 15 at a position facing the camera 11 has a narrower angle of view than the camera 11. In this case, the camera 15 obtains a captured image 28 in which the object is further enlarged (that is, the resolution of the texture is high) and a silhouette image 29 thereof. In this way, by setting different angles of view for cameras installed facing each other with the gazing point therebetween, the resolution of the object can be reduced without lowering the accuracy of the generated three-dimensional shape data. High texture images can be obtained. As a result, it is possible to generate a high-quality virtual viewpoint image while securing the degree of freedom of the viewpoint.

  In the present embodiment, the case where seven cameras are installed has been described as an example. However, the number of cameras may be at least two or more, and the cameras face each other with the point of gaze therebetween (more preferably, the center of gaze is set at the center axis). It is only necessary that a set of cameras be included. At this time, each camera needs to face the same gazing point, but a slight difference within a range that can be evaluated when the cameras are facing the same gazing point is allowed. Note that when completely different points of interest are set for each camera group, the present embodiment is applicable to each point of interest. For example, if there are two camera groups that are respectively responsible for the left half and the right half of the soccer field, and each camera group has a different gazing point (for example, near the front of each goal), This embodiment may be applied to each camera group corresponding to the viewpoint.

  Further, in the present embodiment, a pair of cameras in a pair relationship are cameras equidistant from the gazing point, but the distances from the gazing point may not be the same. Furthermore, the cameras to be combined only need to be at positions substantially facing each other across the point of regard. For example, in FIG. 1B, even when the angle of view of the camera 13 existing at a position slightly deviated from the base line 19 ′ indicated by a broken line connecting the camera 17 and the gazing point 18 with respect to the camera 17 is changed. Good. The degree of deviation allowed is determined in consideration of various conditions such as the size of the space to be imaged and the number of cameras installed. In this manner, a set of cameras may be associated with a camera and a plurality of cameras at a two-fold symmetric position or a two-fold symmetric position of the camera about an axis passing through the point of interest and perpendicular to the imaging field. And the camera installed closest to the camera. In addition, one set of cameras does not necessarily have to face each other with the gazing point therebetween.

  Further, in the present embodiment, the angle of view of one of the paired cameras in the pair is narrower than that of the other camera, but the present invention is not limited to this. For example, the angle of view of the camera whose characteristic is not changed may be set narrower than the standard, and the angle of view of the camera whose characteristic is changed may be widened. Also, the angles of view of the cameras whose characteristics are not changed need not be the same, and may differ for each camera.

  In the present embodiment, the angle of view is changed as a parameter of the camera characteristic that affects the texture resolution of the object. However, the parameter to be changed is not limited to the angle of view. Other parameters that affect the texture resolution of the object, such as the focal length, the size of the imaging sensor, the type of sensor, and the type of lens may be used. For example, by increasing the focal length or changing the lens to a telephoto lens, the same effect as narrowing the angle of view can be obtained. Further, the camera that supports 4K may be changed to a camera that supports 8K.

  By determining the configuration of the cameras installed so as to surround the imaging space in accordance with the concept described above, an image processing system capable of achieving both a degree of freedom of viewpoint and an improvement in image quality in generating a virtual viewpoint image while suppressing the number of cameras. Can be constructed.

Embodiment 2

  In the first embodiment, the basic concept for determining a camera configuration capable of achieving both improvement in the image quality of the virtual viewpoint image and improvement in the degree of freedom of the viewpoint while suppressing the number of cameras to be installed has been described. Next, an image processing system that determines a camera configuration according to the concept described in the first embodiment, acquires a plurality of viewpoint images under the camera configuration, and generates a virtual viewpoint image will be described.

  An apparatus for determining a camera configuration described below acquires camera information indicating the positions and orientations of a plurality of cameras installed to image a predetermined position in an imaging space from different directions, and obtains position information of a gazing point. It is a device that obtains and determines the configuration of the camera based on the information. The apparatus comprises a first camera and a second camera installed at or closest to a two-fold symmetric position about an axis passing through a predetermined position and perpendicular to an imaging field. The configuration is determined for at least one set of cameras consisting of Specifically, with this device, the parameters that affect at least one of the texture resolution of the object in the captured image and the shooting range of the camera differ between the first camera and the second camera that constitute one set of cameras. Next, the camera configuration is determined. In addition, the camera configuration is such that a set of cameras facing each other across a predetermined position has different parameters affecting the texture resolution of the object in the captured image among the parameters defining the characteristics of the two cameras. Is determined. That is, by using this device, it is possible to input the camera information and the position information of the gazing point from the outside, and determine an appropriate camera configuration.

(System configuration)
An image processing system for generating a virtual viewpoint image according to the present embodiment will be described with reference to a system configuration diagram in FIG. The image processing system 100 includes imaging modules 110a to 110g, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end user terminal (not shown). That is, the image processing system 100 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The image collection domain includes the imaging modules 110a to 110g, the data storage domain includes the DB 250 and the server 270, and the image generation domain includes the control device 300 and the end user terminal.

  The control device 300 performs various controls, such as management of an operation state and setting of various parameters, on each block constituting the image processing system 100 via a network. With the control device 300, a series of processes such as determination of the camera configuration, acquisition of a plurality of viewpoint images, setting of a virtual viewpoint, and generation of a virtual viewpoint image are realized. Details of the control device 300 will be described later.

  Each of the imaging modules 110a to 110g has one camera 112a to 112g. In the following, the seven sets of systems including the imaging modules 110a to 110g may be simply referred to as “imaging module 110” without distinction. Similarly, the device in each imaging module 110 may be described as “camera 112” or “camera adapter 120”. Note that the number of the imaging modules 110 is set to seven, but this is merely an example and the present invention is not limited to this. The imaging modules 110a to 110g are connected by a daisy chain. With this connection form, there is an effect that the number of connection cables can be reduced and the wiring work can be saved in increasing the capacity of image data accompanying higher resolution of a captured image to 4K or 8K and higher frame rate. The connection form is arbitrary. For example, a star-type network configuration in which the imaging modules 110a to 110g are connected to the switching hub 180 and data is transmitted and received between the imaging modules 110 via the switching hub 180 may be adopted. The camera adapter 120 performs data communication with other camera adapters 120, the server 270, and the control device 300. In addition, for the image data captured by the camera 112 and the image data received from the other camera adapter 120, processing such as foreground / background separation processing, foreground three-dimensional shape data source information generation processing, and dynamic calibration is performed. The camera 112 is controlled based on a control signal from the control device 300. More specifically, start and stop of imaging, setting of imaging parameters (number of pixels, color depth, frame rate, and white balance), and status information of the camera 112 (imaging, stopping, synchronizing, error, etc.) Acquisition, setting change of angle of view and focal length, etc. are performed. The image captured by the camera 112a in the imaging module 110a is subjected to predetermined image processing in the camera adapter 120a, and then transmitted to the camera adapter 120b of the imaging module 110b. Similarly, the imaging module 110b transmits the image captured by the camera 112b to the imaging module 110c together with the image acquired from the imaging module 110a. By continuing such an operation, the captured images acquired by the imaging modules 110a to 110g are transmitted from the imaging module 110g to the switching hub 180, and then transmitted to the server 270.

  The DB 250 stores data of a plurality of viewpoint images captured by each imaging module 110 based on an instruction from the control device 300, and transmits and receives image data and the like to and from the server 270 by high-speed communication such as InfiniBand. In addition, it holds 3D data of facilities and structures in the imaging space and functions as a background 3D data library. The 3D data of the facility or the structure is prepared in advance according to the shooting scene (sports event or event), and is transmitted to the server 270 in response to a request from the control device 300.

  The server 270 writes the image data acquired from the imaging module 110g into the DB 250 according to the camera identification number, data type, and frame number. Then, the server 270 receives information (virtual viewpoint information) related to the virtual viewpoint based on the user designation or the like from the control device 300. Then, rendering processing is performed according to the received virtual viewpoint information to generate a virtual viewpoint image. A rendered image (virtual viewpoint image) representing the appearance from the virtual viewpoint obtained by the rendering process is transmitted from the server 270 to the control device 300. Then, the user can browse images according to the virtual viewpoint designated by the user.

(Details of control device)
FIG. 4A is a diagram illustrating an example of a hardware configuration of the control device 300. The control device 300 includes a CPU 301, a RAM 302, an HDD 303, a network I / F 304, a general-purpose I / F 305, a monitor 308, and a main bus 309. The network I / F 304 is an interface for connecting the DB 250, the server 270, and the switching hub 180 to the main bus 309. The general-purpose I / F 305 is an interface for connecting an input device 306 such as a mouse and a keyboard, and an external memory 307 such as a memory card to the main bus 309. FIG. 4B is a functional block diagram illustrating a software configuration of the control device 300 according to the present embodiment. The camera configuration determination unit 401, the camera control unit 402, the virtual viewpoint setting unit 403, and the virtual viewpoint image generation It comprises a unit 404. These units are realized by the CPU 301 expanding a predetermined program stored in the HDD 303 in the RAM 302 and executing the program.

(System control flow)
Subsequently, a flow of control of the image processing system by the control device 300 according to the present embodiment will be described. FIG. 5 is a flowchart showing a control flow in the image processing system shown in FIG. 3 according to the present embodiment. Hereinafter, details of the processing performed by each unit illustrated in FIG. 4B will be described with reference to the flow illustrated in FIG. The symbol “S” in the following description represents a step.

  In step S501, the camera configuration determination unit 401 acquires camera information indicating the positions and orientations (optical axis directions) of a plurality of cameras installed. The camera information defines three-dimensional coordinate axes in which the bottom surface of the imaging space is the x and y axes and the height is the z axis, and the position of each camera is the position pi (pxi, pyi, pzi) in the coordinate space, This is information expressing the posture by a vector vi [vxi, vyi, vzi]. Here, i is the identification number of each of the installed cameras. Note that the contents of the camera information are not limited to those described above, but may be information on the coordinate position of each camera and the coordinate position of the gazing point.

  FIG. 6 is a diagram illustrating an example of an arrangement of each camera specified by the camera information. Now, seven cameras 112 a to 112 g shown in FIG. 3 are installed facing the gazing point 601. In this case, for example, it is assumed that the angles of view of the cameras 112e to 112g in which another camera exists at a position facing the own camera are narrower than the angles of view of the cameras 112b to 112d at the respective facing positions. In this case, the texture of the object when viewed from the direction of the cameras 112e to 112g has a high resolution. However, the texture when viewed from the opposite direction 602 has a low resolution, and the image quality of the entire object cannot be improved. Therefore, in the subsequent steps S502 to S507, cameras having a narrow angle of view but capable of capturing an object with high resolution are arranged at substantially equal intervals on the entire circumference of the imaging space.

  In S502, the camera configuration determining unit 401 refers to the camera information acquired in S501, and determines one camera of interest (camera of interest) from all the installed cameras. Normally, the camera of interest is determined in ascending order of the number i for identifying the camera. In subsequent S503, the camera configuration determining unit 401 uses the camera information acquired in S501 to perform a process of determining whether or not another camera exists at a position facing the gazing point with respect to the target camera. This determination process is performed, for example, in the following procedures 1) to 3).

  1) A vector indicating the line-of-sight direction of each camera is projected on a two-dimensional plane, and a direction vector vi [vpi, vqi] obtained by normalization is calculated for each camera. Here, i is a number for identifying each camera, and i = 1 to 7 now.

  2) The inner product vo · vj of the direction vector vo [vpo, vqo] of the camera of interest and the direction vector vj [vpj, vqj] of the camera other than the camera of interest is calculated. Here, o represents the identification number of the camera of interest, and j represents the identification number of a camera other than the camera of interest. For example, when i = 1, o = 1 and j = 2-7.

  3) If there is a camera in which the value of the inner product vo · voj calculated in the above 2) is negative and the value obtained by inverting the sign of the inner product vo · vj is 1, another camera exists at a position opposite to the camera of interest. I judge it.

  The result of the determination process obtained in this manner is held in association with the camera of interest as information on the presence or absence of the opposite camera. The on-coming camera information includes, for example, a first value and a second value. The first value is 1-bit information indicating “1” when another camera exists at the opposite position and “0” when there is no other camera. Then, the second value is information on the identification number of the other camera when the first value is “1” (when another camera is present at the opposing position). Therefore, if the first value is "0", the second value is blank (0). At the start of this processing, it is assumed that the opposite camera presence information of each camera is initialized, the first value is “0”, and the second value is blank. Here, the presence / absence of a camera in a facing positional relationship is determined using the two-dimensional plane and the inner product, but the determination method is not limited to this. Any method may be used as long as it can determine whether or not there is another camera facing the target camera. As a result of this determination processing, when another camera exists at a position facing the own camera, the first value is updated to “1” and the second value is updated to the identification number of the other camera. Hereinafter, a camera whose first value in the on-coming camera existence information is “1” is referred to as “a camera-to-camera”.

  In S504, it is determined whether or not the determination processing in S503 has been completed for all cameras having the same gazing point according to the camera information acquired in S501. If there is an unprocessed camera, the process returns to S502, sets the next camera as the camera of interest, and continues the processing. On the other hand, if the determination process has been completed for all cameras, the process proceeds to S505.

  In step S505, the camera configuration determining unit 401 derives an angle (reference angle) that is used as a reference when determining a camera whose characteristics are to be changed, using the presence / absence information of the opposite camera of each camera. The reference angle when there is no particular limitation on the direction in which the virtual viewpoint can be set is a value obtained by dividing half the total number of the first values in the on-camera presence information from 360 degrees. In the example of FIG. 6 described above, the first value is “1” in the on-camera presence information linked to the six cameras (112b, 112c, 112d, 112e, 112f, and 112g). 6 ÷ 2)) = 120 degrees is the reference angle. That is, in the case of the camera arrangement shown in FIG. 6, it is determined in this step that a camera having a narrow angle of view but capable of imaging an object with high resolution is arranged at a uniform angle of 120 degrees all around the imaging space.

  In step S <b> 506, the camera configuration determination unit 401 determines the camera that serves as a reference when determining the camera whose characteristics are to be changed, among all installed cameras 112 a to 112 g, in which the first value in the opposite camera existence information is “1”. Is determined from among the paired cameras. Here, one camera randomly selected from the paired cameras is set as a reference camera (hereinafter, referred to as a “reference camera”). The determination of the reference camera is not limited to this. For example, a camera having the shortest distance between the paired cameras may be set as the reference camera.

  In step S507, the camera configuration determining unit 401 changes the characteristic based on the camera information acquired in step S501, the reference angle derived in step S505, and the reference camera determined in step S506 (hereinafter, referred to as a “characteristic changed camera”). ). This determination process is performed, for example, in the following procedures 1) to 3).

  1) Calculate the angle θbk between the reference camera b and the camera k adjacent to the reference camera b in the clockwise direction. The angle θbk can be obtained, for example, by the inner product of the direction vectors between the cameras. Here, k indicates an identification number of an adjacent camera pair.

  2) Compare the calculated angle θbk with the reference angle. As a result of the comparison, if the angle θbk is equal to or larger than the reference angle, the paired camera k adjacent to the reference camera is determined as the characteristic-changed camera, and the reference camera is replaced with the paired camera k. On the other hand, if the angle θbk is smaller than the reference angle, it is determined that the camera k is not a characteristic-change camera.

3) The above processes 1) and 2) are sequentially performed clockwise in all combinations of cameras until one of the cameras is determined as the characteristic-changed camera.
In this way, it is determined whether or not all paired cameras including the reference camera are to be the characteristic-changed cameras.

  Here, how the characteristic-change camera is determined by the respective processes from S502 to S507 will be described, taking the case of FIG. 6 as an example. First, referring to the camera information of the seven cameras 112a to 112g, it is determined whether or not there is a combination of cameras at positions facing each other with the point of regard 601 interposed therebetween. As a result of the determination, a camera belonging to each combination of the cameras 112b and 112e, the cameras 112c and 112f, and the cameras 112d and 112g is determined as "a camera-to-camera". Next, it is assumed that the reference camera is randomly selected from the paired cameras, and for example, the camera 112d is selected as the reference camera. Here, the paired camera 112c clockwise adjacent to the reference camera 112d is determined not to be a characteristic-changed camera since the angle with the reference camera 112d is less than the reference angle of 120 degrees. Further, since the angle between the adjacent paired camera 112b and the reference camera 112d is equal to or larger than the reference angle of 120 degrees, the camera is determined as a characteristic-changed camera, and the camera 112b becomes a new reference camera. In the new reference camera 112b, since the adjacent paired camera 112g is smaller than the reference angle and the adjacent paired camera 112f is larger than the reference angle, the camera 112f is determined as the characteristic-changed camera. In all combinations of cameras (cameras 112b and 112e, cameras 112c and 112f, and cameras 112d and 112g), this process was repeated until one of the cameras was determined to be a characteristic-changed camera. 112d and the camera 112f are determined as the characteristic-changed cameras. If the range in which the virtual viewpoint can be set is limited, in the derivation of the reference angle in S505, the value to be divided may be not 360 degrees but an angle within the limited range (for example, 180 degrees). In this case, the reference camera determined in S507 is selected from cameras within the limited range. In this way, information on the identification number of the camera 112 determined as the characteristic-change camera is sent to the camera control unit 402. Returning to the description of the flow of FIG.

  In step S508, the camera control unit 402 transmits a control signal for changing a parameter that defines the characteristics of the camera 112 determined as the characteristic-change camera to the imaging module 110 including the camera 112. The parameter in the present embodiment is the angle of view, and a control signal instructing a change to a narrower angle of view is transmitted. The camera adapter 120 in the imaging module 110 that has received the control signal performs setting for changing the angle of view of the camera 112 to a narrower angle of view. Thus, the characteristic-change camera has a smaller angle of view than the angle of view of the camera located at the position opposite thereto. As a result, the object in the image captured by the camera becomes larger (that is, a captured image with a higher texture resolution of the object is obtained). At this time, how much the angle of view is changed may be in accordance with a user instruction via a user interface, or may be changed to a predetermined value determined in advance. Further, as described in the first embodiment, the parameter to be changed is not limited to the angle of view.

  In step S509, the camera control unit 402 transmits a control signal for instructing start of imaging to the imaging modules 110a to 110g. As a result, imaging is started in each imaging module 110, and an image captured by each camera 112 is obtained. In this way, data of a multi-view image obtained by imaging the object from a plurality of different viewpoints is obtained and stored in the DB 250. When the captured image is a moving image, the following processes are executed for each frame.

  In S510, the virtual viewpoint image generation unit 404 instructs the server 270 to create three-dimensional shape data (shape estimation) of each object included in the multiple viewpoint images acquired in S509. In response to this instruction, the server 270 cuts out the object area from each captured image and estimates the three-dimensional shape of each object. For estimating the three-dimensional shape based on the multiple viewpoint images, a known method such as a visual volume tolerance method or a stereo matching method may be used.

  In step S511, the virtual viewpoint setting unit 403 sets a virtual viewpoint indicating the three-dimensional position and orientation (optical axis direction) of the virtual camera. At this time, the angle of view and the resolution in the generated virtual viewpoint image may be set together. The setting is performed based on a user instruction input via a virtual viewpoint setting UI screen (not shown) displayed on the monitor 308. Alternatively, information of a predetermined virtual viewpoint may be read from the HDD 303 or the like and automatically set.

  In S512, the virtual viewpoint image generation unit 404 instructs the server 270 to generate a virtual viewpoint image according to the virtual viewpoint set in S511. In response to this instruction, the server 270 generates a virtual viewpoint image by using a known viewpoint-dependent rendering method using the multiple viewpoint images acquired in S509 and the three-dimensional shape data created in S510. The generated virtual viewpoint image is output to the monitor 308, and the user can browse the virtual viewpoint image from an arbitrary viewpoint set by the user.

  The above is the flow of the system control according to the present embodiment. In the flow of FIG. 5, the reference camera is selected from the paired cameras, and the characteristic-change camera is determined. However, the present invention is not limited to this. For example, a reference camera is selected from cameras other than the paired camera, a camera whose characteristics are not changed (characteristic unchanged camera) is determined according to the above-described method, and the remaining paired cameras are finally determined as characteristics. It may be a change camera. Although the determination is made clockwise in the flow of FIG. 5, the determination may be made counterclockwise. In the flow of FIG. 5, the reference angle is calculated by calculation, but the characteristic-changed cameras may be arranged evenly according to a predetermined angle.

<Modification>
In the present embodiment, the control device 300 controls a series of processes such as determination of a camera configuration, acquisition of a plurality of viewpoint images, setting of a virtual viewpoint, and generation of a virtual viewpoint image, but is not limited thereto. For example, each processing up to S508 for determining the camera configuration is performed by a separate and independent information processing device (PC), and the installation and adjustment of the camera are performed in advance, and the control device 300 performs only the processing after S509. It may be a configuration.

  In the present embodiment, the camera of interest is determined as a paired camera only when another camera exists at a position completely symmetrical with respect to the camera of interest. However, the present invention is not limited to this. That is, if there is another camera that can be evaluated as having an approximately symmetrical positional relationship when viewed from the own camera, the camera of interest may be set to the opposite camera. In that case, the camera of interest may be set to the paired camera on the condition that there is another camera in which the value obtained by inverting the sign of the inner product vo · vj is larger than a predetermined threshold. Here, the completely symmetrical relationship refers to a relationship in which positions are symmetrical twice with respect to an axis passing through the point of gaze and perpendicular to the imaging field.

  In the present embodiment, the determination as to whether or not the camera is a camera is made based on the two-dimensional direction vector, but is not limited to this. For example, the determination may be made based on a three-dimensional positional relationship including the height direction, and if the heights are different, it may be determined that the camera is not a camera. For example, if the installation floors are different, such as the first floor and the second floor of a stadium, the camera on the first floor and the camera on the second floor can be determined as a paired camera based on the two-dimensional direction vector, Is so different that the camera is not determined.

  Further, in the present embodiment, the reference camera is determined only once, but is not limited to this. The camera whose first value in the oncoming camera existence information is “1” is sequentially determined as the reference camera, and the processing up to S508 is repeatedly performed so that the arrangement of the characteristic-changed camera is within the settable range of the virtual viewpoint. A camera configuration that provides the closest uniformity may be adopted.

Embodiment 3

  In the second embodiment, among all the installed cameras, all the cameras in which another camera exists at a position opposite to the common gazing point are set as candidates of the characteristic changing camera, and one of the paired cameras is Was determined to be a characteristic changing camera. Next, an embodiment in which a predetermined number of cameras installed among a plurality of installed cameras are determined as characteristic-change cameras will be described as a third embodiment. This aspect is effective in a case where, although a plurality of cameras installed are roughly opposed to each other with a common gazing point therebetween, both cameras are not completely symmetrical. The description of the contents common to the second embodiment will be omitted or simplified, and the following description will focus on the differences.

  In the present embodiment, the characteristic-change camera is determined based on the degree of symmetry of the camera and the degree of proximity between the baselines. The reason will be described first.

  In cases where the baseline extending from each camera does not exist evenly over the entire circumference of the imaging space, and there is a large gap between the cameras, a sufficient silhouette image can be obtained compared to a system without such a gap. I can't. In this case, since the accuracy of the three-dimensional shape data of the object in the captured scene is reduced, the generated virtual viewpoint image also has low image quality. Therefore, it is desirable that the intervals between the cameras to be installed be as close and uniform as possible. Further, as described above, the two silhouette images based on the captured images of the cameras at positions opposing each other with the point of regard interposed therebetween are merely left and right inverted. Therefore, there is no significant difference in the information obtained from the silhouette image of each captured image between the system including the camera arrangement facing each other with the gazing point in between, and the image quality of the generated virtual viewpoint image. There is no significant difference.

  Summarizing these facts, if there is another camera at the position facing the own camera and there is another camera near the own camera, even if it is excluded from the camera group constituting the system, It is considered that the image quality of the generated virtual viewpoint image does not decrease so much. Thus, in the present embodiment, a camera having a high degree of symmetry of the opposing camera and a high degree of base line proximity is determined as the characteristic-changed camera.

  An outline of the present embodiment will be described with reference to FIGS. FIGS. 7A and 7B are diagrams illustrating an example of the arrangement of cameras according to the present embodiment, in which five cameras 112a to 112e are installed so as to surround an imaging space. In such a camera arrangement, it is assumed that the characteristics of one camera are changed.

  FIG. 7A shows the case where the camera 112a is selected as the characteristic changing camera, and FIG. 7B shows the case where the camera 112d is selected as the characteristic changing camera. In FIG. 7A, a baseline 701 extending from the camera 112d is adjacent to a baseline 702 extending from the camera 112c and a baseline 703 extending from the camera 112b on the other side of the gazing point 700. 702 is clearly closer. That is, the angle α1 between the base line 701 and the base line 702 is considerably smaller than the angle α2 between the base line 701 and the base line 703. On the other hand, in FIG. 7B, the base line 704 extending from the camera 112a is adjacent to the base line 702 extending from the camera 112c and the base line 703 extending from the camera 112b, and is substantially intermediate between the base line 702 and the base line 703. In the position. That is, the angle β1 formed by the base line 704 and the base line 702 is not so different from the angle β2 formed by the base line 704 and the base line 703.

  When generating three-dimensional shape data of an object by a method such as the visual volume tolerance method, it is necessary to image the object from all directions and acquire a silhouette image of the object in order to improve the accuracy. In the situation of FIG. 7A, when the characteristics of the camera 112a are changed, it becomes difficult to obtain a silhouette image in an area between the base line 701 and the base line 703, and as a result, the accuracy of the three-dimensional shape data deteriorates. Then, the image quality of the virtual viewpoint image generated based on such three-dimensional shape data also deteriorates. On the other hand, in FIG. 7B, since the intervals between the base lines are approximately equal, the silhouette images can be acquired without bias from different directions. Therefore, even if the characteristics of the camera 112d are changed, the image quality of the virtual viewpoint image is not significantly deteriorated as compared with the case where the characteristics are not changed. Therefore, as shown in FIG. 7B, a camera at a position that has less influence on acquisition of a silhouette image is determined as a characteristic-changed camera. For this purpose, the proximity of the baseline on the other side of the gazing point is used as a new judgment index. In addition, the case where the number of installed cameras is five in total and one of them is a characteristic changing camera has been described as an example, but the number of cameras to be installed and the number of cameras for changing characteristics are arbitrary. Needless to say.

(System control flow)
Next, the control flow of the image processing system according to the present embodiment will be described with reference to the flowchart of FIG. The flow of FIG. 8 corresponds to the flow of FIG. 5 of the second embodiment, and the hardware configuration and the software configuration of the control device 300 are the same as those described in the second embodiment (FIGS. 4A and 4B). See).

  In step S801, in addition to camera information indicating the positions and orientations (optical axis directions) of a plurality of installed cameras, information on the number of cameras whose characteristics are to be changed among all the installed cameras is acquired. In subsequent S802, as in S502 of the flow in FIG. 5, one camera of interest is determined from all the installed cameras by referring to the camera information acquired in S801.

  In step S803, the camera configuration determination unit 401 derives the degree of symmetry between the camera of interest determined in step S802 and the camera at the most opposing position among cameras other than the camera of interest using the camera information acquired in step S801. I do. This derivation process is performed, for example, in the following procedures 1) to 3).

  1) A vector indicating the line-of-sight direction of each camera is projected on a two-dimensional plane, and a direction vector vi [vpi, vqi] obtained by normalization is calculated for each camera. Here, i is the identification number of each camera, and i = 1 to 5 now.

  2) The inner product vo · vj of the direction vector vo [vpo, vqo] of the camera of interest and the direction vector vj [vpj, vqj] of the camera other than the camera of interest is calculated. Here, o represents the identification number of the camera of interest, and j represents the identification number of a camera other than the camera of interest. For example, when i = 1, o = 1 and j = 2-5.

  3) A camera is obtained in which the value of the inner product vo · vj calculated in the above 2) is negative and the value obtained by inverting the sign of the inner product vo · vj is the maximum, and the sign of vo · vj of the camera is inverted. The value is used as an index indicating the degree of symmetry of the oncoming camera with respect to the camera of interest.

  The index value thus obtained is held in association with the camera of interest as the opposite camera symmetry information. The index value increases as the position of the other camera facing the own camera with the point of gazing between them is closer to the symmetric position (the maximum value is obtained when the camera is completely symmetric), indicating a higher degree of symmetry. Will be.

  In step S <b> 804, the camera configuration determination unit 401 acquires in step S <b> 801 how close the baseline extending from the camera of interest determined in step S <b> 802 is to the baseline extending from another camera adjacent to the reference line across the gazing point. It is derived using the obtained camera information. This derivation process is performed, for example, in the following procedures 1) to 4).

  1) A vector indicating the line-of-sight direction of each camera is projected on a two-dimensional plane, and a direction vector vi [vpi, vqi] obtained by normalization is calculated for each camera. Here, i is the identification number of each camera, and i = 1 to 5 now.

  2) The backward vector vo '[v'po, v'qo] when the target camera is viewed from its symmetric position with respect to the point of regard is calculated based on the direction vector of the target camera. Here, o is the identification number of the camera of interest.

  3) The inner product vo '· vi of the backward vector and the direction vector of the camera other than the camera of interest and the outer product vo' × vi of the direction vector of the camera other than the camera of interest and the vector of the camera other than the camera of interest are calculated for each camera.

  4) A positive value in which the inner product vo '· vi is the largest in a camera having a positive outer product vo' × vi value, and a largest inner product vo '· vi in a camera in which the outer product vo' × vi value is negative. Is compared with the negative value. Then, the smaller value is used as an index indicating the degree of proximity of the baseline to the camera of interest. The index value thus obtained is held in association with the camera of interest as baseline proximity information. With respect to the base line proximity based on the base line extending from the own camera, the smaller the distance between adjacent base lines, the smaller the index value and the higher the closeness.

  In S805, it is determined whether the derivation processing in S803 and S804 has been completed for all cameras (all installed cameras) related to the camera information acquired in S801. If there is an unprocessed camera, the process returns to step S802 to set the next camera as the camera of interest and continues the processing. On the other hand, if the two types of derivation processes have been completed for all the cameras, the process proceeds to S806.

  In step S806, the camera configuration determination unit 401 derives a reference angle for determining a characteristic-change camera using the number-of-characteristic-change-cameras information acquired in step S801. The reference angle when there is no particular limitation on the direction in which the virtual viewpoint can be set is a value obtained by dividing the number of characteristic-change cameras from 360 degrees. In the case of the example of FIG. 7 described above, the number of cameras whose characteristics are changed is one, so that 360 ÷ 1 = 360 degrees.

  In step S807, the camera configuration determining unit 401 determines a reference camera for determining the characteristic-changed camera from all the installed cameras (here, 112a to 112e). Specifically, the camera having the highest degree of symmetry is determined as the reference camera with reference to the opposite camera symmetry information of each camera derived in S803. If there are a plurality of cameras having the highest degree of symmetry, the camera may be determined at random from among them.

  In S808, the camera configuration determining unit 401 determines a characteristic-changed camera based on the on-coming camera symmetry information derived in S803 and the baseline proximity information derived in S804. This determination process is performed, for example, in the following procedures 1) to 3).

  1) The position of the reference camera is determined, and two cameras existing near a position separated by a reference angle clockwise from the reference camera are detected. The reference camera at this time is the reference camera determined in S807 in the first loop, and the reference camera updated in S810 in the second and subsequent loops. Further, under certain conditions such as when the reference angle is 360 degrees as in the example of FIG. 7, the reference camera itself is included in the two cameras to be detected. The fixed condition is determined by the number of installed cameras, the interval between cameras, a reference angle, and the like.

2) With respect to the two cameras detected in the above 1), an evaluation value for determining which of the two cameras is to be the characteristic-changed camera is obtained by using the following equation (1).
Evaluation value = Symmetry of facing camera × (1 / Baseline proximity) Expression (1)

  3) The evaluation values of the two cameras obtained in the above 2) are compared, and the camera with the larger evaluation value is determined as the characteristic-changed camera (the camera with the smaller evaluation value is determined as the camera whose characteristics are not changed). I do. In this way, one characteristic-change camera corresponding to the reference camera is determined.

  In step S809, it is determined whether or not the number of characteristic-change cameras corresponding to the number acquired in step S801 has been determined. If there is an undetermined portion, the process proceeds to S810. On the other hand, if all the characteristic-change cameras have been determined, the process advances to step S811.

  In S810, the reference camera is updated. Specifically, the camera determined to have a larger evaluation value in S808 (the camera determined as the characteristic-change camera) is set as a new reference camera. Upon completion of the update, the process returns to S808, and the process of determining the next characteristic-change camera based on the updated new reference camera is continued.

  Steps S811 to S815 correspond to steps S508 to S512 in the flow of FIG. 5, respectively, and there is no particular difference.

  The above is the control flow of the image processing system according to the present embodiment. As a result, it is possible to construct a system including a camera group in which the camera characteristics are changed by the number specified by the user.

<Modification>
In the present embodiment, the degree of symmetry of the opposite camera and the degree of base line proximity are derived only once for each camera, but the present invention is not limited to this. Each time the characteristic-changed camera is determined, the next characteristic-changed camera may be determined after re-deriving the opposite camera symmetry and the baseline proximity for the remaining cameras excluding the determined camera. .

  Further, when determining the characteristic-change camera, two cameras close to the position distant from the reference camera by the reference angle were compared, but three or more cameras near the position were compared, and the characteristic-change camera was compared. You may decide.

  According to the present embodiment, even when there is no combination of cameras having a completely symmetrical positional relationship among the installed cameras, it is possible to achieve both the degree of freedom of the viewpoint and the improvement of the image quality in generating the virtual viewpoint image. System can be constructed.

<Other Examples>
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Claims (19)

A system for acquiring a plurality of viewpoint images for generating a virtual viewpoint image,
Having a plurality of cameras installed to image a predetermined position from different directions,
The plurality of cameras include a first camera and a two-fold symmetrical position or the two-fold symmetrical position of the first camera about an axis passing through the predetermined position and perpendicular to an imaging field. A second camera located closest to the plurality of cameras, and at least one set of cameras,
A system wherein the first camera and the second camera are different in parameters that affect at least one of a texture resolution of an object in a captured image and a photographing range of the camera among parameters defining characteristics of the first camera and the second camera.   When the set of cameras is included in the plurality of cameras, the cameras constituting the plurality of cameras are arranged such that the intervals between the cameras having the different parameters are substantially equal. The system of claim 1, wherein:   The system according to claim 1, wherein the parameter is an angle of view.   The system according to any one of claims 1 to 3, wherein the parameter is a focal length.   The system according to claim 1, wherein the parameter is a size of an image sensor.   The system according to claim 1, wherein the parameter is a type of an image sensor.   The system according to claim 1, wherein the parameter is a type of a lens. First acquisition means for acquiring camera information indicating positions and orientations of a plurality of cameras installed so as to image a predetermined position in an imaging space from different directions;
Second acquisition means for acquiring position information of the predetermined position;
Based on the camera information obtained by the first obtaining unit and the position information of the predetermined position obtained by the second obtaining unit, the camera field passes through the predetermined position among the plurality of cameras and captures an image. At least one set of cameras, consisting of a first camera and a second camera, installed at a position symmetrical twice or about the position symmetrical about the vertical axis. The camera configuration is determined between the first camera and the second camera such that parameters that affect at least one of the texture resolution of the object in the captured image and the shooting range of the camera among the parameters defining the characteristics of the camera are different. Determining means;
An apparatus comprising:   The apparatus according to claim 8, wherein the determination unit performs the determination based on a two-dimensional or three-dimensional direction vector indicating a line-of-sight direction of each camera. The determining means comprises:
Of the plurality of cameras, obtain information on the number of cameras having the different parameters,
For each of the plurality of cameras, derive a degree of symmetry between the own camera and another camera,
The apparatus according to claim 8 or 9, wherein a camera having a different parameter among the one set of cameras is determined based on the derived degree of symmetry.   The degree of symmetry takes a maximum value when the own camera and the other camera are in a two-fold symmetrical positional relationship about an axis passing through the predetermined position and perpendicular to an imaging field. An apparatus according to claim 10, wherein The determining means comprises:
Of the plurality of cameras, obtain information on the number of cameras having the different parameters,
For each of the plurality of cameras, derive a degree of symmetry between the own camera and another camera,
For each of the plurality of cameras, a base line passing through the predetermined position extending from the own camera and a proximity degree between the base line passing through the predetermined position extending from another camera facing the predetermined position are derived.
The apparatus according to claim 8, wherein a camera having a different parameter among the one set of cameras is determined based on the derived degree of symmetry and degree of proximity.   13. The apparatus according to claim 12, wherein the degree of proximity becomes higher as the distance between the base line of the own camera and the base line of another adjacent camera is smaller. The determining means comprises:
Based on the degree of symmetry, determine a reference camera from among the plurality of cameras,
Detecting at least two cameras existing near a position separated by a predetermined reference angle from the determined position of the reference camera,
14. The apparatus according to claim 12, wherein a camera having a different parameter is determined based on the degree of proximity from at least two detected cameras. 15. The determining means comprises:
All the cameras that can be the reference camera among the plurality of cameras are sequentially determined as the reference camera,
The apparatus according to claim 14, wherein the camera configuration is determined such that the arrangement of the cameras for which the parameters are different is closest to the most uniform within a settable range of the virtual viewpoint.   The apparatus according to any one of claims 8 to 15, further comprising control means for changing a parameter of the camera determined as a camera having the different parameter by the determination means. The control means further comprises:
Instructing the plurality of cameras to capture images to obtain a plurality of viewpoint images,
Based on the acquired multi-viewpoint image, perform three-dimensional shape data by performing shape estimation of the object,
The apparatus according to claim 16, wherein a virtual viewpoint image is generated based on the plurality of viewpoint images and the three-dimensional shape data. A method for determining a camera configuration in a system that acquires a plurality of viewpoint images for generating a virtual viewpoint image using a plurality of cameras,
The system includes:
Having a plurality of cameras installed to image a predetermined position from different directions,
The plurality of cameras include a first camera and a two-fold symmetrical position or the two-fold symmetrical position of the first camera about an axis passing through the predetermined position and perpendicular to an imaging field. A second camera located closest to the plurality of cameras, and at least one set of cameras,
The method comprises:
Obtaining camera information indicating the position and orientation of the plurality of cameras;
Determining the first camera and the second camera among the plurality of cameras;
Setting parameters that differ between the first camera and the second camera, for parameters that affect at least one of the texture resolution of the object in the captured image and the shooting range of the camera among the parameters that define the characteristics of the camera. ,
A method comprising:   A program for causing a computer to function as the device according to any one of claims 8 to 17.

Images