JP2020005109A - Imaging system, arrangement determining device, arrangement determining method, and program - Google Patents

Abstract

To obtain three-dimensional shape data with an accuracy corresponding to the number of cameras.SOLUTION: An imaging system used to generate three-dimensional shape data on the basis of region information of an object based on a plurality of images acquired by a plurality of pieces of imaging means includes a plurality of pieces of imaging means arranged to image a predetermined position from different directions, and another imaging means different from the at least one imaging means of the plurality of pieces of imaging means is arranged at other than the two-fold symmetric position of at least one of the plurality of pieces of imaging means around the axis perpendicular to the imaging field passing through the predetermined position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging system, an arrangement determining device for determining an arrangement of an imaging unit, an arrangement determining method for determining an arrangement of an imaging unit, and a program.

  2. Description of the Related Art A technique for generating an image (virtual viewpoint image) obtained when an object (object) is viewed from an arbitrary virtual viewpoint from an image (multiple viewpoint images) obtained by capturing an object with a plurality of cameras is known. ing. Patent Document 1 describes the following method. First, three-dimensional shape data is generated based on captured images obtained by a plurality of cameras. Next, based on the texture data based on the plurality of captured images and the three-dimensional shape data, a virtual viewpoint image when viewed from a virtual viewpoint where no camera is arranged is generated. Patent Document 1 exemplifies a method of generating three-dimensional shape data, such as a volume intersection method using a silhouette image in which an object region is extracted from a captured image.

  Incidentally, the quality of the virtual viewpoint image depends on the accuracy of the three-dimensional shape data. When a silhouette image as exemplified in Patent Document 1 is used, it is required to use a plurality of object images to improve the accuracy of the three-dimensional shape data. That is, it is required that images are taken from different directions.

JP 2010020487 A

  However, even if a plurality of cameras are arranged to capture images from different directions, there is a possibility that substantially the same silhouette image will be obtained depending on the arrangement positions of the cameras. For example, when two cameras are arranged at positions that are symmetrical twice with respect to a certain central axis, a silhouette image extracted from images captured by the two cameras has a point where the left and right are reversed. , But substantially the same silhouette image. For this reason, there is a possibility that three-dimensional shape data with an accuracy corresponding to the number of cameras cannot be obtained.

  An object of the present invention is to obtain three-dimensional shape data with an accuracy corresponding to the number of cameras.

  An imaging system according to the present invention is an imaging system used to generate three-dimensional shape data based on region information of an object based on a plurality of images acquired by a plurality of imaging means, and a predetermined position is set in a different direction. A plurality of image pickup means arranged so as to pick up an image from at least one of the plurality of image pickup means, the center being about an axis passing through the predetermined position and perpendicular to an image pickup field. In addition to the rotationally symmetric position, another imaging means different from the at least one imaging means among the plurality of imaging means is arranged.

  According to the present invention, it is possible to obtain three-dimensional shape data with an accuracy corresponding to the number of cameras.

FIG. 1 is a diagram illustrating a configuration example of an imaging system according to a first embodiment. FIG. 2 is a diagram illustrating a configuration example of an imaging system according to a comparative embodiment. FIG. 4 is a diagram for explaining a problem of a comparative embodiment and an effect of the first embodiment. FIG. 5 is a diagram illustrating a determination device and an imaging system according to a second embodiment. FIG. 7 is a block diagram showing a functional configuration of a determining device according to a second embodiment. 9 is a flowchart illustrating a flow of processing for determining an arrangement of a camera in the imaging system according to the second embodiment. FIG. 9 is a view for explaining the concept of a method for determining an arrangement of cameras in an imaging system according to the second embodiment. FIG. 13 is a block diagram showing a functional configuration of a determining device according to a third embodiment. 9 is a flowchart illustrating a flow of processing for determining an arrangement of a camera in an imaging system according to the third embodiment. FIG. 9 is a view for explaining the concept of a method for determining an arrangement of cameras in an imaging system according to a third embodiment. FIG. 1 is a diagram illustrating a configuration example of an image processing system according to a first embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in the present embodiments are necessarily essential to the solution of the present invention. The same components will be described with the same reference numerals.

<First embodiment>
In the present embodiment, an arrangement configuration of a camera (imaging unit) for reducing a decrease in quality of a virtual viewpoint image and accuracy of three-dimensional shape data will be described. The virtual viewpoint image is an image generated by an end user and / or an appointed operator or the like freely operating the position and orientation of a camera (virtual camera) corresponding to the virtual viewpoint, and includes a free viewpoint image and Also called an arbitrary viewpoint image. The virtual viewpoint image may be a moving image or a still image. Further, the virtual viewpoint may be automatically set.

  In the present embodiment, an object refers to a dynamic object (moving body) that has a motion (its absolute position may change) when photographing is performed in the same direction in a time series. The object indicates, for example, a person or a ball in a ball game.

  In the present embodiment, the three-dimensional shape data is data representing a three-dimensional shape of the object, and for example, position information of x, y, z in a three-dimensional space in a world coordinate space that uniquely indicates an imaging space to be imaged. Is represented by a point cloud having. In addition, the three-dimensional shape data is not limited to data represented by a point group, but may be represented by another data format. For example, a surface of a simple convex polygon (called a polygon) such as a triangle or a quadrangle may be used. May be represented by voxels or polygon mesh data.

[Imaging system]
An imaging system 10 according to the present embodiment will be described with reference to FIG. FIGS. 1A and 1B are diagrams illustrating an example of a camera arrangement for imaging an object from different directions according to the present embodiment. FIG. 1A is a three-dimensional bird's-eye view, and FIG. 1B is a two-dimensional view seen from directly above. FIG. In this camera arrangement, images are taken by the six cameras 13 to 18 to obtain a plurality of viewpoint images for generating a virtual viewpoint image. The number of six is an example, and it goes without saying that more cameras are installed when imaging is performed on a large space such as soccer or rugby.

  As shown in FIGS. 1A and 1B, the cameras 13 to 18 are arranged so as to image an arbitrary point of gaze 11 set on an imaging field 12 which is a bottom surface of the imaging space. Here, the gazing point indicates an intersection between the optical axis of the camera and the imaging field 12. The cameras 13 to 18 shown in FIG. 1 are arranged so as to capture the same gazing point 11. The optical axes of the cameras 13 to 18 pass through the gazing point 11. The position of the gazing point 11 is arbitrary, and may be set, for example, in the air in the imaging space. However, the gazing points of the cameras do not have to be at the same predetermined position and may be slightly different, and the gazing points are not limited to the positions shown in FIG. The imaging field 12 indicates a ground in a game such as soccer or rugby.

  In the imaging system 10 of the present embodiment, each of the cameras 13 to 18 is arranged at a position other than a position symmetrical twice with respect to an axis perpendicular to the imaging field 12 including the gazing point 11 of any camera. That is, each of the cameras 13 to 18 is arranged such that another camera is not arranged at a position symmetrical twice with respect to an axis perpendicular to the imaging field 12 including the gazing point 11 of any camera. ing.

  This will be specifically described with reference to FIG. For example, another camera is not disposed at a position symmetrical twice with respect to the axis passing through the gazing point 11 and perpendicular to the imaging field 12. Naturally, no camera different from the camera 13 is arranged on the line segment 19a connecting the camera 13 and the gazing point 11. Note that the camera 16 among the cameras 14 to 16 is arranged at the position closest to the line segment 19a, but is not arranged on the line segment 19a. Similarly, in another camera, another camera is not arranged at a position symmetrical twice with respect to an axis including the point of regard 11 and perpendicular to the imaging field 12. In addition, only the camera is arranged on the line segments 19a to 19f connecting the camera and the gazing point 11, and no other camera is arranged. In the following, a position that is twice symmetrical with respect to an axis passing through the point of gaze and perpendicular to the imaging field may be simply referred to as a symmetrical position. Here, the line segment corresponds to a part of a straight line projected on the imaging field surface of the optical axis.

  On the other hand, with reference to FIG. 2, a problem that occurs when another camera is arranged at a symmetric position of any camera will be described. FIG. 2A is a three-dimensional bird's-eye view of an imaging system 20 in which six cameras are arranged at equal angular intervals around a gazing point 21, and FIG. FIG. 3 shows a two-dimensional plan view. In the case of such a camera arrangement, the camera 26 is arranged at a symmetric position of the camera 23 as shown in FIG. Further, the camera 24 and the camera 27 are arranged symmetrically with respect to each other. The cameras 25 and 28 have the same arrangement. In other words, the camera 23 and the camera 26 are arranged on the line segment 29a. The cameras 24 and 27 are arranged on the line segment 29b. Further, the camera 25 and the camera 28 are arranged on the line segment 29c.

  FIG. 3A is a diagram illustrating a captured image acquired by the imaging system 20 illustrated in FIG. 2 and a silhouette image that is object region information indicating a region of the object 31 in the captured image. Specifically, a captured image 32 acquired by the camera 23 and a silhouette image 33 which is object area information indicating an area of the object 31 acquired from the captured image 32 are shown. Further, a captured image 34 acquired by the camera 26 and a silhouette image 35 which is object area information indicating an area of the object 31 acquired from the captured image 34 are shown. It is assumed that the angles of view of the camera 23 and the camera 26 are set to be the same.

  The number of objects 31 included in silhouette image 33 and silhouette image 35 is the same. Further, since the camera 23 and the camera 26 are arranged at symmetric positions with respect to each other, the silhouette image 33 becomes the same as the silhouette image 35 when the camera 23 is inverted right and left. For this reason, when generating the three-dimensional shape data of the object 31 using the silhouette image of the object 31 by a method such as the visual volume tolerance method, there is a difference between the silhouette image 33 and the silhouette image 35 in the left-right inversion. There is no significant difference in the information obtained. In other words, in generating the three-dimensional shape data, the information obtained from the silhouette images 33 and 35 has the same value. The same applies to silhouette images of objects acquired from images captured by the cameras 24 and 27, respectively. The same applies to silhouette images of objects acquired from images captured by the cameras 25 and 28, respectively. Therefore, it can be said that there are three substantially effective silhouette images obtained by the imaging system 20 of FIG.

  On the other hand, FIG. 3B is a diagram illustrating a captured image acquired by the imaging system 10 of the present embodiment and a silhouette image indicating a region of the object 31 in the captured image. A captured image 36 acquired by the camera 13 and a silhouette image 37 which is object area information indicating an area of the object 31 acquired from the captured image 36 are shown. Further, a captured image 38 obtained by the camera 16 and a silhouette image 39 which is object region information indicating a region of the object 31 obtained from the captured image 38 are shown. It is assumed that the angles of view of the camera 13 and the camera 16 are set to be the same.

  Since the camera 16 is not arranged at a symmetric position with respect to the camera 13, the silhouette image 39 does not match the silhouette image 37 even when flipped horizontally. Therefore, the number of silhouette images obtained by the imaging system 10 is six, which is the same as the number of cameras. Therefore, although the number of cameras does not change from the imaging system 20 of FIG. 2, the number of effective silhouette images can be increased more than that of the imaging system 20. Therefore, when generating the three-dimensional shape data based on the silhouette image indicating the area of the object 31, it is possible to reduce a decrease in the accuracy of the three-dimensional shape data. Further, it is possible to reduce a decrease in image quality of the virtual viewpoint image.

  In the present embodiment, the case where six cameras are installed is described as an example, but the present invention is not limited to this, and an imaging system including at least two or more cameras may be used. The two units indicate the minimum number that constitutes a combination of cameras that are not arranged symmetrically to each other. Further, in the present embodiment, an example has been described in which all cameras included in the imaging system are not arranged at symmetric positions of another camera.However, a configuration in which cameras are arranged at symmetric positions in some cameras is not excluded. Absent.

  Further, in the present embodiment, the configuration example in which another camera is arranged so as not to face the camera with an axis perpendicular to the imaging field including the predetermined position interposed therebetween. However, if the two cameras do not satisfy the relationship of the symmetrical position, the two cameras may include a predetermined position and face the imaging field across a vertical axis. . That is, as long as the distance from each of the two cameras to the axis passing through the gazing point perpendicular to the imaging field is different, the two cameras may face each other.

  Further, in the present embodiment, the case where another camera arranged at a position closest to the target position of a certain camera has the same angle of view as that of the certain camera has been described, but the present invention is not limited to this. For example, even when the focal lengths are the same, when two cameras are arranged in a symmetrical positional relationship, there is no significant difference in the obtained silhouette images, and the above-described problem occurs. In addition, if the camera parameters that affect the silhouette image and determine the characteristics of the camera are the same, the above-described problem occurs. Such camera parameters are, for example, the size of the imaging sensor, the type of lens, and the like.

  When a large difference occurs in the area information (silhouette image) of the object obtained from two cameras having different angles of view, focal lengths, and the like, another camera may be present at a symmetric position with respect to the gazing point. Further, in the present embodiment, at least two cameras having no relation of the symmetric position may have different camera parameters. In addition, camera parameters may be different for all cameras.

  Further, in the present embodiment, a configuration in which another camera does not exist at a symmetric position with respect to the gazing point on the two-dimensional plane has been described, but the present invention is not limited to this. For example, the camera on the first floor seat and the camera on the second floor seat of the soccer stadium have different heights from the imaging field of the camera in the three-dimensional space.

  Further, when there are a plurality of gazing points, that is, when the imaging system is classified into different camera groups arranged so as to capture different gazing points, the camera arrangement described above is used for each camera group. Good. That is, cameras of another camera group having a different gazing point may be arranged at symmetric positions of the cameras in one camera group.

  As described above, by using the multi-viewpoint image captured by the imaging system of the present embodiment, it is possible to reduce a decrease in accuracy of the three-dimensional shape data and reconstruct a high-quality virtual viewpoint image.

[Image processing system]
Hereinafter, an image processing system that generates a virtual viewpoint image having the imaging system of the present embodiment will be described.

  A system in which a plurality of cameras and microphones are installed in facilities such as a stadium or a concert hall to perform photographing and sound collection will be described with reference to a system configuration diagram in FIG. The image processing system 100 includes sensor systems 110a to 110f, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.

  The controller 300 has a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting control, and the like for each block constituting the image processing system 100 via the networks 310a to 310c, 291, 180a, 180b, and 170a to 170e. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (registered trademark, hereinafter abbreviated), or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. The network is not limited to these, and may be another type of network.

  First, an operation of transmitting six sets of images and sounds of the sensor systems 110a to 110f from the sensor system 110f to the image computing server 200 will be described. In the image processing system 100 of the present embodiment, the sensor systems 110a to 110f are connected by a daisy chain.

  In the present embodiment, unless otherwise specified, the sensor system 110 is described without discriminating the six sets of the sensor systems 110a to 110f. Similarly, the devices in each sensor system 110 are not distinguished unless otherwise specified, and are described as a microphone 111, a camera 112, a camera platform 113, an external sensor 114, and a camera adapter 120. Although the number of sensor systems is described as six sets, this is merely an example, and the number is not limited to this. The above-described imaging system corresponds to a system including the cameras 112a to 112f in the image processing system 100. Further, each of the cameras 112a to 112f of the imaging system is arranged at a position other than a symmetric position of a different camera.

  In addition, the plurality of sensor systems 110 may not have the same configuration, and for example, may each be configured by a device of a different model. In the present embodiment, unless otherwise specified, the term image is described as including the concept of a moving image and a still image. That is, the image processing system 100 of the present embodiment can process both still images and moving images. Further, in the present embodiment, an example will be described in which the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual listening point sound, but is not limited thereto. For example, sound may not be included in the virtual viewpoint content. Also, for example, the sound included in the virtual viewpoint content may be sound collected by a microphone closest to the virtual viewpoint. In the present embodiment, for simplicity of description, description of audio is partially omitted, but it is assumed that both image and audio are basically processed.

  Each of the sensor systems 110a to 110f has one camera 112a to 112f. That is, the image processing system 100 includes a plurality of cameras 112 for photographing a subject from a plurality of directions. The plurality of cameras 112 will be described using the same reference numerals, but may differ in performance and model. The plurality of sensor systems 110 are connected by a daisy chain. With this connection form, it is possible to reduce the number of connection cables and to save labor for wiring work in increasing the resolution of a captured image to 4K or 8K and increasing the capacity of image data accompanying a higher frame rate. It is specified here.

  The connection configuration is not limited to this, and a star network configuration in which the sensor systems 110a to 110f are connected to the switching hub 180 to transmit and receive data between the sensor systems 110 via the switching hub 180 may be used.

  FIG. 11 shows a configuration in which all of the sensor systems 110a to 110f are cascaded so as to form a daisy chain. However, the configuration is not limited to this. For example, the plurality of sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy-chained in divided groups. Then, the camera adapter 120 serving as the terminal of the division unit may be connected to the switching hub to input an image to the image computing server 200. Such a configuration is particularly effective in a stadium. For example, a case is considered in which a stadium has a plurality of floors, and the sensor system 110 is provided for each floor. In this case, the input to the image computing server 200 can be performed for each floor or every half of the stadium, so that the installation can be simplified even in a place where wiring for connecting all the sensor systems 110 by one daisy chain is difficult. The system can be made more flexible.

  Further, control of image processing in the image computing server 200 is switched according to whether there is one or more camera adapters 120 connected in a daisy chain and inputting an image to the image computing server 200. . That is, the control is switched according to whether the sensor system 110 is divided into a plurality of groups. When the number of camera adapters 120 for inputting an image is one, an image of the entire circumference of the stadium is generated while performing image transmission by daisy chain connection. Has been taken. That is, if the sensor system 110 is not divided into groups, synchronization can be achieved.

  However, when a plurality of camera adapters 120 perform image input, the delay from when an image is captured to when the image is input to the image computing server 200 may be different for each daisy chain lane (path). That is, when the sensor system 110 is divided into groups, the timing at which the image data of the entire circumference is input to the image computing server 200 may not be synchronized. For this reason, it is specified that the image processing server 200 needs to perform image processing in the subsequent stage while checking the aggregation of image data by the synchronization control that waits until all the image data is collected in the entire periphery and performs synchronization.

  In the present embodiment, the sensor system 110a includes a microphone 111a, a camera 112a, a camera platform 113a, an external sensor 114a, and a camera adapter 120a. Note that the present invention is not limited to this configuration, and it is sufficient that at least one camera adapter 120a and one camera 112a or one microphone 111a are provided. Further, for example, the sensor system 110a may be configured with one camera adapter 120a and a plurality of cameras 112a, or may be configured with one camera 112a and a plurality of camera adapters 120a. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image processing system 100 correspond in N to M (N and M are both integers of 1 or more). Further, the sensor system 110 may include devices other than the microphone 111a, the camera 112a, the camera platform 113a, and the camera adapter 120a. Further, the front-end server 230 may have at least a part of the functions of the camera adapter 120. In the present embodiment, the sensor systems 110b to 110f have the same configuration as that of the sensor system 110a, and will not be described. Note that the configuration is not limited to the same as the sensor system 110a, and each sensor system 110 may have a different configuration.

  The sound collected by the microphone 111a and the image captured by the camera 112a are subjected to various processes in the camera adapter 120a, and then transmitted to the camera adapter 120b of the sensor system 110b through the daisy chain 170a. You. Similarly, the sensor system 110b transmits the collected sound and the captured image to the sensor system 110c together with the image and sound obtained from the sensor system 110a.

  The camera adapter 120 performs processing such as foreground / background separation processing, foreground three-dimensional shape data information generation processing, and dynamic calibration on image data captured by the camera 112 and image data received from another camera adapter 120. . The camera adapter 120 generates a silhouette image of the object based on the foreground / background separation process on the captured image. Further, based on a plurality of silhouette images received from another camera adapter 120, three-dimensional shape data corresponding to the object is generated by a visual volume intersection method or the like. A plurality of three-dimensional shape data is integrated by an image computing server 200 described later. In the camera adapter 120, the image computing server 200 may generate the three-dimensional shape data corresponding to a plurality of objects at once without generating the three-dimensional shape data corresponding to the object.

  By continuing the above-described operation, the images and sounds acquired by the sensor systems 110a to 110f are transmitted from the sensor systems 110f to the switching hub 180 by using 180b, and then transmitted to the image computing server 200.

  In this embodiment, the cameras 112a to 112f and the camera adapters 120a to 120f are separated from each other, but they may be integrated in the same housing. In that case, the microphones 111a to 111f may be built in the integrated camera 112 or may be connected to the outside of the camera 112.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 according to the present embodiment processes data acquired from the sensor system 110f. The image computing server 200 includes a front-end server 230, a database 250 (hereinafter, also referred to as DB), a back-end server 270, and a time server 290.

  The time server 290 has a function of delivering a time and synchronization signal, and delivers the time and synchronization signal to the sensor systems 110a to 110f via the switching hub 180. The camera adapters 120a to 120f that have received the time and the synchronization signal perform Genlock of the cameras 112a to 112f based on the time and the synchronization signal to perform image frame synchronization. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112. Accordingly, since the image processing system 100 can generate the virtual viewpoint image based on the plurality of captured images captured at the same timing, it is possible to suppress a decrease in the quality of the virtual viewpoint image due to a shift in the capturing timing. In this embodiment, the time server 290 manages the time synchronization of the plurality of cameras 112. However, the present invention is not limited to this. Each camera 112 or each camera adapter 120 independently performs the process for time synchronization. You may.

  The front-end server 230 converts the data format by reconstructing the segmented transmission packet from the image and sound acquired from the sensor system 110f, and then stores the packet in the database 250 according to the camera identifier, data type, and frame number. Write.

  Next, the back-end server 270 receives the designation of the viewpoint from the virtual camera operation UI 330, reads the corresponding image and audio data from the database 250 based on the received viewpoint, and performs a rendering process to generate a virtual viewpoint image. I do.

  The configuration of the image computing server 200 is not limited to this. For example, at least two of the front-end server 230, the database 250, and the back-end server 270 may be integrally configured. Further, a plurality of at least one of the front-end server 230, the database 250, and the back-end server 270 may be included. Further, a device other than the above device may be included at an arbitrary position in the image computing server 200. Further, the end user terminal 190 and the virtual camera operation UI 330 may have at least a part of the functions of the image computing server 200.

  The rendered image is transmitted from the back-end server 270 to the end user terminal 190, and the user operating the end user terminal 190 can perform image browsing and audio viewing according to the designation of the viewpoint. That is, the back-end server 270 generates virtual viewpoint content based on the captured images (multiple viewpoint images) captured by the plurality of cameras 112 and the viewpoint information. More specifically, the back-end server 270, for example, based on image data of a predetermined area extracted from images captured by the plurality of cameras 112 by the plurality of camera adapters 120 and a viewpoint specified by a user operation, Generate content. Then, the back-end server 270 provides the generated virtual viewpoint content to the end user terminal 190. Note that, in the present embodiment, the virtual viewpoint content is generated by the image computing server 200, and a description will be given mainly of a case where the virtual viewpoint content is generated by the back-end server 270. However, the present invention is not limited thereto, and the virtual viewpoint content may be generated by a device other than the back-end server 270 included in the image computing server 200, or may be generated by the controller 300 or the end-user terminal 190.

  The virtual viewpoint content according to the present embodiment is a content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on a result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily specified by the user. In addition, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to a viewpoint automatically designated by the device are also included in the virtual viewpoint image.

  In the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described, but audio data may not necessarily be included. Further, the back-end server 270 converts the virtual viewpoint image into, for example, H.264. After compression encoding according to an encoding method such as H.264 or HEVC, the data may be transmitted to the end user terminal 190 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 190 without compression. In particular, the former performing compression encoding assumes a smartphone or a tablet as the end user terminal 190, and the latter assumes a display capable of displaying an uncompressed image. That is, it is specified that the image format can be switched according to the type of the end user terminal 190. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

  As described above, the image processing system 100 has three functional domains: the video collection domain, the data storage domain, and the video generation domain. The image collection domain includes the sensor systems 110 to 110f, the data storage domain includes the database 250, the front-end server 230, and the back-end server 270, and the image generation domain includes the virtual camera operation UI 330 and the end user terminal 190. The present invention is not limited to this configuration. For example, the virtual camera operation UI 330 can directly acquire images from the sensor systems 110a to 110f. However, in the present embodiment, a method of arranging a data storage function in the middle is used instead of a method of directly acquiring images from the sensor systems 110a to 110f. Specifically, the front-end server 230 converts the image data and the audio data generated by the sensor systems 110a to 110f and the meta information of the data into a common schema and a data type of the database 250. Thus, even if the camera 112 of the sensor system 110a to 110f changes to a camera of another model, the changed difference can be absorbed by the front-end server 230 and registered in the database 250. This can reduce the risk that the virtual camera operation UI 330 will not operate properly when the camera 112 is changed to another camera.

  Further, the virtual camera operation UI 330 is configured to access via the back-end server 270 without directly accessing the database 250. The common process related to the image generation process is performed by the back-end server 270, and the difference part of the application related to the operation UI is performed by the virtual camera operation UI 330. As a result, in the development of the virtual camera operation UI 330, it is possible to focus on the development for the UI operation device and the function request of the UI for operating the virtual viewpoint image to be generated. Further, the back-end server 270 can also add or delete a common process related to the image generation process in response to a request from the virtual camera operation UI 330. This makes it possible to flexibly respond to the request of the virtual camera operation UI 330.

  As described above, in the image processing system 100, the back-end server 270 generates the virtual viewpoint image based on the image data based on the images captured by the multiple cameras 112 for capturing the object from multiple directions. Note that the image processing system 100 in the present embodiment is not limited to the physical configuration described above, and may be configured logically.

  The imaging system having the cameras 112a to 112f that operates in the image processing system 100 has been described above. The arrangement positions of the cameras 112a to 112f of this imaging system have the above-described positional relationships. For this reason, it is possible to reduce a decrease in the accuracy of the three-dimensional shape data and a decrease in the image quality of the virtual viewpoint image.

<Embodiment 2>
In the first embodiment, an imaging system for reducing a decrease in accuracy of three-dimensional shape data at virtual viewpoints in various directions and an image processing system including the imaging system have been described. In the present embodiment, an arrangement determining apparatus and method for determining an arrangement of cameras in the imaging system described in the first embodiment will be described.

  This arrangement determining apparatus acquires information on the number of cameras arranged to capture an image of a predetermined position from different directions and position information on a gazing point, and determines the arrangement of the imaging unit based on the information. It is. Specifically, the arrangement determining device arranges another camera at a position other than the two-fold symmetric position (symmetric position) of the camera centered on an axis passing through the gazing point and perpendicular to the imaging field. , The arrangement of the plurality of cameras is determined. With this arrangement determining device, it is possible to input information on the number of cameras and positional information on the point of gaze from outside, and determine an appropriate camera arrangement. Further, the arrangement of the cameras may be automatically performed by the arrangement determining device, or a person may perform the camera arrangement based on the determined arrangement.

  In addition, the arrangement determination device acquires the arrangement information indicating the arrangement positions of the plurality of cameras in addition to the information on the number of vehicles and the position information of the gazing point, and based on the information, determines whether another camera exists at the symmetric position. Alternatively, symmetry information indicating whether or not the information may be obtained. Then, when the symmetry information indicating that the camera exists at the symmetric position of a certain camera is obtained, the arrangement determining device determines to change the camera arrangement so that the camera does not exist at the symmetric position. You may do so.

  The arrangement determining device may be, for example, the controller 300 in the image processing system described in the first embodiment, or may be a device that is not included in the image processing system. Hereinafter, an example in which the processing is performed by another device that is not included in the image processing system will be described. In this case, each camera in the imaging system according to the first embodiment is arranged according to the camera arrangement determined by the arrangement determining device, or the arrangement position is changed when the cameras are already arranged. The actual arrangement or change of the arrangement position of the camera may be performed automatically or by a person. Hereinafter, an example in which the arrangement determination device changes the arrangement of the cameras will be described.

  Hereinafter, a specific configuration of the present embodiment will be described. FIG. 4 is a diagram illustrating an arrangement determining device 402 that determines the arrangement of each camera in an imaging system having a plurality of cameras 401a to 401f. The arrangement determining device 402 is connected to a plurality of cameras of the imaging system via a network. Information on the number of cameras 401a to 401f, initial arrangement position information, posture information indicating the direction of the optical axis, and position information on the point of regard are input to the arrangement determination device 402. The arrangement of the cameras 401a to 401f is changed based on the arrangement information of the cameras 401a to 401f updated by the arrangement determining device 402. The arrangement determining device 402 is connected to the display device 403 and the input devices 404a and 404b via a network. The display device 403 displays, for example, initial arrangement information indicating an initial arrangement position of the camera and updated arrangement information, and displays a change amount and a direction of the camera whose arrangement position is changed. The input devices 404a and 404b receive an input of initial arrangement information indicating an initial arrangement position of the camera from the user or an input of a change amount designated by the user.

  Each of the cameras 401a to 401f captures an image of the object 405a, 405b from a plurality of directions surrounding the object 405 arranged in a region on a schematic plane. The object 405a represents a person (player) on the shooting field 406, which is a competition ground, and the object 405b represents a ball.

  Next, an outline of a method for determining the position of the camera performed in the present embodiment will be described with reference to FIG. First, the arrangement determination device 402 determines information on the number of cameras 702 to 707 included in the imaging system illustrated in FIG. 7A, initial arrangement information indicating an initial arrangement position, and the direction of the optical axis. Obtain the indicated posture information. In addition, the arrangement determination device 402 acquires the position information of the point of regard 701 of each camera in the imaging system. Thereafter, based on the information of the number, the initial arrangement position information, the posture information, and the position information of the point of interest, the determination device determines a camera having a symmetrical positional relationship about an axis perpendicular to the imaging field including the point of interest 701. It is determined whether or not a combination exists. Then, the determining device determines the direction of shifting (change direction) for changing the camera position so that the pair of the camera 702 and the camera 705 determined to have a symmetrical positional relationship does not have a symmetrical positional relationship. The shift amount (change amount) is determined. In addition, the determination device also determines the direction (shift direction) and the shift amount for changing the camera position so that the camera 703 and camera 706 pair and the camera 704 and camera 707 pair do not have a symmetrical positional relationship. (Change amount) is determined.

  For example, when the cameras 704 to 706 are changed to the positions shown in FIG. 7B, as described above, six line segments 709a to 709f connecting each camera and the gazing point 701 are formed. In an imaging system having such a camera arrangement, six effective silhouette images can be acquired. On the other hand, in FIG. 7A, there are only three line segments 708a to 708c connecting each camera and the gazing point 701, and only three effective silhouette images can be obtained. Therefore, the imaging system of FIG. 7B can improve the accuracy of the three-dimensional shape data and the quality of the virtual viewpoint image as compared with the imaging system of FIG. 7A.

  However, the imaging system in FIG. 7B cannot acquire a captured image since no camera is arranged between the cameras 703 and 704 and between the cameras 706 and 707. Therefore, a silhouette image cannot be acquired between the cameras 703 and 704 and between the cameras 706 and 707. Therefore, in the present embodiment, the direction in which the camera is shifted and the shift amount thereof are set such that the line segment connecting the camera and the gazing point after changing the arrangement position is arranged at an approximately equal angle around the gazing point 701. May be determined. For example, as shown in FIG. 7C, the position of the camera 705 is changed from the camera 705 to the position of the camera 707 from the camera position of FIG. 7A, and the line segments 709a to 709f connecting the camera and the gazing point 701 are approximately evenly distributed. The imaging system is constructed as described above. With such an imaging system, it is possible to further reduce the deterioration of the accuracy of the three-dimensional shape data and the quality of the virtual viewpoint image. In the present embodiment, a method for determining the position of the camera will be described by taking, as an example, a case where the quality of the entire object is high, but the present invention is not limited to this. For example, when the direction from the virtual viewpoint is limited, the camera whose position is changed and the amount of change thereof may be determined so that the line segments are equal in the area of the limited direction. .

  Hereinafter, processing performed by the determination device of the present embodiment will be described with reference to FIGS. FIG. 5 is a block diagram illustrating an example of the configuration of the arrangement determination device 402. The arrangement determination device 402 includes a camera information acquisition unit 501, a symmetry information acquisition unit 502, an optical axis adjacent information acquisition unit 503, a change information determination unit 504, a camera information update unit 505, and a camera control unit 506. .

  FIG. 6 is a diagram illustrating an example of a processing flow in the arrangement determination device 402. Hereinafter, the flow of processing performed by each component will be described. The symbol “S” in the following description represents a step.

  The camera information acquisition unit 501 acquires camera information such as the number of cameras, initial arrangement information of the cameras, posture information of the cameras, and position information of the gazing points of the cameras. The symmetry information acquisition unit 502 calculates the degree of symmetry with another camera for each camera based on the camera information, and calculates symmetry information. The degree of symmetry is the degree to which another camera exists at a position symmetrical with respect to the camera twice or about the axis perpendicular to the imaging field including the gazing point, or how close another camera exists. This is an index that indicates whether to do so. The symmetry and the symmetry information will be described later.

  The optical axis adjacent information acquisition unit 503 calculates the optical axis adjacent degree for each camera using the camera information, and acquires the optical axis adjacent information. The optical axis adjacency indicates the proximity between the optical axis of its own camera and the optical axis of another camera, and the angle between the straight lines that project the vector showing the optical axis of each camera onto a two-dimensional plane. It is defined by the indicated index. The optical axis adjacency and the optical axis adjacency information will be described later.

  The change information determination unit 504 determines whether to change the initial arrangement of the camera based on the symmetry information. Further, the change information determination unit 504 determines the amount of change in the position of the camera determined to be changed based on the optical axis adjacent information. The change amount may be input by the user using the input devices 404a and 404b.

  The camera information updating unit 505 updates the camera information output from the camera information obtaining unit 501 or the change information determining unit 504. The camera information updating unit 505 outputs to the camera control unit 506 the content determined by the change information determining unit 504, for example, information on the camera whose position is to be changed, the amount of change, and the camera whose position is not to be changed. It should be noted that the display device 403 in FIG. 4 outputs the content to a display control unit (not shown) in the arrangement determination device 402 or the display device 403 so that the content is displayed, and the display control unit displays the content. It may be displayed. In this case, a person may change the arrangement position of the camera according to the content displayed on the display device 403.

  The camera control unit 506 changes the positions of a plurality of cameras included in the imaging system based on the camera information updated by the camera information updating unit 505.

  In step S <b> 601, the camera information acquisition unit 501 acquires camera information including information on the number of cameras included in the imaging system, initial arrangement information, posture information, and position information of the point of regard. The camera information defines three-dimensional coordinate axes in which the bottom surface (imaging field) of the imaging space is the x-axis and the y-axis and the height is the z-axis, and the position of each camera is a position pi (pxi, pyi, pzi) in the coordinate space. ), Information representing the attitude of each camera as a vector vi [vxi, vyi, vzi]. Here, i is the identification number of each of the installed cameras. Note that the contents of the camera information are not limited to those described above, but may be information on the coordinate position of each camera and the coordinate position of the gazing point. The method of expressing the position information and the posture information of the camera is not limited to this, and may be any expression that can specify the position and the posture of the camera. Further, in the present embodiment, the initial position of the camera may be automatically determined so that the angle between the optical axes around the gazing point becomes equal, or may be determined by the user via the input devices 404a and 404b. The position of the camera may be acquired as the initial position of the camera. The camera information acquisition unit 501 outputs the acquired initial arrangement information and posture information of the camera as camera information to the symmetry information acquisition unit 502, the optical axis adjacent information acquisition unit 503, and the camera information update unit 505.

  In S602, the change information determination unit 504 initializes the change information. The change information is information indicating, as a first value, a value indicating whether or not the position of the camera is changed in each camera, and a change amount indicating a change amount as a second value when the position is changed.

  In step S603, the camera information update unit 505 determines a reference camera from which a plurality of cameras included in the imaging system is to be determined whether to update the camera information. Usually, the reference camera is determined in ascending order of the number i for identifying the camera, but is not limited to this. For example, a camera randomly selected from a plurality of cameras may be determined as a reference camera, or may be determined by another method.

  In step S604, the symmetry information acquisition unit 502 calculates symmetry information for each camera using the camera information acquired from the camera information acquisition unit 501 or the camera information update unit 505. As the camera information, the camera information acquired from the camera information updating unit 505 is used preferentially. Hereinafter, a method of calculating the symmetry information will be specifically described. That is, for example, the following steps 1) to 3) are performed.

  1) A vector indicating the optical axis direction of each camera is projected onto a two-dimensional plane, and a direction vector (hereinafter referred to as an optical axis projection vector) vi [vpi, vqi] obtained by normalization is calculated for each camera. I do. Here, i is a number for identifying each camera, and i = 1-6.

  2) Calculate the degree of symmetry with another camera. Specifically, an inner product vo · vj of an optical axis projection vector vo [vpo, vqo] of the reference camera and an optical axis projection vector vj [vpj, vqj] of a camera other than the reference camera is calculated. Here, o represents the identification number of the reference camera, and j represents the identification number of a camera other than the reference camera. For example, when i = 1, o = 1 and j = 2-6.

  3) Symmetry information is acquired using the degree of symmetry calculated in 2) above. Specifically, the value of the inner product vo · voj with another camera is negative, and the maximum value among the values obtained by inverting the sign of the inner product vo · vj is acquired as the symmetric information of the reference camera. If the symmetry information is 1, it indicates that another camera exists at the symmetry position of the reference camera. If the symmetry information is not 1, it indicates that no other camera exists at the symmetric position of the reference camera. That is, the symmetric information is information indicating whether or not another camera exists at the symmetric position of the camera.

  The symmetry information acquisition unit 502 outputs the acquired symmetry information to the change information determination unit 504. In the present embodiment, the calculation of the symmetry information is performed using the two-dimensional plane and the inner product. However, the present invention is not limited to this, and it is determined whether or not the reference camera and another camera are in a symmetrical positional relationship, and the degree thereof. Other methods that can be used may be used.

  In step S605, the change information determination unit 504 determines, using the symmetry information acquired in step S604, whether or not there is a camera that is symmetrically positioned with respect to the reference camera with respect to the gazing point. In the present embodiment, when the symmetric information of the reference camera is 1, as described above, it is determined that another camera exists at the symmetric position (YES in S605), and the process proceeds to S606. However, the present invention is not limited to this. For example, if the symmetry information is 0.8 or more, it may be determined that another camera exists at the symmetric position. In other words, it is not determined that another camera exists at a symmetric position only when another camera exists at a completely symmetric position with respect to the reference camera, but also when another camera exists at an approximate symmetric position. It may be allowed to determine that it exists at the position. Further, in the present embodiment, whether or not another camera exists at the symmetric position of the reference camera is determined using the symmetric information calculated by the two-dimensional vector. However, the present invention is not limited to this. The determination may be performed based on the three-dimensional positional relations included. In such a case, in the case of a two-dimensional case, even if the camera is at a symmetric position, if the height position is different, it may be determined that the camera does not exist at the symmetric position.

  In S606, the change information determination unit 504 updates the first value of the change information to 1 when determining YES in S605. The fact that the first value is 1 indicates that the position of the camera is to be changed.

  If NO in S605, that is, if the change information determining unit 504 determines that another camera does not exist at the symmetric position of the reference camera, the process proceeds to S612, and determines the first value and the second value of the change information. Both are updated to 0. When the first value is 0, it indicates that the position of the camera is not changed. Also, the fact that the second value is 0 indicates that the change amount is 0. In this case, the process proceeds to S610.

  Next, in step S607, the optical axis adjacent information acquisition unit 503 acquires the optical axis adjacency using the camera information acquired from the camera information acquisition unit 501 or the camera information update unit 505. As the camera information, the camera information acquired from the camera information updating unit 505 is used preferentially. Hereinafter, a method for obtaining the optical axis adjacency will be specifically described. That is, for example, the following steps 1) to 3) are performed.

  1) The optical axis projection vector vi [vpi, vqi] of each camera is calculated for each camera. Here, i is a number for identifying each camera, and i = 1-6. Note that this procedure is the same as the method 1) of calculating the symmetry information, and this information may be acquired from the symmetry information acquisition unit 502.

  2) The inner product vo · vj and the outer product vo × vj of the optical axis projection vector vo [vpo, vqo] of the reference camera and the optical axis projection vectors vj [vpj, vqj] of the cameras other than the reference camera are calculated. Here, o represents the identification number of the reference camera, and j represents the identification number of a camera other than the reference camera. For example, when i = 1, o = 1 and j = 2-6. The inner product vo · vj may be obtained from the symmetry information obtaining unit 502.

  3) The value obtained by dividing the outer product vo × vj by the inner product vo · vj from the value of the inner product vo · voj and the outer product vo × vj with the other camera calculated in the above 2) is calculated, and the calculated value is used as the light of the camera. Axis adjacency. The optical axis adjacency is an index indicating an angle between straight lines obtained by projecting a vector indicating the optical axis of the camera onto a two-dimensional plane. In the present embodiment, the calculation of the optical axis adjacency is performed using the two-dimensional plane, the inner product, and the outer product. However, the present invention is not limited to this, and another method may be used. The optical axis adjacent information acquisition unit 503 outputs information summarizing the optical axis adjacent degrees of the reference camera and another camera to the change information determining unit 504 as optical axis adjacent information.

  In step S608, the change information determining unit 504 determines the amount of change when changing the position of the reference camera from the optical axis adjacent information acquired in S606, and updates the second value of the change information. This will be specifically described below. That is, for example, the following steps 1) to 3) (or 1) to 3 ')) are performed.

  1) A value cp at which the optical axis adjacency is positive and minimum and a value cm at which the optical axis adjacency is negative and minimum are extracted from the optical axis adjacency information.

  2) It is determined from the optical axis adjacency information whether a camera having an optical axis adjacency of 0 exists other than the reference camera.

  3) If it is determined from the optical axis adjacency information that the camera having the optical axis adjacency of 0 is other than the reference camera, the absolute value of the minimum value cp having the positive optical axis adjacency and the negative optical axis adjacency are negative. The absolute value of the minimum value cm is compared with the absolute value, and a value c that minimizes the absolute value is extracted. Then, using the detected value c, an intermediate angle θ = arctan (c / 2) between the optical axis of the reference camera and the optical axis closest to the reference camera is calculated.

  3 ') When it is determined that there is no camera other than the reference camera having the optical axis adjacency of 0, the intermediate angle θ is calculated by (arctan (cp) -arctan (cm)) / 2 + arctan (cp).

  The change information determination unit 504 updates the second value of the change information to the value of the intermediate angle θ, using the intermediate angle θ calculated as described above as the change amount of the reference camera. Then, the change information determining unit 504 outputs the updated change information to the camera information updating unit 505. Therefore, the larger the angle between the straight lines obtained by projecting the vectors indicating the optical axes of the two cameras on the two-dimensional plane, the smaller the change amount becomes.

  In the present embodiment, the change amount is calculated using the arc tangent. However, the present invention is not limited to this, and another method capable of calculating the angle or the movement amount between the optical axis of the reference camera and the optical axis of another camera is used. May be used. Alternatively, a change amount arbitrarily specified by the user may be used, and the change amount may be used as the second value of the change information. Although the angle is used as the second value of the change information, the present invention is not limited to this, and another method such as a distance and a vector in a three-dimensional space may be used.

  In step S609, the camera information update unit 505 updates the camera information of the reference camera acquired from the camera information acquisition unit 501 using the change information acquired from the change information determination unit 504. When the first value in the change information is 1, the position and orientation information of the reference camera is updated. In the present embodiment, the position information pi (pxi, pyi, pzi) of the reference camera is updated to a position obtained by rotating the reference camera with respect to the gazing point by the magnitude of the angle indicated by the second value of the change information. I do. In addition, the posture vector vi of the reference camera is updated using the updated position information pi and the gazing point. In the present embodiment, the camera information is updated so that the reference camera is moved by rotating the reference camera according to the second value of the change information. However, the present invention is not limited to this. For example, when there is a limit on the position where the camera can be placed, the camera information may be updated so that the reference camera is moved from the position simply moved to the closest position within the limit. When the first value in the change information is 0, the position and orientation information of the reference camera in the camera information is not updated.

  In step S610, the camera information update unit 505 determines whether all cameras included in the imaging system have been determined as reference cameras and whether or not to update camera information has been determined. If it is determined that the determination has been made for all cameras (YES in S610), camera information update section 505 outputs the updated camera information to camera control section 506. In addition, as needed, the updated camera information is output to a display control unit (not shown) that controls the display of the display device 403.

  On the other hand, when the result of the determination in step S610 is NO, in step S613, the camera information updating unit 505 changes the reference camera. After that, the process returns to S604. In step S <b> 613, in the present embodiment, a camera that has not been selected as a reference camera until then is selected as a new reference camera. When the camera information has already been updated as the reference camera, the degree of symmetry and the optical axis adjacency when another camera is used as the reference camera are calculated using the updated camera information.

  In step S611, the camera control unit 506 changes the positions of a plurality of cameras included in the imaging system according to the updated camera information acquired from the camera information updating unit 505.

  The above is the description of the determining device and the processing method in the present embodiment. According to the configuration described above, it is possible to construct an imaging system that generates a high-quality virtual viewpoint image regardless of the number of cameras included in the imaging system and the direction of the virtual viewpoint.

<Embodiment 3>
In the second embodiment, the method of determining the position of the camera has been described so that the line segment connecting the camera and the point of interest is distributed at substantially equal intervals all around the imaging space. In the present embodiment, a description will be given of an arrangement determining apparatus and an arrangement determining method for determining the arrangement of cameras such that the cameras themselves are also present at substantially equal intervals around the entire circumference, in addition to the line segment connecting the camera and the gazing point. Specifically, the position of the camera is determined based on the camera adjacency indicating the adjacency between the cameras in addition to the optical axis adjacency described in the second embodiment. The outline of the method for determining the camera arrangement performed in the present embodiment and its significance will be described below with reference to FIG.

  An example of an imaging system 1000 including four cameras 1003 to 1006 as shown in FIGS. 10A and 10B will be described. FIG. 10A shows a case where the cameras are arranged in the imaging system 1000 such that only the line segments 1009a to 1009d connecting each camera and the gazing point 1001 have substantially the same angle over the entire circumference. In the imaging system 1000 of FIG. 10A, since optical axes exist in various directions around the entire circumference and silhouette images in various directions are obtained, three-dimensional shape data can be generated with high accuracy. However, the angle 1007 between the optical axes of the cameras 1003 and 1006 is large, and no camera is arranged between the cameras 1003 and 1006. Therefore, in the object to be imaged, there is a possibility that an area that cannot be imaged by both the camera 1003 and the camera 1006 may occur. As a result, even if the three-dimensional shape data is highly accurate, the texture information used for rendering does not exist or decreases. Since the texture information affects the quality of the virtual viewpoint image, when a virtual viewpoint is set between the cameras 1003 and 1006, the virtual viewpoint image has low image quality.

  On the other hand, FIG. 10B shows that, in the imaging system 1000, the cameras are not arranged in a symmetrical positional relationship with respect to the gazing point 1001, and the cameras themselves are arranged at substantially equal angles around the entire circumference. I have. In FIG. 10B, the degree of dispersion of the optical axis of each camera over the entire circumference is inferior to that of FIG. 10A, but the angle 1008 between the camera 1003 and the camera 1006 in FIG. As compared with the configuration in FIG. 10A, the area that cannot be imaged by both the camera 1003 and the camera 1006 is smaller, and texture information of an object used when generating a virtual viewpoint image can be reduced. As a result, it is possible to improve the quality of the virtual viewpoint image. Therefore, in the present embodiment, as shown in FIG. 10B, an imaging system in which cameras are arranged in consideration of not only the degree of dispersion of the optical axis but also the degree of dispersion of cameras is configured.

  In the example described above, the case where the number of cameras is four has been described. However, the present invention is not limited to this, and the present invention can be applied to a case where the number of cameras is arbitrary. Further, in the above-described example, the case where all the cameras of the imaging system do not satisfy the relationship of the symmetrical position has been described. However, the present invention is not limited to this. May be acceptable.

  Hereinafter, a determination device for determining the arrangement of cameras and a processing method thereof according to the present embodiment will be described with reference to FIGS. 8 and 9. FIG. 8 is a block diagram illustrating an example of a configuration of the arrangement determination device 803. The arrangement determining device 803 includes a camera information obtaining unit 501, a symmetry information obtaining unit 502, an optical axis adjacent information obtaining unit 503, a change information determining unit 802, a camera information updating unit 505, and a camera control unit 506. And a camera adjacent information acquisition unit 801. The camera adjacency information acquisition unit 801 acquires camera adjacency indicating a short distance between cameras based on the camera information, and collects the camera adjacency information for each camera to obtain camera adjacency information. The camera adjacency is an index indicating the distance between the imaging units. The change information determination unit 802 differs from the second embodiment in that the change information determination unit 802 determines the amount of change in the camera arrangement position based on the camera adjacency information in addition to the optical axis adjacency.

  FIG. 9 is a diagram illustrating an example of a processing flow in the arrangement determination device 803. Hereinafter, the flow of processing performed by each component will be described. Configurations and processes similar to those in the second embodiment are denoted by the same reference numerals as in the second embodiment, and description thereof is omitted. More specifically, S612 and S613 are the same as those of the second embodiment up to S607 and after S609.

  In step S901, the camera neighboring information acquisition unit 801 acquires camera neighboring information indicating the distance between the reference camera and another camera using the camera information acquired from the camera information acquiring unit 501 or the camera information updating unit 505. As the camera information, the camera information acquired from the camera information updating unit 505 is used preferentially. Hereinafter, a method for acquiring the camera adjacent information will be specifically described.

  In the present embodiment, two values are held for each reference camera as camera adjacent information. The first value indicates the camera proximity indicating the closeness of the distance to the reference camera, and the second value indicates the direction to the reference camera. In the present embodiment, one of two types of positive (1) and negative (0) is held as the second value. However, the method of expressing the camera adjacent information is not limited to the above, and another method indicating how close another camera is to the reference camera and the direction thereof may be used. Hereinafter, a method of calculating the camera adjacent information will be specifically described. That is, for example, the following steps 1) to 3) are performed.

  1) The optical axis projection vector vi [vpi, vqi] of each camera is calculated for each camera. Here, i is a number for identifying each camera, and i = 1-6. Note that this procedure is the same as the method 1) of calculating the symmetry information, and this information may be acquired from the symmetry information acquisition unit 502.

  2) Calculate the camera adjacency. Specifically, an inner product vo · vj and an outer product vo × vj of an optical axis projection vector vo [vpo, vqo] of the reference camera and an optical axis projection vector vj [vpj, vqj] of a camera other than the reference camera are calculated. I do. Here, o represents the identification number of the reference camera, and j represents the identification number of a camera other than the reference camera. For example, when i = 1, o = 1 and j = 2-6. The inner product vo · vj may be obtained from the symmetry information obtaining unit 502. The outer product vo × vj may be acquired from the optical axis adjacent information acquisition unit 503.

  3) The value of the inner product vo · voj calculated in the above 2) is set as the first value of the camera adjacent information. Further, the second value of the camera adjacent information is determined according to the sign of the value of the cross product vo × vj. When the value of the cross product vo × vj is positive, the second value of the camera adjacency information is 1, and when the value of the cross product vo × vj is negative, the second value of the camera adjacency information is 0.

  In the present embodiment, the camera adjacency is calculated using the two-dimensional plane, the inner product, and the outer product. However, the present invention is not limited to this, and another method may be used. The camera adjacency information acquisition unit 801 outputs information obtained by summarizing the acquired camera adjacency for all cameras to the change information determination unit 802.

  In step S <b> 902, the change information determination unit 802 determines the position of the reference camera based on the optical axis adjacent information acquired from the optical axis adjacent information acquisition unit 503 and the camera adjacent information collected by all the cameras acquired from the camera adjacent information acquisition unit 801. The amount of change when changing is determined. Then, the change information determining unit 802 updates the second value of the change information. This will be specifically described below. That is, for example, the following steps 1) to 6) are performed.

  1) A value cp at which the optical axis adjacency is positive and minimum and a value cm at which the optical axis adjacency is negative and minimum are extracted from the optical axis adjacency information.

  2) Similarly, a value dp at which the second value of the camera adjacency information is 1 and the first value is minimum, and a value at which the second value is 0 and the first value is minimum among the camera adjacency information Get dm.

  3) From the obtained dp and dm, calculate θd = (arctan (dp) −arctan (dm)) / 2 indicating the middle of the angle between the reference camera and the adjacent camera.

  4) It is determined from the optical axis adjacency information whether a camera having an optical axis adjacency of 0 exists other than the reference camera.

  5) If it is determined from the optical axis adjacency information that the camera having the optical axis adjacency of 0 is other than the reference camera, the absolute value of the minimum value cp having the positive optical axis adjacency and the negative optical axis adjacency are negative. The absolute value of the minimum value cm is compared with the absolute value, and a value c that minimizes the absolute value is extracted. Then, using the detected value c, an intermediate angle θc = arctan (c / 2) between the optical axis of the reference camera and the optical axis closest to the reference camera is calculated. On the other hand, when it is determined that there is no camera other than the reference camera having the optical axis adjacency of 0, the intermediate angle θc is calculated by (arctan (cp) −arctan (cm)) / 2.

  6) The intermediate angle θ between θc and θd is calculated by (θc−θd) / 2 + θd.

  The change information determination unit 802 updates the second value of the change information with the calculated intermediate angle θ with the change amount of the reference camera. The change information determining unit 802 outputs the updated change information to the camera information updating unit 505. In the present embodiment, the change amount is calculated using the arc tangent. However, the present invention is not limited to this, and another method capable of calculating the angle or the movement amount between the optical axis of the reference camera and the adjacent optical axis is used. You may. Alternatively, a change amount arbitrarily specified by the user may be used, and the change amount may be used as the second value of the change information. Although the angle is used as the second value of the change information, the present invention is not limited to this, and another method such as a distance and a vector in a three-dimensional space may be used.

  In addition, the larger the distance between the cameras, the smaller the change amount of the camera arrangement position.

  In the present embodiment, the change amount is determined by using the camera adjacent information and the optical axis adjacent information at an equivalent ratio. However, the present invention is not limited to this, and the intermediate angle θ is set to (α × θc−β × θd) / 2 + θd. As described above, it may be determined by the weighted sum. In this case, as α becomes larger than β, the degree of equality of the optical axis takes priority, and conversely, as β becomes larger than α, the degree of equality between cameras takes precedence.

  The above is the description of the determining device and the processing method in the present embodiment. According to the configuration described above, it is possible to construct an imaging system that generates a high-quality virtual viewpoint image regardless of the number of cameras included in the imaging system and the direction of the virtual viewpoint.

<Other embodiments>
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Reference Signs List 10 imaging system 12 imaging field 13, 14, 15, 16, 17, 18 camera

Claims (20)

An imaging system used to generate three-dimensional shape data based on region information of an object based on a plurality of images acquired by a plurality of imaging means,
Having a plurality of imaging means arranged to image a predetermined position from different directions,
Centering on an axis passing through the predetermined position and perpendicular to the imaging field, other than the two-fold symmetric position of at least one of the plurality of imaging means, the other of the plurality of imaging means An imaging system, wherein another imaging means different from at least one imaging means is arranged.   Any of the plurality of imaging units may be located at a symmetrical position of any one of the plurality of imaging units around an axis passing through the predetermined position and perpendicular to the imaging field. The imaging system according to claim 1, wherein the plurality of imaging units are arranged so that the imaging unit is not arranged.   The at least one of the plurality of imaging units is arranged so as not to face at least one of the plurality of imaging units across an axis perpendicular to the imaging field through the predetermined position. The imaging system according to claim 1, wherein another imaging unit different from the imaging unit is arranged.   The imaging system according to any one of claims 1 to 3, wherein each of the plurality of imaging units has the same parameter for determining characteristics of the imaging unit.   The imaging system according to any one of claims 1 to 3, wherein at least two of the plurality of imaging units have different parameters for determining characteristics of the imaging unit.   The imaging system according to claim 4, wherein the parameter is at least one of an angle of view, a focal length, a type of a lens, and a size of an imaging sensor. First acquisition means for acquiring information on the number of a plurality of imaging means arranged to image a predetermined position from different directions;
Second acquisition means for acquiring position information of the predetermined position;
Based on the information on the number acquired by the first acquisition unit and the position information on the predetermined position acquired by the second acquisition unit, an axis that passes through the predetermined position and is perpendicular to an imaging field is In addition to the two-fold symmetric position of at least one of the plurality of imaging units as a center, another imaging unit different from the at least one imaging unit of the plurality of imaging units is arranged. Determining means for determining an arrangement of the plurality of image pickup means as described above. A third acquisition unit that acquires arrangement information indicating an arrangement position of the plurality of imaging units;
The determining unit is configured to obtain the number information obtained by the first obtaining unit, the position information of the predetermined position obtained by the second obtaining unit, and the position information of the predetermined position obtained by the third obtaining unit. On the basis of the arrangement information, centering on an axis passing through the predetermined position and perpendicular to the imaging field, in addition to the two-fold symmetric position of at least one imaging unit of the plurality of imaging units, The arrangement determining apparatus according to claim 7, wherein the arrangement of the plurality of imaging units is determined such that another one of the imaging units different from the at least one imaging unit is arranged. . The information of the number obtained by the first obtaining unit, the position information of the predetermined position obtained by the second obtaining unit, and the arrangement information obtained by the third obtaining unit. Based on each of the plurality of imaging units, obtains symmetry information indicating whether or not the imaging unit exists at a two-fold symmetric position about an axis passing through the predetermined position and perpendicular to an imaging field. 4, further comprising:
When the fourth acquisition unit acquires symmetry information indicating that the imaging unit is arranged at a two-fold symmetric position about an axis passing through the predetermined position and perpendicular to the imaging field. The deciding means includes at least one imaging means of the plurality of imaging means such that the imaging means does not exist at a two-fold symmetric position centered on an axis passing through the predetermined position and perpendicular to the imaging field. 9. The arrangement determining apparatus according to claim 8, wherein the arrangement is determined to be changed.   The third acquisition unit projects a vector indicating an optical axis of each imaging unit onto a two-dimensional plane, and acquires the symmetry information based on a direction vector obtained by normalization. An arrangement determining device according to claim 9.   The arrangement determining apparatus according to claim 9, wherein the determining unit further determines a change amount when changing an arrangement position of the at least one imaging unit.   The determining unit determines a change amount for changing an arrangement position of the at least one imaging unit based on a first index indicating an angle between straight lines obtained by projecting a vector indicating an optical axis of each imaging unit on a two-dimensional plane. The arrangement determining apparatus according to claim 11, wherein:   13. The arrangement determining apparatus according to claim 12, wherein the change amount decreases as the angle between the straight lines increases.   14. The apparatus according to claim 12, wherein the determination unit further determines a change amount for changing an arrangement position of the at least one imaging unit based on a second index indicating a distance between the imaging units. Arrangement determining device. A first acquisition step of acquiring information on the number of a plurality of imaging units arranged to image a predetermined position from different directions;
A second acquisition step of acquiring position information of the predetermined position;
Based on the information on the number obtained in the first obtaining step and the position information on the predetermined position obtained by the second obtaining unit, an axis passing through the predetermined position and perpendicular to an imaging field In addition to the two-fold symmetric position of at least one of the plurality of imaging units, another imaging unit different from the at least one imaging unit of the plurality of imaging units is disposed. And determining the arrangement of the plurality of imaging means. A third acquisition step of acquiring arrangement information indicating an arrangement position of the plurality of imaging units,
In the determining step, the information on the number obtained in the first obtaining step, the position information on the predetermined position obtained in the second obtaining step, and the position information obtained in the third obtaining step On the basis of the arrangement information, centering on an axis passing through the predetermined position and perpendicular to the imaging field, in addition to the two-fold symmetric position of at least one imaging unit of the plurality of imaging units, 16. The arrangement determining method according to claim 15, wherein the arrangement of the plurality of imaging units is determined such that another one of the imaging units different from the at least one imaging unit is arranged. . The information of the number obtained by the first obtaining unit, the position information of the predetermined position obtained by the second obtaining unit, and the arrangement information obtained by the third obtaining unit. Based on each of the plurality of imaging units, obtains symmetry information indicating whether or not the imaging unit exists at a two-fold symmetric position about an axis passing through the predetermined position and perpendicular to an imaging field. 4 acquisition process;
In the case where the information indicating that the imaging unit is arranged at a symmetric position twice around the axis perpendicular to the imaging field through the predetermined position is obtained by the fourth obtaining step, The determining step is such that at least one imaging unit of the plurality of imaging units does not include an imaging unit at a two-fold symmetric position about an axis passing through the predetermined position and perpendicular to the imaging field. 17. The arrangement determining method according to claim 16, wherein it is determined to change the arrangement.   The third acquisition step is characterized in that a vector indicating an optical axis of each imaging unit is projected on a two-dimensional plane, and the symmetry information is acquired based on a direction vector obtained by normalization. A method according to claim 17.   19. The arrangement determining method according to claim 17, wherein the determining step further determines a change amount when changing an arrangement position of the at least one imaging unit.   A program for causing a computer to function as each unit of the arrangement determination device according to any one of claims 7 to 14.

Images