WO2021182153A1 - Information processing device, information processing system, information processing method, and program - Google Patents

Abstract

An image generating device (40a)(information processing device) comprises: a camera image input unit (60)(first image acquiring unit) which acquires an image of a subject (18) captured by a plurality of fixed cameras (14) installed to face the subject (18) and having a fixed installed position and installed direction; and a camera image input unit (61)(second image acquiring unit) which acquires an image of the subject captured by a moving camera (30) installed in the vicinity of the fixed cameras and having a variable installed position and installed direction. A moving camera position setting unit (65a)(setting unit) sets the installed position and installed direction of the moving camera so that a 3D shape of the subject obtained on the basis of the image of the subject acquired by the camera image input unit (60) and the image of the subject acquired by the camera image input unit (61) has a predetermined accuracy. Then, a 3D shape computing unit (66)(3D shape acquiring unit) acquires the 3D shape of the subject on the basis of the images of the subject captured by the fixed cameras and the moving camera.

Description

Information processing equipment, information processing systems, information processing methods and programs

The present disclosure relates to information processing devices, information processing systems, information processing methods and programs, and in particular, information processing devices and information processing systems capable of performing high-quality modeling and rendering with a small number of cameras regardless of the subject. , Information processing methods and programs.

Recently, a volumetric technique for reconstructing a three-dimensional shape of a subject by using a plurality of fixed cameras arranged around the subject has been proposed (for example, Patent Document 1). We want to reduce the number of cameras as much as possible in order to shorten the processing time required to reconstruct the 3D shape. However, with a small number of cameras, deterioration of modeling accuracy and rendering quality becomes a problem.

Therefore, conventionally, by integrating stereo matching technology with Visual Hull to generate a 3D model of the subject and performing texture mapping using the View Dependent method on the generated 3D model, the quality is as high as possible with a small number of cameras. Methods for modeling and rendering have been proposed.

Further, a method of installing a mobile camera different from the fixed camera to eliminate the blind spot in the observation area has been proposed (for example, Patent Document 2).

International Publication No. 2017/082076 Japanese Unexamined Patent Publication No. 2015-204512

However, for example, in the invention of Patent Document 1, since the position of the camera arranged around the subject is fixed, it is difficult to maintain the quality of the volumetric image when the subject changes. There was a problem. Further, the invention of Patent Document 2 has a main purpose of eliminating blind spots, and does not aim at improving the modeling accuracy and rendering accuracy of a subject.

This disclosure proposes an information processing device, an information processing system, an information processing method and a program capable of performing high-quality modeling and rendering with a small number of cameras regardless of the subject.

In order to solve the above problems, the information processing apparatus of one form according to the present disclosure is provided by a plurality of fixed cameras whose installation position and installation direction are fixed, which are installed around the subject with the direction of the subject facing. The first imaging unit that images the subject, the second imaging unit that images the subject by a moving camera installed near the fixed camera and whose installation position and direction are variable, and the first imaging unit. The moving camera is installed so that the 3D shape of the subject obtained based on the image of the subject captured by the imaging unit and the image of the subject captured by the second imaging unit has a predetermined accuracy. The second imaging unit is provided by a setting unit that sets the position and installation direction, an image of the subject captured by the first imaging unit, and the moving camera installed in the installation position and installation direction set by the setting unit. An information processing device including an acquisition unit that acquires a 3D shape of the subject based on an image of the subject captured by the camera.

It is a figure which shows the outline of the flow which generates the 3D model of a subject. It is an external view which shows an example of the structure of the image generation system of 1st Embodiment. It is a hardware block diagram which shows an example of the hardware composition of the image generation apparatus of 1st Embodiment. It is a hardware block diagram which shows an example of the hardware configuration of a drone. It is a functional block diagram which shows an example of the functional structure of the image generation apparatus of 1st Embodiment. It is a figure explaining the outline of the process of acquiring a 3D shape by a stereo pair of a fixed camera and a moving camera. It is a figure explaining the setting method of the installation position of a moving camera. It is a flowchart which shows an example of the flow of the process performed by the image generation apparatus of 1st Embodiment. It is a flowchart which shows an example of the flow of the installation position and direction calculation processing of the moving camera performed by the image generation apparatus of 1st Embodiment. It is a functional block diagram which shows an example of the functional structure of the image generation apparatus of 2nd Embodiment. It is a figure explaining the method of acquiring the high-definition texture information by a moving camera. It is a flowchart which shows an example of the flow of the process performed by the image generation apparatus of 2nd Embodiment. It is a flowchart which shows an example of the installation position and direction of the moving camera, and the flow of the angle-of-view calculation process performed by the image generation apparatus of the 2nd Embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, the present disclosure will be described according to the order of items shown below.
1. 1. First Embodiment 1-1. Explanation of prerequisites-3D model generation 1-2. Outline configuration of image generator 1-3. Hardware configuration of image generator 1-4. Drone hardware configuration 1-5. Functional configuration of the image generator of the first embodiment 1-6. Explanation of how to determine the installation position of the mobile camera 1-7. Flow of processing performed by the image generator of the first embodiment 1-8. Effect of the first embodiment 2. Second Embodiment 2-1. Functional configuration of the image generator of the second embodiment 2-2. Explanation of how to determine the installation position of the mobile camera 2-3. Flow of processing performed by the image generator of the second embodiment 2-4. Effect of the second embodiment

(1. First Embodiment)
[1-1. Explanation of prerequisites-3D model generation]
FIG. 1 is a diagram showing an outline of a flow in which an image generator generates a 3D model of a subject.

As shown in FIG. 1, the 3D model 18M of the subject 18 generates a 3D model 18M having 3D information of the subject 18 by imaging the subject 18 by a plurality of fixed cameras 14 (14a, 14b, 14c) and 3D modeling. It is generated through processing.

Specifically, as shown in FIG. 1, the plurality of fixed cameras 14 are arranged outside the subject 18 so as to surround the subject 18 existing in the real world, facing the direction of the subject 18. FIG. 1 shows an example in which the number of fixed cameras is three, and the fixed cameras 14a, 14b, and 14c are arranged around the subject 18. In FIG. 1, a person is the subject 18. Further, the number of fixed cameras 14 is not limited to three, and a larger number of fixed cameras may be provided.

From different viewpoints, 3D modeling is performed using a plurality of viewpoint images taken synchronously by three fixed cameras 14a, 14b, 14c, and in units of video frames of the three fixed cameras 14a, 14b, 14c. A 3D model 18M of the subject 18 is generated.

The 3D model 18M is a model having 3D information of the subject 18. The 3D model 18M has shape information representing the surface shape of the subject 18 in the form of mesh data called, for example, a polygon mesh, which is expressed by a connection between vertices (Vertex) and vertices. Further, the 3D model 18M has texture information representing the surface state of the subject 18 corresponding to each polygon mesh. The format of the information contained in the 3D model 18M is not limited to these, and may be other formats of information.

When reconstructing the 3D model 18M, so-called texture mapping is performed by pasting a texture representing the color, pattern, or texture of the mesh according to the mesh position. For texture mapping, in order to improve the reality of the 3D model 18M, it is desirable to paste a texture according to the viewpoint position (View Dependent: hereinafter referred to as VD). As a result, when the 3D model 18M is imaged from an arbitrary virtual viewpoint, the texture changes according to the viewpoint position, so that a higher quality virtual image can be obtained. However, since the amount of calculation increases, a texture that does not depend on the line-of-sight position (View Independent: hereinafter referred to as VI) may be attached to the 3D model 18M.

The read content data including the 3D model 18M is transmitted to a mobile terminal which is a playback device and played back. By rendering the 3D model 18M and reproducing the content data including the 3D model 18M, an image having a 3D shape is displayed on the viewing device of the user (viewer).

Figure 1 shows an example of a mobile terminal such as a smartphone or tablet terminal used as a viewing device. That is, an image including the 3D model 18M is displayed on the display 120 of the mobile terminal.

[1-2. Outline configuration of image generator]
Next, the schematic configuration of the image generation system 10a of the first embodiment will be described with reference to FIG. FIG. 2 is an external view showing an example of the configuration of the image generation system of the first embodiment.

The image generation system 10a generates a 3D model 18M of the subject 18. The image generation system 10a includes an image generation device 40a and a drone 20. The image generation system 10a is an example of the information processing system in the present disclosure. Further, the image generation device 40a is an example of the information processing device in the present disclosure.

The image generation device 40a generates a 3D model 18M of the subject 18 based on the image of the subject 18 captured by the fixed camera 14 and the moving camera 30.

A plurality of fixed cameras 14 are arranged so as to surround the subject 18 and capture an image of the subject 18.

The drone 20 is equipped with a mobile camera 30 and moves in the position and direction instructed by the image generation device 40a. The drone 20 is an example of the movement control device in the present disclosure. Further, if the moving camera 30 can be moved in the position and direction instructed by the image generator 40a, a moving camera whose direction can be controlled is installed in the probe of a huge three-dimensional digitizer instead of the drone 20. You may use it.

The mobile camera 30 captures an image of the subject 18 from a position and direction instructed by the image generation device 40a. Further, the mobile camera 30 transmits the captured image to the image generation device 40a.

The image generation system 10a may include a plurality of drones 20 (moving cameras 30), but in the present embodiment, the image generation system 10a includes one drone 20 (moving camera 30). do.

[1-3. Image generator hardware configuration]
Next, the hardware configuration of the image generation device 40a will be described with reference to FIG. FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the image generator of the first embodiment.

The image generation device 40a includes a CPU (Central Processing Unit) 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 43, a storage unit 44, a camera controller 45, a communication controller 46, and input / output. The controller 47 has a configuration in which it is connected by an internal bus 48.

The CPU 41 controls the overall operation of the image generation device 40a by expanding and executing the control program P1 stored in the storage unit 44 and various data files stored in the ROM 42 on the RAM 43. That is, the image generation device 40a has a general computer configuration operated by the control program P1. The control program P1 may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. Further, the image generation device 40a may execute a series of processes by hardware. The control program P1 executed by the CPU 41 may be a program in which processing is performed in chronological order according to the order described in the present disclosure, or at necessary timings such as in parallel or when calls are made. It may be a program that is processed by.

The storage unit 44 is configured by, for example, a flash memory, and stores the control program P1, the camera parameter file C1, the camera image file K, and the 3D model storage file M.

The control program P1 is a program executed by the CPU 41.

The camera parameter file C1 is a file that stores the internal parameters and external parameters of the fixed camera 14.

The camera image file K is a file that temporarily stores the images captured by the fixed camera 14 and the moving camera 30.

The 3D model storage file M is a file that stores the 3D model 18M of the subject 18 generated by the image generation device 40a.

The camera controller 45 is connected to the fixed camera 14 and controls the imaging operation of the fixed camera 14 based on a command from the CPU 41. Further, the camera controller 45 temporarily stores the image captured by the fixed camera 14 in the camera image file K.

The communication controller 46 wirelessly communicates with the drone 20 based on a command from the CPU 41, and instructs the drone 20 of the movement target position and direction. Further, the communication controller 46 acquires an image captured by the mobile camera 30 from the drone 20.

The input / output controller 47 is connected to the display 50 and displays various information output by the image generation device 40a on the display 50. Further, the input / output controller 47 is connected to the touch panel 51 and the keyboard 52, and receives various operation instructions for the image generation device 40a.

[1-4. Drone hardware configuration]
Next, the hardware configuration of the drone 20 will be described with reference to FIG. FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the drone.

The drone 20 communicates with the CPU 21, the ROM 22, the RAM 23, the storage unit 24, the camera controller 25, the camera control motor driver 26, the propeller control motor driver 27, and the GPS (Global Positioning System) receiver 28. The controller 29 has a configuration in which it is connected by an internal bus 37.

The CPU 21 controls the overall operation of the drone 20 by expanding and executing the control program P2 stored in the storage unit 24 and various data files stored in the ROM 22 on the RAM 23. That is, the drone 20 has a general computer configuration operated by the control program P2.

The storage unit 24 is configured by, for example, a flash memory, and stores the control program P2 and the camera parameter file C2.

The control program P2 is a program executed by the CPU 21.

The camera parameter file C2 is a file that stores the internal parameters and external parameters of the moving camera 30. The camera parameter file C2 may be stored in the storage unit 44 of the image generation device 40a.

The camera controller 25 is connected to the mobile camera 30 and controls the imaging operation of the mobile camera 30 based on a command from the CPU 21. Further, the camera controller 25 transmits the image captured by the mobile camera 30 to the image generation device 40a via the communication controller 29 based on the command from the CPU 21.

The camera control motor driver 26 operates the pan control motor 31, the tilt control motor 32, and the roll control motor 33 by a specified amount, respectively, based on a command from the CPU 21. The pan control motor 31, the tilt control motor 32, and the roll control motor 33 move the directions of the moving camera 30 in the pan direction, the tilt direction, and the roll direction, respectively. The camera control motor driver 26 may further include a zoom control motor (not shown) that changes the angle of view of the moving camera 30.

The propeller control motor driver 27 controls the rotation of the propeller drive motor 34 based on a command from the CPU 21. The rotation of the propeller drive motor 34 is transmitted to the propeller 35 to control the moving state of the drone 20.

The GPS receiver 28 identifies the current position of the drone 20 based on the radio waves from the GPS satellites received by the GPS antenna 36. If positioning is possible, a means other than the GPS receiver 28 may be used. For example, Wi-Fi positioning (Wi-Fi is a registered trademark) that measures its own position based on signal strength from a plurality of Wi-Fi (registered trademark) routers may be used.

The communication controller 29 wirelessly communicates with the image generation device 40a based on the command from the CPU 21 based on the command from the CPU 21, and transmits the image captured by the mobile camera 30 to the image generation device 40a. Further, the communication controller 29 receives an operation instruction for the mobile camera 30 and the drone 20 from the image generation device 40a.

[1-5. Functional configuration of the image generator of the first embodiment]
Next, the functional configuration of the image generation device 40a of the first embodiment will be described with reference to FIG. FIG. 5 is a functional block diagram showing an example of the functional configuration of the image generator of the first embodiment.

The CPU 41 of the image generation device 40a expands the control program P1 into the RAM 43 and executes it to perform calibration processing on the camera image input unit 60, the camera image input unit 61, the camera image storage unit 62, and the camera image storage unit 62 shown in FIG. The unit 63, the 3D shape extraction unit 64, the moving camera position setting unit 65a, the 3D shape calculation unit 66, the camera work designation unit 67, the movement command unit 68, and the position measurement unit 69 are realized as functional units. ..

The camera image input unit 60 acquires an image of the subject 18 captured by the fixed camera 14. The camera image input unit 60 is an example of the first image acquisition unit in the present disclosure.

The camera image input unit 61 acquires an image of the subject 18 captured by the moving camera 30 installed in the vicinity of the fixed camera and having a variable installation position and installation direction. The camera image input unit 61 is an example of the second image acquisition unit in the present disclosure.

The camera image storage unit 62 temporarily stores the image acquired by the camera image input unit 60 and the image acquired by the camera image input unit 61 in the camera image file K.

The calibration processing unit 63 includes internal parameters related to the optical parameters of the fixed camera 14 and the moving camera 30, relative positional relationships of the plurality of fixed cameras 14, and external parameters related to the relative positional relationship between the fixed camera 14 and the moving camera 30. Is calculated. Then, the calibration processing unit 63 corrects the distortion of the images captured by the fixed camera 14 and the moving camera 30. The internal parameters can be acquired in advance when the focal length is not changed by optical zoom or the like. It is also possible to acquire external parameters between fixed cameras in advance. On the other hand, when these parameters cannot be acquired in advance, the calibration processing unit 63 calculates the internal parameters and the external parameters in real time using the feature points in the captured image. Further, the external parameters related to the relative positional relationship between the fixed camera 14 and the moving camera 30 can be acquired by using the position and direction of the moving camera 30 measured by the position measuring unit 69. The internal parameters and external parameters of the fixed camera 14 acquired in advance are stored in the camera parameter file C1. Further, the internal parameters and external parameters of the moving camera 30 are stored in the camera parameter file C2. The image distortion correction performed by the calibration processing unit 63 is performed according to a known image processing method.

The 3D shape extraction unit 64 extracts a rough 3D shape of the subject 18 based on a plurality of images of the subject 18 captured by the plurality of fixed cameras 14, respectively. The 3D shape extraction unit 64 specifically extracts the approximate 3D shape of the subject 18 by using the Visual Hull method.

In the moving camera position setting unit 65a, the 3D shape of the subject 18 obtained based on the image of the subject 18 acquired by the camera image input unit 60 and the image of the subject 18 acquired by the camera image input unit 61 has a predetermined accuracy. The installation position and installation direction of the moving camera 30 are set so as to have. More specifically, the moving camera position setting unit 65a moves based on the 3D information of the subject 18 based on the image of the subject 18 acquired by the camera image input unit 60 and the installation position and installation direction of the fixed camera 14. The installation position and installation direction of the camera 30 are set. The moving camera position setting unit 65a is an example of the setting unit in the present disclosure.

The 3D shape calculation unit 66 is captured by the image of the subject 18 acquired by the camera image input unit 60 and the moving camera 30 installed at the installation position and the installation direction set by the moving camera position setting unit 65a, and the camera image input unit 61. Acquires the 3D shape of the subject 18 based on the image of the subject 18 acquired by. The 3D shape calculation unit 66 is an example of the 3D shape acquisition unit in the present disclosure.

The camera work designation unit 67 designates the camera work that represents the trajectory of the virtual viewpoint for observing the subject 18. The camera work designation unit 67 is an example of the designation unit in the present disclosure.

The movement command unit 68 moves the moving camera 30 to a designated position and a designated direction by moving the drone 20. The movement command unit 68 is an example of the command unit in the present disclosure.

The position measurement unit 69 measures the position and direction of the moving camera 30. The position measuring unit 69 is an example of the measuring unit in the present disclosure.

[1-6. Explanation of how to determine the installation position of the mobile camera]
Next, a method of determining the installation position of the moving camera 30 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram illustrating an outline of a process of acquiring a 3D shape by a stereo pair of a fixed camera and a moving camera. FIG. 7 is a diagram illustrating a method of setting the installation position of the mobile camera.

As shown in FIG. 6, it is assumed that the 3D shape 130 of the subject 18 is extracted based on the images captured by the plurality of fixed cameras 14 (14a, 14b, ...). The 3D shape 130 is extracted by the 3D shape extraction unit 64 (see FIG. 5) using, for example, Visual Hull. Visual Hull is to extract the image (silhouette) of the subject 18 captured by the fixed camera 14 and obtain the intersection region of the silhouette extracted from the images captured by a plurality of fixed cameras 14 whose relative relationships are known. This is a method for extracting the 3D shape of the subject 18. According to this method, a 3D shape can be extracted even in a region having a poor texture. However, in principle, it is not possible to extract the dent of the object, so it is desirable to use Visual Hull in combination with another distance measuring means instead of using it alone. In this embodiment, Visual Hull and stereo matching are used together.

In stereo matching, the same subject 18 is imaged by a plurality of cameras arranged close to each other, and the position of the same point of the subject 18 in the images captured by each camera is analyzed (stereo matching). This is a method of calculating the positional deviation (parallax) between images by (searching for corresponding points by). The parallax is a value corresponding to the distance from the camera to the subject 18. That is, the larger the parallax, the shorter the distance from the camera to the subject 18. The smaller the parallax, the farther the distance from the camera to the subject 18. Therefore, by performing the same analysis on a plurality of points of the image of the subject 18 captured in the image, the distance from the camera to the subject 18, that is, the 3D shape of the subject 18 can be obtained. According to such stereo matching, if a characteristic texture exists on the surface of the subject 18, the 3D shape can be calculated even if the surface of the subject 18 has a dent.

In order to obtain high distance measurement accuracy by stereo matching, it is important to appropriately set the distance (baseline length) between multiple cameras and maintain the parallelism of the optical axes of each camera.

If the baseline length is long, the resolution of the measured distance will increase, but if the baseline length is too long, the deformation of the image of the subject 18 captured by each camera will increase, making stereo matching difficult. Alternatively, the area where stereo matching can be obtained becomes narrower. In addition, if the baseline length is made too long, the corresponding points disappear in one camera, so-called occlusion occurs, and stereo matching cannot be obtained. For example, when the fixed camera 14a and the fixed camera 14b in the left figure of FIG. 6 are paired and stereo matching is performed, since there is an area where only one fixed camera 14a can be observed, the 3D shape of the subject 18 is covered over a wide range. Cannot be obtained.

Therefore, in order to perform stereo matching, the shortest baseline length that can secure a predetermined distance measurement accuracy over the entire depth of the subject 18 is set, and the optical axes of the cameras are installed so as to be parallel to each other. It is desirable to do. For example, it is desirable to perform stereo matching by pairing the fixed camera 14a and the moving camera 30 in the right figure of FIG.

As described above, the image generation system 10a of the present embodiment combines the fixed camera 14 and the moving camera 30 to generate a camera pair for stereo matching. In the right figure of FIG. 6, the baseline length W, which is the distance between the fixed camera 14a and the moving camera 30, is the farthest point from the fixed camera 14a on the 3D shape 130 of the subject 18 measured by Visual Hull. At the position, the distance measurement accuracy obtained by stereo matching is determined to be the shortest length that is equal to or greater than a predetermined distance measurement accuracy threshold value.

Further, it is desirable that the optical axis A1 of the fixed camera 14a and the optical axis A2 of the moving camera 30 are set in parallel. This is to facilitate the calculation of parallax when performing stereo matching. Further, it is desirable that the angle of view of the fixed camera 14 (14a, 14b, ...) And the angle of view of the moving camera 30 are set to be substantially equal. When the angle of view of each fixed camera 14 is different, the moving camera 30 makes its own angle of view substantially equal to the angle of view of the fixed camera 14 according to the angle of view of the fixed cameras 14 forming a stereo pair. Adjust to. For example, the angle of view of the lens of the moving camera 30 may be adjusted by the zoom control motor (not shown in FIG. 4).

In the image generation system 10a of the present embodiment, the mobile camera 30 is mounted on the drone 20, and the drone 20 has a positioning function by the GPS receiver 28. Therefore, the image generation system 10a can install the mobile camera 30 at the position and direction instructed by the image generation device 40a with high accuracy.

Further, the internal parameters and external parameters (installation position and installation direction) of each fixed camera 14 are measured in advance and stored in the camera parameter file C1. Therefore, the image generation device 40a controls the position and direction of the drone 20 so that the moving camera 30 is in a state where the optical axis is parallel to the arbitrary fixed camera 14 and the baseline length W is high. It shall be possible to install at.

Then, the moving camera position setting unit 65a of the image generation device 40a moves the optical axis A1 of the fixed camera 14a and the optical axis of the moving camera 30 from the state in which the moving camera 30 is closest to each fixed camera 14. While maintaining parallelism with A2, the baseline length W is increased by a predetermined value, and the measurement accuracy of the distance from the fixed camera 14 to the subject 18 at the maximum depth position (farthest point) of the 3D shape 130 of the subject 18 is measured each time. calculate. That is, the moving camera position setting unit 65a sets the installation position and the installation direction of the moving camera 30 while moving the moving camera 30 in the direction away from the fixed camera 14. Then, the moving camera position setting unit 65a first sets a position where the distance measurement accuracy is equal to or higher than a preset threshold value at the installation position of the moving camera 30. The distance measurement accuracy p is a value corresponding to the distance D from the fixed camera 14 to the maximum depth position of the 3D shape 130 of the subject 18 and the parallax ε of the farthest point between the fixed camera 14 and the moving camera 30. Become. Then, in the moving camera position setting unit 65a, for example, when the value of ε / D becomes equal to or higher than a preset threshold value, the entire subject 18 is equal to or higher than the predetermined value by stereo matching between the fixed camera 14 and the moving camera 30. It is judged that the distance can be measured with the accuracy of.

Further, the moving camera position setting unit 65a sets the direction in which the moving camera 30 is moved according to the subject 18 when the moving camera 30 is moved to change the baseline length W.

For example, as shown in the left figure of FIG. 7, when the subject 18 has a pattern (texture) along the vertical direction (Z-axis direction), the pattern extending vertically is important when searching for a corresponding point in stereo matching. It becomes a feature. It is desirable that the two cameras (fixed camera 14 and moving camera 30) that perform stereo matching are arranged apart from each other in a direction orthogonal to the direction of the pattern, that is, in the Y-axis direction of FIG.

On the other hand, as shown in the right figure of FIG. 7, when the subject 18 has a pattern (texture) along the horizontal direction (Y-axis direction), this horizontally extending pattern is important when searching for a corresponding point in stereo matching. It becomes a feature. Then, it is desirable that the two cameras (fixed camera 14 and moving camera 30) that perform stereo matching are arranged apart from each other in a direction orthogonal to the direction of the pattern, that is, in the Z-axis direction of FIG.

[1-7. Flow of processing performed by the image generator of the first embodiment]
Next, the flow of processing performed by the image generator 40a will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of processing performed by the image generator of the first embodiment.

The camera image input unit 60 acquires an image of the subject 18 captured by the fixed cameras 14 (14a, 14b, ...) At the same time (step S10). Although not shown in FIG. 8, the acquired image is temporarily stored in the camera image file K by the action of the camera image storage unit 62. Further, the calibration processing unit 63 refers to the camera parameter file C1 and corrects the distortion of the image acquired from the fixed camera 14.

The 3D shape extraction unit 64 extracts an approximate 3D shape 130 of the subject 18 by, for example, a Visual Hull method (step S11).

The moving camera position setting unit 65a generates a stereo pair between adjacent fixed cameras 14 (step S12). In step S12, the fixed cameras 14 generate a stereo pair with the fixed camera 14 if the 3D shape of the subject 18 can be acquired by the stereo pair generated between the adjacent fixed cameras 14. This is because it is not necessary to make a stereo pair with the moving camera 30.

The moving camera position setting unit 65a determines the distance measurement accuracy (calculated from the baseline length W, the focal length, and the image resolution) of the stereo pair generated in step S12 at the maximum depth position of the 3D shape 130 of the subject 18. It is determined whether the accuracy set in advance is satisfied (step S13). When it is determined that the accuracy set in advance is satisfied (step S13: Yes), the process proceeds to step S15. On the other hand, if it is not determined that the accuracy set in advance is satisfied (step S13: No), the process proceeds to step S14.

The moving camera position setting unit 65a sets the installation position and direction of the moving camera 30 (step S14). The detailed flow of processing performed in step S14 will be described later (see FIG. 9).

The 3D shape calculation unit 66 extracts the 3D shape 130 of the subject 18 by, for example, a method of visual Hull and stereo matching (step S15).

The moving camera position setting unit 65a determines whether or not the processing after step S12 has been performed on all the fixed cameras 14 (step S16). When it is determined that the processes after step S12 have been performed on all the fixed cameras 14 (step S16: Yes), the image generation device 40a ends the process of FIG. On the other hand, if it is not determined that the processing after step S12 has been performed (step S16: No), the process returns to step S12.

In FIG. 8, a stereo camera pair with the moving camera 30 is formed for all the fixed cameras 14 to generate a 3D model of the subject 18, but the observation direction of the subject 18 can be specified in advance. In some cases, the fixed camera 14 used to generate the 3D shape may be limited. That is, when the camera work designation unit 67 specifies the observation direction of the subject 18, only the fixed camera 14 close to the observation direction may be selected to form a stereo camera pair with the moving camera 30.

For example, when the camera work designation unit 67 specifies the movement locus of the virtual viewpoint when observing the subject 18, the moving camera position setting unit 65a is the fixed camera 14 located in the vicinity of the designated movement locus. Based on the installation position and the installation direction, the installation position and the installation direction of the mobile camera 30 are set so that the fixed camera 14 and the mobile camera 30 form a stereo pair. Then, the 3D shape of the subject 18 is acquired by the stereo pair.

Next, with reference to FIG. 9, the flow of processing performed by the moving camera position setting unit 65a in step S14 will be described. FIG. 9 is a flowchart showing an example of the flow of the installation position and direction calculation process of the mobile camera performed by the image generation device of the first embodiment.

The moving camera position setting unit 65a installs the moving camera 30 at a predetermined relative position with respect to the fixed camera 14 (step S20).

The moving camera position setting unit 65a sets the direction of the optical axis A2 of the moving camera 30 in parallel with the optical axis A1 of the fixed camera 14 (step S21).

The camera image input unit 61 acquires an image of the subject 18 captured by the moving camera 30 (step S22). Although not shown in FIG. 9, the image acquired from the moving camera 30 is temporarily stored in the camera image file K by the action of the camera image storage unit 62. Further, the calibration processing unit 63 refers to the camera parameter file C2 and corrects the distortion of the image acquired from the moving camera 30.

The moving camera position setting unit 65a determines whether a predetermined distance accuracy can be obtained with respect to the position of the maximum depth of the subject 18 (step S23). When it is determined that the predetermined distance accuracy can be obtained (step S23: Yes), the process returns to step S15 in FIG. On the other hand, if it is not determined that the predetermined distance accuracy can be obtained (step S23: No), the process proceeds to step S24.

If No is determined in step S23, the moving camera position setting unit 65a determines whether the moving camera 30 has been moved within a predetermined range (step S24). When it is determined that the moving camera 30 has been moved within a predetermined range (step S24: Yes), the process returns to step S15 in FIG. On the other hand, if it is not determined that the moving camera 30 has moved within a predetermined range (step S24: No), the process proceeds to step S25.

If No is determined in step S24, the moving camera position setting unit 65a moves the moving camera 30 in a direction away from the fixed camera 14 by a predetermined amount (step S25). After that, the process returns to step S21 and the above-described processing is repeated.

[1-8. Effect of the first embodiment]
As described above, according to the image generation device 40a (information processing device) of the first embodiment, the camera image input unit 60 (first image acquisition unit) is placed around the subject 18 in the direction of the subject 18. The fixed camera 14 (second image acquisition unit) acquires an image of the subject 18 captured by a plurality of fixed cameras 14 having a fixed installation position and installation direction. The image of the subject 18 captured by the moving camera 30 installed in the vicinity of the above and whose installation position and direction are variable is acquired. Then, the moving camera position setting unit 65a (setting unit) obtains a 3D image of the subject 18 based on the image of the subject 18 acquired by the camera image input unit 60 and the image of the subject 18 acquired by the camera image input unit 61. The installation position and installation direction of the moving camera 30 are set so that the shape has a predetermined accuracy. Then, the 3D shape calculation unit 66 (3D shape acquisition unit) has the image of the subject 18 acquired by the camera image input unit 60, and the moving camera 30 installed in the installation position and installation direction set by the moving camera position setting unit 65a. The 3D shape of the subject 18 is acquired based on the image of the subject 18 acquired by the camera image input unit 61 after taking an image.

This makes it possible to model a high-quality subject 18 with a small number of cameras regardless of the subject 18.

Further, according to the image generation device 40a (information processing device) of the first embodiment, the moving camera position setting unit 65a (setting unit) is the subject 18 acquired by the camera image input unit 60 (first image acquisition unit). The installation position and installation direction of the moving camera 30 are set based on the 3D information of the subject 18 based on the image of the above and the installation position and installation direction of the fixed camera 14.

As a result, the installation position and the installation direction of the mobile camera 30 can be set based on the approximate 3D shape of the subject 18 obtained by the plurality of fixed cameras 14, so that the installation position and the installation direction of the mobile camera 30 can be set. The setting can be done easily.

Further, according to the image generation device 40a (information processing device) of the first embodiment, in the moving camera position setting unit 65a (setting unit), the optical axis A1 of the fixed camera 14 and the optical axis A2 of the moving camera 30 are parallel to each other. In this state, the distance (base line length W) between the fixed camera 14 and the moving camera 30 is set so that the distance accuracy at the maximum depth position of the subject 18 as seen from the fixed camera 14 is higher than a predetermined value. ..

This makes it possible to set the installation position and installation direction of the moving camera 30 for modeling the 3D shape of the subject 18 with a simple procedure.

Further, according to the image generation device 40a (information processing device) of the first embodiment, the mobile camera position setting unit 65a (setting unit) is a fixed camera when setting the installation position and installation direction of the mobile camera 30. The installation position and installation direction of the moving camera 30 are set while moving the moving camera 30 in a direction away from the fixed camera 14 in a direction corresponding to the surface state of the subject 18.

This makes it possible to efficiently set the installation position and installation direction of the mobile camera.

Further, according to the image generation device 40a (information processing device) of the first embodiment, the camera work designation unit 67 (designation unit) designates the camera work representing the trajectory of the virtual viewpoint for observing the subject 18 and moves. The camera position setting unit 65a (setting unit) sets the installation position and installation direction of the moving camera 30 based on the installation position and installation direction of the fixed camera 14 in the vicinity of the camera work designated by the camera work designation unit 67. do.

As a result, it is sufficient to generate a stereo pair with the moving camera 30 and acquire the 3D shape of the subject 18 only for the fixed camera 14 in the vicinity of the designated camera work, which is necessary for modeling the subject 18. The amount of processing can be reduced.

Further, according to the image generation device 40a (information processing device) of the first embodiment, the moving camera 30 moves to the position designated by the movement command unit 68 (command unit) and the designated direction. Further, the position measuring unit 69 measures the position and direction of the moving camera 30.

As a result, when setting the installation position and the installation direction of the mobile camera 30, the self-position of the mobile camera 30 measured by the position measurement unit 69 is fed back to the movement command unit 68, so that the installation position of the mobile camera 30 and the installation position of the mobile camera 30 are set. The installation direction can be set with high accuracy.

Further, according to the image generation system 10a (information processing system) of the first embodiment, the drone 20 (movement control device) is based on the image of the subject 18 by the fixed camera 14 and the image of the subject 18 by the moving camera 30. The installation position and installation direction of the moving camera 30 are moved so that the 3D shape of the subject 18 thus obtained has a predetermined accuracy.

This makes it possible to model a high-quality subject 18 with a small number of cameras regardless of the subject 18.

(2. Second embodiment)
[2-1. Functional configuration of the image generator of the second embodiment]
The image generation system 10b of the second embodiment is a system that renders the subject 18 viewed from an arbitrary rendering viewpoint (virtual viewpoint) and reproduces a volumetric image. The image generation system 10b includes an image generation device 40b and a drone 20. The drone 20 includes a mobile camera 30. The camera control motor driver 26 (see FIG. 4) further includes a zoom control motor (not shown) that changes the angle of view of the moving camera 30 in addition to the configuration of the first embodiment. The image generation system 10b is an example of the information processing system in the present disclosure, and the image generation device 40b is an example of the information processing device in the present disclosure. Further, the image generation system 10b also has a function of modeling the subject 18 to generate a 3D model, that is, a function of the image generation system 10a described in the first embodiment.

First, the functional configuration of the image generation device 40b will be described with reference to FIG. FIG. 10 is a functional block diagram showing an example of the functional configuration of the image generator of the second embodiment.

The image generation device 40b includes a moving camera position setting unit 65b instead of the moving camera position setting unit 65a with respect to the image generation device 40a described in the first embodiment. Further, the image generation device 40b has a functional configuration in which the texture mapping unit 70 is added to the image generation device 40a. Since the other functional configurations are the same as those of the image generator 40a, the description other than the above-mentioned parts will be omitted. Further, in the following description, the same functional parts as those of the image generation device 40a will be described using the same reference numerals.

The moving camera position setting unit 65b detects the region of interest of the subject 18. The region of interest is an region that is considered to be of interest to the user observing the subject 18. For example, when the subject 18 is a person, the region of interest is a facial region or the like. Further, the moving camera position setting unit 65b sets the installation position of the moving camera 30 within the area corresponding to the area of interest. The region corresponding to the region of interest is, for example, a position in the region of interest facing the normal direction N of the subject 18, that is, an region in which the region of interest of the subject 18 can be observed with higher definition. Then, the moving camera position setting unit 65b sets the rendering viewpoint (virtual viewpoint) at a position facing the normal direction N of the subject 18. In this way, the image generation device 40b acquires high-definition texture information of the region of interest of the subject 18 by installing the moving camera 30 at the position of the set virtual viewpoint. At this time, the moving camera position setting unit 65b controls the angle of view of the moving camera 30 so that the mapping surface E of the subject 18 observed by the moving camera 30 is captured at the full angle of view of the moving camera 30. In addition, the angle of view of the moving camera 30 is adjusted by the zoom control motor described above. This is to acquire as high-definition texture information as possible by enlarging the mapping surface E as much as possible and taking an image.

The texture mapping unit 70 maps the texture information acquired by the moving camera 30 to the 3D shape of the subject 18 calculated by the 3D shape calculation unit 66 according to the procedure described in the first embodiment. The texture mapping unit 70 is an example of the drawing unit in the present disclosure.

[2-2. Explanation of how to determine the installation position of the mobile camera]
Next, with reference to FIG. 11, a method in which the image generation device 40b determines the installation position of the mobile camera 30 in the second embodiment will be described. FIG. 11 is a diagram illustrating a method of acquiring high-definition texture information by a moving camera.

As shown in the left figure of FIG. 11, it is assumed that the 3D shape 130 of the subject 18 is extracted based on the images captured by the plurality of fixed cameras 14 (14a, 14b, ...). Then, it is assumed that a volumetric image in which the subject 18 is observed from an arbitrary rendering viewpoint Q (virtual viewpoint) is generated. In this case, the fixed camera 14 needs to acquire high-definition texture information of the mapping surface E facing the rendering viewpoint Q of the subject 18.

Therefore, when the position of the rendering viewpoint Q and the installation position of the fixed camera 14 are close to each other, the image captured by the fixed camera 14 may be acquired.

However, in the state shown on the left of FIG. 11, since the fixed camera 14 near the rendering viewpoint Q does not exist, the image generation device 40b installs the moving camera 30 at a position close to the rendering viewpoint Q. The texture information of the mapping surface E of the subject 18 is acquired.

At this time, in the image generation device 40b, as shown in the right figure of FIG. 11, the moving camera 30 so that the direction of the optical axis A2 of the moving camera 30 faces the normal direction N of the mapping surface E of the subject 18. Set the installation position and installation direction of. Since the mapping surface E is generally not a flat surface but a curved surface, the image generation device 40b calculates the average normal direction of the region facing the imaging range of the moving camera 30 as the normal direction N. do. Whether the direction of the optical axis A2 of the moving camera 30 faces the normal direction N of the mapping surface E of the subject 18 depends on, for example, the line segment connecting the rendering viewpoint Q and the mapping surface E of the subject 18 and the movement. It may be determined whether the angle formed by the line segment connecting the camera 30 and the mapping surface E is equal to or less than a predetermined threshold value. When the fixed camera 14 is installed near the rendering viewpoint Q, a line segment connecting the rendering viewpoint Q and the mapping surface E of the subject 18 and a line segment connecting the fixed camera 14 and the mapping surface E Based on whether the angle formed by the fixed camera 14 is equal to or less than a predetermined threshold value, it is determined whether the optical axis of the fixed camera 14 faces the normal direction N of the mapping surface E, and the image acquired by the fixed camera 14 is used. , The texture information of the subject 18 may be acquired.

[2-3. Flow of processing performed by the image generator of the second embodiment]
Next, the flow of processing performed by the image generator 40b will be described with reference to FIG. FIG. 12 is a flowchart showing an example of the flow of processing performed by the image generator of the second embodiment.

The camera image input unit 60 acquires an image of the subject 18 captured by the fixed cameras 14 (14a, 14b, ...) At the same time (step S30). Although not shown in FIG. 12, the acquired image is temporarily stored in the camera image file K by the action of the camera image storage unit 62. Further, the calibration processing unit 63 refers to the camera parameter file C1 and corrects the distortion of the image acquired from the fixed camera 14.

The 3D shape extraction unit 64 extracts an approximate 3D shape 130 of the subject 18 by, for example, a Visual Hull method (step S31).

The moving camera position setting unit 65b sets the rendering viewpoint Q (step S32).

The moving camera position setting unit 65b determines whether the angle formed by the line segment connecting the rendering viewpoint Q and the mapping surface E of the subject 18 and the line segment connecting the fixed camera 14 and the mapping surface E is equal to or less than the threshold value ( Step S33). When it is determined that the condition is satisfied (step S33: Yes), the process proceeds to step S35. On the other hand, if it is not determined that the condition is satisfied (step S33: No), the process proceeds to step S34.

The moving camera position setting unit 65b sets the installation position and direction of the moving camera 30 (step S34). The detailed flow of processing performed in step S34 will be described later (see FIG. 13).

The 3D shape calculation unit 66 extracts the 3D shape 130 of the subject 18 by, for example, a method of visual Hull and stereo matching (step S35).

The texture mapping unit 70 performs a rendering process for drawing a texture on the mapping surface E of the subject 18 (step S36).

The moving camera position setting unit 65b determines whether all the rendering viewpoints Q have been processed (step S37). When it is determined that all the rendering viewpoints Q have been processed (step S37: Yes), the image generation device 40b ends the processing of FIG. On the other hand, if it is not determined that all the rendering viewpoints Q have been processed (step S37: No), the process returns to step S32.

Next, with reference to FIG. 13, the flow of processing performed by the moving camera position setting unit 65b in step S34 will be described. FIG. 13 is a flowchart showing an example of the flow of the moving camera installation position and direction and the angle of view calculation process performed by the image generation device of the second embodiment.

The moving camera position setting unit 65b calculates the normal direction N of the mapping surface E of the subject 18 as seen from the rendering viewpoint Q (step S40).

The moving camera position setting unit 65b calculates the size of the subject 18 as seen from the direction N of the normal direction of the mapping surface E (step S41).

The mobile camera position setting unit 65b calculates the installation position, installation direction, and angle of view of the mobile camera 30 (step S42). Specifically, as described above, the angle formed by the line segment connecting the rendering viewpoint Q and the mapping surface E of the subject 18 and the line segment connecting the moving camera 30 and the mapping surface E is equal to or less than a predetermined threshold value. As described above, the installation position and the installation direction of the moving camera 30 are determined. Further, the angle of view of the moving camera 30 is determined so that the subject 18 is captured at the full angle of view of the moving camera 30.

The moving camera position setting unit 65b installs the moving camera 30 adjusted to the angle of view calculated in step S42 at the position and direction calculated in step S42 (step S43).

The camera image input unit 61 acquires an image of the subject 18 captured by the moving camera 30 (step S22). Although not shown in FIG. 13, the image acquired from the moving camera 30 is temporarily stored in the camera image file K by the action of the camera image storage unit 62. Further, the calibration processing unit 63 refers to the camera parameter file C2 and corrects the distortion of the image acquired from the moving camera 30. After that, the process returns to step S35 of FIG.

[2-4. Effect of the second embodiment]
As described above, according to the image generation device 40b (information processing device) of the second embodiment, the moving camera position setting unit 65b (setting unit) sets the set position, setting direction, and angle of view of the moving camera 30. The camera image input unit 61 (second image acquisition unit) is set so that the subject 18 is captured in the entire image captured by the moving camera 30 facing the normal direction N of the region of interest of the subject 18. However, the texture information of the subject 18 captured by the moving camera 30 installed in the installation position and the installation direction set by the moving camera position setting unit 65b is acquired. Then, the texture mapping unit 70 (drawing unit) maps the texture information of the subject 18 to the 3D shape of the subject 18 acquired by the 3D shape calculation unit 66 (3D shape acquisition unit).

Therefore, it is possible to generate a volumetric image in which the region of interest of the subject 18 is rendered with high definition.

Further, according to the image generation device 40b (information processing device) of the second embodiment, the mobile camera position setting unit 65b further detects the region of interest of the subject 18 and sets the installation position of the mobile camera 30 as the region of interest. Set in the area according to.

Therefore, by installing the moving camera 30 at a position facing the region of interest, it is possible to acquire higher-definition texture information of the subject 18.

Further, the image generation device 40b (information processing device) of the second embodiment detects the facial area of the subject 18 as the area of interest.

Therefore, when the subject 18 is a person, high-definition texture information of the face of the person can be acquired.

Note that the effects described in this specification are merely examples and are not limited, and other effects may be obtained. Moreover, the embodiment of the present disclosure is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present disclosure.

For example, the present disclosure can have the following structure.

(1)
A first image acquisition unit that acquires an image of the subject captured by a plurality of fixed cameras whose installation position and installation direction are fixed and installed around the subject with the direction of the subject facing the subject.
A second image acquisition unit that acquires an image of the subject captured by a moving camera that is installed near the fixed camera and has a variable installation position and direction.
The 3D shape of the subject obtained based on the image of the subject acquired by the first image acquisition unit and the image of the subject acquired by the second image acquisition unit has a predetermined accuracy. , A setting unit that sets the installation position and installation direction of the mobile camera,
The image of the subject acquired by the first image acquisition unit and the subject acquired by the second image acquisition unit captured by the moving camera installed at the installation position and installation direction set by the setting unit. A 3D shape acquisition unit that acquires the 3D shape of the subject based on the image, and
Information processing device equipped with.
(2)
The setting unit
The installation position and installation direction of the moving camera are set based on the 3D information of the subject based on the image of the subject acquired by the first image acquisition unit and the installation position and installation direction of the fixed camera. ,
The information processing device according to (1) above.
(3)
The setting unit
With the optical axis of the fixed camera and the optical axis of the moving camera parallel to each other,
The distance between the fixed camera and the moving camera is set so that the distance accuracy at the position of the maximum depth of the subject as seen from the fixed camera is higher than a predetermined value.
The information processing device according to (1) or (2) above.
(4)
The setting unit
When setting the installation position and the installation direction of the moving camera, the moving camera is moved with respect to the fixed camera in a direction corresponding to the surface state of the subject and in a direction away from the fixed camera. Set the installation position and direction of the mobile camera,
The information processing device according to any one of (1) to (3) above.
(5)
A drawing unit that maps the texture information of the subject to the 3D shape of the subject acquired by the 3D shape acquisition unit is further provided.
The setting unit faces the set position, setting direction, and angle of view of the moving camera with the normal direction of the region of interest of the subject so that the subject is captured in the entire image captured by the moving camera. Set,
The second image acquisition unit acquires the texture information of the subject captured by the moving camera installed at the installation position and the installation direction set by the setting unit.
The information processing device according to any one of (1) to (4).
(6)
Further provided with a designation unit for designating camera work representing the trajectory of the virtual viewpoint for observing the subject.
The setting unit sets the installation position and the installation direction of the moving camera based on the installation position and the installation direction of the fixed camera in the vicinity of the camera work designated by the designated unit.
The information processing device according to any one of (1) to (5) above.
(7)
The setting unit further detects the region of interest of the subject, and then
The installation position of the mobile camera is set within the area corresponding to the area of interest.
The information processing device according to any one of (1) to (6) above.
(8)
The feature area is a facial area of the subject.
The information processing device according to (7) above.
(9)
The mobile camera
A command unit for moving the moving camera to a designated position and a designated direction,
Further provided with a measuring unit for measuring the position and direction of the moving camera.
The information processing device according to any one of (1) to (8).
(10)
A plurality of fixed cameras with fixed installation positions and directions installed around the subject with the subject facing the direction of the subject.
A mobile camera with variable installation position and direction,
The installation position and installation direction of the moving camera are moved so that the 3D shape of the subject obtained based on the image of the subject by the fixed camera and the image of the subject by the moving camera has a predetermined accuracy. Movement control device to make
Information processing system equipped with.
(11)
An image of the subject taken by a plurality of fixed cameras whose installation position and installation direction are fixed and installed with the direction of the subject facing around the subject is acquired.
An image of the subject captured by a moving camera whose installation position and direction are variable, which is installed in the vicinity of the fixed camera, is acquired.
The installation position and installation direction of the moving camera are set so that the 3D shape of the subject obtained based on the image of the subject by the fixed camera and the image of the subject by the moving camera has a predetermined accuracy. do it,
The 3D shape of the subject is acquired based on the image of the subject by the fixed camera and the image of the subject by the moving camera installed in the set installation position and installation direction.
Information processing method.
(12)
Computer,
A first image acquisition unit that acquires an image of the subject captured by a plurality of fixed cameras whose installation position and installation direction are fixed and installed around the subject with the direction of the subject facing the subject.
A second image acquisition unit that acquires an image of the subject captured by a moving camera that is installed near the fixed camera and has a variable installation position and direction.
The 3D shape of the subject obtained based on the image of the subject acquired by the first image acquisition unit and the image of the subject acquired by the second image acquisition unit has a predetermined accuracy. , A setting unit that sets the installation position and installation direction of the mobile camera,
The image of the subject acquired by the first image acquisition unit and the subject acquired by the second image acquisition unit captured by the moving camera installed at the installation position and installation direction set by the setting unit. A 3D shape acquisition unit that acquires the 3D shape of the subject based on the image, and
A program that works.

10a, 10b ... Image generation system (information processing system), 14, 14a, 14b, 14c ... Fixed camera, 18 ... Subject, 20 ... Drone (movement control device), 30 ... Mobile camera, 40a, 40b ... Image generation device ( Information processing device), 60 ... Camera image input unit (first image acquisition unit), 61 ... Camera image input unit (second image acquisition unit), 62 ... Camera image storage unit, 63 ... Calibration processing unit, 64 ... 3D shape extraction unit, 65a, 65b ... Moving camera position setting unit (setting unit), 66 ... 3D shape calculation unit (3D shape acquisition unit), 67 ... Camera work designation unit (designation unit), 68 ... Movement command unit ( Command unit), 69 ... Position measurement unit (measurement unit), A1, A2 ... Optical axis, E ... Mapping surface, N ... Normal direction, Q ... Rendering viewpoint (virtual viewpoint)

Claims (12)

1. A first image acquisition unit that acquires an image of the subject captured by a plurality of fixed cameras whose installation position and installation direction are fixed and installed around the subject with the direction of the subject facing the subject.
   A second image acquisition unit that acquires an image of the subject captured by a moving camera that is installed near the fixed camera and has a variable installation position and direction.
   The 3D shape of the subject obtained based on the image of the subject acquired by the first image acquisition unit and the image of the subject acquired by the second image acquisition unit has a predetermined accuracy. , A setting unit that sets the installation position and installation direction of the mobile camera,
   The image of the subject acquired by the first image acquisition unit and the subject acquired by the second image acquisition unit captured by the moving camera installed at the installation position and installation direction set by the setting unit. A 3D shape acquisition unit that acquires the 3D shape of the subject based on the image, and
   Information processing device equipped with.
2. The setting unit
   The installation position and installation direction of the moving camera are set based on the 3D information of the subject based on the image of the subject acquired by the first image acquisition unit and the installation position and installation direction of the fixed camera. ,
   The information processing device according to claim 1.
3. The setting unit
   With the optical axis of the fixed camera and the optical axis of the moving camera parallel to each other,
   The distance between the fixed camera and the moving camera is set so that the distance accuracy at the position of the maximum depth of the subject as seen from the fixed camera is higher than a predetermined value.
   The information processing device according to claim 2.
4. The setting unit
   When setting the installation position and the installation direction of the moving camera, the moving camera is moved with respect to the fixed camera in a direction corresponding to the surface state of the subject and in a direction away from the fixed camera. Set the installation position and direction of the mobile camera,
   The information processing device according to claim 3.
5. A drawing unit that maps the texture information of the subject to the 3D shape of the subject acquired by the 3D shape acquisition unit is further provided.
   The setting unit faces the set position, setting direction, and angle of view of the moving camera with the normal direction of the region of interest of the subject so that the subject is captured in the entire image captured by the moving camera. Set,
   The second image acquisition unit acquires the texture information of the subject captured by the moving camera installed at the installation position and the installation direction set by the setting unit.
   The information processing device according to claim 1.
6. Further provided with a designation unit for designating camera work representing the trajectory of the virtual viewpoint for observing the subject.
   The setting unit sets the installation position and the installation direction of the moving camera based on the installation position and the installation direction of the fixed camera in the vicinity of the camera work designated by the designated unit.
   The information processing device according to claim 1.
7. The setting unit further detects the region of interest of the subject, and then
   The installation position of the mobile camera is set within the area corresponding to the area of interest.
   The information processing device according to claim 5.
8. The region of interest is the facial region of the subject.
   The information processing device according to claim 7.
9. A command unit for moving the moving camera to a designated position and a designated direction,
   Further provided with a measuring unit for measuring the position and direction of the moving camera.
   The information processing device according to claim 1.
10. A plurality of fixed cameras with fixed installation positions and directions installed around the subject with the subject facing the direction of the subject.
    A mobile camera with variable installation position and direction,
    The installation position and installation direction of the moving camera are moved so that the 3D shape of the subject obtained based on the image of the subject by the fixed camera and the image of the subject by the moving camera has a predetermined accuracy. Movement control device to make
    Information processing system equipped with.
11. An image of the subject taken by a plurality of fixed cameras whose installation position and installation direction are fixed and installed with the direction of the subject facing around the subject is acquired.
    An image of the subject captured by a moving camera whose installation position and direction are variable, which is installed in the vicinity of the fixed camera, is acquired.
    The installation position and installation direction of the moving camera are set so that the 3D shape of the subject obtained based on the image of the subject by the fixed camera and the image of the subject by the moving camera has a predetermined accuracy. do it,
    The 3D shape of the subject is acquired based on the image of the subject by the fixed camera and the image of the subject by the moving camera installed in the set installation position and installation direction.
    Information processing method.
12. Computer,
    A first image acquisition unit that acquires an image of the subject captured by a plurality of fixed cameras whose installation position and installation direction are fixed and installed around the subject with the direction of the subject facing the subject.
    A second image acquisition unit that acquires an image of the subject captured by a moving camera that is installed near the fixed camera and has a variable installation position and direction.
    The 3D shape of the subject obtained based on the image of the subject acquired by the first image acquisition unit and the image of the subject acquired by the second image acquisition unit has a predetermined accuracy. , A setting unit that sets the installation position and installation direction of the mobile camera,
    The image of the subject acquired by the first image acquisition unit and the subject acquired by the second image acquisition unit captured by the moving camera installed at the installation position and installation direction set by the setting unit. A 3D shape acquisition unit that acquires the 3D shape of the subject based on the image, and
    A program that works.

Images