JP2020182079A - Information processing device, information processing system, information processing method, and program - Google Patents

Abstract

To enable a plurality of imaging apparatuses to identify positions of the imaging apparatuses in a common coordinate system even if the imaging apparatuses are located on both sides of a boundary surface of a plurality of regions filled with substances having different refractive indexes.SOLUTION: A calibration unit 500 detects an airborne marker 810a from an image acquired by a camera 110a located above the water surface of a plurality of cameras 110 for capturing an imaging area containing at least part of the water surface, which is the boundary between air and water. The calibration unit 500 detects an underwater marker 810w from an image acquired by a camera 110w located below the water surface. The calibration unit 500 then identifies positions of the multiple cameras 110 in a common coordinate system based on the results of the detections and the positional information indicating the positions or positional relationships of the airborne marker 810a and the underwater marker 810w.SELECTED DRAWING: Figure 5

Description

The present invention relates to a technique for specifying the position of an imaging device based on an captured image.

There is a technique in which a plurality of imaging devices are installed at different positions to perform synchronous imaging, and a virtual viewpoint image in which the viewpoint can be arbitrarily changed is generated by using the plurality of captured images obtained by the imaging. Specifically, a virtual viewpoint image is generated by generating three-dimensional shape data of a subject included in the captured image based on a plurality of captured images and performing rendering processing based on the position and orientation of the virtual viewpoint. ..

In order to generate three-dimensional shape data of a subject based on captured images acquired by a plurality of imaging devices, it is necessary to specify the position of each imaging device in a common coordinate system. Patent Document 1 discloses that the position of each imaging device in a common coordinate system is specified by using a plurality of images obtained by imaging the same marker with a plurality of imaging devices.

JP-A-2018-207252

In the method described in Patent Document 1, when a plurality of image pickup devices are located on both sides of a boundary surface of a plurality of regions filled with substances having different refractive indexes, the positions of the plurality of image pickup devices are specified. There is a risk of error in the result. For example, when a virtual viewpoint image is generated for aquatics such as artistic swimming, it is conceivable to install imaging devices above and below the water surface, respectively. Since the refractive index of light differs between air and water, light is reflected on the water surface and fluctuations occur on the water surface, so that markers located deeper than the water surface when viewed from the image pickup device are stable from the captured image. Cannot be detected. Therefore, when the image captured by installing the marker only in water is used, an error may occur in the result of specifying the position of the imaging device above the water surface. Similarly, when an image captured by installing a marker only in the air is used, an error may occur in the result of specifying the position of the image pickup device below the water surface.

In view of the above problems, the present invention relates to the same coordinate system even when a plurality of imaging devices are located on both sides of a boundary surface of a plurality of regions filled with substances having different refractive indexes. The purpose is to be able to identify the positions of a plurality of imaging devices.

In order to solve the above problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, among a plurality of imaging devices that image an imaging region including at least a part of a boundary surface between the first region and the second region filled with a substance having a refractive index different from that of the substance satisfying the first region. The first detection means for detecting a predetermined first object in the first region from the image acquired by the image pickup device located on the first region side with respect to the boundary surface, and the plurality of image pickup devices. A second detection means for detecting a predetermined second object in the second region from an image acquired by an imaging device located on the second region side with respect to the boundary surface, the first object, and the second object. An acquisition means for acquiring at least one of the position of an object and the positional relationship between the first object and the second object, a detection result by the first detection means, and detection by the second detection means. It has a specific means for identifying the positions of the plurality of image pickup devices in a common coordinate system based on the result and the position information acquired by the acquisition means.

According to the present invention, even when a plurality of imaging devices are located on both sides of a boundary surface of a plurality of regions filled with substances having different refractive indexes, the plurality of imaging devices in a common coordinate system are used. The position of the device can be specified.

It is a figure which shows the installation example of a camera system. It is a figure which shows the configuration example of an information processing system. It is a figure which shows the hardware configuration example of the apparatus. It is a flowchart which shows the example of the process which concerns on calibration. It is a figure which shows the installation example of a marker. It is a figure which shows the installation example of a marker. It is a figure which shows the installation example of a marker.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

[Information processing system configuration]
A configuration example of the information processing system 10 will be described with reference to FIGS. 1 and 2. The information processing system 10 generates a virtual viewpoint image representing a view from a designated virtual viewpoint based on a plurality of images (multi-viewpoint images) based on images taken by a plurality of imaging devices and a designated virtual viewpoint. It is a system. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The information processing system 10 has a plurality of cameras as an imaging device that images an imaging region from a plurality of directions. In the present embodiment, it is assumed that the imaging region to be imaged is in the vicinity of a pool where aquatics such as artistic swimming, swimming, and water polo are performed. A plurality of cameras are installed at different positions so as to surround such an imaging area, and perform imaging in synchronization. Further, the plurality of cameras included in the information processing system 10 are classified into a plurality of imaging device groups having different installation locations. In the example shown in FIG. 1, the in-air camera system 100a is a group of image pickup devices having a plurality of cameras 110a installed above the water surface, specifically in the air around the pool, and mainly captures the air in the water. To do. On the other hand, the underwater camera system 100w is a group of image pickup devices having a plurality of cameras 110w installed in the water below the water surface, specifically in the corner of the pool, and mainly images the water. However, both the camera 100a in the air and the camera 100w in the water image an imaging region including at least a part of the water surface of the pool, which is the boundary surface between the region in the air and the region in the water.

It should be noted that the plurality of cameras may not be installed over the entire circumference of the imaging area, and may be installed only in a part of the imaging area depending on the limitation of the installation location or the like. Further, FIG. 1 shows cameras 110-1a, 110-2a, 110-1w, and 110-2w, and FIG. 2 shows cameras 110-3a and 110-3w in addition to the cameras 110-1a, 110-2a, 110-1w, and 110-2w. The number is not limited to this. For example, the air camera system 100a and the underwater camera system 100w may each have about 30 cameras. Further, the imaging target is not limited to the above, and the number of cameras may differ depending on the imaging target. Further, cameras having different functions such as a telephoto camera and a wide-angle camera may be installed.

In the present specification, it is assumed that the component having the alphabet a at the end of the code exists in the air, and the component having the letter w at the end of the code exists in the water. Further, when the cameras in the aerial camera system 100a are not distinguished, the camera is described as 110a, when the cameras in the underwater camera system 100w are not distinguished, the camera 110w is described, and when the air or underwater is not distinguished. Is simply referred to as camera 110. The same reference numerals are given to the other components of the air camera system 100a and the underwater camera system 100w.

As shown in FIGS. 1 and 2, a plurality of cameras 110a included in the air camera system 100a are daisy-chained via the corresponding camera control units 210a. Similarly, the plurality of cameras 110w included in the underwater camera system 100w are also daisy-chained via the corresponding camera control units 210w. It is not essential that the underwater camera system 100a and the underwater camera system 100w are connected separately, and the underwater camera system 100a and the underwater camera system 100w may be connected in series. Further, the daisy chain connection is not essential, and a plurality of camera control units 210 may be connected by a network topology such as a star type.

The plurality of cameras 110a included in the aerial camera system 100a are installed at positions and orientations suitable for photographing the water surface of the pool, and the focal length and focus can be obtained so that the attention position such as the vicinity of the water surface can be imaged with a predetermined image quality. Is set. Similarly, the camera 110w of the underwater camera system 100w is installed at a position and orientation suitable for photographing the underwater of the pool, and the focal length and focus are set so that the attention position such as the vicinity of the water surface can be imaged with a predetermined image quality. To In the present embodiment, it is assumed that the camera 110w included in the underwater camera system 100w has a waterproof function and is directly installed underwater. However, the present invention is not limited to this, and the camera 110w may be housed in a housing or the like that is installed underwater to ensure waterproofness. Further, the pool may be made of transparent acrylic glass or the like, and the camera 110w may be installed so that the water can be imaged from the outside of the pool and below the water surface.

As shown in FIG. 2, the in-air camera system 100a has a camera 110a and a plurality of camera control units 210a connected to the camera 110a. Each camera control unit 210a has a synchronization client 211a and an image transmission unit 212a. The plurality of camera control units 210a included in the aerial camera system 100a are daisy-chained, and the endmost camera control unit 210-1a is connected to the time server 300 and the data storage unit 400. The configuration of the underwater camera system 100w is also the same.

The synchronization client 211 of each camera control unit 210 communicates with the time server 300 and the synchronization client 211 of another camera control unit 210 to perform synchronization processing. As the synchronization protocol, PTP (Precision Time Protocol) is used, but the synchronization protocol is not limited to this method. The synchronization of a plurality of cameras 110 is realized by each synchronization client 211 outputting a GenLock signal and Timecode to the cameras 110 according to the result of the synchronization processing. Then, the plurality of cameras 110 perform synchronous imaging, and each camera 110 outputs the acquired captured image and the Timecode indicating the imaging time to the image transmitting unit 212 of the camera control unit 210. The image transmission unit 212 transmits the captured image acquired by the camera 110 to the data storage unit 400.

The data storage unit 400 stores various information necessary for generating a virtual viewpoint image in addition to the captured image acquired by the camera 110. The information stored in the data storage unit 400 also includes information obtained by calibration by the calibration unit 500. The calibration unit 500 has a calibration calculation unit 510 (hereinafter referred to as a calculation unit 510) and a calibration condition input unit 520 (hereinafter referred to as an input unit 520), and an captured image acquired from the data storage unit 400. Calibrate based on. The calibration executed by the calibration unit 500 is an information acquisition process for acquiring the camera parameters of each of the plurality of cameras 110 included in the information processing system 10. The camera parameters obtained by calibration include at least parameters indicating the position of the camera. However, not limited to this, even if the camera parameters acquired by calibration include parameters indicating the orientation of the camera, parameters indicating the focal length of the camera, parameters indicating the state of lens distortion of the camera, and the like. Good. The details of the calibration process will be described later.

The image generation unit 600 acquires a multi-viewpoint image and information (camera parameters) obtained by calibration from the data storage unit 400, and generates a virtual viewpoint image based on these and the viewpoint information acquired from the viewpoint setting unit 700. To do. The viewpoint information used to generate the virtual viewpoint image is information indicating the position and orientation of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may be information that has a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image and indicates the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. .. The viewpoint setting unit 700 generates viewpoint information according to a user operation, and outputs the viewpoint information to the image generation unit 600.

The virtual viewpoint image is generated by, for example, the following method. First, a foreground image in which a foreground area corresponding to a person, a ball, or the like is extracted from a plurality of images (multi-viewpoint images) acquired by taking images from different directions by a plurality of cameras 110 and a background area other than the foreground area are obtained. The extracted background image is acquired. In addition, a foreground model representing a three-dimensional shape of a person or the like and texture data for coloring the foreground model are generated based on the foreground image, and a texture for coloring the background model representing the three-dimensional shape of the background such as a pool. Data is generated based on the background image. Information on the position and orientation of each camera 110 is required to generate this foreground model, and the camera parameters obtained by calibration by the calibration unit 500 are used. Then, the texture data is mapped to the foreground model and the background model, and the virtual viewpoint image is generated by rendering according to the virtual viewpoint indicated by the viewpoint information. However, the method of generating the virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of the captured image without using the three-dimensional model can be used. The image generation unit 600 outputs the generated virtual viewpoint image to a display device or a storage device.

[Hardware configuration]
Next, the hardware configuration of the calibration unit 500, which is one of the information processing devices included in the information processing system 10, will be described with reference to FIG. Other devices included in the information processing system 10, such as the camera control unit 210 and the image generation unit 600, may have the same hardware configuration as the calibration unit 500. The calibration unit 500 includes a CPU 501, a ROM 502, a RAM 503, an auxiliary storage device 504, a display unit 505, an operation unit 506, a communication I / F 507, and a bus 508.

The CPU 501 realizes each function of the calibration unit 500 by controlling the entire calibration unit 500 using computer programs and data stored in the ROM 502 and the RAM 503. The calibration unit 500 may have one or more dedicated hardware different from the CPU 501, and the dedicated hardware may execute at least a part of the processing by the CPU 501. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 502 stores programs and the like that do not require changes. The RAM 503 temporarily stores programs and data supplied from the auxiliary storage device 504, data supplied from the outside via the communication I / F 507, and the like. The auxiliary storage device 504 is composed of, for example, a hard disk drive or the like, and stores various data such as captured images and information obtained by calibration.

The display unit 505 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the calibration unit 500. The operation unit 506 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 501 in response to an operation by the user. The CPU 501 operates as a display control unit that controls the display unit 505 and an operation control unit that controls the operation unit 506. The communication I / F 507 is used for communication with an external device. For example, when the calibration unit 500 is connected to an external device by wire, a communication cable is connected to the communication I / F 507. When the calibration unit 500 has a function of wirelessly communicating with an external device, the communication I / F 507 includes an antenna. The bus 508 connects each part of the calibration part 500 to transmit information.

In the present embodiment, it is assumed that the display unit 505 and the operation unit 506 exist inside the calibration unit 500, but at least one of the display unit 505 and the operation unit 506 may exist as separate devices. These do not have to exist.

[Calibration process]
The calibration by the calibration unit 500 is performed using the calibration image captured before the scene such as the competition for which the virtual viewpoint image is generated is started. Hereinafter, the imaging for acquiring the image required for calibration is referred to as calibration imaging, and the imaging of the scene for which the virtual viewpoint image is generated after the calibration imaging is referred to as scene imaging. .. If the virtual viewpoint image is not generated in real time but is generated based on the recorded captured image, the calibration imaging may be performed after the scene imaging.

Calibration imaging is performed with a reference object located within the imaging area. In the present embodiment, the reference object is the marker 810 that can be detected from the captured image acquired by the camera 110, and the board 800 that displays the marker 810 is installed at a position that can be imaged by the plurality of cameras 110. To do. The marker 810 may be printed on the board 800, or the board 800 may have a display for displaying the marker 810. Further, the shape of the board 800 and the reference object are not limited to these examples.

FIG. 5 shows an example of an imaging region at the time of calibration imaging. The board 800 has two markers 810a and 810w, and is an object that exists across a region above the water surface (in the air) and a region below the water surface (underwater). The marker 810a is located in the air portion of the board 800, and the marker 810w is located in the underwater portion of the board 800. The marker 810a can be imaged by the cameras 110-1a and 110-2a in the air, and the marker 810w can be imaged by the cameras 110-1w and 110-2w in the water. Each marker 810 is a two-dimensional marker having different contents from each other, and for example, the identification information of the marker 810 can be read from the image obtained by capturing the marker 810. However, the content of the marker 810 is not limited to this. In the following, FIG. 4 will be described assuming that the board 800 can move in the pool. However, the board 800 may be fixed in place at the bottom of the pool.

The details of the process related to the calibration by the calibration unit 500 will be described with reference to the flowchart of FIG. The process shown in FIG. 4 is realized by the CPU 501 of the calibration unit 500 expanding the program stored in the ROM 502 into the RAM 503 and executing the program. It should be noted that at least a part of the processing shown in FIG. 4 may be realized by one or a plurality of dedicated hardware different from the CPU 501. The process shown in FIG. 4 is started at the timing when a plurality of captured images acquired by the calibration imaging are stored in the data storage unit 400 and communication between the calibration unit 500 and the data storage unit 400 becomes possible. .. However, the start timing of the process shown in FIG. 4 is not limited to this.

In the description of FIG. 4, it is assumed that calibration imaging is performed for a certain period of time while changing the position of the marker 810, and images of a plurality of frames are stored in the data storage unit 400. By performing calibration using such images of a plurality of frames, the accuracy of calibration can be improved. However, the present invention is not limited to this, and the position of the marker 810 may be fixed during the calibration period. Further, an image captured at a single time may be stored in the data storage unit 400, and calibration may be performed using this image.

In S101, the calculation unit 510 selects at what time the captured image is acquired from the data storage unit 400. In S102, the calculation unit 510 acquires a plurality of images captured by the plurality of cameras 110 at the selected time T from the data storage unit 400. The image acquired in the image acquisition of S102 includes an image obtained by capturing the marker 810a in the air with the camera 110a included in the air camera system 100a. The image acquired in S102 also includes an image obtained by capturing the underwater marker 810w with the camera 110w included in the underwater camera system 100w.

In S103, the calculation unit 510 performs detection processing on each of the acquired images. If the marker 810 is detected from the image by the detection process, the process proceeds to S104, and if the marker 810 is not detected, the process proceeds to S105. Here, the calculation unit 510 performs detection processing of the marker 810a on the water for the image captured by the camera 110a in the air, and detects the marker 810w in the water for the image captured by the camera 110w in the water. Perform processing. After detecting the marker 810 from the image, the calculation unit 510 may determine whether the marker 810 is the marker 810a on the water or the marker 810w in the water.

In S104, the calculation unit 510 sets the time T, the set of marker numbers N read from the detected marker 810 (T, N), and the coordinates (x, y) indicating the position of the marker 810 in the image. The image is recorded in association with the identification information of the camera that captured the image. In S105, the calculation unit 510 determines whether the detection process of the marker 810 has been performed on all the frames of the calibration image stored in the data storage unit 400. If there is an image of a frame that has not been detected yet, the process returns to S101, the time T of the unprocessed frame is selected, and if the images of all frames have been processed, the process proceeds to S106.

In S106, the input unit 520 acquires the position information of the marker 810 and the initial calculation values of the position and the posture of the camera 110. The position information of the marker 810 indicates the positional relationship between the marker 810a in the air and the marker 810w in the water. The input unit 520 acquires position information regarding, for example, three markers 810a located in the air and three markers 810w located in the water. The position information acquired by the input unit 520 is not limited to this. For example, when the position of the marker 810 is fixed, the input unit 520 may acquire information indicating the positions of the markers 810a and the marker 810w in constant coordinates as position information. The input unit 520 shall acquire such information based on the input according to the user operation, but the present invention is not limited to this, and the result of measuring the position of the marker 810 from an external device may be acquired.

In S107, the calculation unit 510 performs calibration based on the information recorded in S104 and the information acquired in S106, using the time T and the marker numbers (T, N) as index IDs. The index used as the reference for calibration is treated as a different index even if the same marker 810 is used if the markers 810 are imaged at different times. Therefore, the set of (T, N) is treated as an index ID. In S108, the calculation unit 510 outputs the camera parameters obtained by the calibration to the data storage unit 400 and stores them. The camera parameters stored in the data storage unit 400 are used by the image generation unit 600 to generate a virtual viewpoint image.

In the calibration process in the present embodiment, the detection result of the marker 810a in the air is mainly used for the calibration of the camera 110a in the air, and the detection of the marker 810w in the water is mainly used for the calibration of the camera 110w in the water. The result is used. When these calibrations are performed independently, the camera parameters of the camera 110a in the air represented by the coordinates in one coordinate system and the camera parameters of the camera 110w in the water represented by the coordinates in another coordinate system are obtained. Be done. When a virtual viewpoint image including an underwater image and an image on the water is generated based on the camera parameters obtained in this way, for example, a virtual viewpoint image near the water surface, the image is disturbed due to the deviation of the coordinate system. It may occur.

Therefore, the calculation unit 510 performs processing based on the position information of the marker 810 acquired by the input unit 520, so that the camera parameters of the camera 110a in the air and the camera 110w in the water are represented by the coordinates in the common coordinate system. To get. For example, the calculation unit 510 acquires a camera parameter representing the position of the camera 110a in the air in the first coordinate system based on the result of detecting the marker 810a in the air from the image captured by the camera 110a in the air. .. Further, the calculation unit 510 acquires a camera parameter representing the position of the underwater camera 110w in the second coordinate system based on the result of detecting the underwater marker 810w from the image captured by the underwater camera 110w.

Then, the calculation unit 510 sets the camera parameters of the underwater camera 110w according to the second coordinate system to the first coordinate system based on the identification results of the markers 810a and the markers 810a specified from the position information acquired by the input unit 520. Convert to the corresponding value. That is, the calibration unit 500 acquires the camera parameters of a plurality of cameras 110 in a common coordinate system based on the image captured by the camera 110a in the air and the image captured by the camera 110w in the water. By generating a virtual viewpoint image using the camera parameters obtained by the calibration in this way, it is possible to suppress image distortion.

The method of acquiring camera parameters represented by coordinate values in a single coordinate system is not limited to this. For example, the calculation unit 510 may convert the camera parameters of the camera 110a in the air corresponding to the first coordinate system into a value corresponding to the second coordinate system. Further, for example, both the camera parameters of the camera 110a in the air corresponding to the first coordinate system and the camera parameters of the underwater camera 110w corresponding to the second coordinate system are converted into values corresponding to the third coordinate system. You may. Further, the calculation unit 510 may correct the deviation of the scale of the coordinate system by using the refractive index of water in the conversion of the camera parameters. However, it is not essential to use the index of refraction in the calculation of coordinates. For example, a plurality of markers as shown in FIG. 5 are used, and correction is performed based on the deviation between the known position information for each of the air marker and the underwater marker and the position information calculated from the detection result of the marker. You may.

To acquire the camera parameters in each of the first coordinate system and the second coordinate system, it is possible to use a known method of performing calibration based on the detection positions of the markers 810 in the plurality of captured images acquired by the plurality of cameras 110. it can. That is, the camera parameters in the first coordinate system of the camera 110a in the air can be acquired based on the detection results of the markers 810a from the plurality of images obtained by the camera system 100a in the air. Further, the camera parameters in the second coordinate system of the underwater camera 110w can be acquired based on the detection results of the markers 810w from the plurality of images obtained by the underwater camera system 100w.

In the above description using FIG. 4, the calibration process is performed by the calibration unit 500 after the calibration imaging is completed and the captured image is stored in the data storage unit 400. However, the present invention is not limited to this, and when the calibration imaging is performed over a certain period of time, the calibration process by the calibration unit 500 may be performed in parallel with the imaging by the camera 110.

[Marker installation example]
In the following, an installation example of the marker 810 when performing calibration imaging will be described. In the installation example shown in FIG. 5 mentioned in the above description, the marker 810a in the air and the marker 810w in the water are displayed on the board 800, which is the same object. That is, the water surface of the pool is located between the two markers of the board 800. On the board 800, the positional relationship between the position where the marker 810a is displayed and the position where the marker 810w is displayed is predetermined. Therefore, by performing calibration using the marker 810a and the position information of the marker 810w acquired by the input unit 520, the camera parameters based on the marker 810a and the camera parameters based on the marker 810w can be appropriately integrated. Can be done.

When calibration is performed using the marker 810 as shown in FIG. 5, the result of detecting the marker 810w in water from the image captured by the camera 110a in the air and the image captured by the camera 110w in the air in the air The result of detecting the marker 810a may be used. For example, the marker 810a in the air represents information that can identify that the marker 810a is located in the air, and the marker 810w in the water represents information that can identify that the marker 810w is located in the water. Specifically, the marker 810a in the air and the marker 810w in the water represent information in which different identifiers can be read. Then, the calculation unit 510 determines whether the marker 810 detected from the captured image is the marker 810a in the air or the marker 810w in the water.

When the calculation unit 510 determines that the marker 810w detected from the image captured by the camera 110a in the air is an underwater marker, the calculation unit 510 corrects the detection result with a known relative refractive index to prevent the refraction of light on the water surface. Correct the misalignment of the marker due to the influence. Further, when the calculation unit 510 determines that the marker 810a detected from the image captured by the underwater camera 110w is a marker in the air, the calculation unit 510 corrects the detection result by a known relative refractive index. Then, the calculation unit 510 adds the above correction to the result of detecting the marker 810a in the air from the image captured by the camera 110a in the air and the result of detecting the marker 810w in the water from the image captured by the camera 110w in the water. Calibration is performed using the detected detection result. According to such a method, the calibration can be performed using the detection results of more markers 810, so that the accuracy of the calibration can be improved.

The method of installing the marker 810 for calibration is not limited to the example of FIG. FIG. 6 shows another installation example of the marker 810. In the poolside air, a board 800-1a for displaying the marker 810-1a and a board 800-2a for displaying the marker 810-2a are installed. In the water in the pool, a board 800-1w for displaying the markers 810-1w and a board 800-2w for displaying the markers 810-2w are installed. The markers 810-1a, 810-2a, 810-1w, and 810-2w are located at positions that can be imaged by the cameras 110-1a, 110-2a, 110-1w, and 110-2w, respectively. Will be installed. These plurality of markers 810 are fixed during calibration imaging, and the positional relationship of the markers 810 does not change. The input unit 520 acquires the position information indicating the position of each marker 810 in the world coordinate system, and the calculation unit 510 obtains the specific result of specifying the position of each marker 810 from the position information and the image captured by each camera 110. Use to calibrate.

When a plurality of markers 810 are installed in this way, the markers 810a in the air are installed so as not to be within the imaging range of the underwater camera 110w, and the markers 810w in the water are not included in the imaging range of the camera 110a in the air. It may be installed as follows. By doing so, it is possible to prevent the calibration accuracy from being lowered by erroneously using the detection result of the underwater marker 810w detected from the captured image of the camera 110a in the air for calibration. Similarly, it is possible to avoid a decrease in calibration accuracy by erroneously using the detection result of the marker 810a in the air detected from the image captured by the underwater camera 110w for calibration. In this case, the content of the marker 810a and the content of the marker 810w may be the same.

On the other hand, as described above in the description of FIG. 5, each marker 810 may represent information in which the identifier of the marker can be read. In this case, since the calculation unit 510 can determine whether each marker 810 detected from the captured image is a marker in the air or a marker in the water, both the marker 810a in the air and the marker 810w in the water can be determined. May be included in the captured image of the same camera 110. This reduces restrictions on the installation of the marker 810. In this case, the result of detecting the marker 810w in the water from the image captured by the camera 110a in the air and the result of detecting the marker 810a in the air from the image captured by the camera 110w in the water are not used for calibration. May be good.

FIG. 7 shows another installation example of the marker 810. In the example of FIG. 7, the board 800 is located on the water surface, which is the boundary surface between the air and the water. The board 800 has a marker 810a on the upper surface (surface in air) and a marker 810w on the lower surface (surface in water). FIG. 7A is a side view of the pool in which the board 800 is floated on the water surface, and FIG. 7B is an arrow view of the board 800 as seen from the direction of arrow A in FIG. 7A. 7 (c) shows an arrow view of the board 800 as viewed from the direction of arrow B in FIG. 7 (a). Since the board 800 is not fixed, calibration imaging can be performed while moving the board 800. The input unit 520 acquires the position information indicating the positional relationship between the marker 810a and the marker 810w, and the calculation unit 510 calibrates using the positional relationship of the marker 810 specified from the position information and the image captured by each camera 110. I do.

If the water surface is rippling during calibration imaging, the inclination of the marker 810 may change and the calibration accuracy may decrease. Therefore, at least one of the marker 810a and the marker 810w may represent information that can specify the inclination of the marker. For example, the board 800 may have a gyro sensor or a spirit level, and markers 810 having different contents depending on the inclination detected by these may be displayed on the display of the board 800. Then, the calculation unit 510 may determine the inclination of the marker 810 by reading the content of the marker 810 detected from the captured image, and calculate the camera parameter using the detection result corrected based on the inclination of the marker. This makes it possible to improve the calibration accuracy when the inclination of the marker 810 changes during calibration imaging. The calculation unit 510 may determine the normal direction of the marker 810 with respect to the surface, and calculate the camera parameters based on the determination result. As another method, a marker including a perfect circle may be used, and the marker direction may be estimated from the shape of the ellipse included in the captured image.

As described above, the calibration unit 500 in the present embodiment has air from the image acquired by the camera 110a located above the water surface among the plurality of cameras 110 that image the imaging region including at least a part of the water surface. The marker 810a inside is detected. Further, the calibration unit 500 detects the underwater marker 810w from the image acquired by the camera 110w located below the water surface. Then, the calibration unit 500 identifies the positions of the plurality of cameras 110 in the common coordinate system based on these detection results and the position information indicating the positions or positional relationships between the markers 810a in the air and the markers 810w in the water. To do.

According to such a configuration, even when a plurality of cameras 110 are located on both sides of the water surface, which is a boundary between a region in air and a region in water having different refractive indexes, a common coordinate system is used. The positions of the plurality of cameras 110 in the above can be specified. Then, by generating the virtual viewpoint image based on the camera parameters indicating the positions of the plurality of cameras 110 specified in this way, it is possible to suppress the deterioration of the image quality of the virtual viewpoint image.

In the above description, there is a case where an air-filled region (water region) and a water-filled region (underwater region), which is a substance having a refractive index different from that of air, exist in the imaging region. Although an example has been described, the application destination of the calibration method in the present embodiment is not limited to this. For example, even when there is a region filled with air and a region filled with glass, resin, oil, or the like in the imaging region, the calibration accuracy is improved by applying the calibration method in the present embodiment. Can be made to. Further, for example, a region filled with water and a region filled with oil may exist in the imaging region. When the substance satisfying the imaging region is a fluid, particularly when it is air and a liquid, fluctuations are likely to occur at the interface, so that the effect of improving the calibration accuracy by the present embodiment is great. The substance that fills the region in the present embodiment refers to a substance that mainly constitutes the three-dimensional region (for example, a substance that occupies more than half of the volume of the three-dimensional region).

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a computer-readable recording medium.

10 Information processing system 110 Camera 500 Calibration unit 810 Marker

Claims (21)

Of a plurality of imaging devices that image an imaging region including at least a part of a boundary surface between the first region and a second region filled with a substance having a refractive index different from that of the substance satisfying the first region, the boundary surface A first detection means for detecting a predetermined first object in the first region from an image acquired by an imaging device located on the first region side.
A second detecting means for detecting a predetermined second object in the second region from an image acquired by the imaging device located on the second region side with respect to the boundary surface among the plurality of imaging devices.
An acquisition means for acquiring position information indicating at least one of the positions of the first object and the second object and the positional relationship between the first object and the second object.
A specific means for specifying the positions of the plurality of image pickup devices in a common coordinate system based on the detection result by the first detection means, the detection result by the second detection means, and the position information acquired by the acquisition means. An information processing device characterized by having. The first detection means detects the first object from an image acquired by an image pickup apparatus installed in the first region, and detects the first object.
The information processing device according to claim 1, wherein the second detection means detects the second object from an image acquired by an image pickup device installed in the second region. The information processing apparatus according to claim 1 or 2, wherein the substance satisfying the first region and the substance satisfying the second region are fluids. The information processing apparatus according to any one of claims 1 to 3, wherein the substance satisfying the first region is air, and the substance satisfying the second region is a liquid. The specific means
Based on the detection result by the first detection means, the first information indicating the position of the image pickup apparatus located on the first region side with respect to the boundary surface in the first coordinate system is acquired.
Based on the detection result by the second detection means, the second information indicating the position of the image pickup apparatus located on the second region side with respect to the boundary surface in the second coordinate system is acquired.
By converting at least one of the first information and the second information based on the position information acquired by the acquisition means, the positions of the plurality of imaging devices in the common coordinate system can be specified. The information processing apparatus according to any one of claims 1 to 4. The first object and the second object are located in different portions of an object existing across the first region and the second region, according to any one of claims 1 to 5. The information processing device described. The information processing apparatus according to claim 6, wherein the object is located at the boundary surface. The information processing apparatus according to any one of claims 1 to 7, wherein the first object and the second object are two-dimensional markers having different contents from each other. The information processing apparatus according to any one of claims 1 to 8, wherein the marker as the first object represents information that can identify that the marker is located in the first region. The information processing apparatus according to any one of claims 1 to 9, wherein the marker as the first object represents information capable of specifying the inclination of the marker. The image acquired by the image pickup apparatus located on the first region side with respect to the boundary surface includes the first object and the second object.
The information processing apparatus according to any one of claims 1 to 10, wherein the detection result by the first detection means used by the specific means is the detection result of the first object. The first detection means detects the first object and the second object from the image acquired by the image pickup device located on the first region side with respect to the boundary surface.
The detection result by the first detection means used by the specific means includes any one of claims 1 to 10, characterized in that the detection result of the first object and the detection result of the second object are included. The information processing device described. The information processing device according to any one of claims 1 to 12, wherein the specifying means further specifies the orientation of the plurality of imaging devices in the common coordinate system. The plurality of imaging devices include a first imaging device group located on the first region side with respect to the boundary surface and a second imaging device group located on the second region side with respect to the boundary surface. ,
The first detection means detects the first object from a plurality of images acquired by the first imaging apparatus group, and detects the first object.
The second detection means detects the second object from a plurality of images acquired by the second imaging apparatus group, and detects the second object.
The specific means acquires the detection result of the first object from a plurality of images by the first detection means, the detection result of the second object from a plurality of images by the second detection means, and the acquisition means. The information processing apparatus according to any one of claims 1 to 13, wherein the positions of the plurality of imaging devices in the common coordinate system are specified based on the obtained position information. The information processing according to any one of claims 1 to 14, wherein the plurality of imaging devices acquire a plurality of images used for generating a virtual viewpoint image by imaging from different directions. apparatus. An image obtained by imaging an object in the first region by a group of first imaging devices located in the first region of the plurality of imaging devices, and a substance satisfying the first region among the plurality of imaging devices. Is based on an image obtained by imaging an object in the second region by a group of second imaging devices located in a second region filled with substances having different refractive indexes. Processing means for calibration and
Virtually based on the information obtained by the calibration by the processing means, the plurality of images obtained by imaging from a plurality of directions by the plurality of imaging devices, and the viewpoint information indicating the position and orientation of the virtual viewpoint. An information processing system characterized by having a generation means for generating a viewpoint image. The information processing apparatus according to claim 16, wherein the substance satisfying the first region is air, and the substance satisfying the second region is a liquid. Of a plurality of imaging devices that image an imaging region including at least a part of a boundary surface between the first region and a second region filled with a substance having a refractive index different from that of the substance satisfying the first region, the boundary surface A first detection step of detecting a predetermined first object in the first region from an image acquired by an imaging device located on the first region side.
A second detection step of detecting a predetermined second object in the second region from an image acquired by the imaging device located on the second region side with respect to the boundary surface among the plurality of imaging devices.
An acquisition step of acquiring position information indicating at least one of the positions of the first object and the second object and the positional relationship between the first object and the second object.
A specific step of specifying the positions of the plurality of imaging devices in a common coordinate system based on the detection result in the first detection step, the detection result in the second detection step, and the position information acquired in the acquisition step. An information processing method characterized by having. In the first detection step, the first object is detected from an image acquired by an image pickup device installed in the first region.
The information processing apparatus according to claim 18, wherein the second detection step detects the second object from an image acquired by an image pickup apparatus installed in the second region. The information processing method according to claim 18 or 19, wherein the substance satisfying the first region is air, and the substance satisfying the second region is a liquid. A program for operating a computer as an information processing device according to any one of claims 1 to 15.

Images