JP2020188395A - Method, information processing device, and program supporting adjustment of posture of imaging device - Google Patents

Abstract

To support to efficiently perform posture adjustment work of an imaging device in a system in which an imaging region is imaged from different directions by a plurality of imaging devices.SOLUTION: A first image which is an image to be captured by an imaging device at the position and orientation in which the imaging device is to be installed, and a second image which is an image captured by the imaging device at the position and orientation in which the imaging device is currently installed are acquired, and the second image is displayed together with the acquired first image.SELECTED DRAWING: Figure 7

Description

The present invention relates to a method of assisting in adjusting the posture of an imaging device.

Attention has been paid to a technique in which multiple cameras (imaging devices) are installed at different positions to perform synchronous shooting from multiple viewpoints, and a virtual viewpoint image based on an arbitrary viewpoint is generated using the multi-viewpoint image obtained by the shooting. (See Patent Document 1). According to such technology, for example, highlight scenes of games such as soccer and basketball, concerts, etc. can be viewed from various angles, giving the user a high sense of presence as compared with ordinary images. Can be done.

When shooting an event such as a sports game or concert, it is necessary to install equipment such as a camera as a preparatory step. In the virtual viewpoint image generation system, it is necessary to adjust the position and orientation of each camera with high accuracy.

Conventionally, a camera installation worker installs a marker in a target facility (field) and adjusts the posture of the camera while looking at a captured image so that the optical axis direction of the camera faces the marker.

JP-A-2008-015756

However, the markers are relatively small with respect to a wide field, and there may be a plurality of markers. Therefore, it was necessary to refer to a manual or the like to determine which marker the camera should be adjusted to. .. As a result, it was a factor that increased the working time and the number of workers. In addition, there is a risk of leading to human error such as target error by the worker. Furthermore, due to various reasons of the facility, it may not be possible to install the marker, and the posture adjustment work itself may be difficult.

An object of the present invention is to support efficient posture adjustment work of an imaging device in a system in which an imaging region is imaged from different directions by a plurality of imaging devices.

The present invention is a method of supporting adjustment of the posture of each imaging device in a system in which an imaging region is imaged from different directions by a plurality of imaging devices, and one of the plurality of imaging devices is installed. A first acquisition step of acquiring a first image which is an image to be captured by the imaging device at a power position and a posture, and
A second acquisition step of acquiring a second image which is an image captured by the image pickup device at the position and orientation in which the image pickup device is currently installed, and the first acquisition step acquired in the first acquisition step. It is characterized by having a display step of displaying the second image acquired in the second acquisition step together with the image.

According to the present invention, in a system in which an imaging region is imaged from different directions by a plurality of imaging devices, it is possible to support the posture adjustment work of the imaging device to be efficiently performed.

It is a figure which shows the configuration example of an image processing system. It is a figure which shows the functional configuration example of an image generation apparatus 150 and a virtual camera operation terminal 160. It is a figure which shows the functional configuration example of a camera adapter. It is a figure which shows the example of the workflow for operating a system. It is a figure which shows the example of the screen display in a camera operation terminal. It is a figure which shows the example of the equipment installation situation in a camera operation terminal. It is a figure for demonstrating the display example in the camera operation terminal at the time of adjusting the posture of a camera. It is a figure which shows the example of the processing flow at the time of posture adjustment of a camera. It is a figure for demonstrating another display example in a camera operation terminal at the time of posture adjustment of a camera. It is a figure which shows another example of the processing flow at the time of posture adjustment of a camera. It is a figure which shows the hardware configuration example of a camera operation terminal.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the configurations described in the following embodiments are merely examples, and the present invention is not intended to be limited to the embodiments described.

<First Embodiment>
(System configuration)
In this embodiment, a system in which a plurality of cameras (imaging devices) are installed in a facility such as a stadium (stadium) or a concert hall to take a picture, and a virtual viewpoint image is generated using the multi-viewpoint image obtained by the shooting. An example of is described. In the present embodiment, in a multi-camera system for generating a virtual viewpoint image, a method of supporting the efficient progress of the work by visually performing the posture adjustment work of each camera will be described.

(Explanation of image processing system 100)
FIG. 1 shows a configuration example of the image processing system 100 according to the present embodiment. The image processing system 100 is a system that generates a virtual viewpoint image representing a view from a designated virtual viewpoint based on a plurality of images taken by a plurality of imaging devices and a designated virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual visual image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The image processing system 100 has a plurality of cameras (imaging devices) that image an imaging region from a plurality of directions. The imaging area is, for example, a stadium where a competition such as soccer is performed, or a stage where a concert or a play is performed. A plurality of cameras are installed at different positions so as to surround such an imaging area, and perform imaging in synchronization. It should be noted that the plurality of cameras may not be installed over the entire circumference of the imaging area, and may be installed only in a part of the imaging area depending on the limitation of the installation location or the like. Further, cameras having different functions such as a telephoto camera and a wide-angle camera may be installed.

The image processing system 100 includes a sensor system 110a to 110z, a switching hub 120, a camera operation terminal 130, an image computing server 140, an image generation device 150, a virtual camera operation terminal 160, and a control station 170.

First, the configuration and operation of the sensor systems 110a to 110z will be described. The sensor systems 110a to 110z have microphones 111a to 111z, cameras 112a to 112z, pan heads 113a to 113z, external sensors 114a to 114z, and batteries 115a to 115z, respectively. Further, the sensor systems 110a to 110z have camera adapters 116a to 116z, respectively, and are connected to each other by daisy chains 121a to 121y via the camera adapters 116a to 116z. The connection form is not limited to the daisy chain connection, and another network topology such as a star type may be adopted. Further, in FIG. 1, an example in which 26 sensor systems 110a to 110z exist will be described, but the number of sensor systems is not limited to this. Further, in FIG. 1, it is assumed that the sensor systems 110a to 110z have a common configuration, but at least a part of the sensor systems may have a configuration different from that of other sensor systems.

In the present embodiment, unless otherwise specified, the sensor systems 110a to 110z are referred to as "sensor system 110" without distinction. Further, each configuration of the sensor system 110 is also referred to as a microphone 111, a camera 112, a pan head 113, an external sensor 114, a battery 115, and a camera adapter 116 when it is not necessary to distinguish them. The sensor system 110 acquires image data by taking a picture with its own camera 112, and also acquires audio data by recording with a microphone 111. That is, the image processing system 100 can obtain image data captured from a plurality of directions by using a plurality of sensor systems 110 that capture the same subject from different directions. Similarly, the image processing system 100 uses these plurality of sensor systems 110 to collect sound at different positions (with different directional characteristics as needed) to collect audio data recorded at the plurality of positions. Obtainable.

It is assumed that the optical axes of the plurality of cameras 112 are fixed, and the shooting range is not changed to the left, right, up, or down by panning, tilting, or the like. It should be noted that the adjustment of the optical axis at the time of initial setting of the camera 112 and the correction of the deviation of the optical axis due to the environment can be performed by the pan head 113 on which the camera 112 is installed. Each of the plurality of sensor systems 110 transmits the image data obtained by photographing to other sensor systems downstream of the daisy chains 121a to 121y. At this time, each sensor system 110 also transfers the image data acquired from the other upstream sensor system to the other downstream sensor system. For example, the sensor system 110a transmits the image data taken by the camera 112a to the camera adapter 116b of the sensor system 110b via the daisy chain 121a after performing the image processing described later on the camera adapter 116a. Similarly, the sensor system 110b transmits the image data captured by the camera 112b to the sensor system 110c together with the image data acquired from the sensor system 110a (in some cases, as a separate signal). The sensor system 110z transmits the image data obtained by photographing and the image data acquired from the upstream sensor system to the switching hub 120 via the network 122b. By using a daisy chain, it is possible to prevent an increase in the number of connection cables and the complexity of wiring work even if the image data capacity increases as the resolution of the captured image increases to 4K or 8K and the frame rate increases. Can be done.

The same applies to the handling of a plurality of microphones 111 and the voice data collected by those microphones 111. That is, it is assumed that the direction of the plurality of microphones 111 is fixed and the direction is not changed. In addition, the voice data obtained by sound collection is transmitted via a daisy chain. By such an operation, the image data and the audio data acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z to the switching hub 120 using the network 122b, and then transferred to the image computing server 140.

The external sensor 114 acquires various environmental information by a method other than photographing or collecting sound. For example, the external sensor 114 can acquire various information such as vibration and inclination of the sensor system 110, air temperature, humidity, wind direction, and wind speed. It should be noted that these are examples, and the external sensor 114 has an arbitrary device configuration for acquiring an arbitrary environmental parameter.

The battery 115 is used when executing a workflow for installing or withdrawing the sensor system 110 in a facility such as a stadium, which will be described later, and is used, for example, when a system power supply is not installed in the stadium. The sensor system 110 may not use the battery 115 after the system power supply is laid after the installation is completed. Further, the sensor system 110 may be configured to operate using the battery 115 without the battery 115 when the system power supply is laid in advance. A power source permanently installed in the facility may be used instead of the system power source.

Although FIG. 1 shows an example in which the camera 112 and the camera adapter 116 are separated from each other, these may be one device included in the same housing. Further, the microphone 111 may also be a microphone built in the camera 112, or a separate device may be prepared as shown in FIG. As described above, the plurality of functional units shown in FIG. 1 may be realized by one device, or one functional unit may be realized by the cooperation of a plurality of devices.

Further, in the present embodiment, unless otherwise specified, the term "image" includes the concepts of both moving images and still images. That is, the image processing system 100 can deal with both still images and moving images. Further, in the present embodiment, the case where the virtual viewpoint image and the virtual listening point sound are provided by the image processing system 100 as the virtual viewpoint contents will be described, but the present invention is not limited to this. For example, the virtual viewpoint content does not have to include sound. Further, for example, the sound included in the virtual viewpoint content may be provided in a format other than the virtual listening point sound, such as the sound recorded by the microphone closest to the virtual viewpoint. Further, in the present embodiment, for the sake of simplicity, the description of the sound is partially omitted, but both the image and the sound are processed.

Next, the configuration and operation of the image computing server 140 will be described. The image computing server 140 processes the data acquired from the sensor system 110z. The image computing server 140 includes a time server 141 and a front-end server 142.

The time server 141 has a function of distributing the time and synchronization signals, and distributes the time and synchronization signals to the sensor systems 110a to 110z via the switching hub 120. When the sensor systems 110a to 110z receive the time and synchronization signals via the camera adapters 116a to 116z, the sensor systems 110a to 112z synchronize the cameras 112a to 112z based on the time and synchronization signals to establish image frame synchronization. That is, the time server 141 synchronizes the shooting timings of the plurality of cameras 112.

The front-end server 142 reconstructs a segmented transmission packet from the image and sound acquired from the sensor system 110z to convert the data format. After that, the data is written in the database (not shown) in the front-end server 142 according to the camera identifier, the data type indicating whether it is an image or sound, and the frame number.

The front-end server 142 temporarily stores the data acquired from the camera adapter 116 on the DRAM, and buffers the foreground image, the background image, the audio data, and the three-dimensional model data until they are prepared.

Here, the foreground image is an image obtained by extracting a region (foreground region) of a foreground object from an captured image captured and acquired by the camera 112. The foreground object extracted as the foreground area refers to a dynamic object (moving object) that moves (the absolute position and shape of the object can change) when images are taken from the same direction in time series. Foreground objects are people such as players and referees in the field in which they are played, balls in the case of ball games, and singers, performers, performers, and moderators in concerts and entertainment.

The background image is an image of at least an area (background area) different from the object that is the foreground. Specifically, the background image is an image in which the foreground object is removed from the captured image. Further, the background refers to an imaged object that is stationary or continues to be in a state close to stationary when imaging is performed from the same direction in time series. Such an imaging object is, for example, a stage such as a concert, a stadium where an event such as a competition is held, a structure such as a goal used in a ball game, a field, or the like. However, the background is at least a region different from the foreground object, and the imaging target may include another object or the like in addition to the foreground object and the background.

The three-dimensional model data is shape data representing the three-dimensional shape of the foreground object, and is generated based on a plurality of foreground images.

In the following, data for generating virtual viewpoint contents such as foreground image, background image, audio data, and three-dimensional model data are collectively referred to as material data. Meta information such as routing information, time code information (time information), and camera identifier is added to the material data, and the front-end server 142 confirms the data attributes based on this meta information. As a result, the front-end server 142 can determine whether or not the material data is complete by checking the buffered data for a certain time. By this determination, when data is transferred from the camera adapter 116 via the network 122 or the like, even if the reception order of network packets does not correspond to the order of the data, it is possible to wait until the necessary data are available. File generation can be performed reliably. The background image may be taken at a frame rate different from that of the foreground image. For example, when the frame rate of the background image is 1 fps (frames per second), one background image is acquired every second. On the other hand, when the foreground image is acquired at, for example, 60 fps, there may be a timing when the foreground image is acquired but the background image is not acquired. Therefore, at such a timing, the front-end server 142 may determine that all the data have been collected without the background image.

The camera operation terminal 130 is a general-purpose personal computer (PC) or tablet, and is an information processing device used for supporting the installation and posture adjustment of the camera 112. Although FIG. 1 shows a case where only one camera operation terminal 130 is provided, a plurality of camera operation terminals 130 may be provided.

The camera operation terminal 130 is configured to be capable of wireless communication with the sensor system 110. Then, the camera operation terminal 130 can operate the operation system application. The operational system application is used when operating a multi-camera system and is used to manage operations from event shooting planning to system installation, shooting, and system withdrawal. Further, the camera operation terminal 130 may be configured to be capable of executing screen display of an operation system application necessary for operating the system and communication by text message, voice, or the like with another camera operation terminal.

The image generation device 150 generates a virtual viewpoint image based on the virtual camera path parameter based on the instruction from the virtual camera operation terminal 160 and the material data stored in the front-end server 142. The virtual camera operation terminal 160 receives an instruction for designating the virtual camera path from the user, and transmits information according to the received instruction to the image generation device 150.

Here, the virtual camera is a virtual camera different from the plurality of cameras 112 actually installed around the imaging region, and is for convenience to explain the virtual viewpoint related to the generation of the virtual viewpoint image. It is a concept. That is, the virtual viewpoint image can be regarded as an image captured from a virtual viewpoint set in the virtual space associated with the imaging region. Then, the position and orientation of the viewpoint in the virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image simulating the captured image obtained by the camera, assuming that the camera exists at the position of the virtual viewpoint set in the space. The virtual camera path is information indicating the transition of the virtual viewpoint over time. However, it is not essential to use the concept of a virtual camera in order to realize the configuration of the present embodiment. That is, at least the information indicating a specific position in the space and the information indicating the direction are set, and the virtual viewpoint image may be generated according to the set information.

The functional configuration of the image generation device 150 and the virtual camera operation terminal 160 will be described with reference to FIG. Since the image generator 150 cooperates with the front-end server 142 described above, FIG. 2 also illustrates a part of the functional configuration of the front-end server 142.

The front-end server 142 has a multi-viewpoint image holding unit 200 and a real space information holding unit 210 as its functions. As described above, the multi-viewpoint image holding unit 200 holds image data related to images synchronously taken by a plurality of cameras installed so as to surround the competition field, the studio, and the like.

Further, the real space information holding unit 210 holds information about a space in a predetermined range including the subject. The real space information holding unit 210 provides 3D model information of an object (background object) reflected as a background in a virtual viewpoint image, such as a stadium field, spectator seats, and studio equipment, and a range in which a virtual viewpoint can be set. Holds the three-dimensional spatial information to be shown.

The image generation device 150 includes a virtual viewpoint image generation unit 221, a virtual camera path calculation unit 222, a virtual camera information holding unit 223, a display image generation unit 224, and a device communication unit 225. Further, the virtual camera operation terminal 160 has a terminal communication unit 231, an image display unit 232, a virtual camera path instruction unit 233, and a user information transmission unit 234.

The virtual viewpoint image generation unit 221 generates a three-dimensional model of the foreground object using the multi-viewpoint image acquired from the multi-viewpoint image holding unit 200. By mapping the generated foreground 3D model and the background 3D model acquired from the real space information holding unit 210 to the texture according to the virtual viewpoint acquired from the virtual camera path calculation unit 222 and rendering. , A virtual viewpoint image is generated. The virtual viewpoint image generation unit 221 calculates the coordinates of the foreground object and the background object reflected in the virtual viewpoint image to be generated in the process of generating the virtual viewpoint image, and performs texture mapping and rendering only for those coordinates. Run. The virtual viewpoint image generation unit 221 outputs the calculated coordinate values as the subject foreground coordinates and the real space background coordinates to the virtual camera information holding unit 223 described later.

The virtual camera path calculation unit 222 calculates temporally continuous virtual camera path parameters based on the content of instructions from the user to the virtual camera path instruction unit 233 described later. The virtual camera path parameter is a parameter indicating the movement of the virtual camera when the user observes the subject, and includes at least the position and orientation (gaze direction) of the virtual camera. The virtual camera path parameter is not limited to these, and may include, for example, an angle of view (focal length). In addition, the virtual camera path parameter is associated with a frame number or time code attached to the multi-viewpoint image so that it can identify which moment in the shooting scene it relates to. The virtual camera path calculation unit 222 sets the virtual camera path within a range in which the virtual viewpoint can be set by referring to the real space information acquired from the real space information holding unit 210 when calculating the virtual camera path parameters.

The virtual camera information holding unit 223 stores the subject foreground coordinates and the real space background coordinates acquired from the virtual viewpoint image generation unit 221 and the virtual camera path parameters calculated by the virtual camera path calculation unit 222.

The display image generation unit 224 generates a display image to be displayed on the image display unit 232 of the virtual camera operation terminal 160. The display image generated here is a virtual viewpoint image generated by the virtual viewpoint image generation unit 221. The device communication unit 225 is a functional unit for performing wired communication or wireless communication between the image generation device 150 and the virtual camera operation terminal 160 via a network (not shown) or the like. The image, voice, and text data and instruction information regarding the virtual camera path instructed by the user when generating the virtual viewpoint image are transmitted and received between the image generation device 150 and the virtual camera operation terminal 160.

In the virtual camera operation terminal 160, the terminal communication unit 231 transmits and receives various information to and from the device communication unit 225 as described above. The image display unit 232 displays an image generated by the display image generation unit 224 described above and received via the terminal communication unit 231 on the screen. The virtual camera path instruction unit 233 receives an instruction for the virtual camera path from the user, and transmits information indicating the instruction to the image generation device 150 using the terminal communication unit 231. The image generation device 150 receives this information using the device communication unit 225, and passes the received information to the virtual camera path calculation unit 222. The user does not have to strictly specify all the virtual camera parameters for the entire time that he / she wants to watch. For example, the user "wants to watch a virtual viewpoint image focusing on a specific subject (player, performer, etc.)", "wants to watch a certain area around the ball", and "a more remarkable event occurs. You may enter instructions based on various viewpoints such as "I want to watch the place where I am". The user information transmission unit 234 attaches user information such as a user ID to the information transmitted from the terminal communication unit 231 to the device communication unit 225.

The control station 170 manages the operating state of each block constituting the image processing system 100 and controls parameter setting through the daisy chains 121a to 121y, the networks 122a to 122b, and 123 to 124.

Subsequently, the functional configuration of the camera adapter 116 will be described with reference to FIG. The camera adapter 116 includes a network adapter 300, a transmission unit 310, an image processing unit 320, and an external device control unit 330. Here, the network adapter 300 includes a data transmission / reception unit 301 and a wireless communication unit 302, and the transmission unit 310 includes an image / audio transmission processing unit 311 and a data compression / decompression unit 312. Further, the image processing unit 320 includes a foreground background separation unit 321, a three-dimensional model information generation unit 322, and a calibration control unit 323. Further, the external device control unit 330 includes a microphone control unit 331, a camera control unit 332, a pan head control unit 333, and a sensor control unit 334. The external device control unit 330 further includes an external device connection state detection unit 335.

The network adapter 300 has various functions for communicating with other devices. The data transmission / reception unit 301 performs data communication with another camera adapter 116, a front-end server 142, a time server 141, and a control station 170 via the daisy chain 121, the network 123, and the network 124. For example, the data transmission / reception unit 301 outputs the foreground image and the background image obtained by separating the image captured by the camera 112 by the foreground background separation unit 321 described later to another camera adapter 116. The output destination camera adapter 116 is a camera adapter 116 on the downstream side adjacent to the daisy chain 121 among the camera adapters 116 of the image processing system 100. By outputting the foreground image and the background image by a large number of camera adapters 116, the foreground image and the background image taken from a plurality of viewpoints are transmitted to the front-end server 142, and the virtual viewpoint image is generated by the image generator 150. Will be done. There may be a camera adapter 116 that outputs a foreground image separated from the captured image and does not output a background image.

The wireless communication unit 302 communicates with the camera operation terminal 130. The data communicated via the wireless communication unit 302 is the data processed by the external device connection state detection unit 335, which will be described later, and is the data for executing the processing in the operation system application operated by the camera operation terminal 130.

The transmission unit 310 executes signal processing on data transmitted and received via the network adapter 300. The image / audio transmission processing unit 311 creates a message for transferring the image data or the audio data to another camera adapter 116 or the front-end server 142 via the data transmission / reception unit 301. The message includes image data or audio data, and meta information of each data. The meta information of the present embodiment includes a time code or sequence number at the time of image shooting or sound sampling, a data type, and an identifier indicating an individual microphone 111 or camera 112. Further, the transmitted image data or audio data may be data-compressed by the data compression / decompression unit 312. Further, the image / audio transmission processing unit 311 receives a message from another camera adapter 116 via the data transmission / reception unit 301. Then, the image / audio transmission processing unit 311 restores the image data or the audio data from the data divided into the packet sizes specified by the transmission protocol according to the data type included in the message. If the data is compressed when the data is restored, the data compression / decompression unit 312 performs the decompression process. The data compression / decompression unit 312 compresses the data transmitted / received via the data transmission / reception unit 301 by applying a predetermined compression method, compression rate, and frame rate, and decompresses the compressed data.

The image processing unit 320 performs image processing on the image data taken by the camera 112 under the control of the camera control unit 332 and the image data received from the other camera adapter 116. The foreground background separation unit 321 separates the image data taken by the camera 112 into a foreground image and a background image. That is, the foreground background separation unit 321 in each of the plurality of camera adapters 116 extracts the foreground region from the captured image captured by the corresponding camera 112 among the plurality of cameras 112. The foreground background separation unit 321 uses the image in the foreground region extracted from the captured image as the foreground image, and the image in the region outside the foreground region (that is, the image in the region other than the foreground image) as the background image. And output the background image. The three-dimensional model information generation unit 322 relates to a three-dimensional model using, for example, the principle of a stereo camera by using the foreground image separated by the foreground background separation unit 321 and the foreground image received from another camera adapter 116. Generate image information. The calibration control unit 323 performs color correction processing for suppressing color variation for each camera on the input image, image alignment processing for blurring caused by camera vibration, image cropping processing, and the like. I do.

The external device control unit 330 has a function of controlling a device connected to the camera adapter 116. The camera control unit 332 is connected to the camera 112 in the sensor system including the camera adapter 116, and controls the camera 112, acquires a captured image, provides a synchronization signal, sets the time, and the like. The control of the camera 112 indicates, for example, the setting and reference of shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), and the state of the camera 112 (shooting, stopped, synchronizing, error, etc.). Includes acquisition of information, start and stop of shooting, focus adjustment, etc. The microphone control unit 331, the pan head control unit 333, and the sensor control unit 334 are connected to the microphone 111, the pan head 113, and the external sensor 114 in the sensor system including the camera adapter 116, respectively, and connect the connected devices. Control. Further, the microphone control unit 331, the pan head control unit 333, and the sensor control unit 334 perform a process of reading information indicating a state such as whether or not the connected device is operating normally from each device. The external device connection state detection unit 335 has a function of reading a self-check result such as the connection state of the device connected to the camera adapter 116 and whether each device is operating normally. The external device connection status detection unit 335 communicates with the microphone control unit 331, the camera control unit 332, the pan head control unit 333, and the sensor control unit 334, and obtains the connection status and self-check result of the external device acquired from each functional unit. , Stored in the memory in the external device connection state detection unit 335. The external device connection state detection unit 335 detects only the cable connection state when power is not supplied to the external device. The cable connection can be detected at the hardware level by setting one of the cable terminals of the camera adapter 116 to the HIGH port so that it becomes LOW when the cable is connected.

(Hardware configuration)
Next, the hardware configuration of the camera operation terminal 130 will be described with reference to FIG. The hardware configurations of the camera adapter 116, the time server 141, the front-end server 142, the image generator 150, the virtual camera operation terminal 160, and the control station 170 are the same as those of the camera operation terminal 130.

The camera operation terminal 130 includes a CPU 1101, a ROM 1102, a RAM 1103, an auxiliary storage device 1104, a display unit 1105, an operation unit 1106, a communication I / F 1107, and a bus 1108.

The CPU 1101 realizes each function of the camera operation terminal 130 shown in FIG. 1 by controlling the entire camera operation terminal 130 by using computer programs and data stored in the ROM 1102 and the RAM 1103. The camera operation terminal 130 may have one or more dedicated hardware different from the CPU 1101, and the dedicated hardware may execute at least a part of the processing by the CPU 1101. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 1102 stores programs and the like that do not need to be changed. The RAM 1103 temporarily stores programs and data supplied from the auxiliary storage device 1104, data supplied from the outside via the communication I / F 1107, and the like. The auxiliary storage device 1104 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 1105 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the camera operation terminal 130. The operation unit 1106 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 1101 in response to an operation by the user. The CPU 1101 operates as a display control unit that controls the display unit 1105 and an operation control unit that controls the operation unit 1106. In the present embodiment, it is assumed that the display unit 1105 and the operation unit 1106 exist inside the camera operation terminal 130, but at least one of the display unit 1105 and the operation unit 1106 is another device outside the camera operation terminal 130. It may exist as.

The communication I / F 1107 is used for communication with an external device of the camera operation terminal 130. For example, when the camera operation terminal 130 is connected to an external device by wire, a communication cable is connected to the communication I / F 1107. When the camera operation terminal 130 has a function of wirelessly communicating with an external device, the communication I / F 1107 includes an antenna. The bus 1108 connects each part of the camera operation terminal 130 to transmit information.

(Workflow)
Next, a workflow executed when a plurality of cameras 112 and microphones 111 are installed in facilities such as a stadium, a stadium, and a concert hall to perform shooting will be described. FIG. 4 schematically shows the overall picture of the workflow. The processing described below is realized by controlling the camera operation terminal 130 unless otherwise specified. The process executed by the camera operation terminal 130 is realized, for example, by the CPU 1101 of the camera operation terminal 130 executing a program stored in the ROM 1102 or the RAM 1103.

In this workflow, first, before a series of processes, an operator (user) who installs or operates the image processing system 100 collects necessary information (preliminary information) before installation and before installation such as planning. Perform processing. The camera operation terminal 130 receives a setting according to the plan from a user who has made a plan such as how to install / withdraw the system based on the prior information by this pre-installation process (S401).

The prior information here includes, for example, facility information, device information, and operation information. The facility information is, for example, stadium information when the imaging location is a stadium, and includes information such as the shape of the stadium, sound, lighting, power supply, transmission environment, and three-dimensional model data of the stadium. The device information includes, for example, information on photographing equipment such as a camera, a pan head, a lens, and a microphone, and information devices such as a LAN, a PC, a server, and a cable. It should be noted that not all of this information must be entered. The camera operation terminal 130 receives input regarding the arrangement information of the camera, the pan head, and the microphone among the photographing equipment in which the device information is input. The placement information can be input using the above-mentioned three-dimensional model data of the stadium. The operation information includes information such as camera mounting form (work at height, etc.), work type (work stage), work time, number of workers, worker skill level, and the like. The operation information may include information such as a shooting target, a shooting time, camera work, and a target position (direction in which the optical axis of the camera is directed, also referred to as a gazing point). For example, the target position is changed and the constraint conditions of camera work are changed according to the type of competition such as whether the subject to be photographed is athletics or soccer that uses the entire field. The camera operation terminal 130 manages and changes a table of setting information composed of a combination of these operation information, and is used for processing various instructions and the like.

Subsequently, the user installs the equipment at the target facility. Then, each device of the image processing system 100 executes a process for confirming the operation of the system according to a command issued from the camera operation terminal 130 based on an operation from the user (S402). Since the system does not operate during the installation of the equipment, the operation management at the time of installation is performed by sequentially communicating with the camera operation terminal 130 and the camera adapter 116. After the installation of the equipment is completed, the adjustment of the installed camera 112 is executed. Adjustment of the camera 112 includes angle of view matching and color matching. This adjustment is performed for all of the installed cameras 112. The adjustment may be performed based on the user operation, or may be realized by the automatic adjustment function. Further, in the angle of view adjustment, zoom, pan, tilt, and focus adjustments are executed at the same time in parallel, and the camera operation terminal 130 acquires and saves the adjustment results. This process will be described later.

Then, before the competition or the like, the user executes pre-shooting processing that may include confirmation of the sound collected by the microphone 111 and the image captured by the camera 112, and adjustment according to the confirmation result (S403). ). After that, the camera operation terminal 130 causes the microphone 111 to collect sound and causes the camera 112 to perform shooting (S404). Note that the shooting in S404 includes sound collection using the microphone 111, but the present invention is not limited to this, and only the shooting of an image may be performed. The process of S404 is continuously executed until the setting in S402 is changed or the shooting is finished. When the setting in S402 is changed or the shooting is finished, the process proceeds to S405.

In S405, when the camera operation terminal 130 changes the setting in S402 and performs a user operation to continue shooting (YES in S405), the process proceeds to S406 and the user operation to complete the shooting is performed. If (NO in S405), the process proceeds to S407. It should be noted that this determination may not be performed based on the user operation, and the camera operation terminal 130 may autonomously determine whether or not the setting needs to be changed.

In S406, the camera operation terminal 130 changes the setting made in S402 to a new setting. The content of the change is determined based on, for example, the user operation acquired in S405. If it is necessary to stop shooting when changing the setting in S406, shooting is stopped once, and shooting is resumed after the setting is changed. If it is not necessary to stop shooting, the setting is changed while shooting is continued.

In S407, the camera operation terminal 130 issues an instruction for editing the image taken by the plurality of cameras 112 and the sound collected by the plurality of microphones 111. The process of S407 may be executed in parallel with the process of S404. For example, in the case of delivering sports competitions, concerts, etc. in real time (for example, images of competitions are delivered during competitions), data obtained by shooting in parallel without waiting for the shooting of S404 to finish. Editing of S407 is executed. On the other hand, in the case of distributing a highlight image in a sports competition after the competition, editing is executed after the shooting is completed in S405.

After the shooting and editing are completed, the equipment is withdrawn (S408). The withdrawal process is to return the facility to the state before installation, for example, by removing the equipment installed in S402. It is assumed that the above-mentioned processing is executed by the camera operation terminal 130, but the control station 170 may execute the processing in a state where the installation is completed and the normal operation of the system is possible. That is, the processing after the pre-imaging processing of S403 may be executed by the control station 170. (Camera posture adjustment)
Here, the adjustment of the camera 112 executed as the installation processing (S402) will be described. As described above, this adjustment includes the angle of view adjustment, but in the angle of view adjustment, the zoom, pan, and tilt adjustments are executed based on the display contents of the camera operation terminal 130.

FIG. 5A shows an example of a screen displayed on the camera operation terminal 130, and shows a field that is an imaging area, installation positions of a plurality of cameras, target positions of each camera, and the like from a bird's-eye view. There is. In the example of FIG. 5A, two target positions A (510) and B (520) are set, and half of the 26 cameras are set to direct the optical axis to each target position. It shows. Specifically, it is shown that the cameras of the A group are installed so as to face the target position A (510), and the cameras of the B group are installed so as to face the target position B (520). Further, the camera to be adjusted and the adjusted camera are displayed so as to be distinguishable, and in the example of FIG. 5A, it is shown that the camera 530 and the camera 550 are the cameras to be adjusted. By sharing such a screen with another camera operation terminal 130 owned by the staff in charge of the installation work, the staff can visually recognize the camera to be installed and perform the installation work without error. Can be done. Further, by sharing such a screen with the control station 170, it is possible to communicate with the administrator who operates the control station 170 and receive appropriate work instructions.

The camera operation terminal 130 communicates with the camera 530 via the wireless communication unit 302 of the camera adapter 116, and collects and displays information on the installed camera. In this system, since the sensor system 110 is connected by a daisy chain, the camera operation terminal 130 can acquire information in another sensor system from the camera adapter 116 of any of the sensor systems 110. When a plurality of camera operation terminals 130 are used, the collected data and the displayed data are shared among the plurality of camera operation terminals 130. For example, the camera operation terminal 130 that communicates with the camera 530 can collect the information of the camera 550 via the camera 540, and similarly, the camera operation terminal 130 that communicates with the camera 550 can collect the information of the camera 530. is there. In addition, the camera operation terminal 130 can collect information on the camera 540 and the camera 560. Note that data input and various judgments may be controlled by setting operation authority for a specific user. Further, information sharing may be performed via an external network (for example, a temporary local area network provided in the facility for constructing the system). According to this, even when the connection between the sensor systems 110 is interrupted, it is possible to collect the installation status of the equipment of the entire system.

FIG. 5B is a diagram for explaining posture adjustment of the camera. In the equipment installation work, the camera installation is installed in place in the facility. The installation position is a world coordinate value and is set in a space defined by the X-axis, the Y-axis, and the Z-axis based on the three-dimensional model data of the stadium prepared in advance. The installed state of the camera is a state in which the coordinate values of the horizontal X-axis, the vertical Y-axis, and the optical-axis Z-axis are fixed. Further, the posture adjustment adjusts Xθ, Yθ, and Zθ in the rotation directions of the respective axes with respect to the positions determined by the coordinate values of the horizontal X-axis, the vertical Y-axis, and the optical-axis Z-axis. In the present embodiment, the pan head 113 is configured to be adjustable on the three axes in the rotation direction. In order to adjust to the set angle of view, it is necessary to adjust the focal length of the camera as well.

FIG. 6 is a diagram showing a state of each camera displayed on the camera operation terminal 130. The horizontal axis shows the task name related to the connection / installation status of each camera, and the vertical axis shows the data for each camera. Regarding the connection status for each camera, for example, the connection status of the cable between the camera 112 and the camera adapter 116 is displayed by the information in the STATUS column next to the camera ID (camera identification information). In addition, the information in the STATUS column next to the lens ID indicates whether, for example, a cable for controlling the lens is connected to the camera adapter 116. The information in the STATUS column next to the lens ID may indicate, for example, whether or not the external sensor 114 is ready to monitor whether or not the lens is properly set. In this case, the information in the STATUS column next to the lens ID may be, for example, information indicating whether or not the external sensor 114 for monitoring the lens is connected to the camera adapter 116. In addition, information indicating the connection status of the cable for communication with the other sensor system 110 and the connection status with other external devices may be managed in the form of a table as shown in FIG. The display regarding the installation state includes not only the cable connection but also the self-check result of the connected device or the mounting state on the mounting bracket for mounting the camera 112 or the camera adapter 116 to the facility.

Further, the installation STATUS column 602 indicates whether or not the installation of the camera 112 at a predetermined position has been completed, and the attitude adjustment STATUS column 603 indicates whether or not the attitude adjustment has been completed. When the camera is ready for shooting, all the information in the line corresponding to the camera is "OK". For external devices that are not used in at least some sensor systems 110, a column indicating the state may not be prepared, and a symbol such as "-" may be described in the column indicating the state. .. For example, in the example of FIG. 6, the camera with the camera ID of Cam03 shows that the connection of the cable and the like and the installation at the position specified in advance are completed, but the posture adjustment is not completed.

In the present embodiment, the posture of the camera is adjusted by using a simulation image (estimated image) showing an image to be photographed (imaging range) at a position and posture in which the camera 112 should be installed. The estimated image used at the time of this posture adjustment can be generated by a general-purpose application. The facility's 3D data (stadium 3D model data) is required to generate the estimated image. Further, in order to generate an estimated image, parameters such as camera sensor information, coordinates indicating a position where the camera should be installed, a posture to be taken by the camera, a focal length, and coordinate information of a target position are required. Using these data, it is possible to obtain an estimated image estimation image indicating an imaging range to be taken when the camera is installed at a predetermined position in the stadium in a three-dimensional space. More specifically, the three-dimensional model data of the stadium is acquired as a point cloud by laser scanning, and this is converted into mesh data. Then, by determining the posture of the camera from the positional relationship between the position of the installed camera and the target position of the camera and setting the focal length, the image that will be taken from that camera using computer graphics technology. To generate. The same can be achieved from the CAD data of the stadium.

This estimated image is generated for each camera 112 to be installed, and the camera operation terminal 130 stores the image data in association with the camera ID. The estimated image may be stored in each camera adapter 116 as a storage location, and the storage location can be appropriately set according to the memory capacity of each device and the system configuration. The estimated image may be generated by the camera operation terminal 130, or may be generated by the camera adapter 116, the control station 170, or the like. Further, it may be generated by using another information device such as a PC which is not included in this system.

FIG. 7A is an example in which the camera operation terminal 130 displays a real-time image captured by the camera to be adjusted during the camera posture adjustment. Further, FIG. 7B shows an example of the estimated image of the camera to be adjusted. As the posture adjustment work, the real-time image (FIG. 7 (A)) is matched with the estimated image (FIG. 7 (B)). That is, the user who is the worker adjusts the pan, tilt, and focal length (zoom) of the camera 112 by using the pan head 113 so that the image displayed as the real-time image is substantially equal to the estimated image. .. As a method of adjustment, it is easier to work by aiming at the feature points of the stadium (for example, the corners of the spectators' seats and the structures peculiar to the competition such as the goal post). Further, guide information such as an arrow or a guide line indicating a direction to be adjusted may be superimposed and displayed on the estimated image, and the posture may be adjusted using this guide information.

As the screen of the camera operation terminal 130 at the time of posture adjustment, for example, as shown in FIG. 7C, a real-time image (display of R) and an estimated image (display of S) are overlaid (superimposed) and displayed. Further, if the angles of view are significantly different during the adjustment, it may be difficult to adjust in the superimposed display. Therefore, the real-time image and the estimated image may be displayed side by side as shown in FIG. 7 (D). Further, the display of FIGS. 7 (C) and 7 (D) may be switched by a user operation or automatically. For example, the degree of agreement (similarity) is determined by image analysis of the estimated image and the real-time image, and when the angles of view are significantly different, they are displayed as shown in FIG. 7D, and the angles of view are within a predetermined range. You may switch to FIG. 7 (C) when approaching the inside. Which display layout (display form) to use, the superimposed display or the adjacent display, may be fixed as a device setting or may be changed by the user.

Subsequently, the flow of processing at the time of adjusting the posture of the camera will be described with reference to FIG. The process of FIG. 8 is realized, for example, by the CPU 1101 of the camera operation terminal 130 executing a program stored in the ROM 1102 or the RAM 1103. This process is started, for example, by designating a camera for adjustment work from the user from the image shown in FIG. 5 (A).

First, in S801, the CPU 1101 reads out the camera ID that is the target of the adjustment work. Next, the CPU 1101 executes reading of the estimated image corresponding to the read camera ID (S802). Similarly, the CPU 1101 executes reading of the real-time image corresponding to the camera ID (S803).

In S804, the CPU 1101 reads out the set values of the display layout of the estimated image and the real-time image in the camera operation terminal 130. This set value includes a setting of displaying the estimated image and the real-time image side by side or superimposing them as described above.

In S805, the CPU 1101 displays the estimated image and the real-time image on the same screen according to the display layout indicated by the read set value. In S806, it is determined whether or not the instruction for changing the display layout from the user of the camera operation terminal 130 has been accepted. When the display layout change instruction is received (YES in S806), the process returns to S805, and the estimated image and the real-time image are displayed according to the changed display layout. If the change instruction is not received (NO in S806), the process proceeds to S807.

The user adjusts the posture of the camera 112 while referring to the real-time image and the estimated image displayed on the camera operation terminal 130. Further, the installation work is executed while switching the display contents and the display layout of the camera operation terminal 130. Then, when the posture adjustment is completed, the camera operation terminal 130 performs an operation to that effect. In S807, the CPU 1101 determines whether or not a user operation indicating that the posture adjustment has been completed has been performed. When the operation to the effect that the posture adjustment is completed is performed (YES in S807), the posture adjustment process ends. Further, when the operation is performed, it is reflected in the posture adjustment STATUS column 603 of FIG. Further, when the posture adjustment process is completed, the display may be switched to the screen shown in FIG. 5 (A).

As described above, in the present embodiment, in order to support the installation work of each camera, an estimated image to be taken at the position and posture in which each camera should be installed is generated in advance. Then, when adjusting the posture of the camera, an estimated image generated in advance is acquired and displayed together with the real-time image being captured by the installed camera. By adjusting the posture of the camera while comparing the displayed estimated image with the real-time image, the user can efficiently adjust the posture of the camera without installing a marker or the like in the imaging area.

<Second embodiment>
In the first embodiment, the posture adjustment of the camera is performed by giving the state in which the corresponding camera has been installed and adjusted as various parameters and using the generated simulation image (estimated image). However, in facilities such as stadiums, since the field of the imaging area is wide, depending on the angle of view, for example, only the lawn and the line of the competition event may be photographed. In such a case, it was sometimes difficult to know which position in the field the angle of view should be adjusted. In the present embodiment, a method for improving the workability of the adjustment work will be described even in such a case.

FIG. 9 is an explanatory diagram for explaining the adjustment work of the present embodiment. Wide-angle side real-time image (Fig. 9 (A)) and wide-angle side estimated image (Fig. 9 (B)) of the camera to be adjusted, real-time image being adjusted to the final angle of view (Fig. 9 (C)), adjustment An example of an estimated image of the angle of view to be taken (FIG. 9 (D)) is shown.

In the present embodiment, instead of displaying the estimated image of the angle of view to be adjusted from the beginning and matching the real-time image, the estimated image on the wide-angle side is first displayed and the real-time image on the wide-angle side is matched. .. If, for example, a stadium structure (audience seats, etc.) is set to fit in the angle of view of the image on the wide-angle side, it is easy to perform the work of matching the real-time image with the estimated image (posture adjustment). Then, if the posture can be adjusted on the wide-angle side, the user can change the focal length and zoom to match the angle of view to be adjusted. It is not necessary to strictly adjust the posture on the wide-angle side, and the posture may be adjusted accurately at the final angle of view.

Further, after changing (zooming) the focal length, an image indicating the direction to be adjusted (for example, the arrow 901) may be superimposed on the real-time image. Further, the focal length to be set may be displayed and the user may adjust the focal length. Furthermore, in the above example, an example of displaying an estimated image at two focal lengths has been described, but a simulation image in which the angle of view is changed stepwise from the wide-angle side to the focal length to be adjusted is prepared. , The display may be switched. That is, it is possible to selectively switch and display an image to be displayed together with a real-time image from a plurality of estimated images having different focal lengths.

The flow of the adjustment process at the time of installing the equipment of the present embodiment will be described with reference to FIG. The process of FIG. 10 is realized, for example, by the CPU 1101 of the camera operation terminal 130 executing a program stored in the ROM 1102 or the RAM 1103. Similar to the process of FIG. 8, this process is started by, for example, designating a camera for adjustment work from the user from the image shown in FIG. 5 (A).

First, in S1001, the camera ID that is the target of the adjustment work is read out. Next, as the adjustment work step 1, reading of the estimated image on the wide-angle side corresponding to the camera ID is executed (S1002). Then, similarly, the real-time image corresponding to the camera ID is read out (S1003).

In S1004, the read estimated image on the wide-angle side and the real-time image are displayed on the same screen. Here, as described above, the user adjusts the posture of the camera 112 so as to match the real-time image with the estimated image. In S1005, it is determined whether or not the posture adjustment in the adjustment work step 1 is completed. The determination is determined by whether or not a completion operation has been performed by the user. When the operation indicating that the posture adjustment in step 1 is completed is performed (YES in S1005), the process proceeds to S1006.

In S1006, the CPU 1101 reads out the focal length information of the target camera ID. The value of the focal length information changed by the user's operation is sequentially transmitted from the camera adapter 116 and held by the camera operation terminal 130. The CPU 1101 uses this focal length information to determine whether or not to switch the image to be displayed for the adjustment work. This determination is executed by storing the value of the focal length to be switched as a threshold value in advance and monitoring whether or not the set value is exceeded. When the focal length information exceeds the set value (YES in S1006), the image for the angle of view adjustment work is switched in the adjustment work step 2.

As the adjustment work step 2, the CPU 1101 reads out the estimated image of the angle of view to be installed corresponding to the camera ID (S1007). Similarly, the CPU 1101 executes reading of the real-time image corresponding to the camera ID (S1008). Then, in S1009, the CPU 1101 displays both the read estimated image and the real-time image. As described above, the user adjusts the posture of the camera 112 so as to match the real-time image with the estimated image. In S1010, the CPU 1101 determines whether or not the user operation to the effect that the posture adjustment in step 2 has been completed has been performed. When the operation indicating that the posture adjustment is completed is accepted (YES in S1010), the posture adjustment process ends.

In the above description, the description of the display layout of the estimated image and the real-time image has been omitted, but as in the first embodiment, the superimposed display and the adjacent display may be appropriately switched.

Further, in the above description, it has been described that the process proceeds to S1006 after waiting for the user's operation to confirm that the posture adjustment of step 1 has been completed by the user in S1005, but the determination of S1005 may be omitted. .. That is, even if it is determined whether or not the focal length value exceeds the threshold value, and if it exceeds the threshold value, it is considered that the posture adjustment in step 1 is completed, and the posture adjustment work in step 2 after S1007 is performed. Good.

Further, the determination process of S1006 may be omitted, and the process of S1007 or later may be proceeded when the operation to the effect that the adjustment work of step 1 is completed by the user is performed in S1005.

As described above, in the present embodiment, in addition to the estimated image of the angle of view to be finally adjusted, an estimated image having a wider angle of view is prepared, and first, the estimated image on the wide angle side is compared with the real-time image. Adjust the posture of the camera with. Then, when the posture adjustment is almost completed, the display is switched to the estimated image of the angle of view to be finally adjusted, and the posture is adjusted so as to match the real-time image. As a result, even the posture adjustment of the camera installed at a position where the target is difficult to understand in the imaging range of the final angle of view can be efficiently performed.

<Other Embodiments>
In the above description, an example has been described in which the user determines that the posture adjustment of the camera 112 has been completed and the posture adjustment is completed by the user's completion operation. However, whether or not the posture adjustment is completed is automatically determined. It may be determined to. For example, in the camera operation terminal 130, the degree of coincidence (similarity) between the estimated image and the real-time image is calculated using a known image matching technique, and it is determined whether or not the posture adjustment is completed based on the calculated value. It may be. Further, the posture adjustment of the camera 112 may be adjusted automatically instead of manually. In that case as well, the camera operation terminal 130 calculates the degree of coincidence between the estimated image and the real-time image, and transmits the posture adjustment instruction to the camera adapter 116 so that the estimated image and the real-time image match. The camera adapter 116 can automatically adjust the posture of the camera 112 by controlling the pan head control unit 333 based on the received instruction. Further, after the automatic adjustment, the user may make further manual adjustment for fine adjustment.

Further, in the above description, an example of setting two target positions and adjusting the posture so that the 26 cameras 112 point the optical axis to any of them has been described, but the number of target positions is not limited to this. For example, it is applicable even when the target positions of all the cameras 112 are set to different positions.

Further, in the above description, a method of displaying an estimated image or a real-time image on the camera operation terminal 130 to support the posture adjustment of the camera 112 has been described, but other devices may be used. For example, the above screen may be displayed on the camera adapter 116 or the control station 170 to assist the user who performs the adjustment work. In that case, for example, the control station 170 may transmit the camera ID to be adjusted to the camera operation terminal 130 or the like that stores the estimated image, and acquire and display the corresponding estimated image. In addition, the administrator operating the control station 170 may instruct the user who is adjusting the camera 112 to make specific adjustments by telephone or the like, or remotely via the camera adapter 116. The posture of the camera 112 may be adjusted by operation.

Further, in the above description, the virtual viewpoint image generation system generated based on the images captured from different directions by a plurality of cameras and the designated virtual viewpoint has been described, but it is also applied to other camera systems. It is possible.

For example, it can be applied to a system in which images captured from different directions by a plurality of cameras are appropriately switched and displayed, and a surveillance camera system in which a plurality of cameras capture different areas. In addition, it is not limited to a system that captures images with multiple cameras, but also for use cases where the installation position and posture of the cameras are adjusted to predetermined positions and postures when imaging with one camera. Applicable.

As described above, in each of the above-described embodiments, the image indicating the imaging range to be imaged at the position and posture in which the imaging device should be installed is captured at the position and posture in which the imaging device is actually installed. The image is displayed in a comparable manner. As a result, the user who installs the image pickup device can easily and efficiently adjust the posture of the image pickup device by adjusting the posture of the image pickup device while comparing the images.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image processing system 110 Sensor system 116 Camera adapter 120 Switching hub 130 Camera operation terminal 140 Image computing server 150 Image generator 160 Virtual camera operation terminal 170 Control station

Claims (17)

It is a method of supporting the adjustment of the posture of each imaging device in a system in which an imaging region is imaged from different directions by a plurality of imaging devices.
A first acquisition step of acquiring a first image which is an image to be captured by the image pickup device at a position and a posture in which one of the plurality of image pickup devices should be installed.
A second acquisition step of acquiring a second image which is an image captured by the image pickup device at the position and orientation in which the image pickup device is currently installed, and
A display step of displaying the second image acquired in the second acquisition step together with the first image acquired in the first acquisition step.
A method characterized by having. The method according to claim 1, wherein in the display step, the first image and the second image are displayed side by side. The method according to claim 1, wherein in the display step, the first image and the second image are superimposed and displayed. In the display step, switching between a display layout in which the first image and the second image are displayed side by side and a display layout in which the first image and the second image are superimposed and displayed are switched. 1. The method according to claim 1. In the first acquisition step, a plurality of images having different focal lengths are acquired as the first image, and the images are acquired.
The method according to any one of claims 1 to 4, wherein in the display step, an image to be displayed together with the second image is selectively switched from the plurality of images having different focal lengths. .. The method according to claim 5, wherein in the display step, the images having the longest focal length are displayed in order from the plurality of images having different focal lengths. The method according to claim 5 or 6, wherein in the display step, an image to be displayed together with the second image is switched from the plurality of images having different focal lengths based on a user's operation. The fifth or sixth aspect of the present invention, wherein the display step switches an image to be displayed together with the second image from the plurality of images having different focal lengths based on a change in the focal length of the imaging device. the method of. The generation step of generating the first image and
The method according to any one of claims 1 to 8, further comprising a storage step of storing the first image generated in the generation step. In the generation step, the first image is generated based on at least the shape data indicating the three-dimensional shape of the facility where the imaging is performed and the information indicating the position and orientation in which the imaging device should be installed. 9. The method according to claim 9. In the generation step, the first image is generated for each of the plurality of imaging devices.
The method according to claim 9 or 10, wherein in the storage step, the first image is stored in association with identification information for identifying each of the plurality of imaging devices. The method according to any one of claims 9 to 11, wherein in the first acquisition step, the first image stored in the storage step is acquired. In the display step, the first image is displayed in accordance with a user operation of designating an image pickup device for adjusting the posture by displaying an image showing the image pickup region and the installation positions of the plurality of image pickup devices from a bird's-eye view. The method according to any one of claims 1 to 12, wherein the display is switched to the display of the image and the second image. One of claims 1 to 13, wherein the control step of controlling the posture of the image pickup apparatus to be adjusted based on the result of image analysis of the first image and the second image is provided. The method described in paragraph 1. One of claims 1 to 14, wherein the system is a system that generates a virtual viewpoint image based on a plurality of images taken by the plurality of imaging devices and a designated virtual viewpoint. The method described in paragraph 1. An information processing device that supports adjustment of the posture of each imaging device in a system that images an imaging region from different directions by a plurality of imaging devices.
The first image, which is an image to be captured by the imaging device at the position and orientation in which one of the plurality of imaging devices should be installed, and the position and orientation in which the imaging device is currently installed. A second image, which is an image captured by the image pickup device, and an acquisition means for acquiring the image.
A display control means for displaying the second image together with the first image acquired by the acquisition means, and
An information processing device characterized by having. A program for causing a computer to perform the method according to any one of claims 1 to 15.

Images