JP7462085B2 - Information processing device, method, and program - Google Patents

Description

The present invention relates to technology for installing and operating a system that uses multiple devices.

A technology that captures images of a subject from multiple directions synchronously using multiple imaging devices installed in different positions, and generates a virtual viewpoint image using the multiple captured images obtained by the imaging, is attracting attention. The virtual viewpoint image allows the user to view images from various viewpoints different from the positions of the imaging devices, and therefore provides the user with a high sense of realism. However, since multiple imaging devices are used to generate a virtual viewpoint image, it is necessary to execute a large number of complicated processing procedures compared to imaging using a single imaging device. Patent Document 1 describes a UI (user interface) that allows the user to easily select the viewpoint from which to observe the virtual viewpoint image.

JP 2014-215828 A

The technology described in Patent Document 1 can simplify the processing required to view a prepared virtual viewpoint image, but does not mention the processing procedure for preparing a virtual viewpoint image, nor does it consider a situation in which a virtual viewpoint image cannot be generated. As a result, the installation, removal, and operation of multiple imaging devices and other devices that make up a multi-camera system for generating virtual viewpoint images remains complicated, and there is an issue that the burden on workers (users) cannot be reduced.

The present invention provides user assistance technology for installing and operating a multi-camera system.

An information processing device according to one aspect of the present invention has an acquisition means for acquiring information indicating the status of multiple devices regarding the progress of multiple processes included in a workflow relating to work in a system including multiple devices, the work being performed between installation of the multiple devices and before imaging of a subject for which three-dimensional shape data is to be generated begins, and a display control means for causing a display device to display a screen including the multiple processes included in the workflow and information indicating the progress of each of the multiple processes.

The present invention can provide appropriate support to users in the installation and operation of a multi-camera system.

FIG. 1 is a diagram illustrating an example of the configuration of an information display system. FIG. 2 is a diagram showing the arrangement of an imaging device. FIG. 1 illustrates an example of the configuration of an imaging device. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. FIG. 4 illustrates an example of workflow definition information. FIG. 11 is a diagram illustrating an example of workflow management information. FIG. 11 is a diagram illustrating an example of a process flow of a workflow UI display. FIG. 11 illustrates an example of a workflow UI. FIG. 13 illustrates an example of a warning UI. FIG. 11 illustrates an example of a workflow UI. FIG. 4 illustrates an example of workflow definition information. 11 is a diagram illustrating an example of a processing flow when an abnormality occurrence notification is obtained. FIG. 11 illustrates an example of a workflow UI.

Below, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment is merely one example of how the present invention can be implemented, and the device configuration and processing flow described may be changed as appropriate. For example, the functional blocks and hardware blocks that make up the device are not limited to the described embodiment, and some blocks may be omitted or other blocks may be added, or some blocks may be combined with other blocks to form a single block. Also, some processing steps may be omitted or other steps may be added, and multiple processing steps may be executed by a single processing step.

In the embodiment described here, a multi-camera system in which multiple imaging devices are installed to generate a virtual viewpoint image is described as an example. Then, in order to guide a user such as a worker to perform appropriate processing when setting up, removing, or operating such a multi-camera system, information such as a guide to the processing that the user should perform and a warning when an incorrect processing is performed is displayed to the user in an easy-to-understand manner. This makes it possible to prevent the user from performing inappropriate processing such as making a mistake in the procedure when setting up, removing, or operating the multi-camera system. Note that the multi-camera system may be used to generate a virtual viewpoint image, but it may also be a system used for purposes other than generating a virtual viewpoint image. Here, in this embodiment, "image" includes a still image and a moving image (video). In addition, the following discussion can be applied to any system that provides a predetermined service using a large number of devices by replacing the equipment such as the imaging devices described below with any equipment used in any system.

Figure 1 shows an example of the configuration of an information display system according to this embodiment. This information display system includes an imaging device group 100 including multiple imaging devices 101-1 to 101-m, a HUB 120, a virtual viewpoint image generating device 140, and an information processing device 160. Each device is connected to each other so that they can communicate with each other via a transmission cable for transmitting images and control information, for example. The transmission cable is, for example, GbE (Gigabit Ethernet) or 10 GbE conforming to the Ethernet (registered trademark) standard, but is not limited to these, and other types of cables may be used. In addition, each device may perform wireless communication, wired communication via a cable may not be performed, or wireless communication and wired communication may be performed in parallel.

As described above, the imaging device group 100 includes multiple (m) imaging devices (imaging devices 101-1 to 101-m). Hereinafter, when it is not necessary to specify a specific imaging device, these imaging devices will be referred to as imaging devices 101-x. Note that the value added after 101, such as m or x, is an integer. Two imaging devices 101-x (for example, imaging device 101-1 and imaging device 101-2) that are adjacent to each other can be directly connected to each other via a transmission cable. Note that the two imaging devices 101-x that are directly connected do not need to be physically adjacent, and another imaging device 101-x may be placed between these two imaging devices 101-x. For example, the multiple imaging devices may be classified into multiple groups each having an imaging target (viewpoint) set at a different point, and in this case, the imaging devices 101-x in a group are directly connected, and the imaging devices 101-x in different groups may be configured not to be directly connected. The imaging device 101-x transmits the captured image and the processing result of the request received from the information processing device 160 to the virtual viewpoint image generating device 140 and the information processing device 160. The imaging device 101-2 is connected to the imaging device 101-1 as shown in FIG. 1, and is also directly connected to the imaging device 101-3 (not shown) adjacent to the imaging device 101-1 on the opposite side. In this way, the imaging device 101-k can be directly connected to the imaging device 101-(k-1) and the imaging device 101-(k+1), which are two adjacent imaging devices. Note that this does not apply to the imaging device (e.g., the imaging device 101-1) located at the end of this connection and the imaging device (e.g., the imaging device 101-n) directly connected to the HUB 120. That is, these imaging devices are not connected to two other imaging devices, but to one other imaging device. In this way, the multiple imaging devices 101-x are daisy-chained. In addition, the multiple imaging devices 101-x may be grouped according to the imaging target, for example, as described above, and may be connected in a daisy chain for each group. In this case, the imaging device 101-x belonging to the first group may be configured not to be connected to the imaging device 101-x of the second group different from the first group without going through the HUB 120. In the example of FIG. 1, there are two such groups, and separate daisy chains are configured for the group from the imaging device 101-1 to the imaging device 101-n and the group from the imaging device 101-(n+1) to the imaging device 101-m. Note that this connection form is merely an example, and imaging devices of different groups may be connected by one daisy chain. In addition, in FIG. 1, an example in which the number of groups is two is shown, but all imaging devices may be configured in one daisy chain without grouping, or three or more groups may be provided. Furthermore, the mutual connections between the multiple imaging devices 101-x do not have to be in a daisy chain format, but may be in any format, such as a ring type, mesh type, star type, bus type, tree type, or a combination of these. Furthermore, grouping may be performed based on any other criteria, such as, for example, imaging devices that are physically close to each other belong to the same group.

In one example, the imaging devices 101-x are arranged to surround a stadium such as a soccer field or a specific subject. FIG. 2 is a diagram showing an example of the arrangement of the imaging devices 101-x. In the example of FIG. 2, multiple imaging devices 101-x are arranged so that all or part of the range of a stadium such as a soccer field is captured.

Figure 3 shows an example of the configuration of the imaging device 101-x. The imaging device 101-x has a camera 301, a lens 302, a camera platform 303, and an image processing device 304. The camera 301 is, for example, a digital camera, and works with the attached lens 302 to capture images and generate image data. The camera 301 is also connected to the image processing device 304, for example, by an SDI (Serial Digital Interface) cable. The camera 301 receives control signals and synchronization signals from the image processing device 304, and transmits image data obtained by capturing images to the image processing device 304. The camera 301 captures images based on the synchronization signal received from the image processing device 304. The synchronization signal is a signal for causing multiple imaging devices 101-x to capture images at the same timing, and this allows all imaging devices 101-x to capture images at the same or approximately the same timing. The cable between the camera 301 and the image processing device 304 may be a cable other than an SDI cable as long as it can transmit a control signal, a synchronization signal, and an image signal. The camera platform 303 adjusts the angle of the camera 301. The camera platform 303 is connected to the image processing device 304, for example, by a serial cable. The camera platform 303 receives angle data of the pan direction and the tilt direction from the image processing device 304, and adjusts the angle of the camera 301 based on the received angle data. The cable between the camera platform 303 and the image processing device 304 may be a cable other than a serial cable as long as it can transmit angle data. The image processing device 304 acquires a control signal from the information processing device 160 and outputs a control signal to the camera 301, the lens 302, and the camera platform 303. The image processing device 304 transmits the captured image received from the camera 301 to the virtual viewpoint image generating device 140 via a transmission cable. Furthermore, the image processing device 304 transmits a control response signal to the information processing device 160 indicating whether processing of the output control signal was completed normally or abnormally.

Returning to FIG. 1, the virtual viewpoint image generating device 140 is connected to the HUB 120 and is configured to be able to communicate with the imaging device 101-x via the HUB 120, and accumulates images captured by the imaging device 101-x. The virtual viewpoint image generating device 140 may generate three-dimensional shape data of the subject and background based on the accumulated images, for example, and prepare for generating a virtual viewpoint image. When virtual viewpoint information is input by operating a user terminal, the virtual viewpoint image generating device 140 generates a virtual viewpoint image corresponding to the input virtual viewpoint information using images captured by the multiple imaging devices 101-x. Here, the virtual viewpoint information includes at least position information and direction information, and is information that specifies from which position and in which direction the subject is observed. The position information is information that indicates a relative position with respect to a predetermined position such as the center of the stadium (for example, a position in the forward/backward direction, left/right direction, and left/right direction when the predetermined position is used as a reference). The direction information is information indicating the direction from the predetermined position (for example, the angle from each axis in a three-dimensional Cartesian coordinate system with the predetermined direction as the reference direction and the forward/backward, left/right, and up/down directions as axes). The virtual viewpoint image generating device 140 is, for example, a server device. The virtual viewpoint image generating device 140 has, for example, a database function for storing images captured by each of the multiple imaging devices 101-x, and a function for managing information on the positions at which the imaging devices 101-x are placed and the directions in which the images are captured. The virtual viewpoint image generating device 140 has an image processing function for generating a virtual viewpoint image using the stored images based on the information managed by the managed information. The database function of the virtual viewpoint image generating device 140 can hold in advance, as background image data, images of a scene of a competition venue without a subject, such as a scene of the competition venue before the start of the competition.

The information processing device 160 is connected to the HUB 120 and is configured to be able to communicate with the imaging device 101-x and the virtual viewpoint image generating device 140 via the HUB 120. The information processing device 160 receives captured images captured by the imaging device 101-x and status information of the imaging device 101-x, and executes processing for displaying them on the display screen. The information processing device 160 also receives status information of the virtual viewpoint image generating device 140, and executes processing for displaying them on the display screen. A person watching this system can constantly monitor the status of the imaging device group 100 and the virtual viewpoint image generating device 140 using the captured images and the like displayed on the display screen by the processing by the information processing device 160. The information processing device 160 can also execute processing for displaying a workflow showing the tasks to be performed from the installation of the system to the capture of images, and displaying information on the progress of the tasks, such as completion, incompletion, or currently being processed. The information processing device 160 can also execute processing for displaying information showing whether the completed tasks are in a normal state or an abnormal state. The display contents will be described later.

The HUB 120 relays communications between the imaging device group 100, the virtual viewpoint image generating device 140, and the information processing device 160. The HUB 120, for example, transfers captured images output from a plurality of imaging devices 101-x to the virtual viewpoint image generating device 140 and the information processing device 160. This allows the virtual viewpoint image generating device 140 and the information processing device 160 to acquire the same captured image.

Next, an example of the configuration of the information processing device 160 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the information processing device 160. The information processing device 160 is configured to include, for example, a controller unit 401, an operation unit 410, and a display device 420 as its hardware configuration. Note that the information processing device 160 may be configured to not include, for example, the operation unit 410 or the display device 420. For example, the information processing device 160 may be configured to receive a command indicating a user operation from an operation device such as an external keyboard, execute processing, and output screen data for displaying the screen on an external display or the like. In addition, the information processing device 160 may be a server or the like arranged on a network, and may receive operation commands from one or more operation terminals via the network and transmit screen data to one or more displays via the network.

The controller unit 401 includes, for example, a CPU 402, a ROM 403, a RAM 404, a HDD 405, an operation unit I/F 406, a display unit I/F 407, and a communication I/F 408. Note that HDD is an acronym for Hard Disk Drive, and I/F is an abbreviation for interface. The CPU 402, ROM 403, RAM 404, the operation unit I/F 406, the display unit I/F 407, and the communication I/F 408 are connected to each other via a system bus 409.

The CPU 402 starts an OS (operating system) by a boot program stored in the ROM 403. The CPU 402 then executes application programs stored in the HDD 405 or the like on this OS. The CPU 402 realizes various processes by executing the application programs. The RAM 404 is used as a working area for the CPU 402. The HDD 405 stores the above-mentioned application programs and the like.

The operation unit I/F 406 is an interface that receives information indicating user operations from the operation unit 410 and controls user operations by outputting information to the operation unit 410 as necessary. The operation unit I/F 406 acquires information indicating user operations by a system monitor received by the operation unit 410 from the operation unit 410 and outputs the information to the CPU 402. The operation unit 410 is configured by devices that receive user operations, such as a mouse or keyboard. However, without being limited to this, any device that can receive user operations, such as a sensor that detects the user's gestures or line of sight, or a sensor that detects the movement of an operator, may be used as the operation unit 410.

The display unit I/F 407 is an interface that outputs image data to be displayed on the display unit 420 to the display unit 420 and controls the screen display by receiving information from the display unit 420 as necessary. The display unit 420 may be a computer display. The operation unit 410 and the display unit 420 may be configured as a single device such as a touch panel display. In this embodiment, information is presented by screen display, but information may be presented in various ways such as audio output or vibration output. In this case, the display unit I/F 407 may output audio data or vibration pattern information to be output to an external speaker or vibrator.

The communication I/F 408 is an interface for connecting to and communicating with other devices, for example, via a transmission cable. The communication I/F 408 inputs and outputs information through communication with external devices (for example, a user terminal, or the HUB 120 and information processing device 160 in FIG. 1). The communication I/F 408 can be a wired communication I/F, but may also be a wireless communication I/F.

FIG. 5 shows an example of the functional configuration of the information processing device 160. The information processing device 160 has a data recording unit 501, a user input unit 502, a control unit 503, a communication unit 504, a workflow management unit 505, a UI image generation unit 506, a display unit 507, and an anomaly detection unit 508. These functional units are logically connected to each other, and can transmit and receive data to each other under the control of the control unit 503. The data recording unit 501 records data in the HDD 405, and the control unit 503, the workflow management unit 505, and the UI image generation unit 506 perform processing by the processing of the CPU 402. The user input unit 502 acquires information indicating user input through the operation unit I/F 406, and the communication unit 504 communicates with an external device through the communication I/F 408. The display unit 507 displays information on the display device 420 via the display unit I/F 407.

The data recording unit 501 holds workflow definition information indicating the procedures and progress from the installation of the system to the imaging. The workflow definition information includes at least one of the following information: "ID", "category", "process", and "request". Here, "category" indicates an item that groups procedures from installation to imaging, "process" indicates a processing item that belongs to a category, and "request" indicates details of the processing corresponding to the process, and indicates the processing content that the information processing device 160 requests the imaging device 101-x. Furthermore, "ID" is identification information that is uniquely assigned to a combination of these items, and is data corresponding to the execution order of the process. For example, "ID" can be a value that is assigned "1" to the process that should be executed first, and increases in order to "2", "3", ....

Figure 6 shows an example of workflow definition information. The workflow definition information in this example defines the content and sequence of 14 processes from installation of the imaging system to imaging. Each process is defined as a "process," and one "category" includes a group of one or more "processes." In addition, for each "process," a "request" is defined as the control content that the information processing device 160 requests of the imaging device 101-x. As shown in Figure 6, the workflow definition information in this example defines the categories of "device connection," "synchronization," "angle of view adjustment," "exposure adjustment," and "calibration (CAL)."

For the device connection category, a group of processes including "camera", "lens", and "head" is defined. These processes are processes for checking whether a camera, lens, and head are connected to each of the imaging devices 101-x. In this process, a process for simply checking that the device exists and is connected may be executed, but other processes such as checking the type of device for each imaging device 101-x may be executed. For example, when multiple types of cameras are used in the imaging system, a checking process such as "a camera of type A is connected to the imaging device 101-1" may be executed. In addition, the ID of the device may be checked for each imaging device 101-x. For example, a unique serial ID may be defined for each camera used in the imaging system, and a check may be performed to check that a camera with ID=001 is connected to the imaging device 101-1. Since the virtual viewpoint image imaging system handles a huge number of devices, management of the equipment is very important. In response to this, the information processing device 160 can check the number of devices and whether the desired equipment is connected through these processes.

For the synchronization category, a "synchronization" process is defined. To generate a virtual viewpoint image, the cameras 301 of all the imaging devices 101-x need to capture images in a synchronized manner. For this reason, the information processing device 160 uses this process to request the image processing devices 304 of all the imaging devices 101-x to output a synchronization signal, and causes them to output a synchronization signal for synchronizing the multiple cameras 301.

For the angle of view adjustment category, a group of processes including "platform angle," "zoom," and "focus" is defined. To generate a virtual viewpoint image, multiple cameras 301 need to capture the same area without blur. For this reason, in these processes, the information processing device 160 outputs a request signal to specify the camera orientation by adjusting the angle of the platform, to determine the camera's imaging range by adjusting the zoom, and to eliminate blur by adjusting the focus.

For the exposure adjustment category, a group of processes including "ISO", "Iris", "Shutter", and "ND" is defined. To generate a virtual viewpoint image, the images captured by the cameras 301 of all image capture devices 101-x must have the same brightness. Therefore, the information processing device 160 outputs a signal requesting exposure adjustment through these processes.

For the calibration (CAL) category, a group of processes including "imaging", "calculation", and "setting" is defined. In order to generate a virtual viewpoint image, it is necessary to manage the positions and orientations of the cameras 301 of all the imaging devices 101-x, and the type of lens distortion they have. This management data is hereinafter referred to as "CAL parameters". The CAL parameters are identified by capturing images at the same time using all the cameras 301, analyzing the captured images, extracting the positions of feature points, and performing calculations. The CAL parameters can be calculated using a method of mapping three-dimensional points expressed in a world coordinate system onto an image. The information processing device 160 uses the calculated CAL parameters to set the image processing devices 304 of all the imaging devices 101-x.

In this example, 14 processes and 5 categories are defined, but the number and contents of the processes and categories can be set arbitrarily and are not particularly limited. For example, the above-mentioned processing is related to a system that generates a virtual viewpoint image, but a process and category necessary for a multi-camera system that is not related to a virtual viewpoint image to fulfill its purpose may be defined. In addition, it is not necessary to use multiple imaging devices, and processes and categories may be defined for the installation and operation of a system that uses a large number of devices. In addition, the information processing device 160 does not need to include the data recording unit 501, and the data recording unit 501 may be realized by a recording device external to the information processing device 160. In addition, the data recording unit 501 may be implemented in each imaging device 101-x with respect to a portion that holds information about the imaging device 101-x. The data recording unit 501 may be implemented in a distributed manner in multiple devices, such as by being implemented separately for each function in the imaging device 101-x and the information processing device 160.

Returning to FIG. 5, the user input unit 502 receives information on user operations received by the operation unit 410 and outputs it to the control unit 503.

The control unit 503 outputs control instructions to each unit of the information processing device 160. For example, the control unit 503 determines the request content to the imaging device 101-x based on operation information from the user input unit 502, and outputs the request content to the communication unit 504. The control unit 503 also transmits the request content to the workflow management unit 505 in a similar manner in order to display the progress of the system toward imaging on a UI (user interface) screen. The control unit 503 may also receive request response information indicating the processing result of the request output to the imaging device 101-x via the communication unit 504, and output the request response information to the workflow management unit 505.

The communication unit 504 acquires request information from the control unit 503, converts the request information into a signal format that can be transmitted to other devices, and transmits it to each of the imaging devices 101-x. The communication unit 504 also receives signals from each of the imaging devices 101-x, extracts request response information contained in the signals, and outputs the extracted request response information to the control unit 503. The communication unit 504 also acquires device information from the signals received from each of the imaging devices 101-x, and outputs the device information to the abnormality detection unit 508. Here, the device information may include information such as at least one of the setting values of the imaging device, the external light sensor value, and the gyro sensor value.

The workflow management unit 505 obtains request information and request response information from the control unit 503, updates the workflow management information according to that information, and then transmits the updated workflow management information to the UI image generation unit 506. FIG. 7 shows an example of workflow management information. The workflow management information includes information indicating which of four types of states each process is in: not running, running, normal completion, or abnormal completion. State transitions will be described later.

The UI image generation unit 506 generates a UI image based on the workflow management information obtained from the workflow management unit 505 and outputs it to the display unit 507.

The display unit 507 overwrites the display image output from the UI image generation unit 506 onto a frame buffer (not shown). The display unit 507 then reads out the display image stored in the frame buffer at a predetermined refresh rate and displays it on the display device 420.

The anomaly detection unit 508 acquires device information from the communication unit 504 and monitors whether any device is operating abnormally. When the anomaly detection unit 508 detects an abnormality in the device, it outputs an anomaly notification to the workflow management unit 505.

(Example of process flow during system installation and operation)
Next, a description will be given of an example of a flow of a process executed by the information processing device 160. In this process, the information processing device 160 displays a UI screen for assisting a user in thoroughly executing a procedure from installation of a number of imaging devices to start of imaging in order to generate a virtual viewpoint image.

Figure 8 shows an example of the flow of this process. This process can be realized, for example, by CPU 402 executing a program stored in HDD 405. Note that the flow of the process will be explained below using the functional configuration of Figure 5.

In this process, the workflow management unit 505 first reads workflow definition information from the data recording unit 501 (S801) and generates workflow management information. For example, the workflow management unit 505 extracts category and process information from the workflow definition information and manages the state of each process by setting it to "not executed". After that, the workflow management unit 505 outputs the created workflow management information to the UI image generation unit 506, and the UI image generation unit 506 generates and displays a workflow UI based on the workflow management information (S802). Note that the workflow definition information may be manually input by a user operation. Also, workflow management information corresponding to multiple systems may be defined in advance and recorded in the data recording unit 501, and the workflow management unit 505 may read out workflow definition information corresponding to a system selected by the user from the multiple systems. Also, the workflow management unit 505 may read out workflow management information corresponding to a specific application from the data recording unit 501 in response to the start of the application, for example. Furthermore, if the information processing device 160 is dedicated to a specific system, the workflow management unit 505 may read specific workflow management information from the data recording unit 501 when the information processing device 160 is started up, without accepting any user operations.

9 is an example of a workflow UI display at the time when workflow management information is generated based on the workflow definition information of FIG. 6. This workflow UI is initially displayed in S802. In the display of FIG. 9, the upper part displays a list of category information included in the workflow. In the lower part, information on processes included in the workflow is displayed in a list, and the information on these processes is displayed below the category to which the process belongs. Between the category information and the process information, a status bar 901 that displays the progress state of the category is arranged, and below the process information, a status bar 902 that displays the progress state of the process is arranged. These status bars express four states: not executed/running/normally completed/abnormally completed. Note that, for example, in the initial display, the workflow UI may display only the category, and in response to the selection of a category, the process belonging to that category may be displayed. That is, for example, the user may select a category that does not transition to normal completion while remaining in execution to grasp the situation, and may not need to know the details of a category that has been normally completed. In such a case, the display may be simplified to make it easier for the user to roughly grasp the state. In addition, when displaying the progress status, an area showing the progress status such as a status bar may be provided separately from the area showing the category or process name as shown in FIG. 9, but is not limited to this. For example, the color/size of the text/frame showing the category or process may be changed or a blinking display may be used, and as long as the display allows the user to determine the status, a status bar display may not be used.

Returning to FIG. 8, the workflow management unit 505 waits for request information to arrive from the control unit 503 (S803). Here, as an example, a case will be described in which the progress status is normal up to the lens process of device connection based on the example of workflow management information shown in FIG. 7. When the workflow management unit 505 acquires request information from the control unit 503 (YES in S803), it checks whether there is a process other than "normal end" ("not executed", "running", "abnormal end") among the processes that should be completed at that time (S804). Then, when the workflow management unit 505 determines that there is a process that is not in a normal end state (YES in S804), it displays a warning UI (S805) and transitions to a request waiting state (S803). For example, when the workflow management unit 505 acquires request information for synchronization setting from the control unit 503, it checks the current progress by the workflow management information. When the workflow management unit 505 detects that the head process is "not executed," it determines that a process other than one that has ended normally exists, and requests the UI image generation unit 506 to display a warning UI.

Figure 10 shows an example of a warning UI display. By visually checking this warning UI, the operator of the information processing device 160 can recognize that there is an incomplete process. This enables the worker to execute the steps of the workflow without missing anything. Note that the display content of the warning UI is not limited to the content of Figure 10, so long as it enables the user to recognize that there is an incomplete process. In addition, this notification may be made by a sound such as audio guidance or a beep, and does not have to be made by a UI display.

On the other hand, if all of the processes that should have been completed by then have ended normally (NO in S804), the workflow management unit 505 changes the state of the process corresponding to the request information to a running state. Then, the workflow management unit 505 requests the UI image generation unit 506 to display that this process is being executed (S806). After that, the workflow management unit 505 waits to receive request response information from the control unit 503 (S807). When the workflow management unit 505 receives the request response information (YES in S807), it determines whether the request of S803 ended normally based on the request response information (S808). If the workflow management unit 505 determines that the request ended normally (YES in S808), it changes the state information of the process corresponding to that request to a normal end state. Then, the workflow management unit 505 requests the UI image generation unit 506 to display that the process has ended normally (S809). On the other hand, if the workflow management unit 505 determines that the request in S803 has ended abnormally (NO in S808), it changes the information on the state of the process corresponding to that request to an abnormal end state. In this case, the workflow management unit 505 requests the UI image generation unit 506 to display that the process has "ended abnormally" (S810). Thereafter, the workflow management unit 505 determines whether or not the display of the category should be changed (S811). If the workflow management unit 505 determines that the display should be changed (YES in S811), it requests the UI image generation unit 506 to display that the category corresponding to the request has ended normally (S812). On the other hand, if the workflow management unit 505 determines that the display of the category should not be changed (NO in S811), it returns to a standby state for request information.

11(A) to 11(D) show an example of a workflow UI display. For the sake of simplicity, only an example of the UI display of the device connection category is illustrated in FIG. 11(A) to 11(D). FIG. 11(A) shows a progress status in which the lens process has been normally completed and the head process has not been executed. When the progress status is not executed, the status bar is displayed in white. FIG. 11(B) shows an example of a display when the head process transitions to a progress status in execution after a head connection confirmation request is acquired from the control unit 503. When the progress status is in execution, the status bar is displayed, for example, in a first predetermined color (for example, gray) and blinks. Next, FIG. 11(C) shows an example of a display when the progress status of the head process transitions to abnormal termination in response to the request response information acquired from the control unit 503 indicating that the process has terminated abnormally. When the progress status becomes abnormal termination, the status bar is displayed, for example, in a second predetermined color (for example, red). This display allows the operator of the information processing device 160 to recognize that the head connection has not been established, and to take appropriate measures such as checking whether the cable between the image processing device 304 and the head 303 is connected and whether the head is turned on. FIG. 11D shows an example of a display in which the progress status of the head process transitions to normal end in response to the request response information acquired from the control unit 503 indicating that the process has ended normally. When the progress status is normal end, the status bar is displayed in the same color as the status bar of the process that has ended normally, such as the camera. In addition, when the workflow management unit 505 confirms that all processes belonging to the category are in a normal end state, the UI image generation unit 506 may change the state of the status bar of the category to indicate a normal end state. Note that the display of the status bar is not limited to the above, and the progress status may be displayed in a distinguishable manner by at least one of the color, size, pattern, and animation of the bar (icon/area indicating the progress status).

The operator of the information processing device 160 can check the progress of the workflow UI to confirm the appropriate process, and execute a request corresponding to the process by operating the operation unit 410. In one example, each key on the keyboard of the operation unit 410 is associated with a separate process, and the user can execute a request corresponding to the process by pressing the key corresponding to the process to be selected. Note that the execution of this request is not limited to key input on the keyboard, and for example, the display portion of each process on the workflow UI may be a button, and the request for that process may be executed by the user pressing the button by operating the mouse. In addition, the information processing device 160 may have a voice recognition function and execute this request by voice input using the operator's voice. In addition to this, any method capable of receiving an instruction to execute a request may be used.

The progress of the process may be arbitrarily changed by user input. For example, in the above description, the exposure adjustment is performed by the information processing device 160 in the workflow, but it may be set directly in the camera 301, in which case there is no need for setting from the information processing device 160. In this case, the exposure adjustment process can be skipped by transitioning all progress of the exposure adjustment process to normal completion by user input.

As described above, in the above-mentioned process, the information processing device 160 displays the process order by means of the workflow UI, and also displays the progress so that it can be easily distinguished, thereby efficiently supporting the user in installing and operating the system. This workflow UI allows the user to easily understand the next process to be performed, and prevents incorrect operations or misidentification of the next process to be performed. Therefore, by executing requests corresponding to the processes in accordance with the displayed process order, the user can easily perform the complex procedures from installation to imaging without hesitation and without omissions, thereby ensuring reliable operation of the system.

(Example of processing flow when an abnormality occurs)
Next, an example will be described in which a UI is displayed for easily and completely executing the processes required for recovery when an abnormality occurs during the process from installation to image capture.

For this process, the workflow definition information stored in the data recording unit 501 includes the above information as well as information defined for the processes affected by the abnormality. Figure 12 shows an example of workflow definition information in this case. In this example, 14 processes and 5 categories are shown, as in Figure 6, and in addition to the information in Figure 6, it includes abnormality information for each process that defines the processing to be performed again when an abnormality occurs. First, the details of each abnormality information will be explained.

A camera disconnection, lens disconnection, or tripod head disconnection refers to a disconnection between the image processing device 304 in the imaging device 101-x and each device. These abnormalities are collectively referred to as "disconnection abnormalities" below. Disconnection abnormalities affect processes in the device connection category but do not affect other processes, and are therefore defined only for each process in the device connection category. A synchronization abnormality refers to a state in which the synchronization signal sent from the image processing device 304 in the imaging device 101-x does not reach the camera 301, or the information in the synchronization signal is incorrect. A synchronization abnormality affects the synchronization process but does not affect other processes, and is therefore defined only for the synchronization category.

Camera movement, zoom lens movement, focus lens movement, and aircraft vibration are generally abnormalities in the angle of view. In order to generate a virtual viewpoint image, it is necessary for multiple imaging devices 101-x to capture the same spatial area. For this reason, it is necessary to determine the positions, camera orientations, and focal lengths of the multiple imaging devices 101-x in advance by calculation. In other words, the angle of view adjustment refers to setting the imaging device 101-x to a previously determined installation state, and if the installation state deviates from the state once installed, it is detected as an abnormality. Camera movement refers to the movement of the camera 301 due to an external force such as the attitude of the camera 301 being adjusted by a human hand, or an object such as a ball colliding with the camera 301 or the camera platform 303. Zoom lens movement refers to the focal length changing due to the movement of the zoom lens, and the imaging range changing. Focus lens movement refers to the focus being shifted due to the movement of the focus lens. Aircraft vibration refers to minute movements of the camera 301 caused by improper installation of the installation jig for the pan head 303 or improper installation of the installation jig between the pan head 303 and the camera 301. These angle of view deviation abnormalities affect each process in the angle of view adjustment category and the calibration category, so they are defined for both of these categories.

ISO value fluctuation, iris value fluctuation, shutter value fluctuation, ND filter value fluctuation, and external light change are generally exposure anomalies. Because a virtual viewpoint image is created by combining multiple captured images, if there are cameras 301 with different exposures, the quality of the virtual viewpoint image will decrease. ISO value fluctuation, iris value fluctuation, shutter value fluctuation, and ND filter value fluctuation refer to a state in which the setting values of at least some of the cameras 301 are different from the setting values of the other cameras 301. External light change refers to a change in the brightness of the surrounding environment due to changes in sunlight caused by the opening and closing of the roof of the venue or the movement of clouds. Exposure anomalies other than iris value fluctuation only affect the exposure adjustment process, and are therefore defined only for the exposure adjustment process. On the other hand, iris value fluctuation affects the depth of field and therefore affects the calibration process, and is therefore defined for both of these categories.

The definition of the abnormality information is also applied to the abnormality detection process of the abnormality detection unit 508. That is, the abnormality detection unit 508 monitors the connection status between the image processing device 304 and each device in the imaging device 101-x, and when it detects a disconnection, it outputs an abnormality notification of a disconnection abnormality to the workflow management unit 505. The abnormality detection unit 508 also monitors the state of the synchronization signal output from the image processing device 304 to the camera 301 in the imaging device 101-x, and when it detects that the synchronization signal is not normal, it outputs an abnormality notification of a synchronization abnormality to the workflow management unit 505. The abnormality detection unit 508 also acquires the value of the gyro sensor mounted on the camera 301 of the imaging device 101-x, and monitors the change in the attitude of the camera 301. When the abnormality detection unit 508 detects a change in attitude, it outputs an abnormality notification of the camera movement to the workflow management unit 505. Furthermore, the anomaly detection unit 508 acquires the exposure setting value of the camera 301 of the imaging device 101-x, and when it detects, for example, a change in the ISO value, it outputs an anomaly notification of the ISO value change to the workflow management unit 505. Note that an anomaly detection method different from the above method may be used as long as it is possible to identify the anomaly. Furthermore, the definition of the anomaly information can be set arbitrarily, and is not limited to the above example.

For example, as described above, processes that are affected by an abnormality are defined, and when an abnormality occurs, the process defined in response to the abnormality is executed again, making it possible to recover from an abnormal state to a normal state. Therefore, when an abnormality is detected, the information processing device 160 executes processing to generate and display a UI that notifies the user of the category or process processing corresponding to the abnormality and prompts the user to execute processing related to the process.

Figure 13 shows an example of the flow of processing executed when the workflow management unit 505 acquires an abnormality occurrence notification for such processing. The workflow management unit 505 monitors whether an abnormality notification has been received from the abnormality detection unit 508, for example, in parallel with the processing of Figure 8 or while the system is providing a service (S1301). When the workflow management unit 505 acquires an abnormality notification (YES in S1301), it checks the abnormality information described in the workflow definition information and checks the process affected by the acquired abnormality (S1302). After that, the workflow management unit 505 changes the progress state of the process affected by the acquired abnormality to "not executed" (S1303). The updated workflow management information is sent to the UI image generation unit 506. The UI image generation unit 506 generates a UI image based on the workflow management information acquired from the workflow management unit 505, and the display unit 507 acquires the UI image and displays it on the display device 420. Note that, although an example has been described in which an abnormality notification is obtained from the abnormality detection unit 508, the workflow management unit 505 may also detect an abnormality in accordance with the control content obtained from the control unit 503.

14(A) and 14(B) are examples of workflow UI display when an abnormality occurs. The screen in FIG. 14(A) displays that the calculation process in the calibration category has been completed normally, and that all processing will be completed by executing the setting process. In this situation, assume that an abnormality in camera movement occurs due to a factor such as an operator bumping into the camera. In this case, the processes affected by the abnormality in camera movement are processes belonging to the angle of view adjustment category and the calibration category. Therefore, the workflow management unit 505 changes the progress status of these influencing processes to "incomplete" and updates the UI display. FIG. 14(B) is an example of the UI display after this update. This allows the operator of the information processing device 160 to recognize that the process in the angle of view adjustment category and the process that were completed in the calibration category have become incomplete. Then, the operator of the information processing device 160 can easily and thoroughly execute the processes required to recover from the abnormal state by re-executing requests corresponding to these incomplete processes.

In this example, the processing required to recover from an abnormality that has occurred is defined in advance in the workflow definition information, and when an abnormality occurs, the state of the processing corresponding to the abnormality is changed to an incomplete state, but this is not limited to this. For example, the selection of the processing required to recover from an abnormality may be performed using AI (Artificial Intelligence) machine learning. By using AI, the accuracy of the recovery processing improves as the number of learning times increases, and the robust performance of the system can be gradually improved. Also, a method other than these methods may be used as long as it is possible to identify the processing required for recovery.

The above-mentioned process is effective in a system having a large number of devices, and an information processing device configured to be able to communicate with the large number of devices can display workflow information related to the work required to operate the system on a UI as described above. The information processing device then communicates with those devices involved in each process in the workflow, among the large number of devices, to check whether the devices are in a state in which they can perform the functions of the system, and displays that state on the UI display. This makes it easy to understand the order of work in a system having a large number of devices, and allows installation and operation to proceed without error.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

100: imaging device group, 301: camera, 302: lens, 303: camera platform, 304: image processing device, 160: information display device, 402: CPU, 403: ROM, 404: RAM, 405: HDD, 406: operation unit I/F, 407: display unit I/F, 408: communication unit, 410: operation unit, 420: display device, 501: data recording unit, 502: user input unit, 503: control unit, 504: communication unit, 505: workflow management unit, 506: UI image generation unit, 507: display unit, 508: anomaly detection unit

Claims (17)

an acquisition means for acquiring information indicating the status of a plurality of devices with respect to a progress status of a plurality of processes included in a workflow relating to a task in a system configured to include a plurality of devices, the task being executed from installation of the plurality of devices until before imaging of a subject for which three-dimensional shape data is to be generated is started;
a display control means for causing a display device to display a screen including the plurality of processes included in the workflow and information indicating a progress status of each of the plurality of processes ;
13. An information processing device comprising: The method further comprises: receiving a user operation related to the process to be executed;
the acquiring means acquires information indicating states of the plurality of devices for the process corresponding to the user operation.
2. The information processing apparatus according to claim 1, the display control means causes the display device to display a warning when the process corresponding to the user operation is not normally completed, the process being expected to be completed at the time of executing the process.
3. The information processing apparatus according to claim 2. the screen includes information indicating a category including one or more of the processes and a progress status of the category;
4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer. the plurality of devices includes a plurality of imaging devices;
The process relates to at least one of connection of the plurality of imaging devices, synchronization of the plurality of imaging devices, adjustment of the angle of view of the images captured by the plurality of imaging devices, adjustment of the exposure of the images captured by the plurality of imaging devices, and calibration of the imaging devices.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer. the display control means causes the display device to display at least one of "not executed,""beingexecuted,""normallycompleted," and "abnormally completed" as the progress status of each of the plurality of processes;
6. The information processing apparatus according to claim 1, the display control means causes the display device to display the screen in which an area showing the progress status is prepared separately from information about the process;
7. The information processing apparatus according to claim 1, the display control means displays the progress status in a distinguishable manner by at least one of a color, a size, a pattern, and an animation of an area in which the progress status is displayed;
8. The information processing apparatus according to claim 1, The acquisition means acquires a notification that an abnormality has occurred for each of the plurality of devices,
when the display control means receives the notification after the process affected by the abnormality is completed, the display control means changes the information indicating the progress of the process on the screen from information indicating that the process is completed to information indicating that the process is not completed.
9. The information processing apparatus according to claim 1, The method further includes: receiving a user operation for changing the progress status.
10. The information processing apparatus according to claim 1, The system is a system for generating a virtual viewpoint image.
11. The information processing apparatus according to claim 1, The system is a system for generating three-dimensional shape data of a subject .
12. The information processing apparatus according to claim 1, A method executed by an information processing device, comprising:
an acquisition step of acquiring information indicating the status of a plurality of devices with respect to a progress status of a plurality of processes included in a workflow relating to a task in a system configured to include a plurality of devices and executed from installation of the plurality of devices until start of imaging of a subject for which three-dimensional shape data is to be generated;
a display control step of displaying, on a display device, a screen including the plurality of processes included in the workflow and information indicating a progress status of each of the plurality of processes;
The method according to claim 1, further comprising: The method further includes the step of accepting a user operation related to the process to be executed,
In the acquiring step, information indicating states of the plurality of devices is acquired for the process corresponding to the user operation.
14. The method of claim 13. In the display control step, when the process corresponding to the user operation is not normally completed, the process is displayed on the display device.
15. The method of claim 14. the screen includes information indicating a category including one or more of the processes and a progress status of the category;
16. The method according to any one of claims 13 to 15. A program for causing a computer to function as each of the means possessed by an information processing device according to any one of claims 1 to 12.

Images