JP2022012398A - Information processor, information processing method, and program - Google Patents

Abstract

To determine the position and direction of an imaging apparatus so that an appropriate virtual viewpoint image fitted to a specified virtual viewpoint can be generated.SOLUTION: An arrangement determination device 105 specifies the position and direction of a virtual viewpoint corresponding to a virtual viewpoint image generated on the basis of a plurality of images obtained by imaging an imaging object area from different positions from each other by a plurality of cameras 101. The arrangement determination device 105 determines the position and direction of the camera 101 on the basis of the position and direction of the specified virtual viewpoint.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for generating a virtual viewpoint image using a plurality of photographing devices.

There is a technique of installing a plurality of photographing devices at different positions to perform synchronous photography, and using a plurality of captured images obtained by the imaging to generate a virtual viewpoint image in which the viewpoint can be arbitrarily changed. When generating a virtual viewpoint image of a moving image, the position and orientation of the virtual viewpoint corresponding to the virtual viewpoint image can be changed over time. However, the image quality of the virtual viewpoint image may deteriorate depending on the position and orientation of the virtual viewpoint. For example, when the position and orientation of the photographing device and the position and orientation of the virtual viewpoint are significantly different, the appearance from the virtual viewpoint cannot be sufficiently reproduced based on the captured image, and the image quality of the virtual viewpoint image is deteriorated.

In Patent Document 1, the positions and orientations of a plurality of installed photographing devices are specified, and an indicator indicating how to move the virtual viewpoint to generate a high-quality virtual viewpoint image is displayed to the user. Is disclosed. By operating the virtual viewpoint while looking at the display, the user can suppress deterioration of the image quality of the virtual viewpoint image.

Japanese Unexamined Patent Publication No. 2019-197348

However, when an attempt is made to generate a high-quality virtual viewpoint image by the method described in Patent Document 1, the range in which the position of the virtual viewpoint and the line-of-sight direction can be set is limited. For example, even if the position and line-of-sight direction of the virtual viewpoint desired by the user are predetermined, the position and line-of-sight direction of the virtual viewpoint are changed in order to suppress the deterioration of the image quality of the virtual viewpoint image. It is possible that it will end up.

The present invention has been made in view of the above problems, and an object of the present invention is to determine the position and orientation of a photographing device so that an appropriate virtual viewpoint image corresponding to a specified virtual viewpoint can be generated.

In order to solve the above-mentioned problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, the information processing device specifies the position and orientation of the virtual viewpoint corresponding to the virtual viewpoint image generated based on the plurality of images obtained by shooting the shooting target area from different positions by the plurality of shooting devices. Specific means and
It has a determination means for determining the position and orientation of the imaging device included in the plurality of imaging devices based on the position and orientation of the virtual viewpoint specified by the specific means.

According to the present invention, the position and orientation of the photographing device can be determined so that an appropriate virtual viewpoint image corresponding to the specified virtual viewpoint can be generated.

It is a block diagram which shows the structure of an image processing system. It is a figure which shows the installation example of a photographing apparatus. It is a figure which shows the hardware configuration example of the arrangement determination apparatus. It is a figure which shows the functional configuration example of the arrangement determination apparatus. It is a figure which shows the example of the request according to the importance of a region. It is a flowchart for demonstrating an example of processing of an arrangement determination apparatus. It is a figure for demonstrating an example of the shooting range and area importance of a virtual camera. It is a figure which shows the example of the arrangement information which shows the camera arrangement. It is a figure which shows the functional configuration example of the arrangement determination apparatus. It is a flowchart for demonstrating an example of processing of an arrangement determination apparatus. It is a figure for demonstrating an example of an object area and an important area.

[System configuration]
FIG. 1 is a block diagram showing a configuration example of the image processing system 100 according to the present embodiment. As shown in FIG. 1, the image processing system 100 has a plurality of cameras 101, a hub 102, a control device 103, an image generation device 104, and an arrangement determination device 105.

The image processing system 100 is a system that generates a plurality of images taken by a plurality of cameras 101 and a virtual viewpoint image representing a scene seen from a designated virtual viewpoint based on a designated virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

As shown in FIG. 2, the plurality of cameras 101 are installed at different positions so as to surround the shooting target area 201 in which the object 202 is located, and a part or the whole of the shooting target area 201 is synchronized from different directions. Take a picture. The plurality of cameras 101 may not be installed over the entire circumference of the photographing target area 201, or may be installed only in a part of the periphery of the photographing target area 201. Further, the number of cameras 101 is not limited to the example shown in the figure. The camera 101 is, for example, a digital camera capable of capturing still images and moving images, and the plurality of cameras 101 may include cameras having different functions such as a telephoto camera and a wide-angle camera.

In this embodiment, a case where a camera 101 having an independent housing and capable of shooting from a single viewpoint is used as a shooting device for shooting the shooting target area 201 from different positions will be described. However, the present invention is not limited to this, and two or more photographing devices may be configured in the same housing. For example, a single camera having a plurality of lens groups and a plurality of sensors and capable of shooting from a plurality of viewpoints may be installed as a plurality of shooting devices.

The shooting target area 201 in the present embodiment is a shooting studio where shooting for producing a virtual viewpoint image is performed. However, the shooting target area 201 is not limited to this, and may be, for example, a stadium where a sports competition is performed or a stage where a performance is performed.

A plurality of images taken by the plurality of cameras 101 are sent to the image generator 104 through the hub 102. The control device 103 is communicably connected to the plurality of cameras 101 via the hub 102, and controls the shooting of the plurality of cameras 101 and manages the arrangement status of the plurality of cameras 101.

The image generation device 104 acquires and stores a plurality of images (multiple viewpoint images) based on images taken by the plurality of cameras 101. The image included in the multi-viewpoint image may be a captured image, or may be an image obtained by performing image processing such as extraction of a predetermined region on the captured image. Further, the image generation device 104 receives an input according to a user operation using a controller or the like for designating a virtual viewpoint, and specifies a virtual viewpoint corresponding to the virtual viewpoint image to be generated. Then, the image generation device 104 generates a virtual viewpoint image representing a scene seen from the specified virtual viewpoint based on the plurality of viewpoint images.

The image generation device 104 generates a virtual viewpoint image by, for example, the following method. First, a plurality of images are acquired by taking pictures from different directions with the plurality of cameras 101. Next, a foreground image in which a foreground area corresponding to a predetermined object 202 such as a person is extracted and a background image in which a background area other than the foreground area is extracted are acquired from the plurality of images. The background is, for example, a studio set or field. Further, a foreground model representing the three-dimensional shape of the predetermined object 202 and texture data for coloring the foreground model are generated based on the foreground image. Further, texture data for coloring the background model representing the three-dimensional shape of the background such as a studio or a stadium is generated based on the background image. Then, a virtual viewpoint image is generated by mapping the texture data to the foreground model and the background model and performing rendering according to the virtual viewpoint indicated by the viewpoint information. However, the method of generating the virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of the captured image without using the three-dimensional model can be used.

The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the direction of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. ..

In this embodiment, the description will be given using the concept of a virtual camera. The virtual camera is a virtual camera different from the plurality of cameras 101 actually installed around the shooting target area 201, and is a concept for expediently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. Is. That is, the virtual viewpoint image can be regarded as an image taken from a virtual viewpoint set in the virtual space associated with the shooting target area 201. Then, the position and orientation of the viewpoint in the virtual shooting can be expressed as the position and posture of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image simulating the captured image obtained by the camera, assuming that the camera exists at the position of the virtual viewpoint set in the space. Further, in the present embodiment, the information indicating the content of the transition of the virtual viewpoint over time is referred to as a virtual camera path. However, it is not essential to use the concept of a virtual camera in order to realize the configuration of this embodiment. That is, at least the information indicating a specific position in the space and the information indicating the direction are set, and the virtual viewpoint image may be generated according to the set information.

The arrangement determination device 105 determines the arrangement of a plurality of cameras 101 based on the virtual camera path set in the image generation device 104 so that the virtual viewpoint image corresponding to the virtual camera path can be generated with high image quality. Details of the arrangement determination of the camera 101 will be described later. The arrangement determination device 105 is connected to the control device 103 and the image generation device 104 by wire or wirelessly.

The configuration of the image processing system 100 is not limited to that shown in FIG. For example, any two or all of the control device 103, the image generation device 104, and the arrangement determination device 105 may be integrally configured. Further, the foreground image and the background image described above may be extracted by the image generation device 104 from the captured image, or may be extracted by another device (for example, the camera 101). When the camera 101 extracts the foreground image and the background image, each of the plurality of cameras 101 may extract both the foreground image and the background image, or some of the plurality of cameras 101 may extract the foreground image. It may be extracted and the background image may be extracted by some other cameras 101. Further, the plurality of cameras 101 may include a camera 101 that does not extract either a foreground image or a background image.

[Hardware configuration]
FIG. 3 is a diagram showing a hardware configuration example of the arrangement determination device 105 as an example of the information processing device included in the image processing system 100. The configurations of the control device 103 and the image generation device 104 as an example of the information processing device are also the same as those of the arrangement determination device 105 described below. The arrangement determination device 105 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, an operation unit 305, a display unit 306, and a communication unit 307.

The CPU 301 realizes each function of the arrangement determination device 105 by executing arithmetic processing and various programs using the control programs stored in the ROM 302 and the HDD 304. The arrangement determination device 105 may have one or more dedicated hardware different from the CPU 301, and the dedicated hardware may execute at least a part of the processing by the CPU 301. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 302 has a storage area for storing a boot program, a control program, parameters set in each part of the image generation device 104, and the like. The RAM 303 is used as a matching storage area for the main memory, work area, etc. of the CPU 301. The HDD 304 stores various data and programs. In addition, other auxiliary storage devices such as SSD may be used instead of HDD 304.

The operation unit 305 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 301 in response to an operation by the user. The display unit 306 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the arrangement determination device 105. The CPU 301 operates as a display control unit that controls the display unit 306. The communication unit 307 communicates with an external device such as the control device 103 and the image generation device 104. The communication by the communication unit 307 may be performed via a wired network such as a LAN, or may be performed via a wireless network.

In the present embodiment, it is assumed that the display unit 306 and the operation unit 305 exist inside the arrangement determination device 105, but at least one of the display unit 306 and the operation unit 305 exists as another device outside the arrangement determination device 105. You may be doing it.

[Functional configuration of placement determination device]
FIG. 4 is a block diagram showing a functional configuration example of the arrangement determination device 105 in the present embodiment. The path acquisition unit 401 acquires a virtual camera path indicating changes in the position, orientation, and angle of view of the virtual viewpoint over time, and transmits the virtual camera path to the shooting range calculation unit 402. The virtual camera path acquired by the path acquisition unit 401 is generated by the image generation device 104 to generate a virtual viewpoint image. However, the virtual camera path acquired by the path acquisition unit 401 is not limited to the one generated by the image generation device 104, and may be one generated by another processing system or an external virtual camera path generation device. Further, the virtual camera path may be generated from the simulation result of the motion of the object 202 assumed in advance, and is not limited thereto.

The path acquisition unit 401 acquires one or more virtual camera paths as a reference for the arrangement determination device 105 to determine the camera arrangement. From the acquired virtual camera path, the position and orientation of the virtual viewpoint at each of the plurality of time points are specified. The path acquisition unit 401 may acquire all the virtual camera paths generated and stored in the past by the image generation device 104, or may specify and acquire only a part of the generated virtual camera paths. .. Further, the virtual camera path corresponding to the virtual viewpoint image whose image quality is desired to be improved may be selected and acquired. Further, the path acquisition unit 401 may acquire a virtual camera path that has been rejected because the image quality of the virtual viewpoint image is insufficient, and sets the virtual camera path that is the target of image quality improvement as the required image quality. You may also get it. Further, the path acquisition unit 401 may acquire a virtual camera path corresponding to a virtual viewpoint image to be generated in the future. The virtual camera path acquired by the path acquisition unit 401 may include a plurality of virtual camera paths corresponding to the same shooting period, or a virtual camera that has been time-processed such as slow, fast forward, and rewind. The path may be included. Further, the path acquisition unit 401 may acquire the virtual camera path generated by the connected image generation device 104 in real time.

The shooting range calculation unit 402 calculates the shooting range of the virtual camera for each frame indicated by the virtual camera path received from the path acquisition unit 401 in the shooting target area 201 indicated by the information acquired from the arrangement condition storage unit 404, and the importance level. It is transmitted to the calculation unit 403. Specifically, the shooting range of the virtual camera is based on the parameters indicating the position, posture, and angle of view of the virtual camera included in the virtual camera path at each time point corresponding to each frame constituting the moving image of the virtual viewpoint image. (The range included in the field of view of the virtual viewpoint) is calculated.

The calculated shooting range may be a three-dimensional region or a two-dimensional region on a plane (for example, a floor surface with z = 0). Further, when the shooting range is calculated as a two-dimensional region, the criterion for determining whether or not each position on the plane is included in the shooting range may differ depending on the size and movement of the object 202. For example, when the object 202 is a person, even if it is determined that the shooting range includes a two-dimensional position in which a region whose height from the floor is 0 to 2 m can be shot by a virtual camera. good.

Further, the range in which the virtual camera can shoot at a certain resolution or higher may be set as the shooting range. That is, even if the position is included in the field of view of the virtual camera, the position where the virtual camera cannot shoot at a certain resolution or higher (for example, the position where the distance from the virtual camera is equal to or higher than a predetermined value) is included in the shooting range of the virtual camera. It may be determined that it cannot be done. The above-mentioned constant resolution may be determined according to the shooting content or the like, or may be specified by the user.

Further, the shooting range of the virtual camera may be represented by the coordinate values of the points surrounding the shooting range, and information indicating whether or not each block (partial area) included in the shooting target area 201 is within the shooting range. May be represented by. The division unit of the block may be determined to have a predetermined size (for example, 10 cm or 1 m) in the real space, or the imaging target area 201 may be determined to be equally divided into a predetermined number. Further, the division unit of the block may be changed according to the size of the shooting target area 201, the number of a plurality of cameras 101, and the shooting content, and is not limited thereto. Further, an image showing a shooting range may be output with respect to the shooting target area 201.

The importance calculation unit 403 calculates the importance of each area in the shooting target area 201 based on the shooting range of the virtual camera received from the shooting range calculation unit 402. As a method of calculating the importance, for example, it is determined for each frame whether each area is included in the shooting range of the virtual camera, and a certain area is included in the shooting range of the virtual camera according to the cumulative time (cumulative number of frames). The importance of the area is calculated. The longer the cumulative time, the higher the importance is set, so that the important area that enters the shooting range of the virtual camera for a long time is specified. The importance may be expressed by a numerical value of the cumulative time, or may be expressed by a plurality of levels such as high importance / medium importance / low importance depending on the cumulative time.

Further, the information used for calculating the importance is not limited to the cumulative time indicating the length of the period during which the area falls within the shooting range of the virtual camera. For example, the attention level of the shooting scene may be used to calculate the importance of the area. That is, the importance may be weighted in units of virtual camera paths or frames according to the degree of attention of the scene shot by the virtual camera. Specifically, when a certain area is included in the shooting range of the virtual camera in the frame corresponding to the scene with high attention, it is more than when it is included in the shooting range in the frame corresponding to the scene with low attention. However, the amount of addition of the importance of the area may be increased.

Also, if the virtual viewpoint image generated based on the virtual camera path is popular, it can be determined that a similar virtual camera path is important, so a virtual camera corresponding to such a popular virtual camera path is used. You may increase the addition amount of the importance of the area included in the shooting range of. Further, for example, the importance of each area may be corrected based on the position and direction of the illumination that illuminates the image target area 201. The importance weighting is not limited to adjusting the amount of importance added. For example, the importance value or importance level of the area satisfying a predetermined condition may be set to the highest value or the highest level, the importance value of the area satisfying a predetermined condition may be added by a certain value, or the importance level may be set to a constant level. It may be raised and is not limited to these.

The arrangement condition storage unit 404 stores information regarding the arrangement conditions of the plurality of cameras 101. For example, information on a venue such as a shooting studio, information on a shooting target area 201, information on a structure in which a camera 101 can be installed in the studio and its position, the number of cameras 101 that can be placed, and spec information on the camera 101 are stored. Will be done. If there is an already installed camera 101, the position and setting information of the camera 101, the information of the camera type such as the fixed camera or the pan head camera, and the like are stored as the information of the current arrangement of the cameras 101.

The request acquisition unit 405 acquires a request regarding the shooting status of the area according to the importance of the area. Specifically, a request regarding the positional relationship of the plurality of cameras 101 with respect to a region of a specific importance (for example, an upper limit of the angle between the plurality of cameras 101 viewed from the region) is acquired. Further, a request regarding the resolution related to the shooting of a region of a specific importance (for example, the lower limit of the resolution corresponding to the size of the region in the image captured by the camera 101) is acquired. The request acquisition unit 405 may acquire a request related to the image quality of the virtual viewpoint image, and the content of the request is not limited to these.

The request acquisition unit 405 may store the upper limit camera-to-camera angle and the lower limit resolution corresponding to the area importance as a table in advance, and may refer to the table. An example of the referenced table is shown in FIG. As shown in FIG. 5, the higher the area importance, the smaller the upper limit camera-to-camera angle so that the model accuracy of the object 202 becomes higher. Further, the higher the region importance is, the higher the lower limit resolution is, so that the resolution of the texture mapped to the model of the object 202 is higher. Further, the request acquisition unit 405 may acquire request information according to a numerical value or the like specified by the operator via the operation unit 305. Further, the image quality requirement value may be determined for a specific virtual camera path and a specific frame.

The arrangement determination unit 406 is based on the importance of each area received from the importance calculation unit 403, the condition acquired from the arrangement condition storage unit 404, and the request acquired from the request acquisition unit 405, and the arrangement determination unit 406 of the plurality of cameras 101. Determine the placement. Specifically, if there is information on the installed camera 101, the preconditions, setting information and position information of the camera 101 that can be arranged, information on the shooting target area 201, and the like are acquired from the arrangement condition storage unit 404. To. Further, the request acquisition unit 405 acquires the request according to the importance, and obtains the upper limit camera-to-camera angle and the lower limit resolution required for each area. Then, the arrangement of the plurality of cameras 101 is determined so as to satisfy the requirements of the upper limit camera-to-camera angle and the lower limit resolution according to the importance of each area. The arrangement information indicating the determined camera arrangement is output as, for example, an image via the display unit 306 and the communication unit 307. The arrangement determination unit 406 determines not only the position of the camera 101 but also the focal length corresponding to the direction (shooting direction) of the camera and the angle of view of the camera so that the requirements according to the importance of each area are satisfied. You may. The arrangement information in the present embodiment may include information indicating the position, orientation, and focal length of the camera 101.

When there are a plurality of patterns of camera arrangements that satisfy the requirements for each area, a plurality of patterns of camera arrangements may be derived, and the optimum camera arrangement may be determined among them. The optimum camera arrangement may be, for example, the one in which the number of required cameras 101 is the smallest, or the one in which the amount of change from the arrangement of the installed cameras 101 is the smallest. Further, the camera arrangement to be adopted may be selected from a plurality of camera arrangement candidates satisfying the requirements according to the operation of the operator via the operation unit 305. If the number of cameras 101 that can be arranged cannot meet the requirements for all the areas in the shooting target area 201, the requirements for the less important areas may be changed. In that case, the request change instruction may be received from the operator via the operation unit 305.

[Operation flow]
Next, the processing flow of the arrangement determination device 105 will be described with reference to FIG. The process shown in FIG. 6 is realized by the CPU 301 of the arrangement determination device 105 expanding the program stored in the ROM 302 into the RAM 303 and executing the program. It should be noted that at least a part of the processing shown in FIG. 6 may be realized by one or a plurality of dedicated hardware different from the CPU 301. The same applies to the process shown in FIG. 10 described later. The process shown in FIG. 6 starts when a virtual camera path is specified by the user via the image processing system 100 and the arrangement determination device 105 receives an instruction from the operator to determine the camera arrangement via the operation unit 305. Will be done. This virtual camera path may be specified before shooting by a plurality of cameras 101, or when a virtual viewpoint image is prototyped based on a shot image acquired by cameras 101 installed at predetermined positions. It may be specified. The start timing of the process shown in FIG. 6 is not limited to the above.

In S601, the path acquisition unit 401 acquires a virtual camera path as a reference when determining the camera arrangement. In S602, the shooting range calculation unit 402 acquires parameters indicating the position, posture, and angle of view of the virtual camera for each frame from the virtual camera path acquired in S601, and the shooting range of the virtual camera in the shooting target area 201. Is calculated. Then, the shooting range calculation unit 402 transmits information indicating the shooting range of the virtual camera for all frames included in the virtual camera path to the importance calculation unit 403.

FIG. 7A shows an example of the virtual cameras 701 to 703 indicated by the virtual camera path acquired in S601 and the shooting ranges 704 to 706 of the virtual cameras calculated in S602. In FIG. 7A, an example of a 3-frame virtual camera path is shown for the sake of clarity, but the number of frames is not limited to this, and the virtual camera path includes a virtual camera corresponding to more frames. Information may be included. FIG. 7A shows a shooting range of a two-dimensional region (xy plane) on the floor surface (z = 0) of the shooting target area 201, but the calculated shooting range is not limited to this.

In S603, the importance calculation unit 403 calculates the cumulative time in which the area is included in the shooting range for each area based on the shooting range of the virtual cameras for all frames included in the received virtual camera path. FIG. 7B shows the cumulative time of each region calculated based on the shooting range 704 to 706 of the virtual camera shown in FIG. 7A. The cumulative time of the region 707 is 3 frames, the cumulative time of the region 708 is 2 frames, and the cumulative time of the region 709 is 1 frame.

In S604, the importance calculation unit 403 calculates the importance of each region based on the cumulative time calculated in S603. FIG. 7C shows an example of correspondence between the cumulative time calculated in FIG. 7B and the importance. In the example of FIG. 7C, the importance is divided into three levels according to the cumulative time, but the correspondence between the cumulative time and the importance is not limited to this, and the correspondence between the cumulative time and the importance varies in the number of frames of the virtual camera path and the cumulative time. It may be determined according to the degree.

In S605, the arrangement determination unit 406 acquires the upper limit camera-to-camera angle and the lower limit resolution as a request according to the importance of the area from the request acquisition unit 405. As shown in FIG. 5, the correspondence table between the importance held in advance and the request may be acquired, or the request is set by the operator via the operation unit 305 for the area importance calculated in S604. May be done. In S606, the placement determination unit 406 may use the placement condition storage unit 404 as camera placement conditions, for example, information on the placement of the current camera 101, information on the camera 101 that can be placed, information on the place where the camera can be placed, and a shooting target area 201. Get information about.

In S607, the arrangement determination unit 406 determines the camera arrangement based on the request acquired in S605 and the camera arrangement condition acquired in S606. As an example of the method of determining the camera arrangement, first, the inter-camera angle and the resolution for each area according to the arrangement of the installed cameras 101 are calculated, and the area that does not satisfy the requirement is specified. Then, at least one of the position, the posture, and the focal length of the installed camera 101 is changed so as to satisfy the request preferentially from the region of high importance. The position and orientation of the camera 101 are controlled by the pan head, and the focal length is controlled by the zoom lens.

Further, if the request cannot be satisfied only by changing the setting of the currently installed camera 101, the camera 101 that can be arranged is added to the place where the camera 101 can be arranged. If there is an installed camera 101, the camera arrangement may be determined so as to change the installed camera 101 and to reduce the number of additional cameras 101 as much as possible. Alternatively, the state of the installed camera 101 may not be changed, and an additional camera 101 may be installed so as to satisfy the request. In that case, at least the requirements of the high-importance area may be satisfied. Further, the installed camera 101 may be fixed, and a camera 101 that can be controlled by a pan head may be installed as an additional camera 101, and these cameras 101 may be controlled.

If there is no installed camera 101, a plurality of cameras 101 are temporarily arranged so that the shooting target area 201 can be photographed under certain conditions, and then the cameras are arranged so that the demand is satisfied from the area of high importance. May be changed. Further, when there are a plurality of types of cameras 101 having different focal lengths, they may be arranged in combination. At that time, a camera with a telephoto lens may be preferentially used for photographing a region of high importance.

In S608, the arrangement determination unit 406 outputs the arrangement information indicating the camera arrangement determined in S607. The arrangement information is presented to users such as operators and camera installation workers via the display unit 306, and the user sees the display and installs or adjusts the camera 101 so that the camera arrangement determined in S607 is realized. conduct. The arrangement information may be transmitted to the control device 103 via the communication unit 307, and the position and orientation of the camera 101 may be automatically changed according to the arrangement information received by the control device 103.

Through the above processing, a camera arrangement that satisfies the requirements for generating an appropriate virtual viewpoint image according to the virtual camera path is realized. Then, the image generation device 104 can generate a virtual viewpoint image according to the virtual camera path with high image quality based on the plurality of captured images acquired by the plurality of cameras 101 so arranged. According to such a method, even if the virtual viewpoint is freely specified, a high-quality virtual viewpoint image can be generated by determining the camera arrangement accordingly, so that the degree of freedom related to the designation of the virtual viewpoint can be improved. Can be done.

FIG. 8 shows an example of the arrangement information output in S608. FIG. 8A shows an example of arrangement information output when there is no installed camera 101. In FIG. 8A, the three-dimensional coordinates and the focal length of the camera position and the gazing point are shown for each of the camera A, the camera B, and the camera C. The gazing point is a position on the optical axis of the camera 101 (a position away from the position of the camera 101 in the shooting direction), for example, at a position where the optical axis of the camera 101 and the ground of the shooting target area 201 intersect. be. Once the position and gaze point of the camera 101 are determined, the orientation of the camera 101 is also determined accordingly. The overall view of the camera arrangement as shown in FIG. 8B may be output as the arrangement information.

FIG. 8C is an example of arrangement information output when there is an installed camera 101. FIG. 8C shows the amount of change in position, posture, and focal length for each of camera A and camera C whose settings should be changed. Further, as shown in FIG. 8C, an overall view of the camera arrangement indicating the camera 101 whose setting should be changed may be output as arrangement information.

In addition, a camera 101 that captures an important area may be added without changing the settings of the installed camera 101. In this case, information regarding the arrangement of the camera 101 to be added as shown in FIG. 8A and an overall view of the camera arrangement showing the position of the camera 101 to be added as shown in FIG. 8D are output. It may be included in the placement information. The format of the arrangement information is not limited to the example of FIG. For example, the camera position may be indicated by a predetermined position where the camera 101 can be installed or a number of the position, instead of the three-dimensional coordinates. Further, a simulation image showing the shooting range of the camera 101 may be displayed instead of the three-dimensional coordinates of the gazing point.

The area importance may be calculated for each shooting direction of the virtual camera according to the virtual camera path, and the camera arrangement may be determined. For example, in the virtual camera path shown in FIG. 7, shooting is performed by the virtual camera only from the lower side of the shooting target area 201. Therefore, the cameras 101 (cameras F to J in the example of FIG. 8D) located on the opposite side of the shooting direction of the virtual camera may not be arranged, or the number of cameras and the set value may be changed.

Further, the example of the arrangement information shown in FIGS. 8B and 8C shows the determined camera arrangement two-dimensionally, but the example of the arrangement information is not limited to this and is determined. The camera arrangement may be three-dimensionally shown. That is, the output placement information may include a diagram showing the height of each camera 101 in the determined camera placement.

Further, in the present embodiment, of the shooting target area 201, the area that is not included in the shooting range of the virtual camera according to the virtual camera path is regarded as low in importance, and certain requirements are satisfied for that area as well. It was assumed that the camera placement would be decided. As a result, even if the virtual camera path is slightly changed or added, a virtual viewpoint image corresponding to the camera path can be generated. However, when the virtual viewpoint image is generated using only the specified virtual camera path, the camera is arranged so that the area not included in the shooting range of the virtual camera corresponding to the virtual camera path is not captured by the camera 101. May be determined.

[Modification example]
Hereinafter, a modified example of the image processing system 100 that determines the camera arrangement based on the position of an object existing in the shooting range of the virtual camera according to the virtual camera path will be described. In this modification, a virtual viewpoint image is generated based on a plurality of captured images acquired by a plurality of cameras 101 and a designated virtual camera path, and the arrangement of the plurality of cameras 101 is performed based on the virtual camera path. The settings are changed. Then, by using the plurality of captured images re-acquired by the plurality of cameras 101 according to the changed arrangement and settings, a high-quality virtual viewpoint image corresponding to the designated virtual camera path is generated. For example, the camera arrangement at the time of actual shooting is determined based on the virtual camera path specified at the time of rehearsal shooting. At the time of actual shooting, the position and orientation of the camera 101 used at the time of rehearsal shooting may be changed, or the camera 101 not used at the time of rehearsal shooting may be added. It is assumed that the virtual camera path in this modification includes the shooting time corresponding to each frame or the time code information at the time of shooting. In the following description, the differences from the above-described embodiments will be mainly described.

FIG. 9 shows the configuration of the image processing system 100 in this modification. In FIG. 9, the same reference numerals are given to the processing units that perform the same processing as that shown in FIG. The object information acquisition unit 901 acquires the object information corresponding to the virtual camera path acquired by the path acquisition unit 401 and transmits it to the object area calculation unit 902. The object information is information indicating the type of the object existing in the shooting target area 201 at the time of shooting (for example, at the time of rehearsal) and the shape and position for each shooting time. The types of objects include the types and names of people and objects. The shape and position of the object are, for example, information indicating the shape and position of the three-dimensional model of the object used to generate the virtual viewpoint image. The object information may include the result of identifying a person's face or the like by using the image recognition technique.

The object information acquired by the object information acquisition unit 901 is not limited to the one generated by the image generation device 104, but may be generated by another processing system or an external object shape model generation device. Alternatively, the object information may be generated based on the result of simulating the movement of the object assumed in advance, and is not limited thereto. The object information acquisition unit 901 acquires object information for all frames at the time of shooting, and extracts object information at the shooting time corresponding to the frame included in the virtual camera path acquired by the path acquisition unit 401. May be good. Alternatively, the object information acquisition unit 901 may acquire only the object information of the shooting time corresponding to the frame included in the virtual camera path from the outside.

The object area calculation unit 902 determines an area in which an object included in the shooting range of the virtual camera exists based on the shooting range of the virtual camera received from the shooting range calculation unit 402 and the object information received from the object information acquisition unit 901. calculate. The calculated object area is transmitted to the importance calculation unit 903. The calculated object area may include the area of all the objects within the shooting range of the virtual camera, or may include only the area of a specific object. The specific object is, for example, a specific person, the face of the person, a product in commercial photography, a company name, a product logo, or the like. The specific object may be determined according to the shooting content, or may be determined based on the instruction of the operator.

The importance calculation unit 903 calculates the importance of each area in the shooting target area 201 based on the object area received from the object area calculation unit 902. As a method of calculating the importance, as in the above method, each block in the shooting target area 201 accumulates the number of frames included in the object area, and the longer the cumulative time, the higher the importance of the block. It may be calculated in. However, the importance is not limited to this, and the importance may be calculated so that all the blocks included in the object area for one or more frames are important areas.

Alternatively, only the area where a specific object exists may be regarded as an important area, or the importance may be weighted according to the type of the object included in the field of view of the virtual camera. The weighting method according to the object may be determined according to the shooting content or may be determined according to the instruction of the operator. Further, the importance may be weighted according to the size of the object in the field of view of the virtual camera. For example, the area in which the object is located may be more important as the virtual camera shoots the object larger. ..

Next, the processing flow of the arrangement determination device 105 of this modification will be described with reference to FIG. The process shown in FIG. 10 starts at the timing when the virtual camera path is specified by the user via the image processing system 100 and the arrangement determination device 105 receives an instruction from the operator to determine the camera arrangement via the operation unit 305. Will be done. This virtual camera path is specified, for example, when prototyping a virtual viewpoint image based on a captured image at the time of rehearsal. The start timing of the process shown in FIG. 10 is not limited to the above. In the description of FIG. 10, the same reference numerals are given to the same processes as those described with reference to FIG. In the following, the differences from the processing shown in FIG. 6 will be mainly described.

In S1001, the object information acquisition unit 901 acquires the object information corresponding to the shooting time of each frame included in the virtual camera path acquired in S601. In S1002, the object area calculation unit 902 has the object area within the shooting range of the virtual camera calculated in S602 based on the object information acquired in S1001 for each frame included in the virtual camera path acquired in S601. Is calculated.

FIG. 11A shows an example of the virtual cameras 701 to 703 indicated by the virtual camera path, the shooting ranges 704 to 706 of the virtual cameras calculated in S602, and the areas 1101 and 1102 in which the object exists in each frame. In FIG. 11A, the area 1101 is the area of the object located within the shooting range of the virtual camera, and the area 1102 is the area of the object located outside the shooting range of the virtual camera. When one object exists within or outside the shooting range of the virtual camera, only a part of the area of the object within the shooting range may be treated as the object area within the shooting range. Alternatively, the entire area of such an object may be treated as an object area within the shooting range or an area outside the shooting range.

In S1003, the importance calculation unit 903 calculates the cumulative area obtained by accumulating the object areas within the shooting range of the virtual camera calculated in S1002 over all the frames included in the virtual camera path. FIG. 11B shows a state in which the area 1101 of the object in the virtual camera shooting range shown in FIG. 11A is accumulated.

In S1004, the importance calculation unit 903 calculates the important area based on the cumulative result of the object area calculated in S1003, and transmits it to the arrangement determination unit 406. FIG. 11C shows the important region 1103 calculated so as to surround the cumulative region of the region 1101. That is, the object area in each frame is included in the important area. However, the important region 1103 does not have to be a quadrangle as in the example of FIG. 11C, and may be, for example, the contour shape of the cumulative region, an ellipse, or the like. Further, the important area 1103 may be set so as to take a margin around the cumulative area. The importance calculation unit 903 may uniformly set the importance of the area included in the important area, or the area included in the important area having a large number of frames, which is an object area, has a higher importance. The importance may be set as follows.

The process after the importance of each area is calculated is the same as the process described with reference to FIG. 6, and the camera arrangement is determined and determined based on the request according to the area importance and the camera arrangement condition. Placement information indicating the camera placement is output. Through the above processing, a camera arrangement that meets the requirements for high-quality display of objects within the shooting range of the virtual camera according to the virtual camera path (that is, objects reflected in the virtual viewpoint image) is realized. Will be done. Then, the image generation device 104 can generate a virtual viewpoint image according to the virtual camera path with high image quality based on the plurality of captured images acquired by the plurality of cameras 101 so arranged.

For example, in a situation where reshooting is possible such as studio shooting, if the image quality of the shot image and the virtual viewpoint image generated based on the specified virtual camera path is insufficient, the following is based on the virtual camera path. You can modify the camera placement for shooting. Further, in shooting in which the performer makes a predetermined movement, it is possible to determine the camera arrangement based on the simulation data representing the position and movement of the performer and the virtual camera path. Further, not only in studio shooting but also in sports shooting, for example, it is possible to improve the camera arrangement for the next shooting by using the past shooting data and the virtual camera path.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

100 Image processing system 101 Camera 102 Hub 103 Control device 104 Image generator 105 Placement determination device

Claims (19)

A specific means for specifying the position and orientation of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of images obtained by shooting a shooting target area from different positions by a plurality of shooting devices.
An information processing apparatus comprising: a determination means for determining the position and orientation of an imaging device included in the plurality of imaging devices based on the position and orientation of the virtual viewpoint specified by the specific means. The specific means identifies the position and orientation of the virtual viewpoint at each of a plurality of time points, and determines the position and orientation of the virtual viewpoint.
The first aspect of the present invention is characterized in that the determination means determines the position and orientation of the imaging device included in the plurality of imaging devices based on the position and orientation of the virtual viewpoint at each of the plurality of time points. Information processing equipment. The specifying means identifies the position and orientation of the virtual viewpoint, and the angle of view corresponding to the virtual viewpoint.
The determining means is a position, orientation and angle of view of the photographing apparatus included in the plurality of photographing devices based on the position and orientation of the virtual viewpoint specified by the specific means and the angle of view corresponding to the virtual viewpoint. The information processing apparatus according to claim 1 or 2, wherein the information processing apparatus is determined. Any of claims 1 to 3, wherein the specifying means specifies the position and orientation of the virtual viewpoint based on an input corresponding to a user operation for designating the position and orientation of the virtual viewpoint. The information processing apparatus according to item 1. The information processing according to any one of claims 1 to 4, wherein the specific means specifies the position and orientation of the virtual viewpoint designated before the imaging by the plurality of imaging devices is performed. Device. According to the position and orientation of the virtual viewpoint specified by the specific means, based on a plurality of images obtained by photographing by the plurality of photographing devices installed according to the position and orientation determined by the determination means. The information processing apparatus according to any one of claims 1 to 5, further comprising a generation means for generating a virtual viewpoint image. The specifying means identifies the position and orientation of the virtual viewpoint designated for generating the first virtual viewpoint image based on the first plurality of images.
The determination means is a position of a photographing device for acquiring a plurality of second images used for generating a second virtual viewpoint image according to the position and orientation of the virtual viewpoint specified by the specific means. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing apparatus is characterized in that the orientation is determined. The information processing according to claim 7, wherein the photographing device whose position and orientation are determined by the determining means includes a photographing device that is not used for acquiring the first plurality of images. Device. The determining means determines the position and orientation of the photographing device for acquiring the second plurality of images based on the position and orientation of the photographing device used for capturing the first plurality of images. The information processing apparatus according to claim 7, wherein the information processing apparatus is to be used. Any of claims 1 to 9, further comprising a control means for controlling at least one of the positions and orientations of the photographing apparatus based on at least one of the positions and orientations of the photographing apparatus determined by the determining means. The information processing apparatus according to claim 1. The determination means is
A partial region included in the visual field according to the position and orientation of the virtual viewpoint specified by the specific means, which is a partial region included in the imaging target region, is specified.
The information processing apparatus according to any one of claims 1 to 10, wherein the position and orientation of the photographing apparatus are determined so that the photographing condition of the partial region by the photographing apparatus satisfies a predetermined requirement. .. The information processing apparatus according to claim 11, wherein the partial area specified by the determination means is an area in which a predetermined object existing in the photographing target area is located. 11. The information processing apparatus according to 12. The specific means identifies the position and orientation of the virtual viewpoint at each of a plurality of time points, and determines the position and orientation of the virtual viewpoint.
The determination means determines the importance of the partial region based on the number of time points in which the partial region is included in the visual field according to the position and orientation of the virtual viewpoint among the plurality of time points.
The information processing apparatus according to any one of claims 11 to 13, wherein the predetermined requirement is determined according to the importance of the partial region. The determining means determines the importance of the partial region, a shooting scene corresponding to a virtual camera path indicating the position and orientation of the virtual viewpoint at each of the plurality of time points, the popularity of the virtual camera path, and the virtual. The information processing apparatus according to claim 14, wherein the information processing apparatus is determined based on at least one of the type of an object included in the field of view according to the camera path and the size of the object in the field of view. A specific process for specifying the position and orientation of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of images obtained by shooting a shooting target area from different positions by a plurality of shooting devices.
An information processing method comprising: a determination step of determining a position and an orientation of an imaging device included in the plurality of imaging devices based on the position and orientation of the virtual viewpoint specified in the specific step. In the specific step, the position and orientation of the virtual viewpoint at each of the plurality of time points are specified.
16. The determination step is characterized in that the positions and orientations of the photographing devices included in the plurality of photographing devices are determined based on the positions and orientations of the virtual viewpoints at each of the plurality of time points. Information processing method described in. The determination step is
A partial region included in the visual field according to the position and orientation of the virtual viewpoint specified in the specific step, which is a partial region included in the imaging target region, is specified.
The information processing method according to claim 16 or 17, wherein the position and orientation of the photographing apparatus are determined so that the photographing condition of the partial region by the photographing apparatus satisfies a predetermined requirement. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 15.

Images