JP2021051374A - Shape data generation device, shape data generation method, and program - Google Patents

Abstract

To appropriately perform processing involved in generation of shape data in an entire captured space.SOLUTION: A space setting unit 110 sets a first partial space of a plurality of partial spaces included in a captured space at a first parameter used in generation of shape data on a subject included in the first partial space. Also, the space setting unit 110 sets a second partial space of the plurality of partial spaces at a second parameter used in generation of shape data on the subject included in the second partial space, the second parameter differing from the first parameter. A shape estimation unit 130 generates the shape data on the subject included in the first partial space on the basis of the first parameter and a plurality of captured images taken from a plurality of image-capturing devices 20. Also, the shape estimation unit 130 generates the shape data on the subject included in the second partial space on the basis of the second parameter and a plurality of captured images.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for generating a virtual viewpoint image.

In recent years, a plurality of image pickup devices are installed at different positions to perform synchronous shooting from a plurality of directions, and an image (virtual viewpoint) viewed from a designated viewpoint (virtual viewpoint) is used by using a plurality of images obtained from the image pickup device. The technology for generating images) is drawing attention. According to the above-mentioned technology for generating a virtual viewpoint image, it is possible to generate an image viewed from a designated viewpoint for various events such as sports games, concerts, and plays.

In Patent Document 1, shape data of a subject is generated based on an image showing a subject (hereinafter, referred to as a foreground image) obtained based on imaging by a plurality of imaging devices installed at different positions, and the shape data thereof. Describes how to generate a virtual viewpoint image using.

JP-A-2015-45920

The imaging space imaged by a plurality of imaging devices may include subjects having different sizes, motion characteristics, and the like. For example, in a rugby game, the subjects are a player and a ball, but the ball is smaller and moves faster than a person who is a player. Then, in the first space near the ground in the imaging space, the player and the ball are included as the subjects, but in the second space above the sky, the player is not included and only the ball may exist. If processing such as foreground image generation and shape data generation is performed in the same manner for the first and second subspaces, the foreground image generation accuracy may vary, and the shape data may not be generated accurately. There was a risk that the processing related to the generation of shape data could not be performed properly.

The present invention has been made in view of the above problems. The purpose is to appropriately perform processing related to shape data generation in the entire imaging space.

The shape data generation device according to the present invention is included in one or more captured images acquired from one or more imaging devices among the plurality of imaging devices, and in a plurality of subspaces in the imaging space imaged by the plurality of imaging devices. A first generation means for generating shape data of a subject included in the first subspace based on a first parameter corresponding to the first subspace, and one or more of the plurality of imaging devices. A second parameter corresponding to one or more captured images acquired from the image pickup apparatus and a second subspace included in the plurality of subspaces, which is different from the first parameter. It is characterized by having a second generation means for generating shape data of a subject included in the second subspace based on the above.

According to the present invention, it is possible to appropriately perform the process related to the generation of shape data in the entire imaging space.

It is a figure which shows an example in which a plurality of image pickup apparatus 2 are arranged. It is a figure for demonstrating the hardware structure of the shape estimation apparatus 1. FIG. It is a figure for demonstrating the structure of the image processing system 1000 including the shape estimation apparatus 1. It is a figure for demonstrating an example of the structure of the image pickup system 2. It is a figure which represented typically an example of the imaging space divided into a plurality of subspaces. It is a flowchart for demonstrating an example of a process performed by a shape estimation apparatus 1. It is a figure for demonstrating the structure of the image processing system 1100 including the shape estimation apparatus 7. It is a figure for demonstrating an example of the structure of the 1st imaging system 8 and the 2nd imaging system 9. It is a flowchart for demonstrating an example of the process performed by a shape estimation apparatus 7.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The components described in the following embodiments show an example of the embodiments of the present invention, and the present invention is not limited to them.

(First Embodiment)
The present embodiment relates to a shape estimation device that generates shape data of a subject in an imaging space imaged by a plurality of imaging devices. First, the problems in this embodiment will be described. When generating shape data of a subject in an imaging space imaged by a plurality of imaging devices, processing related to the generation of shape data is not properly performed when subjects having different sizes and motion characteristics are included in the imaging space. there is a possibility. For example, when a rugby game is imaged, the imaging space includes players and the ball. Since the ball is smaller in size and moves faster than the player, the foreground image may not be generated properly. Therefore, if the ball and the player perform the shape data generation process in the same manner, there is a possibility that the shape data is not properly generated. On the other hand, it is conceivable to perform different processing when generating the shape data depending on the type of the subject (here, the player and the ball). However, it has been difficult to specify the type of subject before performing the process related to the generation of shape data.

By the way, in the example of rugby, a player and a ball can coexist as a subject in the space near the ground of the field, but for example, the space 10 m above the ground does not include the player and only the ball is included as the subject. obtain. That is, there may be a subspace in the imaging space where only a specific subject can exist. Focusing on the above points, the shape estimation device in the present embodiment is configured to perform different processes related to shape data generation for each subspace. Therefore, for example, by making the process related to the generation of shape data different between the subspace near the ground and the subspace 10 m above in a rugby game, the process related to the generation of the shape data of the subject is appropriately performed in the entire imaging space. Will come to be.

Notwithstanding the above example, the present embodiment can be applied even in the following cases. For example, the imaging space may be imaged by a plurality of imaging devices including imaging devices having different angles of view. In this case, for example, a space in which an important scene (for example, a goal scene in soccer) can occur is imaged by a telephoto camera and a wide-angle camera, and the entire other space is imaged only by the wide-angle camera. Further, for example, in order to image a player with high resolution, it is assumed that the space near the ground is imaged with a telephoto camera, and the space above the field where only the ball can be included is imaged only with a wide-angle camera. When a subject is imaged by a wide-angle camera, the area of the subject in the captured image becomes smaller than when the same subject is imaged by a telephoto camera. As a result, the process related to the generation of shape data may not be performed properly. Even in such a case, if the shape estimation device according to the present embodiment is used, the processing related to the generation of the shape data of the subject can be appropriately performed in the entire imaging space.

Hereinafter, the shape estimation device according to the present embodiment will be described. The shape data generated by the shape estimation device is used to generate a virtual viewpoint image. Here, the virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and is selected by the user from a plurality of candidates, for example. The virtual viewpoint image also includes an image corresponding to the viewpoint. Further, the virtual viewpoint may be specified by a user operation, or may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a still image will be mainly described, but the virtual viewpoint image may be a moving image.

Further, the plurality of image pickup devices used for generating the virtual viewpoint image may be arranged so as to surround the image pickup space, for example, as in the image pickup device 20 shown in FIG. Events such as sports games, concerts, and plays are assumed as targets to be imaged by a plurality of image pickup devices. Further, the imaging space refers to a space where the above-mentioned event is performed. For example, in the case of rugby, it is a three-dimensional space composed of a ground in a stadium where rugby is performed and an arbitrary height. Further, the subject refers to an object in the above-mentioned imaging space, for example, a player in the field, a ball in a ball game, or the like. The imaging devices are installed at different positions and perform imaging in synchronization from different imaging directions. The imaging device may not be installed over the entire circumference of the imaging space, and may be installed only in a part of the imaging space depending on the limitation of the installation location or the like. The number of imaging devices is not limited. For example, when the imaging space is a soccer stadium, tens to hundreds of imaging devices may be installed around the stadium.

The imaging system in this embodiment is composed of a plurality of imaging devices including a telephoto camera and a wide-angle camera. For example, by imaging a player in a rugby game using a telephoto camera, a high-resolution captured image can be obtained, and the resolution of the generated virtual viewpoint image is improved. On the other hand, since the ball has a wide range of movement, it is necessary to install a large number of balls in order to take an image with a telephoto camera. Therefore, the number of image pickup devices to be installed can be reduced by taking an image of the ball using a wide-angle camera having a wide angle of view. Further, for example, in a rugby game, it is assumed that the players are distributed near the ground of the field and the ball can be included above the field. Therefore, it is also possible to image a subspace near the ground where the player plays in the imaging space and a subspace where the ball reaches, for example, 10 m above the ground, with imaging devices having different angles of view. In addition to this, for example, a space where players are likely to gather and important scenes (for example, goal scenes) are likely to occur, such as in front of a goal and a penalty area in soccer, is imaged with a telephoto camera, and other spaces are captured with a wide-angle camera. It may be configured to take an image. As described above, according to an imaging system including imaging devices having different angles of view such as a telephoto camera and a wide-angle camera, it is possible to generate a high-resolution virtual viewpoint image while reducing the number of imaging devices installed. There is. The configuration of the imaging system is not limited to this. For example, it may be configured by an image pickup device having the same angle of view, or may be configured to include an image pickup device of a type different from the above.

FIG. 2 is a diagram for explaining the hardware configuration of the shape estimation device 1 in the present embodiment. The shape estimation device 1 includes a CPU 511, a ROM 512, a RAM 513, an auxiliary storage device 514, a communication I / F 515, and a bus 516. The CPU 511 realizes each function of the shape estimation device 1 by controlling the entire shape estimation device 1 by using computer programs and data stored in the ROM 512 and the RAM 513. The shape estimation device 1 may have one or more dedicated hardware different from the CPU 511, and the dedicated hardware may execute at least a part of the processing by the CPU 511. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 512 stores programs and the like that do not require changes. The RAM 513 temporarily stores programs and data supplied from the auxiliary storage device 514, data supplied from the outside via the communication I / F 515, and the like. The auxiliary storage device 514 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The communication I / F 515 is used for communication with an external device of the shape estimation device 1. For example, when the shape estimation device 1 is connected to an external device by wire, a communication cable is connected to the communication I / F 515. When the shape estimation device 1 has a function of wirelessly communicating with an external device, the communication I / F 515 includes an antenna. The shape estimation device 1 in the present embodiment communicates with an image pickup system, an image generation device, and the like, which will be described later, via communication I / F 515. The bus 516 connects each part of the shape estimation device 1 to transmit information. In the present embodiment, the auxiliary storage device 514 is assumed to exist inside the shape estimation device 1, but it may be connected to the outside of the shape estimation device 1.

FIG. 3 is a diagram for explaining the configuration of the image processing system 1000 including the shape estimation device 1 in the present embodiment. The system and the apparatus included in the image processing system 1000 will be described with reference to FIG. The image processing system 1000 includes a shape estimation device 1, an image pickup system 2, an image generation device 3, and a display device 4. The shape estimation device 1 generates shape data of a subject in the imaging space imaged by the image pickup system 2, and transmits the generated shape data to the image generation device 3. A detailed description of the shape estimation device 1 will be described later.

The image pickup system 2 is a system composed of a plurality of image pickup devices as described above. FIG. 4 is a diagram for explaining an example of the configuration of the imaging system 2 in the present embodiment. FIG. 4 shows an imaging system 20 composed of seven imaging devices 20a to 20g. The number of imaging devices included in the imaging system 2 is not limited to this. The image pickup system 20 in the present embodiment has foreground background separation units 21a to 21g, and the image pickup devices 20a to 20g are connected to the foreground background separation units 21a to 21g, respectively. In the following description, unless otherwise specified, the image pickup apparatus 20a to 20g and the foreground background separation portion 21a to 21g are simply referred to as the image pickup apparatus 20 and the separation unit 21.

Each of the plurality of image pickup devices 20 has an identification number for identifying the image pickup device 20 and subject information indicating a subject to be mainly imaged. The subject information in the present embodiment is information indicating the type of the subject indicating whether it is a person or a moving object other than a person such as a ball, such as a "player" or a "ball". In the case of subject information indicating a person, the subject information may have more detailed information such as "player A" and "player B". The identification number and subject information are set in advance. In this embodiment, one subject information is attached to one imaging device 20. The separation unit 21 separates a region corresponding to the foreground and a region corresponding to the background from the captured image obtained by the imaging device 20 capturing the imaging space, and generates a foreground image and a background image. The foreground in the present embodiment refers to a dynamic object (moving object) that moves (the absolute position and shape of the object can change) when images are taken from the same direction in time series among the objects corresponding to the subject. .. Dynamic objects are, for example, singers, performers, performers, moderators, etc. in concerts and entertainment, in addition to the players and balls described above. The foreground image is an image obtained by extracting a region corresponding to the above-mentioned dynamic object in the captured image.

The background refers to an object to be imaged that is stationary or nearly stationary when the image is taken from the same direction in chronological order among the objects corresponding to the subject. Such imaging objects are, for example, stages such as concerts, stadiums where events such as competitions are held, structures such as goals used in ball games, and fields. That is, the background is at least an area different from the object corresponding to the foreground. The background image is obtained by removing the object corresponding to the foreground from the captured image. The subject imaged by the imaging device may include another object in addition to the above-mentioned foreground and background.

The separation unit 21 performs a separation process of extracting the foreground and generating a foreground image according to the subject information possessed by the image pickup apparatus 20. For example, when the subject information possessed by the image pickup apparatus 20 is "player", the separation unit 21 uses the background subtraction method as the separation process. The background subtraction method is a method of extracting the foreground by calculating the difference between the captured image and the background image. In addition, by updating the background image at regular intervals, the foreground is stubbornly extracted even when the brightness fluctuates. Further, for example, when the subject information possessed by the image pickup apparatus 20 is a "ball", the separation unit 21 uses the inter-frame difference method as the separation process. The inter-frame difference method is a method of extracting the foreground by using continuously imaged imaging frames. In the case of a subject that moves at high speed and in a wide range according to the laws of physics, such as a ball, the difference between the captured image of the target for which the foreground is extracted and the frame several frames before in the continuously captured imaging frame is calculated. , The subject is stubbornly extracted. As described above, the separation unit 21 uses at least one of the size and motion characteristics of the subject imaged by each image pickup device 20 by using different separation processes depending on the subject mainly imaged by the image pickup device 20. Even if they are different, the accuracy of the foreground image can be improved. In addition, each separation unit 21 corresponding to each of the image pickup devices 20 may have a configuration in which the subject information is provided. Separation processing is performed using a method corresponding to information.

In addition, the separation unit 21 performs noise removal processing on the generated foreground image. As an example of the noise removal process, there is a method of reducing the noise area by filtering. As for the processing parameters at the time of filtering, different values are set according to the subject information possessed by the image pickup apparatus 20. The above processing parameters are parameters used for determining whether or not a specific region in the foreground image is noise, and are represented by, for example, a pixel area or the like. The pixel area referred to here is the area of the region corresponding to the subject in the captured image expressed by the number of pixels. For example, when the subject information is "player", the processing parameters are set based on the standard body shape of the player. As a result, the region extremely different from the pixel area corresponding to the size of the player is determined to be the noise region. Further, for example, when the subject information is a "ball", processing parameters are set based on the size and shape of a soccer ball or a rugby ball. The processing parameters used for the noise removal processing may be determined according to the angle of view of the image pickup apparatus 20. For example, when a subject is imaged by a wide-angle camera, it is assumed that the pixel area corresponding to the subject in the captured image is smaller than that when the subject is imaged by a telephoto camera. Therefore, when the subject is imaged by the wide-angle camera, the processing parameters are determined so that the region extremely larger than the pixel area of the subject is determined as the noise region. On the contrary, when the subject is imaged by the telephoto camera, the processing parameters are determined so that the region extremely smaller than the pixel area of the subject is determined as the noise region. As a result, even if the angle of view of the image pickup apparatus 20 is different, noise can be removed while leaving the subject area. Further, the separation unit 21 has a plurality of processing parameters, and it is also possible to select a processing parameter to be used from the plurality of processing parameters according to the subject information possessed by the image pickup apparatus 20.

The separation unit 21 transmits the generated foreground image to the shape estimation device 1. The types of separation processing and noise removal processing performed by the separation unit 21 are not limited to the above, and other processing may be performed. Further, in the present embodiment, the separation unit 21 is included in the imaging system 2, but the separation unit 21 is included in each imaging device 20 or the shape estimation device 1, or is connected to the outside as another device. It may be.

Returning to FIG. 3, the image generation device 3 generates a virtual viewpoint image using the shape data generated by the shape estimation device 1. The method of generating the virtual viewpoint image will be described later. The image generation device 3 transmits the generated virtual viewpoint image to the display device 4. The display device 4 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a virtual viewpoint image transmitted from the image generation device 3. In addition to the virtual viewpoint image, the display device 4 also displays a GUI (Graphical User Interface) or the like for the user to perform an input operation or the like necessary for generating the virtual viewpoint image.

Next, the functional configuration of the shape estimation device 1 will be described with reference to FIG. The shape estimation device 1 includes a subject information acquisition unit 100, a space setting unit 110, an imaging information acquisition unit 120, and a shape estimation unit 130. Hereinafter, each processing unit will be described.

The subject information acquisition unit 100 acquires the identification numbers and the subject information of each of the plurality of imaging devices 20 from the imaging system 2. The subject information acquisition unit 100 may be configured to acquire a file in which the identification number of the image pickup apparatus 20 and the subject information are associated with each other from an external storage device or the like.

The space setting unit 110 divides the imaging space imaged by the imaging system 2 into a plurality of subspaces, and sets the subject information to be associated with each subspace. Dividing the imaging space here means virtually dividing the space into a plurality of subspaces. FIG. 5 is a diagram schematically showing an example of an imaging space divided into a plurality of subspaces. The shape estimation device 1 in the present embodiment generates shape data of a subject in the imaging space 300 indicated by a broken line. Although the imaging space and the space where the shape estimation is performed (shape estimation space) are different spaces, in the present embodiment, the imaging space and the shape estimation space will be described as the same.

The imaging space 300 is a three-dimensional space, and is set in association with the coordinates on the captured image acquired by the imaging system 2. In the example shown in FIG. 5A, the ground of a stadium or the like where a sporting event is held is defined as an xy plane represented by the x-axis and the y-axis, and the z-axis is defined in a direction 302 perpendicular to the ground 301. .. The ground 301 has z = 0. Further, a boundary 310 for dividing the imaging space 300 into the subspace 320 and the subspace 330 is set. The boundary 310 is represented using three-dimensional coordinates representing the height, for example, "z = 2m". The imaging space 300 and the boundary 310 are set by the space setting unit 110 acquiring information on three-dimensional coordinates from the auxiliary storage device 514, an external storage device, or the like. Further, the origin and the coordinate axis of the coordinates representing the imaging space can be set at arbitrary positions and directions.

5 (b) and 5 (c) are diagrams showing other examples of the imaging space and the subspace determined based on the boundaries. The imaging space 300 in FIG. 5B is divided into a rectangular parallelepiped subspace 430 and other subspaces 420 by the boundary 410. Further, FIG. 5C is an example of being divided into three subspaces. The imaging space 300 in FIG. 5C is divided into a rectangular parallelepiped subspace 431 and a subspace 432, and other subspaces 421 by the boundary 411. In this way, the space setting unit 110 divides the imaging space into two or more subspaces by an arbitrary boundary.

The space setting unit 110 associates the subject information acquired by the subject information acquisition unit 100 with each of the plurality of subspaces determined based on the imaging space 300 and the boundary 310. Each subspace is associated with subject information indicating a subject having a high proportion contained in each subspace. For example, a case where the subspace shown in FIG. 5A is set with respect to the imaging space in which the rugby game is imaged will be described. At this time, the subspace 330 from the ground 301 to the boundary 310 may include both the player and the ball as subjects. However, since it is assumed that the proportion of players included in the subject near the ground is higher than that of the ball, "player" is associated with the subject information. On the other hand, above the boundary 310, players are rarely included, and high-rise balls may be frequently included. Therefore, a "ball" is associated with the subspace 320 as subject information.

Even when the subspaces shown in FIGS. 5 (b) and 5 (c) are set, the subject information is associated with the same concept as described above. Subject information indicating an "infielder" and an "outfielder" in baseball, for example, is associated with the subspace as shown in FIG. 5B. Further, in the subspace as shown in FIG. 5 (c), "Team A player", "Team B player" and "ball" in which the players are classified by team in a competition such as volleyball in which the players are played by themselves are displayed. The indicated subject information is associated. The space setting unit 110 transmits information indicating each subspace and the identification number of the imaging device associated with the subject information associated with each subspace to the shape estimation unit 130.

Further, the space setting unit 110 sets processing parameters to be used when the shape estimation unit 130, which will be described later, performs the shape estimation process, according to the subject information associated with each subspace. For example, the space setting unit 110 sets the first processing parameter in the subspace 320 which is the first subspace included in the imaging space 300, and sets the first processing parameter in the subspace 330 which is the second subspace. A first processing parameter different from the processing parameter is set. Similarly, processing parameters are set for an imaging space having three or more subspaces (for example, the imaging space 300 shown in FIG. 5C). Therefore, there can be at least one combination of subspaces in which different processing parameters are set. In a plurality of subspaces including the first and second subspaces in which different processing parameters are set, the first or second processing parameters are set for the subspaces other than the first and second subspaces. It may be set. The space setting unit 110 also transmits the set processing parameters to the shape estimation unit 130.

In the present embodiment, an example in which the subspace is set for each subject having a high proportion of the subspace is described, but even if the subspace is set according to the easiness of occurrence of an important scene. Good. For example, before a goal in soccer, it is assumed that important scenes such as a goal scene are likely to occur. In this case, the space setting unit 110 sets a subspace including the front of the goal and a subspace including other areas, and provides information such as "importance: high" and "importance: low" to each. Correspond. Further, the subspace may be set according to the space in which the number of subjects is large and the space in which the number of subjects is small in the imaging space.

The image pickup information acquisition unit 120 acquires the foreground image generated by the image pickup system 2. Further, the imaging information acquisition unit 120 acquires the imaging parameters of each imaging device 20 included in the imaging system 2 from the imaging system 2. The imaging parameter is a parameter including an external parameter indicating the position where the imaging device 20 is installed and the posture of the imaging device 20, and an internal parameter indicating the focal length, the optical center, the lens distortion, and the like of the imaging device 20. The imaging information acquisition unit 120 can also calculate imaging parameters using existing calibration processing based on the captured image obtained from the imaging system 2. As an example of the existing calibration process, the imaging information acquisition unit 120 acquires a plurality of captured images from the imaging system 2, and uses the feature points in the plurality of captured images to calculate the corresponding points between the captured images. The imaging information acquisition unit 120 optimizes the calculated corresponding points so that the error when projected onto each imaging device 20 is minimized, and calculates the imaging parameters by calibrating each imaging device 20. The calibration process used is not limited to the above. Further, the imaging parameters may be acquired at the stage of preparation in advance when the imaging device 20 is installed, or may be calculated each time the imaging information acquisition unit 120 acquires the captured image. In addition, the imaging parameters may be calculated based on the captured images acquired in the past. The imaging information acquisition unit 120 transmits the acquired foreground image and imaging parameters to the shape estimation unit 130 and the image generation device 3.

The shape estimation unit 130 performs shape estimation processing of the subject based on the information indicating the subspace acquired from the space setting unit 110 and the identification number, and the foreground image and the imaging parameter acquired from the imaging information acquisition unit 120. Generate shape data. The shape estimation unit 130 in the present embodiment generates three-dimensional model data by using a known technique of the visual volume crossing method (shape-from-silhouette method). As a method for generating three-dimensional model data, a method other than the visual volume crossing method may be used.

The shape estimation unit 130 identifies one or more image pickup devices 20 having an identification number corresponding to the subspace based on the information indicating the subspace acquired from the space setting unit 110 and the identification number. Further, the shape estimation unit 130 estimates the shape of the subject by using the foreground image generated based on the one or more captured images obtained from the specified one or more image pickup devices 20. For example, it is assumed that the subspace 320 and the subspace 330 shown in FIG. 5A are associated with subject information indicating a “ball” and a “player”, respectively. At this time, the shape estimation unit 130 is generated based on one or more captured images obtained from one or more imaging devices 20 having an identification number associated with the subject information indicating the "ball" in the subspace 320. Shape estimation is performed using the foreground image. Further, the shape estimation unit 130 is a foreground generated based on one or more captured images obtained from one or more imaging devices 20 having an identification number associated with subject information indicating "player" in the subspace 330. Shape estimation is performed using images.

Here, the shape estimation process performed by the shape estimation unit 130 will be described. Based on the foreground image, the shape estimation unit 130 generates a silhouette image in which the region corresponding to the foreground and the other regions in the captured image are represented by two values. The silhouette image is, for example, an image in which the pixel value of the region corresponding to the foreground is 1 and the pixel value of the other region is 0. The shape estimation unit 130 uses imaging parameters to determine the coordinates of the representative points (for example, the center) of the voxels (one unit in the method of expressing a three-dimensional space by a set of cubes of a unit volume) to be processed. It is converted into the coordinates of the captured image system acquired by 20. The shape estimation unit 130 determines whether or not the coordinates of the captured image system are included in the foreground region. Specifically, the shape estimation unit 130 determines whether or not the pixel value on the silhouette image corresponding to the coordinates on the captured image is a value corresponding to the foreground region (pixel value 1 in the above example). .. When it is determined that the above coordinates are the foreground region in all the captured image systems, the shape estimation unit 130 determines that the voxels corresponding to the coordinates are a part of the shape of the subject. On the other hand, when there is an captured image in which the above coordinates are determined to be a region other than the foreground region (the pixel value on the silhouette image corresponding to the coordinates is 0), the shape estimation unit 130 determines the voxels corresponding to the pixels. Judge that it is not a part of the shape of the subject. The shape estimation unit 130 performs the above processing for each voxel constituting the shape estimation space corresponding to the imaging space, and estimates the shape of the subject by deleting the voxels that are not determined to be a part of the shape of the subject. Will be done. In the present embodiment, the shape estimation unit 130 is configured to generate the silhouette image, but the silhouette image may be generated by the separation unit 21 and the imaging information acquisition unit 120 may acquire the silhouette image.

In the above shape estimation process, the shape estimation unit 130 performs shape estimation using the processing parameters set in the space setting unit 110. Here, an example of performing shape estimation processing using shape estimation parameters as processing parameters will be described. The shape estimation parameter is a parameter indicating that, in the shape estimation process described above, a predetermined number of captured images in which the pixels on the captured image corresponding to the voxel to be processed are determined not to be in the foreground region are allowed. For example, when the shape estimation parameter is set to 1, the shape estimation unit 130 determines that the number of pixels on the captured image corresponding to the voxel to be processed is not the foreground region, and the number of captured images is 1 or less. Determines that the voxel is part of the shape of the subject. On the other hand, when the number of captured images determined to be not in the foreground region is two or more pixels on the captured image corresponding to the voxel to be processed, it is determined that the voxel is not a part of the shape of the subject. The shape estimation unit 130 estimates the shape of the subject in each subspace using the shape estimation parameters as described above.

An example of the shape estimation process in the case where the subspace 320 and the subspace 330 shown in FIG. 5A are associated with the subject information indicating the “ball” and the “player”, respectively, will be described. Since the ball is small in size and moves quickly, the foreground extraction in the separation unit 21 tends to fail. Therefore, when performing the shape estimation process in the subspace 320, the shape estimation unit 130 performs the process with the shape estimation parameter set to 1. As a result, the shape estimation unit 130 can perform shape estimation even if there is an captured image in which foreground extraction has failed. On the other hand, players are more likely to extract the foreground more accurately than the ball. Therefore, when performing the shape estimation process in the subspace 320, the shape estimation unit 130 sets the shape estimation parameter to 0 and performs the process. As a result, the shape estimation unit 130 can accurately estimate the shape of the athlete. As described above, for a subject such as a ball whose foreground extraction is likely to fail, accurate shape estimation is possible by performing shape estimation processing using a shape estimation parameter having a larger value than that of a subject whose foreground extraction is unlikely to fail. become. The shape estimation parameters are set for each subspace in the space setting unit 110. Therefore, the shape estimation unit 130 performs the shape estimation process using the shape estimation parameter set to 1, for example, in the subspace 320 which is the first subspace included in the imaging space 300. Further, the shape estimation unit 130 performs the shape estimation process in the subspace 330, which is the second subspace, by using the shape estimation parameter set to 0.

Further, the value of the shape estimation parameter may be determined according to the number of image pickup devices that image the subject. For example, consider the case where there are 10 and 50 image pickup devices associated with subject information indicating "ball" and "player", respectively. Since the number of image pickup devices corresponding to "players" is larger than that of image pickup devices corresponding to "balls", it is expected that there is a high possibility that an image pickup device that fails to extract the foreground due to a failure or the like is included. In this case, for example, the shape parameter is set to 1 when estimating the shape in the subspace corresponding to the "ball", and the shape parameter is set to 4 when estimating the shape in the subspace corresponding to the "player". As a result, the shape estimation unit 130 can accurately estimate the shape in consideration of the number of image pickup devices. The above shape estimation parameter is an example, and values other than the above may be used as the shape estimation parameter. The shape estimation unit 130 generates shape data based on the result of the shape estimation process and transmits it to the image generation device 3.

In the present embodiment, the example in which the shape estimation unit 130 generates the three-dimensional model data has been described, but the shape estimation unit 130 may be configured to generate the two-dimensional shape data. In this case, the shape estimation unit 130 generates a foreground image based on the captured image as a shape data generation process. At this time, the imaging information acquisition unit 120 acquires the captured image from the imaging system 2 and transmits it to the shape estimation unit 130. In addition, the spatial setting unit 110 sets processing parameters for specifying a processing method for generating a foreground image. For example, it is assumed that "1" is assigned as a parameter indicating the background subtraction method and "2" is assigned as a parameter indicating the inter-frame difference method. The space setting unit 110 sets 2 as a processing parameter in the subspace 330 whose corresponding subject information is "player", and sets 1 as a processing parameter in the subspace 320 whose subject information is "ball". To do. As a result, the shape estimation unit 130 can accurately generate a foreground image according to the subject.

Further, as the processing parameter when the shape estimation unit 130 generates the two-dimensional shape data, the space setting unit 110 sets the processing parameter used for the noise removal processing of the foreground image described above. Also in this case, different processing parameters are set according to the subject information (for example, "ball" and "player") corresponding to the plurality of subspaces (for example, the subspace 320 and the subspace 330). As a result, the shape estimation unit 130 can remove a region extracted as the foreground but having an extremely different size (pixel area) from the foreground region as a noise region when generating the two-dimensional shape data. Image accuracy is improved.

FIG. 6 is a flowchart for explaining the process performed by the shape estimation device 1. The process shown in FIG. 6 is executed by the CPU 511 reading and executing the program stored in the ROM 512 or the auxiliary storage device 514. In the following description, as an example of processing, when the subspace 320 and the subspace 330 included in the imaging space 300 shown in FIG. 5A are associated with subject information indicating a “ball” and a “player”, respectively. The processing of the shape estimation device 1 in the above will be described. The shape estimation device 1 starts the process by communicating data such as subject information with the image pickup system 2. In the following description, the processing step is simply referred to as S.

In S600, the subject information acquisition unit 100 acquires the identification numbers and the subject information of each of the plurality of imaging devices 20 from the imaging system 2. The subject information acquisition unit 100 transmits the acquired identification number and subject information to the space setting unit 110. In S601, the space setting unit 110 acquires information for determining the imaging space 300 and the boundary 310 from the auxiliary storage device 514, an external storage device, or the like. Further, the space setting unit 110 sets the subspace based on the acquired information. In this process, the boundary 310 is assumed to be "z = 2 m", and the subspace 320 and the subspace 330 are set.

In S602, the space setting unit 110 associates the subject information with each of the subspace 320 and the subspace 330 set in S601 based on the preset information. The preset information referred to here is information that associates the "ball" as subject information with respect to the subspace above the z-coordinate indicating the boundary 310, and "player" with respect to the subspace below the z-coordinate indicating the boundary 310. Is included as subject information. Therefore, in this process, the "ball" is associated with the subspace 320 above "z = 2m" indicated by the boundary 310, and the "player" corresponds to the subspace 330 below "z = 2m". Attached. The space setting unit 110 transmits information indicating each subspace and the identification number of the imaging device associated with the subject information associated with each subspace to the shape estimation unit 130. As the preset information, information indicating which subject information is associated with which subspace may be set in accordance with the information for determining the boundary 310.

In S603, the imaging information acquisition unit 120 acquires the foreground image generated by the imaging system 2 and the imaging parameters of each imaging device 20. If the installation state of the imaging device 20 does not change, the imaging parameters may be acquired only once and stored in the auxiliary storage device 514 or the like. However, it is assumed that the installation state of the image pickup apparatus 20 may change due to contact with wind or an obstacle. In this case, the imaging information acquisition unit 120 can calculate the imaging parameters as needed. The imaging information acquisition unit 120 transmits the acquired foreground image and imaging parameters to the shape estimation unit 130 and the image generation device 3. The image pickup information acquisition unit 120 may be configured to acquire an image captured from the image pickup system 2 and transmit it to the shape estimation unit 130 and the image generation device 3. In this case, the shape estimation unit 130 generates a foreground image based on the acquired captured image. Here, the foreground image is generated by the image pickup apparatus 20 or the shape estimation unit 130 using appropriate processing parameters according to the subject information associated with the corresponding imaging apparatus.

In S604, the shape estimation unit 130 identifies the image pickup device 20 to be used for the shape estimation process based on the information indicating the subspace and the identification number of the image pickup device 20 associated with each subspace. In this process, since the boundary 310 is set to "z = 2 m", the shape estimation unit 130 determines whether or not z> 2 m is satisfied with respect to the coordinates of the representative point of the voxel to be the target of the shape estimation process. judge. When z> 2m is satisfied, the shape estimation unit 130 determines that the voxel is included in the subspace 320, and proceeds to S605 for processing. If z> 2m is not satisfied, the shape estimation unit 130 determines that the voxel is included in the subspace 330, and proceeds to S606. The shape estimation unit 130 determines whether or not the voxel to be processed is included in the subspace 320, proceeds to S605 when it is determined to be included, and S606 when it is determined not to be included. It may be configured to proceed with processing.

In S605, the shape estimation unit 130 shape-estimates the image pickup device 20 having an identification number associated with the same subject information as the subject information (in this case, the subject information is a "ball") corresponding to the subspace 320. Determined to be used for processing.

In S606, the shape estimation unit 130 shape-estimates the image pickup device 20 having an identification number associated with the same subject information as the subject information (in this case, the subject information is "player") corresponding to the subspace 330. Determined to be used for processing.

In S607, the shape estimation unit 130 generates a silhouette image based on the foreground image acquired from the image pickup apparatus 20 determined to be used in S605 or S606, and performs shape estimation processing using the generated silhouette image. At this time, the shape estimation unit 130 determines whether or not the voxel is a part of the shape of the subject by using different shape estimation parameters according to the subject information associated with each subspace. For example, if it is determined in S605 that the image pickup device 20 corresponding to the "ball" is used, 1 is used as the shape estimation parameter, and if it is determined in S606 that the image pickup device 20 corresponding to the "player" is used, the shape. 0 is used as the estimation parameter. The shape estimation unit 130 determines whether or not the pixel value on the silhouette image corresponding to the coordinates obtained by converting the voxel to be processed into the captured image system is the value corresponding to the foreground region (pixel value 1 in the above example). judge. When the shape estimation parameter used is 1, if the voxel to be processed is determined to have the above coordinates in the foreground region in all the captured image systems, the shape estimation unit 130 sets the voxel to be processed as the subject. Determined to be part of the shape. On the other hand, when there is an captured image in which the above coordinates are determined to be a region other than the foreground region (the pixel value on the silhouette image corresponding to the coordinates is 0), the shape estimation unit 130 sets the voxel to be processed as the subject. Judge that it is not part of the shape. When the value of the shape estimation parameter used is 1, the shape estimation unit 130 targets the voxel to be processed when the number of captured images determined to be in the region other than the foreground region is 1 or less. It is determined that it is a part of. When the value of the shape estimation parameter used is 1, the shape estimation unit 130 targets the voxel to be processed when the number of captured images determined to be in the region other than the foreground region is two or more. Judge that it is not a part of.

In S608, the shape estimation unit 130 determines whether or not the processing of all voxels in the imaging space has been completed. When it is determined that the unprocessed voxels exist, the shape estimation unit 130 performs the processing after S604 again. When it is determined that the processing of all voxels is completed, the shape estimation unit 130 proceeds to processing to S609.

In S609, the shape estimation unit 130 generates shape data based on the result of the shape estimation process and transmits it to the image processing device 3. When the shape estimation unit 130 transmits the shape data, the process ends.

The above is an example of the processing performed by the shape estimation device 1. The shape estimation device 1 can perform the same processing even when the type or boundary of the subject is different from the above example and when the subspace is three or more. In this case, in S604, the boundary determination is performed according to the number of subspaces. Further, the shape estimation unit 130 performs the shape estimation process by using an image pickup device having subject information corresponding to the subspace including the voxels based on the result of the determination in S604. Further, as the shape estimation parameter used in the shape estimation process, any value can be used according to the type of the subject and the subspace.

As described above, the space setting unit 110 in the present embodiment sets the processing parameters used for the process of generating the shape data of the subject differently for the plurality of subspaces included in the imaging space. According to the above configuration, since the shape data can be generated by using appropriate processing parameters for each subspace, it is possible to accurately generate the shape data of the subject in the entire imaging space.

<How to generate a virtual viewpoint image>
The virtual viewpoint image generation process performed by the image generation device 3 will be described. The image generation device 3 executes a process of generating a virtual viewpoint image of the foreground and a process of generating a virtual viewpoint image of the background by using the foreground image, the imaging parameter, and the shape data transmitted from the shape estimation device 1. The image generation device 3 generates a virtual viewpoint image by synthesizing the generated virtual viewpoint image of the foreground and the virtual viewpoint image of the background. Hereinafter, each generation process will be described.

A method of generating a virtual viewpoint image of the foreground will be described. The virtual viewpoint image of the foreground is generated by coloring each voxel in the shape data. A method of calculating the color to be colored in voxels will be described. The image generation device 3 calculates the distance d from the image pickup device 20 to the voxel on the surface of the shape data of the subject based on the shape data and the imaging parameter, and creates a distance image in which the calculated distance and the pixel value are associated with each other. Generate. Further, the image generation device 3 converts the coordinates Xw into the coordinates Xi on the captured image (distance image) by using the imaging parameters of the imaging device 20 including the coordinates Xw within the angle of view for a certain coordinate Xw in the imaging space. To do. Further, the image generation device 3 calculates the distance dx between the coordinates Xw and the image pickup device 20 including the above coordinates Xw within the angle of view. The image generation device 3 compares the distance d with the distance dx by referring to the pixel values of the coordinates Xi on the distance image. When the difference between the distance d and the distance dx is equal to or less than a predetermined threshold value, the coordinates Xw are determined to be visible from the image pickup apparatus 20. The predetermined threshold value at this time is referred to as a visibility determination parameter in the following description. When the coordinates Xw are visible, the image value at the coordinates Xi on the captured image corresponding to the coordinates Xw is calculated as the surface color of the shape data.

By performing the above processing on the plurality of imaging devices 20, a plurality of coordinates determined to be visible can be specified for one voxel on the surface of the shape data. The image generation device 3 determines and colors the voxel color by calculating (blending) the average value of the pixel values corresponding to the specified plurality of coordinates. By performing the above processing for all voxels in the shape data, the shape data is colored and a virtual viewpoint image of the foreground is generated. As the visibility determination parameter used in the above process, different values can be used for each subspace. The smaller the visibility determination parameter, the fewer coordinates are determined to be visible, so the effect of blending fewer colors and making the colors clearer can be expected. Therefore, for example, when coloring the shape data in the subspace corresponding to the "player", by setting the visibility judgment parameter to a small value, the color of the player becomes clear and a high-quality virtual viewpoint image is generated. Can be done. On the other hand, the larger the visibility determination parameter, the more coordinates are determined to be visible, so that many colors are blended. For example, in the case of a subject such as a ball, which is small in size and can move at high speed, the color of the subject does not have to be calculated exactly. Therefore, when coloring the shape data in the subspace corresponding to the "ball", the visibility determination parameter is set to a large value.

The same value may be used for the visibility determination parameter in the entire imaging space. Further, the color calculation method is not limited to the above, and various methods may be used, for example, using the color of the captured image acquired by the imaging device 20 closest to the designated virtual viewpoint. Further, in the present embodiment, the above processing is performed using the imaging device 20 that includes the coordinates of the processing target in the angle of view among all the imaging devices 20, but corresponds to the subspace including the coordinates of the processing target. Only the imaging device 20 may be used.

Next, a method of generating a virtual viewpoint image of the background will be described. The virtual viewpoint image of the background is generated using the three-dimensional shape data of the background. As the background three-dimensional shape data, a CG model of a stadium or the like that is generated in advance and stored in a storage device or the like is used. The CG model reproduces the shape of the background with a plurality of surfaces. At this time, the image generation device 3 identifies the imaging device 20 that includes the surface in the angle of view and faces the surface most by comparing the normal vector of the surface with the imaging direction of the imaging device 20. The image generation device 3 generates a texture image corresponding to the surface using the image captured by the specified image pickup device 20, and attaches the texture to the surface by using an existing texture mapping method. By performing the above processing on each surface, a virtual viewpoint image of the background is generated.

The image generation device 3 generates a virtual viewpoint image by synthesizing the virtual viewpoint image of the foreground and the virtual viewpoint image of the background generated by the above-described processing. Further, the image generation device 3 accepts a user operation for designating the viewpoint position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, or the like by an input device or the like connected to the inside or the outside of the image generation device 3, or is virtual. Obtain information for designating the viewpoint from a storage device or the like. By performing the above processing, the image generation device 3 can generate a virtual viewpoint image viewed from a designated viewpoint and display it on the display device 4.

(Second embodiment)
In the present embodiment, the shape estimation device 7 that generates shape data using a plurality of imaging systems will be described. The hardware configuration, functional configuration, and processing are designated by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

FIG. 7 is a diagram for explaining the configuration of the image processing system 1100 including the shape estimation device 7. The image processing system 1100 includes a shape estimation device 7, a first image pickup system 8, a second image pickup system 9, an image generation device 3, and a display device 4. Since the hardware configuration of the shape estimation device 7 is the same as that of the first embodiment, the description thereof will be omitted.

The first imaging system 8 and the second imaging system 9 are systems composed of a plurality of imaging devices having the same subject information, respectively. FIG. 8 is a diagram for explaining an example of the configuration of the first imaging system 8 and the second imaging system 9. FIG. 8 shows a first imaging system 8 composed of four imaging devices 80a to 80d and a second imaging system 9 composed of three imaging devices 90a to 90c. The number of imaging devices included in each imaging system is not limited to this. The imaging devices 80a to 80d included in the first imaging system have the same subject information (for example, a player or the like), and are installed so as to mainly capture a subject indicated by the subject information. Similarly, the imaging devices 90a to 90d included in the second imaging system have the same subject information (for example, a ball or the like) and are installed so as to mainly capture a subject indicated by the subject information. The image pickup system 8 and the image pickup system 9 generate a foreground image in the separation units 81a to 81d and the separation units 91a to 91 to c in the respective systems, and transmit the foreground image to the shape estimation device 7. In the following description, unless otherwise specified, the image pickup devices 80a to 80d and the image pickup devices 90a to 90c are simply referred to as the image pickup device 80 and the image pickup device 90.

Returning to FIG. 7, the functional configuration of the shape estimation device 7 will be described. The shape estimation device 1 includes a subject information acquisition unit 100, a space setting unit 110, a first imaging information acquisition unit 700, a second imaging information acquisition unit 710, and a shape estimation unit 130. Here, a processing unit different from the shape estimation device 1 in the first embodiment will be described.

The first imaging information acquisition unit 700 acquires the foreground image generated by the first imaging system 8 and the imaging parameters of each imaging device 20 included in the first imaging system 8. The second imaging information acquisition unit 710 acquires the foreground image generated by the second imaging system 9 and the imaging parameters of each imaging device 20 included in the second imaging system 9. The first imaging information acquisition unit 700 and the second imaging information acquisition unit 710 transmit the acquired foreground image and imaging parameters to the shape estimation unit 130 and the image generation device 3, respectively.

FIG. 9 is a flowchart for explaining the process performed by the shape estimation device 7. Instead of S603, S605 and S606 in the flowchart shown in FIG. 6, the processes of S900, S901 and S902 are performed. In the following description, as in FIG. 6, when the subspace 320 and the subspace 330 included in the imaging space 300 shown in FIG. 5A are associated with subject information indicating a “ball” and a “player”, respectively. The processing of the shape estimation device 7 in the above will be described. Further, a case where the imaging device 80 included in the first imaging system 8 has a “player” as subject information and the imaging device 90 included in the second imaging system 9 has a “ball” as subject information will be described. .. Hereinafter, the points different from the processing shown in FIG. 6 will be described.

In S900, the first imaging information acquisition unit 700 acquires the foreground image generated by the first imaging system 8 and the imaging parameters of each imaging device 80. In addition, the second imaging information acquisition unit 710 acquires the foreground image generated by the second imaging system 9 and the imaging parameters of each imaging device 90. The first imaging information acquisition unit 700 and the second imaging information acquisition unit 710 transmit the acquired foreground image and imaging parameters to the shape estimation unit 130, respectively.

In S604, as in FIG. 6, it is determined in which subspace the voxel is contained. When the boundary is set to "z = 2m", the shape estimation unit 130 determines whether or not z> 2m is satisfied with respect to the coordinates of the representative point of the voxel to be processed for shape estimation. When z> 2m is satisfied, the shape estimation unit 130 determines that the voxel is included in the subspace 320, and proceeds to S901 for processing. If z> 2m is not satisfied, the shape estimation unit 130 determines that the voxel is included in the subspace 330, and proceeds to S902. In S901, the shape estimation unit 130 determines that the image pickup device 90 included in the second imaging system 9 is used for the shape estimation process because the subject information corresponding to the subspace 320 is a “ball”. In S902, the shape estimation unit 130 determines that the image pickup device 80 included in the first imaging system 8 is used for the shape estimation process because the subject information corresponding to the subspace 320 is the “player”.

After that, as in the first embodiment, the shape estimation process is performed using different shape estimation parameters for each subspace, and the shape data of the subject is generated. Similar to the first embodiment, the process shown in FIG. 9 can be applied even when the type or boundary of the subject is different from the above example and when the subspace is three or more. As described in the present embodiment, by using an imaging system composed of imaging devices having the same subject information, it becomes easy to specify the imaging device used for the shape estimation process. Further, since the image pickup apparatus included in the image pickup system has the same subject information, the same processing parameters can be assigned to each image pickup system. In the present embodiment, the case where the image processing system 1100 includes two imaging systems has been described, but an arbitrary number of imaging systems may be included depending on the type of the subject.

(Other embodiments)
In the above-described embodiment, an example in which the imaging space is divided into a plurality of subspaces based on the boundary and processing parameters are set for each subspace has been described. Here, an example of setting processing parameters other than the above will be described. For example, in the imaging space 300 shown in FIG. 5A, when the subject is imaged across the boundary 310, different processing parameters are used for the voxels included in the subspace 320 and the voxels included in the subspace 330. Then, the shape data is generated. Therefore, the shape of the generated subject may be distorted near the boundary 310. In order to solve this problem, the space setting unit 110 sets the processing parameters so that the processing parameters near the boundary 310, for example, change slowly. For example, when the boundary 310 is z = 3 m, the space setting unit 110 sets the shape estimation parameter as 1 for z = 3 ± 0.3 m, which is the space near the boundary (hereinafter referred to as the boundary space). Further, the space setting unit 110 sets 2 as a shape estimation parameter in the subspace 320 and 0 as a shape estimation parameter in the subspace 330. By inclining the parameters in this way, it is possible to reduce the distortion of the shape data near the boundary. The setting of the boundary space is not limited to the above values.

In addition, when it is possible to identify that the subject straddles the boundary based on the position of the subject during the shape data generation process, the shape data generation process near the boundary is performed using the same processing parameters. You may. In this case, for example, a processing parameter corresponding to a partial region including a larger portion of the subject straddling the boundary is used.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1 Shape estimation device 110 Space setting unit 130 Shape estimation unit

Claims (15)

A first subspace corresponding to one or more captured images acquired from one or more imaging devices among a plurality of imaging devices and a first subspace included in a plurality of subspaces in the imaging space imaged by the plurality of imaging devices. A first generation means for generating shape data of a subject included in the first subspace based on the parameter of 1.
It is a second parameter corresponding to one or more captured images acquired from one or more imaging devices among the plurality of imaging devices and a second subspace included in the plurality of subspaces, and is the first parameter. A shape data generation device comprising a second generation means for generating shape data of a subject included in the second subspace based on a second parameter different from the parameter of the above. The first parameter and the second parameter according to claim 1, wherein the first parameter and the second parameter include a parameter for determining whether or not the element constituting the imaging space is a part of the shape of the subject. Shape data generator. The first generation means and the second generation means are image data generated based on the one or more captured images, and are based on the image data indicating a region of a subject in the one or more captured images. The shape data generation device according to claim 1 or 2, wherein the shape data of the subject is generated. The first to third image pickup apparatus according to claim 1 to 3, wherein the one or more image pickup apparatus for imaging the first subspace includes an image pickup apparatus having a different angle of view from the one or more image pickup apparatus for imaging the second subspace. The shape data generation device according to any one of the items. Any of claims 1 to 4, wherein the imaging space is divided into a plurality of subspaces including the first subspace and the second subspace based on the information indicating the boundary. The shape data generation device according to item 1. The shape data generation device according to claim 5, wherein the information indicating the boundary includes information indicating a height in the imaging space. Claims 1 to 6 are characterized in that the subject that can be included in the first subspace and the subject that can be included in the second subspace are different in at least one of a size and a motion characteristic, respectively. The shape data generation device according to any one of the above items. The shape data generation device according to any one of claims 1 to 7, wherein the number of subjects included in the first subspace is different from the number of subjects included in the second subspace. .. A first subspace corresponding to one or more captured images acquired from one or more imaging devices among a plurality of imaging devices and a first subspace included in a plurality of subspaces in the imaging space imaged by the plurality of imaging devices. A first generation step of generating shape data of a subject included in the first subspace based on the parameter of 1.
It is a second parameter corresponding to one or more captured images acquired from one or more imaging devices among the plurality of imaging devices and a second subspace included in the plurality of subspaces, and is the first parameter. A second generation step of generating shape data of a subject included in the second subspace based on a second parameter different from the parameter of the above.
A shape data generation method characterized by having. The ninth aspect of claim 9, wherein the one or more imaging devices that image the first subspace include one or more imaging devices that image the second subspace and an imaging device having a different angle of view. Shape data generation method. The 9 or 10 according to claim 9, wherein the imaging space is divided into a plurality of subspaces including the first subspace and the second subspace based on the information indicating the boundary. Shape data generation method. The shape data generation method according to claim 11, wherein the information indicating the boundary includes information indicating a height in the imaging space. Claims 9 to 12, wherein the subject that can be included in the first subspace and the subject that can be included in the second subspace are different in at least one of the size and the motion characteristic, respectively. The shape data generation method according to any one of the above items. The shape data generation method according to any one of claims 9 to 13, wherein the number of subjects included in the first subspace is different from the number of subjects included in the second subspace. .. A computer program for causing a computer to function as the shape data generator according to any one of claims 1 to 8.

Images