JP2022029239A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To control a processing load related to three-dimensional shape data.SOLUTION: A three-dimensional model generation device 104 has: a controlled foreground selection unit 902 that selects objects in a first area in a frame and objects in a second area that is an area other than the first area; a controlled interval determination unit 903 that determines a first frame rate for generating three-dimensional shape data of the objects in the first area, and determines a second frame rate for generating three-dimensional shape data of the objects in the second area to be a frame rate different from the first frame rate; a model generation unit 905 that generates the three-dimensional shape data by using object images; and a data deletion unit 904 that controls such that the three-dimensional shape data of the objects in the first area is generated at the first frame rate, and controls such that the three-dimensional shape data of the objects in the second area is generated at the second frame rate.SELECTED DRAWING: Figure 9

Description

The technique of the present disclosure relates to the generation of three-dimensional shape data of an object.

There is a technique in which a plurality of cameras are installed at different positions, images are taken synchronously from a plurality of viewpoints, and a virtual viewpoint image is generated using a plurality of images obtained by the imaging. In the generation of the virtual viewpoint image, the three-dimensional shape data of the target object is generated. In addition, in the generation of 3D shape data, the amount of calculation depends on factors such as the number and size of target objects, the number of image pickup devices, the size of box cells that make up the 3D shape, and the size of the target 3D space. May be large.

As a method of reducing the amount of calculation, Patent Document 1 describes that a target object is formed by a large number of minimum voxel blocks by repeating voxel division for a voxel that overlaps with a target object in the measurement target space divided by the maximum voxel. It describes how to reconstruct it.

Japanese Unexamined Patent Publication No. 2001-307073

The captured image may include a high-importance area such as before a goal in a soccer match and a low-importance area such as outside the field, and the target object is not only a high-importance area but also a high-importance area. It may also be located in less important areas. Therefore, when the three-dimensional shape data of a plurality of target objects included in the captured image is generated by the method described in Patent Document 1, the processing load of the region of low importance is not sufficiently suppressed, and the three-dimensional shape data is obtained. There is a risk that the processing load involved will increase.

An object of the present disclosure is to suppress a processing load related to three-dimensional shape data.

The image processing apparatus according to the present disclosure is an image processing that generates three-dimensional shape data of an object included in the frame by using an object image showing the shape of the object in the frame obtained by imaging by a plurality of imaging devices. A selection means for selecting an object in the first region in the frame and an object in the second region which is an region other than the first region, and an object in the first region. The first frame rate for generating the three-dimensional shape data is determined, and the second frame rate for generating the three-dimensional shape data of the object in the second region is set to a frame rate different from the first frame rate. A determination means for determining, an image generation means for generating the object image corresponding to the frame, and a data generation means for generating the three-dimensional shape data using the object image generated by the image generation means. The image generation means controls so that the three-dimensional shape data of the object in the first region is generated at the first frame rate, and the second region is generated at the second frame rate. It is characterized by controlling so that the three-dimensional shape data of the object of is generated.

According to the technique of the present disclosure, it is possible to suppress the processing load related to the three-dimensional shape data.

The figure which shows the structure of the virtual viewpoint image generation system. The figure which showed the example of the camera arrangement of the virtual viewpoint image generation system. The figure showing the foreground detected from the captured image. A diagram showing a rectangular texture and a rectangular mask. A diagram showing a foreground mask image. The figure for demonstrating the generation method of the 3D model by the visual volume crossing method. A diagram for explaining a three-dimensional model of the foreground by voxels. The figure which showed the hardware composition of the 3D model generator. The figure which shows the functional structure of a 3D model generator. Diagram to explain the background model and importance information. A flowchart showing the process of generating a three-dimensional model. The figure which showed the importance area and the mask image. The figure for demonstrating the deletion of the rectangular mask of the low importance area. The figure for demonstrating the composition of a three-dimensional model. The figure which shows the functional structure of a 3D model generator. A flowchart showing the process of generating a three-dimensional model. The figure for explaining the importance information according to the number of voxels. The figure for demonstrating the importance area according to the distance from a ball model.

Hereinafter, the technique of the present disclosure will be described in detail based on the embodiments with reference to the accompanying drawings. The following embodiments are merely examples of embodiment of the techniques of the present disclosure and are not intended to limit the technical scope of the present disclosure. The techniques of the present disclosure can be implemented in various forms without departing from their technical ideas or their key features.

<First Embodiment>
In the first embodiment, in the process of generating a three-dimensional model of a plurality of objects included in a series of frames, an update of the three-dimensional model of an object located in a preset region of low importance among the plurality of objects. Performs processing to reduce the frequency (generation frequency). In this embodiment, a method of suppressing the processing load related to the three-dimensional model by this processing will be described.

[System configuration]
FIG. 1 is a block diagram showing a configuration of a virtual viewpoint image generation system 100 of the present embodiment for generating a virtual viewpoint image. The virtual viewpoint image is an image generated based on the position and orientation of a non-existent virtual camera different from the real camera, and is also called a free viewpoint image or an arbitrary viewpoint image. According to the technology for generating a virtual viewpoint image, for example, a highlight scene of soccer or basketball can be viewed from various angles, so that a user can be given a high sense of presence as compared with a normal image. The virtual viewpoint image may be a moving image or a still image. In the present embodiment, the virtual viewpoint image will be described as a moving image.

The virtual viewpoint image generation system 100 of the present embodiment includes a camera array 101 having a plurality of cameras 101a to 101p, a foreground extraction device group 102 having a plurality of foreground extraction devices 102a to 102r, and a control device 103. Further, the virtual viewpoint image generation system 100 includes a three-dimensional model generation device 104 and a rendering device 105. The foreground extraction devices 102a to 102p, the control device 103, the three-dimensional model generation device 104, and the rendering device 105 are general image processing including a CPU for performing arithmetic processing, a memory for storing arithmetic processing results, programs, and the like. Realized by the device.

The camera array 101 is composed of a plurality of cameras 101a to 101p, captures an object from a plurality of directions at various angles, and outputs image data of the captured image to the foreground extraction device group 102.

FIG. 2 is a diagram showing the arrangement of all 16 cameras 101a to 101p constituting the camera array 101 in a bird's-eye view of the field as viewed from directly above. As shown in FIG. 2, the cameras are arranged around the stadium, and images are taken in time synchronization from various angles toward the gazing point on the field common to all cameras 101a to 101p. However, in the present disclosure, some of the cameras 101a to 101p may be arranged toward different gazing points, or all the cameras 101a to 101p may be arranged toward different positions. Further, some of the cameras 101a to 101p may be arranged toward the same gazing point, and the remaining cameras may be arranged toward different positions from each other.

The foreground extraction device group 102 is composed of foreground extraction devices 102a to 102p corresponding to the respective cameras 101a to 101p. Each foreground extraction device 102a to 102p extracts a foreground region showing a silhouette of an object included in the captured image from the image data of the captured image output from the corresponding camera.

The object that is the foreground of the captured image is a subject that can be viewed from an arbitrary angle of the virtual viewpoint, and refers to, for example, a person existing on the field of the stadium. Alternatively, the object may be an object having a predetermined image pattern, such as a ball or a goal. Further, the object may be a moving body or a stationary body.

Then, each of the foreground extraction devices 102a to 102p generates a rectangular mask showing the foreground region and the other regions of the captured image obtained by the corresponding camera, and a rectangular texture which is an image showing the texture of the object. do.

FIG. 3 is a diagram for explaining the extraction process of the foreground region in the foreground extraction devices 102a to 102p. FIG. 3A is an example of an captured image obtained by taking an image with the camera 101d of FIG. 2, and the captured image 300 includes a foreground 3a-3e showing five objects. The foreground extraction device 102d extracts the foreground region included in the captured image 300 of the camera 101d, and calculates five rectangular regions 4a-4e corresponding to the foreground 3a-3e as shown in FIG. 3B. As a method for extracting the foreground region, for example, a method of determining a region having a large difference in brightness and color between the held background image and the captured image as the foreground is used. The background refers to an area other than the foreground of the captured image.

FIG. 4 is a diagram showing a rectangular texture and a rectangular mask. FIG. 4A is a diagram showing a plurality of images obtained by cutting out a rectangular region including the foreground shown in FIG. 3B, and this image is referred to as a rectangular texture. Further, in FIG. 4 (b), among the rectangular areas including the foreground shown in FIG. 3 (b), the foreground area showing the object is white and the background area is black, and the binary silhouette for each object is shown. It is a figure which shows the image 5a-5e. This binary image is called a rectangular mask. The rectangular texture and the rectangular mask are transmitted to the 3D model generator 104 together with the coordinates of the rectangular area.

The control device 103 calculates camera parameters indicating the positions and orientations of the cameras 101a to 101p from the image data of the captured images captured by the cameras of the camera array 101 in time synchronization, and the three-dimensional model generation device 104 and the rendering device 105. Output to.

Camera parameters are composed of external parameters and internal parameters. The external parameters are composed of a rotation matrix and a parallel traveling matrix, and indicate the position and orientation of the camera. On the other hand, the internal parameters include the focal length of the camera, the optical center, and the like, and indicate the angle of view of the camera and the size of the image pickup sensor. The process of calculating camera parameters is called calibration, which includes points in a three-dimensional world coordinate system acquired using multiple images of a specific pattern such as a checkerboard, and points on the corresponding two dimensions. It is obtained by using the correspondence of.

The three-dimensional model generation device 104 acquires camera parameters from the control device 103, acquires a rectangular mask from the foreground extraction device group 102, reconstructs a plurality of rectangular mask images, and generates mask images for each of a plurality of cameras. ..

FIG. 5 is a diagram showing a foreground mask image (also referred to as an object image) showing the silhouette of an object, which corresponds to the captured image 300 obtained by the camera 101d. The foreground mask image is an image showing an area of an object in the captured image, and is an image reconstructed by arranging a rectangular mask at the position of the object in the captured image 300 based on the rectangular mask and the coordinate information of each rectangular mask. be. As shown in FIG. 4B, the rectangular mask is an image of each object cut out by a rectangle containing each object in the captured image, whereas the foreground mask image is a plurality of rectangles as shown in FIG. It is an image synthesized by pasting the mask on the cut out coordinates. Similar to the rectangular mask, the foreground mask image is a binary silhouette image in which the foreground area of the captured image is represented by white and the background area is represented by black. In the present embodiment, the foreground mask image will be described as being generated by the three-dimensional model generation device 104, but may be generated by the foreground extraction devices 102a to 102p.

Further, the three-dimensional model generation device 104 generates a three-dimensional model of an object represented by a voxel set by a visual volume crossing method in a three-dimensional space using foreground mask images of a plurality of cameras. The generation of 3D models and voxels will be described later. The three-dimensional model generation device 104 outputs the data of the three-dimensional model of the generated object to the rendering device 105.

The rendering device 105 acquires a three-dimensional model of the object and a foreground texture image. The foreground texture image is an image reconstructed so as to arrange the rectangular texture at the position of the object in the captured image based on the rectangular texture and the coordinate information of each rectangular texture. Further, the rendering device 105 acquires camera parameters from the control device 103. The rendering device 105 generates a virtual viewpoint image based on these data. Specifically, the positional relationship between the foreground texture image and the three-dimensional model of the object is obtained from the camera parameters. Then, the three-dimensional space is reconstructed by coloring based on the color of the pixels in the foreground texture image corresponding to each voxel constituting the three-dimensional model, and an image viewed from an arbitrary viewpoint is generated.

In this embodiment, it is assumed that the foreground extraction devices 102a to 102p and the three-dimensional model generation device 104 are connected in a star-shaped topology. In addition, the foreground extraction devices 102a to 102p and the three-dimensional model generation device 104 may be connected in a topology such as a ring type or a bus type by daisy chain connection.

[How to generate a 3D model]
Next, the outline of the 3D model generation process of the object performed in the 3D model generation device will be described. The 3D shape of an object represented by a voxel in the 3D shape data is also called a 3D model of the object (foreground). Here, the generation of a three-dimensional model of an object by the visual volume crossing method will be described.

FIG. 6 is a diagram showing the basic principle of the visual volume crossing method. FIG. 6A is a diagram when a target object C, which is an object, is imaged by a camera, which is an image pickup device. As described above, a foreground mask including a two-dimensional silhouette (foreground region) of the target object C by binarizing it based on the difference in color or brightness between the captured image obtained by imaging the target object C and the background image. An image is obtained.

FIG. 6B is a diagram showing a cone extending in a three-dimensional space so as to pass through each point on the contour of the two-dimensional silhouette Da from the projection center (Pa) of the camera. This cone is called the visual volume Va by the camera. FIG. 6C is a diagram showing how a three-dimensional model of an object can be obtained from a plurality of visual volumes. As shown in FIG. 6 (c), a plurality of visual volumes for each camera are obtained from a two-dimensional silhouette Da based on images synchronously captured by a plurality of different cameras having different positions. In the generation of the three-dimensional model by the visual volume intersection method, the three-dimensional model of the target object is generated by finding the intersection (common area) of the visual volumes of a plurality of cameras. The generated 3D model is represented by a set of voxels.

FIG. 7 is a diagram for explaining voxels. A voxel is a minute cube as shown in FIG. 7 (a). FIG. 7B shows the target space of the camera that generates the three-dimensional model as a set of voxels. When one of the voxels in the target space, which is the voxel to be processed, is projected onto the foreground mask image of each camera, it is determined whether or not the projection of the focused voxel fits within the foreground area of the foreground mask image of each camera. Will be done. As a result of this determination, if the projection of the voxel of interest deviates from the foreground region, the voxel of interest is deleted.

By cutting the voxels that do not fit in the foreground area of the camera, a three-dimensional model of the target object of the quadrangular pyramid shown in FIG. 7 (c) is generated by the voxels as shown in FIG. 7 (d). By assigning the pixel value of the texture image corresponding to the position of each voxel constituting the generated 3D model to the voxel, it is possible to construct the 3D shape of the colored object in the 3D space.

Due to factors such as the number and size of target objects, the number of cameras, the size of voxels, and the size of the target 3D space, the amount of calculation for processing related to the 3D model increases, making it difficult to generate virtual viewpoint images. May be. In particular, the fineness of the 3D model, the load at the time of generating the 3D model, or the processing load related to the 3D model is affected by the number of voxels constituting the 3D model.

In this embodiment, the three-dimensional model of the object is represented as a cubic voxel, but the present invention is not limited to this. Alternatively, for example, a three-dimensional model may be represented by a point cloud by using points as elements constituting the three-dimensional space.

[Hardware configuration of 3D model generator]
FIG. 8 is a block diagram showing a hardware configuration example of the three-dimensional model generation device 104 of the present embodiment. The three-dimensional model generation device 104 includes a CPU 801 and a ROM 802, a RAM 803, a storage device 804, a display unit 805, an operation unit 806, and a communication unit 807.

The CPU 801 is a central processing unit, and controls the entire three-dimensional model generation device 104 by executing a program stored in the ROM 802 or the RAM 803.

ROM 802 is a read-only non-volatile memory. The RAM 803 is a memory that can be read and written at any time. As the RAM 803, for example, a DRAM (Dynamic Random Access Memory) can be used. The storage device 804 is a large-capacity storage device composed of, for example, a hard disk. The storage device 804 can store rectangular masks and the like obtained from the foreground extraction devices 102a to 102p.

The display unit 805 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the three-dimensional model generation device 104. The operation unit 806 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 801 in response to an operation by the user. The CPU 801 also operates as a display control unit that controls the display unit 805 and an operation control unit that controls the operation unit 806. In the present embodiment, it is assumed that the display unit 805 and the operation unit 806 exist inside the three-dimensional model generation device 104, but at least one of the display unit 805 and the operation unit 806 is separated from the outside of the three-dimensional model generation device 104. It may exist as a device of.

The communication unit 807 controls communication between the three-dimensional model generation device 104 and the external device. For example, when the communication unit 807 performs communication control, the three-dimensional model generation device 104 is connected to the foreground extraction device group 102 and the rendering device 105 via a network such as a LAN.

Each functional unit of FIG. 9, which will be described later, realized in the three-dimensional model generation device 104 is realized by the CPU 801 executing a predetermined program, but is not limited thereto. In addition, for example, hardware such as a GPU (Graphics Processing Unit) for speeding up an operation or an FPGA (Field Programmable Gate Array) may be used. That is, each functional unit of the three-dimensional model generator 104 may be realized by the cooperation of software and hardware such as a dedicated IC, or even if some or all of the functions are realized only by hardware. good. Further, the processing of each functional unit may be distributed and executed by using a plurality of three-dimensional model generation devices 104.

[Functional configuration of 3D model generator]
FIG. 9 is a block diagram showing a functional configuration of the three-dimensional model generation device 104 in the present embodiment. The three-dimensional model generation device 104 of the present embodiment includes an acquisition unit 901, a suppression foreground selection unit 902, a suppression interval determination unit 903, a data deletion unit 904, a model generation unit 905, a data management unit 906, and a data synthesis unit 907. ..

The acquisition unit 901 is important to indicate the camera parameters of the cameras 101a to 101p constituting the camera array 101, the background model showing the three-dimensional shape of the imaging environment such as the stadium, and the importance of the area in the imaging environment such as the stadium. The degree information is acquired from the control device 103. The importance information will be described later. Further, the acquisition unit 901 extracts the foreground of the rectangular texture shown in FIG. 4A and the rectangular mask shown in FIG. 4B obtained by extracting the foreground from the captured images of the cameras constituting the camera array 101. Obtained from the device group 102. The acquisition unit 901 outputs the camera parameters, the background model, and the importance information to the suppression foreground selection unit 902. Further, the acquisition unit 901 outputs the camera parameters, the rectangular texture, and the rectangular mask to the data deletion unit 904.

The suppression foreground selection unit 902 sets the importance in the region in the captured image of each camera 101a to 101p by using the background model, the camera parameters, and the importance information. As a method of setting the importance of a region in a captured image, for example, a region corresponding to the importance is arranged on a background model showing a three-dimensional space of an imaging environment by using importance information. Then, the importance of the region in the captured image is set by projecting the region on the three-dimensional space corresponding to each importance onto the captured image of the camera using the camera parameters. Information indicating the importance of a region in a captured image is called importance region information. Then, the suppression foreground selection unit 902 selects an object located in a region of predetermined importance in the target frame that is the target of generating the three-dimensional shape data. The suppression foreground selection unit 902 outputs the importance region information in each captured image to the suppression interval determination unit 903.

The suppression interval determination unit 903 determines a time interval indicating the frequency of updating the three-dimensional model of the object located in the region of low importance (low importance region) in the captured image. This time interval is also referred to as a suppression interval. For example, the suppression interval indicates the update frequency when the input frame rate, which is the reception frequency of the captured image, is 60 fps, and the frame rate indicating the update frequency of the three-dimensional model of the object in the low importance region is 30 fps. Information related to the frame rate. The frequency of updating the 3D model can also be said to be the frequency of generating the 3D model.

The frame rate (fps: frames per second) indicates a unit processed in one second, and the input frame rate is the number of captured images (frames) output from the camera in one second. Similarly, the frame rate indicates the number of foreground texture images and foreground mask images generated per second.

The processing frame rate represents the number of 3D models generated per second. Each time a frame is acquired from the camera, the visual volume crossover method produces a three-dimensional model of the object that is not located in the low importance region. Therefore, when the input frame rate is 60 fps, 60 three-dimensional models of each object not located in the low importance region are generated per second. The processing frame rate representing the update frequency of the three-dimensional model of the object not located in the low importance region is 60 fps, which is the same as the input frame rate.

On the other hand, when the interval is 2, a three-dimensional model of the foreground located in the low importance region is generated by the visual volume crossing method at a processing frame rate of 30 fps, which is a frame rate of 1/2 of the input frame rate (60 fps). Will be updated. In this case, out of the 60 frames input from the camera per second, the frames that are not used to generate the three-dimensional model of the object in the low importance region are included. Frames that are not used to generate a 3D model of objects in the low importance region are also called processing suppression frames. Similarly, when the interval is 4, a 3D model of the object in the low importance region is generated and updated at 15 fps, which is a frame rate of 1/4 of the input frame rate.

The suppression interval determination unit 903 determines the processing frame rate of this interval or the low importance region as the suppression interval. The suppression interval determination unit 903 may acquire information on the suppression interval and determine a time interval indicating the frequency of updating the three-dimensional model of the object located in the low importance region. Further, the suppression interval determination unit 903 may determine the suppression interval according to the importance of each. The importance area information and the suppression interval for each importance are output to the data deletion unit 904.

The suppression interval determination unit 903 may dynamically determine the suppression interval. Alternatively, the target frame may be determined as a processing suppression frame when a predetermined condition is satisfied. For example, when the processing load at the time of generating the three-dimensional model becomes high, the target frame may be determined as the processing suppression frame. Alternatively, if the processing load at the time of generating the 3D model becomes high, increase the time interval for updating the 3D model of the object located in the low importance area and lower the processing frame rate for updating the 3D model. The suppression interval may be determined as described above.

The data deletion unit 904 acquires the importance area information output from the suppression interval determination unit 903 and the suppression interval. When the frame used for extracting the rectangular mask or the like acquired by the acquisition unit 901 is a processing suppression frame, the data deletion unit 904 is a rectangle of an object located in the low importance area of the generated rectangular mask. Remove the mask. The data deletion unit 904 generates the foreground mask image shown in FIG. 5 by using a rectangular mask other than the deleted rectangular mask image. The generated foreground mask image is output to the model generation unit 905.

In the present embodiment, the foreground mask image is described as being generated by synthesizing a rectangular mask, but the foreground mask image may be generated without using the rectangular mask. For example, it may be generated by extracting the foreground region from the captured image and binarizing the foreground region into white and the background region into black. In that case, even if the data deletion unit 904 deletes the foreground area of the low importance area by processing the foreground mask image so that the foreground area in the low importance area of the foreground mask image becomes the background area. good.

Further, when the data deletion unit 904 deletes the rectangular mask of the object in the low importance area, the data deletion unit 904 may similarly delete the rectangular texture of the object in the low importance area. By deleting the rectangular texture of the object in the low importance region, it is possible to prevent the rectangular texture of the low importance region from being used in the coloring in the rendering device 105.

The model generation unit 905 uses the camera parameters and the foreground mask images of a plurality of cameras output from the data deletion unit 904 to generate a three-dimensional model of the object included in the frame by the visual volume crossing method. Original shape data generation unit. The generated data is output to the data synthesis unit 907. When generating a three-dimensional model of the processing suppression frame, the foreground mask image used for model generation is a foreground mask image that does not include the foreground region corresponding to the rectangular mask deleted by the data deletion unit 904. Therefore, the model generation unit 905 does not generate a three-dimensional model of the object located in the low importance region in the processing suppression frame, but generates a three-dimensional model of the object located in the region other than the low importance region. Can be done.

The data synthesis unit 907 acquires the foreground mask image and the three-dimensional model of the object generated by the model generation unit 905, and acquires the foreground texture image and the suppression interval from the data deletion unit 904. When the target frame is a processing suppression frame, the data synthesis unit 907 synthesizes a three-dimensional model of the previously generated object in the low importance region on the three-dimensional space deleted by the data deletion unit 904. Then, the data obtained as a result of the synthesis is output to the rendering device 105 as data representing the three-dimensional shape of the object included in the target frame.

Similarly, the data synthesizing unit 907 synthesizes the foreground region of the corresponding region of the previously generated foreground mask image with the low importance region of the foreground mask image. The foreground mask image obtained as a result of the composition is used as the foreground mask image in which the low importance region is located in the next frame. Similarly, the data synthesizing unit 907 synthesizes the corresponding foreground portion of the previously generated foreground texture image into the low importance region of the foreground texture image. Each data obtained as a result of the synthesis is output to the data management unit 906.

The data management unit 906 stores the data output by the data synthesis unit 907 in a storage unit such as a ROM 803 or a storage device 804. The stored data is each of the three-dimensional model, the foreground texture image, and the foreground mask image. Therefore, the data synthesis unit 907 can acquire the three-dimensional model of the object located in the low importance region from the three-dimensional model corresponding to the previous frame via the data management unit 906.

[About the importance of the area in the image]
FIG. 10 is a diagram for explaining the importance of the area in the image. FIG. 10A is a diagram in which importance areas R1 to R3, which are areas for which importance is set, are superimposed and displayed on the three-dimensional space of the background model. Further, FIG. 10A shows the arrangement of the cameras 101a to 101p based on the camera parameters.

The background model is three-dimensional shape data representing the three-dimensional space of the imaging environment, and is a simple stadium in FIG. 10 (a). As the data format of the background model, for example, it is represented by a geometry definition file format that is also used in three-dimensional CG (Computer Graphics).

The importance areas R1 to R3 are displayed based on the importance information. The importance information is information for determining the importance of each area in the three-dimensional space. Based on the importance information, the floor of the background model is divided and importance is given to each area. In the present embodiment, the importance will be described as being set in three stages of 1, 2, and 3 in descending order of importance. Then, the area of importance 1 is set as the area R1, the area of importance 2 is set as the area R2, and the area of importance 3 is set as the area R3 in the background model. Regions R2 and R3, which are regions other than the region R1 having the highest importance, are referred to as low importance regions. The importance information is, for example, information indicating the importance of each area set by the instruction of the user.

In the example of FIG. 10, the area R1 is in front of the goal where the viewer's attention is high, the area R2 is near the center of the field, and the area R3 is outside the field. In the present embodiment, the importance information is described as information for imparting importance to the floor surface which is a divided two-dimensional plane, but is information for imparting importance to the three-dimensional space. You may.

In the present embodiment, the object located in the low importance region is processed so that the frequency of updating the three-dimensional model is low. Therefore, by reducing the update frequency of the 3D model in the unimportant area, it is possible to suppress the overall processing load related to the 3D model while keeping the update frequency of the 3D model in the important area high. can.

FIG. 10 (b) is a diagram showing an image captured by the camera 101e obtained by performing a perspective projection conversion of a point on the three-dimensional space based on the camera parameters, when the three-dimensional space of FIG. 10 (a) is imaged. .. By performing perspective projection conversion in each camera, it is possible to obtain each importance region of the images captured by all the cameras.

The perspective projection transformation is to project a three-dimensional coordinate point onto an image plane imaged from a camera by Equation 1.
sx = A [R | t] X ... Equation 1
In Equation 1, X is a coordinate point in three-dimensional space, R | t is an external parameter of the camera parameter, A is an internal parameter of the camera parameter, x is a coordinate point on a two-dimensional plane, and s is a coordinate point in the camera coordinate system. Indicates the z coordinate.

[Processing flow for generating 3D model]
FIG. 11 is a flowchart showing the flow of generation of three-dimensional shape data showing a three-dimensional model of an object in a frame. The series of processes shown in the flowchart of FIG. 11 is performed by the CPU 801 of the three-dimensional model generation device 104 expanding the program code stored in the ROM 803 into the RAM 802 and executing the program code. Further, some or all the functions of the steps in FIG. 11 may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process means that the step is a step in the flowchart, and the same applies to the subsequent flowcharts.

In S1101, the suppression foreground selection unit 902 determines the importance of the region in the captured image of each camera 101a to 101p based on the camera parameters, the background model, and the importance information.

In S1102, the suppression interval determination unit 903 acquires the suppression interval for each importance. In the description of this flowchart, the suppression interval is expressed by the processing frame rate. It is assumed that the processing frame rate of the object located in the region R1 of the importance 1 having the maximum importance is predetermined to be 60 fps, which is the same as the input frame rate. Further, it is assumed that the processing frame rate of the object located in the region R2 of importance 2 is 30 fps, and the processing frame rate of the object located in the region R3 of the minimum importance 3 is 15 fps. Therefore, the suppression interval determination unit 903 acquires the processing frame rate of each determined importance.

S1103 to S1110 are steps for outputting a three-dimensional model of an object included in one frame (target frame) output from the cameras 101a to 101p. First, in S1103, the acquisition unit 901 acquires a rectangular mask and a rectangular texture of each camera from the foreground extraction device group 102. In S1104, the suppression foreground selection unit 902 determines whether or not there is a rectangular mask located in the low importance region among the acquired rectangular masks.

FIG. 12 is a diagram for explaining the process of this step. 12 (a) is a diagram showing a frame based on the image taken by the camera 101e of FIG. 10 (a). FIG. 12B is a diagram in which a rectangular mask generated based on the frame of FIG. 12A is arranged on an image obtained by taking an image of a background model in which each importance region is superimposed by the camera 101e. be. As shown in FIG. 12B, when there is a rectangular mask located in the area R2 in the center of the field which is a low importance area and the area R3 outside the field, it is determined that there is a rectangular mask located in the low importance area. Rectangle.

As shown by the rectangular mask M1 in FIG. 12B, the rectangular mask straddling the two importance regions of the region R1 and the region R2 is set to the low importance region based on the importance of the region to which the lower end of the rectangular mask belongs. It is determined whether there is a rectangular mask to be located. That is, it is determined that the rectangular mask M1 is located in the high importance region of the region R1.

If there is a rectangular mask located in the low importance region, proceed to S1105. Based on the suppression interval of the odor importance in S1105, it is determined whether the target frame used to generate the rectangular mask acquired in S1103 is the processing suppression frame. For example, the suppression interval determination unit 903 determines whether the target frame is a frame that does not update the three-dimensional model of the object located in the region of a certain importance. The decision is made based on the suppression interval and the timecode of the target frame.

For example, it is assumed that the input frame rate is 60 fps and the processing frame rate of the object in the area R2 is set to 30 fps. Further, it is assumed that the processing frame rate of the object in the area R3 is set to 15 fps. In this case, when the value indicating the number of frames of the time code of the target frame is other than a multiple of 2, the target frame is a processing suppression frame that does not update the three-dimensional model of the object located in the area R2 or the area R3. Is decided. Further, when the value indicating the number of frames of the time code of the target frame is a multiple of 2 and other than a multiple of 4, the target frame does not update the three-dimensional model of the object located in the area R3. Determined to be a suppression frame.

What is time code? [Hour: Minutes: Seconds. Time information expressed in the format of [frame]. Hereinafter, in this flowchart, the processing frame rate of the area R2 will be 30 fps, and the processing frame rate of the area R3 will be 15 fps.

That is, for the object located in the area R2, when the frame value of the time code of the target frame is a multiple of 2, a three-dimensional model is generated using the rectangular mask generated based on the target frame. For the object located in the region R3, when the value of the frame of the time code is a multiple of 4, a three-dimensional model is generated using the rectangular mask generated based on the target frame.

When the target frame is a processing suppression frame (YES in S1105), in S1106, among the rectangular masks located in the low importance region, the rectangular mask located in the region targeted to suppress the generation of the three-dimensional model is selected. Rectangle. Then, the data deletion unit 904 deletes the rectangular mask located in the region to be suppressed from generating the selected three-dimensional model.

FIG. 13 is a diagram for explaining the processing of this step of deleting the rectangular mask of the processing suppression frame. FIG. 13A is a diagram showing a rectangular mask after deleting the rectangular mask located in the area R3 among the rectangular masks included in FIG. 12B. For example, when the time code is 12: 34: 56.02, and the frame value is a multiple of 2 and not a multiple of 4, a three-dimensional model of the object in which the least important region R3 is located. Is a processing suppression frame that is not generated. Therefore, the rectangular mask located in the area R3 is deleted.

FIG. 13 (b) is a diagram showing a rectangular mask mask after deleting the rectangular mask located in the area R3 or the area R2 among the rectangular masks included in the figure of FIG. 12 (b). When the frame value is not a multiple of 2 and is not a multiple of 4, such as 12: 34: 56.03, the positions of the regions R2 and R3, which are regions other than the region R1 having the highest importance. The 3D model of the object is a processing suppression frame that is not generated. Therefore, the rectangular masks located in the areas R2 and R3 are deleted.

In S1107, a three-dimensional model of the object is generated. First, the data deletion unit 904 generates a foreground mask image using a rectangular mask. When the rectangular mask is deleted in S1106, the data deletion unit 904 generates a foreground mask image of each camera without using the deleted rectangular mask. Then, the model generation unit 905 generates a three-dimensional model by the visual volume crossing method using the camera parameters and the foreground mask image of each camera. As shown in FIG. 13A or FIG. 13B, the number of rectangular masks is reduced in the three-dimensional model generation processing in the processing suppression frame. Therefore, the foreground region in the foreground mask image is also reduced, and the number of voxels in the generated 3D model is also reduced. Therefore, in the present embodiment, the processing load related to the three-dimensional model can be reduced.

On the other hand, if NO is determined in S1104 or NO in S1105, S1106 skips and proceeds to S1107. Therefore, a foreground mask image is generated using all the acquired rectangular masks, and a three-dimensional model of the object included in the frame is generated.

In S1108, it is determined whether there is an object for which the three-dimensional model was not generated in S1107. If the rectangular mask is deleted in S1106, it is determined as YES.

If there is an object for which a 3D model is not generated (YES in S1108), the data synthesis unit 907 in S1109 acquires the 3D model of the previous frame located in the area where the 3D model of the object was not generated. do. Then, the three-dimensional model of the object of the immediately preceding frame obtained is added to the three-dimensional space in which the generated three-dimensional model is arranged. The data obtained as a result of the synthesis is output to the rendering device 105 as data representing a three-dimensional model of the object of the target frame.

FIG. 14 is a diagram for explaining the processing of the data synthesis unit 907. FIG. 14A is a diagram showing a three-dimensional model of the object generated as a result of the processing of S1107 in a white region. The figure which superposed the three-dimensional model generated based on the foreground mask image generated based on the deletion of the rectangular masks located in the area R2 and the area R3 shown in FIG. 13B on the background is shown. Is FIG. 14 (a). That is, it is shown that only the three-dimensional model of the object located in the region R1 which is the high importance region is generated as in the case where the time code of the target frame is 12: 34: 56.03.

FIG. 14B is a diagram showing a three-dimensional model of the previously output object corresponding to the previous frame in the shaded area. In this step, the 3D model of the object located in the area where the 3D model was not generated this time is acquired from the 3D model of the previous frame. That is, in the example of FIG. 14, the three-dimensional model located in the region R2 and the region R3 in FIG. 14B is acquired. Then, as shown in FIG. 14 (c), the three-dimensional model located in the region R2 and the region R3 in the previous frame is combined with the region R2 in the three-dimensional space in which the three-dimensional model generated this time is arranged. Add to region R3. That is, the three-dimensional model located in the low importance region R2 and the region R3 is not updated in this frame, and only the three-dimensional model located in the high importance region R1 is processed in the processing suppression frame. Will be updated.

As shown in FIG. 13A, the rectangular mask in the area R2 is not deleted, and even when the rectangular mask located in the area R3 is deleted, the three-dimensional model of the previous frame is additionally combined. Is done. That is, the 3D model located in the region R3 is acquired from the 3D model of the previous frame, and the acquired 3D model is obtained on the 3D space in which the generated 3D model is arranged. Will be added.

In addition, for a 3D model near the boundary of regions of different importance, the 3D model of the previous frame is acquired even though the 3D model is generated due to misjudgment of the position or the like. Is possible. In this case, the three-dimensional models are duplicated due to the composition. In order to avoid this duplication, it is determined whether the 3D model corresponding to the previous frame and the generated 3D model overlap, and if they overlap, the process of not synthesizing the 3D model of the previous frame is performed. You may be broken.

On the other hand, when it is determined that there is no object for which the 3D model has not been generated (S1108 is NO), the 3D model of all the objects has been generated. Therefore, since the process of synthesizing the three-dimensional model of the previous frame is not performed, S1109 is skipped.

In S1110, the data management unit 906 stores the three-dimensional model of the target frame and the like in the storage unit. The three-dimensional model to be stored is the three-dimensional model obtained as a result of the synthesis when the synthesis is performed in S1109. The stored data is used as the data of the three-dimensional model of the previous frame in the synthesis process of the three-dimensional model of S1109 in the next frame. When the previous frame is a processing suppression frame, the stored 3D model data includes the 3D model generated based on the previous frame and the frame past the previous frame. Includes the generated 3D model.

In S1111, it is determined whether or not there is the next frame. If there is a next frame, the process returns to S1103. Then, by repeating the processes of S1103 to S1110, a three-dimensional model of a series of temporally continuous frames is generated, and a temporally continuous three-dimensional model is output.

As described above, in the present embodiment, the frequency of generation of the three-dimensional model located in the low importance region where the viewer's attention of the virtual viewpoint image is low is suppressed. On the other hand, the frequency of updating the 3D model located in the high importance area where the viewer's attention is high, such as the area in front of the soccer goal, is maintained. Therefore, even when the processing related to the three-dimensional model has a high load, the processing load can be suppressed while suppressing the deterioration of the virtual viewpoint image quality.

In the above description, when the three-dimensional model of the object located in the low importance region is not generated in the processing suppression frame, the three-dimensional model of the low importance region in the previous frame is added. explained. Depending on the area within the frame, the effect on the viewer may be small even if a three-dimensional model of the object is not generated. In this case, if the 3D model of the object located in the area is not generated, it is not necessary to add the 3D model output in the previous frame. Even in this case, it is possible to suppress the processing load while suppressing the deterioration of the virtual viewpoint image quality.

<Second embodiment>
In the first embodiment, a method of suppressing the update of the three-dimensional model of the object located in the low importance region in the frame has been described based on the importance information determined in advance. In this embodiment, a method of specifying an object that increases the processing load and suppressing the processing load by reducing the frequency of updating the three-dimensional model of the specified object will be described. The present embodiment will be described focusing on the differences from the first embodiment. The parts not specified in particular have the same configuration and processing as those in the first embodiment.

FIG. 15 is a block diagram showing a functional configuration of the three-dimensional model generation device 104 of the present embodiment. The same processing blocks as those in the first embodiment are designated by the same numbers, and the description thereof will be omitted. The three-dimensional model generation device 104 of the present embodiment further includes an importance information determination unit 1501.

The importance information determination unit 1501 calculates the number of voxels of the three-dimensional model generated by the model generation unit 905, and identifies the three-dimensional model in which the number of voxels is equal to or greater than the threshold value. Then, the area determination for determining the area including the position of the specified three-dimensional model in the three-dimensional space is performed. The position information and importance of the determined area in the three-dimensional space are output to the suppression foreground selection unit 902.

FIG. 16 is a flowchart showing a flow of generation of three-dimensional shape data representing a three-dimensional model of an object in a frame. A method of generating three-dimensional shape data in the present embodiment will be described with reference to FIG.

In S1601, the suppression foreground selection unit 902 sets the importance of all areas in the frame to the highest importance as an initial value. In S1602, similarly to S1102, the suppression interval determination unit 903 acquires the suppression interval for each importance.

S1603 to S1613 are steps for outputting a three-dimensional model of an object included in one frame output from the camera. Since the processing of S1603 to S1607 is the same as the processing of S1103 to S1107, an outline thereof will be described. In S1603, the acquisition unit 901 acquires a rectangular mask and a rectangular texture from the foreground extraction device group 102. In S1604, it is determined whether or not there is a rectangular mask of an object located in the low importance region among the acquired rectangular masks. In the first frame, since the importance of all areas is set to the highest importance in S1601, it is determined as NO in this step. In S1605, it is determined whether the target frame is a processing suppression frame. In S1606, the rectangular mask of the object located in the low importance area is deleted. In S1607, a three-dimensional model is generated by the visual volume crossing method using a plurality of foreground mask images.

In S1608, the importance information determination unit 1501 determines whether or not there is a three-dimensional model in which the number of voxels constituting the three-dimensional model is equal to or greater than the threshold value among the three-dimensional models generated in S1607. As the threshold value, for example, a value predetermined in advance based on the processing performance of the three-dimensional model generation device 104 or the required specifications of the virtual viewpoint image to be generated is used.

If there is a three-dimensional model composed of the number of voxels above the threshold value (YES in S1608), the process proceeds to S1609. In S1609, the importance information determination unit 1501 sets a region on the three-dimensional space of the background model, which includes a three-dimensional model in which the number of voxels is equal to or greater than the threshold value, as a low importance region. The suppression foreground selection unit 902 updates the importance region in the captured image of each camera 101a to 101p based on the set importance information of the background model. Therefore, when the next frame is a processing suppression frame, in the process of generating the 3D model of the next frame, it is possible to suppress the update of the 3D model of the object located in the low importance region determined in this step. ..

FIG. 17 is a diagram for explaining determination of a low importance region by the importance information determination unit 1501. FIG. 17A is a diagram showing a generated three-dimensional model in a three-dimensional space, and the generated three-dimensional model is a three-dimensional model 1701 of a flag object composed of a number of voxels equal to or larger than a threshold value. Is included. FIG. 17B shows that the region on the three-dimensional space in which the three-dimensional model 1701 composed of the number of voxels equal to or larger than the threshold value is located is set in the region R3 which is a low importance region. FIG. 17C shows an image obtained by projecting a three-dimensional space including the region R3 onto the image plane of the camera 101e using the camera parameters of the camera 101e. Therefore, in the next frame, it is determined whether or not there is an object located in the low importance region in the frame based on the importance region shown in FIG. 17 (c).

As described above, in the present embodiment, the importance of the region is determined for each frame according to the frame, so that the frequency of updating the three-dimensional model composed of a large number of voxels can be reduced. The amount of processing related to the 3D model often depends on the number of voxels. Therefore, by reducing the frequency of updating the three-dimensional model composed of a large number of voxels, the processing load related to the three-dimensional model can be suppressed.

On the other hand, if there is no three-dimensional model composed of the number of voxels equal to or larger than the threshold value (S1608 is NO), the process proceeds to S1609. In S1609, as in S1601, the importance area that maximizes the importance of the area in the frame is set.

Since the processing of S1611 to S1613 is the same as the processing of S1108 to S1110, an outline will be described. In S1611, it is determined whether there is an object for which the three-dimensional model has not been generated. If there is an object for which 3D model generation has not been generated, the 3D model located in the low importance region output in the previous frame in S1612 is synthesized and output to the rendering device 105. In S1613, the output three-dimensional model and the like are stored. In S1614, it is determined whether or not there is the next frame. If there is a next frame, the process returns to S1603. Then, by repeating the processes of S1603 to S1613, a three-dimensional model of a series of frames is generated, and a temporally continuous three-dimensional model is output.

In this frame, there may be a three-dimensional model composed of the number of voxels equal to or greater than the threshold value, and in the next frame, there may be a three-dimensional model composed of the number of voxels equal to or greater than the threshold value. In this case, the region set in the low importance region in the previous frame may be returned to the high importance region, and then the region in which the 3D model is located in the current frame may be set in the low importance region.

Further, in the above description, a method of suppressing the update frequency of a three-dimensional model composed of a large number of voxels, such as the number of voxels exceeding the threshold value, has been described. In addition, the importance of the area may be dynamically determined based on a predetermined position of a predetermined object.

FIG. 18 is a three-dimensional model generated based on a frame obtained by imaging a soccer game. FIG. 18 is a diagram showing an example in which the importance of the area in the frame is determined so that the area closer to the three-dimensional model 1801 of a high-importance object such as a ball in a soccer game becomes more important. Is. In this way, by dynamically determining the importance, even if the area of high attention of the user changes for each frame, such as the area around the ball, the area of high attention is highly important. It can be an area. Therefore, it is possible to maintain a high frame rate by updating the three-dimensional model of the object located in the region of high attention.

Further, in the above description, when there is a three-dimensional model composed of the number of voxels above the threshold value, a method of uniformly determining the region in which the three-dimensional model is located as a low importance region has been described. Alternatively, the importance may be changed according to the number of voxels constituting the three-dimensional model to determine the importance of the region in which the three-dimensional model is located. For example, when a plurality of threshold values are set and the number of voxels is equal to or greater than the first threshold value, the region in which the three-dimensional model is located is determined to be the region R3 having the lowest importance. When the number of voxels is less than the first threshold value and greater than or equal to the second threshold value smaller than the first threshold value, the region in which the three-dimensional model is located may be determined to be the region R2 having the next lowest importance. ..

Further, when the movement of the object is small, the influence on the viewer is small even if the update frequency of the three-dimensional model of the object is reduced. Therefore, the movement amount of each object is calculated from the generated three-dimensional model for several frames. Then, for a three-dimensional model of an object with little movement such that the movement amount is equal to or less than the threshold value, a decision may be made to reduce the processing frame rate in order to reduce the update frequency.

Further, in the above description, the importance of the region is set based on the number of voxels of the generated 3D model. In addition, in the case of the processing suppression frame, it is determined whether there is a 3D model in which the number of voxels exceeds the threshold value during the generation of the 3D model, and when the number of voxels exceeds the threshold value, the generation of the 3D model is stopped. You may. Then, the region in which the three-dimensional model whose generation process is terminated may be determined as the low importance region.

Further, the importance of the region or the position of the region may be changed according to the time taken by the camera in order to obtain the target frame. For example, the importance information associated with time may be acquired and the importance of the area may be switched at a specified time.

When the object to be imaged by the image pickup device is a sports competition, the importance information determination unit 1501 identifies the area where the competition is performed based on information such as a line drawn from the captured image to the field, and sets the other area where the competition is performed. It may be set to an area of higher importance than the area of.

As described above, according to the present embodiment, the importance of the area in the frame can be dynamically changed. Therefore, the processing load related to the three-dimensional model can be suppressed, and the quality deterioration can be suppressed even when the load is high, and the generation of the three-dimensional model can be realized.

<Other embodiments>
In the above-described embodiment, the control device 103, the three-dimensional model generation device 104, and the rendering device 105 have been described as being separate devices. In addition, the functions of the control device 103, the three-dimensional model generation device 104, and the rendering device 105 may be realized by one or two devices. For example, a three-dimensional model of the foreground and a virtual viewpoint image may be generated by one image processing device.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

104 3D model generator 902 Suppression foreground selection unit 904 Data deletion unit 905 Model generation unit

Claims (17)

An image processing device that generates three-dimensional shape data of an object included in the frame by using an object image showing the shape of the object in the frame obtained by imaging by a plurality of image pickup devices.
A selection means for selecting an object in the first region in the frame and an object in the second region which is an region other than the first region.
The first frame rate for generating the three-dimensional shape data of the object in the first region is determined, and the second frame rate for generating the three-dimensional shape data of the object in the second region is the first. A means of determining a frame rate different from the frame rate,
An image generation means for generating the object image corresponding to the frame, and
It has a data generation means for generating the three-dimensional shape data by using the object image generated by the image generation means.
The image generation means is
The three-dimensional shape data of the object in the first region is controlled to be generated at the first frame rate, and the three-dimensional shape data of the object in the second region is generated at the second frame rate. An image processing device characterized by being controlled so as to be. The image processing apparatus according to claim 1, wherein the first region is a region having a lower importance than the second region. The image processing apparatus according to claim 2, wherein the value of the first frame rate is smaller than the value of the second frame rate. The image processing apparatus according to claim 2 or 3, wherein the importance is set based on a user's instruction. The image processing apparatus according to claim 2 or 3, further comprising a region determining means for determining the importance of the region in the frame. The area determination means is
When the image pickup target by the plurality of image pickup devices targets a sports competition, the area in the frame where the competition is performed is specified, and the area other than the area where the competition is performed is set from the area where the competition is performed. The image processing apparatus according to claim 5, wherein the image processing apparatus is determined to be a region of low importance. The area determination means is
The image processing apparatus according to claim 5, wherein a region having a low importance is determined based on a processing load at the time of generating three-dimensional shape data by the data generation means. The area determination means is
Of the three-dimensional shapes of the object indicated by the three-dimensional shape data generated by the data generation means, the importance is low based on the region where the three-dimensional shape in which the number of elements constituting the three-dimensional shape is equal to or greater than the threshold value is located. The image processing apparatus according to claim 5, wherein the area is determined. The area determination means is
The image processing apparatus according to claim 5, wherein the less important area is determined based on the area including a predetermined object. The area determination means is
It is characterized in that each area corresponding to a plurality of importance is determined, and each area is determined so that the area closer to the predetermined object is, the more important the area is. The image processing apparatus according to claim 9. The area determination means is
The image processing apparatus according to claim 5, wherein the region of low importance is determined based on the amount of movement of the object. The area determination means is
The image processing apparatus according to claim 5, wherein the region of low importance is determined based on the time when the frame is imaged. The determination means is
The image processing apparatus according to any one of claims 1 to 12, wherein the first frame rate and the second frame rate are determined based on the processing load of the image processing apparatus. When there is an area in the frame that does not generate 3D shape data of the object, the 3D shape in the 3D shape data of the object in the area generated in the past is the 3D shape data generated by the data generation means. The image processing apparatus according to any one of claims 1 to 13, further comprising a processing means for processing so as to be included in the three-dimensional space in the above. Further, it has an acquisition means for acquiring an image showing the shape of each object included in the frame.
The image generation means generates the object image using the acquired image, and the object image is generated.
If there is an area in the frame that does not generate 3D shape data for the object
The image generation means deletes the image corresponding to the region in the frame that does not generate the three-dimensional shape data of the object, and uses the image other than the deleted image among the acquired images to obtain the object. The image processing apparatus according to any one of claims 1 to 14, wherein an image is generated. An image processing method for generating three-dimensional shape data of an object contained in the frame by using an object image showing the shape of the object in the frame obtained by imaging by a plurality of image pickup devices.
A selection step for selecting an object in the first region in the frame and an object in the second region which is an region other than the first region.
The first frame rate for generating the three-dimensional shape data of the object in the first region is determined, and the second frame rate for generating the three-dimensional shape data of the object in the second region is the first. The decision step to decide the frame rate different from the frame rate,
An image generation step that generates the object image corresponding to the frame, and
It has a data generation step of generating the three-dimensional shape data using the object image generated by the image generation step.
In the image generation step,
The three-dimensional shape data of the object in the first region is controlled to be generated at the first frame rate, and the three-dimensional shape data of the object in the second region is generated at the second frame rate. An image processing method characterized by being controlled in such a manner. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 15.

Images