JP2019032742A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an Image processing device, an image processing method, and a program that improve image quality of a virtual viewpoint image including an area where a background shape can vary.SOLUTION: An image processing device for generating a virtual viewpoint image from images photographed by a plurality of imaging devices, includes: specifying means for specifying an area where a person appears in a region to be a background of the virtual viewpoint image; and generation means for generating the virtual viewpoint image using a three-dimensional model having a higher resolution in a region where a person does not appear in the region to be background of the virtual viewpoint image than the area specified by the specifying means.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program capable of generating a free start point video.

  There is a virtual viewpoint image technique as a technique for reproducing an image photographed from a camera (virtual camera) that does not actually exist and is virtually arranged in a three-dimensional space using videos photographed by a plurality of real cameras. For example, when creating a virtual viewpoint image, in addition to the main subjects such as players and balls that are competing, background images such as spectator seats and roofs are used. Need to be generated. As a technique for creating this background image, for example, Patent Document 1 discloses a rendering technique in which a multi-viewpoint image is projected as a texture on a background shape on a cylindrical surface having a constant radius.

JP 2008-217547 A

In the method described in Patent Document 1, the processing load can be reduced by projecting a multi-viewpoint image onto a detailed shape of the actual background by projecting the multi-viewpoint image onto a cylindrical surface different from the actual background shape. . However, there is a problem that the larger the difference between the actual background shape and the cylindrical shape, the larger the difference between the generated virtual viewpoint image and the actually captured image, and the worse the image quality of the virtual viewpoint image. It is also conceivable to reduce the difference from the actual background shape by projecting a multi-viewpoint image onto the detailed shape of the actual background generated in advance. However, even if a multi-viewpoint image is projected onto a detailed shape of an actual background that has been generated in advance, the generated virtual viewpoint image and a photographed image can be used as long as the background shape changes with time, such as a spectator seat. And the image quality of the virtual viewpoint image may be deteriorated.
Therefore, an object of the present invention is to improve the image quality of a virtual viewpoint image including an area where the background shape can vary.

The present invention is an image processing device that generates a virtual viewpoint image from images captured by a plurality of imaging devices, the specifying means for specifying a region in which a person is captured in a region as a background of the virtual viewpoint image, and the specifying Means for generating the virtual viewpoint image using a three-dimensional model having a higher resolution in a region where a person is not captured in a region as a background of the virtual viewpoint image than an area specified by the means. Features.
Another aspect of the present invention is an image processing device that generates a virtual viewpoint image from images captured by a plurality of imaging devices, the first acquisition means for acquiring the degree of variation of the shape as a background, and the background. Determining means for determining the resolution of the shape serving as the background based on the degree of variation of the shape to be generated; and generating means for generating the virtual viewpoint image using the shape serving as the background of the resolution determined by the determining means; It is characterized by having.

  According to the present invention, it is possible to improve the image quality of a virtual viewpoint image including a region where the background shape can vary.

It is a figure which shows the example of 1 structure of the image processing apparatus of this embodiment. It is the figure which showed an example of each camera which comprises a camera group. It is a functional block diagram of an image processing apparatus. It is a flowchart which shows the flow of the process performed with an image processing apparatus. It is a schematic diagram showing the area | region which makes a shape low resolution. It is a flowchart which shows the flow of the process which produces | generates a detailed shape. It is a figure which shows an example of the front view of a shape.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

FIG. 1 is a diagram illustrating a schematic configuration example of a virtual viewpoint image system including an image processing apparatus 100 according to the present embodiment. The virtual viewpoint image system shown in FIG. 1 includes an image processing device 100 and a plurality of imaging devices (camera groups) 109.
The image processing apparatus 100 is an apparatus that generates a virtual viewpoint image. In the present embodiment, the term “image” will be described as a concept including both moving images and still images unless otherwise specified. The image processing apparatus 100 includes a CPU 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, and an external I / F unit 106, and each unit is connected via a bus 107.

  The CPU 101 is an arithmetic processing device that controls the image processing apparatus 100 in an integrated manner, and executes various programs by executing various programs stored in the storage unit 103 and the like. The main memory 102 temporarily stores data and parameters used in various processes and provides a work area for the CPU 101. The storage unit 103 is a large-capacity storage device that stores various programs and various data necessary for GUI (graphical user interface) display. For example, a nonvolatile memory such as a hard disk or a silicon disk is used. The input unit 104 is a device such as a keyboard, a mouse, an electronic pen, or a touch panel, and receives an operation input from a user. The display unit 105 is configured by a liquid crystal panel or the like, and performs GUI display for setting a virtual camera path when generating a virtual viewpoint image. The external I / F unit 106 is connected to each camera constituting the camera group 109 via the LAN 108 and transmits and receives video data and control signal data. A bus 107 connects the above-described units and performs data transfer.

The camera group 109 is connected to the image processing apparatus 100 via the LAN 108. Based on a control signal from the image processing apparatus 100, the start and stop of shooting, change of camera settings (shutter speed, aperture, etc.), and shooting are performed. Transfer video data.
As for the system configuration, there are various components other than the above, but the description thereof is omitted.

  FIG. 2 is a view showing an arrangement example of the cameras 203 constituting the camera group 109 of FIG. There are many possible scenes for generating a virtual viewpoint image, such as sports and concerts. Here, a case where ten cameras 203 are installed in a stadium where rugby is played will be described as an example. A player and a ball as the subject 202 exist on the field 201 where the game is performed, and ten cameras 203 are arranged so as to surround the field 201. When the individual cameras 203 constituting the camera group 109 are set with appropriate camera orientation, focal length, exposure control parameters, etc. so that the entire field 201 or the attention area of the field 201 is within the angle of view. To do.

  FIG. 3 is a functional block diagram showing a functional configuration of the image processing apparatus 100 according to the present embodiment. In the present embodiment, the image processing apparatus 100 generates a stadium shape as a background of the virtual viewpoint image. The image processing apparatus 100 includes a detailed shape generating unit 301, a shape variation region specifying unit 302, a shape low resolution unit 303, an imaging region mapping unit 304, an occlusion region specifying unit 305, and a shape correcting unit 306. Note that these functional configurations are realized by the CPU 101 executing, for example, a control program stored in the storage unit 103 to control calculation and processing of information and each hardware. Also, some or all of the functional configurations shown in FIG. 3 may be configured by dedicated hardware such as ASIC or FPGA.

  The detailed shape generation unit 301 generates a detailed shape of the stadium from the stadium design drawing and the multi-viewpoint captured image. The detailed shape generation unit 301 generates three-dimensional model information indicating the shape of the object that is the background. The shape change area specifying unit 302 specifies an area where the degree of change in the shape of the stadium with time elapses is large. The shape lowering unit 303 changes the resolution in the shape of the region having a large shape variation degree, specifically, lowers the resolution. The shape reduction unit 303 may change the resolution of the shape by changing the fineness, roughness, smoothness, or quantization bit number of the three-dimensional model. The shooting area mapping unit 304 maps the area shot by each camera onto the shape of the stadium. The occlusion area specifying unit 305 specifies an occlusion area in the shape of the stadium. The occlusion area refers to an area that is not photographed by the camera or that is not photographed by the camera because the object in the foreground hides the object behind it. The shape correcting unit 306 corrects the shape corresponding to the occlusion area. Here, the shape in the present embodiment is formed by a plurality of polyhedral polygon groups. The shape may be expressed by a wire frame model, a surface model, or a point group.

  Next, processing performed in the image processing apparatus 100 will be described with reference to a flowchart shown in FIG. This series of processing is realized by the CPU 101 reading a predetermined program from the storage unit 103 and developing it in the main memory 102, which is executed by the CPU 101. Note that it is not necessary for the CPU 101 to execute all of the processes shown below, and the image processing apparatus 100 may be configured such that part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. Good. In the following description, the processing steps S401 to S406 are abbreviated as S401 to S406, respectively, for the sake of simplicity, and this also applies to other flowcharts described later.

  In S401, the detailed shape generation unit 301 generates the detailed shape of the stadium. Specifically, the detailed shape generation unit 301 generates the detailed shape of the stadium based on information such as the design drawing of the stadium developed in the main memory 102 and the edge characteristics of the multi-viewpoint image. Details of this detailed shape generation method will be described later.

In next step S <b> 402, the shape change area specifying unit 302 receives a multi-viewpoint image developed in the main memory 102 and specifies an area having a large shape change degree in the detailed shape of the stadium.
Here, the shape variation degree in the stadium is, for example, the shape of the stadium in the sequence for generating the virtual viewpoint image when the initial shape is the shape of the stadium where no game is played and no people are entering. Is an index that represents how much has changed from the initial shape. For example, when a match starts and a spectator enters the spectator seat, the spectator seat area is specified as an area having a large shape variation. One method for identifying a region having a large shape variation is a method using prior knowledge. For example, in a soccer game, a large number of spectators can enter the spectator seat, so that the degree of change in shape is large, and since a person does not enter the roof, the amount of change in shape can be reduced.

  In addition, the region where the degree of shape variation is large is not necessarily constant in the sequence of virtual viewpoint images. For example, if the weather was good at the start time of the virtual viewpoint image sequence and there were many spectators on the first floor, but it began to rain during the sequence, and the spectator moved to the second floor with a roof, the shape changed. A region with a high degree fluctuates during the corresponding sequence. In this way, when the degree of change in shape changes during a game, shapes with different resolutions are prepared in advance for each stadium area, and when rendering a virtual viewpoint image, the shape of the stadium is determined according to the degree of change in shape. The resolution may be changed for each region.

The stadium shape variation degree does not necessarily have to be manually designated by the user, and an area having a large shape variation degree may be automatically determined. As a method for automatically determining a region having a large degree of shape variation, for example, a method for identifying a region having a large color change in a multi-viewpoint image can be used. In an area where many spectators enter, the spectator moves over time, so the color change with time becomes larger than in other areas. Conversely, the color change is less in the roof area. Therefore, by using the color change, it is possible to automatically determine an area having a large degree of shape variation. For example, for one camera 203 that performs multi-viewpoint shooting, a matrix storing luminance values for each pixel (x, y) of an image of width n and height h in the nth frame is _denoted by In, and n. the sum S _n of the difference in brightness values before and after k frames of th frame can be represented by the formula (1).

  The shape variation area specifying unit 302 determines whether or not the area has a large shape variation degree by setting a threshold value for the difference between the luminance values. The threshold value may be parameter-tuned by the user, or may be automatically determined from the distribution of luminance value differences for each pixel. The shape variation area specifying unit 302 performs this process on images of all cameras that are taking multiple viewpoints. Note that in Equation (1), the total value of the previous and subsequent frames is calculated with a small degree of shape variation, such as a large video display device installed in a stadium, but it is erroneous depending on the displayed video. This is to prevent erroneous determination in a region that is highly likely to be determined. That is, the large-sized video display device displays a clock, scores of each team, and the like, and erroneous determination occurs when the degree of shape variation is determined only by the front and rear frames.

  After the image processing apparatus 100 identifies a region having a large degree of shape variation in the multi-viewpoint image as described above, the region having a large degree of shape variation in the image is determined based on the position and orientation of the camera that captured each image. Is projected onto the shape and mapping is performed. The image processing apparatus 100 associates a region having a large shape variation degree in the multi-viewpoint image captured by the camera group 109 with the background shape generated in S401. Further, after mapping an area having a large degree of shape variation, the image processing apparatus 100 uses an edge 502 that connects the vertices 501 of the quadrilateral polygons so as to include the mapped area, as shown in FIG. Perform the enclosing process. An area surrounded by the edge 502 is an area to be processed in S403 described later. Here, the edge represents a line that is one side of a certain polygon and connects the vertexes 501 of the polygon. Note that the image processing apparatus 100 may directly specify a region having a large shape variation degree from the background shape. Further, the image processing apparatus 100 may specify a region having a large degree of shape variation from the world coordinates. In addition, the image processing apparatus 100 may specify an area where a person is captured (or a person may be captured) as an area where the degree of shape variation is large. In addition, the image processing apparatus 100 may detect a person from a multi-viewpoint image by image processing in a background region in the virtual viewpoint image, and specify the detected region as a region having a large shape variation degree. The image processing apparatus 100 may specify an area where the degree of variation in shape is large among areas serving as backgrounds in a virtual viewpoint image of a facility such as a stadium to be photographed. Further, the image processing apparatus 100 may identify an area where the degree of shape variation is large in an area where no person exists, such as a roof of a facility such as a stadium to be imaged.

  In step S <b> 403, the shape reduction unit 303 determines the resolution of the shape used when generating the virtual viewpoint image based on the shape variation degree. In step S <b> 403, the shape reduction unit 303 changes the resolution in the shape of the region having a large shape variation degree among the detailed shapes of the stadium generated by the detailed shape generation unit 301, specifically, reduces the resolution. Here, the resolution of the shape is an index representing the details of the shape depending on the number of polygons forming the shape. That is, reducing the resolution of the shape corresponds to reducing the number of polygons of the corresponding shape. Note that the shape reduction unit 303 may increase the resolution in the shape of the region having a small shape variation degree, for example, by increasing the number of polygons. In addition, when the shape of the background region in the virtual viewpoint image is expressed by a point group, the shape reduction resolution unit 303 may change the resolution of the shape by increasing or decreasing the point group.

  The simplest method for reducing the resolution of the shape is an approximate polygon polygon (m-square polygon) passing through the vertex 511 of the edge 512 surrounding the projection region 514 as shown in FIG. ) As an alternative to a region with a large degree of shape variation. In general, after replacing the shape with an approximate m-polygon so that the resolution is not too far from the original shape, the m-polygon is divided into polygons equal to or smaller than a quadrangle. Note that the low resolution method of the shape is not limited to the above method. In addition to the above methods, the resolution of the shape in the projection area is subdivided, and a method of preferentially reducing the resolution from a low-resolution part or fitting a convex curved surface passing through the vertex of the edge surrounding the projection area There are also methods that may be used.

  Next, in S <b> 404, the imaging region mapping unit 304 determines the stadium shape output from the shape reduction unit 303 based on the position and orientation of the multi-viewpoint camera developed in the main memory 102 and the angle of view. Map the shooting area of each camera. The method for mapping the imaging region on the shape of the stadium can be the same as the method for mapping the region having a large degree of shape variation on the shape in S402.

  Next, in S405, the occlusion area specifying unit 305 determines whether or not there is an occlusion area for the shape of the stadium to which the shooting area mapping unit 304 maps the shooting area. Here, the occlusion area is an area in the stadium shape where the shooting area is not mapped. If it is determined in S405 that no occlusion area exists, the image processing apparatus 100 ends the shape generation process of the flowchart of FIG.

  On the other hand, if it is determined in S405 that an occlusion area exists, the shape correction unit 306 corrects the shape of the occlusion area in S406. In other words, when it is determined that there is an occlusion area, the shape correction unit 306 surrounds an area where the imaging area is not mapped as shown in step 402 with a nearby edge, and sets it as an area for shape correction. Then, the shape correcting unit 306 corrects the shape corresponding to the occlusion area specified by the occlusion area specifying unit 305. Here, as described in S405, the shape of the occlusion area is surrounded by an edge connecting the vertices of the polygon. Therefore, when the shape is corrected, the same method as described in S403 for reducing the resolution of the polygon can be used. In this case, the resolution of the polygon is reduced using one of the methods mentioned in S403, and the process returns to S405 for processing. Generally, since the fine irregularities are reduced by reducing the resolution of the shape, the occlusion area that cannot be seen from the camera is reduced.

  After the shape correction in S406, the occlusion area specifying unit 305 determines again whether or not there is an occlusion area in S405. If an occlusion area is detected again, shape correction by the shape correction unit 306 is performed in S406. Here, when shape correction is performed, a method that is not yet used in the iterative processing of S405 and S406 is used among a plurality of polygon resolution reduction methods. That is, by repeating S405 and S406, all polygon resolution reduction methods that can be selected for the occlusion region can be applied. When the entire stadium is photographed by a multi-viewpoint camera, the shape occlusion area can be made substantially zero by using any one of the polygon reduction methods.

  Next, the detailed shape generation of the stadium in S401 of FIG. 4 will be described in detail. FIG. 6 is a flowchart illustrating in detail the flow of processing when generating the detailed shape of the stadium from the design drawing according to the present embodiment. Here, the design drawing of the stadium represents, for example, a projection drawing.

  In step S <b> 601, the detailed shape generation unit 301 reads stadium design drawing data and draws a contour line in the design drawing on a 3DCG (three-dimensional computer graphics) space. First, the detailed shape generation unit 301 reads projection drawings including a front view, a side view, a top view, and the like, and arranges each drawing on a 3DCG space as an image plane of a camera corresponding to the viewpoint of each orthographic projection. To do.

  FIG. 7 shows an example of a front view. The front view of the stadium is a diagram in which an outline is drawn when the stadium is photographed from the front by the orthographic camera 700. The detailed shape generation unit 301 arranges the previous drawing in the 3DCG space as the image plane 702 on which the outline 710 of the projection target 701 from the viewpoint of the orthographic projection camera 700 is arranged. Similarly, the side view and the top view are diagrams showing the outline of the stadium when the stadium is photographed from the side and the top. When generating the detailed shape, the detailed shape generation unit 301 draws the contour line in the design drawing as a line in the 3DCG space in order to use these contour lines as shape edges.

  Next, in step S602, the detailed shape generation unit 301 projects the contour line generated in step S601 on the three-dimensional space, and aligns the contour line. First, the detailed shape generation unit 301 translates each contour line toward the corresponding orthographic camera. Furthermore, the detailed shape generation unit 301 sets a point that collides with the vertex of a line generated from another projection when the translation is performed as the distance of the translation of the line. Then, after determining the translational distance, the detailed shape generation unit 301 performs rotation so as to match the vertices of the lines adjacent to these lines. These processes do not necessarily need to be performed fully automatically, but may be subject to user manual position adjustment.

  In step S <b> 603, the detailed shape generation unit 301 generates a polygon for an edge having a shape formed by combining the outlines of the respective design drawings generated in step S <b> 602. When generating a polygon, a line connecting vertices forming a line is set as an edge of the polygon. Also, as described in S403, in order to maintain the shape resolution, the polygon is used with a square or less.

  In the flowchart of FIG. 6, the detailed shape is generated using the outline of the design drawing, but the method of generating the detailed shape using the outline is not necessarily limited to this. For example, a stadium can be photographed from multiple viewpoints, and the shape can be generated from the edges of a live-action image photographed from multiple viewpoints. As in the case of using the design drawing, each captured image is arranged on the 3DCG space as an image plane from each viewpoint of the captured camera. Thereafter, a line is drawn on the 3DCG space based on the edge of the photographed image. The edge of a real image can be obtained by convolving a Sobel filter, a Laplacian filter, a Previt filter, or the like, which is a common edge detection filter in image processing, with a grayscale image. By adjusting the filter coefficient to detect even weak edges, the contour of the shape can be obtained in some detail, but because it will be strongly affected by noise present in the image, Very detailed edges need to be detected visually by the user. In fact, in CG modeling, there are many examples in which a user visually finds an edge of an image and forms a contour of a shape without using an edge detection filter. By using these edges as contours, the stadium shape can be obtained by the procedures of S602 and S603.

  Further, the method for generating the detailed shape is not necessarily limited to the method using the contour line, and other methods may be used. One method is to acquire stadium point cloud data in advance using a laser scanner and convert the point cloud to polygons. Known methods for converting point cloud data into polygons include Poisson surface reconstruction and ball pivot. By transforming the point cloud into a polygon shape by these methods, the shape can be smoothed to reduce noise and obtain a smooth stadium shape. As a smoothing method, a method of convolving a moving average filter or a Gaussian filter with respect to polygonal vertex coordinates is generally known.

  As described above, according to the present embodiment, by reducing the resolution of an area where the background shape has a large degree of variation, and by correcting the shape of the occlusion area of the background shape, even if a multi-viewpoint image is projected, the background etc. It is possible to generate a free start point image with good image quality that does not break down the image. Further, according to the present embodiment, the resolution of the region with a small shape variation such as the roof is made higher than the resolution of the region with a large shape variation such as the auditorium. Therefore, by lowering the resolution of areas with large degree of shape variation such as audience seats, it is possible to round the difference between the shape of the background generated in advance and the shape of the background after equal variation where there is a spectator. It is possible to reduce the deterioration of image quality when generating. On the other hand, in areas where the degree of shape variation is small, such as the roof, the difference between the pre-generated background shape and the actual background shape is small, so a high-resolution virtual viewpoint image can be obtained by using a high-resolution background shape. Can be generated.

  In the above-described embodiment, the example in which the shape of the stadium suitable for virtual viewpoint image generation is generated by reducing the shape from the detailed shape is described. As another embodiment, when generating a stadium shape, a method of gradually transforming a low-resolution shape into a high-resolution shape may be used.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  DESCRIPTION OF SYMBOLS 100: Image processing apparatus, 101: CPU, 102: Main memory, 301: Detailed shape production | generation part, 302: Shape fluctuation area specific | specification part, 303: Shape low resolution part, 304: Imaging area mapping part, 305: Occlusion area specification Part, 306: shape correction part

Claims (19)

An image processing device that generates a virtual viewpoint image from images taken by a plurality of imaging devices,
A specifying means for specifying an area in which a person appears in an area as a background of the virtual viewpoint image;
Generating means for generating the virtual viewpoint image using a three-dimensional model having a higher resolution in a region in which a person is not captured in a region as a background of the virtual viewpoint image than in an area specified by the specifying means;
An image processing apparatus comprising:   2. The image processing according to claim 1, wherein a region in which a person is not captured in a region as a background of the virtual viewpoint image is a region corresponding to a roof of a stadium that is a photographing target of the plurality of imaging devices. apparatus.   The image processing apparatus according to claim 1, wherein the specifying unit specifies an area corresponding to a spectator seat of a stadium, which is a shooting target of the plurality of imaging apparatuses. Changing means for reducing the resolution of the three-dimensional model corresponding to the area specified by the specifying means;
The image processing apparatus according to claim 1, wherein the generation unit generates the virtual viewpoint image using the three-dimensional model whose resolution is reduced by the changing unit.   The specifying means captures an area in which a person appears in the background area of the virtual viewpoint image by user input, detection of a person from an image captured by the plurality of imaging devices, or imaging by the plurality of imaging devices. The image processing apparatus according to claim 1, wherein the image processing apparatus is specified based on a color change of the image. An image processing device that generates a virtual viewpoint image from images taken by a plurality of imaging devices,
First acquisition means for acquiring a degree of variation of a shape as a background;
Determining means for determining the resolution of the background shape based on the degree of variation of the background shape;
Generating means for generating the virtual viewpoint image using the background shape having the resolution determined by the determining means;
An image processing apparatus comprising:   The image processing apparatus according to claim 6, wherein the variation degree of the shape is an index indicating a magnitude of a change in the shape as the background in a sequence for generating a virtual viewpoint image. Having a second acquisition means for acquiring a background shape;
The second acquisition means uses at least one of an outline of a design drawing and an edge detected in an image captured by the plurality of imaging devices as an edge having a shape as the background. 8. The image processing apparatus according to claim 6, wherein the image processing apparatus is characterized in that:   The image processing apparatus according to claim 8, wherein the second acquisition unit acquires the background shape generated based on a point cloud acquired in advance.   The first acquisition unit specifies an area having a large degree of variation of the shape based on a user input or a color change of images captured by the plurality of imaging devices. The image processing apparatus according to any one of 9.   11. The image processing apparatus according to claim 6, wherein the determination unit determines to reduce the resolution of an area having a large degree of variation of the background shape.   The generation unit generates a virtual viewpoint image using the background shape obtained by approximating a region having a large variation degree of the background shape with a polygonal polygon and reducing the resolution. The image processing apparatus according to 11.   The generation unit subdivides the resolution in an area where the degree of variation in the shape of the background is large, and generates the virtual viewpoint image using the shape of the background that is preferentially reduced in resolution from a low-resolution part. The image processing apparatus according to claim 11, wherein the image processing apparatus is an image processing apparatus.   The said generating means produces | generates the said virtual viewpoint image using the shape which fitted the convex curved surface to the area | region where the fluctuation | variation degree of the shape used as the background is large. The image processing apparatus described. A correction unit that corrects the shape of the area that is not photographed by the imaging device in the background shape;
15. The method according to claim 6, wherein when the processing for correcting the shape is repeated, the correcting means uses a method that is not yet used among a plurality of methods for correcting the shape. The image processing apparatus according to item. Mapping means for mapping the images taken by the plurality of imaging devices onto the background shape;
An area specifying means for specifying an occlusion area in which the photographed image is not mapped out of the shape serving as the background;
The image processing apparatus according to claim 15, wherein the correction unit corrects a shape of the occlusion area. An image processing method executed by an image processing device that generates a virtual viewpoint image from images taken by a plurality of imaging devices,
A specifying step of specifying an area in which a person appears in an area as a background of the virtual viewpoint image;
A generating step of generating the virtual viewpoint image using a three-dimensional model having a higher resolution in a region where a person is not captured in a region as a background of the virtual viewpoint image than in the region specified in the specifying step;
An image processing method comprising: An image processing method executed by an image processing device that generates a virtual viewpoint image from images taken by a plurality of imaging devices,
An acquisition step of acquiring the degree of variation of the shape as a background;
A determination step for determining a resolution of the background shape based on the degree of variation of the background shape;
A generation step of generating the virtual viewpoint image using the shape as the background of the resolution determined in the determination step;
An image processing method comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 16.

Images