JP2023021735A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of acquiring a virtual viewpoint image that expresses a part of an object like an afterimage.SOLUTION: An image processing device acquires object shape data corresponding to each of multiple times in a period of continuous synchronous imaging with multiple imaging devices, and determines a specific portion that is a part of the object. The image processing device projects the object shape data at a reference time among the multiple times and the shape data of the object specific portion at a specific time different from the reference time among the multiple times to a virtual viewpoint, and generates a virtual viewpoint image.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to technology for generating a virtual viewpoint image based on a plurality of captured images.

2. Description of the Related Art There is a technique for generating a virtual viewpoint image representing a view from a virtual viewpoint using a plurality of captured images obtained by synchronously capturing a subject (object) by installing a plurality of imaging devices at different positions. As a technique that applies this virtual viewpoint image, Patent Document 1 discloses that virtual viewpoint images of a specific object viewed from a desired virtual viewpoint are generated at different points in time and superimposed on each other to express the trajectory of the object. A method is disclosed.

Japanese Patent Application Laid-Open No. 2021-13095

Koichi Ogawara, Xiaolu Li, Katsushi Ikeuchi, "Estimation of human body motion using flexible deformation model with joint structure", Proc. of MIRU2006, pp.994-999, 2006.

There is a case where it is desired to express only a moving part of an object (for example, only a person's arm) in a virtual viewpoint image as an afterimage. In this regard, in the technique of Patent Document 1, parts other than the arms that do not move are also expressed as afterimages, and it was not possible to obtain a virtual viewpoint image in which only the arms that are in motion are expressed like afterimages. .

An image processing apparatus according to the present disclosure includes acquisition means for acquiring shape data of an object corresponding to each of a plurality of times during a period in which images are continuously captured by a plurality of imaging devices; shape data of the object at a reference time among the plurality of times, and shape data of a specific portion of the object at a specific time different from the reference time among the plurality of times and generating means for generating a virtual viewpoint image corresponding to the virtual viewpoint.

According to the technique of the present disclosure, it is possible to obtain a virtual viewpoint image in which a part of an object is expressed like an afterimage.

The figure which shows an example of a structure of an image processing system. FIG. 2 is a diagram showing an example of the hardware configuration of an image processing apparatus; FIG. 2 is a diagram showing an example of the software configuration of an image processing apparatus; 4 is a flowchart showing the flow of processing for generating a virtual viewpoint image according to the first embodiment; (a) And (b) is a figure explaining detection of a specific part. (a) to (d) are diagrams showing projected images of a 3D model. 4A and 4B are diagrams for explaining synthesis processing of projected images; FIG. (a) to (c) are diagrams showing an example of an afterimage expression. 9 is a flowchart showing the flow of processing for generating a virtual viewpoint image according to the second embodiment; The figure which shows an example of the UI screen for site|part specification. 4A and 4B are diagrams for explaining adjustment during synthesis; FIG. (a) and (b) are diagrams showing an example of an afterimage expression.

Hereinafter, the form for implementing this embodiment is demonstrated with reference to drawings. Note that the following embodiments do not limit the technology of the present disclosure, and not all the configurations described in the following embodiments are essential for solving the problems.

[Embodiment 1]
First, an outline of the virtual viewpoint image will be briefly described. A virtual viewpoint image is an image representing a view from a virtual camera viewpoint (virtual viewpoint) different from the actual camera viewpoint, and is also called a free viewpoint image. The virtual viewpoint is set by a method such that the user operates the controller to directly specify it, or selects, for example, from a plurality of preset virtual viewpoint candidates. Note that the virtual viewpoint image includes both moving images and still images, but the following embodiments will be described using moving images as an example. When generating a moving image virtual viewpoint image, the virtual viewpoint may be fixed, or may be changed so as to follow the movement of the object.

<About system configuration>
FIG. 1 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint image according to this embodiment. The image processing system 100 has a plurality of imaging devices (cameras) 101 , an image processing device 102 , a controller 103 and a display device 104 . In the image processing system 100 , the image processing device 102 generates a virtual viewpoint image based on a plurality of captured images obtained by synchronous imaging by a plurality of cameras 101 and displays it on the display device 104 .

The cameras 101 are installed so as to surround the object, and each camera 101 takes an image of the object in synchronization with the time. However, if there is a limit to where to install the cameras 101, such as in a studio or concert hall, the multiple cameras 101 will be installed only in a partial direction of the imaging target area. Each camera 101 is realized by, for example, a digital video camera having a video signal interface typified by a serial digital interface (SDI). Each camera 101 adds time information represented by a time code to an output video signal and transmits the video signal to the image processing apparatus 102 . At this time, an imaging parameter set such as the three-dimensional camera position (x, y, z), imaging direction (pan, tilt, roll), angle of view, and resolution is also transmitted. The imaging parameter set is calculated and stored in advance by performing known camera calibration for each camera 101 .

The image processing device 102 generates a virtual viewpoint image that expresses a part of an object like an afterimage based on a plurality of captured images obtained by synchronous imaging by a plurality of cameras 101 . Specifically, the generation of the shape data of the foreground object, the setting of a part (specific part) of the object that is subject to movement for afterimage expression, the projection of the shape data of the object to the virtual viewpoint, and the projection image Synthesis of Details of the functions of the image processing apparatus 102 will be described later.

A controller 103 is a control device operated by a user to cause the image processing device 102 to generate a virtual viewpoint image. The operator performs various settings, data input, and the like necessary for generating a virtual viewpoint image through an input device such as a joystick or keyboard of the controller 103 . Specifically, the three-dimensional position of the virtual viewpoint (virtual camera) in the imaging space, line-of-sight direction, angle of view, resolution, time code necessary for image generation, and the like are specified. Here, the time represented by the time code includes the start and end times of virtual viewpoint image generation in the imaging period of the plurality of captured images, key frame time (reference time for afterimage expression), and the like. Information for generating a virtual viewpoint image set based on user designation (hereinafter referred to as “virtual viewpoint information”) is transmitted to the image processing device 102 .

The display device 104 acquires and displays image data (UI screen data for a graphical user interface and virtual viewpoint image data) sent from the image processing device 102 . The display device 104 is implemented by, for example, a liquid crystal display, a projector, a head-mounted display, or the like.

<About hardware configuration>
FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 102. As shown in FIG. The image processing apparatus 102 as an information processing apparatus has a CPU 211 , a ROM 212 , a RAM 213 , an auxiliary storage device 214 , an operation section 215 , a communication I/F 216 and a bus 217 .

The CPU 211 implements each function of the image processing apparatus 102 by controlling the entire image processing apparatus 102 using computer programs and data stored in the ROM 212 or RAM 213 . Note that the image processing apparatus 102 may have one or more dedicated hardware or a GPU (Graphics Processing Unit) different from the CPU 211 . At least part of the processing by the CPU 211 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I/F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and sound volume data.

The operation unit 215 is composed of, for example, a keyboard and a mouse, and inputs various instructions to the CPU 211 in response to user's operations. The CPU 211 operates as a display control unit that controls the display device 104 and as an operation control unit that controls the operation unit 215 . A communication I/F 216 is used for communication with an external device of the image processing apparatus 102 . For example, when the image processing apparatus 102 is connected to an external apparatus by wire, a communication cable is connected to the communication I/F 216 . If the image processing device 102 has a function of wirelessly communicating with an external device, the communication I/F 216 has an antenna.

A bus 217 connects each unit of the image processing apparatus 102 and transmits information. Although the controller 103 and the display device 104 are provided as external devices in this embodiment, they may be provided as one of the functional units of the image processing device 102 .

<About software configuration>
FIG. 3 is a diagram showing an example of the software configuration of the image processing apparatus 102. As shown in FIG. The image processing device 102 has a data acquisition unit 300 , a model generation unit 301 , a model analysis unit 302 and a virtual viewpoint image generation unit 303 . The virtual viewpoint image generating unit 303 has a projecting unit 304 and a synthesizing unit 305 . The function of each part will be described below.

The data acquisition unit 300 acquires various data used for generating a virtual viewpoint image, specifically, a plurality of captured image data synchronously captured by each camera 101, an imaging parameter set for each camera 101, and virtual viewpoint information. . The acquired data is stored in the auxiliary storage device 214 .

The model generation unit 301 generates shape data representing the three-dimensional shape of a foreground object such as a stage performer (hereinafter referred to as a "foreground model") based on a plurality of captured images and the imaging parameter set of each camera 101. For example, it is generated by the visual volume intersection method. The foreground model of this embodiment is represented by a textured polygon mesh or a three-dimensional point group with each point colored, and is generated in a general-purpose format file such as Stanford PLY or Wavefront OBJ. Note that the foreground model may be shape data without color information. In this case, the projection unit 304, which will be described later, performs a process (texture mapping) for adding a color corresponding to the virtual viewpoint. The generated foreground model is stored in the auxiliary storage device 214 .

The model analysis unit 302 analyzes the shape change (movement) of the foreground model generated by the model generation unit 301 over a plurality of times, detects a portion of the foreground model with a large amount of change, and uses it as a target for afterimage expression. Determine specific parts.

A projection unit 304 projects a foreground model onto a virtual viewpoint based on virtual viewpoint information input from the controller 103 to generate a projection image corresponding to the virtual viewpoint. Here, in addition to the above-mentioned foreground model, the projection target includes shape data corresponding to a specific part of the foreground model (hereinafter referred to as "partial model") and shape data of the background object (hereinafter referred to as "background model"). (referred to as “models”). The background model is shape data with color information representing the three-dimensional shape of the background object such as stage equipment, and is generated in advance and stored in the auxiliary storage device 214 . The background model may be generated using design data such as CAD, or shape and color data scanned by a laser scanner or the like. Alternatively, it may be generated from captured images from multiple viewpoints using a computer vision technique such as structure from motion.

A synthesizing unit 305 synthesizes projection images obtained by projecting each 3D model (foreground model, partial model, background model) onto a virtual viewpoint by the projection unit 304, and creates a virtual viewpoint that expresses a specific part like an afterimage. Generate an image. Data of the generated virtual viewpoint image is transmitted to the display device 104 .

Note that each functional unit described above may be distributed among a plurality of image processing apparatuses. For example, the first image processing device may have the data acquisition unit 300 and the model generation unit 301, and the second image processing device may have the model analysis unit 302 and the virtual viewpoint image generation unit 303. FIG.

<Flow of generating virtual viewpoint image>
Next, a flow of generating a virtual viewpoint image in the image processing device 102 will be described with reference to the flowchart of FIG. A series of processes shown in the flowchart of FIG. 4 is realized by the CPU 211 reading out a control program stored in the ROM 212 or the auxiliary storage device 214 or the like, developing it in the RAM 213, and executing it. In the following description, symbol "S" means step.

In S401, the data acquisition unit 300 acquires a plurality of captured image data for a continuous period (for example, 5 seconds), the imaging parameter set of each camera 101, and virtual viewpoint information. The acquired data is developed and held in the RAM 213 .

In S402, the model generation unit 301 uses the captured image data and the imaging parameter set acquired in S401 to generate a foreground model corresponding to each of a plurality of times in the past or future from the reference time t of interest in the continuous period. to generate Here, the time of interest is, for example, the time corresponding to a specific key frame specified by the time code in the acquired continuous period. In addition, a plurality of times in the past or future for which the foreground model is to be generated may be set in advance or may be specified in the virtual viewpoint information.

In S403, the model analysis unit 302 analyzes the time-dependent change in the shape of the foreground model generated in S402, and determines a specific part for which afterimage expression is desired. Specifically, based on the generated foreground model for a plurality of time points, a part that moves significantly over time is detected, and the detected part is determined as a specific part to be subjected to afterimage expression processing. FIG. 5(a) is a diagram for explaining the detection of a specific part, and is a diagram showing foreground models corresponding to different times superimposed on the y-axis in the same space. FIG. 5(b) is a diagram showing the relationship between times. In FIG. 5B, a thin double-headed arrow 501 indicates the generation range of the virtual viewpoint image, and a thick double-headed arrow 502 indicates the range to be represented as an afterimage. The time t−n is a time other than the reference time t within the range 502, and the variable n takes an arbitrary integer corresponding to the above-mentioned “a plurality of past or future times”. For example, "n=1" represents one hour past the reference time t, "n=2" represents two hours past, and "n=-1" represents one hour future of the reference time t. become. Now, in FIG. 5A, the one-dot chain line represents the left arm of the foreground model at time t-1, and the two-dot chain line represents the left arm of the foreground model at time t-2. In other words, it means that only the left arm is moving from time t-2 to reference time t. This means that only the portion of the shape represented by the foreground model that corresponds to the left arm has a large amount of change between different times indicated by the range 502 . In this way, by calculating the amount of change in the foreground model between different times, it is possible to detect a moving part of the foreground object. For calculating the amount of change, for example, if the foreground model is in the form of a polygon mesh, the distance between polygons can be used. That is, each vertex of the foreground model at a certain time is searched for the nearest neighbor point in the normal direction to each vertex of the foreground model at another time, and the point of contact with the mesh of the foreground model at another time is taken as the nearest neighbor point. Then, the distance between the vertex and the nearest point is obtained as a signed distance. The signed distance is a distance that takes a positive value if the closest point is outside the mesh and a negative value if it is inside the mesh when viewed from the vertex. This signed distance is obtained for all the vertices of the foreground model at a certain time, and the vertices whose absolute value of the distance is larger than the standard deviation (σ) of the distribution are extracted. Then, the part of the foreground model composed of the extracted vertex group is detected. In the example of FIG. 5A, each vertex of the portion corresponding to the left arm of the foreground model has a large absolute value of the signed distance, so it is extracted by the above procedure. A part of the foreground model composed of the extracted vertices is determined as a specific part to be subjected to afterimage expression processing. Then, the above-described processing is performed for all multiple times corresponding to the foreground model generated in S402, and the specific part at each time is determined. Shape data corresponding to the determined specific portion (shape data corresponding to a partial portion of the foreground model; hereinafter referred to as "partial model") is stored in the RAM 213 separately from the foreground model.

In S404, the projection unit 304 performs processing for projecting the foreground model at the reference time, the partial model of the specific part determined in S403, and the background model prepared in advance based on the virtual viewpoint information. Known model-based rendering or image-based rendering may be used for projection processing. FIG. 6(a) shows the projected image of the foreground model 501 at the reference time s, and (b) and (c) show the left arm partial model of the foreground model 501 at the time s−1 and the time s−2, respectively. shows a projected image of Also, FIG. 6(d) shows a projected image of the background model. These projection images are, for example, 4-channel images in which each pixel has an 8-bit (0-255) RGB value plus an alpha value. Here, the alpha value is a numerical value representing transparency, and when the maximum value is 255, it is opaque. That is, the alpha value of pixels onto which the shape data of each model is projected is greater than 0, and the alpha value of pixels onto which nothing is projected is 0. Therefore, the alpha value of each pixel is set to 255 for the projected image of the background model that is not transmitted at all and the projected image of the foreground model at the current time s. On the other hand, the projected image of the partial model to be transmitted to express the afterimage is 255×0.6=153 for time s−1 and 255×0.3=77 for time s−2. Set different alpha values accordingly. In this case, "0.6" and "0.3" are coefficients for controlling transmittance, and values less than 1.0 corresponding to the difference value from the reference time are entered. That is, in the examples of (b) and (c) of FIG. 6 described above, the transmittance increases (the alpha value decreases) as the corresponding specific time is past the reference time. . The transmittance control method described here is merely an example, and for example, the transmittance may be increased as the specific time corresponding to the partial model is in the future with respect to the reference time.

In S405, the synthesizing unit 305 synthesizes all the projection images generated in S404 to generate a virtual viewpoint image. FIG. 7(a) is a diagram showing how the four projection images shown in FIGS. 6(a) to 6(d) are alpha-combined. As shown in FIG. 7(a), four projected images 701 to 704 are superimposed in layers and alpha synthesized to express the left arm as an afterimage as shown in FIG. 7(b). A virtual viewpoint image is obtained.

In S406, it is determined whether or not the processing has been completed for all the reference times designated by the virtual viewpoint information. If there is an unprocessed reference time, the process advances to S407 to update the reference time of interest, returns to S402, and continues the process. At this time, if the virtual viewpoint information designates that the virtual viewpoint moves with the passage of time, a virtual viewpoint image projected at an arbitrary camera angle can be obtained. If the processing has been completed for all reference times, this flow ends.

The above is the flow of generating a virtual viewpoint image according to the present embodiment. The data of the virtual viewpoint image thus obtained is transmitted to the display device 104 for viewing by the user. Note that the target foreground object is not limited to a dynamic object such as a person, and may be a static object such as a building. For example, based on the captured images obtained by continuously capturing the construction process of a high-rise building over a certain period, only the newly constructed parts are made opaque and the existing building parts are made transparent. , a virtual viewpoint image corresponding to each reference time is obtained as shown in FIGS. In this case, for example, the building is imaged every day (S401), a foreground model is generated from the obtained captured image (S402), and a portion with a large shape change compared to the day before (that is, the portion built on the day) ) is detected to determine the specific site (S403). Then, the partial model corresponding to the part to be built on the current day and the foreground model for the previous day are projected onto the virtual viewpoint (S404), and alpha synthesis is performed using the obtained projected image (S405). At this time, by setting the alpha value for the current day to "127" and the alpha value for the previous day to "255", etc., a virtual viewpoint image is obtained in which the difference from the previous day can be intuitively grasped. be done.

<Modification 1>
In the above-described embodiment, the specific part is determined based on the shape change of the foreground model during a plurality of times, but the method of determining the specific part is not limited to this. That is, prior to determining the specific part, projection processing is performed on the virtual viewpoint for a plurality of times, the difference between the obtained projection images is obtained for a plurality of times, and the part corresponding to the image region with a large difference value is selected as the specific part. may decide. In this case, a partial model corresponding to the determined specific part is generated from the foreground model, and the partial model is again projected onto the virtual viewpoint to obtain projection images corresponding to FIGS. 7(b) and 7(c). Then, the synthesis processing described above may be performed.

<Modification 2>
Instead of alpha synthesis that changes transmittance, afterimages may be expressed by color change processing that changes brightness or saturation, for example. In this case, each pixel of the projection image becomes a three-channel projection image of RGB without an α value. Then, the brightness or saturation may be decreased as the specific time corresponding to the partial model becomes past the reference time, or the brightness or saturation may be decreased as the specific time becomes future relative to the reference time. This makes it possible to visually represent the movement of the object over time. In addition, highlight processing may be performed to make a specific part stand out at a specific time.

<Modification 3>
In the above-described embodiment, projection images corresponding to a plurality of times are superimposed and synthesized in a layered manner, but in order to accurately express the shielding relationship of the foreground object at the plurality of times, synthesis is performed using depth information. may That is, when generating a projection image for each model in S404, a depth image is also generated in which the distance from the virtual viewpoint to the foreground object is recorded for each pixel. Then, when synthesizing in S405, projection images are selected and synthesized for each pixel such that the smaller the depth (that is, the closer the distance to the virtual viewpoint), the higher the layer. This makes it possible to express a more stereoscopic image that appropriately expresses the actual shielding relationship.

<Modification 4>
In the above-described embodiment, the motion of the foreground object is detected to determine the specific part, but it may be determined based on the positional relationship with the specific area in the three-dimensional space where the image was captured. Specifically, a bounding box is set in a three-dimensional space such that a person's shoulders are located at the boundary of the bounding box and the parts other than the arm are included, and parts inside or outside the bounding box are detected. to determine the specific site. The shape of the bounding box is arbitrary, and any shape such as a rectangular parallelepiped, a cylinder, a sphere, or an elliptical sphere can be used as long as the inside/outside can be determined in a three-dimensional space. Also, the position, orientation, and size of the bounding box may be changed for each time (for each frame). Information such as the shape and size of the bounding box, whether it changes at each time, whether to detect the inside or the outside, etc. may be stored in advance in the auxiliary storage device 214 and read out and used as necessary. This makes it possible to more easily determine the specific site.

As described above, according to the present embodiment, it is possible to easily obtain a virtual viewpoint image in which a moving part of a foreground object is represented by an afterimage.

[Embodiment 2]
In the first embodiment, by analyzing temporal changes in the shape of the foreground model and detecting parts with large movements, specific parts to be subjected to afterimage expression are automatically determined. Next, a mode of analyzing the structure of the generated foreground model, presenting the result to the user, and determining the specific part based on the user's explicit instruction will be described as a second embodiment. Further, in the first embodiment, it is assumed that parts other than specific parts do not change over a plurality of times. It will be explained together. Note that the description of the contents common to the first embodiment, such as the basic system configuration, will be omitted, and the differences will be mainly described below.

<Flow of generating virtual viewpoint image>
A flow of generating a virtual viewpoint image according to this embodiment will be described with reference to the flowchart of FIG. A series of processes shown in the flowchart of FIG. 9 is realized by the CPU 211 reading a control program stored in the ROM 212 or the auxiliary storage device 214 or the like, developing it in the RAM 213, and executing it. In the following description, symbol "S" means step.

S901 and S902 respectively correspond to S401 and S402 in the flowchart of FIG. 4 according to the first embodiment. That is, captured image data, imaging parameter sets, and virtual viewpoint information for a continuous period are acquired (S901), and then foreground models corresponding to each of a plurality of times are generated (S902).

In S903, the model analysis unit 302 analyzes the structure of the foreground model generated in S902. A known method may be applied to the structural analysis. For example, there is a known method of minimizing the distance between the surfaces of the reference bone model and the foreground model to superimpose the reference bone model on the foreground object and associate the bone structure (Non-Patent Document 1). In addition, the method of "OpenPose", which uses a deep learning model to infer the bone structure with the image of the object as an input, or the method of motion capture may be used. In the case of a polygon mesh format foreground model, the structural analysis result is obtained as metadata that associates a part name and part ID with each mesh polygon that is a constituent element. The structural analysis results thus obtained are stored in the RAM 213 .

In S904, the specific site is determined based on user input via the UI screen reflecting the structural analysis results obtained in S903. More specifically, first, data of a UI screen for designating a specific part for which afterimage representation is desired (part designation UI screen) is transmitted to the display device 104 and displayed on the display device 104 . FIG. 10 is an example of a part designation UI screen. In the case of the UI screen 1000 of FIG. 10, the foreground model of the target for designating the specific part is displayed in the center of the screen, and on the right side thereof, a list 1001 of all the parts obtained by the structural analysis and a desired time out of the plurality of times described above are displayed. A seek bar 1002 for designating the time is displayed. The user operates the pointer 1003 using a mouse or the like to designate a specific part where the afterimage representation is desired. For example, when specifying the left arm, an operation of moving the pointer 1003 to the vicinity of the fingertips of the left hand while dragging it is performed after moving the pointer 1003 over the left shoulder. For the part specified by the user in this way, the corresponding part name in the list 1001 is highlighted, for example. Thereby, the user can confirm the fact that the site specified by the user has been selected as the specific site. The above designation method is an example. For example, a button corresponding to each part obtained by structural analysis may be provided so that the desired part can be designated by pressing the button corresponding to the part.

S905 corresponds to S404 of the first embodiment. That is, the process of projecting the foreground model at the reference time s, the partial model of the specific part determined in S904, and the background model prepared in advance based on the virtual viewpoint information is performed. At this time, in this embodiment, the position, posture, etc. of the partial model corresponding to the specific part determined in S904 are adjusted as necessary. FIG. 11A is a diagram for explaining how the position and posture of the left arm determined as the specific part are adjusted according to the shape data of the foreground model at the reference time so that the base of the left arm matches each time. Now, a human foreground object is moving from time t-1 to time t while moving its left arm. Foreground model 1100 corresponds to time t and foreground model 1100' corresponds to time t-1. In the adjustment, geometric transformation is performed so that the left arm base 1101' of the foreground model 1100' at time t-1 overlaps with the left arm base 1101 of the foreground model 1100 at time t. At this time, if the scales of the left arm differ between time t−1 and time t, for example, transformation processing may be performed to match the scales. FIG. 11(b) shows the result of the adjustment, in which the foreground model 1100′ at time t−1 and the foreground model at time t are three-dimensionally aligned with the base of the left arm designated as the specific part. inherently overlapped. Such adjustments are made to the partial model as necessary, and the foreground model and the partial model are projected onto the virtual viewpoint.

S906 to S908 correspond to S405 to S407 of the first embodiment, respectively, and are not particularly different, so description thereof is omitted.

The above is the flow of generating a virtual viewpoint image according to the present embodiment. Note that the structural analysis of the foreground model at each time may be performed by another device prior to the start of the flow of FIG. 9, and the result may be acquired in S903. Further, when the adjustment is performed in S905, the adjustment result may be displayed on the UI screen so that the user can fine-tune the position, orientation, and the like.

<Modification 1>
Instead of changing the transmittance of the specific site designated by the user according to the time, the transmittance around the specific site may be changed according to the distance from the specific site. Such an afterimage representation is useful for analyzing a golf swing, for example. That is, the head of the golf club is determined as a specific portion, and after making the head opaque, the foreground model of the golf club is subjected to transparency processing when projecting so that the more transparent the head is, the further away it is from the head. In this transparency processing, while referring to the structural analysis results, the alpha value is decreased as the distance from the specified head portion increases, and the golf club is made transparent in a gradation pattern, and the foreground model at each time is displayed as a virtual viewpoint. Project. At this time, regarding the relationship between the distance and the alpha value, a transmission start distance and a transmission end distance may be set in advance. Alternatively, the distance from the head portion to the farthest portion may be set as the transmission end distance after detecting a portion with large movement. The transmission start distance at this time may be the distance from the head to the nearest portion. FIG. 12(a) shows an example of a virtual viewpoint image obtained by synthesizing processing according to this modification. The projected image of the foreground model at the most recent time and the projected image of the foreground model at each time in the past, processed so that the golf club becomes gradually transparent from the base of the head to the handle. The projection image obtained by the projection image is synthesized. According to this modified example, it is possible to obtain a virtual viewpoint image that emphasizes the trajectory of the specific part specified by the user.

By applying the method of this modified example and superimposing the method of the above-described embodiment, which makes the image more transparent as the time goes past, a virtual viewpoint image as shown in FIG. 12B is obtained. be done. At this time, the virtual viewpoint may be moved (for example, moving at a constant speed starting from a position where the person is seen from the front and moving to a position where the person is seen from the back at the end of hitting). As a result, a virtual viewpoint image is obtained in which the movement of the club head portion is projected at an arbitrary camera angle.

As described above, according to the present embodiment, it is possible to easily obtain a virtual viewpoint image in which only the user-specified portion of the foreground object is expressed as an afterimage. Further, by adjusting the position and orientation of the foreground model as necessary, a virtual viewpoint image with less sense of discomfort can be obtained.

(Other examples)
The present disclosure provides a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

102 Image processing device 300 Data acquisition unit 302 Model analysis unit 303 Virtual viewpoint image generation unit 304 Projection unit 305 Synthesis unit

Claims (28)

Acquisition means for acquiring shape data of an object corresponding to each of a plurality of times during a period in which images are continuously captured by a plurality of imaging devices;
determining means for determining a specific portion that is part of the object;
A virtual viewpoint corresponding to a virtual viewpoint using shape data of the object at a reference time among the plurality of times and shape data of a specific portion of the object at a specific time different from the reference time among the plurality of times generating means for generating an image;
An image processing device comprising: The generating means projects a first projection image obtained by projecting shape data of the object at the reference time onto the virtual viewpoint, and shape data of a specific portion of the object at the specific time onto the virtual viewpoint. 2. The image processing apparatus according to claim 1, wherein the virtual viewpoint image is generated by synthesizing the second projection image obtained by the above. The first projection image and the second projection image are 4-channel images in which each pixel has an alpha value in addition to RGB values,
The generating means generates the virtual viewpoint image by alpha synthesis using the first projected image and the second projected image.
3. The image processing apparatus according to claim 2, wherein: In the alpha synthesis, the transmittance of pixels corresponding to the specific part at the specific time in the second projection image differs from the transmittance of pixels corresponding to the specific part at the reference time. Item 4. The image processing apparatus according to item 3. When the specific time is earlier than the reference time, the generating means determines that the transmittance of the pixels corresponding to the specific site at the specific time is higher than the transmittance of the pixels corresponding to the specific site at the reference time. 5. The image processing apparatus according to claim 4, wherein said alpha composition is performed so as to be higher than a rate. The generating means performs the alpha synthesis such that the transmittance of pixels corresponding to the specific part in the second projection image increases as the specific time is past the reference time. The image processing apparatus according to claim 5. The generating means, when the specific time is later than the reference time, the transmittance of the pixels corresponding to the specific part at the specific time is higher than the transmittance of the pixels corresponding to the specific part at the reference time. 5. The image processing apparatus according to claim 4, wherein said alpha composition is performed so as to be higher than a rate. The generating means performs the alpha synthesis such that the transmittance of pixels corresponding to the specific portion in the second projection image increases as the specific time is later than the reference time. The image processing apparatus according to claim 7. 4. The method according to claim 3, wherein in said alpha synthesis, the transmittance of pixels corresponding to said specific portion in said second projection image differs from the transmittance of pixels corresponding to the periphery of said specific portion. image processing device. 3. The generating means performs the alpha synthesis so that the transmittance of pixels corresponding to the specific portion is higher than the transmittance of pixels corresponding to the periphery of the specific portion. 9. The image processing apparatus according to 9. The generating means performs the alpha synthesis such that the transmittance of pixels corresponding to the periphery of the specific site in the second projection image increases as the distance from the specific site increases. Item 11. The image processing device according to Item 10. the first projection image and the second projection image are 3-channel images in which each pixel has an RGB value;
The generation means generates the virtual viewpoint image by synthesizing the first projection image and an image obtained by changing the brightness or saturation of the second projection image.
3. The image processing apparatus according to claim 2, wherein: In the synthesis, the brightness or saturation of pixels corresponding to the specific part at the specific time in the second projection image is different from the brightness or saturation of pixels corresponding to the specific part at the reference time. 13. The image processing apparatus according to claim 12. When the specific time is earlier than the reference time, the generating means converts the brightness or saturation of the pixels corresponding to the specific part at the specific time to the brightness of the pixels corresponding to the specific part at the reference time. 14. The image processing apparatus according to claim 13, wherein said synthesis is performed so as to lower color saturation. The generating means performs the synthesis so that the more the specific time is past the reference time, the lower the brightness or saturation of pixels corresponding to the specific part at the specific time in the second projection image. 15. The image processing apparatus according to claim 14, characterized by: When the specific time is later than the reference time, the generating means converts the brightness or saturation of the pixels corresponding to the specific part at the specific time to the brightness of the pixels corresponding to the specific part at the reference time. 14. The image processing apparatus according to claim 13, wherein said synthesis is performed so as to lower color saturation. The generation means performs the synthesis such that the further the specific time is later than the reference time, the lower the brightness or saturation of pixels corresponding to the specific part at the specific time in the second projection image. 17. The image processing apparatus according to claim 16, characterized by: The generating means is
generating a depth image in which the distance from the virtual viewpoint to the object is recorded for each pixel;
Selecting the second projection image for each pixel so that the lower the depth, the higher the layer, and performing the synthesis;
3. The image processing apparatus according to claim 2, wherein: 3. The image processing apparatus according to claim 2, wherein said generating means performs said synthesis by geometrically transforming the shape data corresponding to said specific part. The geometric transformation is processing for adjusting at least one of the position, posture, and scale of the shape data corresponding to the specific part in accordance with the shape data of the object at the reference time. The image processing apparatus according to claim 19. The determining means is
analyzing shape changes of the object at the plurality of times using the shape data;
Based on the results of the analysis, determining a portion with a large amount of change as the specific site;
21. The image processing apparatus according to any one of claims 1 to 20, characterized by: The determining means is
analyzing the shape change of the object at the plurality of times using projection images obtained by projecting the shape data of the object corresponding to each of the plurality of times onto a virtual viewpoint;
Based on the results of the analysis, determining a portion with a large amount of change as the specific site;
21. The image processing apparatus according to any one of claims 1 to 20, characterized by: The determining means is
using the shape data to analyze the positional relationship between the object and a specific region in the three-dimensional space where the imaging was performed at the plurality of times;
Based on the results of the analysis, determining a portion inside or outside the specific region as the specific site;
21. The image processing apparatus according to any one of claims 1 to 20, characterized by: 24. The image processing apparatus according to claim 23, wherein said specific area is a bounding box. further comprising a graphical user interface for accepting designation of the specific part from the user;
the determining means determines the specific part based on the received designation;
21. The image processing apparatus according to any one of claims 1 to 20, characterized by: The graphical user interface displays results of structural analysis of the object,
The user designates the specific site based on the displayed structural analysis result,
26. The image processing apparatus according to claim 25, characterized by: an acquisition step of acquiring shape data of an object corresponding to each of a plurality of times during a period in which images are continuously captured by a plurality of imaging devices;
a determining step of determining a specific portion that is part of said object;
A virtual viewpoint corresponding to a virtual viewpoint using shape data of the object at a reference time among the plurality of times and shape data of a specific portion of the object at a specific time different from the reference time among the plurality of times a generation step for generating an image;
An image processing method comprising: A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 26.

Images