JP2021033347A - Image processing device, image processing method, and program - Google Patents

Abstract

To suppress a data volume of a 3D model generated by interpolation to an existing 3D model for generating virtual view point image data of an arbitrary frame rate.SOLUTION: A model analysis unit 304 acquires basic 3D models divided for each of sections being interpolation 3D model generation objects at consecutive two basic time points, that is, basic 3D models and dividing position information. The model analysis unit 304 calculates a motion vector for each of sections of basic 3D models M(T1) and M(T2) containing the acquired divided sections. An interpolation time point determination unit 305 calculates interpolation time points being time points to perform the interpolation for each of the sections of the basic 3D models of the objects. A model interpolation unit 306 generates an interpolation 3D model for a section to which the interpolation time point is set, and stores the model together with dividing section information, model connection information, and a time code indicating the interpolation time point in a model storage unit 303.SELECTED DRAWING: Figure 5

Description

The present invention relates to a technique for generating virtual viewpoint image data.

A virtual viewpoint that expresses the view from a virtual viewpoint by installing a plurality of imaging devices at different positions and using a plurality of captured image data (multi-viewpoint image data) obtained by synchronously imaging the plurality of imaging devices from a plurality of viewpoints. There is a technology to generate image data. The virtual viewpoint image data is generated by rendering using a 3D model, which is three-dimensional shape data of an object generated from a plurality of viewpoint image data, and captured image data. When the virtual viewpoint image data is moving image data, rendering is performed on the 3D model corresponding to each frame image constituting the virtual viewpoint image data.

When this virtual viewpoint image data is generated using only the multiple viewpoint image data, it is generated at the same frame rate as the multiple viewpoint image data, but in the distribution of the virtual viewpoint image data, various other different frame rates are used. Delivery is also required. This means that the virtual viewpoint image data is played back and adjusted to a different frame rate according to the processing capacity of the device for viewing, or the same frame rate is viewed even if the playback speed is changed. This is to make it possible.

In response to such needs, Non-Patent Document 1 tracks the vertices of the mesh constituting the existing 3D model in time, interpolates the vertices in a time zone in which the existing 3D model does not exist, and shortens the time. 3D models are generated at intervals. This makes it possible to generate a 3D model corresponding to each frame image of the virtual viewpoint image data at an arbitrary frame rate.

Alvaro Collet, et al., “High-quality streamable free-viewpoint video.”, ACM Transactions on Graphics (TOG), 34 (4), 2015.

However, it is not realistic in terms of storage area and processing cost to generate in advance a 3D model corresponding to all frame rates that the user may request by using this conventional technique. On the other hand, if a 3D model corresponding to the specified frame rate is generated each time there is a request to change the frame rate, and the 3D model is rendered to generate virtual viewpoint image data, the frame rate is changed. There is a large delay from the request to playback, which is not practical.

Therefore, it is an object of the present invention to appropriately generate a 3D model using an existing 3D model.

One aspect of the present invention is an acquisition means for acquiring a first 3D model associated with a time code representing a three-dimensional shape of an object, and two first aspects associated with two consecutive time codes. 3D model and a generation means for generating a second 3D model corresponding to the time between the times indicated by the two consecutive time codes based on the moving speeds of the two first 3D models. It is an image processing apparatus characterized by being provided.

According to the present invention, an existing 3D model can be used to appropriately generate a 3D model.

It is a figure which shows the configuration example of the virtual viewpoint image generation system in embodiment. It is a figure which shows the hardware configuration of the information processing apparatus in embodiment. It is a block diagram which shows the structure of the virtual viewpoint image data generation processing in embodiment. It is a flowchart which shows the basic model generation processing in embodiment. It is a flowchart which shows the interpolation 3D model generation processing in embodiment. It is a figure which shows the basic 3D model of the continuous model time in an embodiment. It is a figure which shows the interpolation 3D model in an embodiment. It is a flowchart which shows the virtual viewpoint image data generation processing for distribution in an embodiment. It is a figure explaining the acquisition method of the 3D model corresponding to an arbitrary frame rate in an embodiment. It is a figure explaining the acquisition method of the 3D model corresponding to a predetermined frame rate in an embodiment. It is a figure which shows the interpolation 3D model in an embodiment. It is a figure which shows the interpolation 3D model after synthesis in an embodiment. It is a figure explaining the synthesis method of the interpolation model in an embodiment. It is a flowchart which shows the interpolation time calculation process in an embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

The present invention can be applied regardless of the objects included in the virtual viewpoint image data, but in the present embodiment, the virtual viewpoint image generation system in which a soccer game is imaged and a stadium, a person, a ball, etc. are objects. explain.

Virtual viewpoint image generation is used for generating virtual viewpoint image data. The virtual viewpoint image generation generates virtual viewpoint image data representing the appearance from a specified virtual viewpoint based on a plurality of image data obtained by synchronous imaging by a plurality of imaging devices and a designated virtual viewpoint. It is a stem. The virtual viewpoint image represented by the virtual viewpoint image data in the present embodiment is also called a free viewpoint image. However, the virtual viewpoint image is not limited to the image corresponding to the viewpoint freely (arbitrarily) specified by the user, and includes, for example, an image corresponding to the viewpoint selected by the user from a plurality of candidates. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like.

The configuration of the virtual viewpoint image generation system targeted by this embodiment will be described with reference to FIG. At the stadium, a plurality of cameras 101 are arranged in the stadium. The object 102 is an object of the system, and is, for example, a player, a referee, a ball, or the like. The plurality of cameras 101 transmit the synchronously captured images to the information processing device 103.

Although the plurality of cameras 101 and the information processing device 103 are connected in a star type in FIG. 1, they may be connected in a daisy chain type. The information processing device 103 is connected to a display device 104 that displays a transmitted image and a processing result of the information processing device 103, and an input device 105 for inputting to the information processing device 103.

The information processing device 103 can distribute the generated virtual viewpoint image data in response to a request from user terminals 106 and 107 such as a PC, a tablet, and a smartphone via a network. The user terminals 106 and 107 can specify the virtual viewpoint image data to be reproduced, specify the reproduction section, specify the virtual viewpoint, specify the frame rate, and the like for the information processing device 103.

The hardware configuration of the information processing apparatus 103 will be described with reference to FIG. The information processing device 103 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display I / F 215, an operation I / F 216, a communication I / F 217, and a bus 218. The display I / F 215 and the operation I / F 216 operate as a display control unit in which the CPU 211 controls the display device 104 and an operation control unit in which the operation unit 216 is controlled.

The CPU 211 realizes each function of the information processing device 103 shown in FIG. 1 by controlling the entire information processing device 103 using computer programs and data stored in the ROM 212 and the RAM 213. The information processing device 103 may have one or more dedicated hardware different from the CPU 211, and the dedicated hardware may execute at least a part of the processing by the CPU 211. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 212 stores programs and the like that do not require changes. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I / F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display device 104 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for operating the information processing device 103. The input device 105 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 211 in response to an operation by the user.

The communication I / F 217 is used for communication with an external device of the information processing device 103. For example, when the information processing device 103 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the information processing device 103 has a function of wirelessly communicating with an external device, the communication I / F 217 includes an antenna. Further, the communication I / F 217 is used for connecting to the network 108 and performing data communication with an external device such as the user terminals 106 and 107 via the network 108.

The bus 218 connects each part of the information processing device 103 to transmit information.

In the present embodiment, it is assumed that the display device 104 and the input device 105 exist as separate devices outside the information processing device 103, but at least one of the display device 104 and the input device 105 exists inside the information processing device 103. You may be doing it. In this case, the CPU 211 may operate as a display control unit that controls the display device 104 and an operation control unit that controls the operation unit 216.

FIG. 3 is a block diagram showing a functional configuration of the information processing apparatus 103 that performs the virtual viewpoint image data generation processing in the present embodiment.

The multi-viewpoint image imaging unit 301 acquires image data from a plurality of cameras 101 in synchronization with each other in time. Here, the imaging time at which each frame image data of the plurality of viewpoint image data is generated indicates the time in the imaging time based on the imaging start time point. On the other hand, the time (model time (basic time, interposition time), frame time) used in the processing described later refers to a time on a time axis different from the time axis of the imaging time, and therefore, in the present disclosure, the time in the content time. And. However, the basic time is a time corresponding to each frame image data of the plurality of viewpoint image data on a one-to-one basis, and the interpolation time is a time corresponding to between the frame image data.

The basic model generation unit 302 acquires the captured image data obtained by synchronously imaging from the multi-viewpoint image imaging unit 301, and is three-dimensional shape data having the same frame rate as the captured image data by the basic model generation process described later. Generate a basic 3D model.

The model storage unit 303 stores the basic 3D model generated by the basic model generation unit 302, the model junction information, the interpolation 3D model generated by the model insertion unit 306, and the like.

The model analysis unit 304 acquires the basic 3D model from the model storage unit 303, performs mesh tracking and motion vector calculation, and acquires the velocity information of each 3D model.

The interpolation time determination unit 305 determines the interpolation time in the content time for interpolation for each 3D model. In the present embodiment, since it is determined whether or not the interpolation is performed depending on whether or not the interpolation time is set, the interpolation time determination unit 305 also has an interpolation determination function.

The model insertion unit 306 generates an interpolation 3D model of the interpolation time determined by the interpolation time determination unit 305 based on the basic 3D model and the motion vector which is the velocity information, and represents the interpolation time. The interpolated 3D model generated together with is stored in the model storage unit 303.

The model acquisition unit 307 receives an arbitrary frame rate designation from the user via the input / output unit 309, and is required for the frame rate specified from the 3D model including the interpolated 3D model stored in the model storage unit 303. Acquire the 3D model corresponding to the frame image data.

The virtual viewpoint image data generation unit 308 generates virtual viewpoint image data for distribution at a specified frame rate using the 3D model acquired by the model acquisition unit 307 and sends it to the input / output unit 309.

The distribution virtual viewpoint image data in the present embodiment is data in which the information of the 3D model in each frame is recorded in the OBJ format at the frame rate specified by the user. The user views the image by reproducing it on the user terminal with virtual viewpoint image data reproduction software or the like. The 3D model information may be described in a format other than OBJ, the capacity of the 3D model information may be compressed, or a streaming distribution format may be used.

FIG. 4 is a flowchart of the basic model generation process of the present embodiment. This flow is executed at the timing when the information processing apparatus 103 receives the multi-viewpoint image data obtained by synchronously capturing a new image by the multi-viewpoint image capturing unit 301.

In S401, the basic model generation unit 302 acquires the multi-viewpoint image data obtained by being imaged from the multi-viewpoint image imaging unit 301 by a plurality of cameras 101 in time synchronization.

In S402, the basic model generation unit 302 generates a basic 3D model by an image-based modeling technique using the multi-viewpoint image data acquired in S401. Here, the 3D model including the basic 3D model is represented by a list of the positions of the vertices constituting the 3D model and a list of mesh information regarding the vertices, sides and faces constituting the mesh. The basic 3D model is stored in the model storage unit 303 together with a time code indicating the original model time in the content time as a base 3D model for generating virtual viewpoint image data at an arbitrary frame rate.

Here, among the objects captured in the captured image, the object for which the 3D model is generated in the image-based modeling can be freely selected. In the present embodiment, since the virtual viewpoint image data generation system installed in the soccer stadium is used, "player" and "soccer ball" are targeted as the objects for generating the 3D model.

In S403, the basic model generation unit 302 divides the basic 3D model generated in S402. Here, each basic 3D model is divided into parts that make common movements, and each part is treated as an individual 3D model. By interpolating only the part having a certain amount of movement or more, the amount of processing for generating the interpolated 3D model can be reduced. This makes it possible to efficiently generate a 3D model corresponding to virtual viewpoint image data at an arbitrary frame rate.

As a method of detecting the divided parts of the basic 3D model, for example, there is a method of learning a 3D model of each part for each object by using machine learning and detecting each part based on the learning result. Also, instead of directly detecting the 3D model of each part, look at the cross-sectional view of the 3D model from one end to the other, and detect the joint between the parts from the change in the size of the cross-sectional view. There is also a method of detecting each part with.

As for the parts to be divided, for example, the fineness and speed of movement are often different between the human arm and the body, so it is efficient to treat these two parts as different division models. In the present embodiment, the ball is not divided into a single part, and the person is divided into six parts: the head, the torso, the left arm, the right arm, the left foot, and the right foot. The number of parts to be divided is not limited to six. For example, the arm may be further divided into three parts: shoulder to elbow, elbow to wrist, wrist to fingertip, and the foot may be further divided into three parts. , It may be divided into 14 parts in total. Further, the number of parts to be divided and the division position may be changed according to the characteristics of the movement of the object.

Further, the portion to be divided may not be determined in advance as described above, but may be divided into a portion consisting of vertices having a movement amount equal to or more than the threshold value and a portion consisting of vertices having a movement amount less than the threshold value.

In the division processing of the basic 3D model, specifically, the division part information and the model joining information which is the information of the joints between the divided parts are combined with the basic 3D model in the model storage unit 303. To store. The division site information is a list of the indexes of the vertices for each part to be divided, and the model joining information is a list of the indexes of the vertices shared between the parts to be joined. With this model joining information, it is possible to appropriately join (synthesize) the interpolated 3D model for each divided part in the model acquisition unit 307, which will be described later.

In the present embodiment, when the basic 3D model of the human shape is divided, the model joining information between the torso and the other five parts (head, left arm, right arm, left foot, right foot) is stored in the model storage unit 303. In this case, the number of joints of each joint is two. Each part of the head, left arm, right arm, left foot, and right foot has one joint, and a list of apex indexes shared between the parts is created and used as model joint information. On the other hand, the body part has five joints, and a list of apex indexes shared between the parts for all five joints is created and used as model joint information.

FIG. 5 is a flowchart of the interpolation model generation process of this embodiment. In this flow, when the new basic 3D model of the new original model time is stored in the model storage unit 303 by the basic model generation unit 302, the new basic 3D model of the new original model time and the original model immediately before it are stored. It is executed for the basic 3D model of time. The generation of the interpolated 3D model is performed for each object, and the number of interpolated 3D models generated for each object may be different.

In S501, the model analysis unit 304 acquires the basic 3D model divided for each part, that is, the basic 3D model and the divided part information at the time of two consecutive original models to be the target of the interpolation 3D model generation. Here, of the two basic 3D models, the one with the earlier basic 3D model time T1 is referred to as the basic 3D model M (T1), and the one with the later original model time T2 is referred to as the basic 3D model M (T2).

FIG. 6 is a diagram showing an example of a basic 3D model of continuous model time in the embodiment. In the present embodiment, the player's basic 3D model is divided into a head 11, a right arm 12, a left arm 13, a torso 14, a right foot 15, and a left foot 16 based on the division site information.

In S502, the model analysis unit 304 calculates a motion vector for each of the basic 3D models M (T1) and M (T2) including the divided parts acquired in S501. In this embodiment, mesh tracking is performed using simple vertex matching using distance. Specifically, when there is a vertex group P (T1) of the basic 3D model M (T1) and a vertex group P (T2) of the basic 3D model M (T2), each vertex of the vertex group P (T1) , It is performed by associating the vertices having the shortest distance from the vertex group P (T2). For mesh tracking, the method shown in Non-Patent Document 1 may be used. The motion vector is a vector representing the movement between the vertices of the underlying 3D model associated by mesh tracking. The motion vector is the displacement amount from the three-dimensional position coordinates of each vertex of the vertex group P (T1) to the three-dimensional position coordinates of the corresponding vertex of the vertex group P (T2) associated with mesh tracking on each coordinate axis. Obtained by asking.

In S503, the interpolation time determination unit 305 calculates the interpolation time, which is the time for interpolation, for each part of the basic 3D model of each object. In order to calculate the interpolation time, first, the number of interpolations N for interpolation is determined between the adjacent basic times. The number of interpolations N is determined from the motion vector obtained by the model analysis unit 304, the time interval between adjacent basic times, the permissible speed Va determined by the builder of this system, and the minimum time resolution Tmin. In the example shown in FIG. 6, the time interval between adjacent basic times is the time interval between the basic times T1 and T2 of the basic 3D models M (T1) and M (T2).

In the present embodiment, the average of the motion vectors for all the vertices belonging to a certain part is used as the representative motion vector representing the part. Assuming that the magnitude of the representative motion vector per second, that is, the moving speed representing the part is the representative speed V, the number of interpolations N between adjacent basic times can be obtained by the following equation (1).
N = min (Rounddown (V / Va), (T2-T1) / Tmin) Equation (1)

Here, Rounddown represents a function for finding an integer with the decimal point truncated. According to the equation (1), interpolation is performed when the representative velocity V is equal to or higher than the permissible velocity Va, and the minimum value of the interpolation interval is suppressed to the minimum time resolution Tmin. The number of interpolations N is obtained for each adjacent basic time and for each divided portion. The number of interpolations N increases as the movement is larger, and the number of interpolations N increases as the movement increases. ..

The permissible speed Va and the minimum time resolution Tmin are determined in consideration of the processing capacity of the information processing apparatus 103, the complexity of the basic 3D model, the time resolution assumed as the user's request, and the like. As a result, the maximum number of interpolation times that can be set for each basic time is also determined. When the number of interpolations N determined based on the equation (1) is not 0 (V ≧ Va), the interpolation times T1-i (i = 1,. ., N) is set. The number of interpolation times set for each adjacent basic time is N. When the number of interpolations is 0 (V <Va), the interpolation time T1-i is not set because the interpolation is not performed. The interpolation time represents a time in the same content time as the original model time. In addition, the basic time and the interpolation time are collectively referred to as the model time.

In the present embodiment, interpolation is performed using linear interpolation, the interpolation time T1-i is set at equal intervals between the basic times T1 and T2, and the time interval between the basic times is the number of interpolations at the basic time. By adding the value divided by, it is calculated as in the following equation (2).
T1-i = T1 + (T2-T1) / (N + 1) × i equation (2)

Here, FIG. 14 shows a flowchart of the interpolation time calculation process of S503. In S1401, the interpolation time determination unit 305 selects one unselected part. Next, in S1402, the interpolation time determination unit 305 calculates the number of interpolations N based on the equation (1) from the motion vector, the time interval between adjacent basic times, the permissible speed Va, and the minimum time resolution Tmin. To do. In S1403, the interpolation time determination unit 305 determines whether or not the number of interpolations N is 0, shifts to S1404 if the number of interpolations N is not 0, and S1405 if the number of interpolations N is 0. Move to. When the number of interpolations N is not 0 but shifts to S1404, the interpolation time determination unit 305 calculates the interpolation time based on the equation (2). In S1405, the interpolation time determination unit 305 determines whether or not all the parts have been selected, returns to S1401 if there are unselected parts, and ends the process if all the parts are selected. ..

Whether or not to perform interpolation is determined as described above depending on whether or not the interpolation time is set. In addition, whether or not to perform interpolation depends on whether or not the number of interpolations N is 0. Whether or not the number of interpolations N is 0 depends on whether the moving speed of the basic 3D model is equal to or higher than a predetermined speed (V ≧ Va) or lower than a predetermined speed (V <Va). Whether or not it will be determined by the moving speed of the basic 3D model.

In S504, the model insertion unit 306 generates an interpolation 3D model for the part where the interpolation time is set in S503, and models the model together with the division part information, the model junction information, and the time code indicating the interpolation model time. It is stored in the storage unit 303. The interpolation 3D model is not generated for the part where the interpolation time is not set in S503.

FIG. 7 is a diagram showing an interpolated 3D model M (T1-1) generated when N = 1 with respect to the basic 3D models M (T1) and M (T2) shown in FIG. As a result of calculating the number of interpolations N for the basic 3D models M (T1) and M (T2) shown in FIG. 6, the representative speed V of only the left foot 16 and the soccer ball 17 was equal to or higher than the permissible speed Va. Therefore, as shown in FIG. 7B, the interpolation time was set only for the left foot 16 ″ and the soccer ball 17 ″. In this case, as for the head 11, right arm 12, left arm 13, torso 14, and right foot 15 of the interpolated 3D model M (T1-1), those of M (T1) or M (T2) are used as they are. In this way, by generating the interpolated 3D model only for the part or the object having a large movement, it is possible to suppress the amount of data of the 3D model after the interpolation.

In the present embodiment, the interpolated 3D model is generated by performing linear interpolation of the vertex coordinates using the motion vector calculated for each vertex in S502. Therefore, in this embodiment, linear interpolation is used to generate an interpolated 3D model between the basic 3D models of the two original model times T1 and T2. However, the technique of the present disclosure is not limited to this embodiment, and in order to generate an interpolated 3D model, the number of reference basic 3D models is increased to 3 or more, and non-linear vertex coordinate interpolation such as quadratic interpolation is performed. The method may be used.

FIG. 8 is a flowchart of the virtual viewpoint image data generation process for distribution of the present embodiment. This flow is executed at the timing when a request is made to reproduce the specific virtual viewpoint image data acquired from the user via the input / output unit 309 at a specific frame rate f. The frame rate f may be specified by the user via the user terminals 106 and 107, or may be determined according to the processing capacity of the user terminals 106 and 107 and the connection status with the information processing device 103. ..

In S801, the model acquisition unit 307 sets a time code representing the time (reproduction start time and reproduction end time) of the reproduction section of the distribution virtual viewpoint image data generated in this process and the frame rate f of the distribution virtual viewpoint image data. get. The time code of the reproduction section acquired in S801 may be only the reproduction time and may be the timing when the user requests the end of the reproduction, or the reproduction end time may be the timing of the end of the data of the virtual viewpoint image data for distribution. Good. The frame rate f may be determined based on the input via the input device 105 or the like by the user, or may be automatically determined from the processing capacity of the user terminals 106 and 107. The frame rate f automatically determined by the user terminals 106 and 107 may be, for example, the maximum frame rate of the user terminals 106 and 107, or a frame rate at which playback is not interrupted depending on the condition of the communication line.

In S802, the model acquisition unit 307 acquires the model time T closest to the designated playback start time, and based on the acquired model time T, each frame image data at a time interval corresponding to the designated frame rate f. Determines the frame time Tf of.

In S803, the model acquisition unit 307 is close to the frame time Tf acquired in S801 from the 3D model including the basic 3D model and the interpolated 3D model stored in advance in the model storage unit 303, preferably the closest model time. Acquire a 3D model of T. Further, when the acquired 3D model is an interpolated 3D model, the division site information and the model joining information are also acquired. Further, among the parts included in the model joining information, for the part where the interpolated 3D model does not exist, the corresponding part of the basic 3D model of the model time T closest to the model time T of the interpolated 3D model is acquired.

FIG. 9 is a diagram illustrating a method of acquiring a 3D model according to an arbitrary frame rate f in the embodiment. Here, a case where virtual viewpoint image data of 40 fps and 100 fps is generated by using a 3D model interpolated at N = 3 with respect to a basic 3D model generated from a plurality of viewpoint image data of 60 fps will be described.

When generating 40 fps virtual viewpoint image data, the basic 3D model M (T1) is used as the 3D model for the frame time Tf1, and the 3D model for the frame time Tf2 is within the model time that matches the frame time Tf2. Insert 3D model M (T2-2) is used. As described above, when the model time T that matches the frame time Tf exists, the 3D model of the model time T that matches the frame time Tf is used to generate the frame image data of the frame time Tf.

On the other hand, when generating virtual viewpoint image data of 100 fps, the basic 3D model M (T1) is used as the 3D model for the frame time Tf1, but there is no model time that matches the frame time Tf2. Therefore, as the 3D model for the frame time Tf2, the interpolated 3D model M (T1-2) at the model time T1-2 closest to the frame time Tf2 is used. Similarly, as the 3D model for the frame time Tf3, the interpolated 3D model M (T2-1) of the model time T2-1 closest to the frame time Tf3 is used. As the 3D model for the frame time Tf4, the interpolated 3D model M (T2-3) of the model time T2-3 closest to the frame time Tf4 is used.

As described above, when there is no model time T that matches the frame time Tf, the 3D model of the model time T closest to the frame time Tf is used. As a result, virtual viewpoint image data having an arbitrary frame rate f can be generated. It is preferable to use a 3D model of the model time T that matches the frame time Tf, or to use a 3D model of the model time T whose time interval is narrower than that of the frame time Tf because of high quality virtual viewpoint image data.

FIG. 10 is a diagram illustrating a method of acquiring a 3D model according to a predetermined frame rate f in the embodiment. When the selectable frame rate f of the virtual viewpoint image data is predetermined, by setting the number of interpolation times N appropriately, high quality virtual viewpoint image data can be obtained without holding a lot of 3D models unnecessarily. Can be generated. As shown in FIG. 10, when the selectable frame rates f are 120 fps and 180 fps, there is a 3D model of the model time T that matches all the frame times Tf by setting the number of interpolations N = 5. become. As a result, high-quality virtual viewpoint image data can be generated, and the number of 3D models to be retained can be reduced as compared with the case where the number of interpolations N ≧ 6.

Further, although not shown, when the model time T has a wider time interval than the frame time Tf because the movement is small and the number of interpolations N is small, the same model time T at a plurality of frame times Tf. Use the 3D model of. As described above, in the portion where the amount of change of the object is small and the effect of interpolation is small, the amount of data of the 3D model can be reduced without generating the interpolated 3D model.

In S804, the model acquisition unit 307 generates a synthetic 3D model obtained by synthesizing the interpolated 3D model for each part acquired in S803 so as to be a 3D model for each object before division based on the division part information and the model joining information. To do. For the part where the interpolated 3D model does not exist, the corresponding part of the basic 3D model at a time close to the interpolation time of the interpolated 3D model to be synthesized is used. At this time, the number of interpolated 3D models for each object after synthesis between adjacent basic times is the maximum number of interpolation times N for each part.

Here, in the present embodiment, by referring to the model joint information, it is possible to synthesize the joints of each part so that the joints do not separate. For example, if the same vertices in the basic 3D model before division move to different positions for each part after division, all the vertices that were the same vertices before division are collected at their midpoint positions. And combine. Details will be described later.

In S805, the model acquisition unit 307 acquires a 3D model in the reproduction section including the synthetic 3D model synthesized in S804 and the basic 3D model. Then, the virtual viewpoint image data generation unit 308 acquires and renders the virtual viewpoint information input by the user on the user terminals 106 and 107, and generates distribution virtual viewpoint image data representing the appearance from the virtual viewpoint. The virtual viewpoint information is typically acquired at a predetermined playback frame rate fd on the user terminals 106 and 107. For example, in a user terminal having a default playback frame rate fd of 60 fps, virtual viewpoint information is acquired once every 1/60 second.

If the default playback frame rate fd and the frame rate f specified by the user match, the changed virtual is used by using the 3D model of the model time T that matches the changed frame time Tf of the virtual viewpoint information. Generate virtual viewpoint image data that represents the view from the viewpoint. On the other hand, the default playback frame rate fd may not match the frame rate f of the virtual viewpoint image data for distribution specified by the user. In such a case, the 3D model of the model time T closest to the timing when the user terminal acquires the virtual viewpoint information is used to generate virtual viewpoint image data representing the appearance from the changed virtual viewpoint.

In S806, the model acquisition unit 307 distributes the virtual viewpoint image data generated in S805 from the input / output unit 309 to the user terminals 106 and 107 via the network 108.

If a request for changing the frame rate f is made during distribution of the virtual viewpoint image data, the playback start time may be set as the timing at which the request for changing the frame rate is made, and the processes S801 to S806 may be performed.

Here, the 3D model synthesis process in S804 will be described. FIG. 11 is a diagram showing the interpolated 3D model shown in FIG. 7B, in which the apex B1 of the body 14 and the apex L1 of the left foot 16 ″, which were the same vertices in the basic 3D model, are merged. It is a figure of the previous state. Further, FIG. 12 is a diagram of a state after the apex B1 and the apex L1 of FIG. 11 are merged at their midpoint I1. In FIG. 11, the body 14 and the left foot 16 ″ are transformed into the body 14 ″ and the left foot 16 ″, respectively, so as to move the apex B1 and the apex L1 to the midpoint I1.

The 3D model synthesis process will be further described in detail with reference to FIG. FIG. 13 is a diagram for explaining the details of the method of synthesizing the model for each of the divided parts, and will be described using a simplified 2D model for convenience.

FIG. 13A shows the left arm consisting of vertices v1, v2, v3 of the basic 3D model and the torso consisting of vertices v2, v3, v4, v5. Here, the vertices shared by the left arm and the body are v2 and v3. In this case, in S403 of FIG. 4, the vertex indexes v2 and v3 are stored as model joining information of the body and the left arm. In the example of FIG. 13, there are two joining vertices, but in S403, all the vertices shared with different parts are detected, and all of the vertex indexes are stored as model joining information.

FIG. 13B shows the left arm and the torso interpolated for each part, and the vertices v2 and v3 shared by the left arm and the torso are the vertices v2'and v3' for the left arm and the torso is the apex. It moves to v2'' and v3'', respectively, and shows a state in which the left arm and the torso are separated after interpolation.

FIG. 13 (c) shows a state in which the interpolated left arm and the torso are combined. When the coordinates of the vertices v2 and v3 stored in the model joining information are different between the left arm and the torso as shown in FIG. 13 (b), the vertices v2'and v2'' are moved to their midpoint v2'''. , Move the vertices v3', v3'' to their midpoint v3'''. By the movement of the vertices v2 ′, v2 ″, v3 ′, and v3 ″, the interpolated left arm and the torso are deformed and joined to be synthesized.

By referring to the model joint information in this way, it is possible to combine the separated parts by joining the joints at the time of division.

The above is a configuration example of an image processing device that efficiently provides virtual viewpoint image data having an arbitrary frame rate f in terms of storage area and processing cost. As a result, it is possible to deliver the virtual viewpoint image data in a short time from the reproduction request from the user while suppressing the amount of data of the 3D model to be held to the minimum necessary amount.

Further, in FIG. 8, S804, which is a step of synthesizing the interpolated 3D model, is configured to be performed after S801 to S803, but S804 may be performed before S801. That is, after the interpolation 3D model is generated in the interpolation model generation process of FIG. 5, a process of synthesizing the interpolation 3D model for all of the generated interpolation 3D models may be performed. By synthesizing the interpolated 3D model in advance, the delay from the request by the user to the reproduction can be further shortened.

Further, in the present embodiment, the configuration includes the multi-viewpoint image imaging unit 301, the basic model generation unit 302, the virtual viewpoint image data generation unit 308, and the input / output unit 309, but these are external devices having the same functions. It can be replaced. That is, the information processing apparatus 103 may be configured not to perform the processes shown in FIGS. 4, 5, and 8 for S402, S501, S805, and S806.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

305 Interpolation time determination unit 306 Model interpolation unit 307 Model acquisition unit

Claims (17)

An acquisition means for acquiring a first 3D model associated with a time code representing a three-dimensional shape of an object,
Based on the movement speeds of the two first 3D models associated with the two consecutive timecodes and the two first 3D models, the time between the times indicated by the two consecutive timecodes A generation means for generating the corresponding second 3D model, and
An image processing device comprising. The image processing apparatus according to claim 1, wherein the generation means generates the second 3D model based on the movement speed being equal to or higher than a predetermined speed. The image according to claim 1, wherein the generation means does not generate the second 3D model based on the two first 3D models when the moving speed is less than a predetermined speed. Processing equipment. The generation means determines the time according to the time interval between the two consecutive time codes corresponding to the first 3D model and the number of the second 3D models interpolated between the time codes. Is added to the time of the time code corresponding to the first 3D model immediately before the interpolation, thereby determining the time code associated with the second 3D model. Item 3. The image processing apparatus according to any one of Items 1 to 3. The generation means determines the maximum number of the second 3D models inserted between the two consecutive time codes corresponding to the first 3D model based on the minimum time resolution of the generation means. The image processing apparatus according to claim 4. The generation means determines the number of the second 3D models to be generated for each of the two consecutive time codes corresponding to the first 3D model according to the movement speed of the movement speed. The image processing apparatus according to any one of claims 1 to 5, wherein the image processing apparatus is characterized. The acquisition means divides the first 3D model corresponding to a predetermined object into a plurality of parts, and makes the plurality of parts individual objects, and joins the plurality of parts. The image processing apparatus according to any one of claims 1 to 6, wherein the joining information for designating a part is acquired. The acquisition means acquires the first 3D model in which the first 3D model corresponding to the predetermined object is divided into a plurality of parts according to the moving speed, and the parts are the individual objects. The image processing apparatus according to claim 7, wherein the image processing apparatus is used. Further provided with a synthesis means for generating a third 3D model by synthesizing the second 3D model for each of the divided parts based on the joining information so as to be a 3D model for each object before division. The image processing apparatus according to claim 7 or 8. When the second 3D model is generated only for a part of the 3D model of the object unit before the division, the synthesis means corresponds to a time code close to the time code of the second 3D model. The image processing apparatus according to claim 9, wherein a portion of the attached first 3D model that is not in the second 3D model is synthesized with the second 3D model. Further provided is a selection means for selecting a 3D model corresponding to the frame image data of the virtual viewpoint image data from the first 3D model and the third 3D model based on the specified playback section, virtual viewpoint and frame rate. The image processing apparatus according to claim 9 or 10. The selection means includes the first 3D model or the third 3D model associated with the time code that matches the playback time determined based on the designated playback section, virtual viewpoint, and frame rate. If not, the first 3D model or the third 3D model associated with the time code closest to the playback time is selected as the 3D model corresponding to the frame image data of the virtual viewpoint image data. 11. The image processing apparatus according to claim 11. The image processing apparatus according to claim 11 or 12, further comprising a generation means for generating the virtual viewpoint image data based on a 3D model corresponding to the frame image data selected by the selection means. The image processing device according to any one of claims 11 to 13, wherein the designated frame rate is specified based on the terminal device on which the virtual viewpoint image data is reproduced. Claims 1 to 14 characterized in that the first 3D model acquired by the acquisition means is a 3D model generated based on a plurality of image data obtained by synchronous imaging using a plurality of imaging devices. The image processing apparatus according to any one of the above items. The step of acquiring the first 3D model associated with the time code, which represents the 3D shape of the object,
Based on the movement speeds of the two first 3D models associated with the two consecutive timecodes and the two first 3D models, the time between the times indicated by the two consecutive timecodes Steps to generate the corresponding second 3D model,
An image processing method characterized by having. A program for operating a computer as the image processing device according to any one of claims 1 to 15.

Images