JP2022051978A - Image processing device, image processing method, and program - Google Patents

Abstract

An object of the present invention is to prevent a user viewing a free viewpoint image from losing sight of a desired object. A trajectory information generator generates a trajectory in a three-dimensional space of an object shown in viewpoint images according to designation of a playback position of free viewpoint data generated from a plurality of viewpoint images shot from a plurality of viewpoints. Generate trajectory information to represent The present technology can be applied when viewing a free-viewpoint image. [Selection drawing] Fig. 5

Description

The present technology relates to an image processing device, an image processing method, and a program, and in particular, an image processing device and an image processing that enable a user who views a free-viewpoint image to suppress losing sight of a desired object. Regarding methods and programs.

A method of generating a strobe image showing an object (subject) taken at a plurality of times has been proposed (see, for example, Patent Document 1). Since objects at multiple times are shown in the strobe image, the movement and trajectory of the objects can be easily grasped.

Japanese Unexamined Patent Publication No. 2007-259477

By the way, in recent years, a free-viewpoint image that reproduces the appearance when a three-dimensional space is viewed from a virtual viewpoint has attracted attention.

With respect to the content of the free viewpoint image, it is expected that the user viewing the free viewpoint image loses sight of the desired object when the time seek operation (time seek operation) is performed.

The present technology has been made in view of such a situation, and makes it possible to prevent a user who views a free-viewpoint image from losing sight of a desired object.

The image processing device or program of the present technology is in the three-dimensional space of the object reflected in the viewpoint image according to the designation of the reproduction position of the free viewpoint data generated from the plurality of viewpoint images taken from the plurality of viewpoints. It is a program for making a computer function as an image processing device provided with a locus information generation unit for generating locus information representing the locus of the above, or as such an image processing device.

The image processing method of the present technology represents a locus in a three-dimensional space of an object reflected in the viewpoint image according to a designation of a reproduction position of free viewpoint data generated from a plurality of viewpoint images taken from a plurality of viewpoints. It is an image processing method including generating trajectory information.

In the image processing device, the image processing method, and the program of the present technology, the object reflected in the viewpoint image is specified according to the designation of the reproduction position of the free viewpoint data generated from the plurality of viewpoint images taken from the plurality of viewpoints. Trajectory information representing the locus in the three-dimensional space of is generated.

The image processing device may be an independent device or an internal block constituting one device.

In addition, the program can be distributed by transmitting via a transmission medium or by recording on a recording medium.

It is a figure explaining the example of reproduction of the content of a 2D image. It is a figure explaining the example of the time seek of the content of a free viewpoint image. It is a block diagram which shows the structural example of one Embodiment of the image processing system to which this technique is applied. It is a block diagram which shows the configuration example of the content server 22. It is a block diagram which shows the structural example of the reproduction apparatus 23. It is a flowchart explaining the example of the process performed by the content server 22. It is a flowchart explaining the example of the reproduction processing of the free viewpoint data performed by the reproduction apparatus 23. It is a flowchart explaining the details of the process of the step S35 which generates the locus information when the strobe model is adopted as the locus information. It is a figure which shows the example of the unnatural 3D strobe image. It is a figure which shows the example of a natural 3D strobe image. It is a figure which shows the example of the frame of the viewpoint image of a strobe section. It is a figure which shows the example of the generation of the strobe model using the frame of time t1 to t9 as a strobe section. It is a figure which shows the example of the display of the 3D strobe image generated by the shooting of the strobe model by the virtual camera. It is a figure explaining the effect processing performed on the 3D model arranged in a strobe model. It is a figure explaining the specific example of effect processing. It is a figure explaining the example of the generation of the strobe model when the time seek operation is performed when the strobe model is adopted as the locus information. It is a figure explaining the example of the change of the state of a virtual camera when the strobe model is generated and the 3D strobe image is displayed when the time seek operation is performed. It is a figure explaining the example of the generation of the line data when the time seek operation is performed when the line data is adopted as the locus information. It is a figure explaining the example of the effect processing performed on the 3D model arranged in the strobe model as the locus information. It is a flowchart explaining another example of the reproduction processing of the free viewpoint data performed by the reproduction apparatus 23. It is a figure explaining the first setting example of the virtual camera in the case of setting a virtual camera so that the 3D model of the target object appearing in the frame of a designated reproduction position is included in the shooting range of a virtual camera. It is a figure explaining the 2nd setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object appearing in the frame of a designated reproduction position is included in the shooting range of a virtual camera. It is a figure explaining the 3rd setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object appearing in the frame of a designated reproduction position is included in the shooting range of a virtual camera. It is a figure explaining the 4th setting example of the virtual camera in the case of setting a virtual camera so that the 3D model of the target object appearing in the frame of a designated reproduction position is included in the shooting range of a virtual camera. It is a figure explaining the 5th setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object appearing in the frame of a designated reproduction position is included in the shooting range of the virtual camera. It is a block diagram which shows the structural example of one Embodiment of the computer to which this technique is applied.

<Playback of 2D image content>

FIG. 1 is a diagram illustrating an example of reproducing the contents of a 2D image.

That is, FIG. 1 shows an example of a player that reproduces the contents of a 2D (Dimensional) image.

The player has a display unit 11 for displaying a 2D image, and a seek bar 12 as a GUI (Graphical User Interface) for performing a time seek. Time seek is one of the methods for designating a playback position (time).

The seek bar 12 has a slider 13 that represents a reproduction position, and the position of the slider 13 in the seek bar 12 represents a reproduction position. In the seek bar 12, the left end represents the beginning of the content and the right end represents the end of the content. For example, when the content is normally reproduced, the slider 13 moves from left to right according to the movement of the reproduction position of the normal reproduction.

As a time seek operation, the user can operate the seek bar 12 (slider 13) to specify the reproduction position. That is, when the user operates the seek bar 12 as a time seek operation and the position of the slider 13 is moved, the 2D image of the playback position (frame) corresponding to the position after the movement of the slider 13 is reproduced. (Displayed on the display unit 11).

Therefore, by operating the seek bar 12 as a time seek operation, the user can see (and listen to the sound) the 2D image of the playback position corresponding to the position of the slider 13 moved by the operation of the seek bar 12. You can easily find the desired scene.

In FIG. 1, for the contents of a 2D image showing a soccer game, while the user is operating the seek bar 12 as a time seek operation, an image of the time corresponding to the seek bar operation at that time is displayed. For example, if the camera keeps chasing a player, the user can of course keep looking at the player in the image without losing sight of it.

In addition to the operation of the seek bar 12, the time seek operation can be performed by an operation of instructing fast forward reproduction, rewind reproduction (reverse reproduction), an operation of instructing chapter jump, and the like.

<Time seek of free viewpoint image>

FIG. 2 is a diagram illustrating an example of time seek of the content of a free viewpoint image.

Here, in recent years, distribution of images (videos) via the Internet has become common, and as for the method of viewing images, in addition to the method using a TV (Television) receiver or a screen in a movie theater, a PC ( It is becoming more diverse, such as methods using personal computer) browsers, smartphones, tablets, etc.

In addition to 2D image (video) content with an aspect ratio of 16: 9 and 4: 3, content with free viewpoint data has appeared.

The free viewpoint data is data expressing the three-dimensional structure of the three-dimensional space, and according to the free viewpoint data, an image that can be seen when the object that is the subject of the three-dimensional space is viewed from a predetermined viewpoint, that is, a predetermined It is possible to generate a 2D image obtained by projecting a viewing volume (or view volume) in a three-dimensional space represented by free viewpoint data on a projection plane that is perpendicular to the line-of-sight direction when the object is viewed from the viewpoint. can. Since this 2D image can be generated by freely setting the viewpoint, it can be called a free viewpoint image. Since the viewpoint (including the line of sight (direction)) required for generating the free viewpoint image can be virtually set regardless of the actual viewpoint of the user, such a viewpoint is also referred to as a virtual viewpoint.

The free viewpoint image (moving image) can be displayed on a two-dimensional planar display such as a PC monitor or a smartphone.

In addition, the free viewpoint image can be displayed on a head-mounted display (HMD). In this case, the head tracking technology can realize a stereoscopic view as if it exists in a three-dimensional space.

Furthermore, the free-viewpoint image can be displayed on a head-up display (HUD) using a transparent display such as AR (Augmented Reality) glass that allows the other side to be seen through. In this case, an object such as a person or an object photographed in another three-dimensional space can be superimposed and displayed on the three-dimensional space in which the user actually exists.

Like the 2D image, the free viewpoint data has a degree of freedom in the time direction, and also has a virtual viewpoint, that is, a shooting position and a shooting posture (3D space represented by the free viewpoint data) of the virtual camera. It has a degree of freedom in the shooting direction). The shooting position of the virtual camera can be represented by, for example, the coordinates of the xyz coordinate system as the world coordinate system, and the shooting posture of the virtual camera can be represented by the rotation angle or the like around each axis of the world coordinate system. The shooting position of the virtual camera can be moved in the x-axis, y-axis, and z-axis directions, and the shooting posture of the virtual camera can be rotated around the x-axis, y-axis, and z-axis. Since it is possible, there is a degree of freedom of 6DoF (Degree of Freedom) in the shooting position and shooting posture of the virtual camera. Therefore, with respect to the free viewpoint data, there are a total of seven degrees of freedom, one degree of freedom in the time direction and six degrees of freedom in the shooting position and shooting posture (virtual viewpoint) of the virtual camera.

Here, the free-viewpoint image generated by projecting the free-viewpoint data (viewing volume in the three-dimensional space represented by) on the projection surface with respect to the virtual viewpoint is as if it were virtual from the virtual viewpoint. With the camera, you can obtain a 2D image as if you were shooting free viewpoint data. Therefore, it can be said that projecting the free viewpoint data on the projection surface with respect to the virtual viewpoint is equivalent to shooting the free viewpoint data from the virtual viewpoint with a virtual camera. That is, projecting the free viewpoint data on the projection surface with respect to the virtual viewpoint to generate the free viewpoint image can be regarded as taking the free viewpoint data with a virtual camera arranged in the virtual viewpoint. .. Therefore, in the present embodiment, the generation of the free-viewpoint image is regarded as the shooting of the free-viewpoint data with a virtual camera, and the description will be given as appropriate. A virtual camera that captures free-viewpoint data is also referred to as a virtual camera.

The shooting position and shooting posture of the virtual camera can be set by the user by operating the joystick or other controller (device) that controls the position and posture.

When the HMD or HUD is used for viewing the free viewpoint data, the shooting position and shooting posture of the virtual camera should be set so as to correspond to the user's viewpoint and line of sight (direction) detected by the HMD or HUD. Can be done.

For the content of 2D images, there is only one degree of freedom in the time direction, but for the content of free viewpoint data, there is one degree of freedom in the time direction, and the shooting position and shooting posture (virtual viewpoint) of the virtual camera. There are 6 degrees of freedom and 7 degrees of freedom.

Therefore, when the time seek operation is performed on the content of the free viewpoint data, the user is more likely to lose sight of the desired object than in the case of the content of the 2D image.

Specifically, for the content of the free viewpoint data, a three-dimensional space represented by the free viewpoint data, that is, a shooting space that is a three-dimensional space taken by (multiple) cameras to obtain the free viewpoint data. The 2D image taken by the virtual camera is displayed as a free viewpoint image.

As for the content of the free viewpoint data, an object that is within the shooting range of the virtual camera at a certain shooting position and shooting posture at the current playback position (time) is within the shooting range of the virtual camera even at another playback position. It does not always enter.

In FIG. 2, a time seek operation is performed in which a time seek is performed from the current reproduction position to the past reproduction position. Further, in FIG. 2, an object that is in the shooting range of the virtual camera at the current playback position is out of the shooting range of the virtual camera at the current playback position at the past playback position.

In this case, at the current playback position, the object appears in the 2D image of the shooting space taken with the virtual camera, but at the past playback position (playback position after the time seek operation), the shooting space was shot with the virtual camera. Objects do not appear in 2D images.

Therefore, the user loses sight of the object when performing a time seek operation to the past playback position.

For example, when the content of the free viewpoint data is the content of a soccer match, as described in FIG. 1, the user shot from where the soccer player who scored the goal came from and how the pass was received. When a time seek operation is performed to perform a time seek in the past direction from the playback position where the scene where the goal is scored is reflected, the soccer player who scored the goal at the playback position after the time seek operation is asked. If the user is not within the shooting range of the virtual camera at the playback position where the scene that scored the goal appears, the user loses sight of the soccer player who scored the goal.

Therefore, in the present technology, the locus information representing the locus of the object in the three-dimensional space is generated according to the designation of the reproduction position of the free viewpoint data by the time seek operation or the like. By displaying the locus image which is a free viewpoint image captured by the virtual camera, it is possible to prevent the user who views the free viewpoint image from losing sight of the desired object.

<Embodiment of an image processing system to which this technology is applied>

FIG. 3 is a block diagram showing a configuration example of an embodiment of an image processing system to which the present technology is applied.

The image processing system of FIG. 3 includes a photographing device 21, a content server 22, and a reproducing device 23.

The photographing device 21 is composed of at least a plurality of cameras, and photographs an object from a plurality of viewpoints. For example, a plurality of cameras constituting the photographing device 21 are arranged so as to surround the object, and each camera photographs the object from the viewpoint as the position where the camera is arranged. A 2D image taken from the position of the camera by each camera, that is, a viewpoint image (moving image) of a plurality of viewpoints which is a 2D image taken from a plurality of viewpoints is obtained from the shooting device 21 to the content server on a frame-by-frame basis. It is supplied to 22.

Here, the photographing device 21 may be provided with a plurality of distance measuring devices in addition to the plurality of cameras. The distance measuring device may be arranged at the same position (viewpoint) as the camera, or may be arranged at a position different from that of the camera. The distance measuring device measures the distance from the position (viewpoint) where the distance measuring device is arranged to the object, and generates a depth image which is a 2D image whose pixel value is the depth which is information about the distance. The depth image is supplied from the photographing device 21 to the content server 22.

When the photographing device 21 is not provided with a distance measuring device, the distance to the object is measured by the principle of triangulation using the viewpoint images of two viewpoints out of the viewpoint images of a plurality of viewpoints. , Can generate depth images.

The content server 22 is, for example, a web server or a server on the cloud. The content server 22 generates content (moving image content) of free viewpoint data by using, for example, viewpoint images of a plurality of viewpoints supplied from the photographing device 21. The content server 22 transmits the content of the free viewpoint data to the reproduction device 23 in response to the request from the reproduction device 23.

The playback device 23 is, for example, a client of a mobile terminal such as a PC (Personal Computer), a TV (television receiver), or a smartphone. The reproduction device 23 requests and acquires the content of the free viewpoint data from the content server 22. The reproduction device 23 reproduces the content of the free viewpoint data acquired from the content server 22. That is, the reproduction device 23 generates and displays a 2D image as a free viewpoint image obtained by capturing free viewpoint data with a virtual camera.

Further, the reproduction device 23 determines the trajectory in the shooting space (three-dimensional space) of the predetermined object reflected in the viewpoint image according to the designation of the reproduction position of the free viewpoint data by the user's time seek operation or the like (of the reproduction device 23). Generates the locus information to be represented. Further, the reproduction device 23 generates and displays a 2D image (free viewpoint image) as a locus image taken by a virtual camera of the locus information.

Here, as the locus information, for example, a strobe model can be adopted. The strobe model is a model in which a 3D model of the same object (subject) of a plurality of frames (time) of a viewpoint image is arranged in a shooting space (3D model) as a three-dimensional space shot by the shooting device 21. And can be generated using free viewpoint data.

As shown in FIG. 3, the content server 22 and the playback device 23 may be configured as separate devices, or may be configured as one device as a whole (for example, a PC (Personal Computer) or the like). You can also do it. Further, the content server 22 can be composed of a plurality of devices as a whole. The same applies to the reproduction device 23.

<Configuration example of content server 22>

FIG. 4 is a block diagram showing a configuration example of the content server 22 of FIG.

The content server 22 has a free viewpoint data generation unit 31, a storage unit 32, and a communication unit 33.

The free viewpoint data generation unit 31 generates the content of the free viewpoint data by generating free viewpoint data in frame units from the viewpoint images and depth images of a plurality of viewpoints from the photographing device 21.

Here, the free viewpoint data means a wide range of 3D image data (3D data) that can generate a free viewpoint image. As the free viewpoint data, for example, a set of viewpoint images and depth images of a plurality of viewpoints from the photographing device 21 can be adopted as they are. In addition, as the free viewpoint data, for example, 3D data having a 3D shape model which is information on a three-dimensional shape and color information, or a set of a 2D image and a depth image of a plurality of viewpoints should be adopted. Can be done.

When a set of viewpoint images and depth images of a plurality of viewpoints from the photographing device 21 is adopted as the free viewpoint data, the free viewpoint data generation unit 31 includes the viewpoint images and the depth images of the plurality of viewpoints from the photographing device 21. The set of is used as it is as free viewpoint data. When the viewpoint of the depth image from the photographing device 21 is different from the viewpoint of the viewpoint image from the photographing device 21, the free viewpoint data generation unit 31 photographs using the depth images of a plurality of viewpoints from the photographing device 21. It is possible to generate a depth image of the viewpoint of the viewpoint image from the device 21.

When 3D data having a 3D shape model and color information is adopted as the free viewpoint data, the free viewpoint data generation unit 31 models by Visual Hull or the like using the viewpoint images of a plurality of viewpoints from the photographing device 21. To generate a 3D shape model of the object reflected in the viewpoint image, and use it as free viewpoint data together with the viewpoint images of multiple viewpoints that serve as textures.

When a set of 2D images and depth images of a plurality of viewpoints is adopted as the free viewpoint data, the free viewpoint data generation unit 31 uses, for example, 3D data having a 3D shape model and color information as described above. A set of 2D images and depth images viewed from a plurality of viewpoints (the same viewpoint as the camera constituting the photographing device 21 or different viewpoints) is generated as free viewpoint data. do.

As described above, the free viewpoint data generation unit 31 generates the content of the free viewpoint data and supplies it to the storage unit 32.

As the free viewpoint data, the amount of free viewpoint data can be reduced by adopting a set of 2D images and depth images of a plurality of viewpoints as compared with 3D data having a 3D shape model. The technique of generating and transmitting a set of 2D images and depth images of multiple viewpoints as free viewpoint data is described in International Publication No. 2017/082076 previously proposed by the applicant. A set of 2D images and depth images from multiple viewpoints as free viewpoint data includes, for example, 2D images such as MVCD (Multiview and depth video coding), AVC (Advanced Video Coding), and HEVC (High Efficiency Video Coding). It can be encoded by the target coding method.

Here, the free viewpoint data (expression format) includes a 3D model called View Independent (hereinafter, also referred to as VI model) and a 3D model called View Dependent (hereinafter, also referred to as VD model).

The VD model is 3D data in which information on a three-dimensional shape such as a 3D shape model and information on an image as a texture are separated. In the VD model, the 3D shape model is colored by mapping an image that becomes a texture (texture mapping). According to the VD model, it is possible to express the degree of reflection on the surface of an object that differs depending on the (virtual) viewpoint. The VD model is called View Dependent because its appearance depends on the viewpoint.

The VI model is, for example, 3D data in which polygons and points as components of information on a three-dimensional shape have color information. As a VI model, for example, there is a set of a colored point cloud, a 3D shape model, and a UV map as color information of the 3D shape model. According to the VI model, the colors of polygons and points are observed from any (virtual) viewpoint. The VI model is called View Independent because its appearance does not depend on the viewpoint.

The storage unit 32 stores the content of the free viewpoint data from the free viewpoint data generation unit 31.

The communication unit 33 communicates with the reproduction device 23. That is, for example, the communication unit 33 transmits the content of the free viewpoint data stored in the storage unit 32 to the reproduction device 23 in response to a request from the reproduction device 23.

The free viewpoint data generation unit 31 can be provided in the reproduction device 23.

<Configuration example of playback device 23>

FIG. 5 is a block diagram showing a configuration example of the reproduction device 23 of FIG.

The reproduction device 23 includes a communication unit 41, a storage unit 42, a free viewpoint image generation unit 43, a display unit 44, a virtual camera setting unit 45, an operation unit 46, and a locus information generation unit 47.

The communication unit 41 communicates with the content server 22. That is, the communication unit 41 transmits, for example, a request for the content of the free viewpoint data in response to the operation of the operation unit 46. Further, the communication unit 41 receives the content of the free viewpoint data transmitted from the content server 22 and supplies the content to the storage unit 42 in response to the request.

The storage unit 42 stores the content of the free viewpoint data from the communication unit 41.

The free viewpoint image generation unit 43 captures the free viewpoint data (three-dimensional space represented by the storage unit 42) stored in the storage unit 42 with a virtual camera (in a state) represented by camera information supplied from the virtual camera setting unit 45. 2D images (2D images obtained by shooting with a virtual camera) (including a set of 2D images for the left eye and 2D images for the right eye) are generated (rendered) as free viewpoint images (data). It is supplied to the display unit 44.

Further, the free viewpoint image generation unit 43 is a 2D image taken by a virtual camera represented by the camera information supplied from the virtual camera setting unit 45, with the free viewpoint data as the locus information supplied from the locus information generation unit 47. A free viewpoint image as a locus image is generated and supplied to the display unit 44.

The display unit 44 displays a free viewpoint image or the like from the free viewpoint image generation unit 43.

The display unit 44 can be configured by, for example, a 2D HMD / HUD, a 2D monitor, a 3D HMD / HUD, a 3D monitor, or the like. A 3D HMD / HUD or monitor is a display device that realizes stereoscopic vision by displaying, for example, a 2D image for the left eye and a 2D image for the right eye.

The virtual camera setting unit 45 captures the free viewpoint data (free viewpoint data stored in the storage unit 42 and free viewpoint data as locus information generated by the locus information generation unit 47). Is set, and the camera information representing the setting contents is supplied to the free viewpoint image generation unit 43.

For example, the virtual camera setting unit 45 sets the state of the virtual camera such as the shooting position, shooting posture, angle of view (focal length), and zoom magnification of the virtual camera according to the user's operation.

Here, when the display unit 44 is, for example, an HMD or HUD, the virtual camera setting unit 45 corresponds to the user's viewpoint and line of sight according to the user's viewpoint and line of sight detected by the HMD or HUD. As such, the shooting position and shooting posture of the virtual camera can be set.

Further, when the playback device 23 is a mobile terminal such as a smartphone or a tablet, the shooting position and shooting posture of the virtual camera correspond to the position and posture according to the position and posture of the mobile terminal. Can be set.

The operation unit 46 is operated by the user and supplies operation information according to the user's operation to a necessary block. The operation unit 46 can be integrally configured with the display unit 44 by, for example, a touch panel or the like. Further, the operation unit 46 can be configured as a mouse, a controller, or the like separate from the display unit 44. As the operation of the operation unit 46 of the user, for example, an operation of setting a virtual camera, a time seek operation, or the like can be performed.

The locus information generation unit 47 specifies the playback position of the free viewpoint data by the user, that is, the locus information representing the locus of a predetermined object reflected in the viewpoint image (and by extension, the free viewpoint image) in response to, for example, a time seek operation by the user. To generate. As the locus information, free viewpoint data (hereinafter, also referred to as line data) of a curve (including a straight line) representing the locus of an object or a strobe model can be adopted.

The line data as locus information includes, for example, a section including a playback position (hereinafter, also referred to as a designated playback position) designated by a time seek operation by the user, for example, a predetermined first time with reference to the designated playback position. It is possible to adopt the free viewpoint data of the curve representing the locus of a predetermined object (reflected in the frame) in the interval from the past time t1 by Δt1 to the future time t2 by the predetermined second time Δt2.

As the strobe model as the locus information, for example, a 3D model of a predetermined object in a section including a designated playback position, for example, a section from time t1 to time t2 as described above with reference to the designated playback position is arranged. It is possible to adopt a strobe model.

The locus information generation unit 47 generates locus information as free viewpoint data according to a time seek operation by the user, and supplies the locus information to the free viewpoint image generation unit 43.

When the locus information is supplied from the locus information generation unit 47, the free viewpoint image generation unit 43 photographs the line data and the strobe model as the locus information with a virtual camera represented by the camera information from the virtual camera setting unit 45. Generate a locus image as a free viewpoint image. Then, the free viewpoint image generation unit 43 supplies the locus image generated from the locus information to the display unit 44 in place of the free viewpoint image generated from the free viewpoint data stored in the storage unit 32, or , The locus image generated from the locus information is superimposed on the free viewpoint image generated from the free viewpoint data stored in the storage unit 32, and supplied to the display unit 44. As a result, the display unit 44 displays the locus image generated from the locus information while the user is performing the time seek operation.

When a strobe model is adopted as the locus information, the free viewpoint image generation unit 43 uses a 2D image taken by the virtual camera represented by the camera information from the virtual camera setting unit 45 for the strobe model as the locus information. A certain 3D strobe image is generated as a locus image and supplied to the display unit 44.

Here, the strobe image is an image showing one or more identical objects (images) taken at a plurality of times. A strobe image that shows an object reflected in a 2D image is also called a 2D strobe image. Also called a strobe image. When a strobe model is adopted as the locus information, the free viewpoint image generation unit 43 generates a 3D strobe image.

When a strobe model is adopted as the trajectory information, the trajectory information generation unit 47 uses the free viewpoint data from the free viewpoint data generation unit 31 to create a shooting space (3D model) which is a three-dimensional space reflected in the viewpoint image. Generate a strobe model in which a 3D model, which is free viewpoint data of the same object in multiple frames (different times) of the viewpoint image, is arranged.

That is, the locus information generation unit 47 sets a strobe section that is a section of a frame for which a strobe model is generated, and from the frame of the strobe section, a frame used for generating the strobe model (hereinafter, also referred to as a generation frame) is used. select. Further, the locus information generation unit 47 sets one or more of the objects reflected in the generation frame as the target object for generating the locus information (hereinafter, also referred to as a target object). Then, the locus information generation unit 47 generates a strobe model in which the 3D model of the target object reflected in the generation frame is arranged.

In the locus information generation unit 47, for example, all the moving objects reflected in the generation frame can be set as the target object.

Further, in the locus information generation unit 47, for example, an object specified by the user among the objects displayed in the generation frame can be set as the target object. The user can specify the target object (object to be) by the operation of the operation unit 46 or the like, and can also specify the target object by the line of sight.

Here, the reproduction device 23 may have at least a function of communicating with the outside, a function of displaying an image, and a function of accepting a user's operation. In FIG. 5, other functions, for example, the functions realized by the free viewpoint image generation unit 43, the virtual camera setting unit 45, and the trajectory information generation unit 47, are, for example, in the content server 22 and the server on the cloud. Can be provided.

<Processing of content server 22>

FIG. 6 is a flowchart illustrating an example of processing performed by the content server 22 of FIG.

In step S11, the content server 22 obtains viewpoint images (and depth images) of a plurality of viewpoints shot by the shooting device 21 in frame units. The content server 22 supplies viewpoint images of a plurality of viewpoints to the free viewpoint data generation unit 31, and the process proceeds from step S11 to step S12. The viewpoint images of the plurality of viewpoints taken by the photographing device 21 can be transmitted from the reproduction device 23 to the content server 22 via the reproduction device 23.

In step S12, the free viewpoint data generation unit 31 generates free viewpoint data in frame units using viewpoint images of a plurality of viewpoints and supplies the data to the storage unit 32. The storage unit 32 stores the content (content of the free viewpoint data) composed of the free viewpoint data in frame units from the free viewpoint data generation unit 31. After that, the process proceeds from step S12 to step S13.

In step S13, for example, the communication unit 33 transmits the content of the free viewpoint data stored in the storage unit 32 to the reproduction device 23 in response to a request from the reproduction device 23, and the process ends.

<Processing of playback device 23>

FIG. 7 is a flowchart illustrating an example of a free viewpoint data reproduction process performed by the reproduction device 23 of FIG.

In step S31, for example, the communication unit 41 requests the content of the free viewpoint data from the content server 22 in response to the user's operation, and the process proceeds to step S32.

In step S32, the communication unit 41 waits for the content of the free viewpoint data to be transmitted from the content server 22, and then receives the content of the free viewpoint data. The communication unit 41 supplies and stores the content of the free viewpoint data from the content server 22 to the storage unit 42, and the process proceeds from step S32 to step S33.

In step S33, the reproduction device 23 reproduces the content of the free viewpoint data stored in the storage unit 42, and the process proceeds to step S34.

That is, in the playback device 23, the virtual camera setting unit 45 sets the virtual camera (state), and the free viewpoint image generation unit 43 shoots the free viewpoint data with the virtual camera set by the virtual camera setting unit 45. Generated by rendering a free viewpoint image. Then, the free viewpoint image generation unit 43 supplies the free viewpoint image to the display unit 44 and displays it.

In step S34, the locus information generation unit 47 determines whether or not the time seek operation by the user has been started.

If it is determined in step S34 that the time seek operation has not been started, the process returns to step S33, and reproduction of the content of the free viewpoint data stored in the storage unit 42 is continued.

If it is determined in step S34 that the time seek operation has started, the process proceeds to step S35.

In step S35, the locus information generation unit 47 generates locus information based on the designated reproduction position designated by the time seek operation, supplies the locus information to the free viewpoint image generation unit 43, and the process proceeds to step S36.

That is, the locus information generation unit 47 is a predetermined object in a section from the past time t1 by a predetermined first time Δt1 to the future time t2 by a predetermined second time Δt2 with reference to the designated reproduction position. Generates locus information (free viewpoint data) that represents the locus.

In step S36, the free viewpoint image generation unit 43 generates the locus information taken by the virtual camera set by the virtual camera setting unit 45, and supplies the locus information from the locus information generation unit 47 to the display unit 44. It is displayed, and the process proceeds to step S37.

In step S37, the locus information generation unit 47 determines whether or not the time seek operation by the user has been completed.

If it is determined in step S37 that the time seek operation has not been completed, the process returns to step S35, and the same process is repeated thereafter.

If it is determined in step S37 that the time seek operation has been completed, the process returns to step S33 and is stored in the storage unit 42 from the designated playback position designated when the time seek operation is completed. The content of the free viewpoint data is played.

The content of the free viewpoint data stored in the storage unit 42 is designated by the time seek operation while the time seek operation is being performed, that is, while the processes of steps S35 to S37 are being performed. Free viewpoint data at the specified playback position can be played back.

FIG. 8 is a flowchart illustrating the details of the process of step S35 for generating the locus information when the strobe model is adopted as the locus information.

In step S51, the locus information generation unit 47 sets the strobe section, which is the section of the target frame for generating the strobe model, and the process proceeds to step S52.

For example, the locus information generation unit 47 uses the designated playback position designated by the time seek operation as a reference from the past time t1 by the predetermined first time Δt1 to the future time t2 by the predetermined second time Δt2. Set the section of to the strobe section. The first time Δt1 and the second time Δt2 may be the same time or different times.

In step S52, the locus information generation unit 47 sets a target object for arranging the 3D model on the strobe model from the object reflected in the viewpoint image, and the process proceeds to step S53.

For example, all the objects reflected in the frame of the strobe section, the objects specified according to the user's operation and the line of sight, and the like are set as the target objects.

In step S53, the locus information generation unit 47 selects a generation frame to be used for generating the strobe model from the frames of the strobe section, and the process proceeds to step S54.

Here, if all the frames of the strobe section are used as the generation frames and used to generate the strobe model, the same number of 3D models of the same object as the number of frames of the strobe section are overlapped and arranged in the strobe model, and 3D. The strobe image may be difficult to see.

Therefore, the locus information generation unit 47 can select some frames as generation frames from the frames of the strobe section, and generate a strobe model using the generation frames (3D model of the object reflected in).

The locus information generation unit 47 can select, for example, a frame in which the interference degree of the 3D model is equal to or less than the threshold value from the frames in the strobe section as the generation frame. That is, the locus information generation unit 47 calculates the degree of interference indicating the degree of overlap between the 3D models in a state where the 3D model of the target object reflected in the frame of the strobe section is arranged in the three-dimensional space. The degree of interference is calculated, for example, in a three-dimensional space, where 100% is the case where 3D models of arbitrary two frames completely overlap, and 0% is the case where they do not overlap at all. Then, the locus information generation unit 47 selects a frame whose interference degree is equal to or less than a predetermined threshold value as a generation frame. As described above, from the frames of the strobe section, select the frame in which the interference degree of the 3D model is equal to or less than the threshold value as the generation frame, and generate the strobe model in which the 3D model of the target object reflected in the generation frame is arranged. As a result, in the strobe model, the 3D models are arranged so as to overlap each other, and it is possible to prevent the 3D strobe image from becoming an image that is difficult to see.

In addition, in the selection of the generation frame, for example, the frame of the strobe section can be simply selected as the generation frame for each predetermined number of frames. Further, in the selection of the generated frame, all the frames in the strobe section can be selected as the generated frame.

In step S54, the locus information generation unit 47 uses the free viewpoint data stored in the storage unit 42, and the 3D model of the target object reflected in the plurality of generation frames selected from the frames of the strobe section is photographed by the target object. Generates a strobe model placed in the background (3D model) as the shooting space when it is shot. Then, the process of generating the strobe model as the locus information is completed.

<Generation of 3D strobe image>

FIG. 9 is a diagram showing an example of an unnatural 3D strobe image.

FIG. 9 shows a 3D strobe image generated from a strobe model generated by using 5 frames of the frame of the viewpoint image obtained by rolling the ball as an object from the front side to the back side as the generation frame. An example is shown.

In FIG. 9, the 3D model of the ball reflected in the generated frame of 5 frames is arranged (rendered) so as to give priority to the later 3D model in time. Therefore, the 3D model (of the ball) after the time is arranged so as to hide the 3D model on the front side in time even though it is located on the back side. As a result, the 3D strobe image of FIG. 9 is an unnatural image.

FIG. 10 is a diagram showing an example of a natural 3D strobe image.

FIG. 10 shows a 3D strobe image generated from a strobe model generated by using 5 frames of the viewpoint image frames obtained by rolling a ball as an object from the front side to the back side as a generation frame. An example is shown.

In FIG. 10, the 3D model of the ball reflected in the generated frame of 5 frames is arranged so as to give priority to the 3D model on the front side. Therefore, the 3D model on the front side is arranged so as to hide the 3D model on the back side, that is, the 3D model on the front side is preferentially reflected. As a result, the free viewpoint image is a natural image.

The free viewpoint image generation unit 43 uses the depth (depth) of the 3D model of each object placed in the strobe model to generate a 3D strobe image in which the 3D model of the object on the front side is preferentially reflected as described above. (Shoot with a virtual camera).

FIG. 11 is a diagram showing an example of a frame of a viewpoint image of a strobe section.

In FIG. 11, nine frames at times t1 to t9 are frames for the viewpoint image of the strobe section. The frame at time t1 to t9 shows the ball as an object rolling from left to right. In FIG. 11, a frame of a viewpoint image of a certain viewpoint is illustrated in order to avoid complicating the figure.

FIG. 12 is a diagram showing an example of generation of a strobe model using a frame at time t1 to t9 as a strobe section.

In FIG. 12, of the frames at time t1 to t9 as the strobe section, the frames at time t1, t3, t5, t7, and t9 are selected as the generation frames, and the frames are used as the generation frames for the viewpoint images of a plurality of viewpoints. A strobe model is generated in which a 3D model of the ball as an object reflected in the frames at times t1, t3, t5, t7, and t9 is placed.

FIG. 13 is a diagram showing an example of displaying a 3D strobe image generated by shooting a strobe model with a virtual camera.

As a 3D strobe image, a frame in which a 3D model of a ball as an object reflected in a frame at time t1, t3, t5, t7, and t9 can be generated and displayed. As the 3D strobe image, from the strobe model of FIG. 12, a frame in which the 3D model of the ball as an object reflected in the frame at time t1 is displayed, and a frame in which the 3D model of the ball as an object reflected in the frames at time t1 and t3 is displayed. , A frame showing a 3D model of the ball as an object in frames t1, t3, and t5, a frame showing a 3D model of the ball as an object in frames t1, t3, t5, and t7, and , Times t1, t3, t5, t7, and t9 can generate frames that show the 3D model of the ball as an object and display them in sequence.

In the 3D strobe image of FIG. 13, the shooting position and the like of the virtual camera that shoots the strobe model are not changed, but the shooting position and the like of the virtual camera can be changed. For example, a strobe model in which a 3D model of a ball as an object reflected in a frame at time t1, t3, t5, t7, and t9 is arranged can be shot with a virtual camera while changing the shooting position. When the shooting position is changed, the viewpoint for viewing the strobe model is changed, and a 3D strobe image with a changing camera angle is displayed.

<Effect processing>

FIG. 14 is a diagram illustrating effect processing performed on a 3D model arranged in the strobe model.

The locus information generation unit 47 can perform effect processing on the 3D model arranged in the strobe model. The effect processing is a standard 3D model that performs effect processing on the 3D model of the latest or oldest time among the 3D models placed in the strobe model, the 3D model specified according to the user's operation, etc. As a reference 3D model, it can be performed on one or both past and future 3D models from the reference 3D model.

Here, in order to make the explanation easy to understand, a strobe model is generated, and then the effect processing is performed on the 3D model arranged in the strobe model. However, the generation of the strobe model and the effect processing for the 3D model arranged in the strobe model can be performed in parallel or in an appropriate order. For example, after performing effect processing on a 3D model, it is possible to generate a strobe model in which the 3D model after the effect processing is placed.

FIG. 14 is a diagram illustrating a 3D model that is a target of effect processing in a strobe model.

In the strobe model, the effect processing is performed on one or both of the reference 3D model and the past and future from the reference 3D model among the 3D models of multiple generated frames as multiple times selected from the frames of the strobe section. Can be done on 3D models of.

The target model, which is the target 3D model to be subjected to the effect processing, is specified by the effect direction indicating the time direction (past direction and future direction) with respect to the reference 3D model and the effect distance indicating the degree of distance from the reference 3D model.

As the effect direction, past direction past, future direction future, or both past direction past and future direction future can be set.

When the past direction past is set as the effect direction, the effect processing is performed from the reference 3D model to the past direction 3D model. When the future direction future is set as the effect direction, the effect processing is performed from the reference 3D model to the future direction 3D model. When the past direction past and the future direction future are set as the effect direction, the effect processing is performed from the reference 3D model to the past direction 3D model and the future direction 3D model.

The effect distance can be specified by the number of models of the 3D model from the reference 3D model, the distance distance, or the time time.

According to the model number number, the 3D model placed in the strobe model, that is, the 3D model (of the object) reflected in the generation frame used to generate the strobe model, is separated from the reference 3D model by the model number number or more. The 3D model can be specified as the target model.

According to the distance distance, among the 3D models placed in the strobe model, the 3D model separated from the reference 3D model by the distance distance or more can be specified as the target model.

According to the time time, among the 3D models arranged in the strobe model, the 3D model separated from the reference 3D model by the time time or more can be specified as the target model.

In the strobe model, the effect processing targets the 3D model that is separated from the reference 3D model by the number of models, the distance distance, or the time time or more, in the past direction, the future direction, or in both the past and future directions. It can be carried out.

<Specific example of effect processing>

FIG. 15 is a diagram illustrating a specific example of effect processing.

In FIG. 15, as the effect processing, there is an effect processing represented by the effect modes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. For effect modes 1 to 14, the effect direction and effect distance described with reference to FIG. 14 can be set.

If the effect direction is not set, the effect process can be performed by default, for example, the past direction past is set as the effect direction.

As described with reference to FIG. 14, the effect distance is specified by the number of models of the 3D model from the reference 3D model, the distance distance, or the time time. For example, if the effect direction is set to the past direction past and the effect distance is set to the number of models number = 1, the target is a 3D model that is separated from the reference 3D model by the number of models number = 1 or more in the past direction. As a model, the effect processing represented by the effect mode is performed.

Effect mode 0 indicates that no effect processing is performed.

Effect mode 1 represents an effect process that makes the 3D model transparent. In the effect processing of effect mode 1, the target model can be made transparent with the same transparency, or gradually made transparent, that is, a 3D model (target) that is temporally or distant from the reference 3D model. The more transparent the model), the more transparent it can be. How to make the 3D model transparent can be specified, for example, by defining the parameters associated with effect mode 1 and using those parameters. When the transparency is set to 100%, the target model becomes completely transparent, and in this case, the result of the effect processing in the effect mode 1 is substantially the same as the effect mode 4 described later.

Effect mode 2 represents an effect process that gradually eliminates the 3D model.

Effect mode 3 represents an effect process that reduces the number of textures (the number of 2D images used as textures) in the 3D model. In the effect processing of effect mode 3, the number of textures of the target model can be reduced to the same number, or gradually reduced, that is, the texture of the 3D model that is farther from the reference 3D model in time or distance. You can also reduce the number. How to reduce the number of textures in the 3D model can be specified, for example, by defining the parameters associated with effect mode 3 and using those parameters.

Note that the effect processing in effect mode 3 targets the 3D model to which texture mapping is performed, that is, the VD model, and does not target the VI model to which texture mapping is not performed.

Effect mode 4 represents an effect process that erases a 3D model.

Effect mode 5 represents an effect process that reduces at least one of the brightness and saturation of the 3D model. In the effect processing of effect mode 5, the brightness and saturation of the target model can all be reduced by the same percentage, or gradually reduced, that is, 3D that is temporally or distant from the reference 3D model. The model can also increase the rate of decreasing brightness and saturation. How to reduce the brightness and saturation of the 3D model and whether to reduce the brightness or saturation can be specified by, for example, defining a parameter associated with the effect mode 5.

Effect mode 6 represents an effect process that limits the number of 3D models placed on the strobe model. In the effect processing of effect mode 6, the 3D model placed on the strobe model is limited to the 3D model other than the target model among the 3D models of the generated frame.

The effect mode 7 represents an effect process that makes the 3D model low polygon, that is, an effect process that reduces the number of meshes (the number of polygons) of the 3D model. In the effect processing of effect mode 7, the number of meshes of the target model can be reduced to the same number, or gradually reduced, that is, the mesh of the 3D model that is temporally or distant from the reference 3D model. You can also reduce the number. How to reduce the number of meshes in the 3D model can be specified, for example, by defining the parameters associated with effect mode 7.

Note that the effect processing in effect mode 7 targets a 3D model composed of polygons, and does not target a 3D model not composed of polygons, that is, for example, a wire frame.

Effect modes 8 and 9 represent effect processing that changes the representation format of the 3D model.

That is, the effect mode 8 represents an effect process of changing a 3D model composed of polygons into a wire frame.

The effect mode 9 represents an effect process that changes the representation format of the 3D model from View Dependent to View Independent, that is, an effect process that changes the VD model to a VI model (for example, a point cloud).

Effect mode 10 represents an effect process that erases the 3D model and leaves evidence of the 3D model.

Effect mode 11 represents an effect process that changes the texture (texture material) of the 3D model. What kind of texture to change the texture of the 3D model can be specified by, for example, defining a parameter attached to the effect mode 11.

The effect mode 12 represents an effect process that blurs the (shape) of the 3D model. The degree of 3D model blurring can be specified, for example, by defining a parameter associated with the effect mode 12 and using that parameter.

Effect mode 13 represents an effect process that changes the color of the 3D model. What kind of color to change the color of the 3D model can be specified by, for example, defining a parameter attached to the effect mode 13.

Effect mode 14 represents an effect process that resizes the 3D model. The extent to which the 3D model is resized can be specified, for example, by defining the parameters associated with effect mode 14.

For effect modes 1 to 14, the effect direction and effect distance can be set, but if necessary, the default effect direction and effect distance can be defined.

For example, the past direction past can be defined as the default effect direction for effect modes 1 to 14.

Further, for example, as the default effect distance of the effect mode 1, the number of models number = 1 can be defined.

In this case, if the effect direction and effect distance of effect mode 1 are not set, the 3D model that is one or more models away from the reference 3D model in the past direction, that is, the 3D before the next 3D model in the past direction of the reference 3D model. The effect processing of effect mode 1 is performed with the model as the target model.

Further, for example, the distance distance = 5 [m] can be defined as the default effect distance of the effect mode 4.

In this case, if the effect direction and effect distance of effect mode 4 are not set, the effect processing of effect mode 4 is performed on the target model of the 3D model that is 5 m or more away from the reference 3D model in the past direction.

Further, for example, the time time = 10 [sec] can be defined as the default effect distance of the effect mode 5.

In this case, if the effect direction and effect distance of effect mode 5 are not set, the effect processing of effect mode 5 is performed on the target model of the 3D model that is 10 seconds or more away from the reference 3D model in the past direction.

Further, for example, as the default effect distance of the effect mode 7, the number of models number = 3 can be defined.

In this case, if the effect direction and effect distance of effect mode 7 are not set, the 3D model that is 3 or more models away from the reference 3D model in the past direction, that is, before the third 3D model in the past direction of the reference 3D model. The effect processing of effect mode 7 is performed with the 3D model as the target model.

For effect processing, a plurality of effect modes can be set. For example, when effect modes 1 and 3 are set for effect processing, the effect processing that makes the 3D model transparent and reduces the number of textures is performed.

<Trajectory information>

FIG. 16 is a diagram illustrating an example of generation of a strobe model when a time seek operation is performed when a strobe model is adopted as locus information.

In the locus information generation unit 47, a strobe model in which a 3D model of the target object reflected in the section (frame) with respect to the designated reproduction position is arranged is generated as locus information according to the time seek operation.

In FIG. 16, the 3D model of the target object reflected in (the frame of) the designated playback position, the 3D model of the target object reflected in one frame in the future from the designated playback position, and the target object reflected in the past two frames from the designated playback position. In total of the 3D models, a strobe model in which the 3D models of the target objects reflected at four times (frames) are arranged is generated.

For example, when the user performs a time seek operation for operating the seek bar 12 realized by the operation unit 46, the frame of the 3D model arranged in the strobe model is changed according to the change in the designated playback position specified by the time seek operation. Section (strobe section) also changes.

FIG. 17 is a diagram illustrating an example of a change in the state of a virtual camera when a strobe model is generated and a 3D strobe image is displayed when a time seek operation is performed.

In the playback device 23, a 3D strobe image obtained by a virtual camera of a strobe model in which a 3D model of the target object displayed in a section (frame) with respect to the designated playback position is arranged is displayed according to the time seek operation.

By viewing the 3D strobe image, the user can easily guess the destination of the target object at the time from the designated playback position to the future direction or the past direction. Therefore, the user can see the destination of the target object according to the estimation result of the destination, that is, the state of the virtual camera as a virtual viewpoint so that the target object is within the shooting range of the virtual camera. Can be changed. As a result, it is possible to prevent the target object from being lost during the time seek operation.

FIG. 18 is a diagram illustrating an example of generation of line data when a time seek operation is performed when line data is adopted as locus information.

In the reproduction device 23, line data representing the locus of the target object reflected in the section based on the designated reproduction position is generated according to the time seek operation, and the line image obtained by capturing the line data with the virtual camera is displayed. By looking at the line image, the user can easily guess the movement destination of the target object at the time from the designated playback position to the future direction or the past direction. Therefore, the user can see the destination of the target object according to the estimation result of the destination, that is, the state of the virtual camera as a virtual viewpoint so that the target object is within the shooting range of the virtual camera. Can be changed. As a result, it is possible to prevent the target object from being lost during the time seek operation.

As the locus information, in addition to the strobe model and line data, arbitrary information that can display the locus of movement of the target object in an easy-to-understand manner can be adopted.

In the case shown in FIGS. 17 and 18, the reproduction device 23 generates locus information according to the time seek operation and displays a locus image (3D strobe image or line image) corresponding to the locus information. However, the setting of the virtual camera is done regardless of the time seek operation.

Therefore, the user guesses the movement destination of the target object by looking at the locus image, and the virtual camera setting unit 45 sets the virtual camera so that the target object falls within the shooting range of the virtual camera according to the estimation result. Need to be done.

Therefore, the generation of the locus information and the display of the locus image corresponding to the locus information shown in FIGS. 17 and 18 detect the user's viewpoint and line of sight (including the movement and posture of the head and body). HMD / HUD that sets the shooting position and shooting posture of the virtual camera so as to correspond to the user's viewpoint and line of sight, and AR that changes the appearance of objects displayed on the smartphone according to the position and posture of the smartphone etc. It is particularly suitable for applications such as applications in which the shooting position and shooting posture of a virtual camera are set in conjunction with an operation that changes the user's viewpoint or line of sight. According to such an application, for example, a user looks at a locus image, guesses a destination of a target object, and moves a line of sight to the destination to prevent the user from losing sight of the target object during a time seek operation. be able to.

FIG. 19 is a diagram illustrating an example of effect processing performed on a 3D model arranged in a strobe model as locus information.

Effect processing can be performed on the 3D model placed on the strobe model. For example, as shown in A of FIG. 19, it is possible to perform an effect process in which the transparency of the most future 3D model in the strobe section is the lowest, and the transparency is higher as the past 3D model is.

Also, depending on the direction of the time seek, that is, whether the time seek operation is an operation such as fast-forwarding or rewinding, the effect processing performed on the 3D model arranged in the strobe model is performed. You can control the degree (the degree to which the effect is applied).

For example, a 3D model in the direction of time seek can be subjected to weak effect processing, and a 3D model in the opposite direction can be subjected to strong effect processing.

Specifically, for example, when the direction of the time seek is the future direction (when the time seek operation is an operation such as fast-forwarding), the direction of the time seek is toward the past direction opposite to the future direction. , If the time seek direction is the past direction (when the time seek operation is an operation that rewinds), the future in the opposite direction of the past direction, which is the direction of the time seek, with a higher degree of effect processing. A higher degree of effect processing can be performed in the direction.

In this case, for example, when the effect process is a process of applying an effect that makes the 3D model transparent, when a time seek operation in the future direction is performed, as shown in A of FIG. 19, the most future of the strobe section. The effect processing is performed to make the transparency of the 3D model the lowest and the transparency of the 3D model in the past direction higher. On the other hand, when the time seek operation in the past direction is performed, as shown in B of FIG. 19, the effect of making the transparency of the oldest 3D model in the strobe section the lowest and the transparency of the 3D model in the future direction higher. Processing is done.

In addition, for example, it is possible to perform an effect process in which the transparency of the 3D model at the designated playback position is set to the lowest level, and the transparency of the 3D model that appears in the frame farther from the designated playback position in the past and future directions is increased.

As the effect processing performed on the 3D model arranged in the strobe model as the trajectory information, in addition to the effect processing for making the 3D model transparent, various effect processing described with reference to FIG. 15 can be adopted.

As described above, by performing effect processing on the 3D model arranged in the strobe model as locus information, the user can easily recognize the time direction (future direction, past direction) in the strobe model (3D strobe image). be able to.

As the strobe model as the locus information, for example, it is possible to generate a strobe model in which the 3D model of the target object becomes sparser as the distance from the designated reproduction position increases. In this case, the user can recognize the degree of distance from the designated playback position due to the density of the target objects reflected in the 3D strobe image.

<Processing of playback device 23>

FIG. 20 is a flowchart illustrating another example of the free viewpoint data reproduction process performed by the reproduction device 23 of FIG.

In FIG. 20, in steps S71 to S75, the same processing as in steps S31 to S35 of FIG. 7 is performed.

Then, in step S75, similarly to step S35 of FIG. 7, the locus information generation unit 47 generates locus information based on the designated reproduction position designated by the time seek operation, and then the process proceeds to step S76. ..

In step S76, the virtual camera setting unit 45 sets the virtual camera so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range of the virtual camera, and the process proceeds to step S77.

In step S77, similarly to step S36 of FIG. 7, the free viewpoint image generation unit 43 generates the locus information taken from the locus information generation unit 47 by the virtual camera set by the virtual camera setting unit 45. Then, it is supplied to the display unit 44 to be displayed, and the process proceeds to step S78.

In step S78, the same processing as in step S37 of FIG. 7 is performed.

As described above, in FIG. 20, in step S76, the virtual camera is set so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range (angle of view) of the virtual camera. Therefore, the display content displayed on the display unit 44 in step S77 includes at least the position in the three-dimensional space of the 3D model of the target object at the designated reproduction position (time). As a result, it is possible to more strongly prevent the user from losing sight of the target object during the time seek operation.

That is, in the reproduction process of FIG. 7, the reproduction device 23 generates the trajectory information according to the time seek operation and displays the trajectory image corresponding to the trajectory information, but the setting of the virtual camera is the time seek operation. Regardless of, it is performed according to the user's operation, the user's viewpoint, and the like.

Therefore, the user guesses the movement destination of the target object by looking at the locus image, and the virtual camera setting unit 45 sets the virtual camera so that the target object falls within the shooting range of the virtual camera according to the estimation result. It is necessary to perform operations and change the viewpoint, etc. so that

Therefore, depending on the user's operation and how to change the viewpoint, the virtual camera may not be set so that the target object falls within the shooting range of the virtual camera, and the trajectory information may deviate from the shooting range of the virtual camera. ..

When the locus information deviates from the shooting range of the virtual camera, the locus image corresponding to the locus information is not displayed, so that the user may lose sight of the target object.

On the other hand, in the reproduction process of FIG. 20, the virtual camera is set so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range of the virtual camera. Therefore, at least a part of the trajectory information (the part of the position in the 3D space of the 3D model of the target object reflected in the frame of the specified reproduction position) always falls within the shooting range of the virtual camera, so that the trajectory corresponding to the trajectory information At least a part of the image is always displayed on the display unit 44. As a result, it is possible to prevent the user from losing sight of the target object during the time seek operation, as compared with the case of FIG. 7.

FIG. 21 is a diagram illustrating a first setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object displayed in the frame at the designated reproduction position is included in the shooting range of the virtual camera.

In FIG. 21, a strobe model is adopted as the locus information. Then, in FIG. 21, for example, the shooting position of the virtual camera is fixed to the shooting position when the time seek operation is started, and only the shooting posture (shooting direction) of the virtual camera is reflected in the frame at the designated playback position. The 3D model of the object is set to be included in the shooting range of the virtual camera.

FIG. 22 is a diagram illustrating a second setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object displayed in the frame at the designated reproduction position is included in the shooting range of the virtual camera.

In FIG. 22, line data is adopted as the locus information. Then, in FIG. 22, as in FIG. 21, the shooting position of the virtual camera is fixed to the shooting position when the time seek operation is started, and only the shooting posture of the virtual camera is reflected in the frame at the designated playback position. The 3D model of the object is set to be included in the shooting range of the virtual camera.

FIG. 23 is a diagram illustrating a third setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range of the virtual camera.

In FIG. 23, a strobe model is adopted as the locus information. Then, in FIG. 23, for example, the shooting posture of the virtual camera is fixed to the shooting posture when the time seek operation is started, and only the shooting position of the virtual camera is a 3D model of the target object reflected in the frame of the designated playback position. Is set to be included in the shooting range of the virtual camera. That is, in FIG. 23, the shooting posture of the virtual camera is fixed, and the shooting position of the virtual camera determines the positional relationship of the target object reflected in the frame at the designated playback position with the 3D model, for example, when the time seek operation is started. It is set to maintain the positional relationship of.

FIG. 24 is a diagram illustrating a fourth setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range of the virtual camera.

In FIG. 24, a strobe model is adopted as the locus information. Then, in FIG. 24, for example, the shooting posture of the virtual camera is fixed to the shooting posture when the time seek operation is started, and only the shooting position of the virtual camera is a 3D model of the target object reflected in the frame of the designated playback position. Is set to be included in the shooting range of the virtual camera. That is, in FIG. 24, the shooting posture of the virtual camera is fixed, and the shooting position of the virtual camera is such that the 3D model of the target object reflected in the frame that has become the designated playback position after the time seek operation is started is all of the virtual camera. It is set to be included in the shooting range. Therefore, in FIG. 24, the virtual camera is moving backward, that is, in a direction away from the 3D model of the target object.

FIG. 25 is a diagram illustrating a fifth setting example of the virtual camera when the virtual camera is set so that the 3D model of the target object reflected in the frame at the designated reproduction position is included in the shooting range of the virtual camera.

In FIG. 25, a strobe model is adopted as the locus information. Then, in FIG. 25, for example, the shooting position and shooting posture of the virtual camera are fixed to the shooting posture when the time seek operation is started, and only the angle of view (FOV (Field Of View)) of the virtual camera is specified. The 3D model of the target object reflected in the frame at the playback position is set to be included in the shooting range of the virtual camera. In FIG. 25, the angle of view of the virtual camera expands with the time seek operation.

In the case shown in FIGS. 21 to 25, the reproduction device 23 generates trajectory information in response to the time seek operation, displays the trajectory image corresponding to the trajectory information, and sets the frame at the designated reproduction position. The virtual camera is set so that the 3D model of the object to be reflected is included in the shooting range of the virtual camera.

Therefore, the user does not need to perform an operation or change the viewpoint or the like so that the virtual camera is set so that the target object falls within the shooting range of the virtual camera.

Therefore, setting the virtual camera shown in FIGS. 21 to 25 depends on the position and posture of a PC monitor, a smartphone, or the like on which the virtual camera is not set, depending on the viewpoint of the user or the like. It is especially suitable for applications where the appearance of objects displayed on smartphones does not change.

As a method for setting the virtual camera so that the 3D model of the target object displayed in the frame at the specified playback position is included in the shooting range of the virtual camera, the method described with reference to FIGS. 21 to 25 is used. In addition, a method in which any two or more of the methods described in FIGS. 21 (or 22), 23, 24, and 25 are combined can be adopted.

<Explanation of computer to which this technology is applied>

Next, the series of processes described above can be performed by hardware or software. When a series of processes is performed by software, a program constituting the software is installed on a general-purpose computer or the like.

FIG. 26 is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-mentioned series of processes is installed.

The program can be pre-recorded on the hard disk 905 or ROM 903 as a recording medium built in the computer.

Alternatively, the program can be stored (recorded) in the removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as so-called package software. Here, examples of the removable recording medium 911 include a flexible disc, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a magnetic disc, and a semiconductor memory.

In addition to installing the program on the computer from the removable recording medium 911 as described above, the program can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, for example, the program transfers from a download site to a computer wirelessly via an artificial satellite for digital satellite broadcasting, or transfers to a computer by wire via a network such as LAN (Local Area Network) or the Internet. be able to.

The computer has a built-in CPU (Central Processing Unit) 902, and the input / output interface 910 is connected to the CPU 902 via the bus 901.

When a command is input by the user by operating the input unit 907 or the like via the input / output interface 910, the CPU 902 executes a program stored in the ROM (Read Only Memory) 903 accordingly. .. Alternatively, the CPU 902 loads the program stored in the hard disk 905 into the RAM (Random Access Memory) 904 and executes it.

As a result, the CPU 902 performs the processing according to the above-mentioned flowchart or the processing performed according to the above-mentioned block diagram configuration. Then, the CPU 902 outputs the processing result from the output unit 906, transmits it from the communication unit 908, and records it on the hard disk 905, for example, via the input / output interface 910, if necessary.

The input unit 907 is composed of a keyboard, a mouse, a microphone, and the like. Further, the output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, or the like.

Here, in the present specification, the processes performed by the computer according to the program do not necessarily have to be performed in chronological order in the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or individually (for example, processing by parallel processing or processing by an object).

Further, the program may be processed by one computer (processor) or may be distributed processed by a plurality of computers. Further, the program may be transferred to a distant computer and executed.

Further, in the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, the present technology can be configured as cloud computing in which one function is shared by a plurality of devices via a network and jointly processed.

Further, each step described in the above-mentioned flowchart may be executed by one device or may be shared and executed by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

Further, the effects described in the present specification are merely exemplary and not limited, and other effects may be used.

The present technology can have the following configurations.

<1>
Trajectory information generation that generates locus information representing the locus of an object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of the free viewpoint data generated from a plurality of viewpoint images taken from a plurality of viewpoints. An image processing device equipped with a unit.
<2>
The image processing device according to <1>, wherein the locus information generation unit generates locus information representing the locus of the object in a section including a designated reproduction position which is a designated reproduction position.
<3>
The image processing apparatus according to <2>, wherein the section including the designated reproduction position is a section from a past time to a future time with the designated reproduction position as a reference.
<4>
The image processing device according to <2> or <3>, wherein the locus information is a strobe model in which a 3D model of the object displayed in a frame of a section including the designated reproduction position is arranged.
<5>
The image processing device according to <4>, wherein the locus information generation unit performs effect processing on the 3D model arranged in the strobe model.
<6>
The image processing device according to <2> or <3>, wherein the locus information is free viewpoint data of a curve representing the locus of the object reflected in a frame of a section including the designated reproduction position.
<7>
The image processing apparatus according to any one of <1> to <6>, further comprising a free viewpoint image generation unit for generating a locus image which is a free viewpoint image obtained by capturing the locus information with a virtual camera.
<8>
The image processing device according to <7>, further comprising a virtual camera setting unit for setting the virtual camera.
<9>
The image processing device according to <8>, wherein the virtual camera setting unit sets the virtual camera according to a user operation.
<10>
The image processing device according to <9>, wherein the virtual camera setting unit sets a position and a posture of the virtual camera according to a user operation.
<11>
The virtual camera setting unit sets the virtual camera so that the 3D model of the object reflected in the frame at the designated playback position, which is the designated playback position, is included in the shooting range of the virtual camera <8>. The image processing device described.
<12>
The image processing device according to <8>, wherein the virtual camera setting unit sets the virtual camera so that the position and posture correspond to the user's viewpoint and line of sight.
<13>
It includes generating trajectory information representing the trajectory of the object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of the free viewpoint data generated from the plurality of viewpoint images taken from a plurality of viewpoints. Image processing method.
<14>
Trajectory information generation that generates locus information representing the locus of an object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of free viewpoint data generated from a plurality of viewpoint images taken from a plurality of viewpoints. As a part, a program to make a computer work.

11 Display unit, 12 seek bar, 13 slider, 21 shooting device, 22 content server, 23 playback device, 31 free viewpoint data generation unit, 32 storage unit, 33, 41 communication unit, 42 storage unit, 43 free viewpoint image generation unit, 44 Display unit, 45 Virtual camera setting unit, 46 Operation unit, 47 Trajectory information generation unit, 901 bus, 902 CPU, 903 ROM, 904 RAM, 905 hard disk, 906 output unit, 907 input unit, 908 communication unit, 909 drive, 910 input / output interface, 911 removable recording medium

Claims (14)

Trajectory information generation that generates locus information representing the locus of an object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of the free viewpoint data generated from a plurality of viewpoint images taken from a plurality of viewpoints. An image processing device equipped with a unit. The image processing device according to claim 1, wherein the locus information generation unit generates locus information representing a locus of the object in a section including a designated reproduction position which is a designated reproduction position. The image processing apparatus according to claim 2, wherein the section including the designated reproduction position is a section from a past time to a future time with the designated reproduction position as a reference. The image processing apparatus according to claim 2, wherein the locus information is a strobe model in which a 3D model of the object displayed in a frame of a section including the designated reproduction position is arranged. The image processing apparatus according to claim 4, wherein the locus information generation unit performs effect processing on the 3D model arranged in the strobe model. The image processing apparatus according to claim 2, wherein the locus information is free viewpoint data of a curve representing the locus of the object reflected in a frame of a section including the designated reproduction position. The image processing apparatus according to claim 1, further comprising a free viewpoint image generation unit that generates a locus image which is a free viewpoint image obtained by capturing the locus information with a virtual camera. The image processing device according to claim 7, further comprising a virtual camera setting unit for setting the virtual camera. The image processing device according to claim 8, wherein the virtual camera setting unit sets the virtual camera according to a user operation. The image processing device according to claim 9, wherein the virtual camera setting unit sets a position and a posture of the virtual camera according to a user operation. The virtual camera setting unit sets the virtual camera so that the 3D model of the object reflected in the frame of the designated playback position, which is the designated playback position, is included in the shooting range of the virtual camera according to claim 8. The image processing device described. The image processing device according to claim 8, wherein the virtual camera setting unit sets the virtual camera so that the position and the posture correspond to the viewpoint and the line of sight of the user. It includes generating trajectory information representing the trajectory of the object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of the free viewpoint data generated from the plurality of viewpoint images taken from a plurality of viewpoints. Image processing method. Trajectory information generation that generates locus information representing the locus of an object reflected in the viewpoint image in the three-dimensional space according to the designation of the reproduction position of free viewpoint data generated from a plurality of viewpoint images taken from a plurality of viewpoints. As a part, a program to make a computer work.

Images