JP6544719B2 - Moving image generation system, moving image generation device, moving image generation method, and computer program - Google Patents

Description

  The present invention relates to moving image generation.

  As shown in Non-Patent Document 1, there is known a service that provides still images taken of various places around the world.

Google StreetView, https: // www. google. com / maps / views / streetview? gl = jp & hl = ja Hyperlapse, http: // labs. teenanlax. com / project / hyperlapse

  Here, in order to present a virtual sense of movement to the user, it is required to present the user with a moving image of a landscape image that changes according to the movement. In order to realize this, it is conceivable that a camera mounted on a car presents to the user a moving image taken during movement of the car. The user can be photographed without actually visiting the photographed location by presenting the user with a moving image captured in advance according to or in conjunction with the user's action such as driving operation or treadmill traveling. You can get a virtual sense of movement as if you are moving a place.

  In order to provide virtual sense of movement in various places, moving images taken in advance along all the paths connecting those places are required. However, since the amount of data of a moving image is larger than that of a still image, it is not always easy to prepare and provide a moving image captured in advance along all paths.

  As with the aforementioned StreetView, services already exist that provide still images taken of various locations around the world. In addition, Hyperlapse (see Non-Patent Document 2) proposes using still images of various places provided in StreetView as frame images of moving images. By continuously presenting still images at close points, a moving image can be obtained and motion sensation can be presented.

  In StreetView, still images of various places are taken at relatively wide shooting intervals, and it is said that the shooting intervals are approximately 11.5 m intervals. Even with still images shot at relatively wide shooting intervals, if they are switched at very high speed and displayed, it is possible to present a moving image in which the landscape changes smoothly. However, if the still images captured at a wide shooting interval are switched at high speed, only a very fast sense of movement can be obtained.

  On the other hand, when trying to provide a sense of movement at a low speed such as walking or running, it becomes a fragmented moving landscape. That is, one still image is displayed for several seconds when the still images shot at wide shooting intervals are switched at low speed and displayed according to the moving speed of the walking speed of a person or the like. For this reason, when the moving speed is low, the user is only presented with still images in fragments, and the quality of the moving sensation obtained by the user is degraded.

  Although it is possible to sufficiently narrow the shooting interval of the still image which is a frame constituting a moving image for presenting a natural sense of movement at low speed, if the shooting interval is narrowed to such an extent that a natural sense of movement can be obtained. The number of still images to be photographed increases, and the amount of data increases.

  Therefore, it is desirable to improve the presented sense of movement while suppressing an increase in the amount of data.

  According to an aspect of the present invention, there is provided a storage unit that stores a plurality of key frame images captured at a plurality of positions in association with the imaging positions of each of the plurality of key frame images; And a selection unit for selecting, from among the plurality of key frame images stored in the storage unit, the plurality of key frame images whose photographing position is in the vicinity of the user position; And an interpolation processing unit for obtaining an interpolation frame image at the user position from at least one of the plurality of selected key frame images by interpolation processing based on the distance between each of the plurality of selected key frame images and the photographing position. And a moving image generation system. The above key frame image is preferably an omnidirectional image.

  In the above aspect, the part excluding the storage unit in the moving image generation system can constitute a moving image generation apparatus. Another aspect of the present invention is a computer program for causing a computer to function as a moving image generation device. Another aspect of the present invention may be a computer program for causing a computer to function as a moving image generation system. Another aspect of the present invention is a recording medium recording the computer program.

  Still another aspect of the present invention acquires a virtually moving user position, selects, from among a plurality of key frame images, a plurality of the key frame images whose shooting positions are in the vicinity of the user position; Obtaining an interpolated frame image at the user position from at least one of the selected plurality of key frame images by interpolation processing based on a distance between the user position and the photographing position of each of the plurality of selected key frame images; It is a moving image generation method including.

  According to the present invention, it is possible to improve the presented sense of movement while suppressing an increase in the amount of data.

It is a block diagram of a moving image generation system. It is a flowchart which shows a moving image generation process. It is an explanatory view of interpolation frame picture generation by alpha blending. It is an example of setting of alpha value. It is an example of a key frame image database. It is an explanatory view showing other examples of interpolation frame picture generation. It is a flowchart which shows a moving image generation process. It is an explanatory view of interpolation frame picture generation by image modification. It is a figure which shows a two-dimensional movement space. It is an example of a key frame image database. It is a figure which shows three-dimensional movement space. It is an example of a key frame image database. It is a flowchart which shows an image acquisition procedure. It is an example of setting of the priority of key frame image acquisition.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

[1. Moving image generation system]
As shown in FIG. 1, the moving image generation system 10 includes a server 20 and a moving image generation device 30. The server 20 (storage unit; first storage device) is provided on a network such as the Internet, and includes a key frame image database 22 that stores key frame images constituting a moving image. In the present embodiment, the server 20 and the moving image generation device 30 are provided as separate devices, but both may be provided as an integral device. That is, the moving image generation system 10 may be configured by providing the key frame image database 20 in the moving image generation apparatus 30 shown in FIG. 1. In addition, the moving image generation system 10 may be configured by providing the function of the processing device 40 in the server 20 illustrated in FIG. 1. The server 20 may be provided with a part of the functions of the processing device 40 illustrated in FIG. 1.

  The key frame image is used to generate a spatial image (moving image) for realizing a virtual traveling environment. The key frame image of the present embodiment is a still image obtained by photographing each point in the real world with a camera. The still image may be a standard aspect ratio image captured by a normal camera or may be a panoramic image. As a panoramic image, an omnidirectional panoramic image (all-sky image) is preferable. In the following, the key frame image is described as an omnidirectional panoramic image.

  The key frame image database 22 stores a plurality of key frame images and shooting positions of the plurality of key frame images in association with each other. When a position (virtual user position) is designated as a search key by the moving image generation apparatus 30, the key frame image database 22 extracts a plurality of key frame images in the vicinity of the position where the photographing position is designated. The key frame image database 22 provides the moving image generation apparatus 30 with the plurality of extracted key frame images and their shooting positions.

  The photographing interval of the plurality of key frame images can be, for example, about 80 cm to 20 m. Preferably, the same subject is photographed at least in part in a plurality of key frame images at adjacent photographing positions. If only the key frame image is switched and displayed according to the walking or traveling speed of the person with the shooting interval of the above degree, it does not appear as a smoothly changing image, but the shooting interval is relatively wide. The amount of data accumulated in the server 20 can be suppressed by suppressing the increase in the number of Further, since the number of key frame images provided from the server 20 to the processing device 40 can be reduced, an increase in communication traffic between the server 20 and the processing device 40 can also be prevented.

  The image generation device 30 acquires a key frame image stored in the server 20, and generates an interpolated frame image according to the user position from the acquired key frame image. The image generation device 30 uses the key frame image and the interpolation frame image as frame images constituting a moving image. The moving image according to the present embodiment is an image in which the scenery presented to the user changes according to the movement of the virtual user, and the user can feel the movement feeling of moving the place without visiting the actual place. To present.

  If the virtual user position matches the shooting position of the key frame image, the key frame image is presented to the user. On the other hand, when the virtual user position is located between the shooting positions of each of the plurality of key frame images, the interpolation frame image is presented to the user. The interpolation frame image changes in accordance with the change in the positional relationship between the shooting positions of each of the plurality of key frame images and the current user position. When the user moves between the shooting positions of each of the plurality of key frame images, an interpolation frame image that changes according to the user position is presented to the user. Thereby, the user can obtain a natural sense of movement.

  The image generation device 30 is provided in an exercise apparatus such as, for example, a treadmill or a fitness bike. In this case, the image generation device 30 displays a moving image in which the landscape changes in accordance with the speed at which the user travels on the treadmill or the speed traveled by the fitness bike. As a result, it is possible to present a virtual sense of movement to the user in motion, and to impart fun to motion.

  The image generation device 30 includes a processing device 40, a display device 50, a speed detector 52, and a direction detector 54. When the image generation device 30 only generates a moving image and does not perform moving image display, the display device 50 may be omitted. Further, if unnecessary, at least one of the velocity detector 52 and the direction detector 54 may be omitted.

  The processing device 40 is configured by a computer having a processor 41 and a memory 42. The processor 41 performs processing such as user position acquisition processing 43, key frame image selection processing 44, key frame image acquisition processing 45, and interpolation processing 46. The memory 42 (storage unit; second storage device) stores the plurality of key frame image databases 22 acquired from the server 20 and the imaging devices thereof in association with each other.

  The user position acquisition process 43 is a process that causes the processor 41 to function as an acquisition unit for acquiring a virtually moving user position. The key frame image selection process 44 is a process that causes the processor 41 to function as a selection unit for selecting a key frame image whose shooting position is near the virtual user position. The key frame image acquisition process 45 is a process that causes the processor 41 to function as an image acquisition unit for acquiring the selected key frame image and the imaging position thereof from the server 20 and storing the image in the memory 42. The interpolation process 46 is a process that causes the processor 41 to function as an interpolation processing unit that obtains an interpolated frame image at a user position from at least one of the selected plurality of key frame images.

  The processor 41 executes the processes 43, 44, 45, and 46 by executing the computer program of the processing device 40, and acquires the processor 41 (computer) as the acquisition unit, the selection unit, and the image acquisition unit. And function as an interpolation processing unit.

  The display device 50 displays the generated moving image. The display device 50 may be, for example, a flat display, an immersive display that covers the entire periphery of the user by combining a plurality of displays, a dome-shaped display that covers the periphery of the user, and a head mounted display mounted on the head of the user. Composed of In the case of a display device 50 whose displayable range at one time is narrower than the entire range of the omnidirectional panoramic image as in a head mounted display, the orientation of the user's head or body is detected as a virtual movement direction. It may be. When the display device 50 is a head mounted display, an image to be displayed on the display device 50 is generated according to the direction and movement of the head detected by the head mounted display.

  The speed detector 52 detects the virtual movement speed of the user. When the motion image generating apparatus 30 is provided in an exercise apparatus that simulates a user's walking or running, such as a treadmill or a fitness bike, the exercise apparatus has a speed detection function. In this case, the speed detection function of the exercise apparatus can be used as the speed detector 52 of the present embodiment. If it is assumed that the moving speed of the user does not change and is constant or if the processing device 40 sets the moving speed, the speed detector 52 is unnecessary. Also, with regard to virtual movement of the user, it is not necessary for the user to actually carry out exercise such as walking or running, but merely the user's position and / or movement speed is manipulated by the operation device operated by the user. It may be In this case, the operation device operated by the user functions as the speed detector 52 of the present embodiment.

  The direction detector 54 detects the virtual movement direction of the user in the virtual space. The direction detector 54 can be configured, for example, as a controller that manually operates the moving direction of the user. In addition, the direction detector 54 may detect the direction of the user's head or body for movement direction selection, and may detect the direction of the user's head or body as a virtual movement direction. When the display device 50 is a head mounted display, an attitude sensor provided on the head mounted display can be used as the direction detector 54. In the case of exercise equipment that assumes one-dimensional movement, such as a treadmill or fitness bike, the direction detector 54 can be omitted.

[2. Moving image generation process]
[2.1 First example of moving image generation processing]
FIG. 2 shows a first example of the moving image generation process performed by the processor 41 of the processing device 40. In this first example, a one-dimensional space is assumed as a virtual movement space. In the first example, the moving direction of the user is not considered. As described later, the movement space may be a two-dimensional space or a three-dimensional space.

  The processing device 40 acquires the velocity v of the user and the user position Px (step S11). The user velocity v is given from, for example, the velocity detector 52. The processing device 40 performs a user position acquisition process 43 to acquire the user position Px from the user velocity v. Specifically, the processing device 40 calculates the amount of movement from the initial position based on the virtual initial position (current position) of the user at the start of the moving image generation processing and the acquired velocity v. Then, the current position (user position) Px after the virtual movement of the user is obtained. The user position Px may be set by the processing device 40 itself.

The processing device 40 executes key frame image selection processing 44 and key frame image acquisition processing 45 (step S12). That is, the processing device 40 performs the key frame image selection processing 44 to obtain a plurality of key frame images A, which should be acquired from the plurality of key frame images stored in the server 20 based on the velocity v and the user position Px. Select B. The plurality of key frame images to be acquired are a plurality of (two) key frame images A and B in which the shooting positions P _A and P _B are in the vicinity of the user position P x. As shown in FIG. 3, when the movement space is one-dimensional, for example, two key frame images A and B closest to the user position Px are selected. Selection of a plurality of keyframes image, the image A is selected, the shooting position P _A of _B, and an intermediate P _B, user position Px is performed so as to be located.

  In the present embodiment, even if the user position Px is the same, different key frame images are selected according to the virtual movement velocity v of the user. This point will be described later.

The processing device 40 performs the key frame image acquisition processing 45 to acquire the selected plurality of key frame images A and B and their photographing positions P _A and P _B from the server 20. The plurality of acquired key frame images A and B and their shooting positions P _A and P _B are stored in the memory 42. The processing device 40 performs the subsequent processing based on the plurality of key frame images A and B stored in the memory 42 and their shooting positions P _A and P _B.

The processing device 40 calculates distances d ₁ and d ₂ between the user position Px and the shooting positions P _A and P _{B of} the plurality of acquired key frame images A and B, respectively. The distance d ₁ (first distance d ₁ ) is the distance between the user position Px and the selected first key frame image A, and the distance d ₂ (second distance d ₂ ) is selected as the user position Px This is the distance to the second key frame image B. The distances d ₁ and d ₂ may be calculated by the server 20 and provided to the processing device 40.

The processing device 40 performs an interpolation process 46 based on the distances d ₁ and d ₂ to obtain an interpolated frame in which the user position Px becomes a virtual shooting position (viewpoint position) from the plurality of acquired key frame images A and B. Obtain an image X. In the first example, the interpolation frame image X is obtained by alpha blending (first processing) of a plurality of key frame images A and B. In step S13, the processing device 40 sets an α value (transparency) used for alpha blending based on the distances d ₁ and d ₂ . The α value may be set by the server 20 and provided to the processing device 40. Also, alpha blending may be performed by the server 20, and the interpolated frame image X obtained by alpha blending may be provided to the processing device 40.

In this embodiment, the alpha ₁ as alpha value to be multiplied to the first key frame image A, and alpha ₂ as alpha value to be multiplied to the second key frame image B, and is set. In addition, in this embodiment, since it is (alpha) ₂ = 1- (alpha) ₁ evidently from the below-mentioned formula, (alpha) ₂ is obtained if (alpha) ₁ is calculated.

Each of the α values (α ₁ , α ₂ ) to be multiplied to each of the images A and B takes a value of 0 to 1, and is transparent when 0 and opaque when 1. alpha _1, the more the distance d ₁ between the imaging position P _A of the user position Px and the first key frame image A is small, is the set to a large value (transparency is set low). alpha ₂ also, as the distance d ₂ between the imaging position P _B of the user position Px and the second key frame image B is small, is the set to a large value (transparency is set low).

In the present embodiment, the α value (α ₁ , α ₂ ) is expressed as a function of the distances d ₁ and d ₂ . α ₁ and α ₂ are calculated, for example, by the following equation.

FIG. 4A shows the values (vertical axis) of α ₁ and α ₂ set by the above equation. The horizontal axis is a virtual user position indicates a position range from a position P _A to the position P _B. In FIG. 4 (a), alpha ₁ is the maximum (1) at a position _{P A,} decreases toward the position _{P B,} the minimum (0) at the position _{P B.} alpha ₂ is the minimum (0) at the position _{P A,} increases toward the position _{P B,} the maximum (1) at a position _{P B.}

The interpolation frame image X for the user position Px is obtained by multiplying each of the plurality of key frame images A and B by an α value (α ₁ , α ₂ ) as in the following equation.

As shown in FIG. 3, a plurality of key frames image A different photographing positions, to B, and that is reflected is building in the vicinity of the position P _D, both the key frame image A, the shooting position P _A of _{B, and} P _B Because they are different, the sizes and positions of the buildings shown in both key frame images A and B are somewhat different. By superimposing such a plurality of key frame images A and B by alpha blending, it becomes an image in which a building (object) in the images A and B appears double.

In the case where a virtual user position Px is changed from the position _{P A} to the position _{P B,} if user position Px is positioned _{P A, α 1 = 1,} α 2 = 0 , and the interpolation frame image X first key Since it is equal to the frame image A, only the image of the building shown in the image A is presented to the user.

As shown in FIG. 4 (a), the user position Px is, when it is between positions P _B and the position P _A is a position closer to the position P _A, the transparency of the image A is relatively low image B Because the transparency is relatively high (α ₁ > α ₂ ), the building appears double in the state in which the image of the building shown in the image A is dark and the image of the building shown in the image B is thin.

User position Px is, when it is between positions P _B and the position P _A is a position closer to the position P _B, since the transparency of the image A is relatively high transparency of the image B is relatively low (alpha ₁ <α ₂ ), the image of the building shown in the image A is thin, and the image of the building shown in the image B is dark, and the building appears double.

If the user position Px is the position P _B , then α ₁ = 0 and α ₂ = 1, and the interpolation frame image X becomes equal to the second key frame image B. Only the image of is presented.

As shown in FIG. 4A, by continuously changing α ₁ and α ₂ according to the change of the user position Px, the interpolation frame image X also changes continuously according to the movement of the user. User is moving between the position P _B and the position P _A, interpolation frame images X which varies Ji movement of the user is displayed on the display device 50. Although the interpolation frame image X is an image in which the subject may appear doubled when viewed as a still image, the duplication state of the subject changes according to the movement of the user, so the eyes of the person are natural due to the movement. Pictured as a change in landscape. Therefore, it can be presented to the user as a smooth continuous moving image.

The manner of change of α ₁ and α ₂ according to the change of the user position Px is not limited to the example shown in FIG. As shown in FIG. 4A, α ₁ and α ₂ may not change linearly but may change in a curve as shown in FIG. 4A. As shown in FIG. 4 (c), the user position Px is, even away from the position _{P A,} if the position _{P A} near, [alpha] 1 = 1, while maintaining the [alpha] 2 = 0, the key frame image A is displayed the so that, similarly, the user position Px is, even away from the position P _B, if the position P _B vicinity, [alpha] 1 = 0, to maintain the [alpha] 2 = 1, as the key frame image B is displayed May be When image processing other than alpha blending (for example, the processing shown in the second example and / or the third example) is also applied to the key frame images A and B, α ₁ and α ₂ are set as shown in FIG. It may be changed as follows.

When an α value is set as shown in FIG. 4C, if the user position Px is near to the key frame images A and B, even if they are not the shooting positions P _A and P _B , the key frame image A and B can be displayed, and the interpolation frame image can be displayed after some distance from the imaging positions P _A and P _B. As a result, it is not necessary to generate the interpolation frame image X if the user position is only slightly away from the photographing positions P _A and P _B, and a time margin for generation of the interpolation frame image X can be secured.

FIG. 5 is selected by the key frame image selection process 44 when the virtual movement speed v of the user is low (eg, walking) and high (eg, running). Show key frame images. User position Px is in the case where the position shown in FIG. 3 (a position between the positions P _A and the position P _B), if slow, the key frame images A, B is selected, if fast, keyframes Images A and C are selected.

If the moving speed v is low, a more natural feeling of movement can be obtained if the interpolation frame image is generated from a plurality of key frame images captured at relatively narrow shooting intervals. On the other hand, when the moving speed v is high, a natural feeling of movement can be maintained even using a plurality of key frame images photographed at a relatively wide photographing interval. For example, assuming that the shooting interval (key frame interval) between adjacent shooting positions in FIG. 3 is 1, if the moving speed v is smaller than a predetermined threshold, the key frame intervals of a plurality of key frame images selected are 1 so that the position P _A, key taken with P _B frame images A, B are selected. In addition, when the moving speed is larger than a predetermined threshold, the key frame images A and C captured at the positions P _A and P _c are selected such that the key frame interval of the plurality of key frame images to be selected becomes two. Be done.

[2.2 Example of Moving Image Generation]
When generating the interpolation frame image X, a key frame for which image processing (second processing) according to the distances d ₁ and d ₂ is performed instead of alpha blending on the key frame images A and B themselves Alpha blending may be performed on an image (key frame image after correction). The image processing (second processing) corresponding to the distances d ₁ and d ₂ is the key frame image A, according to the positional relationship between the user position Px and the photographing positions P _A and P _B by image processing other than alpha blending. It is a process of emphasizing B or de-emphasizing (such as blurring).

For example, as shown in FIG. 6, the user position Px is, of the two imaging position _P A, _{P B,} close towards imaging position _{P A,} if far from the shooting location _{P B,} processor 40, key Enhancement processing is performed on the frame image A to generate a corrected key frame image A1 that is more strongly presented to the human eye, and de-emphasis processing is performed on the key frame image B to generate human eye To generate a corrected key frame image B1 which is weakly presented. FIG. 6 shows an after-correction key frame image A1 that presents a building that is a subject of human eyes more strongly and an after-correction key frame image B1 that presents a building weaker. By alpha blending the key frame images A1 and B1 after correction, in addition to the difference in transparency set to both images A1 and B1, the difference between the distances d ₁ and d ₂ is due to the enhancement / de-emphasis of the subject itself. The user can be presented with a good sense of movement according to the situation.

  The enhancement process and the de-emphasis process are performed by one or more processes selected from, for example, the brightness adjustment process of the image, the contrast adjustment, the resolution adjustment, the increase / decrease color process, and the like. Color increase / decrease processing is a generic term for color increase processing to increase the color in an image, such as turning a black and white image into a color image, and color reduction processing to reduce a color in an image, such as changing a color image to a black and white image. is there. The color enhancement process can enhance the image, and the color reduction process can de-emphasize the image. The emphasizing process can also be performed by edge emphasizing process, and the de-emphasis process can also be performed by blurring process.

  The corrected key frame image A1 shown in FIG. 6 is obtained by increasing the brightness, enhancing the contrast, and increasing the resolution with respect to the key frame image A, and the corrected key frame image B1 is The key frame image B is obtained by reducing the brightness, reducing the contrast, and reducing the resolution. The emphasizing process and the de-emphasis process need not both be performed, but only one may be performed.

  The second example is the same as the first example in that it is not particularly described.

[2.3 Example of Moving Image Generation]
The interpolation processing for generating the interpolation frame image X is not limited to alpha blending. The interpolation frame image X may be generated by performing image processing (image deformation) on one of the selected plurality of key frame images based on the distances d ₁ and d ₂ .

FIG. 7 shows moving image generation processing according to the third example. Steps S21 and S22 in FIG. 7 are the same as steps S11 and S12 in FIG. In step S23, the processing device 40 performs an image recognition process for recognizing an individual subject in the selected plurality of key frame images A and B. In step S24, the processing device 40 deforms the individual objects in the key frame image closest to the user position Px based on the distances d ₁ and d ₂ to generate an interpolated frame image Px.

As shown in FIG. 8, the buildings in the key frame image A appear relatively small, the buildings in the key frame image B appear relatively large, and the user position Px is close to the shooting position PA of the key frame image A. It shall be in position. In this case, the processing device 40 determines the position of the user from the size (and position) of the building in the key frame image A, the size (and position) of the building in the key frame image B, and the distances d ₁ and d _2. Determine the size (and position) of the building at Px. This process is performed on all objects recognized in the key frame image. Based on this, the processing device 40 obtains conversion parameters for converting the key frame image A captured at the shooting position P _A closer to the user position P x into a landscape at the user position P x, and based on the conversion parameters The key frame image A is subjected to image conversion to obtain an interpolated frame image X.

[2.4 Selection of keyframe image in 2D space]
Although the virtual moving space has been described as being a one-dimensional space in the above description, the virtual moving space may be two-dimensional as shown in FIG. In the key frame image database 22 of the server 20, key frame images _{A to H} photographed at points P _{A to} P _{H in} a two-dimensional space are stored in association with their photographing positions P _{A to} P _H It is done.

In FIG. 9, it is assumed that the user position Px is at a position surrounded by the positions P _A , P _B , P _C , and P _D. In FIG. 9, a shooting interval (key frame interval) between shooting positions adjacent in the vertical direction and the horizontal direction is set to 1. In the key frame image selection processing 44, for example, a plurality of key frame images P _A , P _B and P _C whose distance from the user position Px is 1 or less are selected. The number of key frame images selected may be four or more, or two in some cases.

When three key frame images A, B, C are selected, the α values (α ₁ , α ₂ , α ₃ ) by which the respective images A, B, C are multiplied are the images A, B from the user position Px. , C, which is a function of the distances d ₁ , d ₂ , d _{3 to} the shooting positions P _A , P _B , P _C. α ₁ , α ₂ and α ₃ are calculated, for example, by the following equation.

The interpolation frame image X for the user position Px is obtained by multiplying each of a plurality of key frame images A, B and C by an α value (α ₁ , α ₂ , α ₃ ) as in the following equation.

Even when the moving space is two-dimensional, when the moving speed v is smaller than a predetermined threshold, the positions P _A , P _B , P are set so that the key frame intervals of the plurality of key frame images to be selected become one. shot key frame image A with _C, B, _C are selected. In addition, when the moving speed is larger than a predetermined threshold, the key frame images captured at positions P _A , P _E, P _F , and P _G so that the key frame intervals of the plurality of key frame images to be selected become two. A, E, F, G are selected (see FIG. 10).

[2.5 Selection of Keyframe Image in 3D Space]
The virtual movement space may be three-dimensional as shown in FIG. In the key frame image database 22 of the server 20, key frame images _{A to H} captured at points P _{A to} P _{H in} a three-dimensional space are stored in association with their shooting positions P _{A to} P _H It is done.

In FIG. 11, it is assumed that the user position Px is at a position surrounded by the positions P _{A to} P _H. In FIG. 13, an imaging interval (key frame interval) between imaging positions adjacent in each of three-dimensional directions is set to 1. In the key frame image selection processing 44, for example, a plurality of key frame images P _A , P _B , P _C , and P _F whose distances from the user position Px are one or less are selected. The number of key frame images selected may be five or more, or two or three in some cases.

When three key frame images A, B, C, F are selected, the α values (α ₁ , α ₂ , α ₃ , α ₄ ) by which the respective images A, B, C, F are multiplied are the user position It is expressed as a function of distances d ₁ , d ₂ , d ₃ and α ₄ from Px to shooting positions P _A , P _B , P _C and P _F of the respective images A, B, C _and F. α ₁ , α ₂ , α ₃ and α ₄ are calculated, for example, by the following equation.

The interpolation frame image X for the user position Px multiplies each of a plurality of key frame images A, B, C, and F by an α value (α ₁ , α ₂ , α ₃ , α ₄ ) as in the following equation It is obtained by

  Even in the case where the movement space is three-dimensional, the key frame images to be selected can be made different according to the movement speed v.

[3. Image acquisition process from server]
The processing device 40 needs to generate the interpolation frame image X at high speed according to the movement of the user. For this reason, the processing device 40 does not acquire from the server 20 a key frame image used to generate the interpolation frame image X after the user position Px is obtained, but based on the current user position Px, etc. Alternatively, key frame images to be obtained may be acquired in advance.

FIG. 13 shows a processing procedure for proactive acquisition of key frame images. When acquiring the user's initial position P ₀ with the start of the moving image generation processing (step S 31), the processing device 31 starts acquisition of a plurality of key frame images whose positions around the position P ₀ are shooting positions (step S 31) S32).

  Thereafter, when the processing device 40 acquires the moving speed of the moving user, the user position Px, and the moving direction (step S33), the priority for key frame image acquisition is obtained based on the moving speed, the user position Px, and the moving direction. Are set (step S34). The processing device 40 acquires, from the server 20, a plurality of key frame images whose shooting positions are around the current user position Px in the order according to the set priority.

FIG. 14 shows an example of setting the priority. 14, the user position Px is between the position _{P C} and the position _{P B,} close better position PB. The moving direction of the user is from the left to the right in the figure. In this case, the key frame image processing apparatus 40 is to be acquired by most wired from the server 20 is a key frame image B closest photographing position P _B to the user position Px. When the moving speed of the user is low, the priority decreases as the user moves away from the user position Px in the moving direction of the user. Priority keyframe image A photographing position P _A is the user moving direction in the opposite direction as viewed from the closest photographing position P _B to the user position Px is relatively low.

When the moving speed of the user is high, the priority decreases as the user moves away from the user position Px in the moving direction of the user, but the shooting positions P _D and P where the key frame interval is a multiple of 2 as viewed from the shooting position PB _F keyframe image D, the priority of F is the other key frame image C, E, relatively higher than G.

  When determining the priority, at least one of the moving speed and the moving direction may be omitted.

[4. Application example of moving image generation system]
The moving image generation system 10 is used together with exercise equipment such as a treadmill, and can also be used for location guidance and the like.

[5. Appendix]
The present invention is not limited to the above embodiment, and various modifications are possible.

10 Moving image generation system 20 server (storage unit; first storage device)
30 moving image generation device 40 processing device 41 processor 42 memory (storage unit; second storage device)
43 User position acquisition process (acquisition unit)
44 Key frame image selection processing (selection section)
45 Key frame image acquisition processing (image acquisition unit)
46 Interpolation processing (Interpolation processing unit)
50 display device 52 speed detector 54 direction detector X interpolation frame image A, B key frame image Px user position P _A , P _B shooting position

Claims (10)

A storage unit that stores a plurality of key frame images shot at a plurality of positions in association with shooting positions of each of the plurality of key frame images;
An acquisition unit that acquires a virtually moving user position;
A selection unit that selects, from among the plurality of key frame images stored in the storage unit, the plurality of key frame images whose photographing position is near the user position;
An interpolation frame image at the user position is obtained from at least one of the selected plurality of key frame images by interpolation processing based on the distance between the user position and the photographing position of each of the plurality of selected key frame images. And an interpolation processing unit,
The interpolation process is
A first process of overlapping the plurality of selected key frame images with the transparency set based on the distance between the user position and each of the plurality of selected key frame images;
A moving image including a second process for image enhancement or image de-emphasis according to the distance between the user position and the shooting position for at least one of the plurality of selected key frame images Generation system. The moving image generation system according to claim 1, wherein the transparency is set lower for the key frame image in which the distance between the user position and the photographing position is smaller. The second processing is brightness adjustment processing, contrast adjustment processing, blur processing, edge enhancement processing, resolution adjustment, color reduction, and claim 1 or comprising one or more treatment selected from the group consisting of increasing color processing The moving image generation system as described in 2 . The selection unit includes a photographing position distance between the plurality of the key frame image to be selected, varied according to a virtual moving speed of the user, according to claim 1 to 3 for selecting a plurality of said key frame image A moving image generation system according to any one of the items. The moving image generation system according to any one of claims 1 to 4 , wherein the user position and the imaging position are a position in a one-dimensional space, a position in a two-dimensional space, or a position in a three-dimensional space. The storage unit is
A first storage device that stores a plurality of key frame images and shooting positions of the plurality of key frame images in association with each other;
The plurality of key frame images and their shooting positions are acquired from the first storage device, and the number of the plurality of key frame images and the number of the key frame images are smaller than the number in the first storage device. A second storage device which stores the imaging position in association with each other;
Including
The selection unit is configured to select a plurality of the key frame images near the user position from among the plurality of key frame images stored in the second storage device.
The second storage device obtains a plurality of the key frame images stored in the first storage device and their shooting positions according to the priority determined based on the user position and the virtual movement direction of the user. The moving image generation system according to any one of claims 1 to 5 . The storage unit is
A first storage device that stores a plurality of key frame images and shooting positions of the plurality of key frame images in association with each other;
The plurality of key frame images and their shooting positions are acquired from the first storage device, and the number of the plurality of key frame images and the number of the key frame images are smaller than the number in the first storage device. A second storage device which stores the imaging position in association with each other;
Including
The selection unit is configured to select a plurality of the key frame images near the user position from among the plurality of key frame images stored in the second storage device.
The second storage device acquires a plurality of the key frame images stored in the first storage device and their shooting positions according to the priority determined based on the user position and the virtual movement speed of the user The moving image generation system according to any one of claims 1 to 6 . An acquisition unit that acquires a virtually moving user position;
A selection unit that selects, from among a plurality of key frame images, a plurality of key frame images whose shooting position is near the user position;
Interpolation to obtain an interpolated frame image at the user position from at least one of the selected plurality of key frame images by interpolation processing based on the distance between the user position and the photographing position of each of the plurality of selected key frame images And a processing unit,
The interpolation process is
A first process of overlapping the plurality of selected key frame images with the transparency set based on the distance between the user position and each of the plurality of selected key frame images;
A moving image including a second process for image enhancement or image de-emphasis according to the distance between the user position and the shooting position for at least one of the plurality of selected key frame images Generator. Get the user position to move virtually,
From the plurality of key frame images, select the plurality of key frame images whose shooting position is near the user position;
Obtaining an interpolated frame image at the user position from at least one of the selected plurality of key frame images by interpolation processing based on a distance between the user position and a photographing position of each of the plurality of selected key frame images; Including
To obtain the interpolated frame image,
Superimposing the plurality of selected key frame images with the transparency set based on the distance between the user position and each of the plurality of selected key frame images.
And performing processing for image enhancement or image de-emphasis according to the distance between the user position and the imaging position on at least one of the selected plurality of key frame images. Moving image generation method. An acquisition unit that acquires a virtually moving user position;
A selection unit that selects, from among a plurality of key frame images, a plurality of key frame images whose shooting position is near the user position;
Interpolation to obtain an interpolated frame image at the user position from at least one of the selected plurality of key frame images by interpolation processing based on the distance between the user position and the photographing position of each of the plurality of selected key frame images A processing unit,
Equipped with
The interpolation process is
A first process of overlapping the plurality of selected key frame images with the transparency set based on the distance between the user position and each of the plurality of selected key frame images;
A moving image including a second process for image enhancement or image de-emphasis according to the distance between the user position and the shooting position for at least one of the plurality of selected key frame images A computer program for causing a computer to function as a generation device.