JP7498616B2 - VR image generation device and program - Google Patents

Description

The present invention relates to a VR image generation device and program.

In recent years, the use of live-action video in virtual reality (VR) for viewing news reports, sports, and other events has been increasing. However, with spherical VR video, the shooting resolution is insufficient, and the resolution of the areas that users want to focus on may not be sufficient.

On the other hand, display devices such as HMDs (Head Mounted Displays) are becoming increasingly high quality, with some devices capable of displaying images in high resolution being announced. If VR cameras were to be made to have higher image quality in response, the size of the shooting device would increase. Furthermore, due to the nature of VR, it is not possible to shoot with sufficient resolution unless the camera is placed close to the subject, and the hassle of shooting close up is preventing its widespread use.

Also, depending on the content, the focal point may be only a part of the spherical image, and the rest may be necessary to get a feel for the atmosphere of the scene.

For these reasons, there is a demand for a VR image generating device that can display a portion of interest in high resolution. For example, Patent Document 1 discloses a technology that compensates for a low-quality spherical image (360-degree image) with a planar image by superimposing the planar image on the spherical image.

JP 2018-109946 A

However, although Patent Document 1 discloses a technique for integrating a still spherical image with a planar image, it does not specifically disclose any technique for generating VR video.

In view of the above, the object of the present invention is to provide a VR image generation device and program capable of generating partially high-resolution VR images.

A VR image generating device according to one embodiment receives a VR image captured by a VR camera and a 2D image captured by a 2D camera at a higher resolution in a range narrower than that of the VR camera, and generates a VR image onto which the 2D image is mapped. The device includes a calibration unit that receives the VR image and the 2D image captured at the same position and timing, generates first orientation information indicating the orientation of the VR camera from the VR image, and generates second orientation information indicating the orientation of the 2D camera and lens characteristic information indicating the characteristics of the lens of the 2D camera from the 2D image, an image synthesis unit that generates a synthetic image by mapping the 2D image onto the VR image based on the first orientation information, the second orientation information, and the lens characteristic information, and a rotation correction unit that generates a VR image by matching the shooting direction of the current frame of the synthetic image with the shooting direction of the first frame.

In one embodiment, a VR image generating device inputs a VR image captured by a VR camera and a 2D image captured by a 2D camera with a higher resolution in a narrower range than the VR camera, and generates a VR image onto which the 2D image is mapped. The VR image generating device includes: a calibration unit that inputs the VR image and the 2D image captured at the same position and timing, generates first orientation information indicating the orientation of the VR camera from the VR image, and generates second orientation information indicating the orientation of the 2D camera and lens characteristic information indicating the characteristics of the lens of the 2D camera from the 2D image; a still area extraction unit that aligns the positions of a previous frame and a current frame of the 2D image from the second orientation information and the lens characteristic information, and extracts an area in the current frame where the difference from the previous frame is equal to or less than a threshold as a still image; an image synthesis unit that generates a synthetic image by sequentially mapping the still images onto the VR image based on the first orientation information, the second orientation information, and the lens characteristic information; and a rotation correction unit that generates a VR image by matching the shooting direction of the current frame of the synthetic image with the shooting direction of the first frame.

Furthermore, in one embodiment, the VR camera and the 2D camera may be fixed via a connection part during shooting.

In one embodiment, the VR image generating device inputs a current VR image currently captured by a VR camera and a past VR image generated in the past, and generates a VR image onto which the past VR image is mapped, the past VR image being an image generated by mapping a 2D image captured in the past by a 2D camera with a higher resolution over a narrower range than the VR camera. The VR image generating device includes a calibration unit that inputs the current VR image and the past VR image, generates first orientation information from the current VR image that indicates the orientation of the VR camera, and generates third orientation information from the past VR image that indicates the orientation of a virtual VR camera that is assumed to have captured the past VR image; a rotation correction unit that generates a corrected VR image by correcting the current VR image so that the shooting direction matches that of the past VR image based on the first orientation information and the third orientation information; and a display switching unit that switches between the current VR image and the past VR image to be displayed on a display based on a user instruction.

Furthermore, in one embodiment, the past VR image generating device that generates the past VR image may include a calibration unit that inputs the 2D video and generates from the 2D video second attitude information indicating the attitude of the 2D camera and lens characteristic information indicating the characteristics of the lens of the 2D camera, a still area extraction unit that aligns the positions of the previous frame and the current frame of the 2D video from the second attitude information and the lens characteristic information and extracts an area in the current frame where the difference from the previous frame is equal to or less than a threshold as a still image, and an image synthesis unit that generates the past VR image by sequentially mapping the still images based on the second attitude information and the lens characteristic information.

In one embodiment, the program causes a computer to function as the VR image generation device.

The present invention makes it possible to generate partially high-resolution VR images.

1 is a block diagram showing an example of the configuration of a VR video generation device according to a first embodiment. FIG. FIG. 1 is a diagram showing an example of a VR camera and a 2D camera integrated by a connection portion. FIG. 11 is a block diagram showing an example of the configuration of a VR video generation device according to a second embodiment. A block diagram showing an example configuration of a past VR image generation device that generates past VR images input by a VR video generation device according to a third embodiment. A block diagram showing an example of the configuration of a VR video generation device according to a third embodiment.

The following describes the embodiments in detail with reference to the drawings.

First Embodiment
First, a description will be given of a VR video generation device according to the first embodiment. Fig. 1 shows an example of the configuration of a VR video generation system 1 including a VR video generation device 10 according to the first embodiment.

The VR video generation system 1 includes a VR camera 31, a 2D camera 32, a connection unit 33, a VR video storage unit 41, a 2D video storage unit 42, a VR video generation device 10, a feature point position information storage unit 21, and a first frame attitude storage unit 22. Note that the multiple storage units shown in FIG. 1 may be appropriately integrated into a smaller number of storage units, but are shown separately for ease of explanation.

The VR camera 31 is equipped with multiple cameras and stitches (integrates) images captured by the multiple cameras to generate VR images. The VR camera 31 then outputs the generated VR images to the VR image storage unit 41. The VR images are, for example, spherical images, but may also be wide-field images of less than 360 degrees.

The 2D camera 32 is a camera that captures a narrower range than the VR camera 31 at a higher pixel density (resolution). The 2D camera 32 is fitted with a lens, captures two-dimensional images, and outputs the 2D images to the 2D image storage unit 42. The lens may be a zoom lens or a single-focus lens. The number of pixels in the 2D camera 32 does not necessarily need to be greater than that of the VR camera 31, and because the 2D camera 32 captures a narrower range, it has a higher pixel density than the VR camera 31.

The VR camera 31 and the 2D camera 32 are fixed and integrated via a connection part (e.g., a bar) 33 so that their relative positional relationship does not change when shooting. Figure 2 shows an example of the VR camera 31 and the 2D camera 32 integrated by the connection part 33. The 2D camera 32 to which the VR camera 31 is fixed may perform rotational and translational motion.

The VR image generating device 10 inputs VR images captured by the VR camera 31 and 2D images captured by the 2D camera 32, and generates VR images onto which the 2D images are mapped. The VR images output by the VR image generating device 10 are partially high-resolution VR images, since high-resolution 2D images are mapped onto the VR images. In this specification, this partially high-resolution VR image is referred to as a "partial high-resolution VR image."

When inputting recorded video, the VR video generation device 10 inputs VR video from the VR video storage unit 41 and inputs 2D video from the 2D video storage unit 42. Also, when the VR video generation device 10 inputs live output from a camera, the VR video generation system 1 does not need to be equipped with the VR video storage unit 41 and the 2D video storage unit 42, and the VR video generation device 10 inputs VR video directly from the VR camera 31 and inputs 2D video directly from the 2D camera 32.

The VR image generating device 10 includes a calibration unit 11, an image synthesis unit 12, and a rotation correction unit 13. The VR image generating device 10 may be a workstation, or may be hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

The calibration unit 11 inputs VR video and 2D video captured at the same position and timing. To capture images at the same timing, the frames may be aligned based on the sound of a clapperboard or the like, or a time code may be used. In addition, to precisely align the timing, a synchronization signal may be input to the VR camera and 2D camera 32.

The calibration unit 11 uses a SLAM (Simultaneous Localization and Mapping) technique to obtain the orientation of the VR camera 31 from the VR video and generate first orientation information indicating the orientation of the VR camera 31. Then, the calibration unit 11 outputs the VR video and the first orientation information to the video synthesis unit 12. An example of the SLAM technique is a tool such as OpenVSLAM. The first orientation information includes a position O _VR and a direction (rotation matrix R _VR ).

Furthermore, the calibration unit 11 uses SLAM technology to obtain the attitude of the 2D camera 32 and the characteristics of the lens of the 2D camera 32 from the 2D video, and generates second attitude information indicating the attitude of the 2D camera 32 and lens characteristic information indicating the characteristics of the lens of the 2D camera 32. Then, the calibration unit 11 outputs the 2D video, the second attitude information, and the lens characteristic information to the video synthesis unit 12. The second attitude information includes a position O _2D , a translation vector T _2D , and a direction (rotation matrix R _2D ). The lens characteristic information includes distortion coefficients κ ₁ , κ ₂ , a focal length F of the camera, and center positions C _u , C _v of the image sensor. The calibration unit 11 acquires the attitudes of the VR camera 31 and the 2D camera 32 in the same coordinate system.

The calibration unit 11 also generates feature point position information indicating the positions of feature points of the 2D and VR images extracted during the calibration process, and stores the information in the feature point position information storage unit 21. The feature points can be extracted using existing techniques such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF).

The calibration unit 11 also stores first frame orientation information indicating the orientation of the VR camera 31 in the first frame in the first frame orientation storage unit 22. The calibration unit 11 also saves directional information of the 2D camera 32 relative to the VR camera 31 in the previous frame, extracts feature points only in its surroundings, and limits the matching search range, thereby shortening the processing time for calibration and improving accuracy.

The image synthesis unit 12 inputs the first attitude information, the second attitude information, and the lens characteristic information along with the VR image and the 2D image, and generates a synthetic image by mapping (overwriting) the 2D image onto the VR image based on these three pieces of information. The image synthesis unit 12 then outputs the first attitude information and the synthetic image to the rotation correction unit 13. For example, the image synthesis unit 12 uses a program to place a virtual projector that reflects the first attitude information, the second attitude information, and the lens characteristic information in the virtual space, and projects from the projector to overwrite the texture on the inner surface of the sphere, thereby generating a synthetic image.

Alternatively, the image synthesis unit 12 obtains the coordinates (u, v) of the corresponding pixel in the 2D image from the position W = (X, Y, Z) of the pixel on the sphere using the following formula (1), and generates a synthetic image by replacing the RGB values with the RGB values of the texture on the inner surface of the sphere. z is the distance from the optical principal point to the imaging surface, R _2D is a rotation matrix, and T _2D is a translation vector. A is a camera parameter and is expressed by the following formula (2). a is the aspect ratio (length/width) of the image. _Cu and _Cv are the center positions of the imaging sensor.

By processing this image synthesis unit 12, the position where the 2D image is mapped moves in accordance with the rotational movement, and the area to be mapped is enlarged or reduced in accordance with the zoom operation of the 2D camera 32.

The image synthesis unit 12 may also blend images near the four sides of the 2D camera 32 with the VR image to smooth the boundary between the VR image and the 2D image on the inside of the sphere.

The rotation correction unit 13 inputs the first orientation information and the composite image from the image composition unit 12, reads the feature point position information from the feature point position information storage unit 21, and reads the orientation information of the VR camera 31 in the first frame from the first frame orientation storage unit 22. From these pieces of information, the rotation correction unit 13 calculates the rotation difference _Rd between the first frame and the current frame of the composite image, multiplies the current frame by the inverse matrix of the rotation difference _Rd , and matches the shooting direction of the current frame of the composite image with the shooting direction of the first frame. Then, the rotation correction unit 13 converts the spherical image into a 2D image of an equidistant projection method. The rotation correction unit 13 repeatedly performs the above process on the next and subsequent frames to generate a partial high-resolution VR image.

If the processing by the rotation correction unit 13 is not performed, when the 2D camera 32 mounted with the VR camera 31 is rotated to capture an image, the center of the 2D camera 32 will always be the center of the display side, and the VR image will rotate. As a result, the image rotates regardless of the user's head movement, which may cause discomfort to the user. Therefore, by performing processing by the rotation correction unit 13, it is possible to generate an image in which the 2D image moves relatively while keeping the orientation of the VR image fixed. Processing by the rotation correction unit 13 also makes it possible to display the image in a form that enlarges and shrinks in accordance with the zoom operation of the 2D camera 32.

As described above, the VR image generating device 10 maps 2D images captured by a 2D camera with a higher resolution than the VR camera onto the VR image. Therefore, according to this embodiment, it is possible to display a notable subject in high image quality. In other words, the area of the omnidirectional image captured by the 2D camera often overlaps with a notable area of the omnidirectional image, and this part can be displayed in high resolution. In addition, the intention of the creator at the time of shooting can be concentrated in the high-resolution part, and the production intention and gaze guidance of the video production side can be conveyed to the user without difficulty.

In addition, when a VR camera and a 2D camera are installed separately and photographed as in the past, the camera parameters of the VR camera and the 2D camera must be calculated for each frame, which increases the burden during photographing. In this regard, in this embodiment, photographing is performed at the same position by fixing the VR camera 31 and the 2D camera 32 with the connection part 33, for example. Therefore, according to this embodiment, only one photographing position is required, and there is no need to set up a new VR camera, and since there is no significant change from the conventional photographing flow, it is possible to suppress an increase in the burden during photographing. In addition, there is no need to calculate the camera parameters (camera position and attitude, and lens characteristics) of the VR camera 31 and the 2D camera 32 for each frame, and only the parameters of the lens characteristics of the 2D camera 32 need to be calculated, which increases the speed.

Second Embodiment
Next, a VR image generation device according to a second embodiment will be described.

Figure 3 shows an example configuration of a VR video generation system 2 including a VR video generation device 10a according to the second embodiment. The VR video generation system 2 includes a VR camera 31, a 2D camera 32, a connection unit 33, a VR video storage unit 41, a 2D video storage unit 42, the VR video generation device 10a, a feature point position information storage unit 21, and a first frame attitude storage unit 22. Note that the multiple storage units shown in Figure 3 may be appropriately integrated into a smaller number of storage units, but are shown separately for ease of explanation. The same reference numbers are used for the same configuration as in the first embodiment, and explanations are omitted as appropriate.

The VR image generating device 10a includes a calibration unit 11a, an image synthesis unit 12a, a rotation correction unit 13a, and a still area extraction unit 14. The VR image generating device 10a differs from the VR image generating device 10 of the first embodiment in that it further includes a still area extraction unit 14. The VR image generating device 10a may be a workstation, or may be hardware such as an ASIC or FPGA.

Similar to the calibration unit 11 of the first embodiment, the calibration unit 11a generates first posture information indicating the posture of the VR camera 31, second posture information indicating the posture of the 2D camera 32, lens characteristic information indicating the characteristics of the lens of the 2D camera 32, and feature point position information. Then, the calibration unit 11a outputs the VR video and the first posture information to the video synthesis unit 12a, outputs the 2D video, the second posture information, and the lens characteristic information to the still area extraction unit 14, and stores the feature point information in the feature point position information storage unit 21.

The still region extraction unit 14 aligns the positions of the previous and current frames of the 2D video from the second orientation information and the lens characteristic information, extracts the region in the current frame where the difference from the previous frame is equal to or less than a threshold (a region with no movement) as a still image, and outputs this to the image synthesis unit 12a together with the second orientation information and the lens characteristic information.

The still region extraction unit 14 also stores the image, posture information, and lens characteristic information of the current frame of the 2D video as previous frame information in the previous frame information storage unit 23. Note that this process is not performed for the first frame, as there is no previous frame.

The image synthesis unit 12a inputs the VR image, the first attitude information (position and rotation), the 2D image, the second attitude information, and the lens characteristic information. The image synthesis unit 12a generates a mapping VR image by sequentially mapping and overwriting the still images, and stores the mapping VR image in the mapping VR image storage unit 24. That is, the image synthesis unit 12a reads the mapping VR image that has been mapped so far from the mapping VR image storage unit 24, maps the still images to the mapping VR image based on the input second attitude information and lens characteristic information, and writes it again in the mapping VR image storage unit 24. Then, the image synthesis unit 12a generates a composite image by synthesizing (overwriting) the mapping VR image with the VR image input from the calibration unit 11, and outputs the composite image and the first attitude information to the rotation correction unit 13a. Note that the image synthesis unit 12a does not overwrite the already overwritten part from the second time onwards, but stores it in each pixel, and it is also possible to animate it by sequentially repeating playback or mixing forward playback and reverse playback over time. In other words, if multiple mappings are made to a pixel, playing the mapped RGB values in sequence can easily create a video. However, if you play them in sequence, go back to the beginning when it reaches the end, and start playing again, a shock (discontinuity) will occur due to the beginning of the video, so if you play to the end, play in reverse (rewind), play to the beginning, and then play forward again, you can play a continuous video without any shocks.

The rotation correction unit 13a generates a partial high-resolution VR image in the same manner as in the first embodiment and outputs it to the outside.

In this embodiment, it is assumed that the 2D camera 32 equipped with the VR camera 31 only performs rotational motion and does not perform large translational motion. This is because if translational motion is performed, the 2D image is geometrically transformed, and still images extracted from frames other than the current frame will no longer match the VR image of the current frame.

In the first embodiment, the area to be made high resolution is limited to the area captured by the 2D camera 32. On the other hand, in the present embodiment, a process is added to extract still areas from the 2D video, so that areas not captured by the 2D camera 32 in the current frame can also be made high resolution by mapping still areas from frames other than the current frame onto the VR video.

Third Embodiment
Next, a VR image generation device according to a third embodiment will be described.

Figures 4 and 5 show an example configuration of a VR video generation system 3 including a VR video generation device 10b according to the third embodiment. In this embodiment, after a past VR image is generated by a past VR image generation device 50 shown in Figure 4, a partial high-resolution VR image is generated using the past VR image by a VR video generation device 10b shown in Figure 5.

The VR video generation system 3 includes a 2D camera 32, a 2D video storage unit 42, a past VR image generation device 50, a feature point position information storage unit 61, a previous frame information storage unit 62, a mapping VR image storage unit 63, a past VR image storage unit 64, a VR camera 31, a VR video storage unit 41, a VR video generation device 10b, an input I/F 81, a head mounted display 82, and a head direction detection sensor 83. Note that the multiple storage units shown in Figures 4 and 5 may be appropriately integrated to form a smaller number of storage units, but are described separately for ease of explanation. The same reference numbers are used for the same configuration as in the first embodiment, and explanations are omitted as appropriate.

First, the past VR image generation device 50 will be described with reference to FIG. 4. The 2D camera 32 is equipped with a zoom lens or a single focus lens. The 2D camera 32 stores 2D images captured in various directions in the 2D image storage unit 42. In this embodiment, it is assumed that the 2D camera 32 only involves rotational movement and does not involve large translational movement.

The past VR image generating device 50 inputs 2D images captured in various directions from the 2D image storage unit 42, extracts still areas, and stitches them together to generate a VR image.

The past VR image generating device 50 includes a calibration unit 51, a still area extraction unit 52, and a video synthesis unit 53. The past VR image generating device 50 may be a workstation, or may be hardware such as an ASIC or FPGA.

The calibration unit 51 inputs the 2D video from the 2D video storage unit 42. The calibration unit 51 uses SLAM technology to generate second attitude information indicating the attitude of the 2D camera 32 from the 2D video, and lens characteristic information indicating the characteristics of the lens of the 2D camera 32, and outputs the 2D video, the second attitude information, and the lens characteristic information to the still area extraction unit 52. The calibration unit 51 also stores the positions of the feature points extracted when performing the calibration process in the feature point position information storage unit 61. In this embodiment, the image synthesis unit 53 maps images for each frame, and the feature point position information is used to determine the relative positions of the images between frames. Note that when the video (image group) is input all at once, the positional relationship between the frames is known, and therefore the feature point position information is not necessary.

The still area extraction unit 52 inputs the 2D video, the second orientation information, and the lens characteristic information, and outputs the 2D video, the second orientation information, and the lens characteristic information of the previous frame to the previous frame information storage unit 62. When the still area extraction unit 52 inputs the 2D video, the second orientation information, and the lens characteristic information of a new frame, it reads the 2D video, the second orientation information, and the lens characteristic information of the previous frame from the previous frame information storage unit 62. The still area extraction unit 52 aligns the positions of the previous and current frames of the 2D video from the second orientation information and the lens characteristic information, extracts an area of the current frame where the difference from the previous frame is equal to or less than a threshold as a still image, and outputs this to the image synthesis unit 53 together with the camera orientation information and the lens characteristic information. Note that this process is not performed for the first frame because there is no previous frame.

The video synthesis unit 53 reads mapping VR images up to the previous frame from the mapping VR image storage unit 63. The video synthesis unit 53 inputs still images, second attitude information, and lens characteristic information from the still area extraction unit 52, and generates a past VR image by sequentially mapping the still images based on the second attitude information and lens characteristic information. Specifically, the video synthesis unit 53 maps (overwrites) still images onto the mapping VR images read from the mapping VR image storage unit 63. When the video synthesis unit 53 has completed processing for all frames, it stores the final mapping VR image in the past VR image storage unit 64 as a past VR image. Anything that has already been overwritten once is not overwritten from the second time onwards, but is saved in each pixel, and it is also possible to play it repeatedly in sequence over time, or to animate it by mixing forward and reverse playback.

Next, the VR image generating device 10b will be described with reference to FIG. 5. The VR image generating device 10b receives a current VR image currently captured by the VR camera 31 and a past VR image generated in the past, and generates a VR image onto which the past VR image is mapped.

The VR camera 31 is equipped with multiple cameras, and stitches together (integrates) images captured by the multiple cameras at the same time to generate VR images, which are then output to the VR image storage unit 41. It is assumed that the VR camera 31 captures images from the same position as the 2D camera 32.

The VR image generating device 10b includes a calibration unit 11b, a rotation correction unit 13b, and a display switching unit 15. The VR image generating device 10b may be a workstation, or may be hardware such as an ASIC or FPGA.

The calibration unit 11b inputs the current VR image (current VR image) captured by the VR camera 31 and the past VR image generated by the past VR image generation device 50. The calibration unit 11b uses SLAM technology to determine the attitude of the VR camera 31 from the current VR image and generates first attitude information indicating the attitude of the VR camera 31. The calibration unit 11b also uses SLAM technology to determine the attitude of a virtual VR camera that is assumed to have captured the past VR image from the past VR image and generates third attitude information indicating the attitude of the virtual VR camera. The calibration unit 11b then outputs the current VR image, the past VR image, the first attitude information, and the third attitude information to the rotation correction unit 13b.

The rotation correction unit 13b inputs the current VR video, the past VR image, the first attitude information, and the third attitude information from the calibration unit 11b. The rotation correction unit 13b calculates a rotation difference _Rd of the current VR video as viewed from the past VR image based on the first attitude information and the third attitude information. The rotation correction unit 13b then applies an inverse correction _Rd ^-1 of the rotation difference _Rd to the current VR video, corrects the current VR video so that the shooting direction of the current VR video matches that of the past VR image, and generates a corrected VR video. The rotation correction unit 13b then outputs the corrected VR video and the past VR image to the display switching unit 15.

The head mounted display (HMD) 82 is a display device that is fixed to the user's head and is made up of two displays for the right and left eyes, with two convex lenses for each. Instead of two displays, a single display may be split into left and right halves.

The head direction detection sensor 83 is attached to the HMD 82 and is a sensor for detecting the forward direction of the head of the user wearing the HMD 82. Methods for detecting the forward direction of the head include a method of detecting a marker on the HMD 82 using an acceleration sensor or a camera placed around the device, and a method of obtaining and detecting an external image from a camera installed on the HMD 82.

The input I/F 81 is an interface that accepts instructions from the user to the VR image generating device 10b, and specifically includes a lever that can be moved in the up, down, left, and right directions, as well as buttons. The input I/F 81 outputs control signals to the display switching unit 15 that indicate user instructions, such as the up, down, left, and right directions of the lever, the position of the lever, and the on/off state of a button.

The display switching unit 15 installs two virtual cameras at the center of the sphere with a pupil distance separation to render images for the left and right eyes, and renders images in the direction corresponding to the head direction detected by the head direction detection sensor 83.

The display switching unit 15 switches between the current VR video and the past VR image to be displayed on the HMD 82 based on a control signal (i.e., a user's instruction) input from the input I/F 81. Since the past VR image is an image captured by the 2D camera 32, which has a higher resolution than the VR camera 31, the video that the display switching unit 15 displays on the HMD 82 is a partial high-resolution VR video. The display switching unit 15 may store the past VR image input in the first frame in a storage unit (not shown), and read it from the storage unit from the second frame onwards.

The display switching unit 15, for example, arranges identical spheres in a virtual space, overlapping them, and maps the current VR image onto the inside of one sphere and the past VR image onto the inside of the other sphere. The display switching unit 15 can switch the current VR image and the past VR image between a visible state and an invisible state, respectively, and in the initial state, half of the celestial sphere may be the current VR image and the other half the past VR image. For example, the user can select the area he or she wants to view by rotating the celestial sphere, and in this case, the display switching unit 15 rotates the celestial sphere based on a rotation instruction input from the input I/F 81.

As another display switching method, the display switching unit 15 may make the current VR video visible in the celestial sphere, and in response to the lever position and button on control from the input I/F 81, invert the visible/invisible state of the past VR image and the current VR video for the area of the sphere in the corresponding direction, thereby performing display switching so that the user feels that only the area traced with the brush is visible to the past VR image. In addition, the display switching unit 15 may first make the current VR video visible in the celestial sphere, and in accordance with the button on/off control input from the input I/F 81, make the current VR video visible when the button is on, and make the past VR image visible when the button is off, thereby performing display switching between the current VR video and the past VR image in the celestial sphere.

The display switching unit 15 may also have a function of enlarging and outputting an image in the line of sight, as if viewed through binoculars, when it detects that a specific button on the input I/F 81 has been pressed. If the image in the line of sight is high resolution, the user can view it with a sense of realism.

According to this embodiment, a user wearing the HMD 82 can operate the input I/F 81 to switch between the current VR video and a past VR image captured in the same shooting position in the past. This makes it possible, for example, to display a comparison of the difference between past and present scenery from the same viewpoint. In the second embodiment, the VR video generation device 10a sequentially overwrites the 2D video, so that the high-definition area gradually expands, but in this embodiment, the VR video generation device 10b can generate high-definition VR video in all areas captured by the 2D camera 32 from the beginning.

<Program>
A computer capable of executing program instructions can be used to function as the VR image generation devices 10, 10a, and 10b. The computer stores a program describing the processing contents for realizing each function of the VR image generation devices 10, 10a, and 10b in the storage unit of the computer, and reads and executes the program by the processor of the computer. A part of the processing contents may be realized by hardware. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be a program code, a code segment, or the like for executing a required task. The processor may be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or the like.

A program for functioning as the VR image generating device 10 causes a computer to execute the steps of, for example, inputting VR image and 2D image captured at the same position and timing, generating first orientation information indicating the orientation of the VR camera from the VR image, and generating second orientation information indicating the orientation of the 2D camera and lens characteristic information indicating the characteristics of the lens of the 2D camera from the 2D image, generating a composite image by mapping the 2D image onto the VR image based on the first orientation information, second orientation information, and lens characteristic information, and generating a VR image in which the shooting direction of the current frame of the composite image is aligned with the shooting direction of the first frame.

A program for functioning as the VR image generating device 10a causes a computer to execute the following steps, for example: inputting VR images and 2D images captured at the same position and timing; generating first orientation information indicating the orientation of the VR camera from the VR images; generating second orientation information indicating the orientation of the 2D camera from the 2D images and lens characteristic information indicating the characteristics of the lens of the 2D camera; aligning the positions of the previous and current frames of the 2D images from the second orientation information and lens characteristic information, and extracting an area in the current frame where the difference from the previous frame is equal to or less than a threshold as a still image; generating a composite image by sequentially mapping the still images onto the VR image based on the first orientation information, second orientation information, and lens characteristic information; and generating a VR image in which the shooting direction of the current frame of the composite image is aligned with the shooting direction of the first frame.

A program for functioning as VR video generation device 10b causes a computer to execute the following steps, for example: inputting a current VR video and a past VR image, generating first orientation information indicating the orientation of the VR camera from the current VR video, and generating second orientation information indicating the orientation of the 2D camera from the past VR image; generating a corrected VR video in which the current VR video is corrected so that the shooting direction matches that of the past VR image; and switching between the current VR video and the past VR image to be displayed on the display based on a user instruction.

The program may also be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a CD-ROM, DVD-ROM, or other recording medium. The program may also be provided by downloading via a network.

The above-mentioned embodiment has been described as a representative example, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes are possible without departing from the scope of the claims. For example, it is possible to combine multiple configuration blocks shown in the configuration diagram of the embodiment into one, or to divide one configuration block.

Furthermore, the VR image generating device 10 according to the first embodiment and the VR image generating device 10a according to the second embodiment may also have a function of inputting a control signal from the input I/F 81, enlarging the high-resolution area captured by the 2D camera 32 based on the control signal, and displaying it on the HMD 82, as in the third embodiment. This allows the user to view high-resolution images with a sense of realism, even if the resolution of the HMD 82 in use is insufficient.

1, 2, 3 VR image generation system 10, 10a, 10b VR image generation device 11, 11a, 11b Calibration section 12, 12a, 12b Image synthesis section 13 Rotation correction section 14 Still area extraction section 15 Display switching section 21 Feature point position information storage section 22 First frame attitude storage section 23 Previous frame information storage section 24 Mapping VR image storage section 31 VR camera 32 2D camera 33 Connection section 41 VR image storage section 42 2D image storage section 50 Past VR image generation device 51 Calibration section 52 Still area extraction section 53 Image synthesis section 61 Feature point position information storage section 62 Previous frame information storage section 63 Mapping VR image storage section 64 Past VR image storage section 81 Input I/F
82 Head mounted display 83 Head direction detection sensor

Claims (6)

A VR image generating device that receives a VR image captured by a VR camera and a 2D image captured by a 2D camera with a higher resolution and a range narrower than that of the VR camera, and generates a VR image onto which the 2D image is mapped,
a calibration unit that inputs the VR image and the 2D image captured at the same position and timing, generates first orientation information indicating an orientation of the VR camera from the VR image, and generates second orientation information indicating an orientation of the 2D camera from the 2D image and lens characteristic information indicating characteristics of the lens of the 2D camera;
an image synthesis unit that generates a synthetic image by mapping the 2D image onto the VR image based on the first posture information, the second posture information, and the lens characteristic information;
a rotation correction unit that generates a VR image by aligning a shooting direction of a current frame of the composite image with a shooting direction of a first frame;
A VR image generating device comprising: A VR image generating device that receives a VR image captured by a VR camera and a 2D image captured by a 2D camera with a higher resolution and a range narrower than that of the VR camera, and generates a VR image onto which the 2D image is mapped,
a calibration unit that receives the VR image and the 2D image captured at the same position and timing, generates first orientation information indicating an orientation of the VR camera from the VR image, and generates second orientation information indicating an orientation of the 2D camera from the 2D image and lens characteristic information indicating characteristics of the lens of the 2D camera;
a still region extraction unit that aligns positions of a previous frame and a current frame of the 2D video based on the second orientation information and the lens characteristic information, and extracts, as a still image, a region in the current frame where a difference between the current frame and the previous frame is equal to or smaller than a threshold;
an image synthesis unit that generates a synthetic image by sequentially mapping the still images onto the VR image based on the first attitude information, the second attitude information, and the lens characteristic information;
a rotation correction unit that generates a VR image by aligning a shooting direction of a current frame of the composite image with a shooting direction of a first frame;
A VR image generating device comprising: The VR image generating device according to claim 1 or 2, wherein the VR camera and the 2D camera are fixed via a connection part during shooting. A VR image generating device that receives a current VR image currently captured by a VR camera and a past VR image generated in the past, and generates a VR image onto which the past VR image is mapped, comprising:
The past VR image is an image generated by mapping a 2D image captured in the past by a 2D camera at a higher resolution in an area narrower than the VR camera,
a calibration unit that receives the current VR video and the past VR image, generates first orientation information from the current VR video that indicates an orientation of the VR camera, and generates third orientation information from the past VR image that indicates an orientation of a virtual VR camera that was assumed to have captured the past VR image;
a rotation correction unit that generates a corrected VR image by correcting the current VR image so that a shooting direction of the current VR image coincides with a shooting direction of the past VR image based on the first attitude information and the third attitude information;
a display switching unit that switches between the current VR video and the past VR image to be displayed on a display based on an instruction from a user;
A VR image generating device comprising: A past VR image generating device for generating the past VR image includes:
a calibration unit that receives the 2D image and generates, from the 2D image, second posture information indicating a posture of the 2D camera and lens characteristic information indicating characteristics of a lens of the 2D camera;
a still region extraction unit that aligns positions of a previous frame and a current frame of the 2D video based on the second orientation information and the lens characteristic information, and extracts, as a still image, a region in the current frame where a difference between the current frame and the previous frame is equal to or smaller than a threshold;
an image synthesis unit that generates the past VR image by sequentially mapping the still images based on the second attitude information and the lens characteristic information;
The VR image generating device according to claim 4 . A program for causing a computer to function as a VR image generating device according to any one of claims 1 to 5.

Images