JP7020024B2 - Information processing equipment and programs - Google Patents

Description

The present invention relates to an information processing apparatus and a program.

In recent years, with the spread of omnidirectional cameras, there are increasing opportunities to view omnidirectional images taken in all directions of shooting points using a dedicated viewer. Here, the dedicated viewer for the all-sky image is to map the all-sky image to the surface of a predetermined three-dimensional object to generate a three-dimensional model, and to perform the user's display range change operation (pan, tilt, zoom). Correspondingly, it is an application that displays a two-dimensional image obtained by projecting and transforming a part of a region of a three-dimensional model of an all-sky sphere image.

Here, when trying to search the area of interest in the spherical image using the conventional dedicated viewer, the user performs a manual operation (pan / tilt / zoom) to change the display area. I had to visually check the entire spherical image.

Regarding this point, Japanese Patent Application Laid-Open No. 2015-18013 (Patent Document 1) responds to the fact that the areas of interest included in the spherical image are thumbnailed in advance and displayed in a list, and the thumbnails are selected. Disclose a display control device that displays the entire area of interest.

According to the display control device of Patent Document 1, the time and effort of searching the attention area of the spherical image can be saved, but on the other hand, the time and effort of manually selecting the thumbnail still remains, and the time and effort is the number of attention areas. It increases as it increases.

The present invention has been made in view of the above, and an object of the present invention is to provide an information processing apparatus capable of presenting a region of interest of a spherical image in an easy-to-understand manner while reducing the time and effort of a user's operation. ..

As a result of diligent studies on the configuration of an information processing device that can present the area of interest of the spherical image in an easy-to-understand manner while reducing the time and effort of the user's operation, the present inventors have come up with the following configuration. It led to the invention.

That is, according to the present invention, in the spherical image, the viewpoint management unit that registers the viewpoints designated by the user in the designated order, and the first viewpoint in the spherical image and the first viewpoint are designated next to the first viewpoint. A viewpoint control unit that transitions the viewpoint along a path that interpolates between the second viewpoints, a calculation unit that generates a partial image centered on each transition viewpoint, and a plurality of generated partial images of the viewpoint. An information processing device including a control unit that executes an animation display connected in the order of transition is provided.

As described above, according to the present invention, there is provided an information processing apparatus capable of presenting a region of interest of a spherical image in an easy-to-understand manner while reducing the time and effort of a user's operation.

The figure which shows the structural example of the image pickup system which includes the information processing apparatus of this embodiment. The hardware configuration diagram of the information processing apparatus of this embodiment. The external view of the information processing apparatus of 1st Embodiment. The functional block diagram of the information processing apparatus of 1st Embodiment. The figure for demonstrating a partial image. The figure which shows the application screen at the time of viewpoint registration. The figure which shows the application screen at the time of viewpoint registration. The figure which shows the animation at the time of preview playback. The figure which shows the partial image at the time of viewpoint registration. The figure which shows the application screen at the time of viewpoint editing. The figure which shows the application screen at the time of a detailed setting. The figure which shows the application screen at the time of moving image generation. A sequence diagram showing the processing executed when registering the viewpoint. The figure which shows the viewpoint information management table. A sequence diagram showing the processing executed during preview playback. The figure which shows the application screen at the time of viewpoint registration of 2nd Embodiment. The figure for demonstrating the setting of a transition direction. The figure which shows the preset data. The figure which shows the application screen when using the preset data. The figure which shows the application screen when using the preset data. The figure which shows the application screen when using the preset data. The figure which shows the application screen when using the preset data. A diagram illustrating parameters used to change the visual representation. The figure which shows the example of the projection type. The figure which shows the example of the projection type. The figure which shows the example of the projection type. The figure which shows the application screen at the time of setting a projection type. A sequence diagram showing the processing executed when registering the viewpoint. A diagram illustrating animation transitions between projection types. A diagram illustrating animation transitions between projection types. A diagram illustrating animation transitions between projection types. A sequence diagram showing the processing executed during preview playback.

Hereinafter, the present invention will be described with reference to embodiments, but the present invention is not limited to the embodiments described later. In each of the figures referred to below, the same reference numerals are used for common elements, and the description thereof will be omitted as appropriate.

FIG. 1 shows a configuration example of an imaging system including the information processing apparatus 100 of the present embodiment. The image pickup system includes at least one information processing device 100 and at least one image pickup device 110, and is communicably connected to each other. In the example shown in FIG. 1, it is composed of three information processing devices 100a to 100c and one image pickup device 110, and each of the three information processing devices 100a to 100c and the image pickup device 110 are wirelessly connected.

As the image pickup device 110, a spherical camera that captures a spherical image can be used. Here, the "omnidirectional image" is an image obtained by capturing a substantially omnidirectional image centered on the spherical camera in one imaging. The omnidirectional image does not necessarily have to be an image captured in all directions of 360 °, and includes, for example, an image excluding the direction of several degrees or several tens of degrees with the handle portion of the omnidirectional camera. The spherical camera includes at least two fisheye lenses, at least two image sensors, a controller, and a communication interface as hardware.

The fisheye lens has an angle of view of 180 ° or more. The angle of view represents the range of the image captured by the camera as an angle. The image pickup element is arranged at a position (imaging position) where light is focused by a fisheye lens, and converts an optical image formed by the focused light into image data of an electric signal and outputs it. As the image pickup element, an image sensor such as a CMOS sensor or a CCD sensor can be used.

The image pickup apparatus 110 includes, for example, two fisheye lenses and two image pickup elements, and the two fisheye lenses are arranged so as to face each other back to each other and capture an image (hemispherical image) in which an orientation of 180 ° or more is captured.

The controller performs predetermined processing on each hemispherical image, joins the two hemispherical images to generate a spherical image, and processes the information processing devices 100a to 100c that request the spherical image via the communication interface. Send. Although the spherical camera has been briefly described, since it is not directly related to the present invention, further detailed hardware configurations and functions will be omitted.

The three information processing devices 100a to 100c perform wireless communication with the image pickup device 110, acquire the spherical image captured by the image pickup device 110 from the image pickup device 110, and display the image. Therefore, the information processing devices 100a to 100c can perform wireless communication with the image pickup device 110, display the acquired spherical image, and execute a predetermined process on the spherical image. Any device may be used as long as it is used. In the example shown in FIG. 1, the information processing device 100a is a smartphone, the information processing device 100b is a tablet PC, and the information processing device 100c is a notebook PC. The information processing device 100 is not limited to these devices, and may be a desktop PC, a digital camera, a projector, an electronic blackboard, an MFP (Multifunction Peripheral), or the like.

In the example shown in FIG. 1, the three information processing devices 100a to 100c and the image pickup device 110 are wirelessly connected by a wireless LAN or the like, but the present invention is not limited to this, and the three information processing devices 100a to 100c are connected by wire using a cable or the like. May be. Further, the three information processing devices 100a to 100c and the image pickup device 110 are not limited to being directly connected by wireless or wired, and may be connected via a network such as the Internet.

The system configuration is as described above. Next, the hardware configuration of the information processing apparatus 100 of the present embodiment will be described with reference to FIG. Any of the information processing devices 100a to 100c shown in FIG. 1 can adopt the hardware configuration shown in FIG.

The information processing apparatus 100 functions as a processor 10 that controls the operation of the entire apparatus, a ROM 12 that stores a boot program, a firmware program, and the like, a RAM 13 that provides an execution space for the program, and the information processing apparatus 100 as means described later. A communication interface for communicating between an auxiliary storage device 14 for storing a program, an operating system (OS), and the like, an input / output interface 15 for connecting a display 18 and the like, and an image pickup device 110. It is equipped with 16.

The processor 10 reads a program from the auxiliary storage device 14 and executes it, and requests the image pickup device 110 to acquire a spherical image via the communication interface 16. Further, the processor 10 stores the spherical image acquired via the communication interface 16 in the RAM 13 or the auxiliary storage device 14 under the execution of the program, displays it on the display 18, and displays the spherical image on the spherical image. Executes a predetermined process. Details of these will be described later.

(First Embodiment)
FIG. 3 shows the appearance of the information processing apparatus 100 according to the first embodiment of the present invention. The information processing device 100 of the present embodiment is an information processing device including a display 18, and is a moving image that continuously displays an arbitrary area included in the whole celestial sphere image as a program for executing the above-mentioned predetermined processing. An application program (hereinafter referred to as a dedicated application) for generating information is installed. In the example shown in FIG. 3, a smartphone is shown as the information processing device 100, but the information processing device 100 is not limited to the smartphone, and may be the above-mentioned tablet PC, notebook PC, desktop PC, or the like. You may. In the following, a case where a smartphone is adopted as the information processing apparatus 100 will be described as an example.

As shown in FIG. 3, the operation screen of the dedicated application includes an area 20, an area 22, and an area 23. Here, in the area 20, a two-dimensional image obtained by projecting and transforming a part of the whole celestial sphere image (three-dimensional model) is displayed. Further, in the area 22, thumbnails of registered viewpoints (described later) are displayed. Further, in the area 23, a slider for setting the transition speed (described later), a "preview" button for previewing the generated video, and a "deletion" for deleting the registration viewpoint (described later). Various operation buttons such as a button, a "save" button for saving the generated video, and a "detailed setting" button for making detailed settings are displayed.

Subsequently, the functional configuration of the information processing apparatus 100 will be described with reference to the functional block diagram shown in FIG. Since the information processing device 100 causes the information processing device 100 to function as each means by executing the above-mentioned dedicated application by the processor 10, the information processing device 100 that executes the dedicated application is said to have each means. be able to. Here, it is described that it functions as each means by executing a dedicated application, but the present invention is not limited to this, and some or all of the functions may be realized by hardware such as a dedicated circuit.

As shown in FIG. 4, the information processing apparatus 100 has a display unit 102, a control unit 103, a calculation unit 104, a viewpoint management unit 105, a viewpoint control unit 106, and a moving image data generation unit as each of the above means. It includes 107, and further includes a storage area 108 realized by the RAM 13 and the auxiliary storage device 14.

The display unit 102 is a means for generating an image to be displayed on the display 18 and displaying a UI on the display 18 to accept a user's operation.

The viewpoint management unit 105 is a means for registering and managing viewpoints designated by the user in a three-dimensional model in which a spherical image is mapped to the surface of a predetermined three-dimensional object (for example, the inner surface of a sphere).

The viewpoint control unit 106 is a means for transitioning the viewpoint from the first viewpoint along a path that interpolates the second viewpoint designated after the first viewpoint in the three-dimensional model of the spherical image. ..

The calculation unit 104 is a means for executing various image processing and geometric calculation, and is a means for generating a partial image centered on each viewpoint transitioning on the path. Here, the "partial image" in the present embodiment means a two-dimensional image obtained by projecting and transforming a part of a region centered on an arbitrary viewpoint on the three-dimensional model of the whole celestial sphere image.

The control unit 103 is a means for executing an animation display in which a plurality of generated partial images are connected in the order of transition of viewpoints.

The moving image data generation unit 107 is a means for converting a plurality of generated partial images into moving image data in a general-purpose file format.

The storage area 108 is a means for storing various data.

The functional configuration of the information processing apparatus 100 has been described above, but subsequently, a method of using the dedicated application of the present embodiment will be described.

The display unit 102 of the information processing apparatus 100 of the present embodiment displays a partial image generated by the calculation unit 104 in the area 20 of the display screen (touch panel) of the smartphone, and the spherical image is omnidirectional in response to the user's operation on the area 20. Accepts changes in the viewpoint and angle of view on the image (pan, tilt, zoom in, zoom out). 5 (a) to 5 (d) schematically show partial images pan / tilt / zoom in / zoom out according to a user's operation.

In using the dedicated application of the present embodiment, the user first, as shown in FIG. 6A, a desired position in the partial image displayed in the area 20 by a predetermined operation such as a long press or a double tap. Is specified as the first viewpoint (viewpoint 1). In response to this, the application screen transitions to the state shown in FIG. 6B, and the icon 21 indicating that the viewpoint 1 is registered is displayed at the position specified by the user in the partial image, and the area is displayed. An icon 24 for calling the viewpoint 1 is displayed as a thumbnail on 22.

Subsequently, the user operates the area 20 to change the viewpoint, displays another partial image, and then, as shown in FIG. 7 (a), the desired position in the partial image is the second viewpoint. Designate as (viewpoint 2). In response to this, the application screen transitions to the state shown in FIG. 7B, and the icon 21 indicating that the viewpoint 2 is registered is displayed at the position specified by the user in the partial image, and the area is displayed. Next to the icon 24 mentioned earlier in 22, the icon 25 for calling the viewpoint 2 is displayed as a thumbnail.

Here, in the present embodiment, when the number of registered viewpoints (hereinafter, may be referred to as registered viewpoints) becomes two or more, the two registered viewpoints can be connected and preview-played as an animation. ..

In the above example, on the application screen shown in FIG. 7B, when the user selects the "preview" button displayed in the area 23, the animation as shown in FIG. 8 is reproduced and displayed in the area 20. This preview playback allows the user to confirm whether or not the animation of the spherical image is intended. When the reproduction display of the animation is completed up to the viewpoint 2, the reproduction start icon 30 is displayed. By tapping the playback start icon 30, the animation preview may be executed again.

In the present embodiment, as described above, the state in which the two viewpoints (start point and end point) of "viewpoint 1" and "viewpoint 2" are registered is the minimum configuration, and thereafter, the most recently registered viewpoint (end point). The number of registered viewpoints can be increased by repeating the procedure of registering a new viewpoint (end point) with the above as the next start point. The application may register the starting point (viewpoint) of the first animation by default, and in that case, the minimum configuration is established if the user specifies at least one viewpoint.

For example, following the two "viewpoint 1" and "viewpoint 2" described above, four viewpoints ("viewpoint 3", "viewpoint 4", "viewpoint 5", and "viewpoint 6") as shown in FIG. 9 are used. Can be added. When registering the viewpoint, the angle of view can be changed by performing a zoom-in / zoom-out operation on the partial image displayed in the area 20. By registering viewpoints with different angles of view in this way, it becomes possible to incorporate zoom-in / zoom-out effects into the animation.

FIG. 10A shows an application screen after registering the above-mentioned six viewpoints (viewpoints 1 to 6). In this case, six icons corresponding to the six viewpoints are displayed as thumbnails in the area 22.

In the present embodiment, when three or more viewpoints are registered in this way, preview playback can be performed from the middle with any viewpoint other than the first registered viewpoint and the last registered viewpoint as the starting point. This makes it possible to shorten the animation confirmation time.

Further, in the present embodiment, each viewpoint can be called and its contents can be edited via the viewpoint icon displayed as a thumbnail in the area 22.

For example, as shown in FIG. 10B, when the user selects the "viewpoint 4" icon, the display of the area 20 is switched to a partial image centered on the "viewpoint 4". In response to this, the user can edit the content of the fourth viewpoint as shown in FIGS. 10 (b) and 10 (c) by manipulating the switched partial image. Although FIGS. 10 (b) and 10 (c) exemplify a case where the coordinates of the registered viewpoint are changed, other parameters associated with the viewpoint can be edited by calling the viewpoint in the same procedure. ..

Further, the viewpoint can be deleted by selecting the "Delete" button displayed in the area 23 after selecting the icon of any viewpoint. Further, after selecting the icon of any viewpoint, the "transition speed (described later)" can be changed by the slider displayed in the area 23.

Further, in the present embodiment, as shown in FIG. 11A, when the user selects the "detailed setting" button displayed in the area 23, the application screen transitions to the detailed setting screen shown in FIG. 11B. .. As shown in FIG. 11B, the detailed setting screen includes a region 26, a region 27, and a region 28.

In the area 26, a slider for setting the transition speed and a numerical input box for setting the transition time are associated with each registration viewpoint, and both are selectively selected by a radio button. It has become. As a result, the user can adjust the transition mode of the viewpoint by the transition speed or the transition time.

Here, the "transition speed" refers to the viewpoint along the path of interpolating between the first viewpoint in the three-dimensional model of the spherical image and the second viewpoint designated after the first viewpoint. The transition speed is meant, and the "transition time" means the time required for the viewpoint to start from the first viewpoint and reach the second viewpoint.

Further, in the area 26, a pull-down menu for selecting an easing curve is associated with each registered viewpoint so that the user can produce a transition mode of the viewpoint using the easing curve. Become. Here, the easing curve is a curve showing the time change of the speed of the animation, and when "linear" is set, it is an effect of transitioning between viewpoints at a constant speed, and when "easeIn" is set. Is an effect that makes a slow transition at the beginning and gradually accelerates to a constant speed, and when "easeOut" is set, it is the opposite effect of "easeIn", and when "easeInOut" is set, it becomes an effect. It will be a combination of "ease In" and "ease Out".

In this embodiment, each of the above-mentioned parameters is referred to as a "viewpoint parameter". In the present embodiment, by individually setting each viewpoint parameter for each viewpoint, it is possible to execute a dynamic animation display rich in variation.

On the other hand, in the area 27, an icon for selecting "overall setting" is displayed, and for the icon, a slider for setting the transition speed and a numerical input box for setting the transition time are displayed. , A pull-down menu for selecting the easing curve is associated. When "Global setting" is selected, each of the above parameters is applied to the entire animation from the first viewpoint to the last viewpoint, so that a sense of unity is created in the entire animation and natural behavior can be produced. .. The "overall setting" is suitable, for example, when you want to execute an animation display that keeps rotating in the same direction.

Further, in the area 28, an icon for selecting a viewpoint parameter preset (presets 1 to 5) is displayed. In the present embodiment, one or more sets of a representative viewpoint transition pattern (transition path) and a corresponding viewpoint parameter can be prepared in advance as preset data and stored in the storage area 108. When the user selects an icon to be displayed in the area 28, the preset data corresponding to the icon is read from the storage area 108 and applied to the spherical image read by the user to display the animation. It will be executed automatically. Also in this case, the icon for calling each viewpoint is displayed as a thumbnail in the area 22. In this case, the user may use the animation automatically executed based on the preset data as it is, or may edit an arbitrary viewpoint displayed in the area 22 to customize the animation. Further, in another embodiment, a set of viewpoint parameters corresponding to the animation display created by the user may be stored as a preset in the storage area 108 so that it can be reused.

Further, in the present embodiment, when the user selects the "save" button displayed in the area 23 as shown in FIG. 12 (a), the application screen transitions to the state shown in FIG. 12 (b). During this time, the animation for which the display method is created by the above procedure is cropped to the size of the display screen, converted into video data in a general-purpose video file format such as MPEG or AVI, and stored in the storage area 108. In another embodiment, the animation may be cropped in an arbitrary crop area other than the size of the display screen, or an arbitrary background sound may be combined with the animation.

The video converted to the general-purpose file format in this way can be easily played by a standard video playback application without a dedicated viewer for spherical images. FIG. 12 (c) shows how the converted video is played back in a standard video playback application. Therefore, according to the present embodiment, a moving image obtained by converting a partial image moving between viewpoints created based on a spherical image into a general-purpose video file format can be posted on a video posting site or shared on SNS. Can be done.

The method of using the dedicated application of the present embodiment has been described above, and subsequently, the content of specific processing executed by each of the above-mentioned means (see FIG. 4) of the information processing apparatus 100 will be described. Here, first, the contents of the processing executed by the information processing apparatus 100 at the time of viewpoint registration will be described with reference to the sequence diagram shown in FIG.

First, when the user performs an operation of registering the viewpoint according to the procedure described above (S1), in response to this, the display unit 102 reads and acquires the viewpoint parameters of the following (1) and (2) (S2). ..
(1) Currently set transition speed or transition time (2) Currently set easing curve

Next, the display unit 102 notifies the control unit 103 that the viewpoint registration has been requested by the user, using the parameter acquired in S2 as an argument (S3). In response to this, the control unit 103 orders the calculation unit 104 to calculate the spherical coordinates (θ, φ) of the viewpoint corresponding to the image coordinates (x, y) of the viewpoint specified by the user. The calculation result is acquired (S4), the calculation of the current display magnification (zoom magnification with respect to the initial angle of view) is ordered, and the calculation result is acquired (S5).

Next, the control unit 103 manages the viewpoint by combining the viewpoint parameters (1) and (2) acquired from the display unit 102 with the viewpoint coordinates (θ, φ) and the display magnification acquired from the calculation unit 104. Notify the unit 105 and request the registration of a new viewpoint (S6).

In response to this, the viewpoint management unit 105 newly generates the viewpoint information management table 500 shown in FIG. 14A, and registers each value notified from the control unit 103. As shown in FIG. 14A, the viewpoint information management table 500 includes a field 501 for storing the index of the registered viewpoint, a field 502 for storing the horizontal angle (θ) of the registered viewpoint, and the registered viewpoint. A field 503 for storing the elevation angle (φ), a field 504 for storing the display magnification, a field 505 for storing the transition speed, a field 506 for storing the transition time, and an easing curve are stored. The viewpoint management unit 105 stores each value (hereinafter referred to as viewpoint information) notified from the control unit 103 in S6 in the corresponding field.

Upon completing the registration of the viewpoint, the control unit 103 returns the thumbnail icon of the registered viewpoint to the display unit 102 (S7). In response to this, the display unit 102 displays the icon as a thumbnail in the area 22 (S8) to indicate to the user that the viewpoint registration is completed.

After that, every time the user performs the viewpoint registration operation, the above-mentioned processes S1 to S8 are repeatedly executed, and along with this, the viewpoints designated by the user are registered in the viewpoint information management table 500 in the order of their designation. FIG. 14B exemplifies the viewpoint information management table 500 after the six viewpoints are registered. The latest registration viewpoint is the end point at that time, and information (NONE) indicating that there is no value is stored in the fields 505 to 507 of the record corresponding to the end point. In the present embodiment, the viewpoint information management table 500 is stored in the storage area 108 in a state of being associated with the metadata of the spherical image.

Needless to say, the value of each field of the viewpoint information management table 500 is updated in response to the above-mentioned viewpoint editing operation. In addition to the use of the viewpoint information management table 500 as metadata for spherical images, the contents of the viewpoint information management table 500 can be registered as preset data by itself. Further, the viewpoint information management table 500 can be output to an external database in a state associated with the spherical image or by itself.

The process executed by the information processing apparatus 100 at the time of registering the viewpoint has been described above, but subsequently, the content of the process executed by the information processing apparatus 100 at the time of preview reproduction will be described with reference to the sequence diagram shown in FIG.

First, when the user performs the preview reproduction operation according to the procedure described above (S1), the display unit 102 notifies the control unit 103 that the user has requested the reproduction of the animation (S2).

In response to this, the control unit 103 requests the viewpoint information related to the corresponding spherical image from the viewpoint management unit 105, and the viewpoint management unit 105 stores the viewpoint information in the storage area 108 in association with the corresponding spherical image. The set of viewpoint information stored in the viewpoint information management table 500 is acquired (S3).

Next, the control unit 103 requests the viewpoint control unit 106 to start controlling the animation by using a set of viewpoint information related to the first registered viewpoint (start point) and the second registered viewpoint (end point) as arguments (S4). ).

In response to this, the viewpoint control unit 106 implements control for transitioning the viewpoint along a path that interpolates between the first registered viewpoint (start point) and the second registered viewpoint (end point). Specifically, the distance between the two registered viewpoints and the transition speed or transition time for transitioning from the first registered viewpoint (start point) to the second registered viewpoint (end point) (viewpoint parameter related to the first registered viewpoint). Based on the above, the displacement amount of the viewpoint for each frame (individual partial image constituting the animation) is calculated (S5). In S5, the displacement amount of the viewpoint is calculated on the premise that the viewpoint transitions by interpolating the shortest distance between the first registered viewpoint (start point) and the second registered viewpoint (end point).

When the calculation of the displacement amount of the viewpoint for each frame is completed, the viewpoint control unit 106 notifies the control unit 103 of the start of the animation (S6). After that, the viewpoint control unit 106 repeatedly notifies the control unit 103 of the displacement amount of the viewpoint for each frame until the value obtained by multiplying the number of frames by the frame period exceeds the transition time (S7).

On the other hand, in response to the notification of the displacement amount of the viewpoint from the viewpoint control unit 106, the control unit 103 requests the calculation unit 104 to perform the operation of moving the viewpoint, and as the calculation result, the displaced viewpoint is the center. Acquire a partial image (S9). In response to this, the control unit 103 passes the partial image acquired in S9 to the display unit 102 and requests the display to be updated (S10).

On the other hand, when the value obtained by multiplying the number of frames by the frame period exceeds the transition time, the viewpoint control unit 106 ends the animation from the first registered viewpoint (start point) to the second registered viewpoint (end point). Is notified to the control unit 103 (S8), and the process proceeds to the next transition process together with the post-processing. After that, the processes of S4 to S10 are repeatedly executed for the number of registered viewpoint sets (start point and end point), and when the number of animations reaches the total number of registered viewpoints, the control unit 103 indicates that the animation is completed. (S11).

By the above procedure, the control unit 103 executes an animation in which a plurality of partial images generated by the calculation unit 104 are connected in the order of viewpoint transition.

In the present embodiment, in addition to the interpolation of the viewpoints, it is preferable to interpolate the angle of view between the two registered viewpoints to make a transition. In this case, the viewpoint control unit 106 calculates the displacement amount of the angle of view for each frame from the difference in the display magnification between the two registered viewpoints, and the calculation unit 104 calculates the displacement amount of the angle of view for each frame based on the calculated displacement amount of the angle of view. Generates a partial image with a different angle of view. This enables smooth zoom-in / zoom-out expression in animation.

Further, the moving image data generation unit 107 converts a plurality of partial images generated by the calculation unit 104 into moving image data in a general-purpose moving image file format such as MPEG or AVI, and stores the plurality of partial images in the storage area 108.

As described above, according to the present embodiment, the region of interest of the spherical image can be presented in an easy-to-understand manner as a moving image without the user having to search for and select a viewpoint.

The first embodiment of the present invention has been described above, and then the second embodiment of the present invention will be described. In the following, the description of the parts common to the contents of the first embodiment will be omitted, and only the differences from the first embodiment will be described.

(Second Embodiment)
In the first embodiment described above, the viewpoint control unit 106 determines that the viewpoint transitions by interpolating the shortest distance between the first registered viewpoint (start point) and the second registered viewpoint (end point). Although the displacement amount of the viewpoint has been calculated, the second embodiment is different from the first embodiment in that the user can set the direction in which the viewpoint transitions as a viewpoint parameter.

In the second embodiment, as shown in FIG. 16A, when the user specifies a viewpoint in the partial image displayed in the area 20, the application screen transitions to the state shown in FIG. 16B, and the portion An icon 29 indicating that the viewpoint has been registered is displayed in the image.

The icon 29 includes four arrow-shaped direction indicating buttons for designating the vertical direction (tilt direction) and the horizontal direction (pan direction) of the partial image, and as shown in FIG. 16 (c), the user. In response to selecting one of the direction instruction buttons, the transition direction for transitioning to the next registered viewpoint is set. Although FIG. 16 exemplifies a direction instruction button for instructing four directions of up, down, left, and right, a transition direction such as eight directions may be set in more detail. In addition, the direction instruction button displays the direction of the arrow so that it faces the registered viewpoint when registering the second and subsequent viewpoints, and transitions the viewpoint from the previous viewpoint to the registered viewpoint. You may specify whether to do it.

17 (a) to 17 (d) show the correspondence between the selection of the direction instruction button in the icon 29 and the transition direction from the first registration viewpoint to the second registration viewpoint. In the second embodiment, by accepting the designation of the transition direction at the time of registering the viewpoint, as shown in FIG. 17, it is possible to set a detour route in addition to the shortest route for the transition route of the viewpoint. There will be a wide variety of animation effects.

In the second embodiment, the transition direction set at the time of viewpoint registration is registered in the viewpoint information management table as a viewpoint parameter. FIG. 14C shows the viewpoint information management table 550 created in the second embodiment. As shown in FIG. 14 (c), the viewpoint information management table 550 includes a field 508 for storing the transition direction in addition to the above-mentioned fields 501 to 507.

The second embodiment of the present invention has been described above, and subsequently, the third embodiment of the present invention will be described. In the following, the description of the parts common to the contents of the previous embodiment will be omitted, and only the differences will be described.

(Third Embodiment)
In the previous embodiment, (1) automatic execution of animation using preset data, (2) cropping of animation, and (3) composition of animation and background sound have been schematically described, but in the third embodiment, The above (1) to (3) will be specifically described based on other implementation examples.

FIG. 18A shows the preset data 600 of this embodiment. The preset data 600 of this embodiment is registered in order of two or more viewpoints in the spherical image, and is data that plays the same role as the viewpoint information management table 500 (see FIG. 14) in the generation of animation. be. In this embodiment, it is assumed that one or more preset data 600s are held in the storage area 108.

As shown in FIG. 18A, the preset data 600 includes a field 601 for storing the index of the registered viewpoint, a field 602 for storing the horizontal angle (θ) of the registered viewpoint, and an elevation angle of the registered viewpoint (as shown in FIG. 18A). It is configured to include a field 603 for storing φ) and a field 604 for storing a display magnification (zoom magnification with respect to the initial angle of view). In this embodiment, when the preset data 600 is used, a common "transition speed" and "easing curve" are applied to the entire viewpoint as in the "overall setting" described above, so that the preset is used. The "transition speed" and the "easing curve" are not registered in the data 600.

Further, in the present embodiment, it is assumed that one or more "cropped areas" are held in the storage area 108. The "crop area" here is a rectangular frame for defining an area for cropping the generated animation image, and in the present embodiment, a rectangular frame having an arbitrary aspect ratio is registered as a "crop area". be able to.

Further, in the present embodiment, it is premised that one or more "background sounds" are held in the storage area 108. The "background voice" referred to here is arbitrary voice data, and in a preferred embodiment, the user may be able to register the desired voice data as the "background voice".

FIG. 19A shows an application screen in this embodiment. As shown in FIG. 19A, in the present embodiment, a "preset" button for recalling preset data is displayed in the area 23 of the application screen, and when the user selects the "preset" button, the application screen is displayed. The screen transitions to the preset usage screen shown in FIG. 19 (b).

As shown in FIG. 19B, the preset usage screen of the present embodiment includes a region 40, a region 42, a region 43, and a region 44. In the present embodiment, a partial image of the spherical image is displayed in the area 40. Further, in the area 43, a "preset" button for calling the preset data 600, a "sound" button for calling the background sound, a "crop" button for calling the crop area, and a transition speed are set. "Transition speed" button is displayed. Further, in the area 44, a "preview" button for requesting preview playback and a "save" button for saving the generated moving image are displayed. The area 42 is an area in which the UI corresponding to each button displayed in the area 43 is displayed.

In the present embodiment, when the user selects the "preset" button, as shown in FIG. 20 (a), the icon corresponding to the preset data 600 held in the storage area 108 is selectively displayed in the area 42. .. In response to this, when the user selects any of the icons, the preset data 600 corresponding to the icon is read from the storage area 108 and set in the temporary storage. In the example shown in FIG. 20A, an image in which the contents of the preset data 600 are abstracted is displayed on each icon.

Further, when the user selects the "sound" button, as shown in FIG. 20 (b), an icon corresponding to the background sound held in the storage area 108 is displayed in the area 42 so as to be selectable. When the user selects any of the icons, the corresponding voice data is read from the storage area 108 and set as the background voice. If the user does not select the background sound, the default value (silence) is set.

Further, when the user selects the "transition speed" button, as shown in FIG. 20 (c), a slider for setting the transition speed is displayed in the area 42. When the user operates this slider to specify the transition speed, the specified transition speed is set as the transition speed of the viewpoint when generating the animation. If the user does not specify the transition speed, a default value (predetermined transition speed) is set. A default value (for example, "linear") is always set for the easing curve.

Further, when the user selects the "crop" button, as shown in FIG. 21A, an icon corresponding to the crop area held in the storage area 108 is selectively displayed in the area 42. In response to this, when the user selects any of the icons, the corresponding rectangular pattern is read out from the storage area 108 and set as the crop area.

For example, if the user selects "2: 1" as the crop area, the animation will be cropped in the manner shown in FIG. 21 (b). Further, when the user selects "free" as the crop area, a rectangular frame 45 is displayed on the area 40 as shown in FIG. 21 (c). Here, the size of the frame 45 is variable, and the user can freely set the crop area by manually changing the size of the frame 45 arbitrarily. If the user does not select the crop area, a default value (for example, the size of the display screen) is set.

In the present embodiment, the user can reset each of the above-mentioned values (“preset data”, “background sound”, “transition speed”, “crop area”) as many times as necessary. It can be previewed every time.

For example, as shown in FIG. 22A, when the user selects the icon Q displayed in the area 42 and then selects the “preview” button, the control unit 103 responds to the viewpoint management unit 105. The preset data 600 corresponding to the icon Q is acquired from the storage area 108 via the storage area 108. After that, based on the set of the viewpoint and the display magnification (angle of view) stored in the acquired preset data 600, the control unit 103, the viewpoint control unit 106, the calculation unit 104, and the display unit 102 cooperate to perform preview playback (animation). Display) is executed. The content of the process executed during preview playback is substantially the same as the content described with reference to FIG. 15, except that the preset data 600 is used instead of the viewpoint information management table 500. Omit.

FIG. 22B shows a screen at the time when the preview reproduction is completed. At this time, the final frame image of the animation is projected in the area 40 in a stopped state. Here, when considering publishing this animation to a video posting site or SNS and receiving an evaluation, the impression of the final frame image that the viewer will see last may determine the evaluation of the animation. There is. In addition, the final frame image is often used as the thumbnail of the published animation.

Here, when the preset data 600 is used, the final frame image of the animation is determined by the viewpoint and the angle of view registered in the last order of the preset data 600, but the content of the final frame image thus accidentally determined. Does not always meet the user's wishes. For example, if the entire animation is very interesting, but the final frame image happens to be a featureless image, it can leave a negative impression on the viewer and the featureless image. The thumbnails may not be enough to attract the viewer's interest.

Regarding this point, in the present embodiment, as shown in FIG. 22B, the end point adjustment button 50 is displayed in the area 40 at the end of the preview reproduction. By selecting the end point adjustment button 50, the user can change the final frame image of the animation to a desired content.

When the user selects the end point adjustment button 50, the display unit 102 operates the area 40 on the user's screen in order to receive the change of the viewpoint in the last order registered in the preset data 600 and the angle of view associated with the viewpoint. Transition to the state of accepting (pan / tilt / zoom in / zoom out). In response to this, when the user operates the area 40 to specify a new viewpoint and angle of view as shown in FIG. 22 (c), the control unit 103 sets the preset in the temporary storage at that time. The viewpoint in the last order registered in the data 600 and the angle of view associated with the viewpoint are replaced with the viewpoint and the angle of view newly specified by the user and updated. FIG. 18B shows how the coordinates of the viewpoint 6 in the last order registered in the preset data 600 and the value of the display magnification (angle of view) are replaced. At this time, the icon 21 indicating that the new viewpoint has been registered is displayed in the area 40. The process executed when the user selects the end point adjustment button 50 is the process executed when the viewpoint icon is selected and the contents of the viewpoint are edited in the above embodiment (see FIG. 10). Since they are substantially the same, detailed explanations are omitted here.

When the user selects the "preview" button at this timing, an animation is generated based on the updated preset data 600. That is, in this case, the viewpoint control unit 106 changes the viewpoint along the route interpolated based on the changed last-order viewpoint, and the arithmetic unit 104 changes the viewpoint along the changed route and the changed viewpoint. A partial image is generated based on the angle of view in the last order, and the control unit 103 displays an animation in which the plurality of partially images generated in this way are joined together in the area 40. At this time, the control unit 103 crops the animation based on the crop area set at that time, and if the background sound is set at that time, reproduces the background sound in synchronization with the animation.

On the other hand, when the user selects the "Save" button at this timing, the video data generation unit 107 converts a plurality of partial images created by the same procedure as described above into video data in a general-purpose video file format. Then, it is stored in the storage area 108. At this time, the moving image data generation unit 107 crops the animation based on the crop area in which the crop area is set at that time, and if the background sound is set at that time, the background sound is used as the moving image data. Synthesize.

As described above, according to the present embodiment, it is possible to easily generate a moving image that presents the region of interest of the spherical image by using the preset data prepared in advance. Further, according to the present embodiment, even when the preset data is used, the end of the moving image can be concluded with the intended image.

The third embodiment of the present invention has been described so far, and then the fourth embodiment of the present invention will be described. In the following, the description of the parts common to the contents of the previous embodiment will be omitted, and only the differences will be described.

(Fourth Embodiment)
From the first embodiment to the third embodiment described above, many video expressions are possible by changing parameters such as the viewpoint and the angle of view in the spherical image. However, the parameters are not limited to these, and by adjusting parameters other than these, for example, the projection method (projection type) can be changed, and a dynamic (dynamic) image expression becomes possible. ..

The projection type is a method of expressing (mapped) an all-sky image attached (mapped) to the surface of a three-dimensional object (three-dimensional object) (for example, the inner surface of a sphere) on a two-dimensional plane, similar to the map projection method. It is something to do. Before explaining the projection type, the parameters for changing the video expression will be described.

The spherical image is mapped to the surface of a predetermined 3D object. FIG. 23 is a diagram showing a sphere 200 on a two-dimensional plane represented by the Y-axis and the Z-axis in a three-dimensional space represented by the three axes of the X-axis, the Y-axis, and the Z-axis. The Y-axis is in the height direction (vertical direction), and the Z-axis is one in the horizontal direction parallel to the ground. In FIG. 23, the center of the sphere 200 is the origin 201, and a camera 202 that captures and acquires the partial image in order to generate a partial image is arranged outside the sphere 200 on the Z axis passing through the origin 201. .. These spheres 200 and cameras 202 are virtual.

As described in the first to third embodiments, the parameters include a viewpoint designated as a desired position in the partial image and an angle of view α indicating the angle of the viewing range of the camera 202.

Other parameters include camera coordinates, which are information indicating the position of the camera 202 as shown in FIG. 23, camera UP, which is information indicating the upward direction of the camera 202 and expressing the upward direction as a vector, and camera 202. There are camera gaze point coordinates, which are information indicating the position (gaze point), and the radius of the sphere 200, which is a three-dimensional object that maps the spherical image.

These parameters are examples and may include other parameters. It should be noted that these parameters are necessary parameters for changing the projection type.

In the example shown in FIG. 23, the radius of the sphere 200 is 1, the gazing point is the origin 201 of the sphere 200, the angle of view α is 120 °, the camera coordinates are (0,0, -1.8), and the camera UP is (0, It is set to 1,0).

Actually, what kind of projection type is available will be described with reference to FIGS. 24 to 26. The projection types shown in FIGS. 24 to 26 are examples, and are not limited to these types. Hereinafter, as in the example shown in FIG. 23, the whole celestial sphere image is assumed to be mapped to the surface of the sphere 200, the radius of the sphere 200 is 1, the center of the sphere 200 is the origin 201, and the Y-axis and the Z-axis. This will be described using a diagram representing the sphere 200 on a two-dimensional plane represented by.

FIG. 24 shows a projection type (first type) that expresses a state in which the camera 202 is arranged outside the sphere 200 and viewed. The first type can express an image in which a wide range of images (panoramic images) are attached to the surface of a mirror ball.

In the first type, as shown in FIG. 24A, the gazing point is the origin 201 indicating the center of the sphere 200, and the camera 202 is on the Z axis outside the sphere 200 and is larger than the radius of the sphere 200 from the origin 201. For example, it is located at a distance of about 1.8, and the angle of view is, for example, 120 °. Therefore, as shown in FIG. 24B, the parameters are an angle of view of 120 °, camera gazing point coordinates (0,0,0), camera coordinates (0,0, -1.8), and camera UP (0). , 1,0).

FIG. 25 shows a projection type (second type) in which the Y coordinate of the camera 202 is made equal to the radius of the sphere 200 and the entire inside of the sphere 200 is viewed from above. The second type can express an image of a person or a building standing on a small planet.

In the second type, as shown in FIG. 25 (a), the camera 202 is placed inside the sphere 200 as close to the boundary as possible, the angle of view is set to an ultra-wide angle (170 ° or more), and the gazing point is set to the origin 201. do. Therefore, as shown in FIG. 25 (b), the parameters have, for example, an angle of view of 170 °, the camera gazing point coordinates (0,0,0), the camera coordinates (0,1,0), and the camera UP ( It is set to 0, 0, 1).

FIG. 26 shows a projection type (third type) in which the camera 202 is arranged at the origin 201 and the sphere 200 is viewed from the origin 201. The third type can express an image stretched radially from the center toward the periphery.

In the third type, as shown in FIG. 26A, the camera 202 is arranged at the origin 201, and the point of interest is the intersection of the circumference representing the surface of the sphere 200 and the Z axis. Therefore, as shown in FIG. 26 (b), the parameters have, for example, an angle of view of 205 °, the camera gazing point coordinates (0,0,1), the camera coordinates (0,0,0), and the camera UP ( 0,1,0).

From the first embodiment to the third embodiment, the projection type is registered as the first type, for example, the viewpoint 1, the viewpoint 2, ..., And the two registered viewpoints are connected to execute preview playback as an animation. There is. Therefore, the projection type is the same from all viewpoints. Further, since the gazing point is also the origin 201, it is not considered as a parameter.

In the fourth embodiment, each viewpoint and each gaze point are registered with different projection types, such as the first type for the viewpoint 1 and the second type for the viewpoint 2, and the transition of the viewpoint and the gaze point is performed. Perform preview playback so that the projection type is changed accordingly.

A method of selecting a projection type will be described with reference to FIG. 27. The user selects a file of the spherical image to be displayed, displays a partial image of an arbitrary projection type, and then performs operations such as pan / tilt / zoom in / zoom out on the displayed partial image area 20. If necessary, select the "type selection" button displayed in the area 23 of the display screen (touch panel) of the smartphone shown in FIG. 27 (a), and select the projection type shown in FIG. 27 (b). Switch to the screen to select.

The user selects one of the "first type", "second type", and "third type" buttons displayed in the screen area 46 shown in FIG. 27 (b). Any projection type is a projection type set by default, and the user can select a desired projection type when he / she wants to register a viewpoint with a projection type other than any projection type. In the example shown in FIG. 27 (b), the "second type" is selected. After selection, the screen is switched to the selected projection type partial image, and the button 51 for returning to the original screen above the area 20 where the partial image is displayed is pressed to return to the screen shown in FIG. 27 (a). Can be done.

In the information processing apparatus 100 of the present embodiment, the display unit 102 accepts the selection of the projection type according to the operation of the user. The display unit 102 switches and updates the currently displayed partial image to the selected projection type partial image. If the projection type is not selected and the projection type is not changed, this process is not executed.

Specifically, the display unit 102 accepts the selection of the projection type, and is selected from the parameters as projection information as shown in FIGS. 24 (b), 25 (b), and 26 (b). Acquires the parameters associated with the type for projection by the projection type, uses the acquired parameters to generate a partial image of the selected projection type, and the part generated from the currently displayed partial image. Switch to image.

After that, the user designates a desired position in the partial image displayed in the area 20 as the viewpoint 1 by a predetermined operation such as a long press or a double tap, as in the first to third embodiments. When designating the viewpoint 1, the display magnification may be changed by zooming in or out. In response to this, the application screen displays an icon indicating that the viewpoint 1 has been registered at a position specified by the user in the partial image, and displays a thumbnail of an icon for calling the viewpoint 1 in the area 22. ..

Subsequently, the user selects the "view selection" button displayed in the area 23, switches the screen, and displays the "first type", "second type", in the area 46, if necessary. Select one of the "third type" buttons. The display unit 102 switches and updates the currently displayed partial image to the selected projection type partial image. Again, if the projection type is not selected and the projection type does not change, this process will not be executed.

After that, the user designates a desired position in the displayed partial image as the viewpoint 2. When designating the viewpoint 2, the display magnification may be changed by zooming in or out. In response to this, the application screen displays an icon indicating that the viewpoint 2 has been registered at a position specified by the user in the partial image, and displays a thumbnail of an icon for calling the viewpoint 2 in the area 22. do.

By repeating such an operation, the viewpoint 3, the viewpoint 4, ... Can be added after the viewpoint 1 and the viewpoint 2. Also in this case, when the number of registered viewpoints is two or more, the two registered viewpoints can be connected and previewed as an animation.

Here, the content of the process executed by the information processing apparatus 100 at the time of viewpoint registration will be described with reference to the sequence diagram shown in FIG. 28. The user selects the projection type before performing the operation of registering the viewpoint (S1). In response to this, the display unit 102 acquires the parameter associated with the selected projection type.

The parameters are values such as camera coordinates used to display the partial image set for each projection type, and are managed by the viewpoint management unit 105 using a table. Specifically, the table is a table as shown in FIGS. 24 (b), 25 (b), and 26 (b). Therefore, the display unit 102 notifies the viewpoint management unit 105 of the selected projection type, and acquires the parameter associated with the notified projection type. This parameter is an initial value (default value) such as preset camera coordinates.

When the selected projection type is the first type and the camera coordinates, the angle of view, the camera gaze point coordinates, and the camera UP are acquired as parameters, the display unit 102 displays an all-sky image on the surface of the sphere 200 having the radius. Map and place the camera 202 at the camera coordinates, set the direction of the camera UP to the upward direction of the camera 202, generate a partial image focused on the camera gaze point coordinates at the angle of view, and display it now. Switch to the generated partial image from the existing partial image and update.

Since the operation from the operation of registering the viewpoint by the user (S3) to the acquisition of the calculation result of the current display magnification (S7) is the same as the processing from S1 to S5 shown in FIG. 11, the description thereof is omitted here. In the operation of registering the viewpoint, the viewpoint is changed and the display magnification is also changed. Therefore, the viewpoint coordinates are calculated in S6 and the display magnification is calculated in S7. In the operation of registering the viewpoint, the upward direction of the camera 202 and the camera coordinates change due to the change of the viewpoint, etc., so these must also be calculated.

Therefore, the control unit 103 instructs the calculation unit 104 to calculate the camera UP, and acquires the calculation result (S8). The camera UP is calculated based on the information in the upward direction of the spherical image when designating the viewpoint, the viewpoint coordinates (spherical coordinates of the viewpoint) calculated in S6, and the camera gazing point coordinates. The camera gazing point coordinates do not need to be calculated because they are the coordinates (0,0,0) of the origin 201 of the sphere 200 for the first type and the second type, but the origin for the third type. It is calculated as the coordinates of the point where the straight line extending in the viewing direction of the camera 202, which is determined from 201 based on the viewpoint coordinates, intersects with the circumference representing the surface of the sphere 200. The camera UP is a direction perpendicular to a straight line extending in the viewing direction of the camera 202, and is calculated as a vector in which the upward direction of the image is the upward direction of the camera. If it is necessary to calculate the camera gaze point coordinates, the calculation can be ordered here and the calculation result can be obtained.

The control unit 103 commands the calculation unit 104 to calculate the camera coordinates, and acquires the calculation result (S9). The camera coordinates are calculated based on the viewpoint coordinates calculated in S6, the display magnification calculated in S7, and the camera gazing point coordinates. The display magnification changes the angle of view according to the magnification, and changes the focal length that determines the distance from the gazing point to the camera 202 according to the angle of view. Therefore, the camera coordinates can be calculated from, for example, the distance from the gazing point to the camera 202 from the display magnification, and the viewing direction and the distance of the camera 202 determined based on the viewpoint coordinates. Since the calculation method exemplified here is an example, it is not limited to these methods as long as the camera UP and the camera coordinates can be calculated.

After that, since it is the same as the processing from S6 to S8 shown in FIG. 11, the description thereof is omitted here. In the parameter registration (S10), the viewpoint management unit 105 registers each value in the viewpoint information management table, but the viewpoint information management table in the fourth embodiment has the selected projection type, camera UP, camera coordinates, and camera. Contains a field for storing the gaze point coordinates, and stores the selected type and each calculated value in the corresponding field. The angle of view may be stored as a display magnification which is a zoom magnification with respect to the initial angle of view, or the value of the angle of view may be stored separately from the display magnification.

After that, each time the user performs the viewpoint registration operation, the processes S1 to S12 are repeatedly executed, and along with this, the viewpoints designated by the user are registered in the viewpoint management table in the order specified.

Next, after registering the viewpoints, the transition of the viewpoints during preview reproduction between two registered viewpoints having different projection types will be described with reference to FIGS. 29 to 31.

FIG. 29 is a diagram illustrating a transition from the viewpoint 1 set in the first type to the viewpoint 2 set in the second type. In FIG. 29 (a), "A" is in the same state as in FIG. 24 (a) showing the first type, and "C" is in the same state as in FIG. 25 (a) showing the second type. be. "B" is a state in the middle of transition. In FIGS. 29 to 31, in order to facilitate the explanation, the viewpoint in “B” is assumed to be located between the viewpoint 1 in “A” and the viewpoint 2 in “C”.

When transitioning from the "A" state to the "C" state, the gazing point is at the same position (origin 201) in each state, but the angle of view, camera coordinates, and camera UP change. Therefore, the viewpoint is transitioned along the path of interpolating between the viewpoints, and the angle of view, the camera coordinates, and the camera UP are also interpolated and transitioned between the viewpoints.

Since the parameters in the states of "A" and "C" are the same as the parameters shown in FIGS. 24 (b) and 25 (b), the description thereof will be omitted here. The angle of view in the state during the transition is calculated by the viewpoint control unit 106 using the angle of view in the states of "A" and "C". Specifically, the viewpoint control unit 106 calculates the displacement amount of the angle of view for each frame based on the difference between the angle of view at the viewpoint 1 and the angle of view at the viewpoint 2.

As shown in FIG. 29 (b), the angle of view at viewpoint 1 in the "A" state is 120 °, and the angle of view at viewpoint 2 in the "C" state is 170 °, so the difference is 50 °. be. The angle of view at the time of transition to the "B" state is calculated by adding the sum of the displacement amounts of the angle of view of the number of frames from the viewpoint 1 to the intermediate position to the angle of view at the viewpoint 1. In this example, it is calculated as 145 ° by adding 25 °, which is the sum of the displacement amounts, which is half of the difference of 50 °, to the angle of view of 120 ° at the viewpoint 1.

The camera coordinates in the state during the transition are calculated by the viewpoint control unit 106 using the camera coordinates in the states of "A" and "C". Specifically, the viewpoint control unit 106 calculates the displacement amount of the camera coordinates for each frame based on the distance between the camera coordinates at the viewpoint 1 and the camera coordinates at the viewpoint 2.

The coordinates when the state transitions to the "B" state are calculated by adding the sum of the displacement amounts of the camera coordinates of the number of frames from the viewpoint 1 to the intermediate position to the camera coordinates at the viewpoint 1. In this example, the distance 1 in the Y-axis direction and the distance 1. It is calculated as (0, 0.5, -0.9) by adding 0.5 and 0.9, which are half of 8, for the y-coordinates and z-coordinates to be displaced. Both are added because the y-coordinate and the z-coordinate transition in the direction of increasing values from the viewpoint 1 to the viewpoint 2.

The camera UP in the state during the transition is calculated by the viewpoint control unit 106 using the camera UP in the states of "A" and "C". Specifically, the viewpoint control unit 106 calculates the displacement amount of the camera UP for each frame based on the rotation angles obtained from the camera UP at the viewpoint 1 and the camera UP at the viewpoint 2.

The camera UP at the time of transition to the "B" state adds, subtracts, or both the sum of the displacement amounts of the camera UP of the number of frames from the viewpoint 1 to the intermediate position to the camera UP at the viewpoint 1. It is calculated by. As shown in FIG. 29 (b), since the camera UP at the viewpoint 1 is (0,1,0) and the camera UP at the viewpoint 2 is (0,0,1), the upward direction of the camera 202 is the Y axis. It is shown that the camera is rotated 90 ° in the Z-axis direction from the direction. In this example, 0.5, 0.5, which corresponds to 45 °, which is half of the rotation angle of 90 °, which is the sum of the displacement amounts, is subtracted from the camera UP (0,1,0) for the y-coordinate to be displaced. It is calculated as (0, 0.5, 0.5) by adding each z coordinate. The reason why the y-coordinate is subtracted and the z-coordinate is added is that the y-coordinate transitions from the viewpoint 1 to the viewpoint 2 in the direction in which the y-coordinate is small and the value is large.

FIG. 30 is a diagram illustrating a transition from the viewpoint 1 set in the second type to the viewpoint 2 set in the third type. In FIG. 30 (a), "A" is in the same state as in FIG. 25 (a) showing the second type, and "C" is in the same state as in FIG. 26 (a) showing the third type. be. "B" is a state in the middle of transition.

When transitioning from the "A" state to the "C" state, the angle of view, camera coordinates, camera UP, and camera gazing point coordinates all change. Therefore, the viewpoint is transitioned along the path of interpolating between the viewpoints, and the angle of view, the camera coordinates, the camera UP, and the camera gazing point coordinates are also interpolated and transitioned between the viewpoints.

Since the parameters in the states of "A" and "C" are the same as the parameters shown in FIGS. 25 (b) and 26 (b), the description thereof will be omitted here. The angle of view in the state during the transition is calculated by the viewpoint control unit 106 by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 29 using the angle of view in the states of “A” and “C”.

In this example, the angle of view when transitioning to the "B" state is 187. The angle of view at viewpoint 1 is 170 ° plus 17.5 °, which is the sum of the displacements, which is half of the difference of 35 °. It is calculated as 5 °.

The camera coordinates in the state during the transition are calculated by the viewpoint control unit 106 by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 29 using the camera coordinates in the states of “A” and “C”.

In this example, the camera coordinates when the state transitions to the "B" state are the sum of the displacement amounts to the camera coordinates (0, 1, 0) of the viewpoint 1, and the Y-axis direction between the viewpoint 1 and the viewpoint 2. It is calculated as (0,0.5,0) by subtracting 0.5, which is half of the distance 1 of, for the y-coordinate to be displaced.

The camera UP in the state during the transition is calculated by the viewpoint control unit 106 by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 29 using the values of the camera UP in the states of “A” and “C”.

As shown in FIG. 30B, since the camera UP at the viewpoint 1 is (0,0,1) and the camera UP at the viewpoint 2 is (0,1,0), the upward direction of the camera 202 is the Z axis. It is shown that the camera is rotated 90 ° in the Y-axis direction from the direction. In this example, 0.5 and 0.5, which are the sum of the displacement amounts and correspond to 45 °, which is half of the rotation angle of 90 °, are added to the camera UP (0, 0, 1) for the y-coordinates to be displaced. It is calculated as (0,0.5,0.5) by subtracting the z-coordinates.

The camera gazing point coordinates in the state during the transition are calculated by the viewpoint control unit 106 using the camera gazing point coordinates in the states of "A" and "C". Specifically, the viewpoint control unit 106 calculates the displacement amount of the gazing point for each frame based on the distance between the gazing point 1 at the viewpoint 1 and the gazing point 2 at the viewpoint 2.

The camera gazing point coordinates at the time of transition to the "B" state are the sum of the displacement amounts to the camera gazing point coordinates (0, 0, 0) at the viewpoint 1, and the gazing point at the viewpoint 1 and the gazing point at the viewpoint 2. 0.5, which is half of the distance 1 in the Z-axis direction, is added for each z coordinate to be displaced, and calculated as (0, 0, 0.5).

FIG. 31 is a diagram illustrating a transition from the viewpoint 1 set in the third type to the viewpoint 2 set in the first type. In FIG. 31 (a), "A" is in the same state as in FIG. 26 (a) showing the third type, and "C" is in the same state as in FIG. 24 (a) showing the first type. be. "B" is a state in the middle of transition.

When transitioning from the "A" state to the "C" state, the camera UP is the same in all states, but the angle of view, camera coordinates, and gazing point change. Therefore, the viewpoint is changed along the path of interpolating between the viewpoints, and the angle of view, the camera coordinates, and the camera gazing point coordinates are interpolated between the viewpoints to make the transition.

Since the parameters in the states of "A" and "C" are the same as the parameters shown in FIGS. 26 (b) and 24 (b), description thereof will be omitted here. The angle of view in the state during the transition is calculated by the viewpoint control unit 106 by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 29 using the angle of view in the states of “A” and “C”.

In this example, the angle of view when transitioning to the "B" state is 162. The angle of view at viewpoint 1 is obtained by subtracting 42.5 °, which is the sum of the displacements, which is half of the difference of 85 °. Calculated as 5 °.

The camera coordinates in the state during the transition are calculated by the viewpoint control unit 106 by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 29 using the camera coordinates in the states of “A” and “C”.

In this example, the camera coordinates when the state transitions to the "B" state are the sum of the displacement amounts of the camera coordinates (0, 0, 0) of the viewpoint 1, and the Z-axis direction between the viewpoint 1 and the viewpoint 2. It is calculated as (0,0, -0.9) by subtracting 0.9, which is half of the distance of 1.8.

The camera gazing point coordinates in the state in the middle of the transition are calculated by the viewpoint control unit 106 by the same method as the method described with reference to FIG. 30 using the camera gazing point coordinates in the states of “A” and “C”. The coordinates.

In this example, the camera gazing point coordinates at the time of transition to the "B" state are the camera gazing point coordinates (0, 0, 1) at the viewpoint 1, and the gazing point at the viewpoint 1 and the gazing point at the viewpoint 2. It is calculated as (0,0,0.5) by subtracting 0.5, which is half of the distance 1 in the Z-axis direction, for each z coordinate.

With reference to FIGS. 29 to 31, an example of transitioning from the first type to the second type, from the second type to the third type, and from the third type to the second type has been described. In the same way, when transitioning from the second type to the first type, from the third type to the second type, from the first type to the third type, etc., the angle of view for each frame, A partial image in which the camera coordinates, the camera UP, and the camera gaze point coordinates are changed is generated, and smooth viewpoint, gaze point, camera UP transition, zoom-in / zoom-out expression can be performed in the animation. In addition, by changing the type, dynamic video expression using spherical images becomes possible.

Here, the contents of the processing executed by the information processing apparatus 100 at the time of preview reproduction will be described with reference to the sequence diagram shown in FIG. 32. The processing at the time of preview reproduction is substantially the same as the processing in the first embodiment shown in FIG. 13, but only the processing in S5 and S7 is different, so only the processing in S5 and S7 will be described here.

In S5, the viewpoint control unit 106 controls to transition the viewpoint along a path that interpolates between the first registered viewpoint (start point) and the second registered viewpoint (end point). In the control, the displacement amount of the viewpoint for each frame is calculated as described above, and the angle of view, camera coordinates, camera UP, and camera gazing point coordinates for each frame are calculated according to the projection types registered at the start point and the end point. Calculate the displacement of.

In S7, the viewpoint control unit 106 registers the displacement amount of the viewpoint, the start point and the end point for each frame with respect to the control unit 103 until the value obtained by multiplying the number of frames by the frame period exceeds the transition time. It repeats notifying the displacement amount of the angle of view, the camera coordinates, the camera UP, and the camera gaze point coordinates according to the type. At this time, it is also possible to notify the displacement direction, that is, whether to add or subtract.

By changing the projection type between the registered viewpoints in this way, it is possible to express various spherical images that cannot be expressed only by changing the viewpoint or the angle of view.

Also in the fourth embodiment, the user can set the direction in which the viewpoint described in the second embodiment changes as a viewpoint parameter, and the viewpoint can be changed in the set direction. Further, also in the fourth embodiment, it is possible to automatically execute the animation using the preset data described in the third embodiment, crop the animation, and synthesize and reproduce the animation and the background sound.

Each function of the above-described embodiment can be realized by a program described in C, C ++, C #, Java (registered trademark), etc., and the program of this embodiment is a hard disk device, CD-ROM, MO, DVD, flexible. It can be stored and distributed on recording media such as disks, EEPROMs, and EPROMs, and can be transmitted over a network in a format that other devices can.

Although the present invention has been described above with embodiments, the present invention is not limited to the above-described embodiments, and as long as the present invention exerts its actions and effects within the range of embodiments that can be inferred by those skilled in the art. , Is included in the scope of the present invention.

10 ... Processor 12 ... ROM
13 ... RAM
14 ... Auxiliary storage 15 ... Input / output interface 16 ... Network interface 18 ... Display 20 ... Area 21,24,25,29 ... Icon 22,23,26,27,28 ... Area 40,42,43,44,46 ... ... Area 50 ... End point adjustment button 51 ... Back button 30 ... Playback start icon 100, 100a, 100b, 100c ... Information processing device 102 ... Display unit 103 ... Control unit 104 ... Calculation unit 105 ... Viewpoint management unit 106 ... Viewpoint control unit 107 ... Movie data generation unit 108 ... Storage area 110 ... Image pickup device 200 ... Sphere 201 ... Origin 202 ... Camera 500 ... Viewpoint information management table 501,502,503,504,505,506,507,508 ... Field 550 ... Viewpoint information management Table 600 ... Preset data 601,602,603,604 ... Field

JP-A-2015-18013

Claims (26)

In the spherical image, the viewpoint management unit that registers the viewpoints specified by the user in the specified order,
A viewpoint control unit that shifts the viewpoint along a path that interpolates between the first viewpoint and the second viewpoint designated after the first viewpoint in the spherical image.
An arithmetic unit that generates a partial image centered on each transition viewpoint,
Display and
On the display screen of the display unit, a control unit that executes an animation display in which a plurality of generated partial images are connected in the order of transition of viewpoints, and a control unit.
Including
The display unit is
Accepts the selection of the projection method when projecting the partial image on the display screen and displaying it.
The viewpoint management unit
Register the viewpoint specified by the user and the projection information for projection by the selected projection method in association with each other.
The arithmetic unit
Generate the partial image based on the projection information associated with the viewpoint specified by the user.
Information processing equipment. The viewpoint management unit
Register the viewpoint specified by the user and the angle of view specified by the user in association with each other.
The arithmetic unit
Generate the partial image based on the angle of view associated with the viewpoint specified by the user.
The information processing apparatus according to claim 1. The viewpoint management unit
Register the transition speed or transition time of the viewpoint specified by the user and the viewpoint specified by the user in association with each other.
The viewpoint control unit
Transitioning the viewpoint based on the transition speed or the transition time specified by the user.
The information processing apparatus according to claim 1 or 2. The viewpoint management unit
Register the transition direction of the viewpoint specified by the user and the viewpoint specified by the user in association with each other.
The viewpoint control unit
The viewpoint is transitioned along the path determined based on the transition direction specified by the user.
The information processing apparatus according to any one of claims 1 to 3. Includes a video data generator that converts the generated plurality of partial images into video data.
The information processing apparatus according to any one of claims 1 to 4. The display unit displays thumbnails of icons corresponding to the viewpoint specified by the user on the display screen .
The information processing apparatus according to claim 5. The display unit is
Accepts editing of the viewpoint specified by the user and information registered in association with the viewpoint.
The information processing apparatus according to claim 6. The display unit is
Display one or more background sounds in a selectable manner,
The moving image data generation unit is
Synthesize the background sound selected by the user with the moving image data.
The information processing apparatus according to any one of claims 5 to 7. The display unit is
One or more crop areas are displayed selectably,
The moving image data generation unit is
An animation in which the plurality of partial images are stitched together is cropped based on the crop area selected by the user.
The information processing apparatus according to any one of claims 5 to 8. The information processing device is
Holds 1 or more preset data registered in order of 2 or more viewpoints in the spherical image,
The display unit is
One or more of the preset data can be displayed selectably,
The viewpoint control unit
The viewpoint is changed along a path that interpolates between the first viewpoint registered in the preset data selected by the user and the second viewpoint in the next order of the first viewpoint.
The information processing apparatus according to any one of claims 1 to 9. The preset data includes an angle of view registered in association with each viewpoint.
The arithmetic unit
The partial image is generated based on the angle of view associated with the viewpoint registered in the preset data selected by the user.
The information processing apparatus according to claim 10. The display unit is
Accepts changes in the last-order viewpoint registered in the preset data selected by the user and the angle of view associated with the viewpoint.
The viewpoint control unit
When the last-order viewpoint is changed, the viewpoint is changed along the interpolated path based on the changed last-order viewpoint.
The arithmetic unit
When the angle of view associated with the last-order viewpoint is changed, the partial image is generated based on the changed angle of view.
The information processing apparatus according to claim 11. The viewpoint management unit
The projection information of the projection method different between the first viewpoint and the second viewpoint is linked and registered.
The arithmetic unit
Based on the projection information associated with the first viewpoint and the projection information associated with the second viewpoint, a partial image centered on each viewpoint transitioning from the first viewpoint to the second viewpoint is generated. do,
The information processing apparatus according to any one of claims 1 to 12 . The viewpoint management unit
As the projection information, the angle of view of the image pickup device when the whole celestial sphere image is attached to the surface of a three-dimensional object and imaged by the image pickup device to acquire the partial image, the position information of the image pickup device, and the said. Information indicating the upward direction of the image pickup device and gaze point information indicating the position of the gaze point of the image pickup device are registered.
The information processing apparatus according to any one of claims 1 to 13 . The arithmetic unit
The angle of view of the image pickup device and the image pickup device based on the upward information of the spherical image at the time of designating the viewpoint specified by the user, the angle of view specified by the user, and the selected projection method. Position information, information indicating the upward direction of the image pickup device, and gaze point information indicating the position of the gaze point of the image pickup device are generated.
The information processing apparatus according to claim 14 . Computer,
A viewpoint management means for registering viewpoints specified by the user in the specified order in the spherical image.
A viewpoint control means for transitioning a viewpoint along a path that interpolates between a first viewpoint in a spherical image and a second viewpoint designated after the first viewpoint.
An arithmetic means that generates a partial image centered on each transition viewpoint,
Display means,
A control means that executes an animation display by connecting multiple generated partial images in the order of viewpoint transitions.
To function as
The display means is
Accepts the selection of the projection method when projecting the partial image on the display screen and displaying it.
The viewpoint management means is
Register the viewpoint specified by the user and the projection information for projection by the selected projection method in association with each other.
The calculation means is
A program for generating the partial image based on the projection information associated with the viewpoint specified by the user . The computer,
A moving image data generation means for converting the generated plurality of partial images into moving image data,
16. The program of claim 16 . The display means is
Thumbnail the icon corresponding to the viewpoint specified by the user ,
The program according to claim 17 . The display means is
Display one or more background sounds in a selectable manner,
The moving image data generation means is
Synthesize the background sound selected by the user with the moving image data.
The program according to claim 17 or 18 . The display means is
One or more crop areas are displayed selectably,
The moving image data generation means is
An animation in which the plurality of partial images are stitched together is cropped based on the crop area selected by the user.
The program according to any one of claims 17 to 19. The display means is
One or more preset data registered in order of two or more viewpoints in the spherical image is displayed in a selectable manner.
The viewpoint control means is
The viewpoint is changed along a path that interpolates between the first viewpoint registered in the preset data selected by the user and the second viewpoint in the next order of the first viewpoint.
The program according to any one of claims 16 to 20 . The preset data includes an angle of view registered in association with each viewpoint.
The calculation means is
The partial image is generated based on the angle of view associated with the viewpoint registered in the preset data selected by the user.
21. The program of claim 21 . The display means is
Accepts changes in the last-order viewpoint registered in the preset data selected by the user and the angle of view associated with the viewpoint.
The viewpoint control means is
When the last-order viewpoint is changed, the viewpoint is changed along the interpolated path based on the changed last-order viewpoint.
The calculation means is
When the angle of view associated with the last-order viewpoint is changed, the partial image is generated based on the changed angle of view.
22. The program of claim 22 . The viewpoint management means is
The projection information of the projection method different between the first viewpoint and the second viewpoint is linked and registered.
The calculation means is
Based on the projection information associated with the first viewpoint and the projection information associated with the second viewpoint, a partial image centered on each viewpoint transitioning from the first viewpoint to the second viewpoint is generated. do,
The program according to any one of claims 16 to 23 . The viewpoint management means is
As the projection information, the angle of view of the image pickup device when the whole celestial sphere image is attached to the surface of a three-dimensional object and imaged by the image pickup device to acquire the partial image, the position information of the image pickup device, and the said. Information indicating the upward direction of the image pickup device and gaze point information indicating the position of the gaze point of the image pickup device are registered.
The program according to any one of claims 16 to 24 . The calculation means is
The angle of view of the image pickup device and the image pickup device based on the upward information of the spherical image when the viewpoint specified by the user, the angle of view specified by the user, and the viewpoint are specified, and the selected projection method. Position information, information indicating the upward direction of the image pickup device, and gaze point information indicating the position of the gaze point of the image pickup device are generated.
25. The program of claim 25 .

Images