JP2018110375A - Display device, program, and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device capable of properly reproducing a superimposed moving image.SOLUTION: A display device 30 for displaying a first image on which a second image displayable as a moving image is superimposed includes: determination means 35 for determining whether the second image is browsed; and display switching means 36 that switches whether to reproduce the second image as a moving image or as a still image in accordance with the determination result of the determination means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device, a program, and a display method.

  Imaging devices capable of capturing images with a wider angle than conventional digital cameras are becoming popular. As one of such imaging apparatuses, there is known an omnidirectional camera that combines a plurality of fish-eye lenses and the like to create an omnidirectional image in which 360 degrees around is captured by one imaging. The user can arbitrarily determine the viewing direction for the omnidirectional image and view the omnidirectional image from various directions.

  However, considering the load of image processing and the like, image processing of a spherical image with a wide imaging range is likely to be restricted, and the image quality is deteriorated compared to an image of a normal digital camera (for example, the resolution is low and whiteout is likely to occur) The color reproducibility is low). For this reason, when a user enlarges and browses a part of the omnidirectional image, there is an inconvenience that the image quality is deteriorated and it is difficult to obtain detailed information.

  For such inconvenience, a technique for fitting a telephoto image into a wide-angle image has been devised (see, for example, Patent Document 1).

  However, there is a problem that the conventional technique does not consider the superimposition of moving images. If the user can browse the target with a moving image, it is easy to grasp the situation. However, since the omnidirectional image has a large number of pixels, if the display device displays the omnidirectional image of the moving image, a large amount of resources may be consumed. On the other hand, there are many cases where a user wants to see a region or a region with a movement. Therefore, it is considered that the display device superimposes a moving image on a part of the omnidirectional image. However, since the omnidirectional image has a wide angle of view, the user does not always see the moving image area. There is a low need to play a moving image that is not always seen, and there is a risk of increasing the processing load of the display device.

  In view of the above problems, an object of the present invention is to provide a display device capable of appropriately reproducing a superimposed moving image.

  The present invention is a display device that displays the first image in which a second image that can be displayed as a moving image is superimposed on the first image, and determines whether or not the second image is browsed And a display switching means for switching whether the second image is reproduced as a moving image or displayed as a still image according to a determination result of the determining means.

  It is possible to provide a display device that can appropriately reproduce the superimposed moving image.

It is an example of the figure explaining an omnidirectional image. It is an example of the figure explaining the display method of the moving image superimposed on the omnidirectional image. 1 is a diagram illustrating a system configuration example of an image processing system having an omnidirectional camera and a display device. It is an example of the hardware block diagram of an imaging device. It is an example of the hardware block diagram of a display apparatus. It is a use image figure of a spherical camera. It is a figure explaining the outline | summary of the process until an omnidirectional image is produced from the image imaged with the omnidirectional camera. It is a figure explaining the outline | summary of the process until an omnidirectional image is produced from the image imaged with the omnidirectional camera. It is an example of the figure explaining the pasting to the solid sphere of the omnidirectional image by equirectangular projection. It is an example of the figure explaining a user's eyes | visual_axis. It is an example of the figure which illustrates typically expansion and reduction of a viewing area of a spherical image. It is an example of the functional block diagram which shows the function which a digital camera, a spherical camera, and a display apparatus have in a block form. It is a figure which shows an example of the position parameter which matches the projection system conversion to the coincidence area of a plane image, and a plane image and a coincidence area. It is an example of the figure explaining the projection system conversion image and mask data which a projection system conversion part produces. It is an example of the figure which illustrates the superimposition of the plane image on the omnidirectional image typically. It is a figure which shows typically the plane image superimposed on the omnidirectional image. It is an example of the flowchart figure which shows the procedure in which a display apparatus superimposes a planar image on an omnidirectional image. It is an example of the functional block diagram explaining the function of an image superimposition part to a block form. It is an example of the figure explaining the relative position of the plane image with respect to a browsing area. It is an example of the figure explaining how to obtain the angle of view a and the angle of view b. It is an example of the figure explaining the relationship between a browsing area | region and the diagonal vertex of a planar image. It is a figure explaining the example of the plane image from which reproduction | regeneration is started by a user enlarging an omnidirectional image. It is an example of the flowchart figure which shows the procedure which controls whether a display apparatus reproduces | regenerates a moving image. It is a figure which shows an example of the menu with which a user operates animation reproduction | regeneration. It is an example of the flowchart figure which shows the procedure which controls whether a display apparatus reproduces | regenerates a moving image. It is an example of the figure explaining the process of the automatic expansion of the planar image to a browsing area. It is an example of the figure explaining the process of the automatic expansion of a plane image in case the whole plane image is not contained in the browsing area | region. It is an example of the flowchart figure which shows the procedure which judges whether a display apparatus reproduces | regenerates a moving image. It is an example of the figure explaining the metadata of animation reproduction. It is an example of the figure which illustrates typically the frame obtained by projection system conversion. It is an example of the figure which illustrates the superimposition of a frame typically. It is a figure which shows an example of the frame displayed in the aspect from which a moving image and a still image differ. It is an example of the flowchart figure which shows the procedure in which a display apparatus displays the frame of a plane image on a spherical image.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Overview of spherical image>
FIG. 1 is an example of a diagram illustrating the omnidirectional image CE. 1A shows an omnidirectional image CE represented by a three-dimensional solid sphere CS, and FIG. 1B shows an omnidirectional image CE represented by equirectangular projection. The omnidirectional image CE generated by the omnidirectional camera has a three-dimensional structure in which an image as shown in FIG. The virtual camera IC corresponds to the viewpoint of the user. In FIG. 1, the viewpoint is at the center of the omnidirectional image CE. The user rotates three axes around the X, Y, and Z axes that pass through the virtual camera IC, and is displayed on an arbitrary viewing area T (display or display software) of the omnidirectional image CE. Range) can be displayed. The user can also enlarge or reduce the browsing area T.

<Outline of Display Device of Present Embodiment>
FIG. 2 is an example of a diagram illustrating a method for displaying a moving image superimposed on an omnidirectional image. 2 is superimposed on the omnidirectional image CE, and the celestial sphere image CE is displayed so that the user can enlarge and view the planar image P from FIGS. 2 (a) to 2 (b). Is expanding. The user may be conscious of the moving image or unconscious. The display device may display that there is a planar image P (such as a frame to be described later) to make the user aware of it.

  2A to 2B, the user gradually enlarges the omnidirectional image CE. The planar image P in FIG. 2A corresponds to the entire planar image P in FIGS. As shown in FIG. 2B, when the user enlarges the omnidirectional image CE to some extent, it is estimated that the user wants to see the planar image P, and the display device starts reproducing the moving image. In FIG. 2C, a moving image in which the monorail train 351 is moving from the right to the left is displayed.

  As described above, when it is estimated that the user views the planar image P, the moving image is reproduced. Therefore, when the user wants to view the planar image P, the moving image can be reproduced, and reproduction of unnecessary moving images can be suppressed. In addition, the user can view the moving image from the beginning. If it is always played back, it will not be known how long it has passed since the playback started when the user started browsing the video. In addition, since the reproduction of the moving image starts when the user enlarges the planar image P, it seems that the screen suddenly starts moving, and the user's attention can be drawn.

<Terminology>
Superposition means that two or more objects are combined into one. In the present embodiment, the superimposition may include the meanings of pasting, fitting, synthesis, and superposition.

  The browsing of the claims means that the user views the second image with consciousness. The first image is, for example, an omnidirectional image, and the second image is, for example, a planar image.

  A moving image refers to a moving image obtained by continuously displaying a series of images (frames) captured at substantially constant time intervals at short intervals.

  Also, since the definition of high image quality varies depending on the image, it may vary depending on the user's viewing purpose, but generally it means that the image faithfully represents a scene. For example, an image having a high resolution, a wide dynamic range, high color reproducibility, little noise, and the like.

<System configuration example>
FIG. 3 is a diagram illustrating a system configuration example of the image processing system 100 including the omnidirectional camera 20 and the display device 30. FIG. 3A shows an image processing system 100 having an omnidirectional camera 20 and a normal digital camera 9. The normal digital camera 9 is generally an image pickup apparatus (camera) that picks up an image with a focal length of 35 mm or more when converted into a 35 mm film, but may include an image pickup apparatus that can pick up a wide-angle image of about 24 mm to 35 mm. Simply, it can be said that the imaging device captures the planar image P instead of the omnidirectional camera 20. In other words, the imaging apparatus captures an image that can be displayed entirely in the browsing area T.

  The digital camera 9 has an image sensor that has at least a higher resolution than the omnidirectional image CE (a large number of pixels with respect to the angle of view to be imaged). In addition, since the imaging conditions (exposure time, shutter speed, white balance, etc.) are optimized within an imaging range narrower than that of the omnidirectional camera 20, whiteout is difficult, color reproducibility is high, and noise is often low. Thus, the digital camera 9 generates a so-called high-quality planar image.

  Thereby, in the omnidirectional image CE, an image that becomes rough when enlarged can be supplemented with a high-quality image of the planar image P. In the configuration of FIG. 3A, the digital camera 9 also serves as the display device 30. A specific example of the digital camera 9 is, for example, a digital still camera or a digital video camera, but may be other devices including an imaging device. For example, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a wearable PC (Personal Computer), or the like may be used. In this case, the omnidirectional camera 20 may be externally attached to a smartphone or the like.

  The omnidirectional camera 20 and the digital camera 9 can communicate with each other by USB, Bluetooth (registered trademark), wireless LAN, or the like. The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the digital camera 9. Similarly, the digital camera 9 captures the planar image P. Since the 360-degree landscape is reflected in the omnidirectional image CE, the planar image P becomes a part of the omnidirectional image CE.

  As the display device 30, the digital camera 9 superimposes the planar image P on the omnidirectional image CE. Specifically, for example, the user U selects the planar image P and the omnidirectional image CE and instructs superposition. Alternatively, when the digital camera 9 is set to the superimposing mode and the planar image P is captured, the digital image is automatically superimposed on the omnidirectional image CE with which the planar image P matches. There is little inconvenience because the images are not superimposed if the degree of coincidence of the images is low. Further, the digital camera 9 as the display device 30 switches between reproduction and stop of moving images and displays them on the display of the digital camera 9. As described above, when the digital camera 9 is the display device 30, the omnidirectional camera 20 and the digital camera 9 are configured to superimpose the omnidirectional image CE and the planar image P, or to change the playback and stop of the moving image. can do.

  FIG. 3B shows an image processing system 100 having an omnidirectional camera 20, a normal digital camera 9, and a display device 30. The omnidirectional camera 20 and the display device 30 and the digital camera 9 and the display device 30 can communicate with each other by USB, Bluetooth (registered trademark), wireless LAN, or the like. The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the display device 30. A planar image P captured by the digital camera 9 is also transmitted to the display device 30. In addition to transmission, the display device 30 may read the omnidirectional image CE or the planar image P stored in the storage medium from the storage medium.

  The display device 30 is an information processing device such as a PC, a smartphone, a tablet terminal, or a PDA. In addition, a multifunction device having a function as an information processing device, a projector, a video conference terminal, or the like may be used.

  In the configuration of FIG. 3B, the display device 30 superimposes the planar image P on the omnidirectional image CE. In addition, the playback and stop of the moving image are switched and displayed on the display device 30. More specifically, the user U can know which planar image P is a part of which omnidirectional image CE by comparing the planar image P with the omnidirectional image CE. A planar image P to be pasted is selected, and the display device 30 is pasted. You may superimpose automatically like the digital camera 9. Thus, the display device 30 can acquire the omnidirectional image CE and the planar image P, superimpose the planar image P on the omnidirectional image CE, and switch between playback and stop of the moving image. In the description of FIG. 4 and subsequent figures, the configuration of FIG. 3B will be described in mind unless otherwise specified.

  FIG. 3C shows an image processing system 100 in which an omnidirectional camera 20, a normal digital camera 9, a display device 30, and an image superimposing server 70 are connected via a network N. The network N is constructed by a LAN constructed in the user U environment, a provider network of a provider that connects the LAN to the Internet, a line provided by a line operator, and the like. When the network N has a plurality of LANs, the network N is called a WAN or the Internet. The network N may be constructed by either wired or wireless, and wired and wireless may be combined. In addition, when the omnidirectional camera 20 and the digital camera 9 are directly connected to the public network, they can be connected to the provider network without going through the LAN.

  The image superimposing server 70 may be a general information processing device or server device, and a specific example of the display device 30 may be the same as that in FIG. However, the image superimposing server 70 is preferably compatible with cloud computing. Cloud computing refers to a usage pattern in which resources on a network are used without being aware of specific hardware resources.

  The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the image superimposing server 70. A planar image P captured by the digital camera 9 is also transmitted to the image superimposing server 70. In the configuration of FIG. 3C, the image superimposing server 70 superimposes the planar image P on the omnidirectional image CE. More specifically, the display device 30 receives the operation of the user U, accesses the image superimposing server 70, and displays a list of omnidirectional images CE and planar images P (thumbnail images and the like). The user U can know which planar image P is a part of which celestial sphere image CE by comparing the planar image P with the celestial sphere image CE. Is notified to the image superimposing server 70. The image superimposing server 70 may automatically superimpose.

  The image superimposing server 70 superimposes the planar image P on the omnidirectional image CE and transmits the image data of the viewing area T to the display device 30. It is assumed that the viewing direction and the angle of view in the initial state are determined in advance. The image superimposing server 70 transmits, as a Web server, image data in the viewing area T and a display program (described in script language) for accepting an operation of the user U to the display device 30. The display device 30 executes the display program, accepts an operation of the user U, and transmits the line-of-sight direction and the angle of view to the image superimposing server 70. The image superimposing server 70 updates the viewing area T according to the line-of-sight direction and the angle of view, and transmits image data of the viewing area T to the display device 30. Since the image superimposing server 70 switches between reproduction and stop of the moving image, the display device 30 can display an image in which the moving image is reproduced and an image that is not reproduced.

  The image superimposing server 70 may transmit the omnidirectional image CE, the planar image P, and the display program to the display device 30. In this case, the display device 30 determines the viewing area T according to the operation of the user U, and switches between playback and stop of the moving image.

  Further, the image superimposing server 70 may calculate a position parameter (described later) from the omnidirectional image CE and the planar image P, and the display device 30 may download the omnidirectional image CE, the planar image P, and the positional parameter. Although the user arbitrarily changes the line-of-sight direction, the display device 30 can superimpose the planar image P on the omnidirectional image CE using the same position parameter. A position parameter creation unit 8 of FIG. 13 to be described later is provided in the image superimposing server 70, and the conversion display unit 7 is included in the display device 30.

<Hardware configuration>
<< Hardsphere camera hardware configuration >>
FIG. 4 is a hardware configuration diagram of the omnidirectional camera. In the following, the omnidirectional camera 20 is an omnidirectional imaging apparatus using two imaging elements, but the number of imaging elements may be three or more. In addition, it is not always necessary to use an apparatus dedicated to omnidirectional imaging. By attaching a retrofit omnidirectional imaging unit to a normal digital camera 9, a smartphone, or the like, it has substantially the same function as the omnidirectional camera 20. May be.

  As shown in FIG. 4, the omnidirectional camera 20 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, and acceleration A sensor 120 or the like is also connected.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electric signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103 a and 103 b of the imaging unit 101 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then superimposes the image data. Then, image data is created by equirectangular projection as shown in FIG.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data of the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some imaging devices may have a preview display function using a display or a function corresponding to moving image display. In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the imaging device is not provided with a display unit, but a display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the omnidirectional camera 20 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed image data based on equirectangular projection.

  The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user U inputs various imaging modes and imaging conditions by operating the operation buttons.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. Further, the network I / F 116 may be a network interface regardless of wireless or wired. The image data of the equirectangular projection stored in the DRAM 114 is recorded on an external medium via the network I / F 116 or displayed via the network I / F 116 as a network I / F as necessary. Or transmitted to an external device such as the device 30.

  The communication unit 117 communicates with an external device such as the display device 30 by a short-range wireless technology such as Wi-Fi (wireless fidelity) or NFC via the antenna 117 a provided in the omnidirectional camera 20. The communication unit 117 can also transmit image data based on equirectangular projection to an external device such as the display device 30.

  The electronic compass 118 calculates the direction of the omnidirectional camera 20 from the earth's magnetism and outputs the direction information. This orientation information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

  The gyro sensor 119 detects a change in angle (roll angle, pitching angle, yaw angle) accompanying the movement of the omnidirectional camera 20. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

  The acceleration sensor 120 detects acceleration in three axis directions. Based on the detected acceleration, the attitude of the omnidirectional camera 20 (angle with respect to the direction of gravity) is detected. By having both the gyro sensor 119 and the acceleration sensor 120, the accuracy of image correction is improved.

  It should be noted that even if the single-lens reflex hardware configuration is the same as or different from that shown in FIG.

<< Hardware configuration of display device >>
FIG. 5 is a hardware configuration diagram of the display device. The display device 30 includes a CPU 501 that controls the operation of the entire display device, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL (Initial Program Loader), a RAM 503 that is used as a work area for the CPU 501, and a program for the display device HD (Hard Disk) 504 for storing various data such as. In addition, an HDD (Hard Disk Drive) 505 that controls reading or writing of various data with respect to the HD 504 according to the control of the CPU 501 and a media drive 507 that controls reading or writing (storage) of data with respect to a recording medium 506 such as a flash memory. In addition, a display 508 for displaying various information such as a cursor, menu, window, character, or image, a network I / F 509 for data communication using the network N, a character, a numerical value, various instructions for inputting, etc. A keyboard 511 having a plurality of keys, and a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. Further, as shown in FIG. 5, an optical media drive 514 that controls reading or writing of various data with respect to an optical storage medium 513 as an example of a removable recording medium, and the above-described components are electrically connected. For this purpose, a bus line 510 such as an address bus or a data bus is provided.

  It should be noted that the hardware configuration of the image superimposing server 70 is the same as or different from that in FIG.

<Creation of spherical image CE>
The omnidirectional image CE will be described with reference to FIGS. FIG. 6 is a usage image diagram of the omnidirectional camera 20. As shown in FIG. 6, the omnidirectional camera 20 is used by the user U to take an image of a subject around the user U with his hand. The omnidirectional camera 20 has a structure in which the back surfaces of the two imaging elements are opposed to each other, and obtains two hemispherical images by imaging the subject around the user U, respectively.

  Next, an outline of processing until an omnidirectional image CE is created from an image captured by the omnidirectional camera 20 will be described with reference to FIGS. 7A shows a hemispheric image (front side) captured by the omnidirectional camera 20, FIG. 7B shows a hemispheric image captured by the omnidirectional camera 20 (rear side), and FIG. 7C. These are the figures which showed the omnidirectional image by equirectangular projection. FIG. 8A is a conceptual diagram showing a state where a sphere is covered with an image obtained by equirectangular projection, and FIG. 8B is a diagram showing an omnidirectional image CE.

  As shown in FIG. 7A, the image obtained by the omnidirectional camera 20 is a hemispherical image (front side) curved by the fisheye lens. Further, as shown in FIG. 7B, the image obtained by the omnidirectional camera 20 is a hemispherical image (rear side) curved by the fisheye lens. Then, the hemispherical image (front side) and the hemispherical image inverted at 180 degrees (rear side) are joined together by the omnidirectional camera 20, and as shown in FIG. An image is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 8A, an image by equirectangular projection is pasted so as to cover the spherical surface. An omnidirectional image CE as shown in (b) is created. As described above, the omnidirectional image CE is represented as an image in which an image obtained by equidistant cylindrical projection faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The omnidirectional image CE may be a still image or a moving image. Specifically, the conversion from the image by the equirectangular projection of FIG. 8A to the three-dimensional omnidirectional image of FIG. 8B uses the equation (1) described in FIG.

  FIG. 9A is a diagram showing the omnidirectional image CE by the equirectangular projection, and FIG. 9B is an omnidirectional image in which the omnidirectional image CE of the equirectangular projection is pasted on the omnidirectional sphere. A spherical image CE is shown. The conversion from the image in FIG. 9A to the image in FIG. 9B is a process performed by the omnidirectional camera 20 when the omnidirectional camera 20 described in FIG.

The coordinates of a three-dimensional point (x, y, z) corresponding to an arbitrary point (λ, φ) in equirectangular projection can be obtained by the following equation, assuming that the radius of the sphere is 1.0.
X = cos (φ) * cos (λ)
Y = sin (φ) (1)
Z = cos (φ) * sin (λ)
Thus, a three-dimensional omnidirectional image CE is obtained from the omnidirectional image CE of equirectangular projection. Since the omnidirectional image CE is an image pasted so as to cover the spherical surface, it is curved and has a sense of incongruity when viewed by humans. Therefore, the display device 30 displays a part of the viewing area T of the omnidirectional image CE as a flat image P with a small curvature so as not to give a sense of incongruity to humans. The browsing area T is indicated by coordinates (X, Y, Z) in a three-dimensional virtual space. On the other hand, since the display 508 is a two-dimensional plane, the display device 30 cannot display the viewing area T as it is.

  Therefore, the display device 30 obtains an image with less curvature by perspective projection conversion in which a three-dimensional object is projected onto a two-dimensional plane using a 3D computer graphic technique. As described above, the viewing area T of the omnidirectional image CE is displayed on the display 508.

  FIG. 10 is an example of a diagram illustrating the line of sight of the user U. Since the omnidirectional image CE has three-dimensional coordinates, the line-of-sight direction is specified by information specifying the coordinates of the sphere such as three-dimensional coordinates and latitude / longitude. In the present embodiment, the center cp of the viewing area T is the line-of-sight direction.

  Although the user U can change the line-of-sight direction with the keyboard 511 or the mouse 512, assuming that the virtual camera IC does not translate, the virtual camera IC is rolled as a rigid body (rotation about the Z axis), yaw (Y axis) Rotation around the X axis) and pitch (rotation around the X axis) are possible. The line-of-sight direction changes regardless of which of these three rotations occurs. For example, when the user U rotates the omnidirectional image CE in the horizontal direction, the yaw angle changes. When the user U rotates it in the vertical direction, the pitch angle changes, and the omnidirectional image CE is rotated about the center of the display 508. And roll angle changes. In the present embodiment, the operation of the user U on the Web page is reflected in the line-of-sight direction (roll angle, yaw angle, pitch angle) and the like. How it is reflected is described in advance in a program executed by the display device 30.

  FIG. 11 is an example of a diagram schematically illustrating enlargement and reduction of the viewing area T of the omnidirectional image CE. FIG. 11A shows an initial browsing area T. FIG. When the virtual camera IC exists in the center of the solid sphere CS, the viewing area T is determined by the angle of view α. The angle of view α is an angle at which the diagonal vertex of the viewing area T faces the center of the solid sphere CS, for example.

  The angle of view α in the initial state is α0. If the angle of view is reduced as shown in FIG. 11B (α1 <α0), the viewing area T becomes narrower, so the image is enlarged on the display 508, and the angle of view is increased as shown in FIG. 11C. Then (α2> α0), the viewing area T becomes wide, so that the image is reduced on the display 508. When the user U performs an operation for enlarging or reducing the omnidirectional image CE, the display device 30 decreases or increases the angle of view according to the operation amount.

  Even if the angle of view increases as shown in FIG. 11C, as long as the virtual camera IC is in the center, the display device 30 cannot display an image behind the virtual camera IC. Therefore, when the user U performs an operation of further reducing from the state of FIG. 11C, the display device 30 moves the virtual camera IC backward. In FIG. 11 (d), the angle of view α2 is not different from that in FIG. 11 (c), but the viewing area T is further expanded by moving the virtual camera IC backward. For this reason, it is possible to further reduce the image as compared with FIG.

<Function related to superimposition of planar image P on omnidirectional image CE>
The function regarding the superimposition of the planar image P on the omnidirectional image CE will be described with reference to FIGS. As described above, the configuration of FIG. 3B will be described in mind.

  FIG. 12 is an example of a functional block diagram showing the functions of the digital camera 9, the omnidirectional camera 20, and the display device 30 in a block shape. The digital camera 9 has a planar image acquisition unit 11. The planar image acquisition unit 11 is realized by the imaging function of the digital camera 9, and creates image data (planar image P) in which each pixel of a predetermined resolution represents light and shade. The planar image P may be one still image, a plurality of still images, or a moving image, but is assumed to be a moving image unless otherwise specified in the present embodiment.

  The omnidirectional camera 20 has an omnidirectional image acquisition unit 12. The omnidirectional image acquisition unit 12 is realized by the imaging function of the omnidirectional camera 20, and creates 360 degree image data (omnidirectional image CE) around the omnidirectional camera 20. The omnidirectional image CE may be one still image, a plurality of still images, or a moving image, but is assumed to be a still image unless otherwise specified in the present embodiment.

  The display device 30 mainly includes a position parameter creation unit 8 and a conversion display unit 7. These functions of the display device 30 are functions or means realized by the CPU 501 of the display device 30 illustrated in FIG. 5 executing a program expanded from the HD 504 to the RAM 503.

  The position parameter creation unit 8 further includes an omnidirectional image reading unit 21, a planar image reading unit 22, a position parameter calculation unit 23, and a position parameter writing unit 24. The omnidirectional image reading unit 21 reads the omnidirectional image CE from the omnidirectional image acquisition unit 12. The planar image reading unit 22 reads the planar image P from the planar image acquisition unit 11. Reading means acquiring, receiving, reading from a storage medium, or receiving input.

  The position parameter calculation unit 23 specifies the matching area of the omnidirectional image CE that matches the planar image P, and obtains the position parameter PP for specifying the matching area. Details of the position parameter PP will be described with reference to FIG. Since the omnidirectional image CE is distorted by the equirectangular projection, it is preferable that the flat image P is subjected to the same conversion to be distorted. Then, the position parameter PP can be obtained by extracting both feature points and performing matching. For detection of feature points, various detection methods such as edge detection, corner detection, SIFT feature amount and SURF feature amount, and continuous central points of the same color are known. Alternatively, the sum of the absolute values of the difference between the pixel values of the omnidirectional image CE and the planar image P or the sum of squares of the differences is calculated by shifting one pixel at a time, and the plane when the sum of the absolute values or the sum of squares becomes the smallest The position of the image P may be the matching area.

  In this embodiment, since the omnidirectional image CE is a still image and the planar image P is a moving image, only a frame at a certain time (each still image included in the moving image) matches the omnidirectional image CE. The position parameter calculation unit 23 performs matching with the omnidirectional image in order from the first frame of the planar image P. Then, the position parameter PP is calculated based on a frame having a matching degree equal to or greater than a threshold. Alternatively, the display device 30 may be instructed by the user U of the matching frame. The position parameter writing unit 24 notifies the conversion display unit 7 of the frame ID 6 of the frame that matches the omnidirectional image CE. Note that ID is an abbreviation of Identification and means an identifier or identification information. ID refers to a name, a code, a character string, a numerical value, or a combination of one or more of these used to uniquely distinguish a specific target from a plurality of targets. The same applies to IDs other than the frame ID.

  Further, if the planar image P is divided into a mesh area and matched with the omnidirectional image CE, it is possible to take into account deviation correction due to lens distortion or the like.

  Since the correspondence between the omnidirectional image CE and the planar image P is uniquely determined, the position parameter PP need only be obtained once. When the planar image reading unit 22 reads a plurality of planar images P, the position parameter PP is calculated for each planar image P.

  The conversion display unit 7 further includes a planar image reading unit 25, a line-of-sight direction angle-of-view instruction unit 26, an omnidirectional image reading unit 27, a projection method conversion unit 1 (reference numeral 28), and a projection method conversion unit 2 (reference numeral 29). The image superimposing unit 33, the position parameter reading unit 32, and the image display unit 31. The function of the omnidirectional image reading unit 27 is the same as that of the position parameter creating unit 8. Further, the position parameter reading unit 32 acquires the position parameter PP and the frame ID from the position parameter creating unit 8.

  The planar image reading unit 25 reads each frame of the planar image P. Until the later-described image superimposing unit 33 notifies the planar image reading unit 25 of the moving image reading ON, the frame with the frame ID instructed by the position parameter calculating unit 23 is read. After the image superimposing unit 33 notifies the planar image reading unit 25 that the moving image reading is ON, the subsequent frame in time (the previous frame in the case of reverse reproduction) is read according to the reproduction speed of the moving image. When the image superimposing unit 33 notifies the plane image reading unit 25 that moving image reading is OFF, the frame ID frame or the last read frame is repeatedly read.

  The line-of-sight direction angle-of-view instruction unit 26 receives the line-of-sight direction and the angle of view (enlargement / reduction) operated by the user U. The viewing area T is determined because the line-of-sight direction and the angle of view are input by operation.

  In addition, since the user U arbitrarily operates the line-of-sight direction and the angle of view (enlargement / reduction), only a part of the planar image P that has undergone the projection method conversion enters the viewing area T, or the projection method has been converted. The planar image P may not enter the browsing area T at all. For this reason, the projection method conversion unit 1 generates mask data indicating a region included in the browsing region T in the projection method conversion image. Details will be described with reference to FIG.

  The projection method conversion unit 2 determines the viewing area T corresponding to the viewing direction and the viewing angle received by the viewing direction view angle instruction unit 26, and the image display unit 31 for the viewing area T of the read omnidirectional image CE. Perspective projection conversion according to the display size is performed to create a display image. Therefore, the display image is a two-dimensional plane image.

  The image superimposing unit 33 generates a superimposed image by superimposing the projection method conversion image on the display image using the mask data. Details will be described with reference to FIG. The image display unit 31 displays the superimposed image on the display 508.

<Position parameter>
FIG. 13 is a diagram showing an example of the projection method conversion to the matching area of the planar image P and the position parameter PP for associating the planar image P with the matching area. It is assumed that correspondence between the planar image P and the omnidirectional image CE has already been obtained by matching. The position parameter PP is the latitude and longitude of the omnidirectional image CE where the planar image P exists when the feature points are the best match.

  FIG. 13A shows a planar image P, and FIG. 13B shows an omnidirectional image CE of equirectangular projection. FIG. 13B shows a matching area 301. The position parameter calculation unit 23 divides the planar image P into a grid pattern, and calculates the point (λ, φ) of the coincidence region 301 of the omnidirectional image CE corresponding to the coordinates (x, y) of the intersection of the grids. It is assumed that the position parameter PP. Note that λ is longitude and φ is latitude. FIG. 13C shows an example of the position parameter PP. The coordinates of the grid of the planar image P and the latitude and longitude of the matching area 301 of the omnidirectional image CE are associated with each grid.

<Processing of Projection Method Conversion Unit 1>
FIG. 14 is an example of a diagram illustrating a projection method conversion image 302 and mask data 303 created by the projection method conversion unit 1. FIG. 14A shows a projection method conversion image 302. The projection system conversion image 302 has an area having the same size as the viewing area T of the omnidirectional image represented by the equirectangular projection, and the planar image P after the projection system conversion is pasted on the matching area 301. The planar image P is in a trapezoidal distortion state by the projection system conversion of the planar image P to the coincidence area 301. A portion where the planar image P is not present in the projection system converted image 302 is a gray uniform image 307. Since the uniform image 307 is not used for superposition, any pixel value may be used.

  FIG. 14B shows mask data 303 corresponding to the planar image P of FIG. The mask data 303 is data for extracting the planar image P included in the viewing area T from FIG. The white pixel area of the mask data indicates the browsing area T and the plane image P area. Therefore, the white pixel area of the mask data is equal to or smaller than the area of the planar image P. In the mask data of FIG. 14B, when the viewing area T is larger than the planar image P, the matching area 301 is almost equal to the white pixel area of the mask data 303 in the example of FIG. . When the viewing area T does not overlap the flat image P at all, the entire mask data is black pixels. When the viewing area T is smaller than the planar image P, the mask data 303 is only white pixels. Thus, the white pixel and the matching area 301 of the mask data 303 are always the same size and the same position.

  The image superimposing unit 33 performs a mask process using the projection method converted image 302 and the mask data 303. This mask process is a process for extracting pixels corresponding to white pixels of the mask data 303 from the projection method conversion image 302. The image superimposing unit 33 extracts the pixel value at the position of the white pixel from the projection method conversion image 302 and superimposes it on the same position as the projection method conversion image 302 of the display image.

  The boundary between the black pixel and the white pixel in the mask data 303 is preferably provided with a slope that gradually changes from white to black from the white pixel to the black pixel. Thereby, the boundary between the low-resolution spherical image CE and the high-resolution planar image P can be made inconspicuous.

<Processing of image superimposing unit>
The process of the image superimposing unit 33 will be described with reference to FIG. FIG. 15 is an example of a diagram schematically illustrating the superimposition of the planar image P on the omnidirectional image CE. As shown in FIG. 15, a display image 304 obtained by perspective projection conversion of the viewing area T by the projection method conversion unit 2, a projection method conversion image 302 (FIG. 14A), and mask data 303 (FIG. 14B). Input to the image superimposing unit 33.

  The image superimposing unit 33 superimposes the projection method conversion image 302 having a wider angle of view on the display image 304. By the mask processing using the mask data 303, the projection method conversion image 302 at the pixel position of the white pixel in the mask data 303 is taken out and overwritten on the display image 304 (in this case, the pixels of the display image 304 are lost at this time). ) In this way, in the superimposed image 305, the high-resolution planar image P is arranged on the low-resolution omnidirectional image CE.

  Alternatively, a transparent layer may be prepared, and the projection method conversion image 302 at the pixel position of the white pixel in the mask data 303 may be taken out and placed in the transparent layer. In this case, there is no loss in the display image 304, and for example, the user U can switch between display and non-display of the planar image P.

  The image superimposing unit 33 outputs a superimposed image 305. In the superimposed image 305, a high-resolution planar image P is arranged on a low-resolution omnidirectional image CE.

  In this embodiment, the superimposing method is also characterized. As described above, the display device 30 performs the perspective projection conversion of the omnidirectional image CE and the superimposition of the planar image P in parallel. If the perspective projection conversion is performed after the planar image P is pasted on the omnidirectional image CE, the resolution of the omnidirectional image CE needs to be the same as that of the planar image P, and the amount of data increases. In the present embodiment, since the planar image P is superimposed after the display image 304 is created, an increase in the data amount can be suppressed.

  The display device 30 repeats the superimposition of the omnidirectional image CE and the planar image P in the display cycle of the display 508 (for example, 30 to 60 times per second). By doing so, the display device 30 can hold the planar image P and the omnidirectional image CE one by one, and generate the superimposed image 305 in real time from the viewpoint according to the operation of the user U. In addition, a moving image can be reproduced.

<Plane image P in the spherical image CE>
FIG. 16 is a diagram schematically illustrating a planar image P superimposed on the omnidirectional image CE. As described above, since the planar image P is superimposed on the display image 304 as it is, the planar image P is not pasted on the solid sphere CS as shown in FIG. However, when viewed from the user U, as shown in FIG. 16, it appears that the planar image P is superimposed on a part of a sphere having the same center position and the same radius as the omnidirectional image CE.

  In addition, as shown in FIG. 16, the planar image P may be superimposed on the three-dimensional omnidirectional image CE. Since the longitude and latitude corresponding to the planar image P can be known by matching (by the position parameter PP), the display device 30 can paste the planar image P on the celestial sphere according to Expression (1). In this case, the hidden surface removal method using the depth information (so-called Z buffer method) or the like is not used, and rendering may be performed with priority.

  In the case where there are a plurality of plane images P, if the position parameter calculation unit 23 calculates the position parameters PP for the respective plane images P, they can be superimposed in the same manner. In this case, when the image is captured by the digital camera 9 having the same number of pixels with the wide-angle image first and the telephoto later, the lower-resolution planar image P is placed on the higher-resolution planar image P. Can be avoided.

<Superposition procedure>
FIG. 17 is an example of a flowchart illustrating a procedure in which the display device 30 superimposes the planar image P on the omnidirectional image CE. The process in FIG. 17 is started when the user gives an instruction to superimpose. Steps S10 to S30 and S40 to S60 are executed in parallel, but may be executed in order.

  The planar image reading unit 22 reads the planar image P (S10). The omnidirectional image reading unit 21 reads the omnidirectional image CE (S20). Then, the position parameter calculation unit 23 calculates the position parameter PP (S30).

  The planar image reading unit 25 reads the planar image P (S40). The omnidirectional image reading unit 27 reads the omnidirectional image CE (S50). The line-of-sight direction angle-of-view instruction unit 26 receives the line-of-sight direction and the angle of view (S60). Step S60 is performed as needed.

  Next, the projection method conversion unit 1 performs the projection method conversion on the plane image P with the position parameter PP, and generates the projection method conversion image 302 (S70).

  Next, the projection method conversion unit 1 generates mask data 303 according to the viewing area T determined by the line-of-sight direction and the angle of view and the planar image P (coincidence region) converted by the projection method (S80).

  Next, the projection method conversion unit 2 performs perspective projection conversion of the omnidirectional image CE according to the viewing area T, and generates a display image 304 (S90).

  Next, the image superimposing unit 33 superimposes the projection method conversion image 302 and the display image 304 using the mask data, and generates a superimposed image 305 (S100).

  Next, the image display unit 31 displays the superimposed image 305 (S110). The display device 30 repeats the processing of steps S60 to S110 in FIG.

<Functions related to video playback>
Next, functions relating to the reproduction of the planar image P superimposed on the omnidirectional image CE will be described with reference to FIGS. FIG. 18 is an example of a functional block diagram illustrating the function of the image superimposing unit 33 in a block shape. The image superimposing unit 33 further includes a determination unit 35 and an image switching unit 36. The determination unit 35 determines whether it is estimated that the user U sees the planar image P superimposed on the superimposed image 305, and sends the determination result to the image switching unit 36. The image switching unit 36 notifies the planar image reading unit 25 of moving image reading ON or moving image reading OFF according to the determination result.

<About judgment of video playback>
When it is estimated that the user U sees the planar image P superimposed on the superimposed image 305 as described above, the display device 30 starts reproducing the moving image. Specifically, the following three conditions (i) to (iii) are criteria for reproducing a moving image.
(i) When the ratio of the angle of view occupied by the planar image P to the angle of view of the current viewing area T is greater than a predetermined value
(ii) When the area of the planar image P exceeds a predetermined angle of view
(iii) When User U Clicks Plane Image P Note that the estimation of viewing includes the fact that the user U may actually see and possibly see. In addition, the condition (iii) may be referred to as reproducing a moving image in accordance with an operation for browsing by the user U.

  First, a determination method of “(i) the case where the ratio of the planar image P to the angle of view of the current viewing area T exceeds a predetermined value” will be described.

  FIG. 19 is an example of a diagram illustrating the relative position of the planar image P with respect to the viewing area T. In FIG. 19A, the entire planar image P is included in the viewing area T. Since the viewing area T is the area of the omnidirectional image displayed on the display 508, the user U is viewing the planar image P almost from the front. If the viewing angle of the viewing area is a and the viewing angle of the planar image P is b, if the viewing angle a> the viewing angle b, the entire planar image P and a part of the omnidirectional image CE are visible. Become. In the situation as shown in FIG. 19A, it is estimated that the user U wants to see the planar image P.

  However, when the omnidirectional image CE is somewhat larger than the planar image P, the user U does not have to determine that he / she wants to view the planar image P. Therefore, the determination unit 35 determines whether, for example, the angle of view a is 1.2 times or less of the angle of view b. That is, when the angle of view a is 1.2 times or less of the angle of view b, it is determined that the ratio of the angle of view occupied by the planar image P to the current angle of view is greater than or equal to a predetermined value. The image switching unit 36 sends a moving image reproduction request ON to the planar image reading unit 25. When it is determined that the angle of view a is larger than 1.2 times the angle of view b, the image switching unit 36 sends a moving image reproduction request OFF to the planar image reading unit 25.

  The browsing area T is known. Further, since the latitude and longitude of the planar image P are registered in the position parameter PP, the angle of view b can be obtained from the position parameter PP. Details will be described with reference to FIGS.

  In FIG. 19B, the viewing area T is narrower than the planar image P. That is, the user U sees only the planar image P. In this case, since the angle of view a is 1.2 times or less than the angle of view b, a moving image is naturally reproduced.

  In FIG. 19 (c), only the upper side of the planar image P is the viewing area T. That is, the user U is looking above the planar image P. However, since the viewing area T is completely included in the planar image P (the angle of view a is 1.2 times or less of the angle of view b), the image switching unit 36 sends a moving image reproduction request ON.

  In FIG. 19D, the deviation (angle) between the center of the viewing area T and the center of the planar image P is large, and a part of the viewing area T protrudes from the planar image P. In this case, since the planar image P is smaller than the viewing area T (the angle of view a is greater than 1.2 times the angle of view b), the image switching unit 36 sends the moving image reproduction request OFF.

  It should be noted that the threshold value for determining whether or not to play a moving image is 1.2 times as an example, and may be a value larger than 1.0. For example, it can be determined as 1.1 to 1.5 times. Moreover, the user U may be able to set a threshold value.

FIG. 20 is an example of a diagram for explaining how to obtain the angle of view a and the angle of view b. The angle of view a is obtained from the viewing area T determined by the user U by operation. When the distance from the virtual camera IC to the viewing area T is f and the length of the diagonal line of the viewing area T is 2L, the angle of view α, the distance f, and L have the relationship of Expression (2).
Lf = tan (α / 2) (2)
Since the coordinates of the diagonal vertices of the viewing area T are known by the operation of the user U, the diagonal length 2L can be easily calculated. The distance f is known from the radius of the celestial sphere or that the user U has operated. Therefore, the angle of view α can be calculated by the equation (2).

  The angle of view b can be calculated in the same manner, but the angle of view b may be calculated when the viewing area T includes even a portion of the planar image P. FIG. 21 is a diagram showing the relationship between the viewing area T and the planar image P. The determination unit 35 determines whether each of the four vertices determines whether the planar image P is in the viewing area T. The four vertices of the viewing area T are A1 to A4, and the four vertices of the planar image P are B1 to B4. The vertices B1 to B4 are specified by, for example, the longitude and latitude of the omnidirectional sphere, but may be specified by coordinates on the display 508.

  For example, the vertex B1 is determined as follows.

The longitude of vertex A1 ≦ the longitude of vertex B1 ≦ the longitude of vertex A2, and
It can be similarly determined for the latitude of the vertex A1 ≧ the latitude of the vertex B1 ≦ the latitude vertices B2, B3, and B4 of the vertex A4. If the diagonal vertices (B1 and B3, B2 and B4) are in the viewing area T, the angle of view b can be calculated from the position parameter PP according to equation (2). As shown in FIG. 21, when all of the vertices B <b> 1 to B <b> 4 of the planar image P are not in the browsing area T, the angle of view b is calculated from the vertex B <b> 1 in the browsing area T by the vertex A <b> 3 of the browsing area T. The diagonal vertex of vertex B1 is vertex A3, the diagonal vertex of vertex B2 is vertex A4, the diagonal vertex of vertex B3 is vertex A1, and the diagonal vertex of vertex B4 is vertex A2. This is set in the display device 30.

<Display example>
FIG. 22 is a diagram illustrating an example of a planar image P that is reproduced by the user U enlarging the omnidirectional image CE. In FIG. 22A, a small planar image P is included in the viewing area T. In the state of FIG. 22A, since the angle of view a is larger than 1.2 times the angle of view b, a moving image is not reproduced.

  In FIG. 22B, since the angle of view a is 1.2 times or less of the angle of view b, the reproduction of the moving image starts. Since the omnidirectional image CE is not enlarged from FIG. 22C to FIG. 22E, it has the same composition (angle of view). However, since the moving image is reproduced, the monorail train 351 gradually moves from right to left.

  That is, when the user U enlarges the celestial sphere image CE and the ratio of the planar image P becomes equal to or greater than the threshold, the reproduction of the moving image automatically starts, and the user U can easily grasp the situation by the moving image (enjoy the moving image) Can do that.) In addition, since the reproduction of the moving image starts when the user U enlarges the planar image P, the screen suddenly starts to move, and the user U's attention can be drawn.

<Operation procedure>
FIG. 23 is an example of a flowchart illustrating a procedure for controlling whether or not the display device 30 reproduces a moving image. The procedure of FIG. 23 starts when the display device 30 displays the omnidirectional image CE. It is assumed that the planar image P has already been superimposed on the omnidirectional image CE.

  The determination unit 35 determines whether the viewing area T has been changed or whether the mouse has been operated (S210). The browsing area T being changed means that the user U has changed the line-of-sight direction, or has performed an enlargement or reduction operation. Specifically, the line-of-sight direction angle-of-view instruction unit 26 changes the operation of the user U to the line-of-sight direction and the angle of view and sends it to the image superimposing unit 33. Further, it may be determined whether or not the mouse is simply operated. Thereby, even if the browsing area T is not changed, the processing after step S220 proceeds with some mouse event as a trigger.

  The image superimposing unit 33 identifies the viewing area T based on the viewing direction and the angle of view (S220). That is, the latitude and longitude ranges corresponding to the viewing area T in the omnidirectional image CE are specified.

  Next, the determination unit 35 determines whether or not a part of the planar image P is included in the viewing area T (S230). If there is no entry at all, there is no need to reproduce the moving image, so the process proceeds to step S290.

  If at least a part of the planar image P is included in the viewing area T (Yes in S230), the determination unit 35 calculates the angle of view b of the planar image P (S240). Also, the viewing angle a of the viewing area is calculated (S250).

  Then, the determination unit 35 compares the ratio a / b between the angle of view a and the angle of view b with the threshold value 1.2, and determines whether the ratio a / b is larger than the threshold value 1.2 (S260).

  When the determination in step S260 is Yes (when it is greater than the threshold value of 1.2), since it is not estimated that the user U views the planar image P, the image switching unit 36 notifies the planar image reading unit 25 of moving image reading OFF ( S270).

  If the determination in step S260 is No (when the threshold value is equal to or smaller than the threshold value of 1.2), it is estimated that the user U views the planar image P. Therefore, the image switching unit 36 notifies the planar image reading unit 25 of the moving image reading ON. (S280).

  The image display unit 31 displays the superimposed image 305 in which the planar image P is superimposed as a moving image or a still image on the omnidirectional image CE on the display 508 (S290). If it is determined No in step S230, the planar image P is not superimposed.

  According to the above process, the moving image can be reproduced if the user U displays the planar image P in a large size, and the moving image can be stopped if the user U browses another place. When the moving image is reproduced, the image display unit 31 may display a menu for the user U to control the reproduction of the moving image, and the user U may accept the stop, reproduction, rewind, fast forward, and the like of the moving image.

  FIG. 24 is a diagram illustrating an example of a menu (UI: User Interface) on which a user operates to play a moving image. The menu 200 in FIG. 24 includes a play button 201, a pause button 202, a stop button 203, a fast forward button 206, a fast reverse button 205, a next moving image button 207, and a previous moving image button 204. In addition, a replay button or the like may be provided. The next moving image button 207 and the previous moving image button 204 are effective when the planar image P has a plurality of moving images. The design of each button is just an example.

  The menu 200 as shown in FIG. 24 may be displayed when a moving image is reproduced and when the user clicks on the moving image reproduction area, or when reproduction of the moving image ends.

  When the reproduction of the moving image of the planar image P is completed to the end, the planar image reading unit 25 may repeat reading of the planar image P from the beginning of the frame or may end reading of the planar image P.

<Judgment method of (ii)>
Next, a determination method of “(ii) When the area of the planar image P is equal to or larger than a predetermined angle of view” will be described. This determination is substantially the same as (i), and is based on the idea that when the user U displays the planar image P in a large size, it is estimated that he / she wants to see the planar image P. The determination method may use the coordinates of the vertices B1 to B4 of the planar image P as described with reference to FIG. Alternatively, the coordinates may be determined using coordinates on the display 508 obtained by perspective projection conversion.

  Alternatively, it may be estimated that the user wants to view the planar image P when the distance between the center of the planar image and the center of the viewing area T is less than the threshold, instead of the angle of view. As an example, the threshold value is a fraction of the viewing area T, such as 1/4 of the angle of view of the viewing area T.

  FIG. 25 is an example of a flowchart illustrating a procedure for controlling whether or not the display device 30 reproduces a moving image. In the description of FIG. 25, differences from FIG. 23 are mainly described. In FIG. 25, since the determination unit 35 does not need to calculate the angle of view a of the viewing area T, step S250 is not necessary.

  In step S260-2, the determination unit 35 determines whether or not the angle of view b is larger than the threshold value Th. When it is determined No in step S260-2 (when the angle of view b is equal to or smaller than the threshold), it is not estimated that the user U views the planar image P. Therefore, the image switching unit 36 sets the moving image reading OFF to the planar image reading unit 25. Notification is made (S270).

  When it is determined Yes in step S260-2 (when the angle of view b is larger than the threshold value), it is estimated that the user U views the planar image P. Therefore, the image switching unit 36 sets the moving image reading ON to the planar image reading unit 25. (S280).

  The threshold Th is, for example, an angle of view corresponding to the number of pixels that allows the user U to recognize the target. If the number of pixels of a general display 508 is 768 × 1024, the angle of view corresponding to the number of 500 pixels can be determined. By doing so, a moving image can be reproduced when the planar image P is captured to a certain extent.

<Judgment method of (iii)>
Next, a determination method of “(iii) When the user U clicks the planar image P” will be described. When the user U clicks on the planar image P, it becomes clear that the user U wants to view the planar image P, so that it is considered that a movie should be played even if the conditions (i) and (ii) are not satisfied. . In addition, when displaying on the apparatus provided with the touch panel, the same operation | movement is performed when the plane image P is touched. Further, since it is clear that the user U wants to view the planar image P, if the projection method conversion unit 2 displays the planar image P on the entire surface of the viewing area T, the user U operates so that the planar image P is in front. It is not necessary and convenient. Specifically, the display method changes the viewing area T from FIG. 22A to FIG. 22B when the user U clicks the planar image P in FIG. Such a display method will be described as automatic enlargement of the planar image P.

  The automatic enlargement of the planar image P will be described with reference to FIG. FIG. 26 is an example of a diagram illustrating a process of automatically expanding the planar image P to the browsing area T. It is assumed that the user U clicks in a state where the mouse cursor 311 is overlapped with the planar image P. In the case of a touch panel, it corresponds to the user U tapping the planar image P with a finger. A mouse cursor or touch panel is called a pointing device. Hereinafter, in order to simplify the description, the mouse cursor 311 will be described as an example.

  The line-of-sight direction angle-of-view instruction unit 26 converts the coordinates on the display 508 where the mouse cursor 311 is clicked into the three-dimensional coordinates of the omnidirectional image CE. This conversion corresponds to a conversion opposite to the perspective projection conversion. The determination unit 35 converts the three-dimensional coordinates of the mouse cursor 311 into latitude and longitude, and determines whether or not the coordinates of the mouse cursor 311 are included in the plane image P. Judgment is made as follows.

Longitude of vertex B1 ≦ longitude of coordinates of mouse cursor ≦ longitude of vertex B2, and
The latitude of the vertex B1 ≧ the latitude of the coordinates of the mouse cursor ≦ the latitude of the vertex B4 When the coordinates of the mouse cursor 311 are included in the planar image P, the determination unit 35 determines that the planar image P has been clicked.

  In this case, the image display unit 31 automatically enlarges the planar image P. The projection method conversion unit 2 performs image processing for gradually enlarging the planar image P to match the viewing area T. Specifically, a line 312 connecting the vertex A1 and the vertex B1, a line 312 connecting the vertex A2 and the vertex B2, a line 312 connecting the vertex A3 and the vertex B3, and a line 312 connecting the vertex A4 and the vertex B4 are equally spaced. Interpolate. In FIG. 26, the line 312 is divided into four. Interpolation points on the line 312 obtained by the interpolation are P1i to P4i (i is an integer of 1 to 4).

  The process of automatically enlarging the planar image P is a process of reducing the angle of view a of the viewing area T. The projection method conversion unit 2 reduces the viewing area T to an area connecting the interpolation points P13, P23, P33, and P43 (decreases the angle of view a). Next, the viewing area T is reduced to an area connecting the interpolation points P12, P22, P32, and P42. Next, the viewing area T is reduced to an area connecting the interpolation points P11, P21, P31, and P41. Next, the viewing area T is reduced to an area connecting the vertices B1, B2, B3, and B4 of the planar image P.

  By such image processing, it seems to the user U that the planar image P gradually increases. Since the perspective projection conversion is performed on the viewing area T and displayed on the entire surface of the display 508 (the entire surface of the display software), the user U can see the planar image P large.

  In addition, even when the entire planar image P is not included in the browsing area T, it can be automatically enlarged similarly. FIG. 27 is an example of a diagram illustrating the process of automatically enlarging the planar image P when the entire planar image P is not included in the browsing area T. In FIG. 27, only the vertex B4 of the planar image P is in the viewing area T. However, since the coordinates of the vertices B1 to B3 of the planar image P are known only by not being displayed in the viewing area T, they can be automatically enlarged as in FIG.

<Operation procedure>
FIG. 28 is an example of a flowchart illustrating a procedure for determining whether or not the display device 30 reproduces a moving image. In the description of FIG. 28, differences from FIG. 23 are mainly described.

  The processing in steps S210 and S220 may be the same as in FIG. Next, the determination unit 35 determines whether or not the planar image P is clicked (S222). If the determination in step S222 is No, the subsequent processing is the same as in FIG.

  When the determination in step S222 is Yes, the image switching unit 36 notifies the plane image reading unit 25 that the moving image reading is ON (S224). As a result, it is determined that there is moving image reproduction.

  Then, the projection method conversion unit 2 automatically enlarges the planar image P (S226). With the above processing, the user U does not need to operate so that the planar image P is in front.

  In addition, it is preferable that the user U can set whether to reproduce a moving image when the planar image P is clicked. After the moving image is reproduced, the moving image can be stopped when the user U displays another place and satisfies the condition of step S260. In this case as well, an operation menu may be displayed when a moving image is displayed.

<When the planar image P is a still image and the omnidirectional image CE is a moving image>
In the present embodiment, the planar image P is assumed to be a moving image and the omnidirectional image CE is assumed to be a still image. However, the same processing can be realized even if the planar image P is a still image and the omnidirectional image CE is a moving image. In this case, the omnidirectional image reading unit 27 may read the moving image frame of the omnidirectional image. For example, when the display device 30 displays an omnidirectional image in a museum hall, a picture is prepared as a flat image P, and an omnidirectional image CE in which the entire hall is imaged is prepared as a moving image. Since the attendees are not shown in the planar image P, the user U can enlarge and browse the planar image P and can easily grasp the atmosphere from the manner in which the attendees of the omnidirectional image CE move.

  In this case, the omnidirectional image is displayed as a moving image until the conditions (i) to (iii) are satisfied, and is displayed as a still image when the conditions (i) to (iii) are satisfied. Thereby, when the user U displays the planar image P in a large size, the omnidirectional image becomes a still image, so that it is easy to see.

  Further, both the planar image P and the omnidirectional image CE may be moving images. In this case, the omnidirectional image CE and the planar image P are displayed as moving images until the conditions (i) to (iii) are satisfied, and the omnidirectional image is displayed when the conditions (i) to (iii) are satisfied. CE and planar image P are displayed as still images. Thereby, when the user U displays the planar image P in a large size, the omnidirectional image CE becomes a still image so that it is easy to see.

<Video playback metadata>
In FIG. 24, it has been described that the user can operate the reproduction of the moving image by using the menu 200. However, if there is metadata regarding the reproduction of the moving image, the display device 30 can reproduce the moving image according to the metadata. Video playback metadata is data that specifies how to play back a video.

FIG. 29 is an example of a diagram for explaining metadata of moving image reproduction. Video playback metadata includes playback start time, playback end time, playback direction, playback speed, presence / absence of repetition, number of repetitions, repetition method, and state after stop. This will be described with reference to FIGS. 29 (a) to 29 (c) as appropriate.
Playback start time As shown in FIG. 29 (a), this is a parameter for playing a moving image from the middle instead of from the beginning. It is specified by time or frame number. The initial value when the playback start time is not specified is set to zero.
-Playback end time This is a parameter for ending the playback of a movie halfway rather than the end. It is specified by time or frame number. The initial value when the playback start time is not specified is the length of the moving image. The movie playback time and movie end time are specified when only a part of the movie file is to be played. The actually reproduced moving image is between the reproduction start time and the reproduction end time.
-Playback direction Indicates the playback direction at the start of video playback. There are forward and reverse directions.
• Playback speed slow playback and fast forward playback parameters. The normal playback speed is 1.0, and when it is less than 1 (for example, 0.5), it is slow playback, and when it is greater than 1 (for example, 2.0), fast-forward playback.
-Repetition presence / absence This parameter specifies whether or not to repeat the reproduction when reproducing the video. FIG. 29B shows a state in which reproduction is repeated from the head to the tail. If there is no repetition, playback of the video ends when playback starts from the beginning to the end. If the playback direction is the reverse direction and there is no repetition, playback of the moving image ends when playback from the tail to the beginning is performed.
-Number of repetitions This is the number of repetitions of video playback. When the specified number of times is repeated, the video playback ends. An infinite number of repetitions can be set as the number of repetitions. In this case, the moving image is repeatedly reproduced until a stop condition is satisfied.
-Repeat method This parameter specifies the playback method for repeated playback. There are loop playback and alternate playback. Loop playback is playback in the same direction, and alternate playback refers to reversing the playback direction for each playback as shown in FIG. 29 (c). In both loop playback and alternate playback, when the playback direction is reverse playback, the playback direction is reversed.
-State after stop This parameter specifies the state when playback of the video stops. As it is, there are a head (the tail in the case of reverse playback) and a transparent state.

  Therefore, by attaching metadata to the omnidirectional image, the image superimposing server 70 can control the reproduction of the moving image on the display device 30.

  As shown in FIG. 3C, in the configuration in which the image superimposing server 70 and the display device 30 communicate with each other, at least one of the image superimposing server 70 or the display device 30 can store metadata together with the position parameter.

<Summary>
As described above, according to the display device 30 of the present embodiment, a moving image is reproduced when it is estimated that the user views the planar image P. Therefore, when the user wants to view the planar image P, the moving image can be reproduced. , Unnecessary video playback can be suppressed. In addition, the user can view the moving image from the beginning. In addition, since the reproduction of the moving image starts when the user enlarges the planar image P, it seems that the screen suddenly starts moving, and the user's attention can be drawn.

  In the present embodiment, a display device 30 that displays a frame indicating that the planar image P exists together with the planar image P on the omnidirectional image CE will be described. In the present embodiment, the constituent elements described with the same reference numerals in the first embodiment perform the same functions, and therefore, description of the constituent elements once described may be omitted or only differences may be described.

  It may be difficult to understand where the planar image P is superimposed on the omnidirectional image CE. In addition, when the still image planar image P is also superimposed, it is difficult for the user U to determine which of the moving image and the still image is the planar image P. Therefore, it is effective that the image superimposing unit 33 displays the frame 40 indicating that the planar image P exists. If the mode of the frame 40 is changed between the moving image and the still image, the user U can determine whether the moving image is a still image or not by looking at the frame 40.

  The moving image and the still image planar image P have higher image quality than the omnidirectional image CE, and when the user U enlarges the omnidirectional image CE and displays the larger planar image P, the image quality becomes higher (for example, higher resolution). You can view images.

  The frame 40 can be displayed in the same manner as the projection method conversion of the planar image P. If the projection method conversion unit 1 performs the projection method conversion of the rectangle at the outer edge of the planar image P, the frame 40 is obtained. The projection method conversion unit 1 performs the projection method conversion of the rectangle at the outer edge in the same manner as the case where the planar image P is subjected to the projection method conversion using the position parameter PP.

  FIG. 30 is an example of a diagram schematically illustrating the frame 40 obtained by the projection method conversion. The projection system conversion unit 1 prepares a transparent layer 308 having the same size (number of pixels) as the omnidirectional image CE of equirectangular projection. The rectangle 321 at the outer edge of the planar image P is converted into a projection system for this layer. It should be noted that a dotted line serving as the frame 40 may be drawn at the latitude and longitude of the lattice having the values x = 0.5, −0.5 or y = 0.5, −0.5 with the position parameter PP.

  The projection method conversion unit 1 changes the mode of the frame 40 according to a moving image and a still image. For example, the color is changed to red for a moving image and blue for a still image. In addition, the line type may be changed such as a solid line in the case of a moving image and a dotted line in the case of a still image. You may change the thickness of the line. A moving image or a still image may be displayed in text near the frame 40. The planar image reading unit 25 notifies the projection method conversion unit 1 whether the planar image P is a moving image or a still image. The distinction between a moving image and a still image may be determined by, for example, a file extension (jpeg is a still image, mpeg is a moving image), or may be determined from the file name of the planar image P.

  FIG. 31 is an example of a diagram that schematically illustrates the overlap of the frames 40. In addition to the display image 304, the projection method conversion image 302, and the mask data 303, the transparent layer 308 is input to the image superimposing unit 33. The method of superimposing the projection method conversion image 302 on the display image 304 has been described with reference to FIG.

  The image superimposing unit 33 uses the mask data 303 to extract an area corresponding to the white pixel of the mask data 303 from the transparent layer 308. Then, the frame 40 representing the outer edge of the planar image P is displayed by superimposing the extracted transparent layer 308 on the superimposed image 305.

  For example, the display and non-display of the frame 40 may be switched by the operation of the user U. In this case, the image superimposing unit 33 switches between display and non-display of the transparent layer according to the operation of the user U. Further, when the mouse cursor overlaps with the planar image P, the frame 40 of the overlapping planar image P may be displayed.

  FIG. 32 is a diagram illustrating an example of a frame 40 displayed in a different manner between a moving image and a still image. The moving image is surrounded by a dotted frame 40a, while the still image is surrounded by a solid frame 40b. The user can easily determine whether it is a moving image or a still image. Further, the position and size of the planar image P can be determined.

  FIG. 33 is an example of a flowchart illustrating a procedure in which the display device 30 displays the frame 40 of the planar image P on the omnidirectional image CE. In the description of FIG. 33, differences from FIG. 17 are mainly described. Processing in steps S10 to S30 is the same as that in FIG. However, in step S30, the position parameter calculation unit 23 calculates the position parameters PP of all the planar images P.

  The next steps S40 to S90 may be the same as in FIG. Subsequent to step S90, the projection method conversion unit 1 prepares the outer edge rectangle 321 of the planar image P depending on whether it is a moving image or a still image, and performs the projection method conversion of the outer edge rectangle 321 to the transparent layer 308 with the position parameter PP ( S92). Thereby, different frames 40 are obtained in the transparent layer 308 depending on the moving image and the still image.

  Next, the image superimposing unit 33 generates a superimposed image 305 (S100), and extracts the frame 40 from the transparent layer 308 using the mask data 303 (S102).

  The image superimposing unit 33 determines whether all the planar images P are superimposed (S104). Until all the planar images P are superimposed, the process returns to step S70 and steps S70 to S102 are executed.

  As described above, the display device 30 can superimpose the planar image P on the omnidirectional image CE, and can display the frame 40 having different modes for the moving image and the still image. The frame 40 is preferably deleted when the above conditions (i) to (iii) are satisfied. By erasing, the user U is not hindered from seeing the planar image P.

  In addition, the image superimposing unit 33 may display the frame 40 only when the user U overlays the mouse cursor on the planar image P. Conversely, when the user U does not overlay the mouse cursor on the planar image P. It may be displayed only for. The user U can switch between displaying and hiding the frame 40 by moving the mouse cursor. Further, the image superimposing unit 33 may display the frame 40 until a predetermined time elapses after the viewing area T is changed, and then delete the frame 40.

  Further, the focal length may be displayed on the frame 40. Since the frame 40 and the focal length information are displayed near the plane image P, the user U can grasp how much the plane image P of the focal length is superimposed.

<Summary>
As described above, according to the display device 30 of the present embodiment, the position where the planar image P is superimposed can be displayed with the frame 40 or the like. In addition, the user can determine a moving image and a still image.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, although the above embodiment automatically switches between playback and stop of a moving image, the user U may intentionally specify playback / stop of a moving image.

  Further, the display of the omnidirectional image may be performed by browser software, or may be performed by application software for displaying the omnidirectional image CE.

  In the configuration of FIG. 3C, the image superimposing server 70 performs the process up to the creation of the display position parameter, and returns the created display position parameter and frame ID to the display device 30 as metadata. The device 30 may display using the metadata, the omnidirectional image, and the planar image (including moving images).

  Further, the omnidirectional image of the present embodiment may be image data having an angle of view that cannot be displayed in the viewing area T. For example, an image having an angle of view of 180 degrees to 360 degrees only in the horizontal direction may be used.

  12 and 18 are divided according to main functions in order to facilitate understanding of processing by the omnidirectional camera 20, the display device 30, and the digital camera 9. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the omnidirectional camera 20, the display device 30, and the digital camera 9 can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes. 12 and 18, either the display device 30 or the image superimposing server 70 may perform part or all of the processing.

  For example, the processing up to the image superimposing unit 33 may be performed by the image superimposing server 70, and the image display unit 31 may be a separate display device 30.

  The determination unit 35 is an example of a determination unit, the image switching unit 36 is an example of a display switching unit, the line-of-sight direction angle-of-view instruction unit 26 is an example of an operation reception unit, and the projection method conversion unit 2 is an image display unit. The image superimposing unit 33 is an example of a superimposed image display unit.

7 conversion display unit 8 position parameter creation unit 9 digital camera 20 omnidirectional camera 23 position parameter calculation unit 24 position parameter writing unit 26 gaze direction angle of view instruction unit 30 display device 32 image superimposing unit 33 image superimposing unit 35 determining unit 36 Image switching unit 100 Image processing system

Japanese Patent Laid-Open No. 2006-96487

Claims (10)

A display device that displays the first image in which a second image that can be displayed as a moving image is superimposed on the first image,
Determining means for determining whether or not the second image is viewed;
Display switching means for switching whether to reproduce the second image as a moving image or display as a still image according to the determination result of the determination means;
A display device. When the determination means determines that the second image is browsed, the display switching means reproduces the second image as a moving image,
The display device according to claim 1, wherein when the determination unit determines that the second image is not browsed, the display switching unit displays the second image as a still image. The determination means compares the ratio of the viewing area of the first image displayed on the display device with respect to the area where the second image is displayed, with a threshold,
The display switching unit displays the second image as a still image when the ratio is greater than a threshold value, and displays the second image as a moving image when the ratio is equal to or less than the threshold value. The display device described. The determination means compares the size of the second image displayed on the display device with a threshold value,
When the size is larger than the threshold, the display switching means displays the second image as a moving image,
The display device according to claim 1, wherein when the size is equal to or smaller than a threshold value, the second image is displayed as a still image. Operation receiving means for receiving an operation on the first image;
The determination means determines whether or not the coordinates received by the operation reception means overlap the second image,
The display device according to claim 1, wherein the display switching unit displays the second image as a moving image when the determination unit determines that the coordinates overlap with the second image. When the determination unit determines that the coordinates overlap the second image when the second image is displayed on the display device,
The display device according to claim 5, further comprising an image display unit that gradually enlarges and displays the second image to a display size of the display device. The second image that can be displayed as a moving image and the second image that can be displayed only as a still image are superimposed on the first image,
7. The superimposing image display means for displaying that the second image that can be displayed as a moving image and the second image that can be displayed only as a still image are superimposed, respectively. The display device described.   2. The reproduction method of the second image, wherein one or more of a reproduction start time, a reproduction end time, a reproduction direction, a reproduction speed, whether or not to repeat, the number of repetitions, and a repetition method are designated. The display apparatus of any one of -7. A display device that displays the first image in which a second image that can be displayed as a moving image is superimposed on the first image;
Determining means for determining whether or not the second image is viewed;
Display switching means for switching whether the second image is reproduced as a moving image or displayed as a still image according to a determination result of the determination means;
Program to function as. A display method performed by a display device that displays the first image in which a second image that can be displayed as a moving image is superimposed on the first image,
A step of determining whether or not the second image is viewed;
A step of switching whether the display switching means reproduces the second image as a moving image or displays it as a still image according to the determination result of the determination means;
Display method.