JP7424031B2 - Communication terminal, photographing system, image processing method and program - Google Patents

Description

The present invention relates to a communication terminal, a photographing system, an image processing method, and a program.

Instead of a camera system consisting of a pan/tilt and zoom lens using a rotating table, a fisheye lens etc. is used to photograph the subject, and the image data obtained is distributed. Techniques for generating and displaying images, ie electronic pan-tilt-zoom (PTZ) techniques, are already known.

However, the technology that distributes image data obtained by photographing a subject has the problem of increasing the amount of data communication because it also distributes image data of areas that are not attracting attention.

On the other hand, in order to reduce the amount of communication, the photographing device delivers a low-definition overall image that is a reduced version of the entire wide-angle image, and data of a high-definition partial image that is the area of interest within the wide-angle image. A technique has been disclosed in which a partial image is inserted and synthesized into the entire image on the receiving side (see Patent Document 1).

However, when the entire image and partial images are moving images, it is difficult to reduce the amount of data communication even if the entire image is made low-definition. On the other hand, viewers do not always carefully view videos, and there is a need to be able to carefully view only partial images when a video of interest is played.

The present invention has been made in view of the above-mentioned circumstances, and is designed to suppress the amount of data communication for wide-angle videos such as whole images and narrow-angle videos such as partial images, while preventing the reduction in definition of partial images as much as possible. The purpose is to enable the viewer to view the area of interest more clearly.

The invention according to claim 1 is a communication terminal that displays a moving image by receiving the moving image data transmitted by a photographing device that obtains moving image data of a predetermined definition by photographing a subject, a receiving means for receiving video data; a display control means for displaying a wide-angle video that is all or a part of the video related to the video data on a display means; and a display control means for enlarging the wide-angle video displayed on the display means. a reception means for accepting an operation; and when the reception means accepts a predetermined enlargement of the wide-angle video displayed on the display means, the reception means displays a part of the subject to the photographing device; transmitting means for transmitting request information for requesting still image data whose definition is higher than that of video data; Receiving the image data, the display control means displays the wide-angle video in a composite state with the narrow-angle still image related to the still image data generated when the accepting means accepts a predetermined enlargement. This is a communication terminal characterized by displaying information on the means.

As explained above, according to the embodiments of the present invention, while suppressing the amount of data communication for wide-angle videos such as whole images and narrow-angle videos such as partial images, the definition of partial images is not lowered as much as possible. This has the effect that the viewer can more clearly view the area of interest.

(a) is a left side view of the omnidirectional imaging device, (b) is a rear view of the omnidirectional imaging device, (c) is a plan view of the omnidirectional imaging device, and (d) is a top view of the omnidirectional imaging device. FIG. 3 is a bottom view of the omnidirectional photographing device. FIG. 3 is an image diagram of how the omnidirectional photographing device is used. (a) is a hemisphere image taken with a spherical camera (front), (b) is a hemisphere image taken with a spherical camera (back), and (c) is expressed using equirectangular projection. It is a diagram showing an image. (a) is a conceptual diagram showing a state where a sphere is covered by an equirectangular projection image, and (b) is a diagram showing a celestial sphere image. FIG. 7 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. (a) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a state in which an image of a predetermined area is displayed on a display of a communication terminal. FIG. 3 is a diagram illustrating an outline of partial image parameters. 1 is a schematic diagram of a photographing system according to the present embodiment. FIG. 2 is a hardware configuration diagram of a celestial sphere photographing device. FIG. 3 is a hardware configuration diagram of an intermediary terminal. It is a hardware configuration diagram of a smartphone. 1 is a hardware configuration diagram of an image management system. FIG. 2 is a functional block diagram of the omnidirectional photographing device according to the present embodiment. It is a functional block diagram of the smartphone of this embodiment. FIG. 2 is a functional block diagram of an image management system. (a) A conceptual diagram of horizontal and vertical angle of view threshold information, and (b) an explanatory diagram of a horizontal angle of view and a vertical angle of view. FIG. 2 is a sequence diagram showing the process of generating and playing back each data of a whole moving image and a partial moving image. FIG. 2 is a conceptual diagram of an image in the process of image processing performed by the omnidirectional photographing device. FIG. 3 is a diagram illustrating partial image parameters. FIG. 2 is a conceptual diagram showing a state in which frame images of a whole video and a partial video are combined. It is a conceptual diagram of an image in the process of image processing performed by a smartphone. FIG. 3 is a diagram illustrating creation of a partial three-dimensional sphere from a partial plane. It is a two-dimensional conceptual diagram when a partial image is superimposed on a celestial sphere image without performing partial three-dimensional sphere creation of this embodiment. It is a two-dimensional conceptual diagram when partial three-dimensional sphere creation of this embodiment is performed and a partial image is superimposed on a celestial sphere image. (a) Example of wide image display without superimposed display, (b) Display example of tele image without superimposed display, (c) Display example of wide image with superimposed display, (d) Example of display of wide image with superimposed display FIG. 2 is a conceptual diagram showing a display example of a teleimage. FIG. 3 is a diagram showing the configuration of video playback metadata. FIG. 2 is a sequence diagram showing processing for recording a moving image and creating metadata. FIG. 3 is a sequence diagram showing a process of generating a partial still image and updating metadata. FIG. 6 is a diagram illustrating an example of an operation method when requesting generation of a still image. It is a timing chart at the time of video shooting and still image generation. It is a timing chart at the time of video playback and still image playback. FIG. 2 is a sequence diagram showing processing for playing back moving images and still images. It is a figure showing an example of a selection button and a decision button. FIG. 2 is a two-dimensional conceptual diagram showing a display area and an attention area. This is an example of enlarging a video and displaying a high-definition still image superimposed on the video. This is an example of a display pattern (display pattern 1) corresponding to an enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image. This is an example of another display pattern (display pattern 2) regarding the enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image. This is an example of another display pattern (display pattern 3) regarding the enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image. This is an example of another display pattern (display pattern 4) regarding the enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image. This is an example of another display pattern (display pattern 5) regarding the enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image.

Embodiments of the present invention will be described below with reference to the drawings.

[Overview of embodiment]
An overview of this embodiment will be described below.

A method for generating a spherical image will be described with reference to FIGS. 1 to 6.

First, the appearance of the omnidirectional photographing device 1 will be explained using FIG. 1. The spherical photographing device 1 is a digital camera for obtaining photographed images that are the basis of a spherical (360°) panoramic image. Note that FIG. 1(a) is a right side view of the omnidirectional imaging device 1, FIG. 1(b) is a front view of the omnidirectional imaging device 1, and FIG. 1(c) is a right side view of the omnidirectional imaging device 1. 1, and FIG. 1(d) is a bottom view of the omnidirectional photographing device 1.

As shown in FIGS. 1(a), 1(b), 1(c), and 1(d), the upper part of the omnidirectional photographing device 1 has a fisheye lens 102a on the front side (front side). Inside the omnidirectional photographing device 1, which is provided with a fisheye lens 102b on the back side (rear side), image sensors (image sensors) 103a and 103b, which will be described later, are provided. By photographing objects (including landscapes) using the camera, a hemispherical image (angle of view of 180° or more) can be obtained. A shutter button 115a is provided on the surface opposite to the front side of the omnidirectional photographing device 1. Furthermore, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the omnidirectional shooting device 1. The shutter button 115a, the power button 115b, and the Wi-Fi button 115c are all turned on and off each time they are pressed. Further, each time the shooting mode switching button 115d is pressed, the shooting mode for still images, the shooting mode for moving images, and the mode for distributing moving images are switched. Note that the shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are a type of operation section 115, and the operation section 115 is not limited to these buttons.

Furthermore, a tripod screw hole 151 is provided at the center of the bottom portion 150 of the omnidirectional photographing device 1 for attaching the omnidirectional photographing device 1 to a camera tripod. Furthermore, a Micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom portion 150. An HDMI (High-Definition Multimedia Interface) terminal is provided on the right end side of the bottom portion 150. Note that HDMI is a registered trademark.

Next, using FIG. 2, the usage status of the omnidirectional photographing device 1 will be explained. Note that FIG. 2 is an image diagram of how the omnidirectional photographing device 1 is used. As shown in FIG. 2, the omnidirectional photographing device 1 is used, for example, by a user while holding it in his or her hand to photograph objects around the user. In this case, two hemispherical images can be obtained by capturing images of objects around the user using the imaging device 103a and the imaging device 103b shown in FIG. 1, respectively.

Next, using FIGS. 3 and 4, an outline of the process from which an equirectangular projection image EC and a celestial sphere image CE are created from an image photographed by the celestial sphere photographing device 1 will be explained. In addition, FIG. 3(a) is a hemispherical image (front side) taken by the omnidirectional imaging device 1, FIG. 3(b) is a hemispherical image (rear side) taken by the omnidirectional imaging device 1, and FIG. c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). FIG. 4(a) is a conceptual diagram showing a state where a sphere is covered by an equirectangular projection image, and FIG. 4(b) is a diagram showing a spherical image.

As shown in FIG. 3A, the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a, which will be described later. Further, as shown in FIG. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b, which will be described later. The hemisphere image (front side) and the hemisphere image (rear side) inverted by 180 degrees are combined by the omnidirectional imaging device 1, and as shown in FIG. 3(c), an equirectangular projection is performed. Image EC is created.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the spherical surface, as shown in Figure 4(a). A spherical image CE as shown in b) is created. In this way, the spherical image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image. In this embodiment, unless otherwise specified, "image" includes both "still image" and "video".

As described above, since the spherical image CE is an image pasted to cover a spherical surface, it feels strange when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as a "predetermined area image") of a part of the spherical image CE as a flat image with less curvature, it is possible to perform a display that does not give a sense of discomfort to humans. This will be explained using FIGS. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the viewer who views the spherical image CE displayed as a three-dimensional solid sphere. Further, FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined area image when displayed on a display. Further, in FIG. 6(a), the omnidirectional image shown in FIG. 4 is represented by a three-dimensional solid sphere CS. If the spherical image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC will be located inside the spherical image CE, as shown in FIG. The predetermined area T in the spherical image CE is the imaging area of the virtual camera IC, and the position coordinates (x(rH), y (rV), angle of view α (angle)). Note that zooming of the predetermined area T can be expressed by expanding or contracting the range (circular arc) of the angle of view α. Furthermore, zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. Here, the predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE.

The predetermined area image Q shown in FIG. 6(a) is displayed on a predetermined display as an image of the imaging area of the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by predetermined area information that is initially set (default setting). Note that the predetermined area image may be indicated by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the predetermined area information or the position coordinates of the virtual camera IC.

FIG. 7 is a diagram illustrating an outline of partial image parameters. Here, a method for specifying a part of the spherical image will be explained. In a spherical image, the center point CP of a partial image can be expressed by the direction of the camera that takes the image (here, the virtual camera IC). The center of the entire image, which is a spherical image, is the front of the entire image, and the azimuth angle is "aa" and the elevation angle is "ea". Further, in order to express the range of the partial image, it is expressed using, for example, a diagonal angle of view α, which is an angle of view in a diagonal direction. Further, in order to represent the range in the vertical and horizontal directions, it is possible to represent the image aspect ratio (width w÷height h). Here, the diagonal angle of view and aspect are used to express the range, but vertical angle of view and horizontal angle of view, vertical angle of view and aspect ratio, etc. may also be used. Further, in addition to the azimuth angle and the elevation angle, a rotation angle may also be used.

[Outline of the shooting system]
First, the outline of the configuration of the imaging system of this embodiment will be described using FIG. 8. FIG. 8 is a schematic diagram of the configuration of the imaging system of this embodiment.

As shown in FIG. 8, the photographing system of this embodiment includes a spherical photographing device 1, an intermediary terminal 3, a smartphone 5, and an image management system 7. In addition, in the photographing system of this embodiment, the omnidirectional photographing device 1 is remotely controlled by a predetermined operation by a user A (also referred to as a viewer or an operator) of the smartphone 5, for example.

Among these, the spherical photographing device 1 is a special digital camera that photographs a subject and obtains two hemispherical images that are the basis of a spherical (panoramic) image, as described above. In this embodiment, the omnidirectional photographing device 1 can function as a photographing device that performs a stream video distribution service in which wide-angle images are photographed and distributed. Further, as will be described later, the omnidirectional photographing device 1 receives a still image request acquired by the smartphone 5 (an example of a request based on a predetermined operation) and sends it to the image management system 7 (an example of an image management device). , has a function of generating still image data based on a still image data request transmitted from the image management system 7 and transmitting it to the image management system 7.

The intermediary terminal 3 is an example of a communication terminal that can directly communicate with a communication network 100 such as the Internet. The intermediary terminal 3 performs short-range wireless communication with the omnidirectional imaging device 1 that cannot directly communicate with the communication network 100, thereby acting as an intermediary for communication between the omnidirectional imaging device 1 and the image management system 7. Near field communication is, for example, communication performed using technologies such as Wi-Fi, Bluetooth (registered trademark), and NFC (Near Field Communication). In FIG. 8, the intermediary terminal 3 is used like a dongle receiver in which the omnidirectional photographing device 1 is installed.

The smartphone 5 can communicate with the image management system 7 via the communication network 100 by wire or wirelessly. Furthermore, the smartphone 5 can display images acquired from the omnidirectional photographing device 1 on a display 517, which will be described later, provided on the smartphone 5.

The image management system 7 is constructed by a computer, and transmits a request for a moving image or a still image from the smartphone 5 to the omnidirectional photographing device 1 via the intermediary terminal 3. Further, the image management system 7 transmits the video data and still image data sent from the omnidirectional photographing device 1 via the intermediary terminal 3 to the smartphone 5.

Although FIG. 1 shows only one set of the omnidirectional photographing device 1 and the intermediary terminal 3, a plurality of sets may be used. Further, although only one smartphone 5 is shown, a plurality of smartphones 5 may be used. The smartphone 5 is an example of a communication terminal. Communication terminals include, in addition to the smartphone 5, a PC (Personal Computer), a smart watch, a game device, a car navigation terminal, and the like. Furthermore, the image management system 7 may be constructed not only by a single computer but also by a plurality of computers.

[Hardware configuration of embodiment]
Next, the hardware configurations of the omnidirectional photographing device 1, the intermediary terminal 3, the smartphone 5, and the image management system 7 of this embodiment will be described in detail using FIGS. 9 and 12.

<Hardware configuration of spherical imaging device>
First, the hardware configuration of the omnidirectional photographing device 1 will be explained using FIG. 9. FIG. 9 is a hardware configuration diagram of the omnidirectional photographing device. In the following, the spherical photographing device 1 is assumed to be a spherical (omnidirectional) photographing device capable of photographing at 4π radians using two image sensors, but any number of image sensors may be used. Also, it does not necessarily have to be a device exclusively for omnidirectional photography, but by attaching an aftermarket omnidirectional imaging unit to a normal digital camera, smartphone, etc., it is possible to have substantially the same functions as the omnidirectional photography device 1. You may also do so.

As shown in FIG. 9, the omnidirectional photographing device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM ( Read Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I/F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, acceleration sensor 120 and a terminal 121.

Among these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by fisheye lenses 102a and 102b into electrical signal image data and output the image data. It has a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 via a serial I/F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Further, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, a network I/F 116, a communication unit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a terminal 121, and the like.

The image processing unit 104 takes in image data output from the image sensors 103a and 103b through the parallel I/F bus, performs predetermined processing on each image data, and then synthesizes these image data. , create equirectangular cylindrical projection image data as shown in FIG. 3(c).

Generally, the imaging control unit 105 uses the I2C bus to set commands and the like in register groups of the imaging devices 103a and 103b, using the imaging control unit 105 as a master device and the imaging devices 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to take in status data and the like of the register groups of the imaging elements 103a and 103b, and sends it to the CPU 111.

Further, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the omnidirectional photographing device 1, it may have a preview display function or a function corresponding to video display on a display (for example, the display 517 of the smartphone 5). In this case, image data is continuously output from the image sensors 103a and 103b at a predetermined frame rate (frames/second).

The imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111, as will be described later. Note that in this embodiment, the omnidirectional photographing device 1 is not provided with a display, but may be provided with a display section.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I/F bus, and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the omnidirectional photographing device 1 and executes necessary processing. Note that the CPU 111 may be a single CPU or a plurality of CPUs. ROM112 stores various programs for CPU111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data that is being processed by the image processing unit 104 and data of processed equirectangular projection images.

The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user inputs various shooting modes, shooting conditions, etc. by operating the operation unit 115.

The network I/F 116 is a general term for interface circuits (USB I/F, etc.) with external media such as an SD card, a personal computer, and the like. Further, the network I/F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 may be recorded on an external media via this network I/F 116, or may be transferred to an external terminal (device) such as a smartphone 5 via the network I/F 116 as necessary. ).

The communication unit 117 communicates with an external terminal (device) such as a smartphone 5 via an antenna 117a provided in the omnidirectional photographing device 1 using short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth. . This communication unit 117 can also transmit the data of the equirectangular projection image to an external terminal (device) such as the smartphone 5.

The electronic compass 118 calculates the direction of the omnidirectional photographing device 1 from the earth's magnetism and outputs direction information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of a captured image. Note that the related information also includes data such as the date and time when the image was taken and the data capacity of the image data.

The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) accompanying movement of the omnidirectional imaging device 1. A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image.

The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The omnidirectional imaging device 1 calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device 1) based on the acceleration detected by the acceleration sensor 120. By providing the omnidirectional photographing device 1 with both the gyro sensor 119 and the acceleration sensor 120, the accuracy of image correction is improved.

As the terminal 121, for example, a concave terminal for Micro USB is used.

<Hardware configuration of intermediary terminal>
Next, the hardware configuration of the intermediary terminal 3 will be explained using FIG. 10. Note that FIG.
It is a hardware configuration diagram of the intermediary terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 10, the intermediary terminal 3 operates according to the control of the CPU 301 which controls the entire operation of the intermediary terminal 3, the ROM 302 which stores basic input/output programs, the RAM 303 used as a work area for the CPU 301, and the CPU 301. It is equipped with an EEPROM (Electrically Erasable and Programmable ROM) 304 that reads or writes data, and a CMOS sensor 305 as an image sensor that images a subject and obtains image data under the control of the CPU 301.

Note that the EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data. Further, a CCD sensor may be used instead of the CMOS sensor 305.

Furthermore, the intermediary terminal 3 includes an antenna 313a, a communication unit 313 that communicates with the image management system 7 via the communication network 100 by using a wireless communication signal using the antenna 313a, and a GPS (Global Positioning Systems) satellite or indoor GPS. A GPS receiving section 314 receives a GPS signal including location information (latitude, longitude, and altitude) of the intermediary terminal 3 using an IMES (Indoor MEssaging System), and an address bus and an address bus for electrically connecting each of the above sections. A bus line 310 such as a data bus is provided.

<Hardware configuration of smartphone>
Next, the hardware of the smartphone will be explained using FIG. 11. FIG. 11 is a hardware configuration diagram of the smartphone. As shown in FIG. 11, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I/F 513a, an acceleration/direction sensor 506, a media I/F 508, and a GPS receiver 509. There is.

Among these, the CPU 501 controls the operation of the smartphone 5 as a whole. Note that the CPU 501 may be a single CPU or a plurality of CPUs. The ROM 502 stores programs used to drive the CPU 501, such as the CPU 501 and an IPL (Initial Program Loader). RAM 503 is used as a work area for CPU 501. The EEPROM 504 reads or writes various data such as smartphone programs under the control of the CPU 501. The CMOS sensor 505 photographs a subject (mainly a self-portrait) under the control of the CPU 501 and obtains image data. The image sensor I/F 513a is a circuit that controls driving of the CMOS sensor 512. The acceleration/direction sensor 506 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect geomagnetism. The media I/F 508 reads or writes data to or from the recording medium 507 such as a flash memory.
control). GPS receiving section 509 receives GPS signals from GPS satellites.

The smartphone 5 also includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I/F 513b, a microphone 514, a speaker 515, a sound input/output I/F 516, a display 517, an external device connection I/F 518, and a short distance It includes a communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521.

Among these, the long distance communication circuit 511 is a circuit that communicates with other devices via a communication network such as the Internet. The CMOS sensor 512 is a type of built-in imaging means that photographs a subject and obtains image data under the control of the CPU 501. The image sensor I/F 513b is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a type of built-in sound collecting means for inputting audio. The sound input/output I/F 516 is a circuit that processes input and output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as a liquid crystal or organic EL display that displays an image of a subject, various icons, and the like. External device connection I/F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 521 is a type of input means by which the user operates the smartphone 5 by pressing the display 517.

The smartphone 5 also includes a bus line 510. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501.

<Hardware configuration of image management system>
Next, the hardware configuration of the image management system will be described using FIG. 12. Figure 1
2 is a hardware configuration diagram of the image management system.

FIG. 12 is a hardware configuration diagram of the image management system 7. As shown in FIG. As shown in FIG. 12, the image management system 7 is constructed by a computer, and includes a CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) controller 705, It includes a display 708, a media I/F 707, a network I/F 709, a bus 710, a keyboard 711, a mouse 712, and a DVD-RW (Digital Versatile Disk Rewritable) drive 714.

Among these, the CPU 701 controls the operation of the entire image management system 7. The ROM 702 stores programs used to drive the CPU 701 such as IPL. RAM 703 is used as a work area for CPU 701. The HD 704 stores various data such as programs. The HDD controller 705 controls reading and writing of various data to the HD 704 under the control of the CPU 701. The display 708 displays various information such as a cursor, menu, window, characters, or images. A media I/F 707 controls reading or writing (storage) of data to a recording medium 706 such as a flash memory. The network I/F 709 is an interface for data communication using the communication network 100. The bus 710 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 701 shown in FIG. 12.

Further, the keyboard 711 is a type of input means that includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 714 controls reading and writing of various data on a DVD-RW 713, which is an example of a removable recording medium. Note that the disc is not limited to DVD-RW, but may be DVD-R, Blu-ray Disc, or the like.

[Functional configuration of embodiment]
Next, the functional configuration of this embodiment will be explained using FIGS. 13 to 16. Note that the main function of the intermediary terminal 3 is a transmitting/receiving section for intermediating communication between the omnidirectional photographing device 1 and the image management system 7, and therefore a description thereof will be omitted.

<Functional configuration of spherical photography device>
FIG. 13 is a functional block diagram of the omnidirectional photographing device of this embodiment. The omnidirectional photographing device 1 includes a transmitting/receiving section 11, a partial image parameter creation section 12, an imaging control section 13, imaging sections 14a and 14b, an image processing section 15, a temporary storage section 16, a definition change section 17, and a projection method conversion section. 18, a combining section 19, a moving image encoding section 20a, a still image encoding section 20b, a receiving section 22, a determining section 25, and a still image storage section 29. Among these, the definition change section 17 includes a low definition change section 17a and a low definition change section 17b. The low-definition changing unit 17a generates an entire video (low-definition) in which the definition is lowered for the ultra-high-definition video data stored in the temporary storage unit 16. On the other hand, the low-definition changing unit 17b converts the output image sent from the projection method converting unit 18, which will be described below, to the ultra-high-definition video data stored in the temporary storage unit 16. It is generated as a partial video (high definition) with a reduced rate of reduction in definition (relatively higher definition than the low definition changing section 17a). Note that the low-definition change section 17a is an example of a first low-definition change section, and the low-definition change section 17b is an example of a second low-definition change section. The projection method conversion unit 18 generates a partial still image (ultra high definition) by converting the projection method from the ultra high definition video data stored in the temporary storage unit 16. Each of these parts is realized by one of the components shown in FIG. 9 operating in response to an instruction from the CPU 111 according to a program for the spherical photographing device developed from the SRAM 113 onto the DRAM 114. It is a function or means. The omnidirectional photographing device 1 also includes a threshold management section 1001 implemented by an SRAM 113. As mentioned above, the definition conversion including the low-definition changing section 17a and the low-definition changing section 17b and the projection method conversion shown by the projection method converting section 18 are described separately in terms of functional arrangement; Since they are usually performed together, they may be written in the same block. In this embodiment, the functions are simply illustrated as shown in FIG. 13 for convenience in order to clearly distinguish the functions. In the functional block diagram shown in FIG. 13, dotted lines indicate each functional block that functions during video distribution and recording, solid lines indicate each block that functions during video playback, and thick lines indicate video distribution, recording, and video playback. For convenience, they are written separately to indicate each block that functions in both.

(Horizontal/vertical angle of view threshold information)
As shown in FIG. 16A, the threshold management unit 1001 stores horizontal and vertical view angle threshold information, for example, before shipment from the factory. Note that FIG. 16(a) is a conceptual diagram of the horizontal and vertical viewing angle threshold information, and FIG. 16(b) is an explanatory diagram of the horizontal viewing angle and the vertical viewing angle. Note that although the horizontal and vertical view angle threshold information is expressed in a table format in FIG. 16A, the information is not limited to this and need not be expressed in a table format.

As shown in FIG. 16(a), the horizontal/vertical angle of view threshold information is determined by a column of candidates for the maximum imaging resolution (width W x height H) of the omnidirectional imaging device 1, and by the user (viewer). A row of candidates for the maximum display resolution (horizontal W' x vertical H') that can be displayed on the smartphone 5 to be set, and a determination as to whether to transmit it from the omnidirectional photographing device 1 to the smartphone 5 via the image management system 7 Each threshold value of the horizontal angle of view (AH) x the vertical angle of view (AV) of the image used for the image is managed in association with each other.

Here, the horizontal angle of view (AH) and the vertical angle of view (AV) will be explained using FIG. 16(b). Note that in FIG. 16(b), the center point CP, the azimuth angle "aa", and the elevation angle "ea" are the same as those in FIG. 7, so their explanations will be omitted. In a spherical image (video, still image), the size of the entire area of one frame of a rectangular partial image (video, still image) can be expressed in terms of the direction of the shooting camera (here, the virtual camera IC). be. The horizontal length of the partial image can be expressed by the horizontal angle of view (AH) from the virtual camera IC, and the vertical length of the partial image can be expressed by the horizontal angle of view (AH) from the virtual camera IC.

The horizontal angle of view (AH) and the vertical angle of view (AV) are angles of view at which there is no difference between the original resolution and the converted resolution, and are used for determination by the determination unit 25. This angle of view can be determined using the following (Equation 1) and (Equation 2).

AH=(360/W)*W'...(Formula 1)
AV=(180/H)*(H'/2)...(Formula 2)
Note that even if the maximum imaging resolution of the omnidirectional imaging device 1 is (W: 4000, H: 2000), the omnidirectional imaging device 1 has a horizontal and vertical angle of view that shows each setting pattern shown in FIG. Manages threshold information. However, in this case, the omnidirectional photographing device 1 only refers to the patterns of settings 1 and 2 in FIG. 16. Similarly, even if the maximum display resolution of the smartphone 5 set by the user is (W': 1920, H': 1080), the spherical photographing device 1 will not accept each setting pattern shown in FIG. It manages horizontal and vertical viewing angle threshold information.

For example, if the maximum imaging resolution of the spherical imaging device 1 is 4000 x 2000, and the user sends a request from the smartphone 5 to the spherical imaging device 1 with a width x 1080 resolution. Under the condition (C1) where the setting is made to request image data (moving images, still images) with a resolution of If the angle is less than the threshold value represented by 48.6° and 48.6° vertically, it is determined that the projection method conversion unit 18 is not allowed to perform projection method conversion to ultra-high-definition still image data.

(Each functional configuration of the spherical photography device)
Next, each functional configuration of the omnidirectional photographing device 1 will be described in more detail using FIG. 13.

The transmitting/receiving unit 11 transmits image data to the outside of the omnidirectional photographing device 1 and receives data from the outside. For example, the image data includes ultra-high-definition partial still images, high-definition partial moving images, and low-definition whole moving images. The transmitting/receiving unit 11 also receives instruction data, which will be described later, and transmits partial image parameters, which will be described later.

In addition, in this embodiment, since images with three levels of definition are handled, the image with the highest definition, the image with the second highest definition, and the image with the lowest definition are respectively classified as "ultra high definition" for convenience. ”, “high definition”, and “low definition”. Therefore, for example, "ultra high definition" in this embodiment does not mean general ultra high definition.

The partial image parameter creation unit 12 creates partial image parameters based on instruction data sent from the smartphone 5 via the image management system 7. This instruction data is received by the user's operation at the reception unit 52 of the smartphone 5, and is data indicating an instruction for specifying a superimposed area, which will be described later.

The imaging control unit 13 outputs an instruction to synchronize the output timing of image data of the imaging units 14a and 14b. The imaging control unit 13 also has a function of inputting time information obtained from an external source (for example, a GPS satellite) and managing timing when generating a still image from a video being played (hereinafter also referred to as display). have. This timing management will be explained in conjunction with a sequence diagram related to generation of partial still images, which will be described later.

The imaging units 14a and 14b respectively image a subject etc. according to instructions from the imaging control unit 13, and, for example, as shown in FIGS. 3(a) and 3(b), based on spherical image data. Outputs hemispherical image data.

The image processing section 15 synthesizes and converts the two hemispherical image data obtained by the imaging sections 14a and 14b into equirectangular projection image data, which is an equirectangular projection image. In other words, the image processing unit 15 is an example of a spherical image generation unit that generates spherical image data from moving image data of a predetermined definition obtained by photographing a subject.

The temporary storage unit 16 serves as a buffer that temporarily stores the data of the equirectangular projection image synthesized and transformed by the image processing unit 15. Note that the equirectangular projection image in this state is extremely high definition.

The definition change unit 17 converts the equirectangular projection image from an ultra-high-definition image (here, a video) to a low-definition image (video) by reducing the size of the equirectangular cylindrical projection image according to the instruction data from the smartphone 5 received by the transmitting/receiving unit 11. and change the low-definition image to a relatively high-definition image (video). As a result, a low-definition equirectangular projection image (here, the entire moving image) is generated. Note that the definition changing unit 17 is an example of a changing means.

The projection method conversion unit 18 converts the smartphone according to the instruction data and video data request received by the transmitting/receiving unit 11, that is, the direction, angle of view, and aspect of a partial image that is a partial image of the entire image, and the image management system 7. The projection method is converted from the equirectangular cylindrical projection method to the perspective projection method according to the size of the image data to be transmitted to 5. As a result, a partial image (here, a partial video) is generated while maintaining ultra-high definition. In this way, the overall image data output from the definition changing unit 17 has a lower definition (or resolution) than the partial image data output from the projection method converting unit 18. That is, the partial image data output from the projection method converter 18 has a higher definition (resolution) than the entire image data output from the definition change unit 17.

Furthermore, the projection method converter 18 converts the direction, angle of view, and aspect of the partial image, which is a partial image of the entire image, and the image management system 7, according to the instruction data and still image data request received by the transmitter/receiver 11. The projection method is converted from the equirectangular cylindrical projection method to the perspective projection method according to the size of the image data to be transmitted to the smartphone 5 via the smartphone 5. As a result, a partial image (here, a partial still image) is generated in an ultra-high definition state. In this embodiment, the original images of the partial moving images and partial still images generated by the projection method converter 18 are also ultra-high definition; The degree of definition may be changed to differentiate between degrees.

Here, a case will be described in which, for example, a 2K, 4K, or 8K equirectangular projection image is output as an ultra-high-definition image. Based on the data of these ultra-high-definition equirectangular projection images, the partial image data output from the projection method conversion unit 18 remains at the resolution (either 2K, 4K, or 8K) of the equirectangular projection image. , is data converted into a predetermined projection method. On the other hand, the entire image data output from the definition change unit 17 is data of relatively lower definition (lower resolution) than the equirectangular projection image, such as 1K, 2K, 4K, etc.

The combining unit 19 combines each frame of the low-definition whole video changed by the definition change unit 17 and each frame of the ultra-high-definition partial video converted by the projection method converter 18, as shown in FIG. By compressing the vertical resolution in half, the definition of each frame is lowered and combined into one frame. As a result, the partial video has higher definition than the entire video, but lower definition than the partial still image. By lowering the definition of each frame into one frame in this way, it is possible to suppress the amount of data communication. Furthermore, since the two frames to be combined are generated from the same equirectangular projection image stored in the temporary storage unit 16, there is no need to use metadata or the like to associate the two frames to be combined. It is also possible to associate two frames captured and generated at the same timing.

The video encoding unit 20a encodes the frame data of the entire video and partial videos combined by the combination unit 19. The still image encoding unit 20b encodes data of partial still images. Note that the video encoding unit 20a can manage the time of the omnidirectional photographing device 1 using, for example, time information obtained by a GPS clock function.

The reception unit 22 receives operations performed by the user on the operation unit 115 of the omnidirectional photographing device when making various requests.

The determining unit 25 determines whether the entire area of one frame of the partial image, which is a partial area of the entire video, is indicated by a predetermined threshold of the horizontal/vertical view angle threshold information (see FIG. 16) managed by the threshold managing unit 1001. It is determined whether the area is smaller than a predetermined area. If the difference is smaller, the determining unit 25 prevents the projection method conversion unit 18 from converting the ultra-high definition still image data into the projection method. If the difference is large, the determining unit 25 causes the projection method conversion unit 18 to convert the ultra-high-definition still image data into a projection method.

For example, if the maximum imaging resolution of the spherical photographing device 1 is 4000 x 2000, the user may send a message from the smartphone 5 to the spherical photographing device 1 in the horizontal direction of the maximum display resolution on the smartphone 5. × Under the condition (C1) when the height is set to 1920 x 1080, the horizontal angle of view of the image requested by the instruction data is less than 172.8° and the vertical angle of view is less than 48.6° In this case, the determining unit 25 determines not to cause the projection method conversion unit 18 to convert the ultra-high definition still image data into a projection method. In this way, when the resolution (definition) of the image requested by the smartphone 5 is low, the omnidirectional photographing device 1 sends data of a high-definition partial video or an ultra-high-definition partial still image. Even if the data of This is because there is no need to send data. Note that "less than" shown in the threshold range may also be "less than or equal to".

On the other hand, under the same condition (C1) as above, if the horizontal angle of view of the image requested by the instruction data exceeds 172.8° or the vertical angle of view exceeds 48.6°, the determination unit 25 , it is determined that the projection method converter 18 should convert the ultra-high definition still image data into a projection method. In this case, the smartphone 5 switches and displays a high-definition partial video to an ultra-high-definition still image, allowing the user (also referred to as a viewer) to view the area of interest more clearly (clearly). This allows for careful viewing. Note that "exceeding" shown in the threshold range may also be "greater than".

<Functional configuration of smartphone>
FIG. 14 is a functional block diagram of the smartphone of this embodiment. The smartphone 5 includes a transmitting/receiving section 51, a reception section 52, a video decoding section 53a, a still image decoding section 53b, a superimposed display control section 63 (superimposed area creation section 54, image creation section 55, image superimposition section 56, projective conversion section 57, a display control section 58), a reproduction time management section 59, a reproduction frame extraction section 60, a superimposed image determination section 61, and a storage control section 62. Each of these units is a function or means realized by one of the components shown in FIG. 11 operating in response to an instruction from the CPU 501 according to a smartphone program loaded from the EEPROM 504 onto the RAM 503. . Here, the superimposed display control unit 63, which functions according to instructions from the CPU 501 according to a smartphone program, has a function of superimposing and displaying a still image on a moving image, and is an example of a superimposed display control means. The superimposed display control section 63 has the functions of a superimposed area creation section 54, an image creation section 55, an image superimposition section 56, a projective transformation section 57, and a display control section 58. In the functional block diagram shown in FIG. 14, dotted lines indicate functional blocks that function during video distribution and recording, solid lines indicate blocks that function during video playback, and thick lines indicate video distribution, recording, and video playback. For convenience, they are written separately to indicate each block that functions in both.

(Each function configuration of smartphone)
Next, each functional configuration of the smartphone 5 will be described in more detail using FIG. 14.

The transmitter/receiver 51 transmits data to the outside of the smartphone 5 and receives data from the outside. For example, the transmitting/receiving unit 51 may receive image data from the transmitting/receiving unit 11 of the spherical photographing device 1, or may be configured to and send instruction data. That is, the transmitting/receiving unit 51 is an example of a transmitting means and a receiving means that transmits and receives data indicating a predetermined operation corresponding to image data and instruction data to and from the image management system 7. Further, the transmitting/receiving unit 51 separates the image data (whole video data and partial video data shown in FIG. 20) sent from the transmitting/receiving unit 11 of the omnidirectional photographing device 1 from partial image parameters.

The receiving unit 52 receives from the user an operation for specifying the direction, angle of view, and aspect of the partial image, as well as the size of the image data to be received by the smartphone 5. The reception unit 52 also receives settings for the maximum display resolution of the smartphone 5 (see width W'×height H' in FIG. 16) from the user. This reception unit 52 is an example of reception means.

The video decoding unit 53a generates video shooting start time and corresponding time information for each data of the low-definition whole video and high-definition partial video encoded by the video encoding unit 20a of the omnidirectional photographing device 1. It receives video data output from a playback frame extraction unit 60 (described later) which performs processing together with metadata such as, and decodes it into low-definition whole video data and high-definition partial video data. The still image decoding unit 53b adds metadata such as video shooting start time and corresponding time information to the ultra-high-definition partial still image data encoded by the still image encoding unit 20b of the omnidirectional photographing device 1. Still image data output from a reproduction frame extracting unit 60 (described later) that performs processing together with the data is received and decoded into ultra-high definition partial still image data.
The superimposed area creation unit 54 acquires the partial image parameters (metadata) read out by the storage control unit 62, and creates a superimposed area specified by the acquired partial image parameters. This superimposition area indicates the superimposition position and range of the superimposition image S and mask image M, which are partial moving images (or partial still images), on the spherical image CE, which is the entire moving image.

The image creation section 55 creates a superimposed image S and a superimposed image S according to each data acquired from the superimposed region creation section 54, the moving image decoding section 53a, the still image decoding section 53b, and the superimposed image determination section 61 and information related to their superimposed regions. A mask image is created, and a celestial sphere image CE is created from the low-definition overall image.

The image superimposing unit 56 acquires the superimposed image S and the mask image created by the image creating unit 55, and superimposes the second image on the first image. For example, the image superimposition unit 56 creates the final omnidirectional image CE by superimposing the superimposed image S and the mask image M on the superimposed region on the omnidirectional image CE.

The projective transformation unit 57 obtains the final spherical image CE created by the image superimposition unit 56, and performs perspective projection on the final spherical image CE according to the user's instructions received at the reception unit 52. Convert to a method image. This projective transformation unit 57 is an example of projective transformation means.

The display control unit 58 acquires the perspective projection image converted by the projection conversion unit 57, and performs control to display the image after conversion into the perspective projection mode on the display 517 or the like. For example, the display control unit 58 displays the second image superimposed on the first image by the image superimposing unit 56.

The playback time management unit 59 manages the reference video playback time and outputs the playback time. For example, if you want to play a video at 30fps (frames per second), the start time is the video recording start time that can be obtained from the metadata, and the time is output 30 times in increments of 1/30 seconds per second.

The reproduction frame extraction unit 60 extracts and outputs video data and still image data to be reproduced based on the reproduction time output by the reproduction time management unit 59. At this time, the video data is output as a frame at the reception timing of the playback time. On the other hand, still image data is output by searching for the latest still image file before the playback time using the corresponding time information of the metadata. In this way, it is possible to play back the videos and still images while maintaining the timing relationship (output conditions) when the videos and still images were taken.

The superimposed image determining unit 61 determines which image, a partial moving image or a partial still image, should be superimposed on the entire moving image photographed by the omnidirectional photographing device 1. The superimposed area displayed here is a partial video (or The superimposed position and superimposed range of the superimposed image S and the mask image M, which are partial still images), are shown.

The storage control unit 62 stores and manages various data including metadata consisting of video shooting start time, corresponding time information, partial image parameters, etc. in the storage unit 5000 realized by the ROM 502 shown in FIG. control. Note that the metadata may be stored and managed in the storage unit 7000 of the image management system 7.

<Functional configuration of image management system>
FIG. 15 is a functional block diagram of the image management system. The image management system 7 includes a receiving section 71a, a transmitting section 71b, a video playback metadata creation section 72, a data selection section 73, and a storage control section 74. The receiving section 71a, the transmitting section 71b, the video playback metadata creation section 72, the data selection section 73, and the storage control section 74 are configured to use the network I/F 709 shown in FIG. This is a function or means that is realized by operating according to instructions from the CPU 701 according to a management system program. Note that the receiving section 71a is an example of a system-side receiving section, and the transmitting section 71b is an example of a system-side transmitting section. The image management system 7 also includes a storage unit 7000 implemented by an HD 704.

(Functional configuration of image management system)
Next, each functional configuration of the image management system 7 will be described in more detail using FIG. 15.

The transmitter 71b transmits data to the outside of the image management system 7, and the receiver 71a receives data from the outside. For example, the receiving section 71a receives image data from the transmitting/receiving section 11 of the omnidirectional photographing device 1 via the intermediary terminal 3, and the transmitting section 71b sends the image data to the transmitting/receiving section 11 of the omnidirectional photographing device 1 via the intermediary terminal. The instruction data is transmitted via 3. The transmitting unit 71b also transmits image data (whole video data and partial video data shown in FIG. 20) sent from the transmitting/receiving unit 11 of the omnidirectional photographing device 1 via the intermediary terminal 3, and partial image parameters. is sent to the smartphone 5. Further, the transmitting unit 71b transmits ultra-high definition still image data sent from the transmitting/receiving unit 11 of the omnidirectional photographing device 1 via the intermediary terminal 3 and temporarily stored in the storage unit 7000 to the smartphone 5. do.

The video playback metadata creation unit 72 creates video playback metadata, which will be described later, based on video data captured by the omnidirectional imaging device 1. This video playback metadata includes common information, video information, and at least one piece of still image information.

The data selection unit 73 is a functional unit that selects which data is to be outputted via the transmission unit 71b among various data stored in the storage unit 7000, including the entire video, partial still images, and video playback metadata. be.

The storage control unit 74 controls the storage unit 7000 to store and manage various data including various data including the above-mentioned whole moving image, partial still image, and metadata for playing the moving image. In the functional block diagram shown in FIG. 15, dotted lines indicate functional blocks that function during video distribution and recording, solid lines indicate blocks that function during video playback, and thick lines indicate functional blocks that function during video distribution and recording. For convenience, they are written separately to indicate each block that functions in both cases during playback.

[Processing or operation of embodiment]
Next, the processing or operation of this embodiment will be described using FIGS. 17 to 40. In this embodiment, while playing the entire video and partial video related to the video generated and distributed by the spherical shooting device 1 on the smartphone 5, the viewer can take a closer look at one scene of the partial video or the subject of interest. We will explain the case of carefully viewing high-definition (ultra-high definition) still images. This is because viewers can view the page more carefully when viewing still images rather than moving images.

In this embodiment, wide-angle images forming a wide-angle video acquired from the omnidirectional imaging device 1 and partial images forming a partial video generated from the wide-angle images are recorded as a moving image (moving image) arranged in the same frame. At the same time, the shooting start time is recorded as metadata. Also, when recording a partial still image with a higher resolution than the video acquired from the omnidirectional camera 1 during video recording, the shooting time is recorded as metadata. When a user plays back a video and a still image at the same time (superimposed), the system can control the playback timing of each by referring to the metadata related to the video and partial still images. It is characterized by maintaining the timing relationship between the two. In other words, the shooting time of each wide-angle video, partial video, and partial still image is recorded and managed as metadata, and the playback timing of videos and still images is controlled by referencing the metadata when playing the video. shall be.

<Generation and playback of whole video and partial video>
First, with reference to FIG. 17, the generation and playback process of each data of the entire video and partial video will be described. FIG. 17 is a sequence diagram showing the process of generating and reproducing each data of the entire moving image and partial moving image.

As shown in FIG. 17, when the viewer (user) operates the smartphone 5, the reception unit 22 receives a request to start a video distribution service (step S11). As a result, the transmitting/receiving unit 51 of the smartphone 5 next transmits a request for video data to the receiving unit 71a of the image management system 7 (step S12). At this time, the partial image parameters specified by the viewer are also transmitted. Note that regarding the timing of starting transmission, the transmitting/receiving unit 11 of the omnidirectional photographing device 1 may perform band control of the communication network. By performing this band control, it becomes possible to transmit and receive data more stably.

Next, the transmitter 71b of the image management system 7 transfers a video data recording request to the transmitter/receiver of the intermediary terminal 3 (step S13). Then, the transmitting/receiving section of the intermediary terminal 3 transfers the video data recording request to the transmitting/receiving section 11 of the omnidirectional photographing device 1 (step S14).

Next, the omnidirectional photographing device 1 performs video data generation processing (step S15). The process of step S15 will be explained in detail later (see FIGS. 18 to 20).

Next, the transmitting/receiving unit 11 of the omnidirectional photographing device 1 transmits the video data according to the request to the transmitting/receiving unit of the intermediary terminal 3 (step S16). This video data includes each data of a low-definition whole video and a high-definition partial video. Thereby, the transmitting/receiving section of the intermediary terminal 3 transfers the video data to the receiving section 71a of the image management system 7 (step S17). Further, the transmitter 71b of the image management system 7 transfers the video data to the transmitter/receiver 51 of the smartphone 5 (step S18).

Next, the smartphone 5 performs video data playback processing (step S19). The process of step S19 will be explained in detail later (see FIGS. 21 to 40).

<Video data generation process for omnidirectional imaging device>
Next, with reference to FIGS. 18 to 20, the video data generation process of the omnidirectional imaging device shown in step S15 above will be described in detail. FIG. 19 is a conceptual diagram of an image in the process of image processing performed by the omnidirectional photographing device.

The image processing unit 15 synthesizes and converts the two hemispherical image data obtained by the imaging units 14a and 14b into equirectangular projection image data, which is an equirectangular projection image (here, a moving image) (step S120). ). The data after this conversion is temporarily stored in the temporary storage unit 16 as a high-definition image.

Next, the partial image parameter creation unit 12 creates partial image parameters based on the instruction data sent from the smartphone 5 (step S130).

Next, the low-definition changing unit 17a of the definition changing unit 17 changes the ultra-high definition image to a low-definition image according to the instruction data from the smartphone 5 received by the transmitting/receiving unit 11 (step S140). As a result, a low-definition equirectangular projection image (here, the entire moving image) is generated.

Furthermore, the projection method conversion unit 18 converts the direction of the partial video, which is a part of each frame of the entire video, the angle of view and aspect of the frame of the partial video, and the smartphone according to the instruction data received by the transmitting/receiving unit 11. According to the size of the moving image data to be transmitted to No. 5, the projection method is converted from the equirectangular projection method to the perspective projection method, and the image is changed to a low-definition image (step S150). As a result, a partial video is generated in ultra-high definition.

Next, the combining unit 19 combines each data of the low-definition whole video and the ultra-high-definition partial video (step S160). This combination process will be described in detail later (see FIG. 20).

Here, the process shown in FIG. 18 will be explained in more detail using FIGS. 19 and 20. FIG. 19 is a diagram illustrating partial image parameters.

(partial image parameters)
First, partial image parameters will be explained in detail using FIG. 19. FIG. 19(a) shows the entire image after image synthesis in step S120. FIG. 19(b) is a diagram showing an example of partial image parameters. FIG. 19(c) shows the partial image after the projection method has been converted in step S150.

The azimuth angle explained in FIG. 7 becomes the horizontal direction (latitude λ) in the equidistant azimuth projection method shown in FIG. 19(a), and the elevation angle explained in FIG. 7 becomes the vertical direction (longitude φ) in the equirectangular projection method. The parameters shown in FIG. 19(b), including the diagonal angle of view and aspect ratio in FIG. 7, are partial image parameters. FIG. 19(c) is an example of a partial image obtained by cutting out a portion surrounded by a frame on the equirectangular projection of FIG. 19(a) using partial image parameters.

Here, the conversion of the projection method will be explained. As shown in FIG. 4(a), a celestial sphere image is created by covering a three-dimensional sphere with an equirectangular projection image. Therefore, each pixel data of the equirectangular projection image can be made to correspond to each pixel data on the surface of the three-dimensional sphere of the three-dimensional omnidirectional image. Therefore, the conversion formula by the projection method conversion unit 18 expresses the coordinates in the equirectangular projection image as (latitude, longitude) = (e, a), and the coordinates on the three-dimensional solid sphere as the orthogonal coordinates (x, y , z), it can be expressed by the following (Equation 3).
(x, y, z) = (cos(e) × cos(a), cos(e) × sin(a), sin(e)) ・・・(Equation 3)
However, the radius of the three-dimensional sphere at this time is 1.

On the other hand, a partial image that is a perspective projection image is a two-dimensional image, but when expressed in two-dimensional polar coordinates (radius, declination) = (r, a), the radius r is the diagonal angle of view. Correspondingly, the possible range is 0 ≦ r ≦ tan (diagonal angle of view/2). Furthermore, when a partial image is represented by a two-dimensional orthogonal coordinate system (u, v), the transformation relationship with polar coordinates (radius, declination) = (r, a) can be expressed by the following (Equation 4). can.
u = r × cos(a), v = r × sin(a) ...(Equation 4)
Next, consider making this (Formula 3) correspond to three-dimensional coordinates (radius, polar angle, azimuth). Since we are currently considering only the surface of the three-dimensional sphere CS, the radius vector in the three-dimensional polar coordinates is "1". Furthermore, in the perspective projection transformation of the equirectangular projection image pasted on the surface of the three-dimensional sphere CS, assuming that the virtual camera is at the center of the three-dimensional sphere, the above-mentioned two-dimensional polar coordinates (radius, declination) = ( r, a), it can be expressed by the following (Formula 5) and (Formula 6).
r = tan(polar angle)...(Formula 5)
a = azimuth angle (Formula 6)
Here, if the polar angle is t, then t = arctan(r), so the three-dimensional polar coordinates (radius, polar angle, azimuth) are (radial, polar angle, azimuth) = (1, arctan( r), a).

Further, a conversion formula for converting from three-dimensional polar coordinates to a rectangular coordinate system (x, y, z) can be expressed by the following (Equation 7).
(x, y, z) = (sin(t) × cos(a), sin(t) × sin(a), cos(t)) ・・・(Equation 7)
By using the above (Equation 7), it is now possible to mutually convert the entire image by the equirectangular projection method and the partial image by the perspective projection method. That is, by using the vector radius r that corresponds to the diagonal angle of view of the partial image to be created, it is possible to calculate the transformation map coordinates that indicate which coordinates of the equirectangular projection image each pixel of the partial image corresponds to. , Based on the transformation map coordinates, a partial image, which is a perspective projection image, can be created from the equirectangular projection image.

By the way, the above projection method conversion refers to a conversion in which the position where (latitude, longitude) of the equirectangular cylindrical projection image becomes (90°, 0°) becomes the center point of the partial image that is the perspective projection image. ing. Therefore, when performing perspective projection transformation with an arbitrary point of the equirectangular projection image as the point of interest, the coordinates (latitude, longitude) of the point of interest can be determined by rotating the three-dimensional sphere to which the equirectangular projection image is pasted. What is necessary is to perform coordinate rotation such that the object is placed at the position (90°, 0°).

The transformation formula regarding the rotation of this three-dimensional sphere is a general coordinate rotation formula, so its explanation will be omitted.

(Join processing)
Next, the combining process in step S160 performed by the combining unit 19 will be described using FIG. 20. FIG. 20 is a conceptual diagram showing a state in which frame images of the entire video and partial video are combined. As shown in FIG. 20, the images are combined so that the entire image is placed at the top of one frame and the partial image is placed at the bottom. In this embodiment, the arrangement is generally made in accordance with the aspect ratio of 16:9 for HD, etc., but the aspect ratio does not matter. Furthermore, the arrangement is not limited to the top and bottom, but may be on the left and right. If there are multiple partial images, the entire video may be placed in the upper half, and the partial images may be divided into the number of partial images in the lower half. By combining the entire moving image and partial images into one frame by frame, synchronization between images can be guaranteed. Further, the omnidirectional photographing device 1 may separately transmit the entire video data and the partial video data to the smartphone 5.

However, since the entire video and partial video are halved vertically, the resolution of each video is halved compared to when the video is not halved. That is, the definition is lower than the ultra-high definition image stored in the temporary storage unit 16. As a result, in descending order of definition, the whole image stored in the temporary storage unit 16, the partial image converted by the projection method conversion unit 18 and then changed in definition by the low definition change unit 17b, and the low definition changed image. The overall image is the one whose definition has been changed by the section 17a.

<Smartphone video playback processing>
Next, the video data playback process of the smartphone 5 will be described using FIG. 21. FIG. 21 is a conceptual diagram of an image in the process of image processing performed by a smartphone. The superimposed area creation unit 54 shown in FIG. 7 creates a partial three-dimensional sphere PS specified by the partial image parameters as shown in FIG. 21 (step S320).

Next, the image creation unit 55 creates a superimposed image S by superimposing a partial image using a perspective projection method on the partial solid sphere PS (step S330). Furthermore, the image creation unit 55 creates a mask image M based on the partial solid sphere PS (step S340). Furthermore, the image creation unit 55 creates a celestial sphere image CE by pasting the entire image, which is an equirectangular projection method, onto the three-dimensional sphere CS (step S350). Then, the image superimposition unit 56 superimposes the superposition S and the mask image M on the omnidirectional image CE (step S360). As a result, a low-definition omnidirectional image CE is completed on which the high-definition superimposed image S is superimposed so that the border is not noticeable.

Next, the projective transformation unit 57 converts a predetermined area in the spherical image CE on which the superimposed image S is superimposed so that it can be viewed on the display 517 based on the predetermined viewing direction and angle of view of the virtual camera. Projective transformation is performed (step S370).

FIG. 22 is a diagram illustrating creation of a partial three-dimensional sphere from a partial plane. Normally, in the perspective projection method, the image is projected onto a plane, so it is often represented as a plane in a three-dimensional space, as shown in FIG. 22(a). In this embodiment, as shown in FIG. 22(b), a partial three-dimensional sphere, which is a part of a sphere, is used in accordance with the omnidirectional image. Here, the conversion from a plane to a partial three-dimensional sphere will be explained.

Consider the projection of each point (X, Y, Z) on a plane arranged at an appropriate size (angle of view) onto a spherical surface as shown in FIG. 22(a). The projection onto the spherical surface is the intersection of the spherical surface and a straight line passing through each point (X, Y, Z) from the origin of the sphere. Each point on the sphere is a point whose distance from the origin is equal to the radius of the sphere. Therefore, assuming that the radius of the sphere is 1, the point (X', Y', Z') on the spherical surface shown in FIG. 22(b) is expressed by the following (Equation 8).

(X', Y', Z')=(X, Y, Z)×1/√(X2+Y2+Z2)...(Formula 8)
FIG. 23 is a two-dimensional conceptual diagram when a partial image is superimposed on a celestial sphere image without performing the partial three-dimensional sphere creation of this embodiment. FIG. 24 is a two-dimensional conceptual diagram when a partial three-dimensional sphere is created according to this embodiment and a partial image is superimposed on a celestial sphere image.

As shown in FIG. 23(a), based on the case where the virtual camera IC is located at the center point of the three-dimensional sphere CS, the subject P1 is represented as an image P2 on the spherical image CE. , is represented on the superimposed image S as an image P3. As shown in FIG. 23(a), the images P2 and P3 are located on the straight line connecting the virtual camera IC and the subject P1, so the superimposed image S is superimposed on the spherical image CE. Even if the spherical image CE and the superimposed image S are displayed in the same state, no deviation occurs between the omnidirectional image CE and the superimposed image S. However, as shown in FIG. 23(b), when the virtual camera IC moves away from the center point of the three-dimensional sphere CS (when the angle of view α is decreased), on the straight line connecting the virtual camera IC and the subject P1, The image P2 is located, but the image P3 is located slightly inside. Therefore, if the image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is an image P3', there is a difference between the spherical image CE and the superimposed image S between the images P3 and P3'. A deviation of the amount g will occur. As a result, the superimposed image S is displayed shifted from the omnidirectional image CE, but the superimposed image S may be displayed in a superimposed manner as described above.

On the other hand, in this embodiment, since a partial 3D sphere is created, the superimposed image S is created along the omnidirectional image CE, as shown in FIGS. 24(a) and 24(b). can be superimposed. As a result, not only when the virtual camera IC is located at the center point of the three-dimensional sphere CS as shown in FIG. 24(a), but also when the virtual camera IC is located at the center point of the three-dimensional sphere CS as shown in FIG. Even if they are away from the center point of the three-dimensional sphere CS, the images P2 and P3 will be located on the straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superimposed image S is displayed superimposed on the omnidirectional image CE, no deviation occurs between the omnidirectional image CE and the superimposed image S.

FIG. 25(a) is an example of a wide image displayed without superimposed display, FIG. 25(b) is an example of a tele image displayed without superimposed display, and FIG. 25(c) is an example of a wide image displayed with superimposed display. , FIG. 25(d) is a conceptual diagram showing a display example of teleimages in the case of superimposed display. Note that the wavy lines in the figure are only shown for convenience of explanation, and may or may not actually be displayed on the display 517.

As shown in FIG. 25(a), when the partial image P is not displayed in a superimposed manner on the omnidirectional image CE, when the area indicated by the dotted line in FIG. 25(a) is enlarged and displayed, FIG. As shown in (b), the image remains in low definition, and the user ends up viewing an image that is not clear. On the other hand, as shown in FIG. 25(c), when displaying the partial image P superimposed on the omnidirectional image CE, the area is enlarged to the area indicated by the dotted line in FIG. 25(c). When displayed, a high-definition image is displayed, as shown in FIG. 25(d), and the user can see a clear image. In particular, when a signboard or the like with characters drawn on it is displayed in the area indicated by the wavy line, unless a high-definition partial image P is superimposed on the display, the characters will become blurry even if the image is enlarged. I don't know if it's written there. However, if high-definition partial images P are superimposed and displayed, the characters can be clearly seen even when the images are enlarged, so that the user can understand what is written.

<Metadata structure>
Next, the configuration of video playback metadata used in this embodiment will be described using FIG. 26. As shown in FIG. 26, the video playback metadata includes common information, video information, and at least one piece of still image information. Among these, common information includes gaze point information and attention area view angle information, which can be calculated from partial parameters specified by the viewer while using the video distribution service. These pieces of information are used when displaying a high-definition partial video or ultra-high-definition partial still image superimposed on a low-definition overall video.

The video identification information in the video information is image identification information for identifying a video including a low-definition whole image and a high-definition partial image. In FIG. 26, the file name of the image is shown as an example of the moving image identification information, but it may also be an image ID for uniquely identifying the image.

The video shooting start time in the video information is time information when the frame at which video recording is started was shot by the omnidirectional camera. To add time information to a video, for example, when encoding with H.264, the User data unregistered area of SEC (Supplemental Enhancement Information) is used. This information is used to determine the timing to superimpose ultra-high-definition partial still images during playback of a low-definition overall video.

The still image identification information in the still image information is image identification information for identifying a still image that is an ultra-high definition partial image. In FIG. 26, the file name of the image is shown as an example of the still image identification information, but it may also be an image ID for uniquely identifying the image.

The corresponding time information in the still image information is time information when the still image was photographed, and is information stored in Exif (Exchangeable image file format), which is standardized as an image recording format. This information is used to determine the timing to superimpose ultra-high-definition partial still images during playback of a low-definition overall video. Still image information is stored for the number of still images taken during video recording.
<Video recording and metadata file creation>
FIG. 27 is a sequence diagram showing the process of recording a moving image and creating metadata. As shown in FIG. 27, when the viewer (user) performs an operation to request video recording on the smartphone 5 at any time during video playback on the smartphone 5, the receiving unit 22 records the video. (Step S21). Thereby, the transmitting/receiving unit 51 of the smartphone 5 transmits a video data recording request to the receiving unit 71a of the image management system 7 (step S22). At this time, the partial image parameters specified by the viewer are also transmitted. Note that regarding the timing of starting transmission, the transmitting/receiving unit 11 of the omnidirectional photographing device 1 may perform band control of the communication network. By performing this band control, it becomes possible to transmit and receive data more stably. In addition, regarding operations on the smartphone 5 by the viewer, general operations on the display 517 of the smartphone 5 (tap operation to select an image, pinch-out operation to enlarge an image, etc.) and predetermined operations It may also be a key operation on a key.

Note that the video generation process performed by the omnidirectional photographing device 1 in the next step S23 and its details, the transmission of video data to the intermediary terminal 3 performed in step S24, and the image of the video data performed in step S25 Each process of transmission to the management system 7 corresponds to the contents of steps S15, S16, and S17 described above in the explanation of FIG. 17, and the same process is performed when distributing the video, so the description will be omitted. In other words, while the video distribution continues, recording of the video starts from the time when the viewer makes a video recording request at an arbitrary timing.

Subsequently, upon receiving the video data from the intermediary terminal 3 via the receiving unit 71a, the storage control unit 74 of the image management system 7 stores the received video data in the storage unit 7000 (step S26).

Furthermore, the storage control unit 74 of the image management system 7 creates a metadata file for the superimposed video at the start of storing video data, and stores it in the storage unit 7000 (step S27). The metadata created at this point stores gaze point information, attention area view angle information, video identification information, and video shooting start time among the metadata configurations described above. Further, recording of the video continues until there is a request from the viewer to stop video recording or until the storage area of the storage unit 7000 runs out. Note that if the storage area of the storage unit 7000 is insufficient, the storage unit 7000 is controlled to be used as a ring buffer and various stored data is overwritten depending on the specifications and settings of the viewer and the image management system 7. Good too.

Note that the video data transmission process (step S28) transmitted from the image management system 7 to the smartphone 5 and the video playback process (step S29) executed by the smartphone 5 are the same as the processes of steps S18 and S19 in FIG. 17 described above. Since this is the same as , the explanation will be omitted.

<Generation of partial still image and update of metadata>
Next, partial still image data generation and metadata update processing will be described using FIG. 28. FIG. 28 is a sequence diagram showing still image recording and metadata updating processing.

In step S19 shown in FIG. 17, while the viewer (user) is viewing a video on which a partial image P as shown in FIG. 25(c) is superimposed, ) If the viewer (user) wishes to view the partial still image, the viewer (user) operates the smartphone 5, and the reception unit 22 receives a request to start a partial still image distribution service (step S31).

(Operation method when requesting still image generation)
A specific example of the operation that the viewer (user) performs on the smartphone 5 when requesting generation of a still image in step S31 will be described using FIG. 29. FIG. 29 is a diagram illustrating an example of an operation method when requesting generation of a still image. During video distribution on the smartphone 5, if a repeated tap operation or pinch-out operation is performed on the display 517 (display screen) at the timing when a subject of interest to the viewer appears (FIG. 29 (A) or (B)) ), the reception unit 52 and the superimposed display control unit 63 perform the following processing. First, the reception unit 52 determines whether the repeated tap operations or pinch-out operations on the partial video performed by the viewer reach a predetermined condition DC (enlargement range, number of taps, etc.). Thereafter, the image creation section 55 and the image superimposition section 56 of the superimposition display control section 63 select the area around the subject that has been enlarged in the partial video displayed on the display 517 of the smartphone 5 from the partial video being played back. is also generated as a partial still image with even higher definition, and this partial still image is superimposed on the partial moving image.

The following examples are examples of the predetermined conditions used by the reception unit 52 to determine whether the viewer repeatedly taps or pinches out the display 517. For example, in the case where a tap operation is performed on the screen of the smartphone 5 as shown in FIG. 29(A), the number of taps, the cumulative time of taps, etc. may be used as the criterion for determination. Furthermore, if a pinch-out operation is performed on the screen of the smartphone 5 as shown in FIG. The vertical angle of view for forming the corner or the other pair of sides may be a predetermined condition (see FIG. 16). These angle of view values may be set in advance as a threshold management table having a plurality of combinations, as shown in FIG. 16(a). Further, when a tap operation or a pinch-out operation is performed on the display 517 of the smartphone 5 and the above-mentioned predetermined condition DC or more is reached, the transmitting/receiving unit 51 transmits a request to generate a partial still image (time). For example, the trigger may be a shutter command). Note that the operation example for the subject displayed on the display 517 of the smartphone 5 is not limited to a tap operation or a pinch-out operation, and may be realized by other operations.

Using the method described above, the transmitting/receiving unit 51 of the smartphone 5 transmits a request for still image data to the receiving unit 71a of the image management system 7 (step S32). At this time, the partial image parameters specified by the viewer and the time information when the still image data was requested are also transmitted to the receiving unit 71a of the image management system 7. The time information when the still image data is requested may be, for example, the time information of a clock obtained from the outside as shown in FIG. Specifically, this clock is input to the reception unit 52, and when requesting still image data, the time information indicated by the clock is acquired, and the reception unit 52 sends the data to the transmission/reception unit 51 to perform spherical photography. It may also be configured to output up to device 1.

Next, the transmitter 71b of the image management system 7 transfers a request for still image data to the transmitter/receiver of the intermediary terminal 3 (step S33). Then, the transmitting/receiving unit of the intermediary terminal 3 transfers the still image data request, partial image parameters, and time information when the still image data is requested to the transmitting/receiving unit 11 of the omnidirectional photographing device 1 (step S34). . By comparing the time information related to the time when step S34 was executed and the time information transmitted from the smartphone 5, the omnidirectional photographing device 1 determines the total time required for each process of steps S32, S33, and S34, i.e. Δtd1 can be obtained. This Δtd1 will be explained in detail later.

Next, the omnidirectional photographing device 1 performs still image data generation processing (step S35). The process of step S35 will be explained below.

<Generation of still images from spherical imaging device>
Next, the still image data generation process of the omnidirectional photographing device shown in step S35 above will be described in detail.

When the omnidirectional photographing device 1 receives a request for still image data, a still image photographing determination is made. This is because if the angle of view of the partial image is relatively narrow, the resolution of the partial still image created by conversion from the original image will not differ from the resolution of the partial moving image being transmitted. In this case, the horizontal angle of view (AH) and vertical angle of view (AV) of the horizontal and vertical angle of view threshold information shown in FIG. 16 are the minimum angle of view, and the angle of view specified by the partial image parameter is If it is wider, still image generation processing is executed (step S210 in FIG. 18).

More specifically, when the transmitting/receiving unit 11 of the omnidirectional photographing device 1 receives a request for still image data while transmitting video data, the projection method converting unit 18 performs the same process as when generating partial video data. , for data of a partial still image, a projection method is converted from an equirectangular cylindrical projection method to a perspective projection method according to the partial image parameters. In this case, the size of the generated still image is variable, and conversion is performed while maintaining the resolution of the ultra-high definition image. Assuming that the horizontal resolution of the ultra-high definition image is W and the specified horizontal angle of view is ah, the horizontal resolution Wp of the generated still image can be determined as follows.

Wp=W/360*ah...(Formula 9)
The size of W here is obtained from the W column of the horizontal/vertical angle of view threshold information (see FIG. 16(a)) read when the omnidirectional photographing device 1 is started.

Similarly, if the specified vertical angle of view is av, the vertical resolution Hp of the generated still image can be found as follows.

Hp=H/180*av...(Formula 10)
Furthermore, the still image encoding unit 20b encodes the data of the partial still image and stores it in the image buffer. Then, similarly to steps S16 and S17, the transmitting/receiving unit 11 controls the communication network band described above and transmits the data of the partial still image to the image management system 7 via the intermediary terminal 3. In this case as well, partial image parameters are transmitted.

Next, processing at the time of still image generation will be explained. FIG. 30 is a timing chart when the omnidirectional photographing device 1 performs video shooting and still image generation. In the process of step S31 in FIG. 28, when the viewer performs a predetermined process such as tapping or pinching out on the subject of interest displayed on the display 517 of the smartphone 5, the still image request is accepted and the partial Still image generation processing is started.

(Absorbing delay time that occurs when shooting still images)
In the photographing system of this embodiment, the delay time that occurs in the generation of the above-described partial still image and a method for absorbing the delay time will be described below with reference to FIGS. 30 and 31.

In FIG. 30, the imaging unit 14a and the imaging unit 14b of the omnidirectional imaging device 1 start moving video from time t1 (for example, video shooting start time t=10:00:00 in the information stored as metadata in FIG. 26). Start shooting. Thereafter, when the imaging control unit 13 receives a still image request from the viewer via the image management system 7 and the intermediary device 3 at time t1+Δta after a Δta time delay, the imaging control unit 13 receives the video data stored in the temporary storage unit 16. Generate a partial still image from . Note that in this embodiment, the imaging control unit 13 also functions as timing management means when generating high-definition still images.

Subsequently, when a still image request is received from the viewer after a time delay of t1+Δtb, the imaging control unit 13 captures (generates) a still image at time t1+Δtb, and further requests a still image from the viewer after a time delay of t1+Δtc. When the still image request is received, the still image is captured at time t1+Δtc. Note that the timing at which a still image request is made from a viewer is assumed to have a relationship of 0<Δta<Δtb<Δtc.

Next, in FIG. 31, if playback of the partial video recorded at the timing of FIG. 30 at time t3 is started on the smartphone 5, still image A is played from time (t3+Δta) to time (t3+Δtb). be done. Similarly, the still image B is played back on the smartphone 5 from time (t3+Δtb) to time (t3+Δtc). Furthermore, the still image C is similarly reproduced on the smartphone 5 from the time (t3+Δtc) to the time t4. Note that the still images A, B, and C are switched at the timing when the next still image is played. Therefore, in consideration of the case where the viewer wants to continue playing the same still image, the smartphone 5 may be provided with a pause function and a rewind function such as those provided in general video playback devices. .

In this embodiment, actually, the generation of still image A is delayed by the processing time of the system in this embodiment (for example, the addition of each processing time of steps S32, S33, and S34 in FIG. 28). The video will be shot as a still image. In other words, the still image data of the still image A is obtained at the time when the omnidirectional photographing device 1 receives the request information (the sum of the processing times of steps S32, S33, and S34 after the smartphone 5 transmits the request information). This is data of an image including the subject obtained at a timing shifted by delay time = Δtd1). This delay time is the same when still images B and still images C are reproduced. Even if information about the shooting time is managed as an example of related information (metadata) according to Exif mentioned above, there is a difference between the still image being played and its playback timing due to the delay due to processing time. will occur.

Therefore, in order to absorb the delay time Δtd1, the following method can be considered. For example, the temporary storage section 16 of the spherical photographing device 1 uses its storage area as a ring buffer to control overwriting of stored video data. Assuming this control, the imaging control unit 13 of the omnidirectional photographing device 1 starts from an arbitrary time when a still image request is received from the viewer (step S31 in FIG. 28), and performs the following step S32. , S33 and S34 (=Δtd1) before the time, that is, the time shown by the following equation 11,
(t1+Δta)−Δtd1...(Formula 11)
It becomes possible to generate a partial moving image as a still image A at a time (point in time) as far back as the current time. Note that in the case of a photographing system in which the storage control section 74 of the image management system 7 uses the storage section 7000 to manage moving images of the omnidirectional photographing device 1, the storage control section 74 uses the storage section 7000 as a ring buffer. May be controlled.

In other words, the imaging control unit 13 determines the time when the operation accepted by the viewer exceeds a predetermined condition (for example, the area enlarged by the viewer's pinch-out operation on the subject reaches a predetermined angle of view or more). It becomes possible to generate the still image A using a video that was distributed at a time that is the time required for processing performed by the image management system 7, the intermediary terminal 3, and the smartphone 5 from the time when the image management system 7, the intermediary terminal 3, and the smartphone 5 performed. However, if the imaging system does not require the intermediary terminal 3, the processing time at the intermediary terminal 3 can be ignored. The above-mentioned concept regarding absorption of delay time is the same for still images B and still images C, and it is possible to eliminate the influence of timing delay when generating a high-definition still image of the subject that has attracted the attention of the viewer. can. At this time, the time management of the system can be made even more accurate if, for example, the time information acquired from the GPS satellite is shared by the smartphone 5, the image management system 7, the intermediary terminal 3, and the omnidirectional photographing device 1. This delay time can be effectively absorbed.

By providing a mechanism for absorbing such delay time in the omnidirectional photographing device 1, a still image of the subject of interest to the viewer can be accurately photographed at the desired timing and played back to the viewer. becomes possible.

Note that the above-mentioned control is based on the standard video playback time (for example, 30 fps) managed by the playback time management unit 59 of the smartphone 5, the smartphone 5 (step S32), the image management system 7 (step S33), and the intermediary terminal. Depending on the unit processing time (for example, clock speed, etc.) in each process in step S34, it is determined whether to ignore the delay time Δtd1 or to perform delay time absorption processing based on the above calculation formula. It may be selected. Note that the sequence diagram shown in FIG. 28 is an example, and the process is not limited to this.

Next, the transmitting/receiving unit 11 of the omnidirectional photographing device 1 transmits the still image data according to the request to the transmitting/receiving unit of the intermediary terminal 3 (step S36). This still image data is ultra-high definition partial still image data. Note that the still image data transmission may be started at a time when the amount of network traffic is low, such as at night, or when the delay time is measured and the delay time is shorter than a predetermined delay time. Thereby, the transmitting/receiving section of the intermediary terminal 3 transfers the still image data to the receiving section 71a of the image management system 7 (step S37).

Next, the video playback metadata creating unit 72 of the image management system 7 updates the metadata for superimposed display by superimposing still image data on the metadata file created in step S17 (step S38). At this time, the video playback metadata creation unit 72 stores still image identification information and still image shooting time information (corresponding time information) indicating time information at which the still image was shot, among the metadata configurations described above. The process of step S38 will be explained in detail later.

Next, when the receiving unit 71a of the image management system 7 receives the still image data, the storage control unit 74 causes the storage unit 7000 to store the still image data (step S39).

Note that the processes from step S31 to step S39 may be performed multiple times during video distribution according to instructions from the viewer. Furthermore, in this embodiment, still images are taken at each request from the viewer, but it is also possible to take still images at regular intervals using the interval shooting function installed in general digital cameras. good. Furthermore, the processes in step S38 and step S39 may be executed with their order changed.

The transmitter 71b of the image management system 7 then transfers the still image data to the transmitter/receiver 51 of the smartphone 5 (step S40). At this time, regarding the timing of starting the transfer, band control of the communication network, etc. is performed as in the case of data transmission from the omnidirectional photographing device 1 to the image management system via the intermediary terminal 3.

Next, the display control unit 58 of the smartphone 5 superimposes an ultra-high-definition partial still image on the wide-angle partial video, instead of the partial video that has been superimposed so far, and displays it on the display 517 (step S41 ). Regarding the display timing of the partial still image performed by the smartphone 5 in this step S41, one of the following two cases can be considered. One is a case where the smartphone 5 displays a generated ultra-high-definition partial still image superimposed on the video being distributed while the omnidirectional photographing device 1 is live streaming a video. The other is that after the live broadcast by the omnidirectional camera 1 has finished and a predetermined time has elapsed, the smartphone 5 plays (displays) the live broadcast video at the same timing as the one displayed during the live broadcast. This is a case where an ultra-high-definition partial still image generated at the same time is displayed superimposed on the video being played. In other words, one is an ultra-high-definition partial still image related to the still image data sent by the omnidirectional camera 1 in response to the request information, and the other is the ultra-high-definition partial still image that was displayed when the request information was sent. This is a composite of wide-angle partial videos. The other is that after a predetermined period of time has elapsed since the omnidirectional imaging device 1 transmitted the still image data, the display control unit 58 displays a wide-angle partial video on the display screen, and the omnidirectional imaging device 1 transmits the still image data. This is a state in which an ultra-high-definition narrow-angle still image related to still image data and a wide-angle partial moving image that was being displayed when the request information was transmitted in the currently displayed wide-angle partial moving image are combined. At any of these display timings, the viewer of the smartphone 5 can view a display screen in which an ultra-high-definition partial still image is superimposed on the video being distributed or the video being played.

Note that if the determining unit 25 determines in step S35 that the projection format converting unit 18 is not to convert the ultra-high-definition still image data into a projection format, in steps S36 to S37, instead of the still image data, Messages such as "still images will not be transmitted" or "the viewing angle is insufficient" may be transmitted. In this case, in step S41, the message may be displayed while the narrow-angle partial video remains superimposed on the wide-angle partial video.

<Updating metadata>
Next, the metadata update process shown in step S38 mentioned above will be explained in detail.

When the receiving unit 71a of the image management system 7 receives still image data from the omnidirectional imaging device 1 via the intermediary device 3 during recording of distributed video data, the storage control unit 74 stores the received still image data. The data is stored in the storage unit 7000. Next, the video playback metadata creation unit 72 adds the still image data information newly stored in the storage unit 7000 to the metadata file created in step S27. At this time, the video playback metadata creation unit 72 stores still image identification information and still image shooting time information, which are examples of still image data information, in the storage unit 7000, among the metadata structures described above.

<Playback of still images on smartphone>
Next, the still image data playback process of the smartphone 5 shown in step S41 above will be described in detail.

In the smartphone 5, the transmitting/receiving unit 51 separates the still image data sent from the omnidirectional photographing device 1 into partial still image data and partial image parameters. The still image decoding unit 53b decodes the separated still image data.

<Download video data>
Next, a case where video data stored and managed in the image management system 7 is downloaded to the smartphone 5 will be described using FIG. 32. First, when the viewer operates the smartphone 5, the receiving unit 22 of the omnidirectional photographing device 1 receives a video playback request (step S51). Thereby, the transmitting/receiving unit 51 of the smartphone 5 transmits a recording data request to the receiving unit 71a of the image management system 7 (step S52).

Upon receiving the recording data request, the receiving unit 71a of the image management system 7 transmits the video file, still image file, and video playback metadata read from the storage unit 7000 via the storage control unit 74 to the transmitting/receiving unit of the smartphone 5. 51 (step S53).

When the transmitting/receiving unit 51 of the smartphone 5 receives the video file, still image file, and video playback metadata from the image management system 7, the storage control unit 62 stores the received video file, still image file, and video playback metadata. The information is stored (downloaded) in the unit 5000, and the moving images and still images are reproduced (step S54).

(Display selection when playing videos and still images)
As described above in step S54, each display pattern may be selectively controlled to be displayed based on, for example, the operating environment of the imaging system and the specifications of each device.

FIG. 33 is a diagram showing an example of a selection button and a decision button. As shown in FIG. 33, a whole video selection button 551 (wide-angle video), a partial video selection button 552 (narrow-angle video), a partial still image selection button 553 (narrow-angle still image), and a determination button 554 are provided on the smartphone 5. It's okay to be hit. Hereinafter, the entire video will be referred to as a wide-angle video, the partial video will be referred to as a narrow-angle video, and the partial still image will be referred to as a narrow-angle still image. The above-described whole video selection button 551 (wide-angle video), partial video selection button 552 (narrow-angle video), and partial still image selection button 553 (narrow-angle still image) are operated by the viewer on the display 517 of the smartphone 5. This is an operation button that is accepted by the reception unit 52, which is an example of reception means. These whole video selection button 551, partial video selection button 552, and partial still image selection button 553 can be used alone or in combination, i.e., at least one of the display formats related to wide-angle video, partial video, and partial still image. The above may be selected. For example, when displaying a wide-angle video and a narrow-angle video on the display 517, or when displaying a narrow-angle video and a narrow-angle still image on the display 517, multiple combinations of the above may be used depending on the usage pattern desired by the viewer. Each selected selection button may be selected.

By providing such selection buttons and decision buttons, priority is given to the video according to the resolution of the wide-angle video or narrow-angle video when the viewer continues to enlarge the subject in the viewer during video playback (resolution The smartphone 5 may be configured so that it is possible to select between giving priority to resolution (sacrificing video) or giving priority to resolution (sacrificing video). In other words, in such a case, the smartphone 5 may provide the reception section 52 with selection buttons, icons, etc. as a selection section that the viewer can select, and accept input from at least one button or icon. good.

Furthermore, by providing a determination button below the selection button or icon, the reception unit 52 can receive determination of the function and combination of functions selected by the viewer. In addition to determining functions and combinations of functions by operating the OK button, the function of the button or icon selected by the viewer is determined after the button or icon selected by the viewer is displayed for a predetermined period of time. may be configured so that the operation of the determination button can be omitted.

(Pattern of superimposed display method)
Next, a specific example of superimposing and displaying at least one of a narrow-angle video and a narrow-angle still image on a wide-angle video in the reproduction process performed in step S54 described above, and its display pattern will be described. The superimposition area creation unit 54, image creation unit 55, image superimposition unit 56, projective conversion unit 57, and display control unit 58 of the smartphone 5 create a high-definition narrow-angle video and a high-definition narrow-angle video for the wide-angle video while playing the wide-angle video. At least one narrow-angle still image with higher definition than the narrow-angle video is superimposed and displayed on the display 517 of the smartphone 5. Here, according to a command from the CPU 501 according to a smartphone program, the superimposed area creation section 54, image creation section 55, image superimposition section 56, projective transformation section 57, and display control section 58 collectively function as a superimposition display control section 63. You may let them. Further, this superimposed display control section 63 is an example of superimposed display control means.

Further, the superimposed image determination unit 61, which is responsible for controlling the superimposition of moving images and still images, determines which image, a narrow-angle moving image or a narrow-angle still image, should be superimposed on a wide-angle moving image, as described above. This superimposed image determining section 61 functions as an example of superimposed image determining means. Note that the processing related to superimposition and display after the determination regarding superimposed display is performed is the same as the control in streaming distribution, so the explanation will be omitted.

Next, variations of the display method for displaying the superimposed (synthesized) image determined by the superimposed image determination unit 61 will be described below. In other words, the superimposed image determination unit 61 configures wide-angle video data constituting the received wide-angle video (whole video), narrow-angle video data constituting the narrow-angle video (partial video), and narrow-angle still images (partial still images). It has the function of a superimposed image determining means that determines which combination of narrow-angle still image data to combine.

(Display method 1)
The superimposed image determination unit 61 determines which image, a narrow-angle video or a narrow-angle still image, should be superimposed on the wide-angle video, based on the display area specified by the viewer and the information on the pre-designated partial area. to decide. The superimposed image judgment unit 61 determines that if the number of vertical pixels in the viewing area on the display 517 is Th, the number of pixels in the partial area is Ph, the number of horizontal pixels in the viewing area is Tw, and the number of pixels in the partial area is Pw, then Ph >H'/2 or Ph>W', an ultra-high-definition still image is superimposed, and in other cases, a moving image is superimposed, thereby automatically switching. Ph and Pw can be approximated by the following equations.

P=b/a*T... (Formula 12)
Note that the parameters a, b, p, and t in the above equation are given based on, for example, the two-dimensional conceptual diagram showing the display area and the attention area shown in FIG. 34. By calculating the required resolution (number of pixels) and automatically switching the superimposed images in this way, the viewer can always view images with the optimal resolution simply by specifying the display area.

(Display method 2)
On the other hand, even under conditions where the resolution can be improved by superimposing still images, it may be more beneficial for viewers to display the video as it is. Therefore, in this embodiment, as will be described later, the smartphone 5 may be provided with a selection button for selecting which of a wide-angle video, a narrow-angle video, and a narrow-angle still image to play, and a determination button for determining those selections. good.

If the viewer wants to view with more emphasis on movement than resolution, the superimposed image determination unit 61 may control the display to switch between narrow-angle video and narrow-angle still images according to the viewer's instructions.

(Display method 3)
In addition, since the area where the still image was shot is likely to be a subject or area that the viewer who plays the video particularly wants to see in detail, if a narrow-angle still image is present during video playback, The superimposed image determination unit 61 may rotate the viewpoint so that the still image is displayed at the center of the screen of the display 517 in cooperation with the superimposed display control unit 63.

(Display method 4)
Furthermore, if the camera moves during shooting, there is a problem that the partial still image and the entire video may be out of sync during playback. In that case, the background moving image may also be displayed in a still state during the period in which the still image is displayed in a superimposed manner. In this way, when there is a still image, the superimposed image determination unit 61 cooperates with the superimposed display control unit 63 to face the direction of the still image and display the still image for a predetermined period of time (for example, an arbitrary set period of time such as 5 seconds). Alternatively, the still image may be displayed only for a period of time until the next still image is displayed.

(Example of video and still image display)
FIG. 34 is a two-dimensional conceptual diagram showing the display area and the attention area. This two-dimensional conceptual diagram is as described in the display pattern 1 above. Further, FIGS. 35 to 40 are examples showing display examples of partial moving images and still images superimposed on a reproduced moving image, and display patterns corresponding to the display examples. Examples of each moving image and still image and each display pattern will be described below.

First, FIG. 35(a) shows a state in which a wide-angle video is played on the display 517 of the smartphone 5. In other words, this state is a state in which a low-definition wide-angle video is displayed. In this state, when a part of the wide-angle video (around the castle tower) is enlarged by the viewer's pinch-out operation, that is, when a part of the low-definition wide-angle video is enlarged, the image creation unit 55 enlarges the video. The generated wide-angle video is used to generate a narrow-angle video with relatively higher definition than the wide-angle video, and the image superimposition unit 56 superimposes the generated narrow-angle video near the castle tower in the wide-angle video. That is, this state is a state in which a low-definition narrow-angle video and a high-definition narrow-angle video are superimposed. Thereafter, the display control unit 58 causes the display 517 to display the narrow-angle video superimposed on the wide-angle video by the image superimposing unit 56 and the video projectively transformed by the projective transform unit 57 (FIG. 35(b)). In addition, in FIG. 35(b), the display control unit 58 displays the four sides of the outer frame of the partial video superimposed on the wide-angle video with dotted lines. It does not have to be displayed on top.

Next, when the enlargement operation is continued by the viewer and the area of the narrow-angle video reaches a predetermined condition (range) or more, the image creation unit 55 converts the enlarged image in the narrow-angle video from the narrow-angle video. The image superimposing unit 56 superimposes the narrow-angle still image around the castle tower in the wide-angle video. After that, the display control unit 58 displays the narrow-angle still image superimposed on the wide-angle video (FIG. 35(c)). That is, this state is a state in which a narrow-angle moving image and an enlarged ultra-high-definition narrow-angle still image are superimposed. In addition, in FIG. 35(c), the four sides of the outer frame of the ultra-high-definition narrow-angle still image superimposed on the wide-angle video are shown by dotted lines, but this is shown for convenience, so the display control unit 58 does not have to be displayed on the actual display 517. Furthermore, the display control unit 58 may control the superimposed narrow-angle still image to be displayed on the display 517 of the smartphone 5 for a predetermined period of time. In addition, the above-mentioned predetermined time refers to the time during which still images A, B, C, etc. are displayed in FIG. 31, which relates to the timing of video playback and still image playback described above, for example, but is calculated by other methods. It may be the time when

On the other hand, the display control unit 58 changes from the state shown in FIG. 35(c) to display an ultra-high-definition narrow-angle still image (instead of the wide-angle video) in the entire display area of the display 517 where the wide-angle video was being displayed. (Fig. 35(d)). That is, in this state, the enlarged ultra-high-definition narrow-angle still image is displayed in the entire displayable area of the display 517 of the smartphone 5.

Each display pattern shown in FIGS. 36 to 40 will be described below. FIG. 36 is an example of a display pattern corresponding to an enlargement of a moving image and a display example of a high-definition still image superimposed on the moving image. 36(a), (b), (c), and (d) correspond to each display pattern of FIG. 35(a), (b), (c), and (d), respectively. In detail, FIG. 36(a) is an example of a state in which a low-definition wide-angle video (wide-angle video WM; hereinafter simply referred to as wide-angle video) is displayed in the entire display area of the display 517, and FIG. 36(b) is an example of a state in which a low-definition wide-angle video and a high-definition narrow-angle video (narrow-angle video NM, hereinafter simply referred to as narrow-angle video) are superimposed, and FIG. This is an example of a state in which a low-definition wide-angle video and an ultra-high-definition narrow-angle still image (narrow-angle still image NS; hereinafter simply referred to as a narrow-angle still image) are superimposed when a narrow-angle video is enlarged. Further, FIG. 36(d) shows a state in which, when the narrow-angle video in FIG. 36(b) is enlarged, an ultra-high-definition narrow-angle still image is replaced by the narrow-angle video and displayed in the entire display area of the display 517. This is an example. Note that when a low-definition wide-angle video is the first image, a high-definition narrow-angle video or an ultra-high-definition narrow-angle still image is the second image. Alternatively, when the high-definition narrow-angle video is the first image, the ultra-high-definition narrow-angle still image is the second image.

FIG. 37 is another display example (display pattern 2) regarding the enlargement of a moving image and the display example of a high-definition still image superimposed on the moving image. In detail, FIGS. 37(a), (b), and (d) are similar to FIGS. 36(a), (b), and (d). FIG. 37(c) is an example of a state in which a high-definition narrow-angle video and an ultra-high-definition narrow-angle still image are superimposed.

FIG. 38 is another display example (display pattern 3) regarding the enlargement of a moving image and the display example of a high-definition still image superimposed on the moving image. In detail, FIGS. 38(a), (b) and (d) are similar to FIGS. 37(a), (b) and (d). FIG. 38(c) is an example of a state in which a low-definition wide-angle video and a high-definition narrow-angle video are superimposed, and a high-definition narrow-angle video and an ultra-high-definition narrow-angle still image are further superimposed.

FIG. 39 is another display example (display pattern 4) regarding the enlargement of a moving image and the display example of a high-definition still image superimposed on the moving image. In detail, FIGS. 39(a) and (b) are similar to FIGS. 38(a) and (b). 39(c) is an example of a state in which a high-definition narrow-angle video is displayed in the entire display area of the display 517, and FIG. 39(d) is an example of a state in which a high-definition narrow-angle video is displayed in the state of FIG. 39(b). When enlarged, the high-definition narrow-angle video is displayed in the entire display area of the display 517 as shown in FIG. 39(c), and then the high-definition narrow-angle video and the ultra-high-definition narrow-angle still image are This is an example of a superimposed state.

FIG. 40 is another display example (display pattern 5) regarding the enlargement of a moving image and the display example of a high-definition still image superimposed on the moving image. In detail, FIGS. 40(a), (b), and (c) are similar to FIGS. 39(a), (b), and (c). FIG. 40(d) shows that when a high-definition narrow-angle video is enlarged in the state of FIG. This is an example of a state where the video is displayed in the entire display area of the display 517 in place of the narrow-angle video.

By performing the above-mentioned control, the superimposed image determination unit 61 controls the playback of wide-angle videos and narrow-angle videos and the narrow-angle still images superimposed on those videos, for example, when the viewer uses the smartphone 5. It is also possible to suppress the amount of data communication (expensive usage fees) that the viewer does not want in connection with the generation and reproduction of .

[Main effects of the embodiment]
As explained above, according to the present embodiment, while suppressing the amount of data communication for wide-angle videos such as whole images and narrow-angle videos such as partial images, the resolution of partial images is not lowered as much as possible, so that viewers can This has the effect that the area of interest can be viewed more clearly.

Furthermore, according to the present embodiment, in order to reduce the amount of communication, the omnidirectional imaging device 1 captures a low-definition overall image obtained by reducing the entire omnidirectional image, and an image of interest in the omnidirectional image (overall image). The receiving smartphone 5 not only superimposes (synthesizes) the partial image on the whole image, but also makes the low-definition whole image and the high-definition partial image different. Even if the projection method is used, it can be combined and displayed on the smartphone 5, so the projection method has a high degree of versatility.

〔supplement〕
In the embodiment described above, the determining unit 25 of the omnidirectional photographing device 1 determines that the entire area of one frame of a partial image, which is a partial area of the entire video, has a horizontal and vertical angle of view managed by the threshold management unit 1001. It is determined whether the area is smaller than a predetermined area indicated by a predetermined threshold value of threshold information (see FIG. 16). If the value is small, the determining unit 25 prevents the projection method converting unit 18 from converting the data of the ultra-high definition still image into the projection method; Although detailed still image data is converted to a projection method, the present invention is not limited to this. For example, if the size is small, the transmitting/receiving unit 11 performs processing to prevent data of the partial still image after projection method conversion stored in the still image storage unit 29 from being transmitted to the smartphone 5 side, and if the size is large, the transmitting/receiving unit 11 stops A process may be performed in which the data of the partial still image after projection method conversion stored in the image storage unit 29 is transmitted to the smartphone 5 side. In this case, the determining unit 25 may instruct the transmitting/receiving unit 11 whether to transmit or not, or the determining unit 25 may instruct the still image storage unit 29 to determine whether the still image is to be stored in the storage unit of the omnidirectional photographing device 1. It may also be possible to instruct the transmitter/receiver 11 side whether or not to transmit the data of the partial still image after projection method conversion.

Further, the omnidirectional photographing device 1 is an example of a photographing device, and photographing devices include digital cameras, smartphones, and the like that obtain ordinary planar images. In the case of a digital camera or smartphone that obtains a normal planar image, it is possible to obtain a relatively wide-angle image (wide-angle image) rather than a spherical image.

Furthermore, the smartphone 5 is an example of a communication terminal or an image processing device, and the communication terminal or image processing device includes PCs such as a tablet PC (Personal Computer), a notebook PC, and a desktop PC. Furthermore, the communication terminal or image processing device includes a smart watch, a game device, a car navigation terminal installed in a vehicle, and the like.

In the above embodiment, the low-definition image is the entire video, which is the entire region of the image data obtained from the imaging units 14a and 14b of the omnidirectional imaging device 1, and the high-definition image is the entire video, which is the entire region of the image data obtained from the imaging units 14a and 14b of the omnidirectional imaging device 1. Although the description has been made regarding a partial video that is a region, the present invention is not limited to this. The low-definition image may be an image of a part of the area A1 of the entire area of the image of the moving image data obtained from the imaging units 14a and 14b. In this case, the high-definition image is an image of a further part of the area A2 of the part of the area A1. That is, the relationship between a low-definition whole video and a high-definition partial video is that the former is a wide-angle video as a whole video, while the latter is a narrow-angle video as a partial video. Note that partial still images can also be called narrow-angle still images in the same way as partial moving images. Note that in this embodiment, the wide-angle image (including both still images and moving images) is an image that is photographed with a photographing device equipped with a so-called wide-angle lens or a fisheye lens, and in which distortion occurs. Further, a narrow-angle image (including both a still image and a moving image) is a part of an image photographed by a photographing device equipped with a wide-angle lens or a fisheye lens, and is an image whose angle of view is narrower than that of a wide-angle image.

In the above embodiment, a case has been described in which a planar image is superimposed on a spherical image, but superimposition is an example of composition. In addition to superimposition, composition also includes pasting, fitting, overlapping, and the like. Further, the combination of images includes a case where a second image is combined with a first image, and a case where a first image is combined with a second image. That is, combining images includes combining a first image and a second image. Furthermore, the superimposed image is an example of a composite image. In addition to the superimposed image, the composite image includes a pasted image, an embedded image, a superimposed image, and the like. Further, the image superimposing section 56 is an example of image compositing means.

Furthermore, the equirectangular projection image EC and the plane image P may both be still images, both may be moving image frames, or one may be a still image and the other may be a moving image frame.

Furthermore, in the embodiment described above, the projection method converting unit 18 converts the projection method of the high-definition image data acquired from the temporary storage unit 16 with the same definition, but the present invention is not limited to this. For example, if the definition is higher than the overall image data output from the low-definition changing unit 17a, the projection method converting unit 18 changes the definition of the image data acquired from the temporary storage unit 16 when converting the projection method. It may be lower. In other words, any method may be used as long as the image data output from the projection method converting section 18 is controlled to have a relatively higher definition than the image data output from the low definition changing section 17a.

Each of the functional configurations shown in FIGS. 13-15 can be implemented in the form of a software functional unit and stored on a computer-readable storage medium when sold or used as an independent product. In this case, the essential part of the technical solution of the present embodiment, or the part that contributes to the prior art, or the part of the technical solution described above is expressed in the form of a computer software product. The above-mentioned computer software product is stored in a storage medium and provides a plurality of instructions that cause a computer device (such as a personal computer, a server, or a network device) to perform all or some of the steps of the above-mentioned method according to each of the above embodiments. include. Note that the above-mentioned storage medium includes various media capable of storing program codes, such as a USB memory, a removable disk, a ROM, a RAM, a magnetic disk, or an optical disk.

Furthermore, the method according to the embodiment described above is applied to or realized by a processor. A processor is an integrated circuit board capable of processing signals. Each step of the method of each of the embodiments described above is implemented by an integrated logic circuit, which is hardware in a processor, or by instructions in the form of software. The processor may be, for example, a general purpose processor, a digital signal processor (DSP), a special purpose integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. , each method, step, and logic box disclosed in each of the above embodiments can be realized or executed. A general purpose processor may be a microprocessor or any general purpose processor. Each step of the method according to each of the above embodiments may be realized by being executed by a decoder that is hardware, or may be realized by a combination of hardware and software in the decoder. The software modules are stored in art-advanced storage media, such as random memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, and the like. From the memory comprising the storage medium on which this software is stored, the processor reads information and adapts it to the hardware to implement the steps of the method.

The embodiments described above may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. In terms of hardware implementation, the processing unit may include one or more special purpose integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gates. The present invention may be implemented by an array (FPGA), general purpose processor, controller, microcontroller, microprocessor, other electronic unit or combination thereof that performs the functions of the invention. Regarding software implementation, the above technology is realized by modules (eg, processes, functions, etc.) that implement the functions described above. Software code is stored in memory and executed by a processor. Note that the memory is implemented inside or outside the processor.

1. Omnidirectional photography device (an example of photography device)
5 Smartphone (an example of a communication terminal, an example of an image processing device)
7 Image management system (an example of image management device)
13 Imaging control unit (an example of timing management means)
14a Imaging unit 14b Imaging unit 15 Image processing unit 16 Temporary storage unit 17 Low definition changing unit (an example of changing means)
18 Projection method conversion unit (an example of projection method conversion means)
19 Coupling part (an example of coupling means)
20a Video encoding section 20b Still image encoding section 51 Transmitting/receiving section (an example of receiving means)
52 Reception Department (an example of reception means)
53a Video decoding section 53b Still image encoding section 54 Superimposition area creation section 55 Image creation section 56 Image superimposition section (an example of image synthesis means)
57 Projective transformation unit (an example of projective transformation means)
58 Display control unit (an example of display control means)
59 Playback time management section 60 Playback frame extraction section 61 Superimposed image judgment section (an example of superimposed image judgment means)
63 Superimposed display control unit (an example of superimposed display control means)
551 Overall video selection button (an example of selection method)
552 Partial video selection button (an example of selection means)
553 Partial still image selection button (an example of selection means)

Japanese Patent Application Publication No. 2006-340091

Claims (17)

A communication terminal that receives video data transmitted by a photographing device that obtains video data of a predetermined definition by photographing a subject, and displays the video,
Receiving means for receiving the video data of the predetermined definition;
Display control means for displaying a wide-angle video that is all or a part of the video related to the video data on a display means;
reception means for accepting an operation to enlarge the wide-angle video displayed on the display means;
When the accepting means accepts a predetermined enlargement of the wide-angle video displayed on the display means, the image capturing device displays a part of the subject that is higher in definition than the video data. a transmitting means for transmitting request information for requesting detailed still image data;
has
The receiving means receives the still image data transmitted by the photographing device based on the request information,
The display control means displays on the display means a combined narrow-angle still image of the still image data generated when the reception means receives a predetermined enlargement, and the wide-angle video. Communication terminal with special features. 2. The communication terminal according to claim 1, wherein the still image data is obtained by photographing the subject when the photographing device receives the request information. The display control means includes:
a state in which a narrow-angle still image related to the still image data transmitted by the photographing device in response to the transmission of the request information and the wide-angle video that was being displayed when the request information was transmitted are combined; or The photographing device displays the wide-angle video after a predetermined time has elapsed after transmitting the still image data, and the narrow-angle still image related to the still image data transmitted by the photographing device and the wide-angle that is being displayed. 2. The communication terminal according to claim 1, wherein a composite state of the wide-angle video that was being displayed when the request information was transmitted is displayed on the display means. The receiving means receives narrow-angle video data that is data of a narrow-angle video that has a narrower angle and higher definition than the wide-angle video and has a wider angle and lower definition than the narrow-angle still image,
When the receiving means receives the first predetermined enlargement as the predetermined enlargement, the display control means displays the wide-angle video and the narrow-angle video in a combined state on the display means, Furthermore, when the receiving means receives a second predetermined enlargement that is larger than the first predetermined enlargement, the display control means is configured to display a composite image of the wide-angle video and the narrow-angle still image. 2. The communication terminal according to claim 1, wherein the communication terminal displays the status on the display means. 5. The communication according to claim 4, wherein when the accepting means accepts the second predetermined enlargement, the display controlling means displays the narrow-angle still image instead of the narrow-angle moving image. terminal. When the accepting means accepts the second predetermined enlargement, the display controlling means displays the combined narrow-angle video and the narrow-angle still image on the display means. The communication terminal according to claim 4. The communication terminal according to any one of claims 4 to 6 , further comprising an image composition unit that composes the narrow-angle still image with the wide-angle video. 8. The communication terminal according to claim 7, further comprising projection conversion means for converting a projection method of the wide-angle video into which the narrow-angle still image has been synthesized by the image synthesis means. comprising a superimposed image determining means for determining whether to synthesize a combination of at least two of the wide-angle video, the narrow-angle video, and the narrow-angle still image according to a maximum display resolution that can be displayed by the display means;
9. The communication terminal according to claim 7, wherein when the superimposed image determining means determines that the image should be combined, the image combining means combines the narrow-angle still image with the wide-angle moving image. The communication terminal according to any one of claims 4 to 6, further comprising an image composition unit that combines the narrow-angle video with the wide-angle video. 11. The communication terminal according to claim 10, further comprising projection conversion means for converting a projection method of the wide-angle video into which the narrow-angle video has been synthesized by the image composition means. a photographing device that obtains video data of a predetermined definition by photographing a subject;
an image management device that receives and manages the video data obtained by the photographing device;
A photographing system in which a communication terminal that receives the video data transmitted from the image management device and displays the video is connected via a network,
The communication terminal is
Receiving means for receiving the video data of the predetermined definition;
Display control means for displaying a wide-angle video that is all or a part of the video related to the video data on a display means;
reception means for accepting an operation to enlarge the wide-angle video displayed on the display means;
When the accepting means accepts a predetermined enlargement of the wide-angle video displayed on the display means, the image capturing device displays a part of the subject that is higher in definition than the video data. a transmitting means for transmitting request information for requesting detailed still image data;
has
The receiving means receives the still image data transmitted by the photographing device based on the request information,
The display control means displays on the display means a combined narrow-angle still image of the still image data generated when the reception means receives a predetermined enlargement, and the wide-angle video. Featured shooting system. The photographing device is
request information receiving means for receiving the request information transmitted from the communication terminal;
still image generation means for generating the still image data based on the received request information;
still image transmitting means for transmitting the generated still image data to the communication terminal;
13. The imaging system according to claim 12, comprising: The photographing device is
When generating the still image data, timing management for generating video data as the still image data at a time that is back by the time required from the time when the accepting means accepts a predetermined enlargement to the time when the request information is received. 14. The imaging system according to claim 13, further comprising means. The photographing device is characterized in that it is a celestial sphere photographing device having a celestial sphere image generation means for generating spherical image data from video data of the predetermined definition obtained by photographing the subject. The imaging system according to any one of claims 12 to 14. An image processing method executed by a communication terminal that receives video data transmitted by a photographing device that obtains video data of a predetermined definition by photographing a subject and displays the video, the method comprising:
a receiving step of receiving the video data of the predetermined definition;
a display control step of displaying a wide-angle video that is all or part of the video related to the video data on a display means;
a reception step of accepting an operation to enlarge the wide-angle video displayed on the display means;
When a predetermined enlargement of the wide-angle video displayed on the display means is accepted in the accepting step, the photographing device is requested to display a part of the subject that is higher in definition than the video data. a sending step of sending request information for requesting detailed still image data;
has
The receiving step includes a process of receiving the still image data transmitted by the photographing device based on the request information,
The display control step is a process of displaying a combined narrow-angle still image of the still image data generated by accepting the predetermined enlargement in the accepting step and the wide-angle video on the display means. An image processing method characterized by comprising: A program for causing a computer to execute the image processing method according to claim 16.