JP2020162118A - Photographing device, photographing system, image processing method, and program - Google Patents

Abstract

To make it possible, in a technology to display a high-definition partial image superimposed on a low-definition entire image, to reduce the amount of data communication of a wide-angle moving image of the entire image and a narrow-angle moving image of the partial image, and allow a browser to more clearly browse an area of attention while not reducing the definition of the partial image as much as possible.SOLUTION: A full spherical photographing device 1 photographing a subject to obtain moving image data with a predetermined definition changes the definition of a wide-angle moving image that is all or part of a moving image related to the moving image data with the predetermined definition to a low definition (S140), converts the projection system of a narrow-angle moving image that is a partial area of the wide-angle moving image with a higher definition than that of the wide-angle moving image after the change (S150), reduces the definition of a frame of the low-definition wide-angle moving image and a frame of the high-definition narrow-angle moving image and couples the frames into one frame (S160), and converts the projection system of a narrow-angle still image that is the partial area of a wide-angle still image as a frame of the wide-angle moving image before the change to low definition (S210).SELECTED DRAWING: Figure 18

Description

The present invention relates to a photographing device, a photographing system, an image processing method, and a program.

Instead of a camera system consisting of a pan / tilt and zoom lens using a turntable, a fisheye lens is used to shoot the subject and the obtained wide-angle image data is distributed, and the image is displayed on the viewer on the receiving side. An electronic pan / tilt / zoom technique for generating and displaying a partial image is already known.

However, the technique of distributing the wide-angle image data obtained by photographing the subject also distributes the image data of the area not attracting attention, so that there is a problem that the amount of data communication increases.

On the other hand, in order to reduce the amount of communication, the photographing device delivers a low-definition whole image obtained by reducing the entire wide-angle image and data of a high-definition partial image which is a region of interest in the wide-angle image. A technique for fitting a partial image into an entire image on the receiving side is disclosed (see Patent Document 1).

However, when the whole image and the partial image are moving images, it is difficult to reduce the amount of data communication even if the whole image has low definition. On the other hand, the viewer does not always browse the video carefully, and there is a need that the viewer should be able to carefully browse only the partial image when the video of interest is played.

The present invention has been made in view of the above circumstances, and while suppressing the amount of data communication of a wide-angle moving image such as a whole image and a narrow-angle moving image such as a partial image, the definition of the partial image is not lowered as much as possible. The purpose is to enable the viewer to browse the area of interest more clearly.

The invention according to claim 1 is a photographing device for photographing a subject and obtaining moving image data of a predetermined definition, and lowers a wide-angle moving image which is all or a part of the moving image related to the moving image data of the predetermined definition. A changing means for changing to fine definition, and a projection method converting means for converting a narrow-angle moving image, which is a part of the wide-angle moving image, into a higher-definition state than the wide-angle moving image after being changed to low-definition by the changing means. , The low-definition wide-angle moving image frame and the high-definition narrow-angle moving image frame are each reduced in definition and combined into one frame, and the projection method conversion means is the changing means. The imaging device is characterized in that a narrow-angle still image, which is a part of the wide-angle still image as a frame of a wide-angle moving image before being changed to low definition, is converted by the projection method.

As described above, according to the present invention, the viewer pays more clear attention by suppressing the data communication volume of the wide-angle moving image and the narrow-angle moving image while keeping the fineness of the narrow-angle image as low as possible. It has the effect of being able to browse the area.

(A) is a left side view of the spherical imaging device, (b) is a rear view of the spherical imaging device, (c) is a plan view of the spherical imaging device, and (d) is a plan view of the spherical imaging device. It is a bottom view of the spherical photography apparatus. It is a use image figure of the spherical photography apparatus. (A) is a hemispherical image (front) taken by the spherical imaging device, (b) is a hemispherical image (rear) taken by the spherical imaging device, and (c) is represented by equirectangular projection. It is a figure which showed the image. (A) is a conceptual diagram showing a state of covering a sphere with an equirectangular cylindrical projection image, and (b) is a diagram showing a spherical image. It is a figure which showed the position of the virtual camera and a predetermined area when the whole celestial sphere image is made into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. It is a figure explaining the outline of a partial image parameter. It is the schematic of the structure of the imaging system of this embodiment. It is a hardware block diagram of the spherical photography apparatus. It is a hardware configuration diagram of an intermediary terminal. It is a hardware configuration diagram of a smartphone. It is a hardware block diagram of an image management system. It is a functional block diagram of the spherical imaging apparatus of this embodiment. It is a functional block diagram of the smartphone of this embodiment. It is a functional block diagram of an image management system. (A) A conceptual diagram of horizontal and vertical angle of view threshold information, and (b) an explanatory diagram of a horizontal angle of view and a vertical angle of view. It is a sequence diagram which showed the process of generation and reproduction of each data of whole moving thing and partial moving thing. It is a conceptual diagram of an image in the process of image processing performed by an omnidirectional imaging apparatus. It is a figure explaining a partial image parameter. It is a conceptual diagram which shows the state which each frame image of a whole moving image and a partial moving image is combined. It is a conceptual diagram of an image in the process of image processing performed by a smartphone. It is a figure explaining the creation of a partial solid sphere from a partial plane. It is a two-dimensional conceptual diagram in the case where the partial image is superposed on the spherical image without creating the partial three-dimensional sphere of this embodiment. It is a two-dimensional conceptual diagram in the case where the partial three-dimensional sphere of this embodiment is created and the partial image is superposed on the spherical image. (A) Wide image display example without superimposition display, (b) Tele image display example without superimposition display, (c) Wide image display example with superimposition display, (d) Superimposition display example It is a conceptual diagram which showed the display example of a teleimage. It is a sequence diagram which showed the process of data generation and reproduction of a partial still image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Outline of Embodiment]
The outline of this embodiment will be described below.

A method of generating a spherical image will be described with reference to FIGS. 1 to 6.

First, the appearance of the spherical imaging device will be described with reference to FIG. The spherical image capturing device is a digital camera for obtaining a captured image that is the basis of a spherical (360 °) panoramic image. Note that FIG. 1A is a right side view of the spherical imaging device, FIG. 1B is a front view of the spherical imaging device, and FIG. 1C is a plan view of the spherical imaging device. FIG. 1D is a bottom view of the spherical imaging device.

As shown in FIGS. 1 (a), 1 (b), 1 (c), and 1 (d), the upper part of the spherical imaging device has a fisheye lens on the front side (front side). The image sensor (image sensor) 103a and 103b described later are provided inside the spherical lens 102a and the spherical lens 102b provided on the back side (rear side), respectively. A hemispherical image (angle of view of 180 ° or more) can be obtained by photographing a subject or a landscape through the lenses 102a and 102b. A shutter button 115a is provided on the surface of the spherical imaging device opposite to the front side. Further, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the spherical imaging device. The shutter button 115a, the power button 115b, and the Wi-Fi button 115c are all switched on and off each time they are pressed. Further, each time the shooting mode switching button 115d is pressed, the shooting mode of a still image, the shooting mode of a moving image, and the distribution mode of a moving image are switched. The shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are a kind of operation unit 115, and the operation unit 115 is not limited to these buttons.

Further, in the center of the bottom 150 of the spherical imaging device, a tripod screw hole 151 for attaching the spherical imaging device to the camera tripod is provided. Further, a Micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom portion 150. An HDMI (High-Definition Multimedia Interface) terminal is provided on the right end side of the bottom 150. HDMI is a registered trademark.

Next, the usage status of the spherical imaging device will be described with reference to FIG. Note that FIG. 2 is an image diagram of the use of the spherical imaging device. As shown in FIG. 2, the spherical imaging device is used, for example, for a user to hold in his / her hand and photograph a subject around the user. In this case, two hemispherical images can be obtained by taking an image of a subject around the user by the image sensor 103a and the image sensor 103b shown in FIG.

Next, with reference to FIGS. 3 and 4, the outline of the process from the image taken by the spherical imaging device to the creation of the equirectangular projection image EC and the spherical image CE will be described. Note that FIG. 3 (a) is a hemispherical image (front side) taken by the all-sky imaging device, FIG. 3 (b) is a hemispherical image (rear side) taken by the all-sky imaging device, and FIG. 3 (c). Is a diagram showing an image represented by the equirectangular projection (hereinafter, referred to as "equirectangular projection image"). FIG. 4A is a conceptual diagram showing a state of covering a sphere with an equirectangular cylindrical projection image, and FIG. 4B is a diagram showing a spherical image.

As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by the fisheye lens 102a described later. Further, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by the fisheye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees are combined by the spherical imaging apparatus, and as shown in FIG. 3C, the equirectangular projection image. EC is created.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4A, the equirectangular projection image is pasted so as to cover the spherical surface, and FIG. 4 (A) An omnidirectional image CE as shown in b) is created. In this way, the spherical image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The spherical image CE may be a still image or a moving image. In the present embodiment, unless otherwise specified, when the term "image" is used, both "still image" and "moving image" are included.

As described above, since the spherical image CE is an image pasted so as to cover the spherical surface, it gives a sense of discomfort to humans. Therefore, by displaying a part of the predetermined region of the spherical image CE (hereinafter, referred to as “predetermined region image”) as a flat image with less curvature, it is possible to display the image without giving a sense of discomfort to humans. This will be described with reference to FIGS. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined region when the spherical image is a three-dimensional three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the image with respect to the spherical image CE displayed as a three-dimensional three-dimensional sphere. Further, FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on a display. Further, in FIG. 6A, the spherical image shown in FIG. 4 is represented by a three-dimensional three-dimensional sphere CS. Assuming that the spherical image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is located inside the spherical image CE as shown in FIG. The predetermined region T in the spherical image CE is a photographing region of the virtual camera IC, and the position coordinates (x (rH), y) including the angle of view of the virtual camera IC in the three-dimensional virtual space including the spherical image CE. (rV), angle of view α (angle)) is specified by the predetermined area information. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. Further, the zoom of the predetermined area T can be expressed by moving the virtual camera IC closer to or further away from the spherical image CE. The predetermined region image Q is an image of the predetermined region T in the spherical image CE.

Then, the predetermined area image Q shown in FIG. 6A is displayed as an image of the photographing area of the virtual camera IC on the predetermined display as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the initial setting (default) predetermined area information. It should be noted that the predetermined area information and the position coordinates of the virtual camera IC may be indicated by the photographing area (X, Y, Z) of the virtual camera IC which is the predetermined area T.

FIG. 7 is a diagram illustrating an outline of partial image parameters. Here, a method of designating a part of the spherical image will be described. In the spherical image, the center point CP of the partial image can be represented by the direction of the camera (here, the virtual camera IC) to shoot. The center of the whole image, which is a spherical image, is the front of the whole image, the azimuth is "aa", and the elevation angle is "ea". Further, in order to represent the range of the partial image, for example, the angle of view α in the diagonal direction is used. Further, in order to represent the range in the vertical and horizontal directions, it can be represented by the aspect ratio (width w ÷ height h) of the image. Here, the diagonal angle of view and the aspect ratio are used to represent the range, but the vertical angle of view and the horizontal angle of view, the vertical angle of view and the aspect ratio, and the like may be used. In addition to the azimuth and elevation angles, the rotation angle may also be used.

[Outline of shooting system]
First, the outline of the configuration of the photographing system of the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the photographing system of the present embodiment.

As shown in FIG. 8, the photographing system of the present embodiment includes an omnidirectional photographing device 1, an intermediary terminal 3, a smartphone 5, and an image management system 7.

Of these, the spherical imaging device 1 is a special digital camera for photographing a subject, a landscape, or the like to obtain two hemispherical images that are the basis of the spherical (panoramic) image, as described above. ..

The intermediary terminal 3 is an example of a communication terminal capable of directly communicating with a communication network 100 such as the Internet. The intermediary terminal 3 plays a role of mediating the communication between the all-sky imaging device 1 and the image management system 7 by performing short-range wireless communication with the all-sky imaging device 1 that cannot directly communicate with the communication network 100. Short-range wireless communication is communication performed using technologies such as Wi-Fi, Bluetooth (registered trademark), and NFC (Near Field Communication). In FIG. 8, the intermediary terminal 3 is used like a dongle receiver on which the spherical imaging device 1 is installed.

The smartphone 5 can communicate with the image management system 7 via the communication network 100 by wire or wirelessly. Further, the smartphone 5 can display the image acquired from the spherical imaging device 1 on the display 517 described later provided in the own device.

The image management system 7 is constructed by a computer, and transmits a request for a moving image or a still image from the smartphone 5 to the spherical photographing device 1 via an intermediary terminal 3. Further, the image management system 7 transmits the moving image data and the still image data sent from the all-sky photographing device 1 via the intermediary terminal 3 to the smartphone 5.

Although only one set of the spherical imaging device 1 and the intermediary terminal 3 is shown in FIG. 1, a plurality of sets may be used. Further, although only one smartphone 5 is represented, a plurality of smartphones 5 may be used. The smartphone 5 is an example of a communication terminal. In addition to the smartphone 5, the communication terminal includes a PC (Personal Computer), a smart watch, a game device, a car navigation terminal, and the like. Further, the image management system 7 may be constructed by not only a single computer but also a plurality of computers.

[Hardware configuration of the embodiment]
Next, the hardware configurations of the spherical imaging device 1, the intermediary terminal 3, the smartphone 5, and the image management system 7 of the present embodiment will be described in detail with reference to FIGS. 9 and 12.

<Hardware configuration of spherical imaging device>
First, the hardware configuration of the spherical imaging device 1 will be described with reference to FIG. FIG. 9 is a hardware configuration diagram of the spherical imaging device. In the following, the omnidirectional imaging device 1 is an omnidirectional (omnidirectional) imaging device capable of photographing 4π radians using two image pickup elements, but the number of image pickup elements may be two or more. In addition, it does not necessarily have to be a device dedicated to omnidirectional photography, and by attaching a retrofitted omnidirectional imaging unit to a normal digital camera, smartphone, etc., it has substantially the same function as the omnidirectional photography device 1. It may be.

As shown in FIG. 9, the celestial sphere photographing apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM ( Read Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, Operation Unit 115, Network I / F 116, Communication Unit 117, Antenna 117a, Electronic Compass 118, Gyro Sensor 119, Acceleration Sensor It is composed of 120 and terminals 121.

Of these, the image pickup unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two image pickups provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images from fisheye lenses 102a and 102b into image data of electrical signals and output them. It has a timing generation circuit that generates a horizontal or vertical synchronization signal, a pixel clock, and the like, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The image sensors 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I / F bus. On the other hand, the image pickup elements 103a and 103b of the image pickup unit 101 are connected to the image pickup control unit 105 by a serial I / F bus (I2C bus or the like). The image processing unit 104, the image pickup control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Further, a ROM 112, a SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image pickup elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data. , Create data for a regular distance cylindrical projected image as shown in FIG. 3 (c).

The image pickup control unit 105 generally uses the image pickup control unit 105 as a master device and the image pickup elements 103a and 103b as slave devices, and sets commands and the like in the register group of the image pickup elements 103a and 103b by using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to take in the status data of the register group of the image pickup elements 103a and 103b and send it to the CPU 111.

Further, the image pickup control unit 105 instructs the image pickup devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some spherical imaging devices 1 may have a preview display function on a display (for example, display 517 of a smartphone 5) or a function corresponding to a moving image display. In this case, the output of the image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frames / second).

Further, as will be described later, the image pickup control unit 105 also functions as a synchronization control means for synchronizing the output timings of the image data of the image pickup elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the spherical imaging device 1 is not provided with a display, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the spherical imaging device 1 and executes necessary processing. The CPU 111 may be single or plural. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data during processing by the image processing unit 104 and data of the processed equirectangular projection image.

The operation unit 115 is a general term for operation buttons such as the shutter button 115a. By operating the operation unit 115, the user inputs various shooting modes, shooting conditions, and the like.

Network I / F116 is a general term for interface circuits (USB I / F, etc.) with external media such as SD cards and personal computers. Further, the network I / F116 may be wireless or wired. The equirectangular projection image data stored in the DRAM 114 is recorded on an external medium via the network I / F116, or is recorded as necessary via an external terminal (device) such as a smartphone 5 via the network I / F116. ) Is sent.

The communication unit 117 communicates with an external terminal (device) such as a smartphone 5 by using short-range wireless communication technology such as Wi-Fi, NFC, and Bluetooth via an antenna 117a provided in the all-sky imaging device 1. .. The communication unit 117 can also transmit the equirectangular projection image data to an external terminal (device) such as a smartphone 5.

The electronic compass 118 calculates the azimuth of the spherical imaging device 1 from the magnetism of the earth and outputs the azimuth information. This orientation information is an example of related information (metadata) along with Exif, and is used for image processing such as image correction of a captured image. The related information also includes each data of the shooting date and time of the image and the data capacity of the image data.

The gyro sensor 119 is a sensor that detects changes in angles (Roll angle, Pitch angle, Yaw angle) that accompany the movement of the spherical imaging device 1. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of captured images.

The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The spherical imaging device 1 calculates the posture (angle with respect to the direction of gravity) of its own device (omnidirectional imaging device 1) based on the acceleration detected by the acceleration sensor 120. By providing both the gyro sensor 119 and the acceleration sensor 120 in the spherical imaging device 1, the accuracy of image correction is improved.

The terminal 121 is a concave terminal for Micro USB.

<Hardware configuration of brokerage terminal>
Next, the hardware configuration of the brokerage terminal 3 will be described with reference to FIG. Note that FIG. 10 is a hardware configuration diagram of the intermediary terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 10, the intermediary terminal 3 follows the control of the CPU 301 that controls the operation of the entire intermediary terminal 3, the ROM 302 that stores the basic input / output program, the RAM 303 that is used as the work area of the CPU 301, and the CPU 301. It includes an EEPROM (Electrically Erasable and Programmable ROM) 304 that reads or writes data, and a CMOS sensor 305 as an image pickup element that images a subject under the control of the CPU 301 and obtains image data.

The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data. Further, a CCD sensor may be used instead of the CMOS sensor 305.

Further, the intermediary terminal 3 is an antenna 313a, a communication unit 313 that communicates with the image management system 7 via the communication network 100 by a wireless communication signal using the antenna 313a, a GPS (Global Positioning Systems) satellite or an indoor GPS. GPS receiver 314 that receives GPS signals including the position information (latitude, longitude, and altitude) of the intermediary terminal 3 by IMES (Indoor MEssaging System), and an address bus for electrically connecting each of the above parts. And a bus line 310 such as a data bus.

<Smartphone hardware configuration>
Next, the hardware of the smartphone will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of a smartphone. As shown in FIG. 11, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I / F 513a, an acceleration / orientation sensor 506, a media I / F 508, and a GPS receiver 509. There is.

Of these, the CPU 501 controls the operation of the entire smartphone 5. The CPU 501 may be single or plural. The ROM 502 stores a program used to drive the CPU 501 such as the CPU 501 and the IPL (Initial Program Loader). The RAM 503 is used as a work area of the CPU 501. The EEPROM 504 reads or writes various data such as a smartphone program under the control of the CPU 501. The CMOS sensor 505 captures a subject (mainly a self-portrait) according to the control of the CPU 501 to obtain image data. The image sensor I / F 513a is a circuit that controls the drive of the CMOS sensor 512. The acceleration / azimuth sensor 506 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect the geomagnetism. The media I / F 508 controls reading or writing (storage) of data to a recording medium 507 such as a flash memory. The GPS receiving unit 509 receives GPS signals from GPS satellites.

Further, the smartphone 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image pickup element I / F 513b, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, an external device connection I / F518, and a short distance. It includes a communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521.

Of these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network such as the Internet. The CMOS sensor 512 is a kind of built-in imaging means for capturing an image of a subject and obtaining image data under the control of the CPU 501. The image sensor I / F 513b is a circuit that controls the drive of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting voice. The sound input / output I / F 516 is a circuit that processes sound signal input / output between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal or an organic EL for displaying an image of a subject, various icons, and the like. The external device connection I / F518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, and Bluetooth. The touch panel 521 is a kind of input means for operating the smartphone 5 by the user pressing the display 517.

In addition, the smartphone 5 is provided with a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

<Hardware configuration of image management system>
Next, the hardware configuration of the image management system will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the image management system.

FIG. 12 is a hardware configuration diagram of the image management system 7. As shown in FIG. 12, the image management system 7 is constructed by a computer, and as shown in FIG. 12, the CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) controller 705, It includes a display 708, media I / F 707, network I / F 709, data bus 710, keyboard 711, mouse 712, and DVD-RW (Digital Versatile Disk Rewritable) drive 714.

Of these, the CPU 701 controls the operation of the entire image management system 7. The ROM 702 stores a program used to drive the CPU 701 such as an IPL. The RAM 703 is used as a work area for the CPU 701. The HD704 stores various data such as programs. The HDD controller 705 controls reading or writing of various data to the HD 704 according to the control of the CPU 701. The display 708 displays various information such as cursors, menus, windows, characters, or images. The media I / F 707 controls reading or writing (storage) of data to a recording medium 706 such as a flash memory. The network I / F709 is an interface for performing data communication using the communication network 100. The bus line 710 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 701 shown in FIG.

Further, the keyboard 711 is a kind of input means including a plurality of keys for inputting characters, numerical values, various instructions and the like. The mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 714 controls reading or writing of various data to the DVD-RW 713 as an example of the removable recording medium. In addition, it is not limited to DVD-RW, and may be DVD-R, Blu-ray Disc (Blu-ray Disc), or the like.

[Functional configuration of the embodiment]
Next, the functional configuration of the present embodiment will be described with reference to FIGS. 13 to 16. Since the main function of the mediation terminal 3 is a transmission / reception unit for mediating communication between the spherical imaging device 1 and the image management system 7, the description thereof will be omitted.

<Functional configuration of spherical imaging device>
FIG. 13 is a functional block diagram of the spherical imaging device of this embodiment. The all-sky imaging device 1 includes a transmission / reception unit 11, a partial image parameter creation unit 12, an imaging control unit 13, imaging units 14a and 14b, an image processing unit 15, a temporary storage unit 16, a low-definition changing unit 17, and a projection conversion unit. It has 18, a coupling unit 19, a moving image coding unit 20a, a still image coding unit 20b, a reception unit 22, a determination unit 25, and a still image storage unit 20. Each of these parts is realized by operating any of the components shown in FIG. 9 by a command from the CPU 111 according to a program for the spherical imaging device developed on the DRAM 113 from the SRAM 113. A function or means. Further, the spherical imaging device 1 has a threshold value management unit 1001 realized by the SRAM 113.

(Horizontal and vertical angle of view threshold information)
As shown in FIG. 16A, the threshold value management unit 1001 stores horizontal and vertical angle of view threshold value information, for example, before shipment from the factory. 16 (a) is a conceptual diagram of the horizontal and vertical angle of view threshold information, and FIG. 16 (b) is an explanatory diagram of the horizontal angle of view and the vertical angle of view. In FIG. 16A, the horizontal / vertical angle of view threshold information is represented in a table format, but the present invention is not limited to this, and the horizontal / vertical angle of view threshold information may not be represented in a table format.

As shown in FIG. 16A, the horizontal / vertical angle-of-view threshold information is determined by a row of candidates for the maximum shooting resolution (horizontal W × vertical H) of the all-sky imaging device 1 and a user (viewer). A row of candidates for the maximum display resolution (horizontal W'x vertical H') on the set smartphone 5, used to determine whether or not to transmit from the all-sky photography device 1 to the smartphone 5 via the image management system 7. Each threshold of the horizontal angle of view (AH) × the vertical angle of view (AV) of the image to be created is managed in association with each other.

Here, the horizontal angle of view (AH) and the vertical angle of view (AV) will be described with reference to FIG. 16 (b). In FIG. 16B, the center point CP, the azimuth angle “aa”, and the elevation angle “ea” are the same as those in FIG. 7, so the description thereof will be omitted. In the spherical image (moving image, still image), the size of the entire area of one frame of the rectangular partial image (moving image, still image) can be represented by the direction of the camera (here, the virtual camera IC) to shoot. is there. The horizontal length of the partial image can be represented by the horizontal angle of view (AH) from the virtual camera IC, and the vertical length of the partial image can be represented by the horizontal angle of view (AH) from the virtual camera IC.

The horizontal angle of view (AH) and the vertical angle of view (AV) are angles of view that eliminate the difference between the resolution of the conversion source and the resolution after conversion, and are used for the determination of the determination unit 25. This angle of view can be obtained by the following (Equation 1) and (Equation 2).

AH = (360 / W) * W'・ ・ ・ (Equation 1)
AV = (180 / H) * (H'/ 2) ・ ・ ・ (Equation 2)
Even if the maximum shooting resolution of the spherical imaging device 1 is (W: 4000, H: 2000), the spherical imaging device 1 has a horizontal and vertical angle of view showing each setting pattern shown in FIG. It manages threshold information. However, in this case, the spherical imaging device 1 refers only to the patterns of setting 1 and setting 2 in FIG. Similarly, even if the maximum display resolution on the smartphone 5 set by the user is (W': 1920, H': 1080), the spherical imaging device 1 sets each setting pattern shown in FIG. The horizontal and vertical angle of view threshold information shown is managed.

For example, in the case where the maximum shooting resolution of the spherical imager 1 is 4000 × 2000 in width × 2000, the width × length is 1920 × 1080 with respect to the spherical imager 1 from the smartphone 5 depending on the user. Under the condition (setting 1) when the data of the resolution image (moving image, still image) is set, the determination unit 25 determines that the angle of view of the partial image requested by the instruction data is horizontal 172. When it is less than the threshold value indicated by 8 ° and 48.6 ° in length, it is determined that the projection method conversion unit 18 does not perform projection method conversion to ultra-high-definition still image data.

(Each functional configuration of spherical imaging device)
Next, each functional configuration of the spherical imaging device 1 will be described in more detail with reference to FIG.

The transmission / reception unit 11 transmits image data to the outside of the spherical imaging device 1 and receives data from the outside. For example, the image data includes ultra-high-definition partial still images, high-definition partial moving images, and low-definition whole moving images. In addition, the transmission / reception unit 11 receives instruction data described later and transmits partial image parameters described later.

In the present embodiment, since an image having three levels of definition is handled, the image with the highest definition, the image with the next highest definition, and the image with the lowest definition are respectively "ultra-high definition" for convenience. , "High definition", "Low definition" to distinguish. Therefore, for example, the "ultra-high definition" of the present embodiment does not mean general ultra-high definition.

The partial image parameter creation unit 12 creates partial image parameters based on the instruction data sent from the smartphone 5 via the image management system 7. This instruction data is received by the operation of the user at the reception unit 52 of the smartphone 5, and is data indicating an instruction for specifying the superimposed area described later.

The photographing control unit 13 outputs an instruction for synchronizing the output timing of the image data of the imaging units 14a and 14b.

The image pickup units 14a and 14b each take an image of a subject or the like according to an instruction from the imaging control unit 13, and for example, as shown in FIGS. 3 (a) and 3 (b), based on the spherical image data. Outputs the hemispherical image data.

The image processing unit 15 synthesizes and converts the two hemispherical image data obtained by the photographing units 14a and 14b into the data of the equirectangular projection image which is the image of the equirectangular projection method.

The temporary storage unit 16 serves as a buffer for temporarily storing the data of the equirectangular projection image synthesized and converted by the image processing unit 15. The equirectangular projection image in this state is ultra-high definition.

The low-definition changing unit 17 reduces the equirectangular projection image from the ultra-high-definition image (here, moving image) to the low-definition image (here, here) by reducing the image of the equirectangular projection image according to the instruction data from the smartphone 5 received by the transmitting / receiving unit 11. , Video). As a result, a low-definition equirectangular projection image (here, the entire moving image) is generated. The low-definition changing unit 17 is an example of the changing means.

The projection method conversion unit 18 follows the instruction data and video data request received by the transmission / reception unit 11, that is, the direction, angle of view and aspect of the partial image which is a part of the entire image, and the smartphone via the image management system 7. According to the size of the image data to be transmitted to 5, the normal-distance cylindrical projection method is converted into the perspective projection method. When the partial moving image data is requested in this way, the projection method conversion unit 18 has a lower definition (or resolution) than the ultra-high-definition whole image stored in the temporary storage unit 16, but the low-definition changing unit A partial image (here, a partial moving image) in a high-definition state having a higher definition (or resolution) than the low-definition image obtained by being changed by 17 is generated.

Further, the projection conversion unit 18 sets the direction, angle of view and aspect of the partial image, which is a part of the entire image, and the image management system 7 according to the instruction data and still image data request received by the transmission / reception unit 11. According to the size of the image data transmitted to the smartphone 5 via the screen, the normal-distance cylindrical projection method can be converted into the perspective projection method. When the partial still image data is requested in this way, the projection method conversion unit 18 generates a partial image (here, a partial still image) in the ultra-high-definition state stored in the temporary storage unit 16. ..

From the above, the difference between the low-definition change unit 17 and the projection method change unit 18 is not only whether or not the projection method conversion is executed, but also the entire image output from the low-definition change unit 17 (here, the entire moving image). The data has a lower definition (or resolution) than the partial image (here, partial moving image and partial still image) data output from the projection conversion unit 18. That is, the partial image data output from the projection method conversion unit 18 has a higher definition (or resolution) than the overall image data output from the low-definition changing unit 17.

Here, as an ultra-high-definition image, for example, a case where a positive-distance cylindrical projected image having a resolution of 2K, 4K, or 8K is output will be described. Based on the data of these ultra-high-definition positive-distance cylindrical projection images, the "partial moving image" data output from the projection method conversion unit 18 has a resolution of 1.5K, 3K, or 6K. Further, the "partial still image" data output from the projection method conversion unit 18 based on the data of these ultra-high-definition regular-distance cylindrical projection images has a resolution of 2K, 4K, or 8K. Is. On the other hand, the "whole moving image" data output from the low-definition changing unit 17 has a resolution of 1K, 2K, or 4K.

FIG. 20 shows, in FIG. 20, each frame of the low-definition whole moving image changed by the low-definition changing unit 17 and each frame of the high-definition partial moving image converted by the projection method conversion unit 18. By compressing the vertical resolution in half, the definition of each frame is reduced and combined into one frame. As a result, the partial moving image has a higher definition than the whole moving image, but has a lower definition than the partial still image. By reducing the definition of each frame to one frame in this way, the amount of data communication can be suppressed. Furthermore, since the two frames to be combined are generated from the same azimuthal equidistant projected image stored in the temporary storage unit 16, it is not necessary to use metadata or the like for associating the two frames to be combined. Can also associate two frames shot and generated at the same timing.

The moving image coding unit 20a encodes the frame data of the entire moving image and the partial moving image combined by the joining unit 19. The still image coding unit 20b encodes the data of the partial still image.

The reception unit 22 receives an operation performed on the operation unit 115 of the spherical imaging device when the user makes various requests.

The determination unit 25 indicates that the entire area of one frame of the partial image, which is a part of the entire moving image, is a predetermined threshold value of the horizontal / vertical angle-of-view threshold information (see FIG. 16) managed by the threshold value management unit 1001. Judge whether it is smaller than the predetermined area. If it is small, the determination unit 25 prevents the projection method conversion unit 18 from generating ultra-high-definition partial still image data from the ultra-high-definition overall moving image data. If it is large, the determination unit 25 causes the projection method conversion unit 18 to generate ultra-high-definition partial still image data from the ultra-high-definition overall moving image data.

For example, in the case where the horizontal × vertical of the maximum shooting resolution of the spherical imaging device 1 is 4000 × 2000, depending on the user, the horizontal of the maximum display resolution of the smartphone 5 with respect to the spherical imaging device 1 from the smartphone 5. Under the condition (setting 1) when the vertical angle is set to 1920 × 1080, the horizontal angle of view of the image required by the instruction data is less than 172.8 ° and the vertical angle of view is 48.6 °. If it is less than, the determination unit 25 determines that the projection method conversion unit 18 does not perform the projection method conversion to the ultra-high-definition still image data. In this way, when the resolution (definition) of the image requested from the smartphone 5 is low, the celestial sphere photographing device 1 may send high-definition partial moving image data or ultra-high-definition partial stillness. Even if image data is sent, it is difficult for the user to distinguish between a high-definition partial moving image and an ultra-high-definition partial still image on the smartphone 5, so the all-sky imaging device 1 purposely uses the ultra-high-definition partial still image. This is because there is no need to send the data of. In addition, "less than" shown in the threshold range may be "less than or equal to".

On the other hand, under the same conditions as above (setting 1), when the horizontal angle of view of the image required by the instruction data exceeds 172.8 ° or the vertical angle of view exceeds 48.6 °, the judgment unit 25 determines that the projection method conversion unit 18 causes the projection method conversion to the ultra-high-definition still image data. In this case, the smartphone 5 switches the high-definition partial moving image to the ultra-high-definition still image and displays it, so that the user (viewer) can browse the area of interest more clearly (clearly). Therefore, it can be browsed carefully. In addition, "super" shown in the threshold range may be "greater than or equal to".

<Functional configuration of smartphone>
FIG. 14 is a functional block diagram of the smartphone of the present embodiment. The smartphone 5 includes a transmission / reception unit 51, a reception unit 52, a video decoding unit 53a, a still image coding unit 53b, a superimposition area creation unit 54, an image creation unit 55, an image superimposition unit 56, a projection conversion unit 57, and a display control unit. Has 58. Each of these parts is a function or means realized by operating any of the components shown in FIG. 11 by a command from the CPU 501 according to a smartphone program developed from the EEPROM 504 to the RAM 503. ..

(Each function configuration of smartphone)
Next, each functional configuration of the smartphone 5 will be described in more detail with reference to FIG.

The transmission / reception unit 51 transmits data to the outside of the smartphone 5 and receives data from the outside. For example, the transmission / reception unit 51 receives image data from the transmission / reception unit 11 of the spherical imaging device 1 or transmits instruction data to the transmission / reception unit 11 of the spherical imaging device 1. Further, the transmission / reception unit 51 separates the image data (total moving image data and partial moving image data shown in FIG. 20) sent from the transmitting / receiving unit 11 of the spherical imaging device 1 and the partial image parameters.

The reception unit 52 receives from the user an operation of specifying the direction, angle of view and aspect of the partial image, and the size of the image data received by the smartphone 5. In addition, the reception unit 52 receives from the user the setting of the maximum display resolution (see horizontal W'x vertical H'in FIG. 16) on the smartphone 5.

The moving image decoding unit 53a decodes each data of the low-definition whole moving image and the high-definition partial moving image encoded by the moving image coding unit 20a. The still image decoding unit 53b decodes the data of the ultra-high-definition partial still image encoded by the still image coding unit 20b.

The superimposition area creation unit 54 creates a superimposition area designated by the partial image parameters. This superimposing region shows the superimposition position and superimposition range of the superimposition image S and the mask image M which are partial moving images (or partial still images) on the spherical image CE which is the whole moving image.

The image creation unit 55 creates a superimposed image S and a mask image according to the superimposed region, and creates a spherical image CE from a low-definition overall image.

The image superimposing unit 56 creates a final spherical image CE by superimposing the superposed image S and the mask image M on the superposed region on the spherical image CE.

The projective conversion unit 57 converts the final spherical image CE into a perspective projection type image according to the user's instruction received by the reception unit 52.

The display control unit 58 controls the display 517 or the like to display the image after conversion to the perspective projection method.

<Functional configuration of image management system>
FIG. 15 is a functional block diagram of the image management system. The image management system 7 has a transmission / reception unit 71. The transmission / reception unit 71 is a function or means realized by operating the network I / F shown in FIG. 12 by a command from the CPU 701 according to a program for an image management system developed on the RAM 703 from the HD 704. is there. Further, the image management system 7 has a storage unit 7000 realized by the HD704.

(Functional configuration of image management system)
Next, the functional configuration of the image management system 7 will be described in more detail with reference to FIG.

The transmission / reception unit 71 transmits data to the outside of the image management system 7 and receives data from the outside. For example, the transmission / reception unit 71 receives image data from the transmission / reception unit 11 of the spherical imaging device 1 via the mediation terminal 3, or the transmission / reception unit 11 of the spherical imaging device 1 via the mediation terminal 3. Send instruction data. Further, the transmission / reception unit 71 inputs image data (total moving image data and partial moving image data shown in FIG. 20) and partial image parameters sent from the transmitting / receiving unit 11 of the spherical imaging device 1 via the intermediary terminal 3. Send to smartphone 5. Further, the ultra-high-definition still image data sent from the transmission / reception unit 11 of the all-sky imaging device 1 via the intermediary terminal 3 can be temporarily stored in the storage unit 7000, or ultra-high from the storage unit 7000. The data of a fine still image is read out and transmitted to the smartphone 5.

[Processing or operation of the embodiment]
Subsequently, the process or operation of the present embodiment will be described with reference to FIGS. 17 to 26. In the present embodiment, while the entire moving image and the partial moving image related to the data generated by the spherical imaging device 1 and then sent are being played back on the smartphone 5, one scene of the partial moving image having a viewer is displayed in higher definition ( In other words, the case of carefully viewing a still image (which is ultra-high definition) will be described.

<< Generation and playback of whole video and partial video >>
First, the process of generating and reproducing the data of the whole moving image and the partial moving image will be described with reference to FIG. FIG. 17 is a sequence diagram showing a process of generating and reproducing each data of the whole moving image and the partial moving image.

As shown in FIG. 17, when the viewer (user) operates the smartphone 5, the reception unit 52 receives a request to start the video distribution service (S11). As a result, the transmission / reception unit 51 of the smartphone 5 next transmits a request for moving image data to the transmission / reception unit 71 of the image management system 7 (S12). At this time, the partial image parameters specified by the viewer are also transmitted. Regarding the timing of starting transmission, the transmission / reception unit 11 of the spherical imaging device 1 may control the band of the communication network. By performing this band control, it becomes possible to transmit and receive data more stably.

Next, the transmission / reception unit 71 of the image management system 7 transfers the request for moving image data to the transmission / reception unit of the intermediary terminal 3 (S13). Then, the transmission / reception unit of the intermediary terminal 3 transfers the moving image data request to the transmission / reception unit 11 of the spherical imaging device 1 (S14).

Next, the spherical imaging device 1 performs a moving image data generation process (S15). The process of step S15 will be described in detail later (see FIGS. 18 to 20).

Next, the transmission / reception unit 11 of the spherical imaging device 1 transmits moving image data in response to the request to the transmission / reception unit of the intermediary terminal 31 (S16). This moving image data includes each data of a low-definition whole moving image and a high-definition partial moving image. As a result, the transmission / reception unit of the brokerage terminal 3 transfers the moving image data to the transmission / reception unit 71 of the image management system 7 (S17). Further, the transmission / reception unit 71 of the image management system 7 transfers the moving image data to the transmission / reception unit 51 of the smartphone 5 (S18).

Next, the smartphone 5 performs a video data reproduction process (S19). The process of step S19 will be described in detail later (see FIGS. 21 to 25).

<Video generation processing of spherical imaging device>
Next, with reference to FIGS. 18 to 20, the process of generating moving image data of the spherical imaging apparatus shown in step S15 will be described in detail. FIG. 18 is a conceptual diagram of an image in the process of image processing performed by the spherical imaging apparatus.

The image processing unit 15 synthesizes and converts the two hemispherical image data obtained by the photographing units 14a and 14b into equirectangular projection image data which is an equirectangular projection image (here, moving image) (S120). .. The converted data is temporarily stored in the temporary storage unit 16 in the state of the ultra-high-definition image.

Next, the partial image parameter creation unit 12 creates a partial image parameter based on the instruction data sent from the smartphone 5 (S130).

Next, the low-definition changing unit 17 changes from the ultra-high-definition image to the low-definition image according to the instruction data from the smartphone 5 received by the transmitting / receiving unit 11 (S140). As a result, a low-definition equirectangular projection image (here, the entire moving image) is generated.

Further, the projection conversion unit 18 follows the instruction data received by the transmission / reception unit 11, that is, the direction of the partial moving image which is a part of each frame of the whole moving image, the angle of view and aspect of the frame of the partial moving image, and the smartphone. According to the size of the moving image data to be transmitted to No. 5, the equirectangular projection method is converted into the perspective projection method (S150). As a result, a partial moving image in a high-definition state is generated.

Next, the joining unit 19 combines the data of the low-definition whole moving image and the high-definition partial moving image (S160). The processing of this binding will be described in detail later (see FIG. 20).

Here, the process shown in FIG. 18 will be described in more detail with reference to FIGS. 19 and 20. FIG. 19 is a diagram illustrating partial image parameters.

(Partial image parameters)
First, the partial image parameters will be described in detail with reference to FIG. FIG. 19A shows the entire image after image composition by S120. FIG. 19B is a diagram showing an example of partial image parameters. FIG. 19C shows a partial image after the projection method has been converted by S150.

The azimuth angle described in FIG. 7 is in the horizontal direction (latitude λ) in the equirectangular projection shown in FIG. 19A, and the elevation angle described in FIG. 7 is in the vertical direction (longitude φ) in the equirectangular projection method. The parameters shown in FIG. 19B, including the diagonal angle of view and the aspect ratio of FIG. 7, are partial image parameters. FIG. 19C is an example of a partial image obtained by cutting out a portion surrounded by a frame in the equirectangular projection of FIG. 19A with partial image parameters.

Here, the conversion of the projection method will be described. As shown in FIG. 4A, a spherical image is created by covering a three-dimensional sphere with an equirectangular projection image. Therefore, each pixel data of the equirectangular projection image can correspond to each pixel data on the surface of the three-dimensional sphere of the three-dimensional all-sky image. Therefore, in the conversion formula by the projection method conversion unit 18, the coordinates in the equirectangular projection image are expressed as (latitude, longitude) = (e, a), and the coordinates on the three-dimensional solid sphere are orthogonal coordinates (x, y). , Z) can be expressed by the following (Equation 3).
(x, y, z) = (cos (ea) × cos (aa), cos (ea) × sin (aa), sin (ea)) ・ ・ ・ (Equation 3)
However, the radius of the three-dimensional sphere at this time is 1.

On the other hand, the partial image which is a perspective projection image is a two-dimensional image, but when this is expressed by two-dimensional polar coordinates (movement diameter, argument) = (r, a), the movement diameter r becomes a diagonal angle of view. Correspondingly, the possible range is 0 ≤ r ≤ tan (diagonal angle of view / 2). Further, when the partial image is represented by the two-dimensional Cartesian coordinate system (u, v), the conversion relationship with the polar coordinates (diameter, argument) = (r, a) can be represented by the following (Equation 4). it can.
u = r × cos (a), v = r × sin (a) ・ ・ ・ (Equation 4)
Next, consider making this (Equation 3) correspond to three-dimensional coordinates (driving diameter, polar angle, azimuth). Since we are only considering the surface of the three-dimensional sphere CS, the moving diameter in three-dimensional polar coordinates is "1". Further, considering that the virtual camera is located at the center of the three-dimensional sphere, the projection that transforms the equirectangular projection image attached to the surface of the three-dimensional sphere CS is the above-mentioned two-dimensional polar coordinates (diameter, argument) = ( Using r and a), it can be expressed by the following (Equation 5) and (Equation 6).
r = tan (extreme angle) ・ ・ ・ (Equation 5)
a = Azimuth ・ ・ ・ (Equation 6)
If the polar angle is t, then t = arctan (r), so the three-dimensional polar coordinates (diameter, polar angle, azimuth) are (diameter, polar angle, azimuth) = (1, arctan (1, arctan ( It can be expressed as r) and a).

The conversion formula for converting from three-dimensional polar coordinates to the orthogonal coordinate system (x, y, z) can be expressed by the following (Equation 7).
(x, y, z) = (sin (t) × cos (a), sin (t) × sin (a), cos (t)) ・ ・ ・ (Equation 7)
According to the above (Equation 7), mutual conversion between the entire image by the equirectangular projection method and the partial image by the fluoroscopic projection method has become possible. That is, by using the moving diameter r corresponding to the diagonal angle of view of the partial image to be created, it is possible to calculate the conversion map coordinates representing which coordinates of the equirectangular projection image each pixel of the partial image corresponds to. , A partial image, which is a perspective projection image, can be created from the equirectangular projection image based on the transformation map coordinates.

By the way, the conversion of the projection method shows a conversion in which the position where the (latitude, longitude) of the equirectangular projection image is (90 °, 0 °) becomes the center point of the partial image which is the perspective projection image. ing. Therefore, when performing perspective projection conversion with an arbitrary point of the equirectangular projection image as the gazing point, the coordinates (latitude, longitude) of the gazing point can be changed by rotating the stereosphere to which the equirectangular projection image is pasted. Coordinate rotation may be performed so that the coordinates are arranged at the position (90 °, 0 °).

Since the transformation formula for the rotation of the solid sphere is a general coordinate rotation formula, the description thereof will be omitted.

(Processing of join)
Next, the coupling process of step S160 performed by the coupling unit 19 will be described with reference to FIG. FIG. 20 is a conceptual diagram showing a state in which each frame image of the entire moving image and the partial moving image is combined. As shown in FIG. 20, the whole image is arranged at the upper part and the partial image is arranged at the lower part of one frame. In the present embodiment, the aspect ratio is generally arranged according to 16: 9, which is an aspect ratio such as HD, but the aspect ratio does not matter. Further, the arrangement is not limited to the top and bottom, but may be left and right. When there are a plurality of partial images, for example, the entire moving image may be arranged in the upper half, and the lower half may be divided and arranged by the number of partial moving images. By combining the entire moving image and the partial image into one for each frame, synchronization between the images can be guaranteed. Further, the spherical imaging device 1 may separately transmit the whole moving image data and the partial moving image data to the smartphone 5.

However, since the entire video and the partial video are halved in height, the resolution is reduced to half as compared with the case where they are not halved. That is, the definition is lower than that of the ultra-high-definition image stored in the temporary storage unit 16. As a result, the fineness is, in descending order, the entire image stored in the temporary storage unit 16, the partial image after being converted by the projection method conversion unit 18, and the entire image after being converted by the low-definition changing unit 17. Become.

<Video playback processing for smartphones>
Subsequently, the reproduction process of the moving image data of the smartphone 5 will be described with reference to FIG. FIG. 21 is a conceptual diagram of an image in the process of image processing performed by the smartphone. The superimposing region creating unit 54 shown in FIG. 7 creates a partial three-dimensional sphere PS instructed by the partial image parameters as shown in FIG. 21 (S320).

Next, the image creation unit 55 creates a superposed image S by superimposing a partial image of the perspective projection method on the partial three-dimensional sphere PS (S330). Further, the image creation unit 55 creates a mask image M based on the partial three-dimensional sphere PS (S340). Further, the image creation unit 55 creates a spherical image CE by pasting an entire image of the equirectangular projection method on the stereoscopic sphere CS (S350). Then, the image superimposing unit 56 superimposes the superposed image S and the mask image M on the spherical image CE (S360). As a result, a low-definition spherical image (here, a low-definition whole moving image) CE on which a high-definition superimposed image (here, a high-definition partial moving image) S is superimposed so that the boundary is not conspicuous is completed. ..

Next, the projective conversion unit 57 enables the display 517 to view a predetermined area in the spherical image CE in which the superimposed image S is superimposed based on a predetermined line-of-sight direction and angle of view of the virtual camera. The projective transformation is performed (S370).

FIG. 22 is a diagram illustrating the creation of a partial three-dimensional sphere from a partial plane. Normally, in the perspective projection method, the image is projected onto a plane, so that it is often represented by a plane on a three-dimensional space as shown in FIG. 22 (a). In the present embodiment, as shown in FIG. 22B, a partial three-dimensional sphere that is a part of the sphere is used according to the spherical image. Here, the conversion from a plane to a partial solid sphere will be described.

Consider the projection of each point (X, Y, Z) on a plane arranged with an appropriate size (angle of view) onto a spherical surface as shown in FIG. 22 (a). The projection onto the sphere is the intersection of a straight line passing through each point (X, Y, Z) from the origin of the sphere and the sphere. Each point on the sphere is a point whose distance from the origin is equal to the radius of the sphere. Therefore, assuming that the radius of the sphere is 1, the points (X', Y', Z') on the sphere shown in FIG. 22 (b) are represented by the following (Equation 8).

(X', Y', Z') = (X, Y, Z) x 1 / √ (X ² + Y ² + Z ² ) ... (Equation 8)
FIG. 23 is a two-dimensional conceptual diagram in the case where the partial image is superimposed on the spherical image without creating the partial three-dimensional sphere of the present embodiment. FIG. 24 is a two-dimensional conceptual diagram in the case where the partial three-dimensional sphere of the present embodiment is created and the partial image is superimposed on the spherical image.

As shown in FIG. 23A, the subject P1 is represented as an image P2 on the spherical image CE, based on the case where the virtual camera IC is located at the center point of the stereoscopic sphere CS. , It is represented as an image P3 on the superimposed image S. As shown in FIG. 23A, since the images P2 and P3 are located on a straight line connecting the virtual camera IC and the subject P1, the superimposed image S is superimposed on the spherical image CE. Even if the image is displayed in the above state, the spherical image CE and the superimposed image S do not deviate from each other. However, as shown in FIG. 23B, when the virtual camera IC is separated from the center point of the three-dimensional sphere CS (when the angle of view α is reduced), the virtual camera IC and the subject P1 are drawn on a straight line. The image P2 is located, but the image P3 is located slightly inward. Therefore, assuming that the image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is the image P3', the spherical image CE and the superimposed image S are displaced between the image P3 and the image P3'. A deviation of the amount g will occur. As a result, the superimposed image S is displayed with a deviation from the spherical image CE, but the superimposed display may be used in this way.

On the other hand, in the present embodiment, since the partial three-dimensional sphere is created, the superimposed image S is aligned with the spherical image CE as shown in FIGS. 24 (a) and 24 (b). Can be superimposed. As a result, not only when the virtual camera IC is located at the center point of the three-dimensional sphere CS as shown in FIG. 24 (a), but also as shown in FIG. 24 (b), the virtual camera can be moved. Even when the image P2 and the image P3 are separated from the center point of the three-dimensional sphere CS, the image P2 and the image P3 are located on a straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superposed image S is displayed in a superposed state on the spherical image CE, there is no discrepancy between the spherical image CE and the superposed image S.

FIG. 25 (a) is a display example of a wide image when the superimposed display is not performed, FIG. 25 (b) is an example of displaying a tele image when the superimposed display is not performed, and FIG. 25 (c) is an example of displaying a wide image when the superimposed display is not performed. , FIG. 25D is a conceptual diagram showing a display example of a teleimage in the case of superimposed display. The wavy lines in the figure are shown only for convenience of explanation, and may or may not be actually displayed on the display 517.

As shown in FIG. 25 (a), the partial image (here, high-definition partial moving image) P is not superimposed and displayed on the spherical image (here, low-definition whole moving image) CE. In this case, when the image is enlarged to the area shown by the wavy line in FIG. 25 (a), the low-definition image remains as shown in FIG. 25 (b), and the user sees an unclear image. It will end up being. On the other hand, as shown in FIG. 25 (c), when the partial image P is superimposed and displayed on the spherical image CE, it is enlarged to the area shown by the wavy line in FIG. 25 (c). When displayed, as shown in FIG. 25 (d), a high-definition image is displayed, and the user can see a clear image. In particular, when a signboard or the like on which characters are drawn is displayed in the area indicated by the wavy line, the characters will be blurred even if the high-definition partial image P is not superimposed and displayed. I don't know if is written. However, if the high-definition partial image P is superimposed and displayed, the characters can be clearly seen even when enlarged and displayed, so that the user can grasp what is written.

<< Generation and playback of partial still images >>
Subsequently, the process of generating and reproducing the data of the partial still image will be described with reference to FIG. FIG. 26 is a sequence diagram showing a process of generating and reproducing data of a partial still image.

The entire image (here, a high-definition partial moving image) P on which the viewer (user) superimposes a partial image (here, a high-definition partial moving image) P as shown in FIG. 25 (c) by step S19 shown in FIG. Here, when the partial moving image is to be viewed as a higher-definition still image while the low-definition whole moving image is being viewed, the viewer (user) operates the smartphone 5 as shown in FIG. By doing so, the reception unit 52 receives the request to start the distribution service of the partial still image (S31). As a result, the transmission / reception unit 51 of the smartphone 5 next transmits a request for still image data to the transmission / reception unit 71 of the image management system 7 (S32). At this time, the partial image parameters specified by the viewer are also transmitted.

Next, the transmission / reception unit 71 of the image management system 7 transfers the request for still image data to the transmission / reception unit of the intermediary terminal 3 (S33). Then, the transmission / reception unit of the intermediary terminal 3 transfers the still image data request to the transmission / reception unit 11 of the spherical imaging device 1 (S34).

Next, the spherical imaging device 1 performs a still image data generation process (S35). The process of this step S35 will be described in detail later.

Next, the transmission / reception unit 11 of the spherical imaging device 1 transmits still image data in response to the request to the transmission / reception unit of the intermediary terminal 31 (S36). This still image data is ultra-high-definition partial still image data. The timing for starting the transmission of the still image data may be a time zone when the amount of network traffic is low such as at night, or a delay time may be measured and transmitted when the delay time is shorter than a predetermined delay time. As a result, the transmission / reception unit of the intermediary terminal 3 transfers the still image data to the transmission / reception unit 71 of the image management system 7 (S37).

Next, when the transmission / reception unit 71 of the image management system 7 receives the still image data, the storage unit 7000 stores the still image data (S38).

Then, the transmission / reception unit 7 of the image management system 7 transfers the still image data to the transmission / reception unit 51 of the smartphone 5 (S39). At this time, regarding the timing of starting the transfer, the band of the communication network 100 is controlled in the same manner as the data transmission from the celestial sphere photographing apparatus 1 to the image management system via the intermediary terminal 3.

Next, the smartphone 5 superimposes and displays an ultra-high-definition partial still image instead of the high-definition partial moving image previously superimposed on the entire moving image (S40). This method of superimposition includes a process common to the case of superimposing a partial moving image on the whole moving image, but since there is a different processing, it will be described in detail later.

If, in step S35, the determination unit 25 determines that the projection method conversion unit 18 does not perform projection conversion to ultra-high-definition still image data, the determination unit 25 replaces the still image data in steps S36 to S37. You may also send a message such as "Do not send a still image" or "Insufficient angle of view". In this case, in step S40, the above message is displayed with the partial moving image still superimposed on the whole moving image.

<Generation of still images of spherical imaging device>
Next, the process of generating still image data of the spherical imaging device shown in step S35 described above will be described in detail.

When the spherical imaging device 1 receives the request for still image data, the still image imaging determination is performed. This is because when the angle of view of the partial image is relatively narrow, the resolution of the partial still image created by the conversion from the original image does not differ from the resolution of the partial moving image being transmitted. In this case, the horizontal angle of view (AH) and the vertical angle of view (AV) of the horizontal / vertical angle of view threshold information shown in FIG. 16 are set as the minimum angle of view, and the angle of view specified by the partial image parameter is the angle of view. If it is wider, the still image generation process is executed (S210).

More specifically, when the transmission / reception unit 11 of the spherical imaging device 1 receives a request for still image data during transmission of moving image data, the projection method conversion unit 18 similarly generates partial moving image data. , The equirectangular projection method is converted to the perspective projection method according to the partial image parameters for the ultra-high-definition partial still image data. In this case, the size of the generated still image is variable, and the conversion is performed while maintaining the resolution of the ultra-high-definition image. Assuming that the horizontal resolution of the ultra-high-definition image is W and the specified horizontal angle of view is ah, the horizontal resolution Wp of the generated still image can be obtained as follows.

Wp = W / 360 * ah ・ ・ ・ (Equation 9)
The size of W here is acquired from the W column of the horizontal / vertical angle-of-view threshold information (see FIG. 16A) read when the spherical imaging device 1 is started.

Similarly, assuming that the specified vertical angle of view is av, the vertical resolution Hp of the generated still image can be obtained by the following equation.

Hp = H / 180 * av ・ ・ ・ (Equation 10)
Further, the still image coding unit 20b encodes the data of the partial still image and stores it in the image buffer. Then, the transmission / reception unit 11 controls the communication network band described above in the same manner as in steps S16 and S17, and transmits the partial still image data to the image management system 7 via the intermediary terminal 3. In this case as well, the partial image parameters are transmitted.

<Playback of still images on smartphones>
Next, the reproduction process of the still image data of the smartphone 5 shown in step S40 described above will be described in detail.

On the smartphone 5 side, the transmission / reception unit 51 separates the still image data sent from the spherical imaging device 1 into the partial still image data received by the communication unit and the partial image parameters. The still image decoding unit 53b decodes the data of the partial still image after the separation. After that, since the above-mentioned processing for each data of the whole moving image and the partial moving image is only replaced with the processing for each data of the whole moving image and the partial still image, the description thereof will be omitted.

[Main effects of the embodiment]
As described above, according to the present embodiment, the area in which the viewer pays more clear attention by suppressing the data communication volume of the entire moving image and the partial moving image while keeping the fineness of the partial image as low as possible. It has the effect of being able to browse.

Further, the spherical image capturing device 1 creates a low-definition whole image (here, a low-definition whole moving image) from the ultra-high-definition spherical image (S140), and the same ultra-high-definition spherical image. A high-definition partial image (here, a high-definition partial moving image) having a different projection method is created from (S150). Then, the spherical imaging device 1 transmits each data of the low-definition whole image and the high-definition partial image to the smartphone 5. On the other hand, the smartphone 5 superimposes a high-definition partial image on the low-definition whole image (S360) and converts the projection method according to the line-of-sight direction and the angle of view specified by the user (viewer). (S370). In this way, the spherical imaging device 1 transmits the partial image, which is a region of interest, in the ultra-high-definition spherical image obtained by photographing the subject or the like in a high-definition manner. However, the whole image (for grasping the whole spherical image) that has not been noticed is transmitted in a low definition, and the high definition partial image is transmitted after the projection method is converted. Further, prior to this transmission, the coupling portion 19 compresses the vertical resolutions of the low-definition whole moving image data and the high-definition partial moving image data in half, as shown in FIG. Reduce the resolution of and combine into one frame. As a result, the smartphone 5 on the receiving side can reduce the amount of data as compared with the conventional case, so that it is possible to quickly display the spherical image in the state where the partial image is superimposed on the whole image.

Therefore, according to the present embodiment, in order to reduce the amount of communication, the whole celestial sphere photographing apparatus 1 reduces the entire celestial sphere image to a low-definition whole image and attention in the whole celestial sphere image (whole image). Not only is the data of the high-definition partial image, which is the area to be processed, distributed, and the partial image is combined with the entire image on the receiving smartphone 5, but also the low-definition overall image and the high-definition partial image are projected differently. Even if there is, since it can be combined and displayed on the smartphone 5, it has the effect of being highly versatile in terms of the projection method.

[Supplement]
In the above embodiment, the determination unit 25 determines the horizontal / vertical angle of view threshold information (see FIG. 16) in which the entire area of one frame of the partial image, which is a part of the entire moving image, is managed by the threshold management unit 1001. It is determined whether or not it is smaller than the predetermined region indicated by the predetermined threshold value. If it is small, the determination unit 25 prevents the projection method conversion unit 18 from converting the ultra-high-definition still image data into a projection method, and if it is large, the determination unit 25 determines the projection method conversion unit. 18 is converted into ultra-high-definition still image data by a projection method, but the present invention is not limited to this. For example, if it is small, the transmission / reception unit 11 performs a process that prevents the data of the partial still image after the projection method conversion stored in the still image storage unit 29 from being transmitted to the smartphone 5, and if it is large, the transmission / reception unit 11 May perform a process of transmitting the data of the partial still image after the projection method conversion stored in the still image storage unit 29 to the smartphone 5. In this case, the determination unit 25 may instruct whether or not to transmit to the transmission / reception unit 11, and the determination unit 25 tells the still image storage unit 29 after the projection method conversion stored in the storage unit 29. You may instruct whether or not to transmit the data of the partial still image to the transmission / reception unit 11.

Further, the spherical photographing device 1 is an example of a photographing device, and the photographing device includes a digital camera, a smartphone, and the like for obtaining a normal plane image. In the case of a digital camera or smartphone that obtains a normal plane image, it is possible to obtain a relatively wide-angle image (wide-angle image) instead of obtaining a spherical image.

Further, the smartphone 5 is an example of a communication terminal or an image processing device, and the communication terminal or the image processing device also includes a PC such as a tablet PC (Personal Computer), a notebook PC, and a desktop PC. Further, the communication terminal or the image processing device includes a smart watch, a game device, a car navigation terminal mounted on a vehicle or the like, and the like.

In the above embodiment, as a low-definition image, an entire moving image which is an entire area of the image of the image data obtained from the imaging units 14a and 14b, and as a high-definition image, a partial moving image which is a part of the entire area will be described. However, it is not limited to this. The low-definition image may be an image of a part of the entire region A1 of the image of the moving image data obtained from the imaging units 14a and 14b. In this case, the high-definition image becomes an image of a part of the area A1 and a part of the area A2. That is, the relationship between the low-definition whole moving image and the high-definition partial moving image is that the former is a wide-angle moving image while the latter is a narrow-angle moving image. The partial still image can also be called a narrow-angle still image in the same way as the partial moving image. Further, in the present embodiment, the wide-angle image (including both a still image and a moving image) is an image in which distortion occurs in an image taken by an imaging device including a so-called wide-angle lens or a fisheye lens. A narrow-angle image (including both still images and moving images) is a part of an image taken by an image pickup device equipped with a wide-angle lens or a fisheye lens, and has a narrower angle of view than a wide-angle image.

In the above embodiment, the case where the planar image is superimposed on the spherical image has been described, but the superimposition is an example of synthesis. In addition to superposition, synthesis also includes pasting, fitting, superposition, and the like. The superimposed image is an example of a composite image. In addition to the superimposed image, the composite image also includes a pasted image, an inset image, a superposed image, and the like. Further, the image superimposition unit 56 is an example of an image composition unit.

Further, the equirectangular projection image EC and the plane image P may be either a still image, both a moving image frame, or one still image and the other moving image frame.

Further, in the above embodiment, the projection method conversion unit 18 converts the high-definition image data acquired from the temporary storage unit 16 with the same definition, but the present invention is not limited to this. For example, if the definition is higher than the overall image data output from the low-definition changing unit 17, the projection method conversion unit 18 determines the definition of the image data acquired from the temporary storage unit 16 when converting the projection method. It may be lowered.

Each functional configuration shown in FIGS. 13 and 14 is realized in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. In this case, in the technical plan of the present embodiment, the essential or part that contributes to the prior art or the part of the above technical plan is expressed in the form of a software product. The computer software product is stored in a storage medium and gives a plurality of instructions to a computer device (personal computer, server, network device, etc.) to perform all or part of the steps of the above method according to each of the above embodiments. Including. The above-mentioned storage medium includes various media such as a USB memory, a removable disk, a ROM, RAM, a magnetic disk, or an optical disk, which can store a program code.

Also, the method according to the above embodiment is applied to or realized by a processor. A processor is an integrated circuit board capable of processing signals. Each step of the method of each of the above embodiments is realized by an integrated logic circuit or software-type directive that is hardware in the processor. The processors are general purpose processors, digital signal processors (DSPs), dedicated integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and each of the above. Each method, step and logical box disclosed in the embodiment is feasible or feasible. The general-purpose processor may be a microprocessor or any general processor. Each step of the method according to each of the above embodiments may be realized by being executed by a decoder which is hardware, or may be realized by a combination of hardware and software in the decoder. Software modules are stored in storage media mature in the art, such as random memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory and registers. The processor reads the information from the memory including the storage medium in which the software is stored, and implements the steps of the above method according to the hardware.

The embodiments described above are realized by hardware, software, firmware, middleware, microcode, or a combination thereof. Among them, regarding the realization of hardware, the processing unit is one or more dedicated integrated circuits (ASIC), digital signal processing processor (DSP), digital signal processor (DSPD), programmable logic circuit (PLD), field programmable. It is implemented by a gate array (FPGA), an application specific integrated circuit, a controller, a microcontroller, a microprocessor, and other electronic units or combinations thereof that perform the functions of the present invention. Regarding the realization of software, the above technology is realized by modules (for example, processes, functions, etc.) that realize the above-mentioned functions. The software code is stored in memory and executed by the processor. The memory is realized inside or outside the processor.

1 Spherical imaging device (an example of imaging device)
5 Smartphones (an example of a communication terminal, an example of an image processing device)
13 Image control unit 14a Image pickup unit 14b Image pickup unit 15 Image processing unit 16 Temporary storage unit 17 Low-definition change unit (example of change means)
18 Projection method conversion unit (an example of projection method conversion means)
19 Bonding part (example of joining means)
20a Video coding unit 20b Still image coding unit 22 Reception unit 51 Transmission / reception unit 52 Reception unit 53a Video decoding unit 53b Still image numbering unit 54 Superimposition area creation unit 55 Image creation unit 56 Image superimposition unit 57 Projection conversion unit 58 Display Control unit

Japanese Unexamined Patent Publication No. 2006-3400091

Claims (10)

A shooting device that shoots a subject and obtains moving image data of a predetermined definition.
A changing means for changing a wide-angle moving image, which is all or a part of the moving image related to the moving image data of the predetermined definition, to a low definition.
A projection conversion means for converting a narrow-angle moving image, which is a part of the wide-angle moving image, into a higher-definition state than the wide-angle moving image after being changed to a low definition by the changing means.
A coupling means for lowering the definition of each of the low-definition wide-angle moving image frame and the high-definition narrow-angle moving image frame to combine them into one frame.
Have,
The imaging method conversion means is characterized in that the narrow-angle still image, which is a part of the region of the wide-angle still image as a frame of the wide-angle moving image before being changed to low definition by the changing means, is converted by the projection method. apparatus. The imaging device according to claim 1.
A transmission means for transmitting the frame data of the wide-angle moving image and the frame data of the narrow-angle moving image combined into one frame by the combining means so that the image management system that manages the image data receives the frame data.
A receiving means for receiving a request for a still image transmitted by a communication terminal, and
Have,
Based on the reception of the request for the still image by the receiving means, the transmitting means transmits the data of the narrow-angle still image after conversion by the projection conversion means so that the image management system receives the data. Shooting equipment to do. When the entire area of one frame of the narrow-angle moving image is smaller than a predetermined area, the projection method conversion means does not convert the narrow-angle still image into the projection method, or the transmission means converts the projection method. A photographing device characterized in that the data of the narrow-angle still image after conversion by means is not transmitted. The shooting according to any one of claims 1 to 3, wherein the wide-angle moving image is composed of the entire area of a frame of moving image data having a predetermined definition obtained by shooting the subject. apparatus. The photographing apparatus according to any one of claims 1 to 4, wherein the projection method conversion means converts the narrow-angle still image into the projection method while maintaining the predetermined definition. The photographing device according to any one of claims 1 to 5, wherein the photographing device is a spherical photographing device that photographs a subject and obtains spherical image data. The imaging device according to any one of claims 1 to 6.
With the image management system
A photography system characterized by having. The imaging device according to any one of claims 1 to 6.
With the image management system
The wide-angle moving image data and the narrow-angle moving image data are received from the image management system, the narrow-angle moving image is synthesized and displayed in a part of the area on the wide-angle moving image, and the narrow-angle moving image is displayed from the image management system. With a communication terminal that receives the data of the narrow-angle still image, synthesizes the narrow-angle image with the part of the region on the wide-angle image, and displays the narrow-angle still image in place of the narrow-angle moving image. ,
A photography system characterized by having. It is an image processing method that shoots a subject and processes moving image data of a predetermined definition.
A change step of changing a wide-angle moving image, which is all or a part of the moving image related to the moving image data of the predetermined definition, to a low definition, and
A projection conversion step that converts a narrow-angle moving image, which is a part of the wide-angle moving image, into a higher-definition state than the wide-angle moving image after being changed to low definition by the change step,
A combination step of lowering the definition of each of the low-definition wide-angle moving image frame and the high-definition narrow-angle moving image frame to combine them into one frame.
And
The projection method conversion step includes a process of converting a narrow-angle still image, which is a part of the wide-angle still image as a frame of a wide-angle moving image before being changed to low definition by the change step, into the projection method. Image processing method. A program comprising causing a computer to perform the method of claim 10.