JP7484237B2 - Photographing device, photographing system, image processing method, and program - Google Patents

Description

The present invention relates to an imaging device, an imaging system, an image processing method, and a program.

Instead of using a camera system consisting of a pan/tilt and zoom lens with a rotating platform, a technology is already known that electronically pans, tilts and zooms the image data obtained by photographing a subject using a fisheye lens or the like, and distributes the image data, and generates and displays a partial image on the receiving viewer.

However, with technology that distributes wide-angle image data obtained by photographing a subject, there is a problem in that the amount of data traffic increases because image data of areas that are not being focused on is also distributed.

In response to this, a technology has been disclosed that reduces the amount of communication traffic by having the imaging device distribute data on a low-resolution overall image that is a reduced version of the entire wide-angle image, and a high-resolution partial image that is a region of interest within the wide-angle image, and the partial image is then fitted into the overall image on the receiving side (see Patent Document 1).

However, when the entire image and partial images are videos, it is difficult to reduce the amount of data traffic even if the resolution of the entire image is reduced. On the other hand, viewers do not always watch videos carefully, and there is a need to be able to view at least a partial image carefully when a video of their interest is played.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to enable a viewer to view the area of interest more clearly by minimizing reduction in the resolution of partial images while suppressing the amount of data communication for wide-angle videos such as entire images and narrow-angle videos such as partial images.

The invention according to claim 1 is a photographing device that photographs a subject to obtain video data of a predetermined resolution, and includes a change means for changing a wide-angle video, which is all or a part of the video related to the video data of the predetermined resolution, to a low resolution, a projection method conversion means for projecting a narrow-angle video, which is a partial area of the wide-angle video, in a state of higher resolution than the wide-angle video after being changed to a low resolution by the change means, and a combination means for lowering the resolution of each of the frames of the low-resolution wide-angle video and the frames of the high-resolution narrow-angle video and combining them into one frame, and the projection method conversion means is characterized in that the projection method conversion is performed on a narrow-angle still image, which is the partial area of a wide-angle still image as a frame of the wide-angle video before being changed to a low resolution by the change means.

As described above, the present invention has the effect of suppressing the amount of data communication for wide-angle video and narrow-angle video while minimizing reduction in the resolution of narrow-angle images, thereby enabling the viewer to view the area of interest more clearly.

1A is a left side view of the omnidirectional imaging device, FIG. 1B is a rear view of the omnidirectional imaging device, FIG. 1C is a plan view of the omnidirectional imaging device, and FIG. 1D is a bottom view of the omnidirectional imaging device. FIG. 1 is a diagram illustrating an example of how the spherical imaging device is used. FIG. 1A is a diagram showing a hemispherical image (front) captured by an omnidirectional imaging device, FIG. 1B is a hemispherical image (back) captured by the omnidirectional imaging device, and FIG. 1C is a diagram showing an image represented by equirectangular projection. 1A is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and FIG. 1B is a diagram showing a spherical image. FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a state in which an image of a predetermined area is displayed on a display of a communication terminal. FIG. 2 is a diagram for explaining an outline of partial image parameters. 1 is a schematic diagram of a configuration of an imaging system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a hardware configuration of the omnidirectional imaging device. FIG. 2 is a diagram illustrating a hardware configuration of an intermediary terminal. FIG. 1 is a hardware configuration diagram of a smartphone. FIG. 2 is a hardware configuration diagram of the image management system. FIG. 2 is a functional block diagram of the omnidirectional imaging device according to the embodiment. FIG. 2 is a functional block diagram of the smartphone according to the present embodiment. FIG. 2 is a functional block diagram of the image management system. 1A is a conceptual diagram of horizontal/vertical angle of view threshold information, and FIG. 1B is an explanatory diagram of the horizontal angle of view and the vertical angle of view. FIG. 11 is a sequence diagram showing the process of generating and playing each data of an entire moving image and a partial moving image. 1A to 1C are conceptual diagrams of images in the process of image processing performed by the omnidirectional imaging device. FIG. 13 is a diagram illustrating partial image parameters. FIG. 13 is a conceptual diagram showing a state in which each frame image of an entire moving image and a partial moving image are combined. 1 is a conceptual diagram of an image during image processing performed by a smartphone. 13A and 13B are diagrams for explaining the creation of a partial solid sphere from a partial plane. FIG. 11 is a two-dimensional conceptual diagram illustrating a case where a partial image is superimposed on a spherical image without creating the partial 3D sphere of the present embodiment. FIG. 11 is a two-dimensional conceptual diagram of a case where a partial 3D sphere is created according to the present embodiment and a partial image is superimposed on a spherical image. FIG. 1 is a conceptual diagram showing (a) an example of a wide image display when there is no superimposed display, (b) an example of a telephoto image display when there is no superimposed display, (c) an example of a wide image display when there is superimposed display, and (d) an example of a telephoto image display when there is superimposed display. FIG. 11 is a sequence diagram showing a process of generating and playing partial still image data.

The following describes an embodiment of the present invention with reference to the drawings.

[Overview of the embodiment]
The outline of this embodiment will be described below.

The method for generating a spherical image will be explained using Figures 1 to 6.

First, the appearance of the omnidirectional imaging device will be described with reference to FIG. 1. The omnidirectional imaging device is a digital camera for obtaining captured images that are the basis of a omnidirectional (360°) panoramic image. FIG. 1(a) is a right side view of the omnidirectional imaging device, FIG. 1(b) is a front view of the omnidirectional imaging device, FIG. 1(c) is a plan view of the omnidirectional imaging device, and FIG. 1(d) is a bottom view of the omnidirectional imaging device.

As shown in Fig. 1(a), Fig. 1(b), Fig. 1(c), and Fig. 1(d), a fish-eye lens 102a is provided on the front side (front side) of the upper part of the omnidirectional imaging device, and a fish-eye lens 102b is provided on the rear side (rear side). Inside the omnidirectional imaging device, imaging elements (image sensors) 103a and 103b (described later) are provided, and a hemispherical image (angle of view 180° or more) can be obtained by photographing a subject or a landscape through the lenses 102a and 102b, respectively. A shutter button 115a is provided on the surface opposite to the front side of the omnidirectional imaging device. In addition, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side of the omnidirectional imaging device. Each time the shutter button 115a, the power button 115b, and the Wi-Fi button 115c are pressed, they are switched on and off. Furthermore, each time the shooting mode switching button 115d is pressed, the mode is switched between a still image shooting mode, a video shooting mode, and a video distribution mode. Note that the shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are types of the operation unit 115, and the operation unit 115 is not limited to these buttons.

A tripod screw hole 151 for attaching the omnidirectional imaging device to a camera tripod is provided in the center of the bottom 150 of the omnidirectional imaging device. A Micro USB (Universal Serial Bus) terminal 152 is provided on the left end of the bottom 150. An HDMI (High-Definition Multimedia Interface) terminal is provided on the right end of the bottom 150. HDMI is a registered trademark.

Next, the use of the omnidirectional imaging device will be described with reference to FIG. 2. FIG. 2 is an image diagram of the omnidirectional imaging device in use. As shown in FIG. 2, the omnidirectional imaging device is used, for example, by being held by a user in his/her hand to capture subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image sensor 103a and the image sensor 103b shown in FIG. 1, respectively.

Next, an outline of the process for creating an equirectangular projection image EC and an omnidirectional image CE from an image captured by an omnidirectional imaging device will be described with reference to Figs. 3 and 4. Fig. 3(a) shows a hemispherical image (front side) captured by an omnidirectional imaging device, Fig. 3(b) shows a hemispherical image (rear side) captured by an omnidirectional imaging device, and Fig. 3(c) shows an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(b) shows an omnidirectional image.

As shown in FIG. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by the fisheye lens 102a described below. Also, as shown in FIG. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by the fisheye lens 102b described below. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the omnidirectional imaging device, and an equirectangular projection image EC is created, as shown in FIG. 3(c).

By using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the sphere as shown in FIG. 4A, and a celestial sphere image CE as shown in FIG. 4B is created. In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image CE may be a still image or a video. In this embodiment, unless otherwise specified, the term "image" includes both a "still image" and a "video".

As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to humans when viewed. Therefore, by displaying a predetermined region of the spherical image CE (hereinafter referred to as a "predetermined region image") as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional sphere. Also, FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing the predetermined area image when displayed on a display. Also, in FIG. 6(a), the omnidirectional image shown in FIG. 4 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, as shown in FIG. 5, the virtual camera IC is located inside the omnidirectional image CE. The predetermined area T in the omnidirectional image CE is the shooting area of the virtual camera IC, and is specified by the predetermined area information indicating the position coordinates (x(rH), y(rV), angle of view α(angle)) including the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. Zooming of the specific area T can be expressed by widening or narrowing the range (arc) of the angle of view α. Zooming of the specific area T can also be expressed by moving the virtual camera IC closer to or farther away from the spherical image CE. The specific area image Q is an image of the specific area T in the spherical image CE.

The predetermined area image Q shown in FIG. 6(a) is displayed on a predetermined display as an image of the shooting area of the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by the predetermined area information that is initially set (default). Note that the predetermined area information may be represented not by the position coordinates of the virtual camera IC, but by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T.

FIG. 7 is a diagram for explaining an outline of partial image parameters. Here, a method for specifying a part of a spherical image will be explained. In a spherical image, the center point CP of a partial image can be expressed in the direction of the camera (here, virtual camera IC) that captures the image. With the center of the entire image, which is a spherical image, facing the entire image, the azimuth angle is "aa" and the elevation angle is "ea". In addition, to express the range of a partial image, for example, a diagonal angle of view α is used. In addition, to express the range in the vertical and horizontal directions, it is possible to express it by the aspect ratio of the image (width w÷height h). Here, the diagonal angle of view and the aspect are used to express the range, but the vertical angle of view and the horizontal angle of view, or the vertical angle of view and the aspect ratio, etc. may also be used. In addition to the azimuth angle and the elevation angle, a rotation angle may also be used.

[Outline of the imaging system]
First, an outline of the configuration of the imaging system of this embodiment will be described with reference to Fig. 8. Fig. 8 is a schematic diagram of the configuration of the imaging system of this embodiment.

As shown in FIG. 8, the photography system of this embodiment is composed of an omnidirectional photography device 1, an intermediary terminal 3, a smartphone 5, and an image management system 7.

Of these, the spherical imaging device 1 is a special digital camera that captures subjects, landscapes, etc., and obtains two hemispherical images that serve as the basis for a spherical (panoramic) image, as described above.

The intermediary terminal 3 is an example of a communication terminal that can directly communicate with a communication network 100 such as the Internet. The intermediary terminal 3 plays a role of mediating communication between the omnidirectional imaging device 1 and the image management system 7 by performing short-distance wireless communication with the omnidirectional imaging device 1 that cannot directly communicate with the communication network 100. The short-distance wireless communication is communication performed using technologies such as Wi-Fi, Bluetooth (registered trademark), and NFC (Near Field Communication). In FIG. 8, the intermediary terminal 3 is used like a dongle receiver in which the omnidirectional imaging device 1 is installed.

The smartphone 5 can communicate with the image management system 7 via the communication network 100, either wired or wirelessly. The smartphone 5 can also display images acquired from the omnidirectional imaging device 1 on a display 517 (described later) provided on the smartphone 5 itself.

The image management system 7 is constructed by a computer, and transmits requests for video and still images from the smartphone 5 to the omnidirectional imaging device 1 via the intermediary terminal 3. The image management system 7 also transmits video data and still image data sent from the omnidirectional imaging device 1 via the intermediary terminal 3 to the smartphone 5.

In FIG. 1, only one set of the omnidirectional imaging device 1 and the intermediary terminal 3 is shown, but multiple sets may be used. Also, only one smartphone 5 is shown, but multiple smartphones may be used. The smartphone 5 is an example of a communication terminal. In addition to the smartphone 5, communication terminals also include PCs (Personal Computers), smart watches, game devices, car navigation terminals, and the like. Furthermore, the image management system 7 may be constructed by not only a single computer but also multiple computers.

[Hardware configuration of the embodiment]
Next, the hardware configurations of the omnidirectional imaging device 1, the intermediary terminal 3, the smartphone 5, and the image management system 7 of this embodiment will be described in detail with reference to FIG. 9 and FIG. 12.

<Hardware configuration of the omnidirectional imaging device>
First, the hardware configuration of the omnidirectional imaging device 1 will be described with reference to Fig. 9. Fig. 9 is a hardware configuration diagram of the omnidirectional imaging device. In the following, the omnidirectional imaging device 1 is an omnidirectional (all-directional) imaging device capable of imaging 4π radians using two imaging elements, but the number of imaging elements may be two or more. In addition, the device does not necessarily need to be dedicated to omnidirectional imaging, and a normal digital camera, a smartphone, or the like may be provided with an omnidirectional imaging unit as a retrofit to have substantially the same functions as the omnidirectional imaging device 1.

As shown in FIG. 9, the omnidirectional imaging device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I/F 116, a communication unit 117, an antenna 117a, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a terminal 121.

Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each wide-angle lens. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image by the fisheye lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 via a serial I/F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the network I/F 116, the communication unit 117, and the electronic compass 118 are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via the parallel I/F bus, performs a predetermined process on each piece of image data, and then synthesizes the image data to create equirectangular projection image data as shown in Figure 3(c).

The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using the I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b, and send it to the CPU 111.

The imaging control unit 105 also instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some omnidirectional imaging devices 1 have a preview display function or a function corresponding to video display on a display (e.g., the display 517 of the smartphone 5). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames/second).

The imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b, as described below. Note that in this embodiment, the omnidirectional imaging device 1 is not provided with a display, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I/F bus and performs a predetermined process on the sound data.

The CPU 111 controls the overall operation of the omnidirectional imaging device 1 and executes necessary processing. There may be a single CPU 111 or multiple CPUs 111. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in the middle of processing. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular projection image.

The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various shooting modes, shooting conditions, and the like.

The network I/F 116 is a general term for an interface circuit (such as a USB I/F) with external media such as an SD card or a personal computer. The network I/F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on external media via the network I/F 116, or transmitted to an external terminal (device) such as a smartphone 5 via the network I/F 116 as necessary.

The communication unit 117 communicates with an external terminal (device) such as the smartphone 5 by using a short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth via an antenna 117a provided in the omnidirectional imaging device 1. The communication unit 117 can also transmit data of the equirectangular projection image to an external terminal (device) such as the smartphone 5.

The electronic compass 118 calculates the direction of the omnidirectional imaging device 1 from the Earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the capture date and time of the image and the data size of the image data.

The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the omnidirectional imaging device 1. The change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images.

The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The omnidirectional imaging device 1 calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device 1) based on the acceleration detected by the acceleration sensor 120. By providing the omnidirectional imaging device 1 with both the gyro sensor 119 and the acceleration sensor 120, the accuracy of image correction is improved.

Terminal 121 is a concave terminal for Micro USB.

<Hardware configuration of the intermediary terminal>
Next, the hardware configuration of the intermediary terminal 3 will be described with reference to Fig. 10. Fig. 10 is a diagram showing the hardware configuration of the intermediary terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 10, the intermediary terminal 3 includes a CPU 301 that controls the overall operation of the intermediary terminal 3, a ROM 302 that stores basic input/output programs, a RAM 303 that is used as a work area for the CPU 301, an EEPROM (Electrically Erasable and Programmable ROM) 304 that reads or writes data under the control of the CPU 301, and a CMOS sensor 305 that serves as an imaging element that captures an image of a subject and obtains image data under the control of the CPU 301.

The EEPROM 304 stores the operating system (OS) executed by the CPU 301, other programs, and various data. A CCD sensor may be used instead of the CMOS sensor 305.

The intermediary terminal 3 further includes an antenna 313a, a communication unit 313 that uses the antenna 313a to communicate with the image management system 7 via the communication network 100 using wireless communication signals, a GPS receiving unit 314 that receives GPS signals containing the position information (latitude, longitude, and altitude) of the intermediary terminal 3 via a GPS (Global Positioning Systems) satellite or an IMES (Indoor Messaging System) as an indoor GPS, and bus lines 310 such as an address bus and a data bus for electrically connecting the above-mentioned units.

<Hardware configuration of smartphone>
Next, the hardware of the smartphone will be described with reference to Fig. 11. Fig. 11 is a hardware configuration diagram of the smartphone. As shown in Fig. 11, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I/F 513a, an acceleration/orientation sensor 506, a media I/F 508, and a GPS receiver 509.

Of these, the CPU 501 controls the overall operation of the smartphone 5. The CPU 501 may be single or multiple. The ROM 502 stores the CPU 501 and programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data, such as smartphone programs, under the control of the CPU 501. The CMOS sensor 505 captures an image of a subject (mainly a self-portrait) under the control of the CPU 501 to obtain image data. The image sensor I/F 513a is a circuit that controls the operation of the CMOS sensor 512. The acceleration/direction sensor 506 is a variety of sensors, such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor, that detect geomagnetism. The media I/F 508 controls the reading or writing (storage) of data from the recording media 507, such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

The smartphone 5 also includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I/F 513b, a microphone 514, a speaker 515, an audio input/output I/F 516, a display 517, an external device connection I/F 518, a short-distance communication circuit 519, an antenna 519a of the short-distance communication circuit 519, and a touch panel 521.

Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network such as the Internet. The CMOS sensor 512 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 501. The image sensor I/F 513b is a circuit that controls the drive of the CMOS sensor 512. The microphone 514 is a type of built-in sound collection means that inputs sound. The sound input/output I/F 516 is a circuit that processes the input/output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I/F 518 is an interface for connecting various external devices. The short-distance communication circuit 519 is a communication circuit such as Wi-Fi, NFC, Bluetooth, etc. The touch panel 521 is a type of input means that allows a user to operate the smartphone 5 by pressing the display 517.

The smartphone 5 also includes a bus line 510. The bus line 510 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 501.

<Hardware configuration of image management system>
Next, the hardware configuration of the image management system will be described with reference to Fig. 12. Fig. 12 is a diagram showing the hardware configuration of the image management system.

Figure 12 is a hardware configuration diagram of the image management system 7. As shown in Figure 12, the image management system 7 is constructed by a computer, and as shown in Figure 12, it has a CPU 701, a ROM 702, a RAM 703, a HD 704, a HDD (Hard Disk Drive) controller 705, a display 708, a media I/F 707, a network I/F 709, a data bus 710, a keyboard 711, a mouse 712, and a DVD-RW (Digital Versatile Disk Rewritable) drive 714.

Of these, the CPU 701 controls the operation of the entire image management system 7. The ROM 702 stores programs used to drive the CPU 701, such as IPL. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD controller 705 controls the reading or writing of various data from the HD 704 according to the control of the CPU 701. The display 708 displays various information such as a cursor, menu, window, character, or image. The media I/F 707 controls the reading or writing (storage) of data from the recording medium 706, such as a flash memory. The network I/F 709 is an interface for data communication using the communication network 100. The bus line 710 is an address bus, data bus, etc. for electrically connecting each component such as the CPU 701 shown in FIG. 12.

The keyboard 711 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 714 controls the reading and writing of various data from the DVD-RW 713, which is an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R, Blu-ray Disc, etc.

[Functional configuration of the embodiment]
Next, the functional configuration of this embodiment will be described with reference to Fig. 13 to Fig. 16. Note that the main function of the intermediary terminal 3 is a transmission/reception unit for intermediating communication between the omnidirectional imaging device 1 and the image management system 7, and therefore a description thereof will be omitted.

<Functional configuration of the spherical imaging device>
Fig. 13 is a functional block diagram of the omnidirectional imaging device of this embodiment. The omnidirectional imaging device 1 includes a transmitting/receiving unit 11, a partial image parameter generating unit 12, an imaging control unit 13, imaging units 14a and 14b, an image processing unit 15, a temporary storage unit 16, a low-resolution changing unit 17, a projection method conversion unit 18, a combining unit 19, a moving image encoding unit 20a, a still image encoding unit 20b, a receiving unit 22, a determining unit 25, and a still image storage unit 20. Each of these units is a function or means realized by operating any of the components shown in Fig. 9 by an instruction from the CPU 111 according to a program for the omnidirectional imaging device expanded from the SRAM 113 onto the DRAM 114. The omnidirectional imaging device 1 also includes a threshold management unit 1001 realized by the SRAM 113.

(Horizontal and vertical angle of view threshold information)
As shown in Fig. 16A, the threshold management unit 1001 stores horizontal and vertical angle of view threshold information, for example, before shipping from a factory. Fig. 16A is a conceptual diagram of the horizontal and vertical angle of view threshold information, and Fig. 16B is an explanatory diagram of the horizontal angle of view and the vertical angle of view. Fig. 16A shows the horizontal and vertical angle of view threshold information in a table format, but the present invention is not limited to this and does not have to be shown in a table format.

As shown in FIG. 16(a), the horizontal and vertical angle of view threshold information is managed in association with each of the thresholds of the horizontal angle of view (AH) x vertical angle of view (AV) of the image used to determine whether to transmit the image from the omnidirectional imaging device 1 to the smartphone 5 via the image management system 7, a row of candidates for the maximum imaging resolution (W x H) of the omnidirectional imaging device 1, a row of candidates for the maximum display resolution (W' x H') of the smartphone 5 set by the user (viewer), and the horizontal angle of view (AH) x vertical angle of view (AV) of the image.

Now, the horizontal angle of view (AH) and the vertical angle of view (AV) will be described with reference to FIG. 16(b). In FIG. 16(b), the center point CP, azimuth angle "aa", and elevation angle "ea" are the same as in FIG. 7, and therefore will not be described. In a celestial sphere image (video, still image), the size of the entire area of one frame of a rectangular partial image (video, still image) can be expressed in terms of the direction of the camera (here, virtual camera IC) that captures the image. The horizontal length of the partial image can be expressed by the horizontal angle of view (AH) from the virtual camera IC, and the vertical length of the partial image can be expressed by the horizontal angle of view (AH) from the virtual camera IC.

The horizontal angle of view (AH) and vertical angle of view (AV) are angles of view at which there is no difference between the original resolution and the converted resolution, and are used for the judgment of the judgment unit 25. These angles of view can be calculated using the following (Equation 1) and (Equation 2).

AH = (360 / W) * W' ... (Equation 1)
AV = (180 / H) * (H' / 2) ... (Equation 2)
Even if the maximum shooting resolution of the omnidirectional imaging device 1 is (W: 4000, H: 2000), the omnidirectional imaging device 1 manages horizontal and vertical angle of view threshold information indicating each setting pattern shown in Fig. 16. However, in this case, the omnidirectional imaging device 1 refers to only the patterns of setting 1 and setting 2 in Fig. 16. Similarly, even if the maximum display resolution of the smartphone 5 set by the user is (W': 1920, H': 1080), the omnidirectional imaging device 1 manages horizontal and vertical angle of view threshold information indicating each setting pattern shown in Fig. 16.

For example, under a condition (setting 1) in which the maximum shooting resolution of the omnidirectional imaging device 1 is 4000 x 2000 in width x height and the user has set the smartphone 5 to request image (video, still image) data with a resolution of 1920 x 1080 in width x height from the omnidirectional imaging device 1, the determination unit 25 determines not to cause the projection method conversion unit 18 to perform projection method conversion to ultra-high definition still image data if the angle of view of the partial image requested by the instruction data is less than the threshold value indicated by 172.8° in width and 48.6° in height.

(Functional configuration of the spherical imaging device)
Next, each functional component of the omnidirectional imaging device 1 will be described in more detail with reference to FIG.

The transmitting/receiving unit 11 transmits image data to the outside of the omnidirectional imaging device 1 and receives data from the outside. For example, the image data includes data for an ultra-high definition partial still image, a high definition partial video, and a low definition full video. The transmitting/receiving unit 11 also receives instruction data (described below) and transmits partial image parameters (described below).

In this embodiment, since images of three levels of resolution are handled, the highest resolution image, the next highest resolution image, and the lowest resolution image are conveniently distinguished by being referred to as "ultra-high definition," "high definition," and "low definition," respectively. Therefore, for example, "ultra-high definition" in this embodiment does not mean general ultra-high definition.

The partial image parameter creation unit 12 creates partial image parameters based on instruction data sent from the smartphone 5 via the image management system 7. This instruction data is received by the reception unit 52 of the smartphone 5 through a user's operation, and is data indicating an instruction for identifying the overlapping area described below.

The imaging control unit 13 outputs instructions to synchronize the output timing of image data from the imaging units 14a and 14b.

The imaging units 14a and 14b each capture an image of a subject or the like in accordance with instructions from the imaging control unit 13, and output hemispherical image data that is the basis of the omnidirectional image data, for example, as shown in Figures 3(a) and (b).

The image processing unit 15 synthesizes and converts the two hemispherical image data obtained by the image capturing units 14a and 14b into equirectangular projection image data, which is an image obtained using the equirectangular projection method.

The temporary storage unit 16 acts as a buffer that temporarily stores the data of the equirectangular projection image that has been synthesized and converted by the image processing unit 15. Note that the equirectangular projection image in this state is ultra-high definition.

The low-resolution change unit 17 changes the equirectangular projection image from an ultra-high resolution image (here, a video) to a low-resolution image (here, a video) by reducing the image or the like in accordance with instruction data from the smartphone 5 received by the transmission/reception unit 11. This generates a low-resolution equirectangular projection image (here, a full video). The low-resolution change unit 17 is an example of a change means.

The projection method conversion unit 18 converts the projection method from equirectangular projection to perspective projection according to the instruction data and video data request received by the transmission/reception unit 11, i.e., according to the direction, angle of view, and aspect of the partial image, which is an image that is a part of the entire image, and the size of the image data to be transmitted to the smartphone 5 via the image management system 7. When partial video data is requested in this manner, the projection method conversion unit 18 generates a partial image (here, a partial video) in a high-definition state that has a lower definition (or resolution) than the ultra-high definition entire image stored in the temporary storage unit 16, but a higher definition (or resolution) than the low-definition image obtained by the change made by the low-definition change unit 17.

Furthermore, the projection method conversion unit 18 can also convert the projection method from equirectangular projection to perspective projection according to the instruction data and still image data request received by the transmission/reception unit 11, i.e., according to the direction, angle of view, and aspect of the partial image, which is an image that is a part of the entire image, and the size of the image data to be transmitted to the smartphone 5 via the image management system 7. When partial still image data is requested in this way, the projection method conversion unit 18 generates a partial image (here, a partial still image) while keeping the ultra-high definition state stored in the temporary storage unit 16.

As described above, the difference between the low-definition change unit 17 and the projection method change unit 18 is not only whether or not projection method conversion is performed, but also that the overall image (here, overall video) data output from the low-definition change unit 17 has a lower definition (or resolution) than the partial image (here, partial video and partial still image) data output from the projection method conversion unit 18. In other words, the partial image data output from the projection method conversion unit 18 has a higher definition (or resolution) than the overall image data output from the low-definition change unit 17.

Here, we will explain the case where, for example, a positive distance cylindrical projection image with a resolution of 2K, 4K, or 8K is output as an ultra-high definition image. Based on the data of these ultra-high definition positive distance cylindrical projection images, the "partial video" data output from the projection method conversion unit 18 has a resolution of 1.5K, 3K, or 6K. Based on the data of these ultra-high definition positive distance cylindrical projection images, the "partial still image" data output from the projection method conversion unit 18 has a resolution of 2K, 4K, or 8K. On the other hand, the "full video" data output from the low definition change unit 17 has a resolution of 1K, 2K, or 4K.

The combining unit 19 reduces the resolution of each frame of the low-resolution overall video modified by the low-resolution modification unit 17 and the high-resolution partial video converted by the projection method conversion unit 18 by compressing the vertical resolution to half, as shown in FIG. 20, to combine them into one frame. As a result, the partial video has higher resolution than the overall video, but lower resolution than the partial still image. By reducing the resolution of each frame into one frame in this way, the amount of data communication can be reduced. Furthermore, since the two frames to be combined are generated from the same equidistant circular projection image stored in the temporary storage unit 16, two frames captured and generated at the same time can be associated without using metadata or the like for associating the two frames to be combined.

The video encoding unit 20a encodes the frame data of the entire video and the partial video combined by the combining unit 19. The still image encoding unit 20b encodes the data of the partial still images.

The reception unit 22 receives operations performed on the operation unit 115 of the spherical imaging device when the user makes various requests.

The determination unit 25 determines whether the entire area of one frame of a partial image, which is a part of the entire video, is smaller than a predetermined area indicated by a predetermined threshold of the horizontal and vertical angle of view threshold information (see FIG. 16) managed by the threshold management unit 1001. If it is smaller, the determination unit 25 prevents the projection method conversion unit 18 from generating ultra-high definition partial still image data from the ultra-high definition entire video data. If it is larger, the determination unit 25 causes the projection method conversion unit 18 to generate ultra-high definition partial still image data from the ultra-high definition entire video data.

For example, when the maximum shooting resolution of the omnidirectional shooting device 1 is 4000×2000 in width×height, and the maximum display resolution of the smartphone 5 is set to 1920×1080 in width×height by the user from the smartphone 5 to the omnidirectional shooting device 1 (setting 1), if the horizontal angle of view of the image requested by the instruction data is less than 172.8° and the vertical angle of view is less than 48.6°, the determination unit 25 determines that the projection method conversion unit 18 will not convert the projection method to ultra-high definition still image data. In this way, when the resolution (definition) of the image requested by the smartphone 5 is low, even if the omnidirectional shooting device 1 sends data of a high-definition partial video or data of an ultra-high definition partial still image, the user will find it difficult to distinguish between a high-definition partial video and an ultra-high definition partial still image on the smartphone 5, so the omnidirectional shooting device 1 does not need to bother to send data of an ultra-high definition partial still image. Note that "less than" in the threshold range can also be changed to "equal to or less."

On the other hand, under the same conditions as above (setting 1), if the horizontal angle of view of the image requested by the instruction data exceeds 172.8° or the vertical angle of view exceeds 48.6°, the judgment unit 25 judges that the projection method conversion unit 18 should perform projection method conversion to ultra-high definition still image data. In this case, the smartphone 5 switches the high definition partial video to an ultra-high definition still image for display, allowing the user (viewer) to view the area of interest more clearly (crisply), thereby enabling careful viewing. Note that the word "ultra" shown in the threshold range may be replaced with "above or equal to."

<Functional configuration of smartphone>
Fig. 14 is a functional block diagram of the smartphone of this embodiment. The smartphone 5 has a transmission/reception unit 51, a reception unit 52, a moving image decoding unit 53a, a still image encoding unit 53b, a superimposition area creation unit 54, an image creation unit 55, an image superimposition unit 56, a projective transformation unit 57, and a display control unit 58. Each of these units is a function or means realized by any of the components shown in Fig. 11 operating in response to an instruction from the CPU 501 in accordance with the smartphone program expanded from the EEPROM 504 onto the RAM 503.

(Smartphone functional configuration)
Next, each functional configuration of the smartphone 5 will be described in more detail with reference to FIG.

The transmitting/receiving unit 51 transmits data to the outside of the smartphone 5 and receives data from the outside. For example, the transmitting/receiving unit 51 receives image data from the transmitting/receiving unit 11 of the omnidirectional imaging device 1 and transmits instruction data to the transmitting/receiving unit 11 of the omnidirectional imaging device 1. The transmitting/receiving unit 51 also separates the image data (the entire video data and partial video data shown in FIG. 20 ) sent from the transmitting/receiving unit 11 of the omnidirectional imaging device 1 from the partial image parameters.

The reception unit 52 receives from the user a specification operation for the direction, angle of view, and aspect of the partial image, as well as the size of the image data to be received by the smartphone 5. The reception unit 52 also receives from the user a setting for the maximum display resolution of the smartphone 5 (see width W' × height H' in FIG. 16).

The video decoding unit 53a decodes each of the low-definition overall video data and high-definition partial video data encoded by the video encoding unit 20a. The still image decoding unit 53b decodes the ultra-high-definition partial still image data encoded by the still image encoding unit 20b.

The overlapping area creation unit 54 creates an overlapping area specified by the partial image parameters. This overlapping area indicates the overlapping position and overlapping range of the overlapping image S and mask image M, which are partial videos (or partial still images), on the omnidirectional image CE, which is the entire video.

The image creation unit 55 creates a superimposed image S and a mask image according to the superimposed area, and creates a spherical image CE from the low-resolution overall image.

The image overlay unit 56 overlays the overlay image S and the mask image M onto the overlay area on the omnidirectional image CE to create the final omnidirectional image CE.

The projection transformation unit 57 converts the final spherical image CE into a perspective projection image in accordance with the user's instructions received by the reception unit 52.

The display control unit 58 controls the display of the image converted to the perspective projection method on the display 517, etc.

<Functional configuration of the image management system>
Fig. 15 is a functional block diagram of the image management system. The image management system 7 has a transmission/reception unit 71. The transmission/reception unit 71 is a function or means realized by the network I/F shown in Fig. 12 operating in response to commands from the CPU 701 in accordance with the image management system program loaded from the HD 704 onto the RAM 703. The image management system 7 also has a storage unit 7000 realized by the HD 704.

(Functional configuration of image management system)
Next, the functional configuration of the image management system 7 will be described in more detail with reference to FIG.

The transmitting/receiving unit 71 transmits data to the outside of the image management system 7 and receives data from the outside. For example, the transmitting/receiving unit 71 receives image data from the transmitting/receiving unit 11 of the omnidirectional shooting device 1 via the intermediary terminal 3, and transmits instruction data to the transmitting/receiving unit 11 of the omnidirectional shooting device 1 via the intermediary terminal 3. The transmitting/receiving unit 71 also transmits image data (the whole video data and the partial video data shown in FIG. 20 ) and partial image parameters transmitted from the transmitting/receiving unit 11 of the omnidirectional shooting device 1 via the intermediary terminal 3 to the smartphone 5. Furthermore, the transmitting/receiving unit 71 temporarily stores the ultra-high definition still image data transmitted from the transmitting/receiving unit 11 of the omnidirectional shooting device 1 via the intermediary terminal 3 in the storage unit 7000, and reads out the ultra-high definition still image data from the storage unit 7000 and transmits it to the smartphone 5.

[Processing or Operation of the Embodiment]
Next, the processing or operation of this embodiment will be described with reference to Fig. 17 to Fig. 26. In this embodiment, a case will be described in which, while an entire video and a partial video related to data generated by and transmitted from the omnidirectional imaging device 1 are being played back on the smartphone 5, a viewer carefully views one scene of a partial video as a still image with higher definition (i.e., ultra-high definition).

<<Creating and playing full videos and partial videos>>
First, the process of generating and playing each data of the entire moving image and the partial moving images will be described with reference to Fig. 17. Fig. 17 is a sequence diagram showing the process of generating and playing each data of the entire moving image and the partial moving images.

As shown in FIG. 17, the viewer (user) operates the smartphone 5, and the reception unit 52 receives a request to start a video distribution service (S11). This causes the transmission/reception unit 51 of the smartphone 5 to transmit a request for video data to the transmission/reception unit 71 of the image management system 7 (S12). At this time, partial image parameters specified by the viewer are also transmitted. Note that the transmission/reception unit 11 of the omnidirectional imaging device 1 may perform bandwidth control of the communication network with respect to the timing of the start of transmission. By performing this bandwidth control, data can be transmitted and received more stably.

Next, the transmitting/receiving unit 71 of the image management system 7 transfers the request for video data to the transmitting/receiving unit of the intermediary terminal 3 (S13). Then, the transmitting/receiving unit of the intermediary terminal 3 transfers the request for video data to the transmitting/receiving unit 11 of the omnidirectional imaging device 1 (S14).

Next, the spherical imaging device 1 performs a process of generating video data (S15). The process of step S15 will be described in detail later (see Figs. 18 to 20).

Next, the transmitting/receiving unit 11 of the omnidirectional imaging device 1 transmits video data in response to the request to the transmitting/receiving unit of the intermediary terminal 31 (S16). This video data includes data for the low-resolution overall video and the high-resolution partial video. As a result, the transmitting/receiving unit of the intermediary terminal 3 transfers the video data to the transmitting/receiving unit 71 of the image management system 7 (S17). Furthermore, the transmitting/receiving unit 71 of the image management system 7 transfers the video data to the transmitting/receiving unit 51 of the smartphone 5 (S18).

Next, the smartphone 5 performs a playback process of the video data (S19). The process of step S19 will be described in detail later (see Figures 21 to 25).

<Video Generation Process of Omnidirectional Camera>
Next, the generation process of video data by the omnidirectional imaging device shown in step S15 will be described in detail with reference to Fig. 18 to Fig. 20. Fig. 18 is a conceptual diagram of an image in the process of image processing performed by the omnidirectional imaging device.

The image processing unit 15 synthesizes and converts the two hemispherical image data obtained by the image capturing units 14a and 14b into equirectangular projection image data, which is an image (here, a video) using the equirectangular projection method (S120). This converted data is temporarily stored in the temporary storage unit 16 in the state of an ultra-high definition image.

Next, the partial image parameter creation unit 12 creates partial image parameters based on the instruction data sent from the smartphone 5 (S130).

Next, the low-definition change unit 17 changes the ultra-high definition image to a low-definition image in accordance with the instruction data received by the transmission/reception unit 11 from the smartphone 5 (S140). This generates a low-definition equirectangular projection image (here, the whole video).

Furthermore, the projection method conversion unit 18 converts the projection method from equirectangular projection to perspective projection in accordance with the instruction data received by the transmission/reception unit 11, i.e., in accordance with the orientation of the partial video, which is a partial area of each frame of the entire video, the angle of view and aspect of the frame of the partial video, and the size of the video data to be transmitted to the smartphone 5 (S150). This generates a high-definition partial video.

Next, the combining unit 19 combines the data of the low-resolution overall video and the high-resolution partial video (S160). This combining process will be described in detail later (see FIG. 20).

The process shown in FIG. 18 will now be described in more detail with reference to FIG. 19 and FIG. 20. FIG. 19 is a diagram explaining partial image parameters.

(Partial image parameters)
First, the partial image parameters will be described in detail with reference to Fig. 19. Fig. 19(a) shows the entire image after image synthesis in S120. Fig. 19(b) shows an example of the partial image parameters. Fig. 19(c) shows the partial image after the projection method has been converted in S150.

The azimuth angle described in FIG. 7 corresponds to the horizontal direction (latitude λ) in the equirectangular projection shown in FIG. 19(a), and the elevation angle described in FIG. 7 corresponds to the vertical direction (longitude φ) in the equirectangular projection method. The parameters shown in FIG. 19(b), combining the diagonal angle of view and aspect ratio in FIG. 7, are the partial image parameters. FIG. 19(c) is an example of a partial image cut out from the area surrounded by the frame in the equirectangular projection of FIG. 19(a) using the partial image parameters.

Here, the conversion of the projection method will be described. As shown in FIG. 4A, a celestial sphere image is created by covering a stereoscopic sphere with an equirectangular projection image. Therefore, each pixel data of the equirectangular projection image can be made to correspond to each pixel data on the surface of the stereoscopic sphere of the three-dimensional celestial sphere image. Therefore, the conversion formula by the projection method conversion unit 18 can be expressed by the following (Formula 3) when the coordinates in the equirectangular projection image are expressed as (latitude, longitude)=(e, a) and the coordinates on the three-dimensional stereoscopic sphere are expressed as Cartesian coordinates (x, y, z).
(x, y, z) = (cos(ea) × cos(aa), cos(ea) × sin(aa), sin(ea)) ... (Equation 3)
In this case, the radius of the three-dimensional sphere is set to 1.

On the other hand, a partial image, which is a perspective projection image, is a two-dimensional image, but when this is expressed in two-dimensional polar coordinates (radius, argument) = (r, a), the radius r corresponds to the diagonal angle of view, and the possible range is 0 ≦ r ≦ tan (diagonal angle of view/2). In addition, when a partial image is expressed in a two-dimensional orthogonal coordinate system (u, v), the conversion relationship with the polar coordinates (radius, argument) = (r, a) can be expressed by the following (Equation 4).
u = r × cos(a), v = r × sin(a) ... (Equation 4)
Next, consider making this (Equation 3) correspond to three-dimensional coordinates (radius, polar angle, azimuth angle). Since only the surface of the solid sphere CS is considered, the radius in the three-dimensional polar coordinates is "1". Furthermore, if we consider the virtual camera to be at the center of the solid sphere, the projection that performs perspective transformation on the equirectangular projection image pasted onto the surface of the solid sphere CS can be expressed by the following (Equation 5) and (Equation 6) using the above-mentioned two-dimensional polar coordinates (radius, deviation angle) = (r, a).
r = tan(polar angle) ... (Equation 5)
a = azimuth angle ... (Equation 6)
Here, if the polar angle is t, then t = arctan(r), so three-dimensional polar coordinates (radius, polar angle, azimuth angle) can be expressed as (radius, polar angle, azimuth angle) = (1, arctan(r), a).

The conversion equation for converting from three-dimensional polar coordinates to a Cartesian coordinate system (x, y, z) can be expressed by the following (Equation 7).
(x, y, z) = (sin(t) × cos(a), sin(t) × sin(a), cos(t)) ... (Equation 7)
The above formula (7) enables mutual conversion between an entire image by equirectangular projection and a partial image by perspective projection. That is, by using the radius vector r corresponding to the diagonal angle of view of the partial image to be created, it is possible to calculate conversion map coordinates that indicate which coordinates of the equirectangular projection image each pixel of the partial image corresponds to, and based on the conversion map coordinates, it is possible to create a partial image that is a perspective projection image from the equirectangular projection image.

The above projection transformation method involves a transformation in which the position where the (latitude, longitude) of the equirectangular projection image is (90°, 0°) becomes the center point of the partial image, which is a perspective projection image. Therefore, when performing perspective projection transformation with an arbitrary point of the equirectangular projection image as the gaze point, it is sufficient to rotate the three-dimensional sphere to which the equirectangular projection image is attached, thereby performing coordinate rotation so that the coordinates (latitude, longitude) of the gaze point are positioned at a position of (90°, 0°).

The transformation formula for the rotation of this three-dimensional sphere is a general coordinate rotation formula, so we will not explain it here.

(Joining process)
Next, the combining process of step S160 performed by the combining unit 19 will be described with reference to FIG. 20. FIG. 20 is a conceptual diagram showing a state in which each frame image of the entire video and the partial video are combined. As shown in FIG. 20, the entire image is combined so as to be arranged at the top of one frame and the partial image is combined at the bottom. In this embodiment, the images are generally arranged according to 16:9, which is an aspect ratio of HD, but the aspect ratio does not matter. The arrangement may be not only top and bottom but also left and right. When there are multiple partial images, for example, the entire video may be arranged in the upper half and the partial video may be arranged in the lower half by dividing them by the number of partial videos. By combining the entire video and the partial images into one for each frame, it is possible to ensure synchronization between the images. In addition, the omnidirectional imaging device 1 may transmit the entire video data and the partial video data separately to the smartphone 5.

However, because the entire video and partial video are halved vertically, their resolution is reduced by half compared to when they are not halved. In other words, the resolution is lower than the ultra-high definition image stored in the temporary storage unit 16. As a result, the resolution is, in descending order, the entire image stored in the temporary storage unit 16, the partial image after conversion by the projection method conversion unit 18, and the entire image after conversion by the low definition change unit 17.

<Video playback processing on smartphones>
Next, the playback process of video data by the smartphone 5 will be described with reference to Fig. 21. Fig. 21 is a conceptual diagram of an image in the process of image processing performed by the smartphone. The overlapping area creation unit 54 shown in Fig. 7 creates a partial 3D sphere PS specified by the partial image parameters as shown in Fig. 21 (S320).

Next, the image creation unit 55 creates a superimposed image S by superimposing a partial image in the perspective projection method on the partial 3D sphere PS (S330). The image creation unit 55 also creates a mask image M based on the partial 3D sphere PS (S340). The image creation unit 55 also creates a celestial sphere image CE by pasting an entire image in the equirectangular projection method onto the 3D sphere CS (S350). Then, the image superimposition unit 56 superimposes the superimposed image S and the mask image M on the celestial sphere image CE (S360). This completes a low-resolution celestial sphere image (here, a low-resolution entire video) CE on which a high-resolution superimposed image (here, a high-resolution partial video) S is superimposed so that the boundary is not noticeable.

Next, the projective transformation unit 57 performs projective transformation based on the line of sight and angle of view of the predetermined virtual camera so that a specific area of the spherical image CE with the superimposed image S superimposed can be viewed on the display 517 (S370).

Fig. 22 is a diagram for explaining the creation of a partial three-dimensional sphere from a partial plane. Normally, in the perspective projection method, projection is performed onto a plane, and therefore it is often expressed as a plane in three-dimensional space, as shown in Fig. 22(a). In this embodiment, a partial three-dimensional sphere that is a part of a sphere is created to match the spherical image, as shown in Fig. 22(b). Here, the conversion from a plane to a partial three-dimensional sphere is explained.

Consider the projection onto a sphere of each point (X, Y, Z) on a plane arranged at an appropriate size (angle of view) as shown in Figure 22 (a). The projection onto the sphere is the intersection of the straight line passing from the origin of the sphere through each point (X, Y, Z) and the sphere. Each point on the sphere is a point whose distance from the origin is equal to the radius of the sphere. Therefore, if the radius of the sphere is 1, then the point (X', Y', Z') on the sphere shown in Figure 22 (b) can be expressed by the following (Equation 8).

(X', Y', Z') = (X, Y, Z) x 1 / √( ^X2 + ^Y2 + ^Z2 ) ... (Equation 8)
Fig. 23 is a two-dimensional conceptual diagram of a case where a partial image is superimposed on a celestial sphere image without creating the partial 3D sphere of this embodiment. Fig. 24 is a two-dimensional conceptual diagram of a case where a partial image is superimposed on a celestial sphere image by creating the partial 3D sphere of this embodiment.

23(a), when the virtual camera IC is located at the center point of the three-dimensional sphere CS, the subject P1 is represented as an image P2 on the omnidirectional image CE and as an image P3 on the superimposed image S. As shown in FIG. 23(a), the images P2 and P3 are located on a straight line connecting the virtual camera IC and the subject P1, so even if the superimposed image S is displayed in a state where it is superimposed on the omnidirectional image CE, no misalignment occurs between the omnidirectional image CE and the superimposed image S. However, as shown in FIG. 23(b), when the virtual camera IC moves away from the center point of the three-dimensional sphere CS (when the angle of view α is reduced), the image P2 is located on the straight line connecting the virtual camera IC and the subject P1, but the image P3 is located slightly inside. For this reason, if the image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is image P3', a shift occurs between the spherical image CE and the superimposed image S by the amount of shift g between the images P3 and P3'. As a result, the superimposed image S is displayed shifted relative to the spherical image CE, but such a superimposed display is also acceptable.

In contrast to this, in this embodiment, since a partial 3D sphere is created, the superimposed image S can be superimposed along the omnidirectional image CE as shown in Figs. 24(a) and (b). As a result, not only when the virtual camera IC is located at the center point of the 3D sphere CS as shown in Fig. 24(a), but also when the virtual camera is away from the center point of the 3D sphere CS as shown in Fig. 24(b), the images P2 and P3 are located on a straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superimposed image S is displayed superimposed on the omnidirectional image CE, no misalignment occurs between the omnidirectional image CE and the superimposed image S.

Fig. 25(a) is a conceptual diagram showing an example of a wide image without superimposed display, Fig. 25(b) is an example of a telephoto image without superimposed display, Fig. 25(c) is an example of a wide image with superimposed display, and Fig. 25(d) is an example of a telephoto image with superimposed display. Note that the wavy lines in the diagrams are shown only for the convenience of explanation, and may or may not actually be displayed on the display 517.

25(a), when the partial image P (here, a high-definition partial video) is not superimposed on the omnidirectional image CE (here, a low-definition overall video) as shown in FIG. 25(a), if the area shown by the wavy line in FIG. 25(a) is enlarged, the image remains low-definition as shown in FIG. 25(b), and the user ends up seeing a blurred image. On the other hand, when the partial image P is superimposed on the omnidirectional image CE as shown in FIG. 25(c), if the area shown by the wavy line in FIG. 25(c) is enlarged, a high-definition image is displayed as shown in FIG. 25(d), and the user can see a clear image. In particular, if a signboard or the like with letters is displayed in the area shown by the wavy line, unless the high-definition partial image P is superimposed, the letters will be blurred even when enlarged, and it will be difficult to understand what is written. However, if the high-definition partial image P is superimposed, the letters will be clearly visible even when enlarged, and the user will be able to understand what is written.

<<Creation and playback of partial still images>>
Next, the process of generating and playing back partial still image data will be described with reference to Fig. 26. Fig. 26 is a sequence diagram showing the process of generating and playing back partial still image data.

When a viewer (user) is viewing an entire image (here, a low-definition entire video) on which a partial image (here, a high-definition partial video) P as shown in FIG. 25(c) is superimposed in step S19 shown in FIG. 17, and wishes to view the partial video as a still image with even higher resolution, the viewer (user) operates the smartphone 5 as shown in FIG. 26, causing the reception unit 52 to receive a request to start a distribution service for the partial still image (S31). This then causes the transmission/reception unit 51 of the smartphone 5 to transmit a request for still image data to the transmission/reception unit 71 of the image management system 7 (S32). At this time, the partial image parameters specified by the viewer are also transmitted.

Next, the transmitting/receiving unit 71 of the image management system 7 transfers the request for still image data to the transmitting/receiving unit of the intermediary terminal 3 (S33). Then, the transmitting/receiving unit of the intermediary terminal 3 transfers the request for still image data to the transmitting/receiving unit 11 of the omnidirectional imaging device 1 (S34).

Next, the spherical imaging device 1 performs processing to generate still image data (S35). The processing of step S35 will be described in detail later.

Next, the transmitting/receiving unit 11 of the omnidirectional imaging device 1 transmits still image data in response to the request to the transmitting/receiving unit of the intermediary terminal 31 (S36). This still image data is data of a partial still image with ultra-high resolution. Note that the timing to start transmitting the still image data may be during a time period when network traffic is low, such as at night, or when the delay time is measured and becomes shorter than a predetermined delay time. As a result, the transmitting/receiving unit of the intermediary terminal 3 transfers the still image data to the transmitting/receiving unit 71 of the image management system 7 (S37).

Next, when the transmission/reception unit 71 of the image management system 7 receives still image data, it stores the still image data in the storage unit 7000 (S38).

Then, the transmission/reception unit 7 of the image management system 7 transfers the still image data to the transmission/reception unit 51 of the smartphone 5 (S39). At this time, in regard to the timing of the start of transfer, the bandwidth control of the communication network 100 is performed in the same manner as in the data transmission from the omnidirectional imaging device 1 to the image management system via the intermediary terminal 3.

Next, the smartphone 5 superimposes and displays an ultra-high definition partial still image in place of the high definition partial video that has been superimposed on the entire video (S40). This method of superimposition includes processes common to the case where a partial video is superimposed on the entire video, but there are also different processes, which will be described in detail later.

If the judgment unit 25 judges in step S35 that the projection method conversion unit 18 should not convert the projection method to ultra-high definition still image data, a message such as "still image not to be transmitted" or "angle of view is insufficient" may be transmitted in place of the still image data in steps S36 to S37. In this case, in step S40, the message is displayed with the partial video still superimposed on the entire video.

<Generation of Still Images from an Omnidirectional Camera>
Next, the process of generating still image data of the omnidirectional imaging device shown in step S35 above will be described in detail.

When a request for still image data is received in the omnidirectional imaging device 1, a still image capture determination is performed. This is because, if the angle of view of the partial image is relatively narrow, there is no difference between the resolution of the partial still image created by conversion from the original image and the resolution of the partial video being transmitted. In this case, the horizontal angle of view (AH) and vertical angle of view (AV) of the horizontal/vertical angle of view threshold information shown in FIG. 16 are set as the minimum angles of view, and if the angle of view specified by the partial image parameters is wider than the relevant angles of view, a still image generation process is executed (S210).

More specifically, when the transmitter/receiver 11 of the omnidirectional imaging device 1 receives a request for still image data while transmitting video data, the projection method conversion unit 18 performs projection method conversion from equirectangular projection to perspective projection for the ultra-high definition partial still image data in accordance with the partial image parameters, in the same way as when generating the partial video data. In this case, the size of the generated still image is variable, and conversion is performed while maintaining the resolution of the ultra-high definition image. If the horizontal resolution of the ultra-high definition image is W and the specified horizontal angle of view is ah, the horizontal resolution Wp of the generated still image can be calculated as follows:

Wp = W / 360 * ah ... (Equation 9)
The size of W here is obtained from the W column of the horizontal/vertical angle of view threshold information (see FIG. 16A ) that is read when the omnidirectional imaging device 1 is started up.

Similarly, if the specified vertical angle of view is av, the vertical resolution Hp of the generated still image can be calculated using the following formula.

Hp=H/180*av... (Equation 10)
Further, the still image encoding unit 20b encodes the data of the partial still image and stores it in the image buffer. Then, the transmitting/receiving unit 11 controls the communication network bandwidth as described above, as in steps S16 and S17, and transmits the data of the partial still image to the image management system 7 via the intermediary terminal 3. In this case, the partial image parameters are also transmitted.

<Playing still images from a smartphone>
Next, the playback process of the still image data by the smartphone 5 shown in step S40 above will be described in detail.

On the smartphone 5 side, the transmission/reception unit 51 separates the still image data sent from the omnidirectional imaging device 1 into partial still image data and partial image parameters received by the communication unit. The still image decoding unit 53b decodes the partial still image data after separation. After that, the above-mentioned processing for each data of the entire video and partial video is simply replaced with processing for each data of the entire video and partial still images, so a description thereof will be omitted.

[Main Effects of the Embodiments]
As described above, according to this embodiment, the amount of data communication for the entire video and partial video is reduced, while the resolution of the partial images is not reduced as much as possible, thereby enabling the viewer to more clearly view the area that interests them.

The omnidirectional imaging device 1 also creates a low-resolution whole image (here, a low-resolution whole video) from an ultra-high-resolution omnidirectional image (S140), and creates a high-resolution partial image (here, a high-resolution partial video) with a different projection method from the same ultra-high-resolution omnidirectional image (S150). Then, the omnidirectional imaging device 1 transmits data of the low-resolution whole image and the high-resolution partial image to the smartphone 5. In response to this, the smartphone 5 superimposes the high-resolution partial image on the low-resolution whole image (S360), and converts the projection method according to the line of sight and angle of view specified by the user (viewer) (S370). In this way, the omnidirectional imaging device 1 transmits the partial image, which is the area of interest, of the ultra-high-resolution omnidirectional image obtained by capturing an object or the like, with the resolution reduced to high, and transmits the whole image (for grasping the entire omnidirectional image) that is not of interest with the resolution reduced to low, and transmits the high-resolution partial image after converting the projection method. Furthermore, before this transmission, the combining unit 19 reduces the resolution of each frame by compressing the vertical resolution of the low-definition overall video data and the high-definition partial video data by half, as shown in Fig. 20, and combines them into one frame. This reduces the amount of data on the receiving smartphone 5 compared to the conventional method, and has the effect of quickly displaying a celestial sphere image in which a partial image is superimposed on an overall image.

Therefore, according to this embodiment, in order to reduce the amount of communication, the omnidirectional imaging device 1 distributes data of a low-resolution overall image obtained by reducing the entire omnidirectional image and data of a high-resolution partial image that is a region of interest within the omnidirectional image (overall image), and the smartphone 5 on the receiving side not only composites the partial image onto the overall image, but also composites and displays the low-resolution overall image and the high-resolution partial image on the smartphone 5 even if they are different projection methods, resulting in an effect of high versatility in terms of projection methods.

〔supplement〕
In the above embodiment, the determination unit 25 determines whether or not the entire area of one frame of a partial image, which is a part of the entire video, is smaller than a predetermined area indicated by a predetermined threshold of the horizontal/vertical angle of view threshold information (see FIG. 16 ) managed by the threshold management unit 1001. If it is smaller, the determination unit 25 prevents the projection method conversion unit 18 from performing projection method conversion to data of an ultra-high definition still image, and if it is larger, the determination unit 25 causes the projection method conversion unit 18 to perform projection method conversion to data of an ultra-high definition still image, but this is not limited thereto. For example, if it is smaller, the transmission/reception unit 11 may perform a process of not transmitting the data of the partial still image after the projection method conversion stored in the still image storage unit 29 to the smartphone 5 side, and if it is larger, the transmission/reception unit 11 may perform a process of transmitting the data of the partial still image after the projection method conversion stored in the still image storage unit 29 to the smartphone 5 side. In this case, the judgment unit 25 may instruct the transmission/reception unit 11 as to whether or not to transmit, or the judgment unit 25 may instruct the still image memory unit 29 as to whether or not to transmit the data of the partial still image after projection method conversion stored in the memory unit 29 to the transmission/reception unit 11.

The spherical imaging device 1 is an example of an imaging device, and imaging devices also include digital cameras and smartphones that capture normal planar images. Digital cameras and smartphones that capture normal planar images can capture relatively wide-angle images (wide-angle images) rather than spherical images.

The smartphone 5 is also an example of a communication terminal or image processing device, and examples of the communication terminal or image processing device include PCs such as tablet PCs (Personal Computers), notebook PCs, and desktop PCs. Examples of the communication terminal or image processing device include smart watches, game devices, and car navigation terminals installed in vehicles, etc.

In the above embodiment, the low-definition image is an entire video, which is the entire area of the image of the image data obtained from the imaging units 14a and 14b, and the high-definition image is a partial video, which is a part of the entire area, but the present invention is not limited to this. The low-definition image may be an image of a part of the area A1 of the entire area of the image of the video data obtained from the imaging units 14a and 14b. In this case, the high-definition image is an image of a part of the area A2 of the part A1. That is, the relationship between the low-definition entire video and the high-definition partial video is that the former is a wide-angle video, whereas the latter is a narrow-angle video. In addition, the partial still image can also be called a narrow-angle still image, in the same way as the partial video. In this embodiment, the wide-angle image (including both still images and videos) is an image that is captured by an imaging device equipped with a so-called wide-angle lens or fisheye lens, and in which distortion occurs. In addition, the narrow-angle image (including both still images and videos) is a part of an image captured by an imaging device equipped with a wide-angle lens or fisheye lens, and has a narrower angle of view than the wide-angle image.

In the above embodiment, a case where a planar image is superimposed on a spherical image has been described, but superimposition is an example of compositing. In addition to superimposition, compositing also includes pasting, embedding, and superimposing. Furthermore, the above superimposed image is an example of a composite image. In addition to superimposition images, composite images also include pasting images, embedding images, and superimposed images. Furthermore, the image superimposition unit 56 is an example of an image composition unit.

Furthermore, the equirectangular projection image EC and the planar image P may both be still images, both be video frames, or one may be a still image and the other a video frame.

In addition, in the above embodiment, the projection method conversion unit 18 converts the projection method of the high-resolution image data acquired from the temporary storage unit 16 while maintaining the same resolution, but this is not limited to this. For example, if the resolution is higher than the overall image data output from the low-resolution change unit 17, the projection method conversion unit 18 may lower the resolution of the image data acquired from the temporary storage unit 16 when converting the projection method.

Each functional configuration shown in FIG. 13 and FIG. 14 can be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. In this case, the technical solution of the present embodiment is expressed in the form of a software product, either essentially or as a part of the prior art. The computer software product is stored in a storage medium and includes a number of instructions that cause a computer device (such as a personal computer, a server, or a network device) to execute all or some of the steps of the method according to each of the above embodiments. The above-mentioned storage medium includes various media capable of storing program code, such as a USB memory, a removable disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The methods according to the above embodiments are applied to a processor or are realized by a processor. The processor is an integrated circuit board capable of processing signals. Each step of the method according to the above embodiments is realized by an integrated logic circuit, which is hardware, or by instructions in the form of software in the processor. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can realize or execute each method, step, and logic box disclosed in each of the above embodiments. The general-purpose processor may be a microprocessor or any general processor. Each step of the method according to the above embodiments may be realized by being executed by a decoder, which is hardware, or may be realized by a combination of hardware and software in the decoder. The software module is stored in a storage medium mature in the field, such as a random memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The processor reads information from the memory comprising the storage medium in which the software is stored, and realizes the steps of the method according to the hardware.

The above-described embodiments may be realized by hardware, software, firmware, middleware, microcode, or a combination thereof. In the case of hardware, the processing unit may be realized by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units that perform the functions of the present invention, or a combination thereof. In the case of software, the above techniques are realized by modules (e.g., processes, functions, etc.) that realize the above-described functions. The software code is stored in a memory and executed by a processor. The memory may be realized inside or outside the processor.

1. Omnidirectional imaging device (an example of an imaging device)
5. Smartphone (an example of a communication terminal, an example of an image processing device)
13 Imaging control unit 14a Imaging unit 14b Imaging unit 15 Image processing unit 16 Temporary storage unit 17 Low-definition change unit (an example of a change means)
18 projection method conversion unit (an example of a projection method conversion means)
19 Joint (an example of a joining means)
20a: moving image encoding unit 20b: still image encoding unit 22: reception unit 51: transmission/reception unit 52: reception unit 53a: moving image decoding unit 53b: still image encoding unit 54: superimposition area creation unit 55: image creation unit 56: image superimposition unit 57: projection transformation unit 58: display control unit

JP 2006-340091 A

Claims (10)

An imaging device for capturing an image of a subject and obtaining video data of a predetermined resolution,
A change means for changing a wide-angle moving image, which is all or a part of the moving image related to the moving image data of the predetermined resolution, to a low resolution image;
a projection method conversion means for converting a narrow-angle moving image, which is a partial area of the wide-angle moving image, into a state of higher resolution than the wide-angle moving image after the narrow-angle moving image is converted to a low resolution by the conversion means;
a combining means for combining the frames of the low-definition wide-angle video and the frames of the high-definition narrow-angle video into one frame by reducing the resolution of each of the frames;
having
The projection method conversion means converts the projection method of a narrow-angle still image, which is a partial area of a wide-angle still image as a frame of a wide-angle video before being converted to low resolution by the conversion means. The imaging device according to claim 1 ,
a transmitting means for transmitting the frame data of the wide-angle video and the frame data of the narrow-angle video, which have been combined into one frame by the combining means, so that the frame data can be received by an image management system that manages image data;
A receiving means for receiving a request for a still image transmitted by a communication terminal;
having
an imaging device characterized in that, based on reception of a request for the still image by the receiving means, the transmitting means transmits data of the narrow-angle still image after conversion by the projection method conversion means so as to be received by the image management system. The photographing device described in claim 2, characterized in that if the entire area of one frame of the narrow-angle moving image is smaller than a predetermined area, the projection method conversion means does not perform the projection method conversion of the narrow-angle still image, or the transmission means does not transmit data of the narrow-angle still image after conversion by the projection method conversion means. The imaging device according to any one of claims 1 to 3, characterized in that the wide-angle video is composed of the entire area of a frame of video data of a predetermined resolution obtained by photographing the subject. The imaging device according to any one of claims 1 to 4, characterized in that the projection method conversion means converts the projection method of the narrow-angle still image while keeping the specified resolution. The imaging device according to any one of claims 1 to 5, characterized in that the imaging device is an omnidirectional imaging device that captures an object and obtains omnidirectional image data. The imaging device according to claim 2 ;
the image management system;
1. An imaging system comprising: The imaging device according to claim 2 ;
the image management system;
a communication terminal that receives data of the wide-angle video and data of the narrow-angle video from the image management system, synthesizes and displays the narrow-angle video in the partial area on the wide-angle video, and receives data of the narrow-angle still image from the image management system, synthesizes the narrow-angle still image in the partial area on the wide-angle still image , and displays the narrow-angle still image in place of the narrow-angle video;
1. An imaging system comprising: An image processing method for photographing a subject and processing video data of a predetermined resolution, comprising the steps of:
a change step of changing a wide-angle moving image, which is all or a part of the moving image related to the moving image data of the predetermined resolution, to a low resolution moving image;
a projection method conversion step of performing projection method conversion on a narrow-angle moving image, which is a partial area of the wide-angle moving image, in a state of higher resolution than the wide-angle moving image after being changed to a low resolution by the change step;
a combining step of combining the low-definition wide-angle video frame and the high-definition narrow-angle video frame into one frame by reducing the resolution of each of the frames;
Run
An image processing method characterized in that the projection method conversion step includes a process of converting the projection method of a narrow-angle still image, which is a portion of a wide-angle still image as a frame of a wide-angle video before being converted to low resolution by the conversion step. A program causing a computer to execute the method according to claim 9 .