JP2019164782A - Image processing apparatus, image capturing system, image processing method, and program - Google Patents

Abstract

To solve the problem in which the difference in the projection type between an equirectangular projection image EC (spherical image CE) and a planar image P causes poor visibility when the high-definition planar image P captured separately from the spherical image CE is superimposed on a partial area of the low-definition spherical image CE.SOLUTION: An image processing apparatus calculates in a first image EC a first corresponding area CA1 corresponding to a second image P, determines a peripheral area PA including the area CA1, and converts the area PA to the projection type of the image P (S140). The image processing apparatus calculates in an image PI after the conversion a second corresponding area CA2 corresponding to the image P (S160). The image processing apparatus generates a rectangular default shape (S170), and converts the default shape to the projection type of the image EC thereby creating a reference shape (S200). The image processing apparatus calculates reference shape conversion information for converting the reference shape on the basis of a relation of the image P to the default shape (S180).SELECTED DRAWING: Figure 19

Description

  The present invention relates to an image processing apparatus, a photographing system, an image processing method, and a program.

  Conventionally, an enlarged planar image obtained by photographing separately from a wide-angle planar image is inserted into a partial area of the wide-angle planar image, so that the partial area is enlarged even when the partial area is enlarged. A technique capable of displaying a simple image is disclosed (see Patent Document 1).

  By the way, in recent years, there has been provided a special digital camera that obtains two hemispherical image data based on a 360-degree omnidirectional image by one shooting (see Patent Document 2). This digital camera creates one equirectangular projection image data based on two hemispherical image data, and transmits the equirectangular projection image data to a communication terminal such as a smartphone. The communication terminal that has obtained the equirectangular projection image data creates an omnidirectional image based on the equirectangular projection image data. However, since the image is curved as it is, it is difficult for the user to see. Therefore, by displaying a predetermined area image indicating a predetermined area of the omnidirectional image on the communication terminal, the user is photographed with a general digital camera. Can be viewed with the same feeling as a flat image.

  However, for example, in the case where a planar image obtained by shooting separately from the omnidirectional image is superimposed on a partial area of the omnidirectional image, one image having a different projection method is combined with the other image. When images are superimposed, there is a problem that it is difficult to see.

  The invention according to claim 1 is the first image of the first projection method, the second image of the second projection method different from the first projection method, and the second image on the first image. Acquisition means for acquiring reference shape information indicating a reference shape corresponding to the image of the image, reference shape conversion information for converting the reference shape on the first image, and based on the reference shape conversion information An image creating unit that creates a composite image corresponding to the second image by converting reference shape information, and an image composition unit that composites the composite image with the first image, An image processing apparatus.

  As described above, according to the present invention, it is possible to suppress the difficulty of seeing even if the other image is combined with one image having a different projection method.

(A) is a left side view of the special photographing apparatus, (b) is a rear view of the special photographing apparatus, (c) is a plan view of the special photographing apparatus, and (d) is a bottom view of the special photographing apparatus. It is. It is a usage image figure of a special imaging device. (A) is a hemispheric image (front) photographed by a special photographing device, (b) is a hemispheric image (rear) photographed by a special photographing device, and (c) is an image represented by equirectangular projection. FIG. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of the virtual camera and predetermined | prescribed area | region at the time of making an omnidirectional image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. 1 is a schematic diagram of an imaging system according to a first embodiment of the present invention. It is a usage image figure of an imaging system. It is a hardware block diagram of a special imaging device. It is a hardware block diagram of a general imaging device. It is a hardware block diagram of a smart phone. It is a functional block diagram of the imaging system concerning a 1st embodiment. (A) is a conceptual diagram of a cooperative photographing device management table, (b) is a conceptual diagram showing a cooperative photographing device setting screen. It is a detailed functional block diagram of the metadata creation part which concerns on 1st Embodiment. It is a detailed functional block diagram of the superimposition part which concerns on 1st Embodiment. It is a block diagram of superimposition display metadata. It is the sequence diagram which showed the imaging | photography method which concerns on 1st Embodiment. It is a conceptual diagram of the image in the process of the production | generation process of a superimposition display parameter. It is a conceptual diagram at the time of specifying a peripheral region image. (A) is a conceptual diagram showing an initial setting shape, (b) is a conceptual diagram showing a peripheral region image and a second corresponding region, and (c) is a conceptual diagram of an initial setting shape divided into regions. (A) Conceptual diagram when the center of the reference shape is arranged at the center of the omnidirectional image, (b) Conceptual diagram when the center of the reference shape is arranged at a point other than the center of the omnidirectional image. It is a conceptual diagram of the image in the process of the superimposition display process. It is a conceptual diagram which shows the interpolation method of an angle parameter. It is the three-dimensional conceptual diagram which showed the flow of the process in a position parameter calculation part. (A) is the conceptual diagram which showed the lattice area | region of the 3rd corresponding | compatible area | region, (b) is the conceptual diagram which showed the lattice area | region of the plane image. It is the conceptual diagram which showed the four grid | lattice area | regions and the common grid point which is one point of a corner. It is a conceptual diagram of the image in the process of the superimposition process. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the position parameter of the present embodiment. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image using the position parameters of the present embodiment. (A) Display example of wide image without superimposing display, (b) Display example of tele image without superimposing display, (c) Display example of wide image with superimposing display, (d) In case of superimposing display It is the conceptual diagram which showed the example of a display of a tele image. It is a figure which shows the example in which the plane image of a moving image is displayed within the predetermined area image of a moving image. It is the schematic of the imaging | photography system which concerns on the 2nd Embodiment of this invention. It is a hardware block diagram of an image processing server. It is a functional block diagram of the imaging system which concerns on 2nd Embodiment. It is a detailed functional block diagram of the metadata creation part which concerns on 2nd Embodiment. It is a detailed functional block diagram of the superimposition part which concerns on 2nd Embodiment. It is the sequence diagram which showed the imaging | photography method which concerns on 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the omnidirectional image described later is an example of the first image, and the superimposed image is an example of the second image. The peripheral area image is an example of a third image.

[Outline of Embodiment]
Hereinafter, an outline of the present embodiment will be described.

  First, a method for generating an omnidirectional image will be described with reference to FIGS.

  First, the external appearance of the special photographing apparatus 1 will be described with reference to FIG. The special photographing apparatus 1 is a digital camera for obtaining a photographed image that is the basis of a panoramic image of a celestial sphere (360 °). 1A is a left side view of the special photographing apparatus, FIG. 1B is a rear view of the special photographing apparatus, and FIG. 1C is a plan view of the special photographing apparatus. (D) is a bottom view of the special photographing apparatus.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and (d), a fish-eye lens is placed on the front side (front side) of the special photographing apparatus 1 at the top. A fisheye lens 102b is provided on the rear side (rear side) 102a. Inside the special imaging device 1, imaging elements (image sensors) 103a and 103b, which will be described later, are provided, and a hemispherical image (angle of view 180 °) is obtained by photographing a subject or a landscape through the lenses 102a and 102b, respectively. Above). A shutter button 115 a is provided on the surface opposite to the front side of the special imaging device 1. In addition, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the special imaging device 1. Each time the power button 115b and the Wi-Fi button 115c are pressed, they are switched on and off. The shooting mode switching button 115d switches between a still image shooting mode and a moving image shooting mode each time the button is pressed. Note that the shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are part of the operation unit 115, and the operation unit 115 is not limited to these buttons.

  A tripod screw hole 151 for attaching the special photographing device 1 or the general photographing device 3 to a camera tripod is provided at the center of the bottom 150 of the special photographing device 1. A micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom 150. An HDMI (High-Definition Multimedia Interface) terminal 153 is provided on the right end side of the bottom 150. HDMI is a registered trademark.

  Next, the use situation of the special photographing apparatus 1 will be described with reference to FIG. In addition, FIG. 2 is a use image figure of a special imaging device. As shown in FIG. 2, the special imaging device 1 is used, for example, for a user to take a picture of a subject around the user with his / her hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, the outline of processing from the image captured by the special imaging device 1 to the creation of the equirectangular projection image EC and the omnidirectional image CE will be described with reference to FIGS. 3 and 4. 3A is a hemispheric image (front side) photographed by the special photographing apparatus 1, FIG. 3B is a hemispheric image photographed by the special photographing apparatus (rear side), and FIG. It is the figure which showed the image (henceforth "an equirectangular projection image") represented by the cylindrical projection. FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fish-eye lens 102a described later. Also, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image reversed 180 degrees (rear side) are synthesized by the special imaging device 1, and as shown in FIG. 3C, the equirectangular projection image EC. Is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4A, the equirectangular projection image is pasted so as to cover the spherical surface, and FIG. An omnidirectional image CE as shown in b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, by displaying a predetermined area (hereinafter referred to as “predetermined area image”) of a part of the omnidirectional image CE as a flat image with little curvature, a display that does not give a sense of discomfort to humans can be achieved. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. In FIG. 6A, the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. The

  Then, the predetermined area image Q shown in FIG. 6A is displayed on the predetermined display as an image of the photographing area of the virtual camera IC as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. The following description will be made using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC.

  The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. As shown in FIG. 7, “ea” represents an elevation angle, “aa” represents an azimuth angle, and “α” represents an angle of view. That is, the attitude of the virtual camera IC is changed so that the gazing point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP of the predetermined area T that is the shooting area of the virtual camera IC. Become. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following (formula 1) is generally established.

L / f = tan (α / 2) (Formula 1)
[First Embodiment]
Subsequently, a first embodiment of the present invention will be described with reference to FIGS.

<< Summary of shooting system >>
First, an outline of the configuration of the photographing system of the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the photographing system of the present embodiment.

  As shown in FIG. 8, the photographing system of this embodiment includes a special photographing device 1, a general photographing device 3, a smartphone 5, and an adapter 9. The special photographing device 1 is connected to the general photographing device 3 through an adapter 9.

  Among these, the special photographing device 1 is a special digital camera for photographing a subject, a landscape, and the like to obtain two hemispherical images that are the basis of the omnidirectional (panoramic) image as described above.

  The general photographing device 3 is a compact digital camera, but may be a digital single-lens reflex camera.

  The smartphone 5 can perform wireless communication with the special imaging device 1 and the general imaging device 3 using a short-range wireless communication technology such as Wi-Fi, Bluetooth (registered trademark), or NFC (Near Field Communication). Further, the smartphone 5 can display images respectively acquired from the special imaging device 1 and the general imaging device 3 on a display 517 described later provided in the device itself.

  Note that the smartphone 5 may communicate with the special imaging device 1 and the general imaging device 3 through a wired cable without using the short-range wireless communication technology. The smartphone 5 is an example of an image processing apparatus, and the image processing apparatus includes a tablet PC (Personal Computer), a notebook PC, and a desktop PC. Note that the smartphone is also an example of a communication terminal described later.

  The adapter 9 includes a bracket 9a and a rotation mechanism 9b. A tripod screw 9c for attaching to the tripod screw hole 151 of the special photographing apparatus 1 is attached to the distal end side of the bracket 9a. A rotation mechanism 9b is attached to the base end side of the bracket 9a. The general photographing device 3 is attached to the rotation mechanism 9b, and as shown in FIG. 8, the general photographing device 3 can be rotated by Pitech, Yaw, and Roll about three axes.

  FIG. 9 is a usage image diagram of the photographing system. As shown in FIG. 9, the user attaches the adapter 9 to which the special photographing device 1 and the general photographing device 3 are attached on the tripod 2. Then, the user can operate the smartphone 5 to remotely drive the rotation mechanism 9b or start and end shooting of the special imaging device 1 and the general imaging device 3. Note that an installation table or the like may be used without using the tripod 2.

<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of the special imaging device 1, the general imaging device 3, and the smartphone 5 according to the present embodiment will be described in detail with reference to FIGS.

<Hardware configuration of special imaging device>
First, the hardware configuration of the special photographing apparatus 1 will be described with reference to FIG. FIG. 10 is a hardware configuration diagram of the special imaging device 1. In the following, the special imaging device 1 is an omnidirectional (omnidirectional) special imaging device using two image sensors, but any number of two or more image sensors may be used. In addition, it is not always necessary to use a dedicated device for omnidirectional photography. By attaching a retrofit omnidirectional imaging unit to a normal digital camera or smartphone, it has substantially the same function as the special photography device 1. Also good.

  As shown in FIG. 10, the special imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only). Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, acceleration sensor 120, And a terminal 121.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by the fisheye lenses 102a and 102b into image data of an electric signal and output the image data. A timing generation circuit for generating a horizontal or vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 through a serial I / F bus (I2C bus or the like). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. The data of the equirectangular projection image as shown in FIG. 3C is created.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data and the like of the register groups of the imaging elements 103a and 103b and send them to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some special imaging devices 1 may have a preview display function or a function corresponding to a moving image display by a display (for example, the display 517 of the smartphone 5). In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the special imaging apparatus 1 is not provided with a display, but a display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the special photographing apparatus 1 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed equirectangular projection image data.

  The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various shooting modes and shooting conditions.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. Data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or an external terminal (device) such as the smartphone 5 via the network I / F 116 as necessary. ).

  The communication unit 117 communicates with an external terminal (device) such as the smartphone 5 through the antenna 117a provided in the special imaging device 1 by a short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth. The communication unit 117 can also transmit the equirectangular projection image data to an external terminal (device) such as the smartphone 5.

  The electronic compass 118 calculates the azimuth of the special imaging device 1 from the earth's magnetism and outputs azimuth information. This orientation information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. Note that the related information includes each data of the image capturing date and time and the data capacity of the image data.

  The gyro sensor 119 is a sensor that detects a change in angle (Roll angle, Pitch angle, Yaw angle) accompanying the movement of the omnidirectional camera 20. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

  The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The special imaging device 3a calculates the attitude (angle with respect to the direction of gravity) of the own device (special imaging device 3a) based on the acceleration detected by the acceleration sensor 120. By providing both the gyro sensor 119 and the acceleration sensor 120 in the special imaging device 3a, the accuracy of image correction is improved.

  The terminal 121 is a concave terminal for Micro USB.

<Hardware configuration of general imaging device>
Next, the hardware of the general photographing apparatus will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of the general photographing apparatus 3. As shown in FIG. 11, the general imaging device 3 includes an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a bus 310, a CPU 311, a ROM 312, an SRAM 313, a DRAM 314, an operation The unit 315 includes a network I / F 316, a communication unit 317, an antenna 317 a, an electronic compass 318, and a display 319. The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310.

  Configurations 304, 310, 311, 312, 313, 314, 315, 316, 317, 317a, and 318 are the configurations 104, 110, 111, 112, 113, 114, and 115 in the special imaging apparatus 1 of FIG. , 116, 117, 117a, and 118, the description thereof is omitted.

  Further, as shown in FIG. 11, the imaging unit 301 of the general imaging apparatus 3 is provided with a lens unit 306 and a mechanical shutter 307 in order from the outside in the direction of the imaging element 303 on the front surface of the imaging element 303. .

  The imaging control unit 305 basically performs the same configuration and processing as the imaging control unit 105, but further drives the lens unit 306 and the mechanical shutter 307 based on a user operation received by the operation unit 315. To control.

  The display 319 is an example of a display unit that displays an operation menu and an image during or after shooting.

<Smartphone hardware configuration>
Next, the hardware of the smartphone will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the smartphone. As shown in FIG. 12, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I / F 513a, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiver 509. Yes.

  Among these, the CPU 501 controls the operation of the entire smartphone 5. The ROM 502 stores programs used for driving the CPU 501 such as the CPU 501 and IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as a smartphone program under the control of the CPU 501. The CMOS sensor 505 captures a subject (mainly a self-portrait) under the control of the CPU 501 and obtains image data. The image sensor I / F 513 a is a circuit that controls driving of the CMOS sensor 512. The acceleration / direction sensor 506 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 508 controls reading or writing (storage) of data with respect to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

  The smartphone 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I / F 513b, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, an external device connection I / F 518, and a short distance. A communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521 are provided.

  Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network 100 described later. The CMOS sensor 512 is a kind of built-in imaging unit that captures an image of a subject under the control of the CPU 501 and obtains image data. The image sensor I / F 513b is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal display or an organic EL display that displays a subject image, various icons, and the like. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 521 is a kind of input means for operating the smartphone 5 when the user presses the display 517.

  In addition, the smartphone 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of the present embodiment will be described with reference to FIGS. 10 to 13. FIG. 13 is a functional block diagram of the special imaging device 1, the general imaging device 3, and the smartphone 5 that constitute a part of the imaging system of the present embodiment.

<Functional configuration of special imaging device>
First, the functional configuration of the special imaging device 1 will be described in detail with reference to FIGS. 10 and 13. As illustrated in FIG. 13, the special imaging device 1 includes a reception unit 12, an imaging unit 13, a sound collection unit 14, an image / sound processing unit 15, a determination unit 17, a short-range communication unit 18, and a storage / readout. It has a part 19. Each of these units is a function realized by any one of the constituent elements shown in FIG. 10 being operated by a command from the CPU 111 according to a program for the special photographing apparatus developed from the SRAM 113 onto the DRAM 114, or Means.

  The special photographing apparatus 1 has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG.

(Functional configuration of special imaging equipment)
Next, each functional configuration of the special imaging device 1 will be described in more detail with reference to FIGS. 10 and 13.

  The reception unit 12 of the special imaging device 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 illustrated in FIG. 10 and receives operation input from the user.

  The imaging unit 13 is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. obtain. As shown in FIGS. 3A and 3B, the captured image data is two hemispherical image data that is the basis of the omnidirectional image data.

  The sound collecting unit 14 is realized by the processing of the microphone 108 and the sound processing unit 109 and the CPU 111 shown in FIG. 10 and collects sounds around the special photographing apparatus 1.

  The image / sound processing unit 15 is realized mainly by a command from the CPU 111 and performs various processes on the captured image data obtained by the imaging unit 13 or the sound data obtained by the sound collecting unit 14. For example, the image / sound processing unit 15 performs equirectangular projection image data based on two hemispherical image data (see FIGS. 3A and 3B) obtained by the two image sensors 103a and 103b, respectively. (See FIG. 3C).

  The determination unit 17 is realized by the processing of the CPU 111 and makes various determinations.

  The short-range communication unit 18 is mainly realized by the processing of the CPU 111, the communication unit 117, and the antenna 117a, and communicates with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. Can do.

  The storage / reading unit 19 is realized mainly by the processing of the CPU 111 shown in FIG. 10, and stores various data (or information) in the storage unit 1000 or stores various data (or information) from the storage unit 1000. Read out.

<Functional configuration of general imaging device>
Next, the functional configuration of the general imaging device 3 will be described in detail with reference to FIGS. 11 and 13. As illustrated in FIG. 13, the general imaging device 3 includes a reception unit 32, an imaging unit 33, a sound collection unit 34, an image / sound processing unit 35, a display control unit 36, a determination unit 37, and a short-range communication unit 38. And a storage / reading unit 39. Each of these units is a function realized by any one of the constituent elements shown in FIG. 11 being operated by a command from the CPU 311 according to a program for the special photographing apparatus developed from the SRAM 313 to the DRAM 314, or Means.

  The general photographing apparatus 3 includes a storage unit 3000 configured by the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG. 11.

(Functional configuration of general imaging equipment)
The reception unit 32 of the general photographing apparatus 3 is mainly realized by the processing of the operation unit 315 and the CPU 311 illustrated in FIG. 11 and receives operation input from the user.

  The imaging unit 33 is realized mainly by the processing of the imaging unit 301, the image processing unit 304, the imaging control unit 305, and the CPU 311 shown in FIG. obtain. This photographed image data is planar image data photographed by the perspective projection method.

  The sound collection unit 34 is realized by the processing of the microphone 308 and the sound processing unit 309 and the CPU 311 illustrated in FIG. 11 and collects sounds around the general photographing apparatus 3.

  The image / sound processing unit 35 is realized mainly by a command from the CPU 311, and performs various processes on the captured image data obtained by the imaging unit 33 or the sound data obtained by the sound collection unit 34.

  The display control unit 36 is realized by the processing of the CPU 311 illustrated in FIG. 11, and causes the display 319 to display the planar image P related to the captured image data during or after capturing.

  The determination unit 37 is realized by the processing of the CPU 311 and performs various determinations. For example, the determination unit 37 determines whether the user has pressed the shutter button 315a.

  The short-range communication unit 38 is mainly realized by the CPU 311, the communication unit 317, and the antenna 317 a, and can communicate with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. .

  The storage / reading unit 39 is realized mainly by the processing of the CPU 311 shown in FIG. 11, and stores various data (or information) in the storage unit 3000 or stores various data (or information) from the storage unit 3000. Read out.

<Functional configuration of smartphone>
Next, the functional configuration of the smartphone 5 will be described in detail with reference to FIGS. As illustrated in FIG. 13, the smartphone 5 includes a long-distance communication unit 51, a reception unit 52, an imaging unit 53, a sound collection unit 54, an image / sound processing unit 55, a display control unit 56, a determination unit 57, A distance communication unit 58 and a storage / reading unit 59 are provided. Each of these units is a function or means realized by any of the constituent elements shown in FIG. 12 being operated by a command from the CPU 501 according to the smartphone 5 program expanded from the EEPROM 504 to the RAM 503. is there.

  The smartphone 5 includes a storage unit 5000 configured by the ROM 502, the RAM 503, and the EEPROM 504 illustrated in FIG. In this storage unit 5000, a cooperative photographing apparatus management DB 5001 is constructed. This cooperative photographing apparatus management DB 5001 is configured by a cooperative photographing apparatus management table in FIG. FIG. 14A is a conceptual diagram of the cooperative photographing apparatus management table.

(Cooperative shooting device management table)
Next, the cooperative photographing apparatus management table will be described with reference to FIG. As shown in FIG. 14A, for each image capturing device, related relationship information indicating the cooperation relationship of each image capturing device, the IP address of the image capturing device, and the device name of the image capturing device are managed in association with each other. Of these, the related relationship information indicates that one imaging device that starts imaging when the shutter of its own device is pressed is “main”, and shooting is performed in response to the shutter being pressed by the “main” imaging device. Another imaging device to start is indicated as “sub”. Note that the IP address is for Wi-Fi communication, and in the case of wireless communication using Bluetooth instead of the manufacturer ID (Identification) and product ID in the case of communication using a USB wired cable. Replaces BD (Bluetooth Device Address).

(Each smartphone functional configuration)
The long-distance communication unit 51 of the smartphone 5 is mainly realized by the processing of the long-distance communication circuit 511 and the CPU 501 illustrated in FIG. 12, and is connected to another device (for example, other network) via a communication network such as the Internet. Various data (or information) is transmitted to and received from a smartphone or server.

  The accepting unit 52 is realized mainly by the processes of the touch panel 521 and the CPU 501 and accepts various selections or inputs from the user. The touch panel 521 may be shared with the display 517. Further, input means (buttons) other than the touch panel may be used.

  The imaging unit 53 is realized mainly by the processing of the CMOS sensors 505 and 512 and the CPU 501 shown in FIG. 12, and images a subject, a landscape, and the like to obtain captured image data. This photographed image data is planar image data photographed by the perspective projection method.

  The sound collection unit 54 is realized by the processing of the microphone 514 and the CPU 501 illustrated in FIG. 12 and collects sounds around the smartphone 5.

  The image / sound processing unit 55 is realized mainly by a command from the CPU 501, and performs various processes on the captured image data obtained by the imaging unit 53 or the sound data obtained by the sound collection unit 54.

  The display control unit 56 is realized by the processing of the CPU 501 illustrated in FIG. 12, and causes the display 517 to display the planar image P related to the captured image data during or after the imaging by the imaging unit 53. In addition, the display control unit 56 uses the superimposed display metadata created by the image / sound processing unit 55 and uses a background moving image (entire sky) specified by the reference shape conversion information with respect to the reference shape information (reference position parameter). The planar image P is superimposed and displayed on the omnidirectional image CE on the basis of the brightness value and the color value indicated by the conversion position parameter on the spherical image CE) and the correction parameter. The conversion position parameter is an example of “conversion position information”. The correction parameter is an example of “correction information”.

  The determination unit 57 is realized by the processing of the CPU 501 shown in FIG. 12, and makes various determinations.

  The short-range communication unit 58 is mainly realized by the processing of the CPU 501 and the short-range communication circuit 519 and the antenna 519a. It is possible to communicate by a short-range wireless communication technology such as Wi-Fi.

  The storage / reading unit 59 is mainly realized by the processing of the CPU 501 shown in FIG. 12, and stores various data (or information) such as superimposed display metadata in the storage unit 5000 or from the storage unit 5000. Various data (or information) such as superimposed display metadata is read out. The storage / reading unit 59 serves as an acquisition unit that acquires various data from the storage unit 5000.

(Detailed functional configuration of image / sound processor)
Here, the functional configuration of the image / sound processing unit 55 will be described in detail with reference to FIGS. 15 and 16. FIG. 15 is a detailed functional block diagram of the metadata creation unit according to the first embodiment. FIG. 16 is a detailed functional block diagram of the superposition unit according to the first embodiment.

  The image / sound processing unit 55 includes a metadata creating unit 55a that performs encoding and a superimposing unit 55b that performs decoding. The metadata creation unit 55a executes a process of step S19 described later shown in FIG. In addition, the superimposing unit 55b executes a process of step S20 described later shown in FIG.

{Each functional configuration of the metadata creation unit}
First, each functional configuration of the metadata creation unit 55a will be described. The metadata creation unit 55a includes a (metadata) creation moving image frame extraction unit 548, an extraction unit 550, a first corresponding region calculation unit 552, a gazing point identification unit 554, a projection method conversion unit 556, and a second corresponding region calculation. A unit 558, an initial setting shape generation unit 559, a region division unit 560, a projection method inverse conversion unit 562, a reference shape conversion information calculation unit 568, and a superimposed display metadata creation unit 570. Reference numerals indicating images and regions described below are shown in FIG. FIG. 19 is a conceptual diagram of an image in the process of creating a superimposed display parameter.

  The creation moving image frame extraction unit 548 extracts a frame image corresponding to each designated time from one moving image that is a background moving image and the other moving image that is a foreground moving image. Although it is well known that moving images have various recording formats, it can be regarded as an image in which a plurality of still images are arranged in time series. The number of still images constituting a moving image for one second is a frame rate, and is represented by, for example, one frame per second (1 frame / 1 second). Therefore, when the first frame at the start of moving image recording is time 0, it is possible to calculate and extract the number of the frame image at the time T from the relationship between the frame rate and the time. Of course, there may be no frame image having the same time as the designated time ti, and processing such as making a frame close to the previous and next frame images into a frame at the time ti can be considered. In this embodiment, two moving images, a moving image that becomes a background moving image and a moving image that becomes a foreground moving image, are handled. In this embodiment, since the superimposition position for superimposing the foreground video on the background video is calculated, it is easier to handle the image with a smaller time difference, and when the recording start times of the foreground video and the background video are shifted, The time reference should be adjusted to either the foreground video or the background video. The method for calculating the time lag for matching the start image frames of the two moving images as the foreground moving image and the background moving image is, for example, when sound is recorded together with the image in each moving image, There are a method of calculating cross-correlation and obtaining a time shift at which the cross-correlation shift is minimized, and a method of obtaining from the image correlation between frame images when there is no sound data.

  Here, a supplementary explanation will be given regarding the synchronization of the background video and the foreground video. For example, it is assumed that there are a background moving image and a foreground moving image recorded with frame rates of 30 fps and 24 fps, respectively. In this case, when the cross-correlation based on the above sound data is calculated and the background moving image is shot 0.5 seconds earlier than the foreground moving image, the background moving image is set to 0.5 to match the background moving image and the starting frame of the foreground moving image. Time synchronization is realized by offsetting by 15 seconds, that is, (30 × 0.5 =) 15 frames. Therefore, the video frames at 1-second intervals after synchronization take into account the frames of the respective videos, the frame count of the background video is 15, 45, 75, 105, etc., and the frame count of the foreground video is 0, 24, 48, 72... Here, taking into account the deviation of the moving image start frames of the foreground moving image and the background moving image, the frame of the background moving image at time ti is the equirectangular projection image ECi, and the frame of the foreground moving image at the same time ti is the plane image Pi. . That is, the equirectangular projection image ECi and the planar image Pi are one frame of a moving image obtained by shooting at the same time. Note that the cross-correlation calculation method is not an essential part of the present embodiment as long as a known technique is used, and detailed description thereof is omitted. As described above, the synchronization between the images is not an essential part of the present embodiment as long as a known technique is used, and detailed description thereof is omitted.

  The extraction unit 550 extracts feature points based on the local features of the images ECi and Pi. A local feature is a pattern or structure found in an image, such as an edge or a blob. A feature value is a numerical value of a local feature. In the present embodiment, the extraction unit 550 extracts each feature point from different images. The two images used by the extraction unit 550 may have different projection methods as long as the distortion is not significantly large. For example, the extraction unit 550 includes, between the rectangular equirectangular projection image ECi obtained by the equirectangular projection method and the rectangular planar image Pi obtained by the perspective projection method, and the projection image Pi. It is used between the peripheral region image PIi after being converted by the method conversion unit 556. The equirectangular projection method is an example of the first projection method, and the perspective projection method is an example of the second projection method. The equirectangular projection image is an example of a first projection image, and the planar image is an example of a second projection image.

  The first corresponding region calculation unit 552 first obtains each feature amount fv1 based on the plurality of feature points fp1 in the equirectangular projection image ECi, and each feature amount based on the plurality of feature points fp2 in the planar image Pi. Find fv2. Several methods have been proposed as a description method of the feature amount. In the present embodiment, it is desirable that the feature amount is invariable or robust to scale and rotation. The first corresponding area calculation unit 552 is based on the similarity between the feature quantity fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature quantity fv2 of the feature points fp2 in the planar image Pi. By calculating the corresponding points between the images, and calculating the homography corresponding to the planar image P in the equirectangular projection image EC from the calculated relationship between the corresponding points, and using this homography for conversion, A first homography transformation is performed. As a result, the first corresponding area calculation unit 552 calculates the first corresponding area CA1. In this case, a quadrangle (rectangular) center point CP1 composed of the four vertices of the planar image Pi is converted into a gazing point GP1 in the equirectangular projection image EC by the first homography conversion.

  If the vertex coordinates of the four vertices of the planar image Pi are p1 = (x1, y1), p2 = (x2, y2), p3 = (x3, y3), and p4 = (x4, y4), the first The corresponding region calculation unit 552 can determine the center point CP1 (x, y) based on (Expression 2) shown below.

  In FIG. 19, the image shape of the planar image Pi is a rectangle, but the center coordinates can be calculated even for partial images of various rectangles such as a square, a trapezoid, and a rhombus by using the intersections of diagonal lines. When the image shape of the planar image Pi is limited to a rectangle or a square, the midpoint of the diagonal line may be used as the center coordinate PC of the partial image in order to omit the calculation. (Equation 3) in the case of calculating the midpoint of the diagonal line P1P3 is shown below.

  The gazing point specifying unit 554 specifies a point (referred to as “gazing point” in the present embodiment) on the equirectangular projection image ECi where the center point CP1 of the planar image Pi is located after the first homography transformation.

  By the way, since the coordinates of the gazing point GP1 are coordinates on the equirectangular projection image EC, it is convenient to convert them into expressions of latitude and longitude. Specifically, the vertical direction of the equirectangular projection image ECi is expressed as latitude coordinates from −90 degrees (−0.5π) to +90 degrees (+ 0.5π), and the horizontal direction from −180 degrees (−π). Expressed as longitude coordinates of +180 degrees (+ π). In this way, pixel position coordinates corresponding to the image size of the equirectangular projection image ECi can be calculated from the latitude / longitude coordinates.

  The projection method conversion unit 556 generates the peripheral region image PIi by converting the peripheral region PA around the gazing point GP1 in the equirectangular projection image ECi into the same perspective projection method as that of the planar image P. In this case, the point after the gazing point GP1 is converted is set as the center point CP2, and it is specified when the same angle of view as the diagonal angle of view α of the planar image Pi is set as the vertical angle of view (or horizontal angle of view). The peripheral area PA of the projective transformation source is specified so that a possible square peripheral area image PIi can be created as a result. This will be described in more detail below.

(Projection method conversion)
First, the conversion of the projection method will be described. As described with reference to FIGS. 3 to 5, the omnidirectional image CE is created by covering the solid sphere CS with the equirectangular projection image EC. Therefore, each pixel data of the equirectangular projection image EC can correspond to each pixel data on the surface of the solid sphere CS of the three-dimensional omnidirectional image. Therefore, the conversion formula by the projection system conversion unit 556 expresses the coordinates in the equirectangular projection image ECi as (latitude, longitude) = (ea, aa), and the coordinates on the three-dimensional solid sphere CS are orthogonal coordinates (x , Y, z), it can be expressed by the following (formula 4).

(x, y, z) = (cos (ea) × cos (aa), cos (ea) × sin (aa), sin (ea)) (Equation 4)
However, the radius of the solid sphere CS at this time is 1.

  On the other hand, the planar image Pi, which is a transmission projection image, is a two-dimensional image. When this is expressed by two-dimensional polar coordinates (radial radius, declination) = (r, a), the radial radius r is a diagonal field angle. Corresponding to α, the possible range is 0 ≦ r ≦ tan (diagonal angle of view / 2). Further, when the planar image P is represented by a two-dimensional orthogonal coordinate system (u, v), the conversion relationship with polar coordinates (radial radius, declination) = (r, a) is represented by the following (formula 5). Can do.

u = r × cos (a), v = r × sin (a) (Equation 5)
Next, let us consider that this (Formula 5) corresponds to three-dimensional coordinates (radial radius, polar angle, azimuth angle). Since only the surface of the solid sphere CS is considered now, the moving radius in the three-dimensional polar coordinate is “1”. Further, the projection for perspective projection transformation of the equirectangular projection image EC attached to the surface of the solid sphere CS is considered to be the above-described two-dimensional polar coordinate (radial radius, declination) when the virtual camera IC is located at the center of the solid sphere CS. ) = (R, a) can be expressed by the following (Expression 6) and (Expression 7).

r = tan (polar angle) (Formula 6)
a = Azimuth angle (Formula 7)
If the polar angle is t, then t = arctan (r), so the three-dimensional polar coordinates (radial radius, polar angle, azimuth) are (radial radius, polar angle, azimuth) = (1, arctan ( r), a).

  A conversion formula for converting from the three-dimensional polar coordinate to the orthogonal coordinate system (x, y, z) can be expressed by the following (Formula 8).

(x, y, z) = (sin (t) x cos (a), sin (t) x sin (a), cos (t)) (Equation 8)
According to the above (Equation 8), the equirectangular projection image ECi by the equirectangular projection method and the planar image Pi by the perspective projection method can be mutually converted. That is, by using the radius r corresponding to the diagonal angle of view α of the planar image P to be created, the conversion map indicating which coordinate of the equirectangular projection image EC corresponds to each pixel of the planar image P. The coordinates can be calculated, and the peripheral area image PIi, which is a perspective projection image, can be created from the equirectangular projection image ECi based on the conversion map coordinates.

  By the way, in the projection method conversion, the position where the (latitude, longitude) of the equirectangular projection image EC is (90 °, 0 °) becomes the center point CP2 of the peripheral region image PI which is a perspective projection image. Shows the conversion. Therefore, when performing perspective projection conversion with an arbitrary point of the equirectangular projection image ECi as a gazing point, the coordinates of the gazing point (latitude, Coordinate rotation may be performed such that (longitude) is arranged at a position of (90 °, 0 °).

  Since the conversion formula relating to the rotation of the solid sphere CS is a general coordinate rotation formula, description thereof is omitted.

(Identification of surrounding area image)
Next, a method for specifying the area of the peripheral area image PIi will be described with reference to FIG. FIG. 20 is a conceptual diagram when specifying the peripheral region image.

  When the first corresponding area calculation unit 552 determines the correspondence relationship between the planar image Pi and the peripheral area image PIi, it is desirable that the second corresponding area CA2 is included as widely as possible in the peripheral area image PIi. Therefore, if the peripheral area image PIi is set to a wide area, a situation in which the second corresponding area CA2 is not included does not occur. However, if the peripheral area image PIi is set to an excessively large area, the number of pixels for which the similarity is calculated increases, and thus processing takes time. Therefore, it is preferable that the area of the peripheral area image PIi is as small as possible within the range including the second corresponding area CA2. Therefore, in the present embodiment, the peripheral area image PI is specified by the following method.

  In the present embodiment, the 35 mm equivalent focal length of the planar image is used for specifying the peripheral area image PIi, and this is acquired from Exif data recorded at the time of photographing. Since the 35 mm equivalent focal length is a focal length based on a so-called 24 mm × 36 mm film size, the following calculation formulas (Equation 9) and (Equation 10) correspond to the diagonal of the film and the focal length. The angle of view can be calculated.

Film diagonal = sqrt (24 * 24 + 36 * 36) (Equation 9)
Composite image angle of view / 2 = arctan ((film diagonal / 2) / 35mm equivalent focal length of composite image) ... (Equation 10)
By the way, the image covering this angle of view is circular, but since the actual image sensor (film) is rectangular, it is a rectangular image inscribed in the circle. In the present embodiment, the vertical field angle α of the peripheral area image PIi is set to be the same as the diagonal field angle α of the planar image Pi. As a result, the peripheral area image PIi shown in FIG. 20B becomes a square circumscribing a circle covering the diagonal angle of view α of the planar image Pi shown in FIG. The angle α can be calculated from the diagonal of the square and the focal length of the planar image P, as shown in (Expression 11) and (Expression 12) below.
Square diagonal = sqrt (film diagonal * film diagonal + film diagonal * film diagonal) ... (Formula 11)
Vertical angle of view α / 2 = arctan ((square square / 2) / 35mm equivalent focal length of planar image)) ... (Formula 12)
By performing projective transformation at such a vertical angle of view α, the peripheral region image PIi that can cover the image at the diagonal angle of view α of the planar image Pi as wide as possible with the gazing point as the center, and the vertical angle of view α is not too large. (Perspective projection image) can be created.

(Calculation of location information)
Subsequently, returning to FIG. 15, the second corresponding region calculation unit 558 calculates the feature amount fv3 of the plurality of feature points fp2 in the planar image Pi and the plurality of feature points fp3 in the peripheral region image PIi. Corresponding points between images are calculated based on the calculated similarities of the respective feature quantities fv2 and fv3, and a homography corresponding to the planar image P is calculated in the peripheral region image PIi from the calculated relationship between the corresponding points between the images. The second homography conversion is performed by using this homography for the conversion. As a result, the second corresponding area calculation unit 558 calculates the second corresponding area CA2.

  Note that before the first homography conversion, the image size of at least one of the planar image Pi and the equirectangular projection image ECi may be resized in order to advance the calculation time of the first homography. For example, if the planar image Pi is 40 million pixels and the equirectangular projection image ECi is 30 million pixels, the planar image Pi is resized to 30 million pixels, or both images are adjusted to 10 million pixels. Or resize it. Similarly, the image size of at least one of the planar image Pi and the peripheral area image PIi may be resized before the second homography calculation.

Here, in general, homography refers to projecting one plane onto another plane using projective transformation. In addition, the outline of the homography in the present embodiment is shown below. The first homography transformation is shown as a transformation matrix as representing the projection relationship between the equirectangular projection image ECi and the plane image Pi, and the homography calculated by the homography calculation process is set to the coordinates in the plane image Pi. By multiplying the transformation matrix, the corresponding coordinates on the equirectangular projection image ECi (the omnidirectional image CE) can be calculated. Specifically, a gazing point GP1 in the equirectangular projection image EC can be calculated from a square (rectangular) center point CP1 composed of four vertices of the planar image Pi by the first homography transformation. Next, the second homography transformation is shown as a transformation matrix as representing a projection relationship between the peripheral area image PIi obtained by converting the equirectangular projection image ECi by a predetermined projection method and the planar image Pi. By multiplying the coordinates in Pi by the matrix of the projective transformation by homography calculated in the homography calculation process, the corresponding coordinates on the peripheral area image PIi can be calculated.

  FIG. 21A is a conceptual diagram showing an initial shape, FIG. 21B is a conceptual diagram showing a peripheral region image and a second corresponding region, and FIG. 21C is a concept of an initial shape that is divided into regions. FIG.

  The initial setting shape generation unit 559 is an equirectangular projection method, has a projective transformation (homography) relationship with the shape of the planar image Pi, and has an initial setting shape DF determined by a predetermined angle of view and the projection method. Is generated. Here, by appropriately setting the initial setting shape DF, it is possible to suppress display displacement when display is performed by rotation, movement, or scaling. The planar image P1 indicates the first frame of the moving image, but may be an arbitrary frame and indicates a specific frame in the moving image. The initial shape generation unit 559 generates the initial shape DF, but this generation may be preset or newly generated. That is, the initial shape may be set to a predetermined shape based on a predetermined angle of view and a projection method, for example, a quadrangle or a circle. In addition, although it is predetermined as in the present embodiment, in other cases, an angle of view, a projection method, and a shape may be generated according to the shape of the planar image Pi. Note that it is well known in the technical field that the angle of view is obtained from a focal length described later. Specifically, in the planar image Pi, an initial setting shape is first generated from the angle of view, the projection method, and the shape according to the shape of the planar image P1, and the initial setting shape for the planar image Pn from the subsequent planar image P2 is The initial setting shape generated for the planar image P1 is used. The initial shape is provided for proper display during display, and the position and shape for composite display are determined using the initial shape and reference shape conversion information described later. be able to.

  As shown in FIG. 21A, the default shape DF is treated as the same projection method as that of the foreground moving image. Here, in consideration of superimposing an image taken with a general digital camera as the foreground moving image. The initial shape DF is determined in advance as a perspective projection method.

  The initial shape DF is a rectangle inscribed in a circle centered on the center of the rectangle as shown in FIG. 21A. Here, for the sake of simplicity, the vertical and horizontal sides of the rectangle are the X axis and the Y axis, respectively. Parallel to the center and the center as the origin. The diagonal field angle can be arbitrarily determined within a range larger than 0 degree and smaller than 180 degrees. However, if an extreme value is specified, a calculation error may increase. . Further, the angle of view of the recorded image that is used when the moving image is actually recorded may be employed in the moving image that becomes the foreground moving image.

  Next, the reference shape conversion information calculation unit 568 uses the information (parameter) when performing projective transformation (homography) from the planar image Pi to the second corresponding area CA2, and performs initial processing on the second corresponding area CA2. Information for converting the set shape DF is calculated. This calculated information is reference shape conversion information for converting the reference shape BF by at least one of rotation and scaling. The reference shape conversion information regarding this rotation is “reference shape rotation information” described later, and the reference shape conversion information regarding zooming is “reference shape scaling information” described later. In other words, this reference shape conversion information is information used to convert the position and shape of the initial shape DF, and specifically includes information meaning three of rotation, scaling, and movement. Yes.

  Further, the reference shape conversion information calculation unit 568 converts the center point CP2 of the second corresponding area CA2 to the equirectangular projection method by the projection method conversion, thereby corresponding points CP3 corresponding on the equirectangular projection image ECi. The position coordinates of are calculated. The reference shape conversion information regarding the position coordinates is “reference shape movement information” to be described later, and is information for converting the position of the reference shape BF by movement.

  Here, the processing of the reference shape conversion information calculation unit 568 will be described in detail with reference to FIG. FIG. 21B is a conceptual diagram showing a peripheral area image and a second corresponding area.

  FIG. 21B shows the peripheral area image PIi and the second corresponding area CA2. As shown in FIG. 20, the vertical field angle α of the peripheral area image PIi is set to be the same as the diagonal field angle α of the planar image Pi. Then, there is a relationship called second homography transformation between the second corresponding area CA2 and the planar image Pi. By the way, since the homography in this case is conversion for projecting a plane onto a plane, it is considered that the initial set shape DF and the second corresponding area CA2 are also expressed by a projection relationship by homography. Depending on how the four vertices of the second corresponding area CA2 are taken, it is not possible to completely describe by homography from the plane expressed by the default shape DF, and the shift of the four points is minimized by the least square method or the like. Such an approximation is performed. The calculation method of homography is omitted because it is a known calculation method. For example, when using OpenCV (Open Source Computer Vision Library), there is a function that calculates a homography matrix using four pairs of corresponding points as inputs. Also good.

  Further, the reference shape conversion information calculation unit 568 calculates “reference shape rotation information” and “reference shape scaling information” for converting the reference shape BF from the calculated homography. A known homography decomposition method can be used as a method for obtaining rotation information and scaling information from the calculated homography. Details of the method of decomposing the homography into a rotation matrix and a translation vector are disclosed in, for example, the following documents, and thus description thereof is omitted.

[Non-patent literature]
Zhang, Z. “A Flexible New Technique for Camera Calibration.” IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 22, No. 11, 2000, pp. 1330-1334.
Here, the reference shape rotation information is indicated by Euler angles with respect to the respective axes in the three-dimensional model space. Further, the reference shape movement information is specified by the gazing point GP1 on the equirectangular projection image EC.

  Next, returning to FIG. 15, the region dividing unit 560 divides the region of the initial shape DF into a plurality of lattice regions. Here, a method of dividing the reference shape region into a plurality of lattice regions will be described with reference to FIGS. FIG. 21C is a conceptual diagram of a reference shape divided into regions.

  As illustrated in FIG. 21A, the region dividing unit 560 uses the four vertices of the reference shape generated by giving the diagonal angle of view and the aspect ratio as described in the generation of the reference shape DF. The rectangle formed is divided into a plurality of lattice regions as shown in FIG. For example, it is divided equally into 8 in the horizontal direction and 8 in the vertical direction.

  Next, a specific method for dividing a plurality of lattice regions will be described.

  First, a calculation formula for equally dividing the reference shape region is shown. When the two points are A (X1, Y1) and B (X2, Y2) and the line segment between the two points is divided into n pieces at equal intervals, the coordinates of the point Pm corresponding to the mth point from A are as follows: Calculated by (Equation 13).

Pm = (X1 + (X2−X1) × m / n, Y1 + (Y2−Y1) × m / n) (Equation 13)
Since the coordinates obtained by equally dividing the line segment can be calculated by the above (Formula 13), the coordinates obtained by dividing the upper side and the lower side of the rectangle may be obtained, and the line segment formed by the divided coordinates may be further divided. As shown in FIG. 21A, when the upper left, upper right, lower right, and lower left coordinates of the rectangular initial setting shape DF are TL, TR, BR, and BL, respectively, a line segment TL-TR. And the coordinate which divided | segmented line segment BR-BL into 8 equally is calculated | required. Next, in each of the coordinates divided from the 0th to the 8th, coordinates obtained by equally dividing the line segment composed of coordinates corresponding to the same order are obtained. Thereby, coordinates for dividing the quadrangular area into 8 × 8 small areas can be obtained.

  Next, returning to FIG. 15, the projection method inverse transformation unit 562 converts the coordinates of the initial shape DF divided by the region division unit 560 into the same normal projection image ECi as the equirectangular projection image ECi which is the same projection method as the background moving image. By inversely converting to the distance cylindrical projection method, the corresponding points of the grid points of the reference shape are calculated in the equirectangular projection image ECi.

FIG. 22A is calculated when each grid point of the initial setting shape DF shown in FIG. 21C is arranged at the center (gazing point (latitude 0, longitude 0)) of the equirectangular projection image ECi. It is the conceptual diagram which showed each lattice point performed. As described above in the description of the conversion of the projection method, the correspondence between the images having different projection methods can be calculated by mapping each of them to a three-dimensional solid sphere. In FIG. 22A, TL coordinates (LO _00,00 , LA _00,00 ) are shown as an example of coordinates.

  Here, the reference shape BF is generated by giving an angle of view and an aspect ratio as a perspective projection method, but in order to describe the relationship with the equirectangular projection image ECi, information on where the reference shape BF is arranged Is required. Therefore, it is considered to define coordinates for arranging the reference shape BF using latitude / longitude coordinates on the equirectangular projection image ECi. That is, it can be expressed as a gazing point as to where it is facing from the center of the solid sphere CS.

  FIG. 22B is a conceptual diagram when the same reference shape BF as FIG. 22A is arranged at a position of −45 degrees latitude and 90 degrees longitude by changing the gazing point. Since the equirectangular projection is a projection system in which the horizontal direction is extended from the vicinity of the equator toward the polar direction, the reference shape is like a fan shape as shown in the figure.

  As shown in FIG. 5 and FIG. 6, information on where the reference shape BF is arranged is indicated as “reference shape movement information” in the present embodiment, and is expressed as latitude / longitude coordinates on the equirectangular projection image. FIG. 22A can be considered that the reference shape BF shown in FIG. 22B has been moved according to the reference shape movement information. Here, the corresponding point CP3 on the equirectangular projection image ECi is obtained by inversely transforming the center point CP2 of the second corresponding area CA2 by the projection method, and this coordinate position is set as “reference shape movement information”. By using the “reference shape conversion information” including the calculated “reference shape rotation information” and “reference shape scaling information”, the reference shape BF is rotated and scaled, and the rotated and scaled reference The shape can be moved to the position indicated by the gazing point GP1 on the equirectangular projection. That is, the corresponding area in the first projection image corresponding to the second corresponding area is obtained by inversely converting the reference shape conversion information and the coordinates obtained by dividing the reference shape into the first projection method. It can be expressed by the obtained position parameter. Note that the grid point is an example of a plurality of points.

  By the way, the reference shape movement information is similar to moving the visual axis of the virtual camera IC by panning (longitude direction) and tilting (latitude direction) the virtual camera IC at the center of the solid sphere CS. Pan / tilt information ”can also be indicated.

  As a result, the superimposing unit 55b described later uses the reference shape conversion information to rotate, scale, and move the reference shape BF of the planar image Pi, so that the whole sky created by the equirectangular projection image ECi is created. It can be displayed superimposed on the spherical image CE. As described above, the positional relationship between the equirectangular projection image ECi and the planar image Pi can be calculated by creating the reference position parameter, the moving image frame count value (see FIG. 15), and the reference shape conversion information.

  Subsequently, returning to FIG. 15, the superimposed display metadata creation unit 570 creates superimposed display metadata as shown in FIG. 17.

(Superimposed display metadata)
Here, the data structure of the superimposed display metadata will be described with reference to FIG. FIG. 17 shows the data structure of the superimposed display metadata.

  As shown in FIG. 17, the superimposed display metadata includes equirectangular projection image information, planar image information, reference shape information, a plurality of reference shape conversion information 1 to N (N: positive number), and metadata. Consists of creation information.

  Among these, the equirectangular projection image information is metadata information attached to the moving image data photographed by the special photographing apparatus 1. The equirectangular projection image information includes image identification information and attached information. The image identification information in the equirectangular projection image information is image identification information for identifying the equirectangular projection image. In FIG. 17, the file name of the image is shown as an example of the image identification information in the equirectangular projection image information, but an image ID for uniquely identifying the image may be used.

  The attached information in the equirectangular projection image information is related information attached to the equirectangular projection image information. In FIG. 17, posture correction information (Pitch, Yaw, Roll) of equirectangular projection image data obtained when captured by the special imaging device 1 is shown as an example of attached information. This posture correction information may be stored in the Exif (Exchangeable image file format) standardized as the image recording format of the special imaging device 1, and various formats provided by GPano (Google Photo Sphere schema) It may be stored in. An omnidirectional image has a feature that if it is taken at the same position, an image of 360 ° omnidirectional image can be taken even if the posture is different. The display position is not determined unless the center of the image is specified (gaze point). Therefore, generally, the omnidirectional image CE is corrected and displayed so that the zenith is directly above the photographer. As a result, a natural display in which the horizon is straightly corrected is possible. Note that since the image handled in the present embodiment is a moving image, posture information may be stored for each frame of the moving image and corrected for each frame.

  Next, the planar image information is information based on moving image data photographed by the general photographing device 3. The planar image information includes image identification information and attached information. The image identification information in the planar image information is image identification information for identifying the planar image P. In FIG. 17, the file name of the image is shown as an example of the image identification information, but an image ID for identifying the image may be used.

  The attached information in the planar image information is related information attached to the planar image information. In FIG. 17, the value of the 35 mm equivalent focal length is shown as an example of the attached information in the planar image information. The value of the 35 mm equivalent focal length is not essential for displaying the celestial sphere image CE by superimposing the planar image P, but serves as reference information for determining the angle of view to be displayed when superimposing display is performed. Take as an example. In addition, since the focal length of a moving image changes when zoom processing is performed, the focal length corresponding to each frame of the moving image may be recorded as attached information.

  Next, the reference shape information includes the area division information and the position of the lattice point (reference position parameter) of each lattice area as information constituting the reference shape. Among these, the region division information indicates the number of divisions in the horizontal (longitude) direction and the number of divisions in the vertical (latitude) direction when dividing the reference shape BF into a plurality of lattice regions.

  The reference position parameter is used together with reference shape conversion information 1 to N described later, and a planar image Pi that can be extracted by moving picture frame specifying information in the reference shape conversion information i is divided into a plurality of grid-like regions. Is a parameter for calculating vertex mapping information indicating in which position in the equirectangular projection image ECi that can be extracted from the omnidirectional video CE by the reference shape conversion information i. is there.

  Next, the reference shape conversion information includes reference shape rotation information, reference shape scaling information, reference shape movement information, and moving image frame specifying information. The reference shape rotation information, the reference shape scaling information, and the reference shape movement information are as described above.

  The moving image frame specifying information is generated by the creating moving image frame extracting unit 548, and the frame is determined from the moving image specified by the image identification information of the equirectangular projection image information and the moving image specified by the image identification information of the planar image information. Used to extract each image. Each information on rotation, scaling, and movement of the reference shape is obtained by rotating, scaling, or moving the corresponding position on the equirectangular projection image indicated by the position parameter of the reference shape information, so that the planar image Pi is latticed. The above-described vertex mapping indicating where each lattice point when divided into a plurality of regions is arranged in an equirectangular projection image ECi (one frame image extracted from the omnidirectional video CE) Convert to information. The reference shape conversion information calculation unit 568 does not calculate the reference shape conversion information for all the frames of the moving image, but the reference shape conversion information for the frame of the moving image (planar image Pi) appropriately sampled from the foreground moving image. In this example, reference shape conversion information of 1 to N is provided. Further, the reference shape conversion information calculation unit 568 samples the frames of the foreground moving image finely on the time axis when the change between frames is large in the foreground moving image, and the second moving image when the change between frames is small. Roughly sample the frames.

  Next, the metadata creation information indicates version information indicating the version of the superimposed display metadata. By configuring the superimposed display metadata as described above, the data size of the parameters required for calculating the superimposed position in the image related to each frame of the moving image is reduced, and the parameters are set for all the frames of the moving image. By recording the reference shape conversion information for an appropriately sampled frame instead of having it, the required parameter data amount can be dramatically reduced. Then, it is considered that the parameter is provided as a parameter set that can reduce the arithmetic processing for the image generation processing as compared with the conventional processing when performing superimposed display performed by a viewer or the like. As a specific example, consider an arithmetic process for calculating data that can be handled by OpenGL, a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. ing. In this way, since it is recorded as metadata that can be handled by an external program, the external program reads the reference shape conversion information of time series rotation, movement, and scaling, and uses it by interpolating it continuously. The effect that the superimposition processing can be realized by real-time processing is achieved.

{Functional structure of superimposition unit}
Subsequently, the functional configuration of the superimposing unit 55b will be described with reference to FIG. The superimposing unit 55b includes a reproduction time management unit 571, a reproduction moving image frame extraction unit 572, an interpolation unit 574, a conversion position parameter calculation unit 576, a shape conversion unit 578, a correction parameter creation unit 580, a pasting region creation unit 582, and a correction unit 584. , An image creating unit 586, an image superimposing unit 588, and a projective transformation unit 590.

  Among these, the playback time management unit 571 manages the video playback time serving as a reference, and outputs the playback time (or video frame count value) that is the current time. For example, when it is desired to reproduce a moving image at 30 fps, the reproduction time management unit 571 increases the time by 1/30 second per second and outputs it 30 times. Here, the synchronization of the background moving image and the foreground moving image will be briefly described using an example. Any image may be used, but a reference image can be specified, and synchronization can be synchronized from the reference image using a temporal difference between the other images. For example, if the recording start of the foreground moving image (planar image P) is 0.5 seconds earlier as the first image of the background moving image (positive-distance cylindrical projection image EC), the temporal difference is -0.5 seconds, conversely If the recording start of a planar video is delayed by 0.5 seconds, the time difference can be set to +0.5 seconds and recorded in the metadata. An offset of the number of frames can be calculated based on this temporal difference, and can be reproduced and displayed in synchronization. In addition, when playing back and displaying, in accordance with the later recording time, skip the reading of the video and display it synchronously, or from the image where the temporal difference between the background video and the foreground video is zero You may make it display.

  Based on the time managed by the reproduction time management unit 571, the reproduction moving image frame extraction unit 572 becomes one moving image that becomes a background moving image (an equirectangular projection image EC) and a foreground moving image (a planar image P). A frame corresponding to each designated time is extracted from each of the other moving images.

  The interpolation unit 574 reads each reference shape conversion information generated by the metadata generation unit 55a in accordance with the frame reproduction time (or moving image frame count value) in the moving image by the reproduction time management unit 571, and if necessary, By performing interpolation (interpolation), a shape conversion parameter is calculated for each reproduction time of a moving image frame. The form of this shape conversion parameter is the same as the above-described reference shape conversion information, but the value indicates the interpolation result. The interpolation method is obtained by reading the reference shape conversion information before and after the reproduction time closest to the reproduction time, for example, by linear interpolation. In the metadata creation unit 55a, for example, the creation moving image frame extraction unit 548 extracts a moving image frame at 1 fps, so that the superimposed display metadata creation unit 570 creates one reference shape conversion information per second. In this case, when the reproduction time management unit 571 outputs 30 frames per second, the interpolation unit 574 performs interpolation for 29 frames.

  The conversion position parameter calculation unit 576 uses the shape conversion parameters obtained by the interpolation unit 574 to convert the conversion position parameter that is the result of converting the reference shape information (reference position parameter) generated by the metadata generation unit 55a. calculate. This calculation is performed for each playback time of each frame of the moving image.

  By the way, although the conversion position parameter has been calculated, when superimposing display is performed as it is, when the equirectangular projection image EC and the planar image P are greatly different in brightness and color, an unnatural superimposition display may occur. . Therefore, the shape conversion unit 578 and the correction parameter creation unit 580 described below play a role of preventing unnatural superimposition display even when the brightness and color tone are greatly different.

  The shape conversion unit 578 converts the region on the equirectangular projection image EC specified by the conversion position parameter calculated by the conversion position parameter calculation unit 576 into the same rectangular shape as the planar image P, thereby Corresponding area CA3 is created.

As shown in FIG. 26A, the correction parameter creation unit 580 divides the third corresponding area CA3 based on the area division number information of the superimposed display metadata as in the process of the area division unit 560. Thus, each lattice area LA3 is created. Further, as illustrated in FIG. 26B, the correction parameter creation unit 580 divides the planar image P based on the area division number information of the superimposed display metadata as in the process of the area division unit 560. Thus, each lattice area LA0 is created. In FIG. 26B, (8 × 8 =) 64 lattice areas LA0 are created. Furthermore, the correction parameter creation unit 580 has the same shape as that of each lattice region LA3 with respect to the brightness value and the color value of each lattice region LA2 ′ in the third corresponding region CA3 after being converted into the same shape. A correction parameter (an example of correction information) for matching the brightness value and the color value of each lattice area LA0 is created. Specifically, the correction parameter creation unit 580 has brightness values and color values (R, G, B) of all the pixels constituting the four grid areas having one grid area LA0 as a grid point in common. The average value a = (R _ave , G _ave , B _ave ) is calculated, and the brightness values of all the pixels constituting the four lattice areas having one common corner point of each lattice area LA3 ′. The average value a ′ = (R ′ _ave , G ′ _ave , B ′ _ave ) of the color values (R ′, G ′, B ′) is calculated. FIG. 27 shows these four lattice regions (LA3a ′, LA3b ′, LA3c ′, LA3d ′) and a common lattice point LP3 ′ that is one corner. When one point of each lattice area LA0 and one point of each lattice area LA3 are the four corners of the third corresponding area CA3 and the third corresponding area CA3, the correction parameter creation unit 580 An average value a and an average value a ′ of the brightness value and the color value are calculated. When one point of each of the lattice areas LA0 and one point of each of the lattice areas LA3 is a point on the outer periphery of the third corresponding area CA3, the correction parameter creation unit 566 calculates brightness values and colors from the two inner lattice areas. An average value a and an average value a ′ are calculated. In this embodiment, the correction parameter is gain data for correcting the brightness value and the color value of the planar image P. Therefore, as shown in the following (Expression 14), the average value a ′ Is divided by the average value a to obtain the correction parameter Pa.
Pa = a '/ a (Formula 14)
As a result, in the superimposed display described later, the gain value indicated by the correction parameter is multiplied for each lattice area LA0, so that the color and brightness of the planar image P become the equirectangular projection image EC (the omnidirectional image CE). ) And the pixel value are close to each other, and the display can be superimposed and displayed without a sense of incongruity. The correction parameter is not only calculated from the average value, but may be calculated using a median value, a mode value, or the like instead of or in addition to the average value.

  In this embodiment, the pixel values (R, G, B) are used for calculating the correction values for the brightness value and the color value. However, the YUV format for luminance and color difference signals, the JPEG sYCC (YCbCr) format, etc. Using the luminance value and color difference value in, the average value of the luminance value and color difference value of all the pixels constituting the grid area is obtained in the same way, and the average value is divided to correct the correction parameter in the superimposed display described later It is good. The method of converting RGB values to YUV, sYCC (YCbCr) is well known and will not be described in detail. However, using (Equation 15) as a reference, JPEG compressed image format JFIF (JPEG file interchange format) format is used. Here is an example of converting from RGB to YCbCr.

  Based on the correction parameter calculated by the correction parameter creation unit 580, the correction unit 584 performs correction to match the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC. Thus, the corrected image C is created. Note that the correction unit 584 does not necessarily have to correct brightness and color. In addition, when the correction parameter is corrected, the brightness may be corrected or the color may not be corrected.

  On the other hand, the pasting region creation unit 582 is a region portion (hereinafter, referred to as “partial solid sphere”) to which the planar image P is pasted in the virtual solid sphere CS based on the conversion position parameter calculated by the conversion position parameter calculation unit 576. Create PS.

  The image creating unit 586 creates the superimposed image S by pasting the planar image P (or the corrected image C after correcting the planar image P) to the partial solid sphere PS. The image creation unit 586 creates mask data M based on the region of the partial solid sphere PS. Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC to the solid sphere CS. The mask data M is used to gradually bring the brightness and color around the boundary when the superimposed image S is superimposed on the omnidirectional image CE from the inner superimposed image S side to the outer omnidirectional image CE side. The transmittance around the mask gradually increases from the superimposed image S to the omnidirectional image CE from the inside to the outside. Thereby, even if the superimposition image S is superimposed on the omnidirectional image CE, it is possible to prevent the superimposition from being recognized as much as possible. Note that the creation of the mask data M is not essential when the brightness around the boundary when the superimposed image S is superimposed on the omnidirectional image CE is not gradually changed.

  The image superimposing unit 588 superimposes the superimposed image S and the mask data M on the omnidirectional image CE. Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  As shown in FIG. 7, the projective transformation unit 590 generates a superimposed image S based on a predetermined line-of-sight direction of the virtual camera IC (a center point CP of the predetermined area T) and the angle of view α of the predetermined area T. Projective transformation is performed so that the predetermined area T in the omnidirectional image CE in a state where is superimposed can be seen on the display. Projection conversion unit 590 also performs processing for matching predetermined area T to the resolution of the display area on the display when performing projective conversion. Specifically, when the resolution of the predetermined area T is smaller than the resolution of the display area of the display, the projection conversion unit 590 enlarges the predetermined area T so as to match the display area of the display. On the other hand, when the resolution of the predetermined area T is larger than the resolution of the display area of the display, the projective transformation unit 590 reduces the predetermined area T so as to match the display area of the display. Thereby, the display control unit can display the predetermined area image Q indicating the predetermined area T over the entire display area of the display.

<< Processing or Operation of Embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIGS. First, a photographing method executed by the photographing system will be described with reference to FIG. FIG. 18 is a sequence diagram illustrating the photographing method according to the present embodiment. In the following description, a case where a subject, landscape, or the like is photographed will be described. However, ambient sounds may be recorded by the sound collection unit 14 simultaneously with photographing.

  As illustrated in FIG. 18, the reception unit 52 of the smartphone 5 receives the start of cooperative shooting from the user (step S11). In this case, as shown in FIG. 14B, the display control unit 56 causes the display 517 to display the cooperative photographing apparatus setting screen. On this screen, a radio button for designating a main imaging device for linked shooting and a check box for designating (selecting) a sub imaging device for linked shooting are displayed for each imaging device. Yes. Further, for each photographing apparatus, the name of the photographing apparatus and the reception intensity of the radio wave are displayed. Then, when the user designates (selects) a desired photographing apparatus as a main and a sub and presses a “confirm” button, the accepting unit accepts the start of cooperative photographing. Since there may be a plurality of sub photographing apparatuses, a plurality of photographing apparatuses can be designated (selected) by using check boxes.

  And the short distance communication part 58 of the smart phone 5 transmits the imaging | photography start request information for requesting a photography start with respect to the short distance communication part 38 of the general imaging device 3 (step S12). Further, the short-range communication unit 58 of the smartphone 5 transmits shooting start request information for requesting the start of shooting to the short-range communication unit 18 of the special imaging device 1 (step S13).

  Next, the general photographing apparatus 3 starts photographing (step S14). In this photographing process, the image capturing unit 33 captures photographed image data (here, planar image data) which is frame data constituting a moving image by capturing an object, a landscape, and the like, and the storage / reading unit 39 stores the storage unit. This is a process until the captured image data is stored in 3000. And the short distance communication part 38 of the general imaging device 3 transmits the plane image data obtained by step S14 with respect to the smart phone 5 (step S15). At this time, image identification information for identifying the plane image data to be transmitted, and attached information are also transmitted. Thereby, the near field communication part 58 of the smart phone 5 receives plane image data, image identification information, and attached information.

  On the other hand, the special photographing apparatus 1 starts photographing (step S16). This shooting process is performed by shooting image data (two hemispheres as shown in FIGS. 3A and 3B) that are frame data that constitutes a moving image by the imaging unit 13 shooting a subject, a landscape, or the like. Image / sound processing unit 15 creates a single equirectangular projection image data as shown in FIG. 3C based on the two hemispherical image data, This is processing until the storage / reading unit 19 stores equirectangular projection image data in the storage unit 1000. Then, the short-range communication unit 18 of the special imaging device 1 transmits the equirectangular projection image data obtained in step S16 to the smartphone 5 (step S17). At this time, image identification information for identifying the equirectangular projection image data to be transmitted, and attached information are also transmitted. Thereby, the near field communication part 58 of the smart phone 5 receives equirectangular projection image data, image identification information, and attached information. The equirectangular projection image data may be created by the smartphone 5 instead of being created by the special imaging device 1. In this case, two hemispherical image data, image identification information, and attached information are transmitted from the special imaging device 1 to the smartphone 5.

  Next, the storage / reading unit 59 of the smartphone 5 stores the electronic file of planar image data received in step S15 and the electronic file of equirectangular projection image data received in step S17 in the same electronic folder. And it memorize | stores in the memory | storage part 5000 (step S18).

  Next, the image / sound processing unit 55 of the smartphone 5 uses the superimposition that is used when the planar image P that is a high-definition image is superimposed and displayed on a partial region of the omnidirectional image CE that is a low-definition image. Display metadata is created (step S19). At this time, the storage / reading unit 59 stores the superimposed display metadata in the storage unit 5000.

  Here, the process of creating superimposed display metadata will be described in detail mainly with reference to FIGS. Even if the resolutions of the imaging devices of the general imaging device 3 and the special imaging device 1 are the same, the imaging device of the special imaging device 1 generates an equirectangular projection image that is the basis of the 360 ° omnidirectional image CE. Since all must be covered, the definition per fixed area in the captured image is low.

{Process for creating superimposed display metadata}
First, a description will be given of a process for creating superimposition display metadata for superimposing a high-definition plane image P on the omnidirectional image CE created by the low-definition equirectangular projection image EC and displaying it on the display 517. To do.

  The extraction unit 550 extracts a plurality of feature points fp1 in the rectangular equirectangular projection image EC obtained by the equirectangular projection method and a plurality of feature points fp2 in the rectangular planar image P obtained by the perspective projection method. (Step S110).

  Next, the first corresponding region calculation unit 552 performs the first homography conversion to feature amounts fv1 of the plurality of feature points fp1 in the equirectangular projection image EC and features of the plurality of feature points fp2 in the planar image P. Based on the similarity to the quantity fv2, as shown in FIG. 19, in the equirectangular projection image EC, a rectangular first corresponding area CA1 corresponding to the planar image P is calculated (step S120). More specifically, the first corresponding area calculation unit 552 calculates the feature amounts fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature amounts fv2 of the plurality of feature points fp2 in the planar image P. FIG. 19 shows a first homography transformation obtained by calculating corresponding points between images based on the similarity and obtaining a homography corresponding to the planar image P in the equirectangular projection image EC. As described above, in the equirectangular projection image EC, a quadrangular first corresponding area CA1 corresponding to the planar image P is calculated. This process is a process (provisional decision process) for roughly estimating the corresponding position for the time being, although it is impossible to accurately associate the flat image P having a different projection method with the equirectangular projection image EC.

  Next, the gazing point identification unit 554 identifies a point (gaze point GP1) on the equirectangular projection image EC where the center point CP1 of the planar image P is located after the first homography transformation (step S130).

  Next, the projection method conversion unit 556 makes the result by making the vertical field angle α of the peripheral region image PI the same as the diagonal field angle α of the planar image P as shown in FIG. Thus, the peripheral area PA centered on the gazing point GP1 in the equirectangular projection image EC is converted into the same perspective projection method as that of the planar image P so that the peripheral area image PI can be created (step S140).

  Next, the extraction unit 550 extracts a plurality of feature points fp3 in the peripheral area image PI obtained by the projection method conversion unit 556 (step S150).

  Next, the second corresponding region calculation unit 558 performs the feature amount fv2 of the plurality of feature points fp2 in the planar image P and the feature amount fv3 of the plurality of feature points fp3 in the peripheral region image PI by the second homography transformation. The second corresponding area CA2 having a square shape corresponding to the planar image P is calculated in the peripheral area image PI based on the similarity to (step S160). Note that the planar image P is a high-definition image of 40 million pixels, for example, and is resized to an appropriate size in advance.

  Next, as shown in FIG. 21A, the initial setting shape generation unit 559 generates the initial setting shape DF by determining the rectangular diagonal angle of view β and the rectangular aspect ratio ( Step S170).

  Next, the region dividing unit 560 divides the region of the initial shape DF into a plurality of lattice regions as shown in FIG. 21C (step S180).

  Next, the projection method inverse transform unit 562 converts the coordinates of the initial shape DF divided by the region dividing unit 560 into the equirectangular projection method that is the same as the equirectangular projection image ECi that is the same projection method as the background moving image. By performing inverse transformation, corresponding points of each grid point of the reference shape are calculated in the equirectangular projection image ECi (step S190). By the process of the projection method inverse transform unit 562, reference shape information (reference position parameter) indicating the coordinates of each lattice point of each lattice region is created.

  On the other hand, the reference shape conversion information calculation unit 568 calculates reference shape conversion information for converting the reference shape BF by rotating, scaling (enlarging or reducing), or moving the reference shape BF (step S200).

  Finally, the superimposed display metadata creation unit 570 includes equirectangular projection image information acquired from the special imaging device 1, planar image information acquired from the general imaging device 3, the reference shape information generated in step S190, and step S200. Superimposition display metadata is created based on the reference shape conversion information calculated in step S4 and the moving picture frame count value output from the creation moving picture frame extraction unit 548 (step S210). The superimposed display metadata is stored in the storage unit 5000 by the storage / readout unit 59.

  Thus, the process of step S19 shown in FIG. 18 ends. Then, the storage / reading unit 59 and the display control unit 56 perform superimposition processing using the superimposition display metadata (step S20).

{Superposition processing}
Next, the superimposition process will be described in detail with reference to FIGS. 23 and 28 are conceptual diagrams of images in the process of superimposed display. FIG. 23 shows a process for calculating the position parameter and the correction parameter, which is the first half of the superimposition process. FIG. 28 shows a process of superimposing the planar image P on the omnidirectional image CE, which is the latter half of the superimposition process.

  First, the storage / reading unit (acquisition unit) reads and acquires the reference shape information and each reference shape conversion information from the storage unit 5000 in advance. In addition, the storage / reading unit (acquiring unit) is obtained by data of the equirectangular projection image EC related to the frame of the moving image obtained by the equirectangular projection method and the perspective projecting method as the moving image playback time elapses. The data of the plane image P related to the frame of the moving image is read and acquired.

  Then, the interpolation unit 574 reads each reference shape conversion information generated by the metadata creation unit 55a in accordance with the time lapse of the video playback by the playback time management unit 571, and interpolates the video as necessary, thereby interpolating the video. A shape conversion parameter is calculated for each frame reproduction time (S270).

  Here, the shape conversion parameters will be described in detail with reference to FIG. FIG. 24 explains the interpolation method. In the present embodiment, the reference shape conversion information is all expressed in angles except for the reference shape scaling information. When the rotation angle is obtained from a homography matrix or the like, as shown in FIG. 24A, the value of the angle is in a range of ± 180 degrees, for example. If the boundary of ± 180 degrees is crossed during interpolation, an unintended operation may occur. For example, consider a case where the frames of a moving image are set to 130 degrees, 150 degrees, 170 degrees, -170 degrees, -150 degrees, and -130 degrees at 1 second intervals. As shown in FIG. 24 (b), it is normal to think that it is moving at a constant speed counterclockwise from 130 degrees to -130 degrees. However, if this video frame is interpolated at 0.5 second intervals, the simple interpolation result of 170 and -170 is 0, and as a result, as shown in FIG. It will move greatly. Therefore, the interpolation unit 574 determines which direction is closer between the interpolated frame data and executes the interpolation. For example, as shown in FIG. 24D, when the reference shape indicated by the reference position parameter is moved or rotated from 130 degrees (point A) to -130 degrees (point B), the counterclockwise direction is close and ± 180 degrees. In this case, interpolation is performed between point A and (point B + 360 degrees). As shown in FIG. 24 (e), when moving or rotating from 130 degrees (point C) to -20 degrees (point D), the clockwise direction is close and does not cross the ± 180 degree boundary. Simple interpolation is performed at point D. When moving or rotating from -130 degrees (point E) to 130 degrees (point F) as shown in FIG. 24 (f), the clockwise direction becomes closer and the point E and point (D) are crossed over the ± 180 degree boundary. -360 degrees). By interpolating in this way, the movement is almost as intended except when the setting interval of moving image frames is large with respect to the speed of movement or rotation.

  Subsequently, returning to FIG. 23, the conversion position parameter calculation unit 576 uses the reference shape information (reference position parameter) generated by the metadata generation unit 55a and the reference shape conversion information obtained by the interpolation unit 574. The conversion position parameter for this time is calculated (step S280).

Here, the calculation method of the conversion position parameter will be described with reference to FIG. FIG. 25 is a three-dimensional conceptual diagram showing the flow of processing in the conversion position parameter calculation unit.
FIG. 25A is a diagram in which the coordinates (partial solid sphere PS) on the three-dimensional solid sphere CS are created based on the reference position parameter. Again, the idea of (Equation 4) above is used.

  Next, as shown in FIG. 25B, the conversion position parameter calculation unit 576 obtains a plane composed of three points of the four corners of the partial solid sphere PS, and with respect to the plane. Each point is projected through the center of the solid sphere CS as a viewpoint. The selection of three points is arbitrary. Since the formula of a plane passing through three points on three dimensions is a known technique, it is omitted. If the projection from the partial solid sphere PS to the plane is a point P (a, b, c) on the partial solid sphere PS, the equation of the straight line from the origin (viewpoint) to the point P uses the parameter t ( x, y, z) = t (a, b, c). What is necessary is just to obtain | require the intersection of a plane and a straight line.

  Next, the conversion position parameter calculation unit 576 translates each point so that the center of the plane comes to the origin, as shown in FIG. For the center of the plane, for example, a vector for moving the point to the origin is obtained as an average of the position coordinates of the four corner points, and the vector is added to each point.

  Next, as shown in FIG. 25 (d), the conversion position parameter calculation unit 576 receives the information (reference shape after interpolation) obtained by the interpolation unit 574 for each point on the plane of FIG. 25 (c). Rotation information and scaling information are applied to the conversion information. The reason for moving to the origin earlier is to apply rotation and scaling information in local coordinates.

  Next, as shown in FIG. 25 (e), the conversion position parameter calculation unit 576 adds a vector opposite to the vector translated to the origin to each point to return to the original position.

  Next, the conversion position parameter calculation unit 576 returns each point on the plane to the solid sphere CS as shown in FIG. In the method of returning to the solid sphere CS, for example, a certain point on the plane is defined as Q (d, e, f). In order to project these points onto a sphere having a radius of 1, the length of the vector from the origin to point Q need only be 1.

  Next, as shown in FIG. 25 (e), the conversion position parameter calculation unit 576 includes reference shape conversion information acquired by the interpolation unit 574 for each point projected onto the solid sphere CS. The remaining reference shape movement information is applied. Since the reference shape movement information is rotation information centered on the origin, the (x, y, z) coordinates of each point may be multiplied by a rotation matrix. The converted position parameter is calculated by returning again from the three-dimensional coordinates to which the reference position parameter is applied to the coordinates in the equirectangular projection image EC. In addition, although the example which produces the partial solid sphere PS once after returning to the coordinate in the equirectangular projection image EC once this time was shown, what made reference shape movement information act as a partial solid sphere PS as it is Also good.

  Subsequently, returning to FIG. 23, the shape conversion unit 578 converts the region on the equirectangular projection image EC specified by the conversion position parameter calculated by the conversion position parameter calculation unit 576 into the same rectangular shape as the planar image P. The third corresponding area CA3 is created by converting to (step S290). Then, the correction parameter creation unit 580 compares the third corresponding area CA3 with the planar image P extracted from the reproduction moving image frame extraction unit 572, and calculates the correction parameter, thereby creating the correction parameter. (Step S300).

  28, first, the equirectangular projection image obtained by the storage / reading unit 59 (acquisition unit) shown in FIG. 13 in advance by the equirectangular projection method from the storage unit 5000 is used. EC data, data of a planar image P obtained by the perspective projection method, and conversion position parameters are calculated. Then, the playback frame selection unit 572 selects data of the equirectangular projection image EC and the data of the planar image P that are video frames.

  Next, as shown in FIG. 28, the pasting region creation unit 582 creates the partial solid sphere PS in the virtual solid sphere CS by converting the reference shape parameter using the conversion position parameter (step S310). ).

  Next, the correction unit 584 matches the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC based on the correction parameters created by the correction parameter creation unit 580. Correction is performed (step S320). Hereinafter, the corrected planar image P is referred to as “corrected image C”.

  Next, the image creation unit 586 creates the superimposed image S by pasting the corrected image C on the partial solid sphere PS (step S330). The image creation unit 586 creates mask data M based on the partial solid sphere PS (step S340). Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC on the solid sphere CS (step S350). Then, the image superimposing unit 588 superimposes the superimposed image S and the mask data M on the omnidirectional image CE (step S360). Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  Next, as shown in FIG. 7, the projective transformation unit 590 is based on the predetermined visual line direction of the virtual camera IC (the center point CP of the predetermined area T) and the angle of view α of the predetermined area T. Projective transformation is performed so that the predetermined area T in the omnidirectional image CE with the superimposed image S superimposed can be seen on the display 517 (step S370). At this time, the projective transformation unit 590 also performs processing for matching the predetermined area T with the resolution of the display area on the display 517. Thereby, the display control part 56 can perform the process of the display of the predetermined area image Q which shows the predetermined area T over the whole display area of the display 517 (step S24). Here, a superimposed image S that is a planar image P ′ in which the planar image P is superimposed is included in the predetermined region image Q.

  Subsequently, the state of the superimposed display will be described in detail with reference to FIGS. 29 to 33. FIG. 29 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. Here, the case where the planar image P is superimposed on FIG. 5 is shown. As shown in FIG. 29, the high-definition superimposed image S is superimposed on the inside of the solid sphere CS according to the conversion position parameter with respect to the low-definition omnidirectional image CE attached to the solid sphere CS. Yes.

  FIG. 30 is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. FIG. 30 illustrates a state in which the omnidirectional image CE and the superimposed image S are pasted on the solid sphere CS and the image including the superimposed image S is the predetermined region image Q.

  FIG. 31 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the position parameter of the present embodiment. FIG. 32 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image using the position parameters of the present embodiment.

  As shown in FIG. 31A, when the virtual camera IC is located at the center point of the solid sphere CS, the subject po1 is represented as an image po2 on the omnidirectional image CE. And represented as an image po3 on the superimposed image S. As shown in FIG. 31A, the image po2 and the image po3 are located on a straight line connecting the virtual camera IC and the subject po1, and thus the superimposed image S is superimposed on the omnidirectional image CE. Even if displayed in the state, the omnidirectional image CE and the superimposed image S are not displaced. However, as illustrated in FIG. 31B, when the virtual camera IC is separated from the center point of the solid sphere CS, the image po2 is positioned on a straight line connecting the virtual camera IC and the subject po1. The image po3 is located slightly inside. For this reason, if an image on the superimposed image S on the straight line connecting the virtual camera IC and the subject po1 is an image po3 ′, the omnidirectional image CE and the superimposed image S are shifted from the image po3 and the image po3 ′. Deviation of the amount g occurs. Thereby, the superimposed image S is shifted from the omnidirectional image CE and displayed.

On the other hand, in the present embodiment, since the position parameter indicated by the plurality of lattice regions is used, the superimposed image S is represented as an omnidirectional ball as shown in FIGS. 32 (a) and 32 (b). It can be superimposed along the image CE. Thereby, as shown in FIG. 32A, not only when the virtual camera IC is located at the center point of the solid sphere CS, but also as shown in FIG. Even when it is away from the center point of the three-dimensional sphere CS, the image po2 and the image po3 are located on a straight line connecting the virtual camera IC and the subject po1. Therefore, even if the superimposed image S is displayed in a state of being superimposed on the omnidirectional image CE, there is no deviation between the omnidirectional image CE and the superimposed image S.

  33A shows a display example of a wide image without superimposing display, FIG. 33B shows a display example of tele image without superimposition display, and FIG. 33C shows a display example of wide image with superimposition display. FIG. 33D is a conceptual diagram showing a display example of a tele image in the case of superimposed display. Note that the wavy lines in the figure are merely shown for convenience of explanation, and may or may not be actually displayed on the display 517.

  As shown in FIG. 33 (a), when the planar image P is not superimposed and displayed on the omnidirectional image CE, if it is enlarged and displayed up to the area indicated by the wavy line in FIG. 33 (a), FIG. As shown in (b), the low-definition image remains as it is, and the user will see an unclear image. On the other hand, as shown in FIG. 33 (c), when the planar image P is displayed superimposed on the omnidirectional image CE, it is enlarged to the area indicated by the wavy line in FIG. 33 (c). When displayed, a high-definition image is displayed as shown in FIG. 33D, and the user can see a clear image. In particular, when a signboard or the like on which characters are drawn is displayed in the area indicated by the wavy line, if the high-definition plane image P is not superimposed and displayed, the characters are blurred even if enlarged and displayed. I don't know if is written. However, if the high-definition planar image P is displayed in a superimposed manner, the characters can be clearly seen even when enlarged and displayed, so that the user can grasp what is written.

  In addition, as in the present embodiment, when the planar image P that is the frame data of the moving image is superimposed on the omnidirectional image CE that is the frame data of the moving image, the appearance as shown in FIG. Become. FIG. 34 is a diagram illustrating an example in which a moving image plane image is displayed in a predetermined area image of the moving image.

  On the display 517, the predetermined area images Q1, Q2, Q3, and Q4 are respectively shown in time series in FIG. 34 (b), FIG. 34 (c), and FIG. 34 (d) in order from FIG. 34 (a). The flat images P1, P2, P3, and P4 are displayed. The predetermined area images Q1, Q2, Q3, and Q4 are low-definition images, but the planar images P1, P2, P3, and P4 are high-definition images, so that the planar images P1, P2, P3, and P4 are enlarged and displayed. However, the user can see a clear image.

  In addition, when the background image (equal-angle cylindrical projection image) and the foreground image (planar image) are still images, a plurality of positions (points), luminance, and color correction parameters (correction information) in the foreground image are set as the background image. These may be stored in association with a plurality of positions (points). However, if this method is applied to a moving image as it is, there arises a problem that the data amount of the superimposed display metadata increases in proportion to the recording time of the moving image to be superimposed. That is, the foreground image at a certain time and the overlay position of the foreground image at a certain time and the overlay position of the foreground image at another certain time may change differently from a still image. It is necessary to record the superimposed position in units of frames of the moving image. For this reason, position information indicating the superimposed position is required for the number of frames of the moving image, and the superimposed display metadata needs to hold a large amount of position information. Accordingly, the superimposition unit 55b needs to perform superimposition processing for each frame of the background moving image and the foreground image using the superimposition display metadata, so that the processing load becomes very large.

  On the other hand, in this embodiment, as shown in FIG. 22A, the reference shape BF is defined on the background image by the reference position parameter, and the reference shape BF is rotated and changed in the three-dimensional model space. By using the reference shape conversion information for moving the image, the data size of the superimposed display metadata necessary for calculating the superimposed position in each frame in the moving image is reduced, and parameters are set for all frames of the moving image. By recording the reference position parameters for the appropriately sampled moving image frames instead of having them, the data amount of the necessary parameters can be reduced. Further, it is considered that the parameter is provided as a parameter set that can reduce the arithmetic processing for the superimposition processing compared to the conventional processing when performing the superimposition display performed by a viewer or the like. As a specific example, consider an arithmetic process for calculating data that can be handled by OpenGL, a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. ing. In this way, since it is recorded as metadata that can be handled by an external program, the interpolation unit 574 realized by executing the external program performs reference shape conversion information indicating time series rotation, scaling, and movement. Can be realized by real-time processing.

<< Main effects of this embodiment >>
As described above, according to the present embodiment, it is possible to suppress the difficulty of seeing even if the other image is combined with one image having a different projection method. Moreover, even if the other image (for example, the planar image P) is superimposed on one image (for example, the equirectangular projection image EC) having a different projection method, an effect of suppressing the image shift can be achieved.

  Further, as shown in FIG. 33 (c), a high-definition plane image P is superimposed and displayed on a partial area of a predetermined area image that is a part of the low-definition omnidirectional image CE. Therefore, there is an effect that the omnidirectional image and the image of the planar image are matched, and the planar image can be well blended into the omnidirectional image.

  Furthermore, according to the present embodiment, by defining the reference shape BF indicated by the reference shape information and using the reference shape conversion information for rotating, scaling, and moving the reference shape BF on the three-dimensional model space, In a plurality of moving images with different projection methods, it is possible to reduce the data size of a parameter necessary for calculating a position parameter for superimposing each frame of the other moving image on each frame of one moving image. Further, by recording parameters for an arbitrarily sampled frame instead of having parameters for all the frames of the moving image, it is possible to reduce the required parameter data amount.

[Second Embodiment]
Subsequently, a second embodiment of the present invention will be described with reference to FIGS.

<Outline of shooting system>
First, an outline of the configuration of the photographing system of this embodiment will be described with reference to FIG. FIG. 35 is a schematic diagram of the configuration of the imaging system of the present embodiment.

  As shown in FIG. 35, in the imaging system of the present embodiment, an image processing server 7 is further added to each configuration according to the first embodiment. The same configurations as those of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The smartphone 5 and the image processing server 7 can mutually communicate with the image processing server 7 via a communication network 100 such as the Internet or an intranet.

In the first embodiment, the smartphone 5 performs the process of creating superimposed display metadata and the process of superimposing images, whereas in the present embodiment, the image processing server 7 performs these processes.
Note that the smartphone 5 of the present embodiment is an example of a communication terminal, and the image processing server is an example of an image processing apparatus.

  The image processing server 7 is a server computer, and includes a case where image processing is performed in a distributed manner by a plurality of server computers.

<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of the image processing server 7 of this embodiment will be described in detail with reference to FIG. FIG. 36 is a hardware configuration diagram of the image processing server. Note that the hardware configurations of the special imaging device 1, the general imaging device 3, and the smartphone 5 according to the present embodiment are the same as those in the first embodiment, and thus the description thereof is omitted.

<Hardware configuration of image processing server>
FIG. 36 is a hardware configuration diagram of the image processing server. The image processing server 7 is constructed by a computer, and as shown in FIG. 6, a CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) 705, recording medium 706, media I / F 707, display 708. A network I / F 709, a keyboard 711, a mouse 712, a CD-RW drive 714, and a bus line 710. Since the image processing server 7 functions as a server, an input device such as a keyboard 711 and a mouse 712 and an output device such as a display 708 may not be provided.

  Among these, the CPU 701 controls the overall operation of the image processing server 7. The ROM 702 stores a program used for driving the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data with respect to the HD 704 according to the control of the CPU 701. The media I / F 707 controls reading or writing (storage) of data with respect to a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, menu, window, character, or image. A network I / F 709 is an interface for performing data communication using the communication network 100. The keyboard 711 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 714 controls reading of various data with respect to a CD-RW (Compact Disc-ReWritable) 713 as an example of a removable recording medium.

  Further, the image processing server 7 includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 701 shown in FIG.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of this embodiment will be described with reference to FIGS. 37 and 39. FIG. 37 is a functional block diagram of the photographing system according to the present embodiment. In addition, since the functional structure of the special imaging device 1, the general imaging device 3, and the smart phone 5 of this embodiment is the same as that of 1st Embodiment, those description is abbreviate | omitted. In the case of the present embodiment, the image / sound processing unit 55 of the smartphone 5 may or may not have each functional configuration shown in FIG.

<Functional configuration of image processing server>
As shown in FIG. 37, the image processing server 7 includes a long-distance communication unit 71, a reception unit 72, an image / sound processing unit 75, a display control unit 76, a determination unit 77, and a storage / reading unit 79. is doing. Each of these units is a function realized by any of the constituent elements shown in FIG. 36 being operated by an instruction from the CPU 701 according to the image processing server 7 program expanded from the HD 704 onto the RAM 703, or Means.

  Further, the image processing server 7 has a storage unit 7000 constructed by the ROM 702, the RAM 703, and the HD 704 shown in FIG.

(Functional configuration of image processing server)
The long-distance communication unit 71 of the image processing server 7 is mainly realized by the processing of the network I / F 707 and the CPU 701 shown in FIG. 36, and other devices (for example, other servers, Send and receive various data (or information) to and from your smartphone.

  The accepting unit 72 is realized mainly by the processing of the keyboard 711, the mouse 712, and the CPU 701, and accepts various selections or inputs from the user.

  The image / sound processing unit 75 is realized mainly by instructions from the CPU 701 and performs various processes on various data sent from the smartphone 5.

  The display control unit 76 is realized mainly by the processing of the CPU 701, and unlike the display control unit 56 of the first embodiment, creates data of a predetermined area image Q for displaying the planar image P on the display 517 of the smartphone 5. To do. Further, the display control unit 76 uses the superimposed display metadata created by the image / sound processing unit 75 to indicate each lattice area LA0 of the planar image P by the position indicated by the position parameter and the correction parameter. By matching with the brightness value and the color value, data for displaying the planar image P superimposed on the omnidirectional image CE is created.

  The determination unit 77 is realized by the processing of the CPU 701 shown in FIG. 36 and makes various determinations.

  The storage / reading unit 79 is realized mainly by the processing of the CPU 701 shown in FIG. 36, and stores various data (or information) such as superimposed display metadata in the storage unit 7000, or from the storage unit 7000. Various data (or information) such as superimposed display metadata is read out. The storage / reading unit 79 serves as an acquisition unit that acquires various data from the storage unit 7000.

(Detailed functional configuration of image / sound processor)
Here, the functional configuration of the image / sound processor 75 will be described in detail with reference to FIG. FIG. 39 is a detailed functional block diagram of the image / sound processing unit.

  The image / sound processing unit 75 is roughly divided into a metadata creating unit 75a that performs encoding and a superimposing unit 75b that performs decoding. The metadata creation unit 75a executes the process of step S121 described later shown in FIG. In addition, the superimposing unit 75b performs a process of step S122 described later illustrated in FIG.

{Each functional configuration of the metadata creation unit}
First, each functional configuration of the metadata creation unit 75a will be described. The metadata creation unit 75a includes a moving picture frame extraction unit 748, an extraction unit 750, a first corresponding region calculation unit 752, a gazing point identification unit 754, a projection method conversion unit 756, a second corresponding region calculation unit 758, and an initial setting shape. A generation unit 759, a region division unit 760, a projection method inverse conversion unit 762, a reference shape conversion information calculation unit 768, and a superimposed display metadata creation unit 770 are included. These are the moving image frame extraction unit 548, the extraction unit 550, the first corresponding region calculation unit 552, the gaze point specification unit 554, the projection method conversion unit 556, and the second corresponding region calculation in the first embodiment, respectively. Unit 558, initial setting shape generation unit 559, region division unit 560, projection method inverse conversion unit 562, reference shape conversion information calculation unit 568, and superimposed display metadata creation unit 570, and these descriptions are Omitted.

{Functional structure of superimposition unit}
Next, the functional configuration of the superimposing unit 75b will be described. The superimposing unit 75b includes a reproduction time management unit 771, a reproduction moving image frame selection unit 772, an interpolation unit 774, a position parameter calculation unit 776, a shape conversion unit 778, a correction parameter creation unit 780, a pasting region creation unit 782, a correction unit 784, an image. A creation unit 786, an image superimposing unit 788, and a projective transformation unit 790 are included. These are the playback time management unit 571, the playback video frame extraction unit 572, the interpolation unit 574, the displacement position parameter calculation unit 576, the shape conversion unit 578, the correction parameter creation unit 580, and the paste area creation, respectively, in the first embodiment. Since the functions are the same as those of the unit 582, the correction unit 584, the image creation unit 586, the image superimposing unit 588, and the projective transformation unit 590, their descriptions are omitted.

<< Processing or Operation of Embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIG. A photographing method executed by the photographing system will be described with reference to FIG. FIG. 40 is a sequence diagram illustrating the photographing method according to the present embodiment. In addition, since the process of step S111-S118 is a process similar to step S11-S18 of 1st Embodiment, these description is abbreviate | omitted.

  In the smartphone 5, the long-distance communication unit 51 transmits, to the image processing server 7 via the communication network 100, superimposition request information indicating a superimposition request for superimposing the other image on one image having a different projection method (step). S119). Data in the electronic folder (planar image data, equirectangular projection image data) stored in the storage unit 5000 is transmitted to the superimposition request information. Thereby, the transmission / reception unit 71 of the image processing server 7 receives the data in the electronic folder.

  Next, in the image processing server 7, the storage / readout unit 79 stores the data in the electronic folder received in step S119 in the storage unit 7000 (step S120). Then, the metadata creating unit 75a shown in FIG. 38 creates superimposed display metadata (step S121). Furthermore, the superimposing unit 75b performs a superimposing process (step S122). Since the processes of steps S121 and S122 have the same contents as the processes of steps S21 and S22, respectively, their descriptions are omitted.

  Next, the display control unit 76 creates data of the predetermined area image Q that displays the predetermined area image Q indicating the predetermined area T over the entire display area of the display 517 of the smartphone 5. Here, a superimposed image S that is a planar image P ′ in which the planar image P is superimposed is included in the predetermined region image Q. The long-distance communication unit 51 transmits the data of the predetermined area image Q created by the display control unit 76 to the smartphone 5 (step S123). Thereby, the long-distance communication part 51 of a smart phone receives the data of the predetermined area image Q.

  Next, in the smartphone 5, the display control unit 56 displays the predetermined area image Q including the superimposed image S on the display 517 (step S124).

<< Main effects of this embodiment >>
As described above, according to the present embodiment, the same effects as those of the first embodiment can be obtained.

  In the present embodiment, the cooperative shooting process is performed by the smartphone 5, and the superimposition display metadata creation and the superimposition process are performed by the image processing server 7. Therefore, the processing capability of the smartphone 5 is relatively low. In addition, there is an effect that image shift can be suppressed.

[Supplement]
In each of the above embodiments, as shown in FIG. 13, the smartphone 5 has equirectangular projection image data, planar image data, and superimposed display parameter data. However, the present invention is not limited to this. For example, a management server that can communicate via a communication network such as the Internet may store at least one of equirectangular projection image data, planar image data, and superimposed display parameter data.

  In each of the above embodiments, the planar image P is superimposed on the omnidirectional image CE, but the present invention is not limited to this. For example, a partial image of the omnidirectional image CE may be replaced with the planar image P, or a partial image of the omnidirectional image CE may be deleted and the planar image P may be inserted into the deleted portion. Good.

  In the second embodiment, the superimposing process is performed by the image processing server 7 (see step S45), but the present invention is not limited to this. For example, the superimposed display metadata may be transmitted from the image processing server 7 to the smartphone 5, and the overlapping process and the display process may be performed on the smartphone 5 side. In this case, in the image processing server 7, the metadata creation unit 75a shown in FIG. 39 creates metadata for superimposed display. On the other hand, in the smartphone 5, the superimposing unit 55b illustrated in FIG. 15 performs the superimposing process, and the display control unit 56 illustrated in FIG. 13 performs the display process.

  Moreover, although the case where a planar image is superimposed on an omnidirectional image has been described above, the superimposition is an example of synthesis. In addition to superposition, the composition includes pasting, fitting, superposition, and the like. The superimposed image is an example of a composite image. In addition to the superimposed image, the composite image includes a pasted image, a fitted image, a superimposed image, and the like. Furthermore, the image superimposing units 588 and 788 are examples of an image synthesizing unit.

  The equirectangular projection image EC and the planar image P may be either a still image, a case where both are moving image frames, a case where one is a still image and the other is a moving image frame.

  Each of the functional configurations shown in FIGS. 13, 15, 16, 36, 37, and 38 is realized in the form of a software functional unit and is read by a computer when sold or used as an independent product. It can be stored in a possible storage medium. In this case, the technical plan of the present embodiment is expressed in the form of a software product, which is essential, or a part that contributes to the prior art or a part of the technical plan. The computer software product is stored in a storage medium, and a plurality of commands for causing a computer apparatus (personal computer, server, network device, etc.) to execute all or some steps of the method according to each of the embodiments. Including. The above-described storage medium includes various media that can store program codes, such as a USB memory, a removable disk, a ROM, a RAM, a magnetic disk, or an optical disk.

  In addition, the method according to the above embodiment is applied to or realized by the processor. A processor is an integrated circuit board capable of processing signals. Each step of the method of each of the above embodiments is realized by an integrated logic circuit, which is hardware in the processor, or a software type instruction. The processor 42 is a general purpose processor, digital signal processor (DSP), dedicated integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and Each method, step, and logic box disclosed in each embodiment can be realized or executed. The general-purpose processor is a microprocessor or any general processor. Each step of the method according to each of the above embodiments may be realized by being executed by a decoder that is hardware, or may be realized by a combination of hardware and software that can be performed by the decoder. The software modules are stored in storage media mature in the field, such as random memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The processor reads information from a memory having a storage medium in which the software is stored, and implements the steps of the method according to the hardware.

  The embodiment described above is realized by hardware, software, firmware, middleware, microcode, or a combination thereof. Among them, regarding hardware implementation, the processing unit can be one or more dedicated integrated circuits (ASIC), digital signal processor (DSP), digital signal processor (DSPD), programmable logic circuit (PLD), field programmable It can be implemented by a gate array (FPGA), a general purpose processor, a controller, a microcontroller, a microprocessor, other electronic units that perform the functions of the present invention, or a combination thereof. As for software implementation, the above technique is implemented by modules (for example, processes, functions, etc.) that implement the above-described functions. The software code is stored in memory and executed by the processor. The memory is realized inside or outside the processor.

1 Special shooting device (example of first shooting device)
3 General photographing device (an example of a second photographing device)
5 Smartphone (an example of an image processing device)
7 Image processing server (an example of an image processing device)
51 Transmission / Reception Unit 52 Reception Unit 55a Metadata Creation Unit 55b Superimposition Unit 56 Display Control Unit 58 Near Field Communication Unit 59 Storage / Readout Unit (Example of Acquisition Unit)
72 Reception unit 75 Image / sound processing unit 75a Metadata creation unit 75b Superimposition unit 76 Display control unit 78 Short-range communication unit 79 Storage / reading unit (an example of an acquisition unit)
517 Display 548 (Metadata) Creation Movie Frame Extraction Unit 550 Extraction Unit (Example of Extraction Unit)
552 1st corresponding area calculation part (an example of the 1st corresponding area calculation means)
554 Gaze point identification part (an example of gaze point identification means)
556 Projection method conversion unit (an example of projection method conversion means)
558 Second Corresponding Area Calculation Unit (Example of Corresponding Area Calculation Unit / Example of Second Corresponding Area Calculation Unit)
559 initial setting shape generation unit (an example of initial setting shape generation means)
560 area dividing unit (an example of area dividing means)
562 Projection method reverse conversion unit (an example of projection method reverse conversion means)
568 Reference shape conversion information calculation unit (an example of reference shape conversion information calculation means)
570 Superimposed display metadata creation unit (an example of position calculation means)
571 Playback time management unit 572 Playback video frame extraction unit 574 Interpolation unit (an example of interpolation means)
575 Conversion position parameter calculation unit 578 Shape conversion unit 580 Correction parameter creation unit 582 Attachment region creation unit (an example of attachment region creation means)
584 Correction unit (an example of correction means)
586 Image creation unit (an example of image creation means)
588 Image superimposing unit (an example of image superimposing means)
590 Projection conversion unit (an example of projection conversion means)
750 extraction unit (an example of extraction means)
752 First corresponding area calculation unit (an example of first corresponding area calculation means)
754 Gaze point identification part (an example of gaze point identification means)
756 Projection method conversion unit (an example of projection method conversion means)
758 Second corresponding area calculation unit (an example of a corresponding area calculation unit / an example of a second corresponding area calculation unit)
760 area dividing unit (an example of area dividing means)
762 Projection method inversion unit (an example of projection method inversion unit)
764 Shape converter (an example of shape converter)
766 Correction parameter creation unit (an example of correction information creation means)
770 superimposed display metadata creation unit (an example of position calculation means)
782 Attachment region creation unit (an example of attachment region creation means)
784 Correction unit (an example of correction means)
786 Image creation unit (an example of image creation means)
788 image superimposing unit (an example of image superimposing means)
790 Projection conversion unit (an example of projection conversion means)
5000 storage unit (an example of storage means)
5001 Collaborative photographing apparatus DB (an example of cooperative photographing apparatus means)
7000 storage unit (an example of storage means)

Japanese Patent Laid-Open No. 2006-96487 JP 2012-178135 A

Claims (12)

A first image of the first projection method, a second image of a second projection method different from the first projection method, and a reference shape corresponding to the second image on the first image Acquisition means for acquiring reference shape information indicating and reference shape conversion information for converting the reference shape on the first image;
Image creating means for creating a composite image corresponding to the second image by converting the reference shape information based on the reference shape conversion information;
Image synthesizing means for synthesizing the synthesized image with the first image;
An image processing apparatus comprising: The image processing apparatus according to claim 1, further comprising:
Conversion position parameter calculation means for calculating a conversion position parameter that is a result of converting the reference shape information based on the reference shape conversion information;
Shape conversion means for converting the corresponding region on the first image specified by the conversion position parameter into the same rectangular shape as the second image;
Correction information creating means for comparing the corresponding area and the second image to calculate correction information for correcting at least one of luminance and color;
Have
The image processing device is characterized in that the image creating unit creates the composite image by correcting at least one of luminance and color of the second image based on the correction information.   The image processing apparatus according to claim 1, wherein the first image or the second image is a frame of the same shooting time in a moving image. The image processing apparatus according to claim 3, further comprising:
The acquisition means acquires a plurality of reference shape conversion information for a single reference shape information,
Interpolating means for creating a shape conversion parameter for converting the reference shape for each playback time of each frame by interpolating using the plurality of reference shape conversion information in accordance with the playback time of each frame in the moving image Have
The image processing apparatus, wherein the conversion position parameter calculation means calculates the conversion position parameter, which is a result of converting the reference shape information for each shape conversion parameter. The image creating means corrects at least one of the brightness and the color of the second image based on the second image or the correction information on the partial solid sphere created based on the conversion position parameter. Create the composite image by pasting the image,
The image processing apparatus according to claim 2, wherein the synthesizing unit synthesizes the synthesized image with a solid sphere image created based on the first image. The image creating means creates mask data based on the position information,
The image processing apparatus according to claim 1, wherein a transparency around the mask data is gradually increased from an inner side to an outer side of the mask data.   The image processing apparatus according to claim 1, wherein the first image is an equirectangular projection image, and the second image is a perspective projection image.   The image processing apparatus according to claim 1, wherein the image processing apparatus is a smartphone, a tablet personal computer, a notebook personal computer, a desktop personal computer, or a server computer. An image processing apparatus according to claim 1;
A first imaging device that images a subject and obtains the first image of the first projection method;
A second imaging device that images a subject and obtains the second image of the second projection method;
An imaging system comprising:   The imaging system according to claim 9, wherein the first imaging device is a camera that captures a subject and obtains an omnidirectional image. A first image of the first projection method, a second image of a second projection method different from the first projection method, and a reference shape corresponding to the second image on the first image An acquisition step of acquiring reference shape information to indicate and reference shape conversion information for converting the reference shape on the first image;
An image creation step of creating a composite image corresponding to the second image by converting the reference shape information based on the reference shape conversion information;
An image synthesis step of synthesizing the synthesized image with the first image;
The image processing method characterized by performing.   A program causing a computer to execute the method according to claim 11.