JP2018157538A - Program, imaging system, and information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program capable of appropriately correcting a planar image to be overlapped on a wide-angle image.SOLUTION: An information processing apparatus overlaps a planar image obtained by a second projection method to a wide-angle image obtained by a first projection method. The information processing apparatus is functioned as: projection method inverse conversion means for calculating a corresponding region to which the planar image corresponds in the wide-angle image; correction means for correcting at least one of the wide-angle image and the planar image according to an amount of displacement between a visual line direction of the wide-angle image and a center of the planar image; and image generating means for overlapping the planar image corrected by the correction means to the corresponding region of the wide-angle image corrected by the correction means.SELECTED DRAWING: Figure 16

Description

  The present invention relates to a program, an imaging system, and an information processing apparatus.

  By providing a flat image obtained by capturing separately from the wide-angle flat image in a partial area of the wide-angle flat image, a clear image of the partial image is provided together with the entire image. can do.

  In recent years, special digital cameras that provide 360-degree celestial sphere images with one imaging have been provided. When a planar image is superimposed on a part of a wide-angle image such as the omnidirectional image, the brightness of the two images is different, and the planar image may be conspicuous in the wide-angle image. A technique for adjusting the brightness of an image has been devised for such inconvenience (see, for example, Patent Document 1). Patent Document 1 discloses an image processing method for outputting designated areas of image data captured under different exposure conditions to a display unit based on luminance characteristics.

  However, there is a problem that if the brightness of the planar image is simply corrected while a part of the wide-angle image is displayed, the exposure may be overexposed (overexposed) or underexposed (underexposed).

  Since wide-angle images have a wide imaging range, a display device such as a display often displays only a part of the wide-angle image, but the exposure of the wide-angle image is determined in consideration of the entire imaging range. The brightness of is not appropriate and may be over or under. In such a case, if the pixel value is corrected so that the brightness value or color value of the planar image approaches the brightness value of the wide-angle image, the brightness value of the planar image is also corrected to be over or under. .

  An object of this invention is to provide the program which can correct | amend the plane image superimposed on a wide-angle image appropriately in view of the said subject.

  The present invention provides an information processing apparatus that superimposes a planar image obtained by the second projection method on a wide-angle image obtained by the first projection method, and displays a corresponding region corresponding to the planar image in the wide-angle image. A projection method inverse transform means for calculating; a correcting means for correcting at least one of the wide-angle image and the planar image in accordance with a displacement amount between a line-of-sight direction of the wide-angle image and the center of the planar image; and the correcting means. A program is provided that causes an image creation unit to superimpose the planar image corrected by the correction unit on the corresponding area of the corrected wide-angle image.

  It is possible to provide a program that can appropriately correct a planar image superimposed on a wide-angle image.

(A) is a left side view of the special imaging device, (b) is a rear view of the special imaging device, (c) is a plan view of the special imaging device, and (d) is a bottom view of the special imaging device. It is. It is a usage image figure of a special imaging device. (A) is a hemispherical image captured by a special imaging device (front), (b) is a hemispherical image captured by a special imaging device (rear), and (c) is an image represented by equirectangular projection. FIG. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of a virtual camera and a predetermined area | region at the time of making a spherical image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. 1 is a schematic diagram of an imaging system according to an embodiment of the present invention. It is a perspective view of an adapter. It is a usage image figure of an imaging system. It is a hardware block diagram of a special imaging device. It is a hardware block diagram of a general imaging device. It is a hardware block diagram of a smart phone. It is a functional block diagram of an imaging system. (A) is a conceptual diagram of a cooperative imaging device management table, (b) is a conceptual diagram showing a cooperative imaging device setting screen. It is a detailed functional block diagram of an image / sound processing unit. It is a block diagram of superimposition display metadata. (A) is a conceptual diagram showing each lattice region in the second corresponding region, (b) is a conceptual diagram showing each lattice region in the third corresponding region. It is the sequence diagram which showed the imaging method. It is a conceptual diagram of the image in the process of the production | generation process of a superimposition display parameter. It is a conceptual diagram at the time of specifying a peripheral region image. It is a conceptual diagram at the time of dividing | segmenting a 2nd corresponding area | region into a some grid | lattice area | region. It is a conceptual diagram which shows the 3rd corresponding | compatible area | region in the equirectangular projection image EC. It is a conceptual diagram of the image in the process of the preparation process of a correction parameter. It is a conceptual diagram of the image in the process of the superimposition display process. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the position parameter of the present embodiment. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image using the position parameters of the present embodiment. (A) Display example of wide image without superimposing display, (b) Display example of tele image without superimposing display, (c) Display example of wide image with superimposing display, (d) In case of superimposing display It is the conceptual diagram which showed the example of a display of a tele image. It is a conceptual diagram of the image in the process of a correction process. It is a conceptual diagram of the image in the process of the correction process of a plane image. It is a conceptual diagram of the image in the process of the correction process of an equirectangular projection image. It is a figure explaining the change method of an image composition ratio. It is an example of the figure explaining the relationship between the gaze direction of a virtual camera, and the center point of a superimposed image. It is an example of the figure explaining the relationship between the gaze direction of a virtual camera, and the center point of a superimposed image. It is a figure which shows the effect of the correction | amendment when a spherical image is overexposed. It is a figure which shows the effect of the correction | amendment in case an omnidirectional image is underexposure. It is a conceptual diagram of the image in the process of the correction processing of the equirectangular projection image (second embodiment). It is a figure explaining the relationship between the position parameter at the time of using the multiple equirectangular projection image from which exposure differs, and a correction parameter. It is a figure explaining the production method of the correction images C2 and D2 by which the brightness value (or color value) was correct | amended. It is a figure which shows an example of the predetermined area | region T with which the several superimposed image was superimposed. 5 is an explanatory diagram when a plurality of superimposed images are superimposed on a predetermined region T. FIG. It is a conceptual diagram of the image in the process of the correction process of a plane image. It is a conceptual diagram regarding creation of the correction image of the brightness value (or color value) when the target object is one plane image. It is a conceptual diagram regarding creation of the correction image of the brightness value (or color value) when the target object is one image. It is a conceptual diagram regarding creation of the correction image of the brightness value (or color value) when the target object is calculated at the ratio of two images. FIG. 10 is a conceptual diagram of an image in a process of correcting an equirectangular projection image (third embodiment). FIG. 49 is an example of a diagram illustrating a correction process (step 500) of the equirectangular projection image EC shown in FIG. 48. It is a flowchart figure of an example explaining the procedure in which a correction | amendment part selects the equirectangular projection image imaged with different exposure for synthesize | combining with a reference area image. It is an example of the figure explaining the specific example of the synthesis | combination of an image (synthesis | combination with an overexposure image). It is an example of the figure explaining the specific example of the synthesis | combination of an image (synthesis | combination with an underexposure image). It is a conceptual diagram of the image in the process of correction processing of an equirectangular projection image (fourth embodiment). It is a figure which shows the relationship between the correction image D and the correction image C. FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< Summary of Embodiment >>
Hereinafter, an outline of the present embodiment will be described.
First, a method for generating an omnidirectional image will be described with reference to FIGS.

  First, the external appearance of the special imaging device 1 will be described with reference to FIG. The special imaging device 1 is a digital camera for obtaining a captured image that is a source of an omnidirectional (360 °) panoramic image. 1A is a left side view of the special imaging apparatus, FIG. 1B is a rear view of the special imaging apparatus, and FIG. 1C is a plan view of the special imaging apparatus. (D) is a bottom view of the special imaging device.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and 1 (d), the upper part of the special imaging device 1 has a fish-eye shape on the front side (front side). A fish-eye lens 102b is provided on the lens 102a and the back side (rear side). The special imaging device 1 includes imaging elements (image sensors) 103a and 103b, which will be described later. A hemispherical image (angle of view 180 °) is obtained by imaging a subject or a landscape through the lenses 102a and 102b, respectively. Above). A shutter button 115 a is provided on the front surface of the special imaging device 1. Further, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and an imaging mode switching button 115d are provided on the side surface of the special imaging device 1. Each time the power button 115b and the Wi-Fi button 115c are pressed, they are switched on and off. In addition, each time the imaging mode switching button 115d is pressed, a still image imaging mode and a moving image imaging mode are switched. The shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the imaging mode switching button 115d are part of the operation unit 115, and the operation unit 115 is not limited to these buttons.

  A tripod screw hole 151 for attaching the special imaging device 1 to the camera tripod or the general imaging device 3 is provided in the center of the bottom 150 of the special imaging device 1. A micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom 150. An HDMI (registered trademark, High-Definition Multimedia Interface) terminal 153 is provided on the right end side of the bottom 150.

  Next, the usage status of the special imaging apparatus 1 will be described with reference to FIG. FIG. 2 is a usage image diagram of the special imaging apparatus. As illustrated in FIG. 2, the special imaging device 1 is used, for example, to capture a subject around the user while the user holds it in his / her hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, an outline of processing until the equirectangular projection image EC and the omnidirectional image CE are created from the image captured by the special imaging device 1 will be described with reference to FIGS. 3 and 4. 3A shows a hemispherical image (front side) captured by the special imaging apparatus 1, FIG. 3B shows a hemispherical image captured by the special imaging apparatus (rear side), and FIG. It is the figure which showed the image (henceforth "an equirectangular projection image") represented by the cylindrical projection. FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fish-eye lens 102a described later. Also, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) are synthesized by the special imaging device 1 and, as shown in FIG. 3C, the equirectangular projection image EC. Is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems) or the like, the equirectangular projection image is pasted so as to cover the spherical surface as shown in FIG. An omnidirectional image CE as shown in (b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, by displaying a predetermined area (hereinafter referred to as “predetermined area image”) of a part of the omnidirectional image CE as a flat image with little curvature, a display that does not give a sense of discomfort to humans can be achieved. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. In FIG. 6A, the omnidirectional image shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is an imaging area of the virtual camera IC. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. The zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or away from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 7).

  Then, the predetermined area image Q shown in FIG. 6A is displayed as an image of the imaging area of the virtual camera IC on a predetermined display as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. Hereinafter, description will be made using the imaging direction (ea, aa) and the angle of view (α) of the virtual camera IC. The predetermined area T may be indicated by the imaging area (X, Y, Z) of the virtual camera IC that is the predetermined area T, instead of the angle of view α and the distance f.

  Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. As shown in FIG. 7, “ea” represents an elevation angle, “aa” represents an azimuth angle, and “α” represents an angle of view. That is, the attitude of the virtual camera IC is changed so that the gazing point of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP of the predetermined area T that is the imaging area of the virtual camera IC. Become. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following formula (1) is generally established.

L / f = tan (α / 2) (Formula 1)
<Outline of imaging system>
Next, an outline of the configuration of the imaging system of the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the imaging system of the present embodiment.

  As illustrated in FIG. 8, the imaging system according to the present embodiment includes a special imaging device 1, a general imaging device 3, a smartphone 5, and an adapter 9. The special imaging device 1 is connected to the general imaging device 3 via the adapter 9.

  Among these, the special imaging device 1 is a special digital camera for imaging a subject, a landscape, and the like and obtaining two hemispherical images that are the basis of a panoramic image as described above.

  The general imaging device 3 is a digital single-lens reflex camera, but may be a compact digital camera. The general imaging device 3 is provided with a shutter button 315a which is a part of an operation unit 315 described later.

  The smartphone 5 is an information processing device that performs wireless communication with the special imaging device 1 and the general imaging device 3 using a short-range wireless communication technology such as Wi-Fi, Bluetooth (registered trademark), or NFC (Near Field Communication). is there. In addition, the smartphone 5 can display images acquired from the special imaging device 1 and the general imaging device 3 on a display 517 described later provided in the device 5.

  Note that the smartphone 5 may communicate with the special imaging device 1 and the general imaging device 3 through a wired cable without using the short-range wireless communication technology. The smartphone 5 is an example of a communication terminal, and the communication terminal includes a tablet PC (Personal Computer), a notebook PC, and a desktop PC. A smartphone is also an example of an image processing terminal.

  FIG. 9 is a perspective view of the adapter. As shown in FIG. 9, the adapter 9 includes a shoe adapter 901, a bolt 902, an upper adjuster 903, and a lower adjuster 904. Among these, the shoe adapter 901 is attached to the accessory shoe of the general imaging device 3 by sliding. At the center of the shoe adapter 901, a bolt 902 that is rotatably attached to the tripod screw hole 151 is provided. The bolt 902 is provided with a rotatable upper adjuster 903 and a lower adjuster 904. The upper adjuster 903 plays a role of fixing an object (for example, the special imaging device 1) attached to the bolt 902. The lower adjuster 904 plays a role of fixing an object (for example, the general imaging device 3) to which the shoe adapter 901 is attached.

  FIG. 10 is a usage image diagram of the imaging system. As shown in FIG. 10, the user puts the smartphone 5 in a pocket of clothes and takes an image of a subject or the like with the general imaging device 3 to which the special imaging device 1 is attached using the adapter 9. The smartphone 5 may be placed in a range where wireless communication with the special imaging device 1 or the general imaging device 3 is possible without being put in a pocket of clothes.

<< Hardware Configuration of Embodiment >>
Next, the hardware configurations of the special imaging device 1, the general imaging device 3, and the smartphone 5 according to the present embodiment will be described in detail with reference to FIGS.

<Hardware configuration of special imaging device>
First, the hardware configuration of the special imaging device 1 will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of the special imaging device 1. Hereinafter, the special imaging apparatus 1 is an omnidirectional (omnidirectional) special imaging apparatus using two imaging elements, but the number of imaging elements may be two or more. In addition, it is not always necessary to use a dedicated device for omnidirectional imaging. By attaching a retrofit omnidirectional imaging unit to a normal digital camera or smartphone, it has substantially the same function as the special imaging device 1. Also good.

  As shown in FIG. 11, the special imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only). Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, and acceleration sensor 120 etc. are also connected.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by the fisheye lenses 102a and 102b into image data of an electric signal and output the image data. A timing generation circuit for generating a horizontal or vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103 a and 103 b of the imaging unit 101 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 105. The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. The data of the equirectangular projection image as shown in FIG. 3C is created.

  The imaging control unit 105 sets a command or the like in the register group of the imaging elements 103a and 103b using the I2C bus with the imaging element 103a as a master device and 103b as a slave device. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data and the like of the register groups of the imaging elements 103a and 103b and send them to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some special imaging devices 1 may have a preview display function or a function corresponding to a moving image display by a display (for example, the display 517 of the smartphone 5). In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the special imaging apparatus 1 is not provided with a display, but a display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the special imaging apparatus 1 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed equirectangular projection image data.

  The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various imaging modes and imaging conditions.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. Data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or an external terminal (device) such as the smartphone 5 via the network I / F 116 as necessary. ).

  The communication unit 117 is connected to an external terminal (device) such as the smartphone 5 via a short-range wireless communication technology such as Wi-Fi, NFC (Near Field Radio Communication), or Bluetooth via an antenna 117a provided in the special imaging device 1. Communicate with. The communication unit 117 can also transmit the equirectangular projection image data to an external terminal (device) such as the smartphone 5.

  The electronic compass 118 calculates the orientation of the special imaging device 1 from the earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) along Exif (Exchangeable image file format), and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

  The gyro sensor 119 detects a change in angle (roll angle, pitching angle, yaw angle) accompanying the movement of the special imaging device 1. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

The acceleration sensor 120 detects acceleration in three axis directions. Based on the detected acceleration, the posture of the special imaging apparatus 1 (angle with respect to the direction of gravity) is detected. By having both the gyro sensor 119 and the acceleration sensor 120, the accuracy of image correction is improved.
<Hardware configuration of general imaging device>
Next, the hardware of a general imaging device will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the general imaging device 3. As shown in FIG. 12, the general imaging device 3 includes an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a bus 310, a CPU 311, a ROM 312, an SRAM 313, a DRAM 314, an operation. The unit 315 includes a network I / F 316, a communication unit 317, an antenna 317 a, an electronic compass 318, and a display 319. The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310.

  Each configuration (304, 310, 311, 312, 313, 314, 315, 316, 317, 317a, 318) is each configuration (104, 110, 111, 112, 113) in the special imaging device 1 of FIG. 114, 115, 116, 117, 117a, 118), the description thereof is omitted.

  Further, as shown in FIG. 12, the imaging unit 301 of the general imaging device 3 is provided with a lens unit 306 and a mechanical shutter 307 in order from the outside in the direction of the imaging element 303 on the front surface of the imaging element 303. .

  The imaging control unit 305 basically performs the same configuration and processing as the imaging control unit 105, but further drives the lens unit 306 and the mechanical shutter 307 based on a user operation received by the operation unit 315. Control.

The display 319 is an example of a display unit that displays an operation menu and an image during or after imaging.
<Smartphone hardware configuration>
Next, the hardware of the smartphone will be described with reference to FIG. FIG. 13 is a hardware configuration diagram of the smartphone. As illustrated in FIG. 13, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, an image sensor I / F 505 a, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiver 509.

  Among these, the CPU 501 controls the operation of the entire smartphone 5. The ROM 502 stores programs used for driving the CPU 501 such as the CPU 501 and IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as a smartphone program under the control of the CPU 501. The image sensor I / F 505a is connected to a CMOS (Complementary Metal Oxide Semiconductor) sensor 505, and captures a subject (mainly a self-portrait) under the control of the CPU 501 to obtain image data. The acceleration / direction sensor 506 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 508 controls reading or writing (storage) of data with respect to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

  The smartphone 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I / F 513, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, an external device connection I / F 518, and a short distance. A communication circuit 519, a short-range communication circuit antenna 519a, and a touch panel 521 are provided.

  Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network described later. The CMOS sensor 512 is a kind of built-in imaging unit that captures an image of a subject under the control of the CPU 501 and obtains image data. The image sensor I / F 513 is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal display or an organic EL display that displays a subject image, various icons, and the like. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 521 is a kind of input means for operating the smartphone 5 when the user presses the display 517.

  In addition, the smartphone 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

  Note that a recording medium such as an HD (Hard Disk) or a CD-ROM in which each of the above programs is stored can be provided domestically or abroad as a program product.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of the present embodiment will be described with reference to FIGS. FIG. 14 is a functional block diagram of the special imaging device 1, the general imaging device 3, and the smartphone 5 that constitute a part of the imaging system of the present embodiment.

<Functional configuration of special imaging device>
First, the functional configuration of the special imaging device 1 will be described in detail with reference to FIGS. 11 and 14. As illustrated in FIG. 14, the special imaging device 1 includes a reception unit 12, an imaging unit 13, a sound collection unit 14, an image / sound processing unit 15, a determination unit 17, a short-range communication unit 18, and a storage / readout. It has a part 19. Each of these units is a function realized by any one of the constituent elements shown in FIG. 11 being operated by a command from the CPU 111 according to a program for the special imaging device developed from the SRAM 113 onto the DRAM 114, or Means.

  Further, the special imaging device 1 includes a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG.

(Functional configuration of special imaging device)
Next, each functional configuration of the special imaging device 1 will be described in more detail with reference to FIGS. 11 and 14.

  The reception unit 12 of the special imaging apparatus 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 illustrated in FIG. 11 and receives operation input from the user.

  The imaging unit 13 is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. obtain. As shown in FIGS. 3A and 3B, the captured image data is two hemispherical image data that is the basis of the omnidirectional image data.

  The sound collecting unit 14 is realized by the processing of the microphone 108 and the sound processing unit 109 and the CPU 111 shown in FIG. 11 and collects sounds around the special imaging device 1.

  The image / sound processing unit 15 is realized mainly by a command from the CPU 111, and performs various processes on the captured image data obtained by the imaging unit 13 or the sound data obtained by the sound collecting unit 14. For example, the image / sound processing unit 15 performs equirectangular projection image data based on two hemispherical image data (see FIGS. 3A and 3B) obtained by the two image sensors 103a and 103b, respectively. (See FIG. 3C).

  The determination unit 17 is realized by the processing of the CPU 111 and makes various determinations.

  The short-range communication unit 18 is mainly realized by the processing of the CPU 111, the communication unit 117, and the antenna 117a, and communicates with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. Can do.

  The storage / reading unit 19 is realized mainly by the processing of the CPU 111 illustrated in FIG. 11, and stores various data (or information) in the storage unit 1000 or stores various data (or information) from the storage unit 1000. Read out.

<Functional configuration of general imaging device>
Next, the functional configuration of the general imaging device 3 will be described in detail with reference to FIGS. 12 and 14. As illustrated in FIG. 14, the general imaging device 3 includes a reception unit 32, an imaging unit 33, a sound collection unit 34, an image / sound processing unit 35, a display control unit 36, a determination unit 37, and a short-range communication unit 38. And a storage / reading unit 39. Each of these units is a function realized by any of the constituent elements shown in FIG. 12 being operated by a command from the CPU 311 according to a program for the special imaging device developed from the SRAM 313 to the DRAM 314, or Means.

  Further, the general imaging device 3 has a storage unit 3000 constructed by the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG. 12.

(Functional configuration of general imaging device)
The reception unit 32 of the general imaging device 3 is mainly realized by the processing of the operation unit 315 and the CPU 311 illustrated in FIG. 12, and receives operation input from the user.

  The imaging unit 33 is mainly realized by the processing of the imaging unit 301, the image processing unit 304, the imaging control unit 305, and the CPU 311 illustrated in FIG. 12, and images a subject, a landscape, and the like, and captures captured image data. obtain. This captured image data is planar image data captured by the perspective projection method.

  The sound collection unit 34 is realized by the processing of the microphone 308 and the sound processing unit 309 and the CPU 311 illustrated in FIG. 12 and collects sounds around the general imaging device 3.

  The image / sound processing unit 35 is realized mainly by a command from the CPU 311, and performs various processes on the captured image data obtained by the imaging unit 33 or the sound data obtained by the sound collection unit 34.

  The display control unit 36 is realized by the processing of the CPU 311 illustrated in FIG. 12, and causes the display 319 to display the planar image P related to the captured image data during or after imaging.

  The determination unit 37 is realized by the processing of the CPU 311 and performs various determinations. For example, the determination unit 37 determines whether the user has pressed the shutter button 315a.

  The short-range communication unit 38 is mainly realized by the CPU 311, the communication unit 317, and the antenna 317 a, and can communicate with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. .

  The storage / reading unit 39 is mainly realized by the processing of the CPU 311 illustrated in FIG. 12, and stores various data (or information) in the storage unit 3000 or stores various data (or information) from the storage unit 3000. Read out.

<Functional configuration of smartphone>
Next, the functional configuration of the smartphone 5 will be described in detail with reference to FIGS. 13 to 16. As illustrated in FIG. 14, the smartphone 5 includes the long-distance communication unit 51, the reception unit 52, the imaging unit 53, the sound collection unit 54, the image / sound processing unit 55, the display control unit 56, and the determination. Unit 57, short-range communication unit 58, and storage / reading unit 59. Each of these units is a function or means realized by any of the constituent elements shown in FIG. 13 being operated by a command from the CPU 501 according to the smartphone 5 program expanded from the EEPROM 504 to the RAM 503. is there.

  The smartphone 5 includes a storage unit 5000 configured by the ROM 502, the RAM 503, and the EEPROM 504 illustrated in FIG. In this storage unit 5000, a cooperative imaging device management DB 5001 is constructed. This cooperative imaging device management DB 5001 is configured by the cooperative imaging device management table of FIG. FIG. 15A is a conceptual diagram of a cooperative imaging device management table.

(Cooperative imaging device management table)
Next, the cooperative imaging apparatus management table will be described with reference to FIG. As illustrated in FIG. 15A, for each imaging device, related relationship information indicating a cooperative relationship between the imaging devices, the IP address of the imaging device, and the device name of the imaging device are managed in association with each other. Among these, the related relationship information is that one imaging device that starts imaging when the shutter of its own device is pressed is “main”, and imaging is performed in response to the shutter being pressed by the “main” imaging device. Another imaging device to be started is indicated as “sub”. Note that the IP address is for Wi-Fi communication, and in the case of wireless communication using Bluetooth instead of the manufacturer ID (Identification) and product ID in the case of communication using a USB wired cable. Replaces BD (Bluetooth Device Address).

(Each smartphone functional configuration)
The long-distance communication unit 51 of the smartphone 5 is realized mainly by the processing of the long-distance communication circuit 511 and the CPU 501 shown in FIG. 13, and other devices (special imaging device 1) via a communication network such as the Internet. , General imaging device 3 etc.), various data (or information) is transmitted / received to / from a smartphone or server.

  The accepting unit 52 is realized mainly by the processes of the touch panel 521 and the CPU 501 and accepts various selections or inputs from the user. The touch panel 521 may be shared with the display 517. Further, input means (buttons) other than the touch panel may be used.

  The imaging unit 53 is realized mainly by the processing of the CMOS sensors 505 and 512 and the CPU 501 shown in FIG. 13, and images a subject, a landscape, and the like to obtain captured image data. This captured image data is planar image data captured by the perspective projection method.

  The sound collection unit 54 is realized by the processing of the microphone 514 and the CPU 501 illustrated in FIG. 13 and collects sounds around the smartphone 5.

  The image / sound processing unit 55 is realized mainly by a command from the CPU 501 and performs various processes on the captured image data obtained by the imaging unit 53 or the sound data obtained by the sound collecting unit 54. Further, the image / sound processing unit 55 uses the superimposed display metadata to set each lattice area LA0 of the planar image P described later to the position indicated by the position parameter, the brightness value indicated by the correction parameter, and By matching the color value, the planar image P is superimposed on the omnidirectional image CE.

  The display control unit 56 is realized by the processing of the CPU 501 illustrated in FIG. 13, and causes the display 517 to display the planar image P related to the captured image data during or after the imaging by the imaging unit 53. The display control unit 56 displays the omnidirectional image CE on which the planar image P is superimposed on the display. Note that the plane image P is superimposed on the omnidirectional image CE without synthesizing the plane image P with the omnidirectional image CE, because the plane image P is not limited to one display form. For example, changing the projection method for display).

  The position parameter is an example of “position information”. The correction parameter is an example of “correction information”.

  The determination unit 57 is realized by the processing of the CPU 501 shown in FIG. 13 and makes various determinations.

  The near field communication unit 58 is mainly realized by the processing of the CPU 501, the near field communication circuit 519 and the antenna 519 a, and the near field communication unit 18 of the special imaging device 1, the near field communication unit 38 of the general imaging device 3, and the like. It is possible to communicate by a short-range wireless communication technology such as Wi-Fi.

  The storage / reading unit 59 is mainly realized by the processing of the CPU 501 shown in FIG. 13, and stores various data (or information) such as superimposed display metadata in the storage unit 5000 or from the storage unit 5000. Various data (or information) such as superimposed display metadata is read out. The storage / reading unit 59 serves as an acquisition unit that acquires various data from the storage unit 5000.

(Detailed functional configuration of image / sound processor)
Here, each functional configuration of the image / sound processor 55 will be described in detail with reference to FIG. FIG. 16 is a detailed functional block diagram of the image / sound processor.

  The image / sound processing unit 55 is roughly divided into a metadata creation unit 55a that performs encoding and a superimposition unit 55b that performs decoding. The metadata creation unit 55a executes a process of step 22 described later shown in FIG. Further, the superimposing unit 55b executes a process of step 23 described later shown in FIG.

{Each functional configuration of the metadata creation unit}
First, each functional configuration of the metadata creation unit 55a will be described. The metadata creation unit 55a includes an extraction unit 550, a first corresponding region calculation unit 552, a gaze point specification unit 554, a projection method conversion unit 556, a second corresponding region calculation unit 558, a region division unit 560, and a projection method reverse conversion. A unit 562, a shape conversion unit 564, a correction parameter creation unit 566, and a superimposed display metadata creation unit 570. Note that the shape conversion unit 564 and the correction parameter creation unit 566 are not necessary when it is not necessary to correct brightness and color. Reference numerals indicating images and regions described below are shown in FIG. FIG. 20 is a conceptual diagram of an image in the process of creating a superimposed display parameter.

  The extraction unit 550 extracts feature points based on local features of each image. A local feature is a pattern or structure that can be seen in an image, such as an edge or a blob, and is a feature value obtained by quantifying a local feature. In the present embodiment, the extraction unit 550 extracts each feature point from different images. The two images used by the extraction unit 550 may have different projection methods as long as the distortion is not significantly large. For example, the extraction unit 550 includes a rectangular equirectangular projection image EC obtained by the equirectangular projection method, a rectangular planar image P obtained by the perspective projection method, and the planar image P. It is used between the peripheral region image PI after being converted by the method conversion unit 556. The equirectangular projection method is an example of the first projection method, and the perspective projection method is an example of the second projection method. The equirectangular projection image is an example of a first projection image, and the planar image P is an example of a second projection image.

  The first corresponding region calculation unit 552 first obtains each feature amount fv1 based on the plurality of feature points fp1 in the equirectangular projection image EC, and each feature amount based on the plurality of feature points fp2 in the planar image P. Find fv2. Several methods have been proposed as a description method of the feature amount. In the present embodiment, it is desirable that the feature amount is invariable or robust to scale and rotation. The first corresponding area calculation unit 552 is based on the similarity between the feature quantity fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature quantity fv2 of the plurality of feature points fp2 in the planar image P. By calculating the corresponding points between the images, and calculating the homography corresponding to the planar image P in the equirectangular projection image EC from the calculated relationship between the corresponding points, and using this homography for conversion, A first homography transformation is performed. As a result, the first corresponding area calculation unit 552 calculates the first corresponding area CA1. In this case, if the center point CP1 of a quadrangle (rectangle) composed of four vertices of the planar image P is used, the center point CP1 is converted into the gazing point GP1 in the equirectangular projection image EC by the first homography conversion matrix. If the vertex coordinates of the four vertices of the planar image P are p1 = (x1, y1), p2 = (x2, y2), p3 = (x3, y3), and p4 = (x4, y4), the first The corresponding region calculation unit 552 can determine the center point CP1 (x, y) based on (Expression 2) shown below.

(Formula 2)
S1 = {(x4-x2) * (y1-y2)-(y4-y2) * (x1-x2)} / 2
S2 = {(x4-x2) * (y2-y3)-(y4-y2) * (x2-x3)} / 2
x = x1 + (x3-x1) * S1 / (S1 + S2)
y = y1 + (y3-y1) * S1 / (S1 + S2)
In FIG. 20, the image shape of the planar image P is a rectangle, but the center point can be calculated even for partial images of various rectangles such as a square, a trapezoid, and a rhombus by using the intersections of diagonal lines. When the image shape of the planar image P is limited to a rectangle or a square, the midpoint of the diagonal line may be set as the center point CP1 in order to omit the calculation. (Equation 3) in the case of calculating the midpoint of the diagonal line p1p3 is shown below.
(Formula 3)
x = (x1 + x3) / 2
y = (y1 + y3) / 2
The gazing point specifying unit 554 specifies a point (referred to as “gazing point” in the present embodiment) on the equirectangular projection image EC where the center point CM1 of the planar image P is located after the first homography transformation.
By the way, since the coordinates of the gazing point GP1 are coordinates on the equirectangular projection image EC, it is convenient to convert and standardize them into expressions of latitude and longitude. Specifically, the vertical direction of the equirectangular projection image EC is expressed as latitude coordinates from −90 degrees (−0.5π) to +90 degrees (+ 0.5π), and the horizontal direction from −180 degrees (−π). Expressed as longitude coordinates of +180 degrees (+ π). In this way, pixel position coordinates corresponding to the image size of the equirectangular projection image EC can be calculated from the latitude / longitude coordinates.

  The projection method conversion unit 556 generates the peripheral region image PI by converting the peripheral region PA around the gazing point GP1 in the equirectangular projection image EC into the same perspective projection method as that of the planar image P. In this case, the point after the gazing point GP1 is converted is a center point CP2, and a square shape that can be specified when the same field angle as the diagonal field angle α of the planar image P is a vertical field angle (or horizontal field angle). A peripheral area PA that is an area before the projection method conversion of the peripheral area image PI is specified. This will be described in more detail below.

(Projection method conversion)
The projection method conversion will be described. As described with reference to FIGS. 3 to 5, the omnidirectional image CE is created by covering the solid sphere CS with the equirectangular projection image EC. Therefore, each pixel data of the equirectangular projection image EC can correspond to each pixel data on the surface of the solid sphere CS of the three-dimensional omnidirectional image. Therefore, the conversion formula by the projection method conversion unit 556 expresses coordinates in the equirectangular projection image EC as (latitude, longitude) = (ea, aa), and coordinates on the three-dimensional solid sphere CS as orthogonal coordinates (x , Y, z), it can be expressed by the following (formula 4).
(x, y, z) = (cos (ea) × cos (aa), cos (ea) × sin (aa), sin (ea)) (Equation 4)
However, the radius of the solid sphere CS at this time is 1.

On the other hand, the planar image P which is a perspective projection image is a two-dimensional image. When this is expressed by two-dimensional polar coordinates (radial radius, declination) = (r, a), the radial radius r is a diagonal field angle. The range corresponding to α is 0 ≦ r ≦ tan (diagonal angle of view α / 2). Further, when the planar image P is represented by a two-dimensional orthogonal coordinate system (u, v), the conversion relationship with polar coordinates (radial radius, declination) = (r, a) is represented by the following (formula 5). Can do.
u = r × cos (a), v = r × sin (a) (Equation 5)
Next, considering that (Equation 5) corresponds to three-dimensional coordinates (radial radius, polar angle, azimuth angle), since only the surface of the solid sphere CS is considered, the radial radius in the three-dimensional polar coordinate is “ 1 ”. Further, the projection for perspective projection conversion of the equirectangular projection image EC pasted on the surface of the three-dimensional sphere CS is considered to have the virtual camera IC at the center of the three-dimensional sphere CS. ) = (R, a) can be expressed by the following (Expression 6) and (Expression 7).
r = tan (polar angle) (Formula 6)
a = Azimuth angle (Formula 7)
If the polar angle is t, then t = arctan (r), so the three-dimensional polar coordinates (radial radius, polar angle, azimuth) are (radial radius, polar angle, azimuth) = (1, arctan (r ), a). A conversion formula for converting from the three-dimensional polar coordinate to the orthogonal coordinate system (x, y, z) can be expressed by the following (formula 8).
(x, y, z) = (sin (t) x cos (a), sin (t) x sin (a), cos (t)) (Equation 8)
By the above (Equation 8), mutual conversion between the equirectangular projection image EC by the equirectangular projection method and the planar image P by the perspective projection method can be performed. That is, by using the radius r corresponding to the diagonal angle of view α of the planar image P to be created, the conversion map indicating which coordinate of the equirectangular projection image EC corresponds to each pixel of the planar image P. The coordinates can be calculated, and the peripheral area image PI, which is a perspective projection image, can be created from the equirectangular projection image EC based on the conversion map coordinates. The above-described projection conversion is performed such that the position where the (latitude, longitude) of the equirectangular projection image EC is (90 °, 0 °) is the center point CP2 of the peripheral region image PI that is a perspective projection image. Is shown. Therefore, when performing perspective projection conversion using an arbitrary point of the equirectangular projection image EC as a gazing point, by rotating the solid sphere CS to which the equirectangular projection image EC is attached, the coordinates of the gazing point (latitude, Coordinate rotation may be performed such that (longitude) is arranged at a position of (90 °, 0 °). Since the conversion formula relating to the rotation of the solid sphere CS is a general coordinate rotation formula, description thereof is omitted.

(Identification of peripheral area image PI)
Next, a method for specifying the area of the peripheral area image PI will be described with reference to FIG. FIG. 21 is a conceptual diagram when a peripheral region image is specified.

  When the first corresponding region calculation unit 552 determines the similarity between the plurality of feature points in the planar image P and the plurality of feature points in the peripheral region image PI, the second corresponding region CA2 is included in the peripheral region image PI. It should be included as widely as possible. Therefore, if the peripheral area image PI is set to a wide area, the situation where the second corresponding area CA2 is not included does not occur. However, if the peripheral area image PI is set to an excessively large area, the number of pixels for which the similarity is calculated increases, and thus processing takes time. Therefore, it is preferable that the area of the peripheral area image PI is as small as possible within the range including the second corresponding area CA2. Therefore, in the present embodiment, the peripheral area image PI is specified by the following method.

In the present embodiment, the 35 mm equivalent focal length of the planar image is used for specifying the peripheral area image PI, and this is obtained from the Exif data recorded at the time of imaging. Since the 35 mm equivalent focal length is a focal length based on a so-called 24 mm × 36 mm film size, the following calculation formulas (Equation 9) and (Equation 10) correspond to the diagonal of the film and the focal length. The angle of view can be calculated.
Film diagonal = sqrt (24 * 24 + 36 * 36) (Equation 9)
Image angle for superimposition / 2 = arctan ((film diagonal / 2) / 35mm equivalent focal length for superimposition image) ... (Equation 10)
By the way, the image covering this angle of view is circular, but since the actual image sensor (film) is rectangular, it is a rectangular image inscribed in the circle. In the present embodiment, the vertical field angle α of the peripheral area image PI is set to be the same as the diagonal field angle α of the planar image P. As a result, the peripheral area image PI shown in FIG. 21B becomes a square circumscribing a circle covering the diagonal angle of view α of the planar image P shown in FIG. The angle α can be calculated from the square diagonal and the focal length of the planar image P as shown in the following (Expression 11) and (Expression 12).
Square diagonal = sqrt (film diagonal * film diagonal + film diagonal * film diagonal) ... (Formula 11)
Vertical angle of view α / 2 = arctan ((square diagonal / 2) / 35mm equivalent focal length of plane image P)) ... (Formula 12)
By performing the projection method conversion at such a vertical angle of view α, the peripheral region image that can cover the image at the diagonal angle of view α of the planar image P as wide as possible with the gazing point as the center, and the vertical angle of view α is not too large. PI (perspective projection image) can be created.

(Calculation of position parameter)
Subsequently, returning to FIG. 16, the second corresponding region calculation unit 558 calculates feature amounts fv3 of the plurality of feature points fp2 in the planar image P and the plurality of feature points fp3 in the peripheral region image PI. ← Fv2 is issued. Corresponding points between images are calculated based on the calculated similarities of the feature quantities fv2 and fv3, and a homography corresponding to the planar image P is calculated in the peripheral region image PI from the calculated relationship between the corresponding points between the images. The second homography conversion is performed by using this homography for the conversion. As a result, the second corresponding area calculation unit 558 calculates the second corresponding area CA2.

  Note that, before the first homography conversion, the image size of at least one of the planar image P and the equirectangular projection image EC may be resized in order to advance the calculation time of the first homography. For example, if the planar image P is 40 million pixels and the equirectangular projection image EC is 30 million pixels, the planar image P is resized to 30 million pixels, or each image is adjusted so that both images become 10 million pixels. Or resizing. Similarly, the image size of at least one of the planar image P and the peripheral area image PI may be resized before calculating the second homography transformation matrix.

  Further, the homography transformation matrix in the present embodiment is a transformation matrix that represents the projection relationship between the equirectangular projection image EC and the plane image P, and by multiplying the coordinates in the plane image P by the homography transformation matrix, Corresponding coordinates on the equirectangular projection image EC (the omnidirectional image CE) can be calculated. The area dividing unit 560 divides a partial area in the image into a plurality of lattice areas. Here, a method of dividing the second corresponding region into a plurality of lattice regions will be described in detail with reference to FIG. FIG. 22 is a conceptual diagram when the second corresponding region is divided into a plurality of lattice regions.

  As shown in FIG. 22A, the area dividing unit 560 has four vertices of the vertex coordinates of the second corresponding area CA2 calculated by the first corresponding area calculation unit 552 by the second homography transformation. Is divided into a plurality of lattice areas LA2, as shown in FIG. 22 (b). For example, it is equally divided by 30 in the horizontal direction and 20 in the vertical direction.

  Next, a specific method for dividing the plurality of lattice areas LA2 will be described.

First, a calculation formula for equally dividing the second corresponding area CA2 is shown. When the two points are A (X1, Y1) and B (X2, Y2) and the line segment between the two points is divided into n pieces at equal intervals, the coordinates of the point Pm corresponding to the mth point from A are as follows: Calculated by (Equation 13).
Pm = (X1 + (X2−X1) × m / n, Y1 + (Y2−Y1) × m / n) (Equation 13)
Since the coordinates obtained by equally dividing the line segment can be calculated by the above (Formula 13), the coordinates obtained by dividing the upper side and the lower side of the quadrangle may be obtained, and the line segment formed by the divided coordinates may be further divided. When the coordinates of the upper left, upper right, lower right, and lower left of the quadrangle are TL, TR, BR, and BL, respectively, the coordinates obtained by equally dividing the line segment TL-TR and the line segment BR-BL into 30 are obtained. Next, in each of the coordinates divided from the 0th to the 30th, coordinates obtained by equally dividing 20 into the line segments composed of the coordinates corresponding to the same order are obtained. Thereby, the coordinates for dividing the quadrangular area into 30 × 20 small areas can be obtained. In FIG. 22B, TR coordinates (LO _00,00 , LA _00,00 ) are shown as an example of coordinates.

  Subsequently, returning to FIG. 16 and FIG. 20, the projection method inverse conversion unit 562 reversely converts the projection method of the second corresponding area CA2 to the equirectangular projection method same as the equirectangular projection image EC, In the equirectangular projection image EC, a third corresponding area CA3 corresponding to the second corresponding area CA2 is calculated. Specifically, the projection method inverse transform unit 562 calculates a third corresponding area CA3 including each lattice area LA3 corresponding to the plurality of lattice areas LA2 in the second corresponding area CA2 in the equirectangular projection image EC. To do. The third corresponding area CA is shown in FIG. 20, but is also shown in FIG. 23 as an enlarged view. FIG. 23 is a conceptual diagram showing a third corresponding region in the equirectangular projection image EC. Thereby, the planar image P is superimposed and displayed on the omnidirectional image CE created by the equirectangular projection image EC so as to be finally matched (mapped) with the third corresponding area CA3. By the process of the projection method inverse transform unit 562, a position parameter indicating the coordinates of each grid point in each grid area LA3 is created. The positional parameters are shown in FIGS. 17 and 18 (b). Note that the grid point is an example of a plurality of points.

  As described above, the positional relationship between the equirectangular projection image EC and the planar image P can be calculated by creating the positional parameters.

(Calculation of correction parameters)
Although the position parameter is calculated as described above, when the superimposed display is performed as it is, when the equirectangular projection image EC and the planar image P are greatly different in brightness and color, an unnatural superimposed display may occur. is there. Therefore, the shape conversion unit 564 and the correction parameter creation unit 566 described below play a role of preventing unnatural superimposition display even when the brightness and color tone are greatly different.

  Prior to color matching described later, the shape conversion unit 564 projects the four vertices of the second corresponding area CA2 onto the four vertices of the planar image P, so that the second corresponding area CA2 has the same shape as the planar image P. Convert. Specifically, in order to match each lattice area LA2 of the second corresponding area CA2 shown in FIG. 24 (a) with each lattice area LA0 of the planar image P shown in FIG. 24 (c). The shape of the second corresponding area CA2 is converted into the same shape as the planar image P. As a result, the shape of the second corresponding area CA2 shown in FIG. 24A is converted into the second corresponding area CA2 ′ shown in FIG. Accordingly, since the lattice area LA2 is converted into the lattice area LA2 ′, it has the same shape as the lattice area LA0 of the planar image P.

The correction parameter creation unit 566 has a planar image P having the same shape as each grid area LA2 ′ with respect to the brightness value and color value of each grid area LA2 ′ in the second corresponding area CA2 ′ converted into the same shape. A correction parameter for matching the brightness value and the color value of each lattice area LA0 is created. Specifically, the correction parameter creation unit 566 calculates brightness values and color values (R, G, B) of all the pixels in the four grid areas having one grid point LA0 as a grid point in common. An average value a = (Rave, Gave, Bave) is calculated, and brightness values and color values (R ′, R ′, R) of all the pixels in the four grid areas having one grid point LA2 ′ as a grid point in common. The average value a ′ = (R′ave, G′ave, B′ave) of G ′, B ′) is calculated. When one point of each of the lattice points LA0 and one point of each of the lattice points LA2 ′ is the positions of the four corners of the second corresponding area CA2 and the third corresponding area CA3, the correction parameter creating unit 566 has 1 The average value a and the average value a ′ of the brightness value and the color value are calculated from the two grid areas. Further, when one point of each of the lattice points LA0 and one point of each of the lattice points LA2 ′ is an outer peripheral position of the second corresponding area CA2 and the third corresponding area CA3, the correction parameter creating unit 566 The average value a and the average value a ′ of the brightness value and the color value are calculated from the two grid areas. In this embodiment, the correction parameter is gain data for correcting the brightness value and the color value of the planar image P. Therefore, as shown in the following (Expression 14), the average value a ′ Is divided by the average value a to obtain the correction parameter Pa.
Pa = a '/ a (Formula 14)
Thereby, in the superimposed display described later, the brightness value and the color tone of the planar image P are multiplied by the equirectangular projection image EC (the omnidirectional image CE) by multiplying the gain value indicated by the correction parameter for each lattice area LAO. ) And the pixel value are close to each other, and the display can be superimposed and displayed without a sense of incongruity. The correction parameter is not only calculated from the average value, but may be calculated using a median value, a mode value, or the like instead of or in addition to the average value.

  In the present embodiment, pixel values (R, G, B) are used for calculation of correction values for brightness values and color values. However, the YUV format for luminance and color difference signals, the JPEG sYCC (YCbCr) format, etc. Using the luminance value and color difference value in, the average value of the luminance value and color difference value of all the pixels constituting the grid area is obtained in the same way, and the average value is divided to correct a correction parameter for superimposed display described later. It is good. The method of converting RGB values to YUV and sYCC (YCbCr) is well known and will not be described in detail. However, using (Equation 15) as a reference, the JPEG compressed image format JFIF (JPEG file interchange format) format is used. Here is an example of converting from RGB to YCbCr.

The superimposed display metadata creation unit 570 uses the position parameter, the correction parameter, and the like to superimpose the position when the planar image P is superimposed on the omnidirectional image CE, and the correction value of the brightness value and the color value. Create display metadata.
(Superimposed display metadata)
Next, the data structure of the superimposed display metadata will be described with reference to FIG. FIG. 17 shows the data structure of the superimposed display metadata.

  As shown in FIG. 17, the superimposed display metadata is composed of equirectangular projection image information, planar image information, superimposed display information, and metadata creation information.

  Among these, the equirectangular projection image information is information sent together with the captured image data from the special imaging device 1. The equirectangular projection image information includes image identification information and attached information. The image identification information in the equirectangular projection image information is image identification information for identifying the equirectangular projection image. In FIG. 17, the file name of the image is shown as an example of the image identification information in the equirectangular projection image information, but an image ID for identifying the image may be used.

  The attached information in the equirectangular projection image information is related information attached to the equirectangular projection image information. In FIG. 17, posture correction information (Pitch, Yaw, Roll) of equirectangular projection image data obtained when captured by the special imaging device 1 is shown as an example of attached information. This posture correction information may be stored in Exif, which is standardized as the image recording format of the special imaging device 1, and stored in various formats provided by GPano (Google Photo Sphere schema) There is also. The omnidirectional image has the feature that if it is taken at the same position, an image of 360 ° omnidirectional image can be taken even if the posture is different. The display position is not determined unless the center of the image is specified (gaze point). Therefore, generally, the omnidirectional image CE is corrected and displayed so that the zenith is directly above the photographer. As a result, a natural display in which the horizon is straightly corrected is possible.

  Next, the planar image information is information transmitted from the general imaging device 3 together with the captured image data. The planar image information includes image identification information and attached information. The image identification information in the planar image information is image identification information for identifying the planar image P. In FIG. 17, the file name of the image is shown as an example of the image identification information, but an image ID for identifying the image may be used.

  The attached information in the planar image information is related information attached to the planar image information. In FIG. 17, the value of the 35 mm equivalent focal length is shown as an example of the attached information in the planar image information. The value of the 35 mm equivalent focal length is not essential for displaying the celestial sphere image CE by superimposing the planar image P, but serves as reference information for determining the angle of view to be displayed when superimposing display is performed. Take as an example.

  Next, the superimposed display information is information created by the smartphone 5, and includes information on the number of area divisions, positions of lattice points in each lattice area (position parameters), and correction values (correction parameters) of brightness values and color values. Is included. Among these, the area division number information indicates the number of divisions in the horizontal (longitude) direction and the number of divisions in the vertical (latitude) direction when dividing the first corresponding area CA1 into a plurality of lattice areas.

  The position parameter is a vertex indicating at which position in the equirectangular projection image EC (global celestial image CE) each lattice point when the planar image P is divided into a plurality of lattice-like regions is arranged. Mapping information. In this embodiment, the correction parameter is gain data for correcting the color value of the planar image P. Since the correction target may be a monochrome image, the correction parameter is a parameter for adjusting at least the brightness value among the brightness value and the color value.

  When the omnidirectional image CE is obtained, if the perspective projection method, which is the same projection method as the planar image P, is used, 360 ° omnidirectional imaging cannot be performed. Therefore, even if an attempt is made to superimpose a planar image P obtained by imaging separately from the omnidirectional image CE on a partial area of the omnidirectional image CE, the equirectangular projection image EC and the planar image P are projected. Since the method is different, the equirectangular projection image EC (the omnidirectional image CE) and the planar image P do not match, and the planar image P does not merge well with the omnidirectional image CE. A corner image is often created by an equirectangular projection method as one of the projection methods. However, if the equirectangular projection method is used, the length in the horizontal direction increases as the distance from the standard parallel as in the so-called Mercator projection, so it is greatly different from the image of the perspective projection method used in general cameras. The image will be different. Therefore, even if only the scales of the images are changed and superimposed, the images do not match, and the planar image P does not merge well with the omnidirectional image CE. Therefore, in the present embodiment, the position parameter is created by the processing shown in FIG.

  Here, the position parameter and the correction parameter will be described in detail with reference to FIG. FIG. 18A is a conceptual diagram showing each lattice region in the second corresponding region. FIG. 18B is a conceptual diagram showing each lattice region in the third corresponding region.

As shown in FIG. 18A, the first corresponding area CA1 that is a partial area of the equirectangular projection image EC is converted into the same perspective projection method as that of the planar image P. In the present embodiment, the second corresponding area CA2 is divided into a plurality of lattice areas having a horizontal division number of 30 and a vertical division number of 20. FIG. 18A shows the coordinates (LO _00,00 , LA _00,00 ), (LO _01,00 , LA _01,00 ),..., (LO _30,20 , LA ₃₀ ) of each lattice area. _{, 20} ), and correction values (R _00,00 , G _00,00 , B _00,00 ), (R _01,00 , G _01,00 , B _01,00 ), ..., (R _30,20 , G _30,20 , B _30,20 ) are shown. In order to simplify the drawing, the coordinates of the grid points at the four vertices and the correction values of the brightness value and the color value are shown, but in reality, the coordinates of all the grid points, the brightness values, and There is a correction value for the color value. Further, the correction values R, G, and B of the brightness value and the color value respectively indicate a red correction gain, a green correction gain, and a blue correction gain. The correction values R, G, and B of each brightness value and color value are actually the brightness of an image within a predetermined range centered on the coordinates of each grid point (a range that does not overlap with the predetermined range of adjacent grid points). The correction values for the color value and the color value are shown.

On the other hand, as shown in FIG. 18B, the third corresponding area obtained by inversely transforming the second corresponding area CA2 into the same equirectangular projection method as the equirectangular projection image EC. In the present embodiment, CA3 is similarly divided into a plurality of lattice regions having 30 horizontal divisions and 20 vertical divisions. The FIG. 18 (b), the coordinate of the lattice point of each grid area _{(LO '00,00, LA' 00,00} ), (LO '01,00, LA' 01,00), ..., (LO '30 _{, 20} , LA ′ _30,20 ), and brightness value and color value correction values having the same values as the correction values of the second corresponding area CA2. In this case, too, the coordinates of the grid points, the brightness values, and the correction values of the color values at the four vertices are shown in order to simplify the drawing. There is a correction value for the color value.

  Next, the metadata creation information indicates version information indicating the version of the superimposed display metadata.

  As described above, the position parameter indicates the positional correspondence between the planar image P and the equirectangular projection image EC (the omnidirectional image CE). However, if the position parameter is intended to represent information on which coordinates of all the pixel positions of the planar image P are associated with the equirectangular projection image EC (the omnidirectional image CE), the general imaging device 3 In the case of a digital camera having a high number of pixels, information corresponding to about 40 million pixels must be represented. For this reason, the amount of position parameter data increases, and handling of data storage and the like is difficult. Therefore, the position parameter of the present embodiment is that the plane image P is divided into 600 (30 × 20) regions, so that the coordinates of the lattice points of the plane image are the equirectangular projection image EC (the omnidirectional image CE). ) Shows only data indicating where to correspond. In addition, when the smartphone 5 performs superimposed display, the superimposed display can be realized by interpolating the position of the image in each region from the coordinates of the grid points.

{Functional structure of superimposition unit}
The functional configuration of the superimposing unit 55b will be described with reference to FIG. The superimposing unit 55 b includes a pasting area creating unit 582, a correcting unit 584, an image creating unit 586, an image superimposing unit 588, and a projection method conversion unit 590.

  Among these, the pasting area creation unit 582 creates an area part (hereinafter referred to as “partial solid sphere”) PS corresponding to the third corresponding area CA3 in the virtual solid sphere CS.

  The correction unit 584 performs correction to match the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC based on the correction parameter in the superimposed display metadata. Note that the correction unit 584 does not necessarily have to correct brightness and color. In addition, when the correction parameter is corrected, the brightness may be corrected or the color may not be corrected.

  The image creating unit 586 creates the superimposed image S by pasting the planar image P (or the corrected image C after correcting the planar image P) to the partial solid sphere PS. The image creation unit 586 creates mask data M based on the region of the partial solid sphere PS. Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC to the solid sphere CS.

  Here, it is transmission ratio data that can be used when the superimposed image S is superimposed on the omnidirectional image CE. The mask data M is used to gradually bring the brightness and color around the boundary when the superimposed image S is superimposed on the omnidirectional image CE from the inner superimposed image S side to the outer omnidirectional image CE side. The transmittance around the mask gradually increases from the superimposed image S to the omnidirectional image CE from the inside to the outside. Thereby, even if the superimposition image S is superimposed on the omnidirectional image CE, it is possible to prevent the superimposition from being recognized as much as possible. Note that the creation of the mask data M is not essential.

  The image superimposing unit 588 superimposes the superimposed image S and the mask data M on the omnidirectional image CE. Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  As shown in FIG. 7, the projection method conversion unit 590 generates a superimposed image based on a predetermined line-of-sight direction of the virtual camera IC (a center point CP of the predetermined area T) and an angle of view α of the predetermined area T. The projection method is converted so that the predetermined area T in the omnidirectional image CE with S superimposed can be seen on the display 517. The projection method conversion unit 590 also performs processing for matching the predetermined area T with the resolution of the display area on the display 517 when performing the projection method conversion. Specifically, when the resolution of the predetermined area T is smaller than the resolution of the display area of the display 517, the projection method conversion unit 590 enlarges the predetermined area T so as to match the display area of the display 517. On the other hand, when the resolution of the predetermined area T is larger than the resolution of the display area of the display 517, the projection method conversion unit 590 reduces the predetermined area T so as to match the display area of the display 517. Thereby, the display control part 56 can display the predetermined area image Q which shows the predetermined area T over the whole display area of the display 517 (step S24). Here, a superimposed image S that is a planar image P ′ in a state where the planar image P is superimposed is included in the predetermined region image Q.

<< Processing or Operation of Embodiment >>
Subsequently, processing or operation of the present embodiment will be described with reference to FIGS. 19 to 30. First, an imaging method executed by the imaging system will be described with reference to FIG. FIG. 19 is a sequence diagram illustrating an imaging method. In the following description, a case where a subject, a landscape, or the like is imaged will be described. However, ambient sounds may be recorded by the sound collecting unit 14 simultaneously with the imaging.

  As illustrated in FIG. 19, the reception unit 52 of the smartphone 5 receives the cooperative imaging start from the user (Step S <b> 11). In this case, as illustrated in FIG. 15B, the display control unit 56 causes the display 517 to display the cooperative imaging device setting screen illustrated in FIG. On this screen, a radio button for designating a main imaging device for cooperative imaging and a check box for designating (selecting) a sub imaging device for cooperative imaging are displayed for each imaging device. Yes. Furthermore, the device name of the imaging device and the reception intensity of the radio wave are displayed for each imaging device. Then, when the user designates (selects) a desired imaging device as a main and a sub and presses a “confirm” button, the accepting unit accepts the start of cooperative imaging. Since there may be a plurality of sub imaging devices, a plurality of imaging devices can be designated (selected) using check boxes.

  And the near field communication part 58 of the smart phone 5 transmits the imaging start inquiry information which shows the inquiry of an imaging start by polling with respect to the near field communication part 38 of the general imaging device 3 (step S12). Thereby, the near field communication part 38 of the general imaging device 3 receives imaging start inquiry information.

  Next, the determination unit 37 of the general imaging device 3 determines whether or not imaging has been started based on whether or not the reception unit 32 receives pressing of the shutter button 315a from the user (step S13).

  Next, the short-range communication unit 38 of the general imaging device 3 transmits response information indicating the response content according to the determination result in step S13 to the smartphone 5 (step S14). If it is determined in step S13 that imaging has started, the response information includes imaging start information indicating the start of imaging. In this case, the response information includes image identification information of the general imaging device 3. On the other hand, when it is not determined in step S13 that imaging has started, the response information includes imaging standby information indicating imaging standby. Thereby, the near field communication part 58 of the smart phone 5 receives response information.

  Subsequently, a case where it is determined in step S13 that imaging has started and a case where the response information received in step S14 includes imaging start information will be described.

  First, the general imaging device 3 starts imaging (step S15). In this image capturing process, after the shutter button 315a is pressed, the image capturing unit 33 captures an image of a subject, a landscape, and the like, thereby capturing image data (planar image data in this case), and the storage / reading unit 39 stores the storage unit 3000. This is the process until the captured image data is stored.

  Next, in the smartphone 5, the short-range communication unit 58 transmits imaging start request information indicating an imaging start request to the special imaging device 1 (step S16). Thereby, the short-range communication unit 18 of the special imaging device 1 receives the imaging start request information.

  On the other hand, the special imaging device 1 starts imaging (step S17). In this imaging process, the imaging unit 13 images a subject, a landscape, and the like to generate captured image data (two hemispherical image data as shown in FIGS. 3A and 3B). The sound processing unit 15 creates single equirectangular projection image data as shown in FIG. 3C based on the two hemispherical image data, and the storage / reading unit 19 stores the storage unit 1000. This is the process until the equirectangular projection image data is stored.

  Next, in the smartphone 5, the short-range communication unit 58 transmits captured image request information indicating that a captured image is requested to the general imaging device 3 (step S18). This captured image request information includes the image identification information received in step S14. Thereby, the near field communication part 38 of the general imaging device 3 receives imaging start request information.

  Next, the short-range communication unit 38 of the general imaging device 3 transmits the planar image data obtained in step S15 to the smartphone 5 (step S19). At this time, image identification information for identifying the plane image data to be transmitted, and attached information are also transmitted. The image identification information and the attached information are shown as planar image information in FIG. Thereby, the near field communication part 58 of the smart phone 5 receives plane image data, image identification information, and attached information.

  On the other hand, the short-range communication unit 18 of the special imaging device 1 transmits the equirectangular projection image data obtained in step S17 to the smartphone 5 (step S20). At this time, image identification information for identifying the equirectangular projection image data to be transmitted, and attached information are also transmitted. The image identification information and attached information are shown as equidistant cylindrical projection image information in FIG. Thereby, the near field communication part 58 of the smart phone 5 receives equirectangular projection image data, image identification information, and attached information.

  Next, the storage / reading unit 59 of the smartphone 5 stores the electronic file of planar image data received in step S19 and the electronic file of equirectangular projection image data received in step S20 in the same electronic folder. And it memorize | stores in the memory | storage part 5000 (step S21).

  Next, the image / sound processing unit 55 of the smartphone 5 creates superimposed display metadata used when the planar image P is superimposed and displayed on a partial region of the omnidirectional image CE (step S22). . At this time, the storage / reading unit 59 stores the superimposed display metadata in the storage unit 5000.

Here, the process of creating superimposed display metadata will be described in detail mainly with reference to FIGS. 20 to 24. Note that even if the resolutions of the image pickup devices of the general image pickup device 3 and the special image pickup device 1 are the same, the image pickup device of the special image pickup device 1 generates an equirectangular projection image that is the basis of the 360 ° omnidirectional image CE. Since all must be covered, the definition per fixed area in the captured image is low.
{Process for creating superimposed display metadata}
First, a description will be given of a process for creating superimposition display metadata for superimposing a high-definition plane image P on the omnidirectional image CE created by the low-definition equirectangular projection image EC and displaying it on the display 517. To do. As shown in FIG. 17, the superimposed display metadata includes a position parameter and a correction parameter. Therefore, a method for creating the position parameter and the correction parameter will be mainly described with reference to FIG.

  The extraction unit 550 extracts a plurality of feature points in the rectangular equirectangular projection image EC obtained by the equirectangular projection method and a plurality of feature points in the rectangular planar image P obtained by the perspective projection method ( Step S110).

  Next, the first corresponding region calculation unit 552 performs the first homography conversion based on the similarity between the plurality of feature points in the equirectangular projection image EC and the plurality of feature points in the planar image P, as shown in FIG. In the equirectangular projection image EC, a quadrangular first corresponding area CA1 corresponding to the planar image P is calculated (step S120). More specifically, the first corresponding area calculation unit 552 calculates the feature amounts fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature amounts fv2 of the plurality of feature points fp2 in the planar image P. FIG. 20 shows a first homography transformation obtained by calculating corresponding points between images based on the similarity and obtaining homography corresponding to the planar image P in the equirectangular projection image EC. As described above, in the equirectangular projection image EC, a quadrangular first corresponding area CA1 corresponding to the planar image P is calculated. This process is a process (provisional decision process) for roughly estimating the corresponding position for the time being, although it is impossible to accurately associate the flat image P having a different projection method with the equirectangular projection image EC.

  Next, the gazing point identification unit 554 identifies a point (gaze point GP1) on the equirectangular projection image EC where the center point CM1 of the planar image P is located after the first homography transformation (step S130).

  Next, the projection method conversion unit 556 sets the vertical field angle α of the peripheral area image PI to be the same as the diagonal field angle α of the planar image P as shown in FIG. Thus, the peripheral area PA centered on the gazing point GP1 in the equirectangular projection image EC is converted into the same perspective projection method as that of the planar image P so that the peripheral area image PI can be created (step S140).

  Next, the extraction unit 550 extracts a plurality of feature points in the peripheral area image PI obtained by the projection method conversion unit 556 (step S150).

  Next, the first corresponding region calculation unit 552 performs peripheral region image PI based on the similarity between the plurality of feature points in the planar image P and the plurality of feature points in the peripheral region image PI by the second homography transformation. In step S160, a quadrangular second corresponding area CA2 corresponding to the planar image P is calculated. Note that the planar image P is a high-definition image of 40 million pixels, for example, and is resized to an appropriate size in advance.

  Next, the area dividing unit 560 divides the second corresponding area CA2 into a plurality of lattice areas LA2 as shown in FIG. 22B (step S170).

  Next, as shown in FIG. 20, the projection method inverse conversion unit 562 converts the projection method of the second corresponding area CA2 into the equirectangular projection method same as the equirectangular projection image EC (inverse conversion). (Step S180). Thereby, as shown in FIG. 23, the projection method inverse transform unit 562 starts from each lattice area LA3 corresponding to the plurality of lattice areas LA2 in the second corresponding area CA2 in the equirectangular projection image EC. A third corresponding area CA3 is calculated. FIG. 23 is a conceptual diagram showing a third corresponding region in the equirectangular projection image EC. By the process of the projection method inverse transform unit 562, a position parameter indicating the coordinates of each grid point in each grid area LA3 is created. The position parameter is shown in FIGS. 17 and 18B as described above.

  Next, correction parameter creation processing will be described with reference to FIGS. 20 and 24. FIG. 24 is a conceptual diagram of an image in the process of the correction parameter creation process.

  After the process of step S180, the shape conversion unit 564 projects the second correspondence by projecting the four vertices of the second corresponding area CA2 as illustrated in FIG. The area CA2 is converted into the same shape as the planar image P, and a second corresponding area CA2 ′ as shown in FIG. 24B is obtained (step S190).

  Next, as shown in FIG. 24C, the area dividing unit 560 converts the planar image P into the same number and the same shape as each lattice area LA2 ′ in the second corresponding area CA2 ′ after conversion. Dividing into a plurality of lattice areas LA0 (step S200).

  Next, the correction parameter creation unit 566 performs the brightness value of each grid area LA0 of the planar image P corresponding to each grid area LA2 ′ with respect to the color value of each grid area LA2 ′ in the second corresponding area CA2 ′. Then, a correction parameter for matching the color value is created (step S210).

Finally, as shown in FIG. 17, the superimposed display metadata creation unit 570 includes predetermined distance cylindrical projection image information acquired from the special imaging device 1 and planar image information acquired from the general imaging device 3. The superimposed display metadata is created based on the area division number information, the position parameter created by the projection method inverse transform unit 562, the correction parameter created by the correction parameter creation unit 566, and the metadata creation information (step) S220).
The superimposed display metadata is stored in the storage unit 5000 by the storage / readout unit 59.

  Thus, the process of step S23 shown in FIG. Then, the storage / reading unit 59 and the display control unit 56 perform superimposed display processing using the superimposed display metadata (step S23).

{Superimposed display processing}
Next, the superimposed display process will be described in detail with reference to FIGS. 25 to 30. FIG. 25 is a conceptual diagram of an image in the process of superimposed display processing.

  First, the storage / reading unit 59 (acquisition unit) shown in FIG. 14 obtains from the storage unit 5000 the data of the equirectangular projection image EC obtained by the equirectangular projection method, the perspective projection method. The obtained data of the plane image P and the superimposed display metadata are read out and acquired.

  Next, as shown in FIG. 25, the pasting area creation unit 582 creates a partial solid sphere PS corresponding to the third corresponding area CA3 in the virtual solid sphere CS based on the position parameter (step). S310). At this time, pixels other than the lattice points not indicated by the position parameter are interpolated using, for example, linear interpolation.

  Next, the correction unit 584 performs correction to match the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC based on the correction parameter in the superimposed display metadata. (Step S320). Hereinafter, the corrected planar image P is referred to as “corrected image C”.

  Next, the image creation unit 586 creates the superimposed image S by pasting the corrected image C on the partial solid sphere PS (step S330). At this time, pixels other than the lattice points not indicated by the position parameter are interpolated using, for example, linear interpolation. The image creation unit 586 creates mask data M based on the partial solid sphere PS (step S340). Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC on the solid sphere CS (step S350). Then, the image superimposing unit 588 superimposes the superimposed image S on the omnidirectional image CE using the superimposed image S and the mask data M (step S360). Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  Next, as shown in FIG. 7, the projection method conversion unit 590 is based on the predetermined viewing direction of the virtual camera IC (the center point CP of the predetermined region T) and the angle of view α of the predetermined region T. Then, the projection system conversion is performed so that the predetermined area T in the omnidirectional image CE with the superimposed image S superimposed can be seen on the display 517 (step S370). At this time, the projection method conversion unit 590 also performs processing for matching the predetermined area T with the resolution of the display area on the display 517. Thereby, the display control unit 56 can display the predetermined area image Q indicating the predetermined area T over the entire display area of the display 517. Here, a superimposed image S that is a planar image P ′ in a state where the planar image P is superimposed is included in the predetermined region image Q.

  Next, the state of the superimposed display will be described in detail with reference to FIGS. FIG. 26 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. Here, the case where the planar image P is superimposed on FIG. 5 is shown. As shown in FIG. 26, the high-definition superimposed image S is superimposed on the inside of the solid sphere CS according to the position parameter with respect to the low-definition omnidirectional image CE attached to the solid sphere CS. .

  FIG. 27 is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. FIG. 27 illustrates a state in which the omnidirectional image CE and the superimposed image S are pasted on the solid sphere CS and the image including the superimposed image S is the predetermined region image Q.

  FIG. 28 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the position parameter of the present embodiment. FIG. 29 is a two-dimensional conceptual diagram when a planar image is superimposed on the omnidirectional image using the position parameters of the present embodiment.

  As shown in FIG. 28A, when the virtual camera IC is located at the center point of the solid sphere CS, the subject P1 is represented as an image P2 on the omnidirectional image CE. , Represented as an image P3 on the superimposed image S. As shown in FIG. 28A, since the image P2 and the image P3 are located on a straight line connecting the virtual camera IC and the subject P1, the superimposed image S is superimposed on the omnidirectional image CE. Even if displayed in the state, the omnidirectional image CE and the superimposed image S are not displaced. However, as shown in FIG. 28B, when the virtual camera IC is separated from the center point of the solid sphere CS, the image P2 is located on a straight line connecting the virtual camera IC and the subject P1. The image P3 is located slightly inside. For this reason, when an image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is an image P3 ′, the omnidirectional image CE and the superimposed image S are shifted from each other between the image P3 and the image P3 ′. Deviation of the amount g occurs. Thereby, the superimposed image S is shifted from the omnidirectional image CE and displayed.

  On the other hand, in the present embodiment, since the position parameters indicated by the plurality of lattice regions are used, the superimposed image S is represented as an omnidirectional ball as shown in FIGS. 29 (a) and 29 (b). It can be superimposed along the image CE. Thereby, as shown in FIG. 29A, not only when the virtual camera IC is located at the center point of the solid sphere CS, but also as shown in FIG. Even when the three-dimensional sphere CS is away from the center point, the image P2 and the image P3 are located on a straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superimposed image S is displayed in a state of being superimposed on the omnidirectional image CE, there is no deviation between the omnidirectional image CE and the superimposed image S.

  FIG. 30A shows a display example of a wide image without superimposing display, FIG. 30B shows a display example of a tele image without superimposition display, and FIG. 30C shows a display example of a wide image with superimposition display. FIG. 30D is a conceptual diagram showing a display example of a tele image when superimposed display is performed. Note that the wavy lines in the figure are merely shown for convenience of explanation, and may or may not be actually displayed on the display 517.

  As shown in FIG. 30A, when the planar image P is not superimposed and displayed on the omnidirectional image CE, when the enlarged display is performed up to the area indicated by the wavy line in FIG. As shown in (b), the low-definition image remains as it is, and the user will see an unclear image. On the other hand, as shown in FIG. 30C, when the planar image P is superimposed and displayed on the omnidirectional image CE, it is enlarged to the area indicated by the wavy line in FIG. When displayed, a high-definition image is displayed as shown in FIG. 30D, and the user can see a clear image. In particular, when a signboard or the like on which characters are drawn is displayed in the area indicated by the wavy line, if the high-definition plane image P is not superimposed and displayed, the characters are blurred even if enlarged and displayed. I don't know if is written. However, if the high-definition flat image P is displayed in a superimposed manner, the characters can be seen clearly even if the enlarged display is performed, so that the user can grasp what is written.

[First embodiment]
<Correction of brightness value and color value of planar image P and equirectangular projection image EC>
In step S320 of FIG. 25, the brightness value and the color value are corrected. However, when only a part of the omnidirectional image CE is displayed, the brightness value and the color value of the superimposed image S are not appropriate. There is a case. For wide-angle images such as omnidirectional images, the appropriate exposure is determined in consideration of the entire 360-degree image, so the brightness of some areas may not be appropriate and may be overexposed or underexposed. is there. Therefore, in the following, the brightness value and the color value of the planar image P and the equirectangular projection image EC are corrected, and the correction amounts of the brightness value and the color value of the planar image P and the equirectangular projection image EC are corrected. Processing of the smartphone 5 that is appropriately controlled will be described.

  FIG. 31 is a conceptual diagram of an image in the process of correction processing according to the embodiment. This will be described in comparison with FIG. In FIG. 31, a correction process for correcting the brightness and color of the equirectangular projection image EC is newly added to step S400. In step S400, the correction unit 584 performs correction to match the brightness value and color value of the equirectangular projection image EC with the brightness value and color value of the planar image P based on the correction parameter in the superimposed display metadata. . Hereinafter, the equirectangular projection image EC after the correction in step S400 is referred to as a “corrected image D”.

  Further, the correction in step S320 in FIG. 31 is different from the correction in FIG. In FIG. 31, the correction amount of the brightness value and the color value of the planar image P is controlled according to the angle β between the viewing direction and the center point of the planar image P. A process similar to this is performed on the equirectangular projection image EC in the process of step S400.

<< S320 Correction of Brightness Value and Color Value of Plane Image P >>
FIG. 32 is a conceptual diagram of an image in the process of correcting the planar image P. The correction process (S320) of the planar image P shown in FIG. 31 will be described in detail.

  In step S321, the correction unit 584 corrects the brightness value and color value of the planar image P to the brightness value and color value of the equirectangular projection image EC based on the correction parameter in the superimposed display metadata. Create

  In step S322, based on the predetermined viewing direction of the virtual camera IC (the center point of the predetermined region T) and the angle β to the center point of the planar image P, the correcting unit 584 performs the correction image C1 and the uncorrected planar image. A corrected image C2 is newly created by changing the composition ratio of P and compositing. The composition in which the composition ratio is changed will be described with reference to FIG.

  In step S330, the image creating unit 586 creates the superimposed image S by pasting the corrected image C2 on the partial solid sphere PS.

<< S400 Correction of brightness value and color value of equirectangular projection image EC >>
FIG. 33 is a conceptual diagram of an image in the process of correcting the equirectangular projection image EC. The correction process (step S400) of the equirectangular projection image EC shown in FIG. 31 will be described in detail.

  In step S411, the correction unit 584 generates the corrected image D1 in which the brightness value and color value of the equirectangular projection image EC are corrected to the brightness value and color value of the planar image P. The correction parameter in the superimposed display metadata stores gain data that matches the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC. Multiplying the equirectangular projection image EC by the inverse of the gain data makes it possible to perform the above correction processing. The correction parameter in the superimposed display metadata is correction data that matches the brightness value and the color value in the region of the superimposed image S (planar image P ′), and is applied to the entire region of the equirectangular projection image EC. Thus, although the accuracy is inferior to the region of the superimposed image S (planar image P ′), it is possible to correct a region other than the superimposed image S (planar image P ′) without a sense of incongruity.

  In step S412, the correction unit 584 determines the corrected image D1 and the pre-correction image based on the angle β between the predetermined line-of-sight direction of the virtual camera IC (the center point of the predetermined region T) and the plane image P determined by the user's operation. A corrected image D2 is newly created by changing and synthesizing the proportion of the equirectangular projection image EC.

  In step S350, the image creation unit 586 creates the omnidirectional image CE by pasting the corrected image D2 onto the solid sphere CS.

<Change of image composition ratio>
FIG. 34 is a diagram for explaining a method of changing the image composition ratio. FIG. 34A shows an angle β as the amount of displacement between the line-of-sight direction and the center of the superimposed image S (planar image P ′). As shown in FIG. 7, an angle between a predetermined line-of-sight direction of the virtual camera IC (a center point CP of the predetermined region T) and the center point CM of the superimposed image S is an angle β. The angle β takes a value from 0 to 180 °.

  FIG. 34B shows the respective correction processes (S320) of the superimposed image S and the equirectangular projection image EC based on the angle β between the viewing direction of the virtual camera IC (the center point of the predetermined region T) and the planar image P. , S400) is a diagram showing a method for changing the composition ratio in the composition processing. As shown in (b-1) of FIG. 34B, an area inside the area where the angle β is th1 around the center point CP of the predetermined area T (predetermined area image Q) is defined as area 1, and the angle β An area that does not include area 1 inside the area where is equal to th2 is defined as area 2. Further, an area that is not included in areas 1 and 2 within the predetermined area T is defined as area 3.

  First, a method for calculating the composition ratio of the corrected image C1 and the planar image P will be described. (B-2) in FIG. 34 (b) shows the relationship of the composition ratio of the corrected image C1 and the planar image P with respect to the angle β. The composite ratio of S in the superimposed image on the vertical axis takes a value from 0.0 to 1.0. The composite ratio of the superimposed image S indicates the composite ratio of the corrected image C1 and the flat image P obtained by matching the brightness value and color value of the flat image P with the equirectangular projection image EC as shown in FIG. For example, when the composition ratio is 0.3, the planar image P is composed at a ratio of 0.7 (= 1-0.3) and the corrected image C1 is composed at a ratio of 0.3. Therefore, when the composition ratio is 0.0, the corrected image C2 is the planar image P, and when the composition ratio is 1.0, the corrected image C2 is the corrected image C1. In this way, the composition ratio controls the amount of correction of how close the planar image P is to the corrected image C1.

  According to (b-2) of FIG. 34 (b), in the area of area 1 (angle β ≦ th1), the composition ratio of the image is 0.0, so that the corrected image C2 becomes the plane image P, and area 2 (th1 <angle) In the region of β ≦ th2), the composition ratio varies, and the composition ratio of the corrected image C1 increases as it goes to the outer peripheral side of area 2 (angle β approaches th2). In the area 3 (angle β> th2), since the composition ratio is 1.0, the corrected image C2 becomes the corrected image C1.

  By using such a composition ratio, when the superimposed image S is displayed in the predetermined region T, if the superimposed image S is positioned at the center of the predetermined region T (the center point CM of the superimposed image S is the center point of the predetermined region T). The image is displayed in the state of the original brightness value and color value of the planar image P when approaching the CP, and the superimposed image S moves away from the center of the predetermined region T (the center point CM of the superimposed image is the center point of the predetermined region T) As the distance from the CP increases, the brightness value and the color value of the equirectangular projection image EC approach the state, and finally the brightness value and the color value of the equirectangular projection image EC are obtained.

  (B-3) in FIG. 34 (b) shows the relationship between the angle β and the composition ratio of the equirectangular projection image EC. The composition ratio of the equirectangular projection image EC on the vertical axis takes a value from 0.0 to 1.0. As described with reference to FIG. 33, the composition ratio of the equirectangular projection image EC is the correction image D1 in which the brightness value and the color value of the equirectangular projection image EC are matched to the planar image P, and the equirectangular projection image. Refers to the rate of synthesis with EC. For example, when the combination ratio is 0.3, the equirectangular projection image EC is combined at a ratio of 0.7 (= 1−0.3) and the corrected image D1 is combined at a ratio of 0.3. Therefore, when the composition ratio is 0.0, the correction image D2 is the equirectangular projection image EC, and when the composition ratio is 1.0, the correction image D2 is the correction image D1. In this way, the composition ratio controls the correction amount that determines how close the equirectangular projection image EC is to the corrected image D1.

  34 (b-2) and (b-3) in FIG. 34, the region of the angle β is the same, and the composition ratio is different from that of the superimposed image S (reverse phase). The combined ratio of area 2 may be 1 in total.

  In (b-3) of FIG. 34, in the area of area 1 (angle β ≦ th1), the composition ratio of the image is 1.0, so that the corrected image D2 becomes the corrected image D1, and area 2 (th1 <angle β ≦ th2). ), The composition ratio fluctuates, and the composition ratio of the equirectangular projection image EC increases as it goes to the outer peripheral side of the area 2 (the angle β approaches th2). In the area 3 (angle β> th2), since the composition ratio is 0.0, the corrected image D2 is an equirectangular projection image EC.

  Thereby, when the superimposed image S is displayed in the predetermined area T, when the superimposed image S is positioned at the center of the predetermined area T (the center point CM of the superimposed image approaches the center point CP of the predetermined area T), the planar image P The equirectangular projection image EC is displayed in the state of the corrected image D1 in accordance with, and the superimposed image S moves away from the center of the predetermined region T (the central point CM of the superimposed image moves away from the center point CP of the predetermined region T). Accordingly, the brightness value and color value of the equirectangular projection image EC are approached, and finally the brightness value and color value of the equirectangular projection image EC are obtained.

  Accordingly, the smaller the displacement between the line-of-sight direction and the center of the superimposed image S with respect to the equirectangular projection image EC, the smaller the correction amount of the pixel value of the planar image, and the pixel value of the equirectangular projection image EC becomes the planar image P. The pixel value of the equirectangular projection image EC is corrected so as to approach the pixel value of. The equirectangular projection so that the pixel value of the equirectangular projection image EC approaches the pixel value of the equirectangular projection image EC as the amount of displacement between the line-of-sight direction of the equirectangular projection image EC and the center of the superimposed image S increases. The pixel value of the image EC is corrected.

<Specific Example of Virtual Camera IC Gaze Direction, Superimposed Image S, and Center Point>
The relationship between the viewing direction of the virtual camera IC (the center point CP of the predetermined area T) and the center point of the superimposed image S will be described with reference to FIGS. FIG. 35D shows a display example of the predetermined area T (predetermined area image Q) in the predetermined line-of-sight direction of the omnidirectional image CE. The center point CP of the predetermined area T corresponds to the visual line direction of the virtual camera IC and is indicated by a black circle. Further, the region of the superimposed image S superimposed on the omnidirectional image CE is indicated by a dotted line, and the center point CM of the superimposed image S is indicated by a white circle. It should be noted that the black circle at the center point CP and the white circle at the center point CP are displayed for explanation and are not actually displayed. As described above, the position of the center point CM of the superimposed image S with respect to the center point CP of the predetermined region T can be arbitrarily changed by a user operation.

  FIG. 35A, FIG. 35B, and FIG. 35C illustrate display examples of the predetermined region T when the user changes the line-of-sight direction of the virtual camera IC with respect to the omnidirectional image CE. . 36 (a), 36 (b), and 36 (c) show the relationship between the predetermined region T and the superimposed image S corresponding to FIGS. 35 (a) to 35 (c). is there.

  FIG. 35A and FIG. 36A show the case where the center point CM of the superimposed image S deviates significantly in the right direction with respect to the predetermined region T with respect to the viewing direction of the virtual camera IC (center point CP of the predetermined region T). Is a display example. As shown in FIG. 36A, the angle β (= β2) between the line-of-sight direction and the center point CM of the superimposed image S is large.

  FIG. 35B and FIG. 36B show the case where the center point CM of the superimposed image S is shifted to the right in the region of the predetermined region T with respect to the viewing direction of the virtual camera IC (center point CP of the predetermined region T). Is a display example. As shown in FIG. 36B, the angle β (= β1) decreases between the line-of-sight direction and the center point CM of the superimposed image S.

  FIG. 35C and FIG. 36C are display examples when the line-of-sight direction of the virtual camera IC (the center point CP of the predetermined area T) and the center point CM of the superimposed image S match. As shown in FIG. 36C, the angle β between the line-of-sight direction and the center point CM of the superimposed image S is zero.

  When the center point CM of the superimposed image S is away from the center point CP of the predetermined region T as shown in FIG. 35A, the user often views the omnidirectional image. Even if the brightness value and the color value based on the exposure are prioritized, a sense of incongruity is unlikely to occur. On the other hand, when the center point CM of the superimposed image S substantially coincides with the center point CP of the predetermined region T as shown in FIG. 35 (c), the user often views the superimposed image S. The high-quality superimposed image S can be viewed by giving priority to the brightness value and color value based on the exposure of S. The correction unit 584 corrects the brightness value and the color value of the correction image C2 and the correction image D2 at the composition ratio of FIG. 34, so that the relationship between the center point CM of the superimposed image S and the center point CP of the predetermined region T is obtained. Correct brightness and color values.

<Effect of correction>
FIG. 37 and FIG. 38 are examples of diagrams showing the effects of the presence or absence of correction according to the present embodiment. FIG. 37 is a diagram showing the effect of correction when the omnidirectional image CE is overexposed. As described with reference to FIGS. 35 and 36, the predetermined region T is displayed when the line-of-sight direction of the virtual camera IC is changed. It shows how it is corrected. It is assumed that the planar image P superimposed on the omnidirectional image CE is captured with appropriate exposure.

  37 (a-1), 37 (b-1), and 37 (c-1), the brightness value and color value of the superimposed image S are corrected to the brightness value and color value of the omnidirectional image CE. A display example in the case of being performed is shown. Although it is difficult to understand from the drawing, the whole celestial sphere image CE is brightened by overexposure. Since the brightness value and the color value of the superimposed image S are corrected so as to approach the brightness value and the color value of the overexposed omnidirectional image CE, even if the line-of-sight direction of the virtual camera IC is changed (the superimposed image S The image that is displayed remains overexposed (even if it comes to the center of the predetermined area T). Although it is difficult to understand from the drawings, overexposure remains in all of FIGS. 37 (a-1), 37 (b-1), and 37 (c-1).

  37A-2, 37B-2, and 37C-2 illustrate the brightness of the superimposed image S so as to approach the brightness value and the color value of the overexposed omnidirectional image CE. The display of the predetermined area | region T when a value and a color value are not correct | amended is shown. Since the superimposed image S is not corrected, it is displayed with the brightness value and the color value of the planar image P captured with appropriate exposure. However, compared with FIGS. 37 (a-1), 37 (b-1), and 37 (c-1), the superimposed image S has a proper exposure, but the omnidirectional image CE and the superimposed image are displayed. The brightness value and the color value are different in S, and the superimposed image S is not well blended with the omnidirectional image CE (the superimposed image S is uncomfortable). Although it is difficult to understand from the drawings, the superimposed images S with appropriate exposure in FIGS. 37 (a-2), 37 (b-2), and 37 (c-2) are conspicuous.

  FIG. 37 (a-3), FIG. 37 (b-3), and FIG. 37 (c-3) show display examples when the correction of the present embodiment is performed. In FIG. 37 (a-3), the center point CM of the superimposed image S is greatly deviated from the line-of-sight direction of the virtual camera IC (center point CP of the predetermined region T), so that the conventional correction of FIG. Same as the display example. In FIG. 37 (b-3), as the center point CM of the superimposed image S approaches the visual line direction of the virtual camera IC (center point CP of the predetermined area T), the omnidirectional image CE and the superimposed image S are appropriately exposed. The composition ratio based on the brightness value and the color value of the planar image P imaged at is increased. Therefore, as described with reference to FIG. 34, the omnidirectional image CE has a higher composition ratio of the correction image D1 in accordance with the brightness value and color of the planar image P, and the superimposed image S has a composition ratio of the planar image P. By becoming higher, the overexposure of the predetermined region T is reduced.

  In FIG. 37 (c-3), the center point CM of the superimposed image S approaches the line-of-sight direction of the virtual camera IC (center point CP of the predetermined region T), and the center point CM of the superimposed image S is near the center of the predetermined region T. Therefore, the composition ratio based on the brightness value and color value of the planar image P captured with appropriate exposure is maximized, and the predetermined area T becomes the brightness value and color value of the planar image P with appropriate exposure. It is displayed.

  Although it is difficult to understand from the drawing, the brightness and color values of the omnidirectional image CE approach the planar image P in the order of FIG. 37 (a-3) → FIG. 37 (b-3) → FIG. 37 (c-3). Yes.

  37 (a-3), 37 (b-3), and 37 (c-3), since both the omnidirectional image CE and the superimposed image S are corrected, FIG. a-2), FIG. 37 (b-2), and FIG. 37 (c-2), there is no difference in brightness value and color value between the omnidirectional image CE and the superimposed image S, and the superimposed image S Is well integrated into the spherical image CE.

  FIG. 38 is an example of the effect of the correction when the omnidirectional image CE is underexposed. As described with reference to FIGS. 35 and 36, the predetermined region T is changed when the line-of-sight direction of the virtual camera IC is changed. It shows how it is corrected. It is assumed that the planar image P superimposed on the omnidirectional image CE is captured with appropriate exposure.

  38 (a-1), 38 (b-1), and 38 (c-1), the brightness value and color value of the superimposed image S are corrected to the brightness value and color value of the omnidirectional image CE. A display example in the case of being performed is shown. The dark drawing is intended for underexposure. Since the brightness value and the color value of the superimposed image S are corrected so as to approach the brightness value and the color value of the underexposed celestial sphere image CE, even if the line-of-sight direction of the virtual camera IC is changed (the superimposed image S The displayed image remains underexposed (even if it comes to the center of the predetermined area T).

  38 (a-2), 38 (b-2), and 38 (c-2) show the brightness of the superimposed image S so as to approach the brightness value and color value of the underexposed omnidirectional image CE. The display of the predetermined area | region T when a value and a color value are not correct | amended is shown. Although the superimposed image S has an appropriate exposure, there is a difference in brightness value and color value between the omnidirectional image CE and the superimposed image S, and the superimposed image S is not well blended with the omnidirectional image CE (superimposed image). S is uncomfortable.)

  FIG. 38A-3, FIG. 37B-3, and FIG. 37C-3 show display examples when the correction of the present embodiment is performed. In FIG. 38 (a-3), the center point CM of the superimposed image S is greatly deviated from the viewing direction of the virtual camera IC (center point CP of the predetermined region T), so that the conventional correction in FIG. 38 (a-1) is performed. Same as the display example. In FIG. 38 (b-3), as the center point CM of the superimposed image S approaches the line-of-sight direction of the virtual camera IC (center point CP of the predetermined area T), the omnidirectional image CE and the superimposed image S are appropriately exposed. The composition ratio based on the brightness value and the color value of the planar image P imaged at is increased. Therefore, as described with reference to FIG. 34, the omnidirectional image CE has a higher composition ratio of the correction image D1 in accordance with the brightness value and color of the planar image P, and the superimposed image S has a composition ratio of the planar image P. By becoming higher, underexposure of the predetermined region T is reduced.

  In FIG. 38 (c-3), the center point CM of the superimposed image S approaches the line-of-sight direction of the virtual camera IC (center point CP of the predetermined region T), and the center point CM of the superimposed image S is near the center of the predetermined region T. Therefore, the composition ratio based on the brightness value and color value of the planar image P captured with appropriate exposure is maximized, and the predetermined area T becomes the brightness value and color value of the planar image P with appropriate exposure. It is displayed. Thus, even in the case of the underexposed celestial sphere image CE, the same effect as in the case of overexposure is obtained.

[Second Embodiment]
In the second embodiment, a correction method using two equirectangular projection images EC having different brightness value and color value correction methods will be described.

  FIG. 39 is a conceptual diagram of an image in the process of correcting an equirectangular projection image. FIG. 39 explains the correction processing (step S400) of the equirectangular projection image EC shown in FIG.

  In the second embodiment, apart from the equirectangular projection image EC, a plurality of equirectangular projection images (EC1 and EC2 in the figure) captured at different exposures are required. It is preferable that the special imaging device 1 captures a plurality of equirectangular projection images EC1 and EC2 by changing only the exposure with the same composition as the equirectangular projection image EC. At least two images, ie, an over-exposed equirectangular projection image EC1 and an under-exposed equidistant cylindrical projection image EC2, are required rather than exposure of the equidistant cylindrical projection image EC. In FIG. 39, the equirectangular projection image EC1 and the underexposed equirectangular projection image EC2 that are over-exposed than the equidistant cylindrical projection image EC are shown. May be three or more.

  In step S421, the correction unit 584 selects an image necessary for generating the corrected image D1 by combining with the equirectangular projection image EC. For selecting an image, correction parameters in the superimposed display metadata are used. The correction parameter in the superimposed display metadata is gain data that matches the planar image P with the brightness value and color value of the equirectangular projection image EC. When the reciprocal of the gain data is calculated, the equirectangular projection image EC is obtained. This is gain data (hereinafter referred to as inverse gain data) that matches the brightness value and color value of the planar image P. The correction unit 584 corrects the equirectangular projection image EC brightly when the calculated inverse gain data is 1.0 or more, and corrects the equirectangular projection image EC darkly when smaller than 1.0.

  In step S422, the correction unit 584 creates a corrected image D1 that matches the brightness value and color value of the planar image P. Therefore, when the inverse gain data is 1.0 or more, the correction unit 584 is more than the equirectangular projection image EC. The equirectangular projection image EC1 that is overexposed is selected. If it is smaller than 1.0, the equirectangular projection image EC2 that is underexposed rather than the equirectangular projection image EC is selected. The method for creating the corrected image D2 is the same as in the first embodiment.

  FIG. 40 is a diagram for explaining the relationship between the position parameter and the correction parameter when using a plurality of equirectangular projection images EC1 and EC2 having different exposures. By creating the superimposed display metadata, position parameters and correction parameters corresponding to the planar image P and the equirectangular projection image EC are stored in the superimposed display metadata.

  Since the equirectangular projection image EC, the equirectangular projection image EC1, and the equirectangular projection image EC2 have the same composition and only different exposure, the positional parameters can be used in common, and the correction parameter is the equirectangular cylinder. It can be used only for the projected image EC. The correcting unit 584 corresponds to the third corresponding area CA3 of the equirectangular projection image EC according to the position parameter, and corresponds to the third correspondence of the equirectangular projection image EC, the equirectangular projection image EC1, and the equirectangular projection image EC2. The area CA3 is specified, and the brightness value and the color value of each third corresponding area CA3 can be measured.

  First, the correction unit 584 calculates the ratio between the brightness value and the color value in the third corresponding area CA3 of the equirectangular projection image EC, the equirectangular projection image EC1, and the equirectangular projection image EC.

  For example, if the brightness value of the third corresponding area CA3 of the equirectangular projection image EC is Y, the brightness value of the equirectangular projection image EC1 is Y1, and the brightness value of the equirectangular projection image EC2 is Y2, The ratio of the brightness value of the equirectangular projection image EC1 to the equirectangular projection image EC is Y1 / Y, and the ratio of the brightness value of the equirectangular projection image EC2 to the equirectangular projection image EC is Y2 / Y. By multiplying the correction parameter by this ratio, a correction parameter for correcting the planar image P to the brightness value of the equirectangular projection image EC1 or equirectangular projection image EC2 can be calculated.

  If the ratio is similarly calculated for the color value, the correction parameter for the color value of the equirectangular projection image EC1 or the equirectangular projection image EC2 can also be computed.

  Instead of using the above method, correction parameters may be created for the equirectangular projection image EC1 and the equirectangular projection image EC2 by the same method as the equirectangular projection image EC when superimposing display metadata is created.

  The correction unit 584 creates a corrected image (creates a corrected image D1) in step S422 using the correction parameters obtained as described above.

  Returning to FIG. 39, the correction image creation in step S422 will be described. The correction parameter of the equirectangular projection image EC is PC, the correction parameter of the equirectangular projection image EC1 is PC1, and the correction parameter of the equirectangular projection image EC2 is PC2. The correction parameter PC is gain data that brings the planar image P close to the brightness value and color value of the equirectangular projection image EC. When the correction parameter is 1.0, the planar image P and the equirectangular projection image have the same brightness value and color value.

  In step S422, a corrected image D1 having the same brightness value and color value as the planar image P is created from a plurality of equirectangular projection images EC1, EC2 having different exposures. For example, if the correction parameter values are PC = 1.2, PC1 = 1.6, and PC2 = 0.8, the inverse gain data is 1 / PC = 0.833, which is smaller than 1.0. In the image selection process, the equirectangular projection image EC2 is selected to darken the equirectangular projection image EC.

Next, in the correction image creation in step S422, the composition ratio of the equirectangular projection image EC and the equirectangular projection image EC2 selected in step S421 is calculated. Here, assuming that the composition ratio k (k is 0.0 to 1.0) between the equirectangular projection image EC and the image selected in step S421, the composition ratio is expressed by the following (Expression 16) or (Expression 17). Can be calculated.
k / PC + (1-k) /PC1=1.0 (Expression 16)
k / PC + (1-k) /PC2=1.0 (Expression 17)
However, in the image selection (step S421), (Equation 16) is used when the equirectangular projection image EC1 is selected, and (Equation 17) is used when the equirectangular projection image EC2 is selected. When the composition ratio k is calculated using (Equation 17) when the equirectangular projection image EC2 is selected, k = 0.60, and the composition ratio is 0.60 for the equirectangular projection image EC, and the equirectangular cylinder. The projected image EC2 is (1-k) = 0.40. A corrected image D1 having the same brightness value and color value as the planar image P can be created by combining the images at the calculated combining ratio. Since the process after step S412 is the same as that described above, the description thereof is omitted.

  FIG. 41 is a diagram illustrating a method of creating corrected images C2 and D2 in which brightness values (or color values) are corrected. FIG. 41 shows the difference in brightness value (or color value) between the equirectangular projection image EC and the planar image P. The value is large in the upward direction and the value is small in the downward direction. A corrected image D2 is created from the equirectangular projection image EC and the corrected image D1 according to the composition ratio, and a corrected image C2 is created from the plane image P and the correction image C1 according to the composition ratio. As shown in FIG. 34, since the composition ratio for creating the corrected image is inverted between the equirectangular projection image and the superimposed image (opposite phase), the brightness of the corrected image D2 and the corrected image C2 as shown in FIG. The value (or color value) matches.

<When multiple planar images are superimposed>
The example in which one superimposed image S is superimposed on one equirectangular projection image EC has been described so far, but two or more superimposed images S are superimposed on one equirectangular projection image EC. There is.

  In the imaging system of FIG. 8, the special imaging device 1 and the general imaging device 3 are configured one by one, and a set of equirectangular projection images EC and a planar image P are simultaneously captured. When the subject is a stationary object, the user can capture the second planar image P2 by changing at least one of the imaging direction or the angle of view of the general imaging device 3 from the same position. The smartphone 5 can superimpose not only the planar image P captured at the same time as the equirectangular projection image EC but also the additionally captured planar image P2 on the equirectangular projection image EC.

  In addition, when there are a plurality of general imaging devices 3 instead of one, and each of them is facing a different imaging direction or has a different angle of view, the imaging system can simultaneously acquire a plurality of planar images. For example, since the smartphone 5 has two cameras, a front camera and a rear camera, two planar images can be acquired simultaneously.

  Hereinafter, a process in the case where a plurality of superimposed images S are superimposed on the equirectangular projection image EC will be described.

  FIG. 42 is a diagram illustrating an example of the predetermined region T on which a plurality of superimposed images S are superimposed. In FIG. 42D, two superimposed images S1 and S2 are superimposed on the predetermined area T.

  42A, 42B, and 42C are display examples of the predetermined region T when the user changes the line-of-sight direction of the virtual camera IC with respect to the omnidirectional image CE. The center point CP of the predetermined region T is indicated by a black circle, and the center points CM1 and CM2 of the two superimposed images S1 and S2 are indicated by white circles. Note that the black circle at the center point CP and the white circle at the center point CM are displayed for explanation and are not actually displayed.

  43 (a), 43 (b), and 43 (c) show the line-of-sight direction and the superimposed image S1 corresponding to each of FIGS. 42 (a), 42 (b), and 42 (c). It shows the relationship of the angle of the center point CM of the superimposed image S2.

  When a plurality of superimposed images S are superimposed on the predetermined region T, only one equirectangular projection image EC is a target object for bringing the brightness value and color value of the plurality of superimposed images S close to each other. However, there are a plurality of images to be targeted that bring the brightness value and color value of the equirectangular projection image EC closer. Hereinafter, a method for determining a target object will be described with some processing methods.

<< First processing method >>
In the first processing method, the target object is determined based on the angle β between the line-of-sight direction (the center point CP of the predetermined region T) and the superimposed image S.
(1-1) A first method of the first processing method targets a superimposed image in which the angle β between the line-of-sight direction (the center point CP of the predetermined region T) and the superimposed image S is the minimum. For example, in the case of FIGS. 42A and 43A, if the angle β1 between the line-of-sight direction and the superimposed image S1 and the angle β2 between the superimposed image S2 are β1, the state of β1> β2, and the correction unit 584 Uses the superimposed image S2 having the smallest angle β as the target object. In the case of FIGS. 42B and 43B, since β1 <β2, the correction unit 584 sets the superimposed image S1 as the target object. In the case of FIG. 42C and FIG. 43C, β1 <β2 and the correction unit 584 sets the superimposed image S1 as the target object.
(1-2) In the second method of the first processing mode, the ratio of the target superimposed image is determined according to the value of the angle β between the line-of-sight direction (the center point CP of the predetermined region T) and the superimposed image S. While changing, the ratio is calculated from the weighted average so that the ratio with the smaller angle β becomes higher.

  For example, in the case of FIGS. 42A and 43A, if the angle β1 between the line-of-sight direction and the superimposed image S1 and the angle β2 between the superimposed image S2 are β2, the ratio of the superimposed image S1 is β2 / (β1 + β2), The ratio of the image S2 is β1 / (β1 + β2). 42B and 43B, and also in the case of FIGS. 42C and 43C, the ratio can be calculated by the same equation. At this ratio, the pixel values of the plurality of planar images P1 and P2 are reflected in the pixel values of the equirectangular projection image EC1. A method for calculating the brightness value (or color value) of the target object from the calculated ratio will be described later.

<< Second processing method >>
In the second processing mode, the target object is determined based on the ratio of the area of the superimposed image S displayed in the predetermined region T.

  (2-1) The first method of the second processing method targets a superimposed image in which the ratio of the area of the superimposed image S displayed in the predetermined region T is maximized. For example, in the case of FIGS. 42A and 43A, assuming that the area of the superimposed image S1 displayed in the predetermined region T is SS1, and the area of the superimposed image S1 is SS2, SS1 <SS2, and thus the superimposed image S2. Is the target. In the case of FIGS. 42B and 43B, since SS1 <SS2, the superimposed image S2 is the target object. In the case of FIGS. 42 (c) and 43 (c), the superimposed image S2 is not displayed in the predetermined area T, and SS1> SS2, so the superimposed image S1 is set as the target object.

  (2-2) The second method of the second processing method is to change the ratio of the target superimposed image according to the ratio of the area of the superimposed image displayed in the predetermined region T, and the area is The ratio is calculated by a weighted average so that the ratio of the larger one is higher.

  For example, in the case of FIGS. 42A and 43A, if the area of the superimposed image S1 displayed in the predetermined region T is SS1, and the area of the superimposed image S1 is SS2, the ratio of the superimposed image S1 is SS1 / ( SS1 + SS2) and the ratio of the superimposed image S2 is SS2 / (SS1 + SS2). In the case of FIG. 42B and FIG. 43B, the ratio can be calculated by the same formula also in the case of FIG. 42C and FIG. At this ratio, the pixel values of the plurality of planar images P1 and P2 are reflected in the pixel values of the equirectangular projection image EC1. A method for calculating the brightness value (or color value) of the target object from the calculated ratio will be described later.

<< Third processing method >>
The third processing method is a method combining the first and second processing methods. The correction unit 584 includes the angle β between the line-of-sight direction (center point CP of the predetermined region T) of the first processing method and the superimposed image, and the two superimposed images S displayed in the predetermined region T of the second processing method. Combine with the proportion of area. For example, the target image is the superimposed image S in which the angle β is equal to or smaller than the predetermined angle and the ratio of the area of the superimposed image S displayed in the predetermined region T is the largest.

  Alternatively, the ratio of the target superimposed image S is changed according to the ratio of the area of the superimposed image S displayed in the predetermined region T in the superimposed image S in which the angle β is equal to or smaller than the predetermined angle. In addition to these, various combinations are possible.

(Method of creating corrected image of equirectangular projection image EC and plane image P)
Next, a method for creating a corrected image of the equirectangular projection image EC and the planar image P for the determined target object will be described.

  FIG. 44 is a conceptual diagram of an image in the process of correcting the planar images P1 and P2. FIG. 44A is the same as FIG. FIG. 44B is a conceptual diagram of an image in the process of correcting a planar image that is not a target object (in FIG. 44, the planar image P2 is not a target object). The corrected image C12 shows an image in which brightness and color values are matched with the equirectangular projection image EC, and the corrected image C32 shows an image in which brightness and color values are matched with the planar image P1.

  The correction unit 584 creates the corrected image C22 that becomes the superimposed image S by changing the composition ratio of the corrected image C12 and the corrected image C32 (step S322).

Hereinafter, a case where the target object is a single image and a case where the target object is calculated based on the distance and area to the line-of-sight direction of the plurality of superimposed images S will be described.
(1) FIGS. 45 and 46 are diagrams illustrating a method of creating a corrected image C2 in which the brightness value (or color value) is corrected when the target object is one superimposed image S. FIG. FIG. 45A shows a difference in brightness value (or color value) between the equirectangular projection image EC, the planar image P1, and the planar image P2. The vertical axis indicates the brightness value (or color value) of the image, and indicates that the value is large in the upward direction and small in the downward direction. FIG. 45A shows that the brightness value (or color value) of the equirectangular projection image EC is smaller than the two planar images P1 and P2.

  FIG. 45B is a diagram showing the relationship between the corrected images C21 and C22 when the target object is the plane image P1 in the state of FIG. In creation of the correction images C1 and C3, correction parameters created between the equirectangular projection image EC and the planar images P1 and P2 are used. Since the correction parameter is gain data for bringing the brightness value (or color value) of the planar image closer to the equirectangular projection image EC, the equirectangular projection image is multiplied by the inverse gain data by the equirectangular projection image EC. EC can be brought close to the brightness values (or color values) of the planar images P1 and P2.

  The gain data of the plane image P1 is g1, the gain data of the plane image P2 is g2, the reverse gain data of the plane image P1 is G1 (= 1 / g1), and the reverse gain data of the plane image P2 is G2 (= 1 / g2). To do. First, the creation of the corrected image C1 close to the brightness value (or color value) of the target target planar image P1 will be described. In the equirectangular projection image EC, a corrected image D1 close to the brightness value (or color value) of the planar image P1 can be created by multiplying the equirectangular projection image EC by the inverse gain data G1. In the planar image P2, a corrected image C32 that is close to the brightness value (or color value) of the planar image P1 can be created by multiplying the planar image P2 by the gain data g2 / g1.

  Next, creation of a corrected image close to the brightness value (or color value) of the equirectangular projection image EC will be described. In the plane image P1, the correction image C11 and the correction image C12 can be created by multiplying the plane image P1 by the gain data g1, and in the plane image P2 by multiplying the plane image P2 by the gain data g2. Then, in the equirectangular projection image EC, as shown in FIG. 33, a correction image D2 is created from the equirectangular projection image EC and the correction image D1. In the plane image P1, a corrected image C21 is created from the plane image P1 and the corrected image C11 as shown in FIG. In the planar image P2, as shown in FIG. 44, a corrected image C22 is created from the corrected image C32 and the corrected image C12.

  FIG. 45 (c) is a diagram showing the relationship between the corrected images C21 and C22 when the target object is the plane image P2 in the state of FIG. 45 (a). As a corrected image close to the brightness value (or color value) of the target planar image P2, a corrected image D1 obtained by multiplying the equirectangular projection image EC by the inverse gain data G2, and the planar image P1 with gain data g1 / g2 The corrected image C31 multiplied by is created. Further, as a correction image close to the brightness value (or color value) of the equirectangular projection image EC, a correction image C11 obtained by multiplying the plane image P1 by the gain data g1 and a correction image obtained by multiplying the plane image P2 by the gain data g2. An image C12 is created.

  Then, in the equirectangular projection image EC, as shown in FIG. 33, a correction image D2 is created from the equirectangular projection image EC and the correction image D1. In the plane image P2, a corrected image C22 is created from the plane image P2 and the corrected image C12 as shown in FIG. In the planar image P1, a corrected image C21 is created from the corrected image C31 and the corrected image C11 as shown in FIG.

  FIG. 46A shows a state in which the brightness value (or color value) of the equirectangular projection image is between two plane images P1 and P2. FIG. 46B is a diagram showing the relationship between the corrected images C21 and C22 when the target object is the plane image P1 in the state of FIG. 46A. FIG. 46C is a diagram showing the relationship between the corrected images C21 and C22 when the target object is the planar image P2 in the state of FIG. 46A. The corrected images C21 and C22 can be created in the same manner as the method shown in FIG.

  (2) FIG. 47 is a diagram illustrating a method of creating a corrected image C2 in which the brightness value (or color value) is corrected when the target object is two superimposed images S. When the target object is calculated at the ratio of the plurality of superimposed images S, the calculated ratio and the correction parameters created between the equirectangular projection image EC and the planar images P1 and P2 are used.

  FIG. 47A shows a state where the brightness value (or color value) of the equirectangular projection image is between two planar images. FIG. 47B shows the relationship between the corrected images C21 and C22 when the ratio of the plane image P1 is a and the ratio of the plane image P2 is 1-a in the state of FIG. It is.

  Here, the gain data of the plane image P1 is g1, and the gain data of the plane image P2 is g2. The method for obtaining the value of a has been described in (1-2) and (2-2) above.

The correction unit 584 creates the following corrected image close to the brightness value (or color value) of the target object.
Corrected image D1 = Multiply corrected image C32 by multiplying equirectangular projection image EC by a / g1 + (1-a) / g2 Corrected image C31 = Flat image P1 {a / g1 + (1-a) / g2} * g1 = Multiplying the planar image P2 by {a / g1 + (1-a) / g2} * g2 Thus, the brightness value (or color value) of the target object is the brightness value of the planar image P1 and the planar image P2 ( (Or color value) is a weighted average of the calculated ratios.

  Further, as a correction image close to the brightness value (or color value) of the equirectangular projection image EC, a correction image C11 obtained by multiplying the plane image P1 by the gain data g1 and a correction image obtained by multiplying the plane image P2 by the gain data g2. An image C12 is created. Then, in the equirectangular projection image EC, as shown in FIG. 33, a correction image D2 is created from the equirectangular projection image EC and the correction image D1. Regarding the planar image P1, as shown in FIG. 44B, a corrected image C21 is created from the corrected image C31 and the corrected image C11. Further, regarding the planar image P2, as shown in FIG. 44B, a corrected image C22 is created from the corrected image C32 and the corrected image C12.

  The case where two superimposed images S are superimposed on the predetermined region T has been described. However, the present invention is not limited to two, and the same processing can be applied to two or more superimposed images S.

  By performing the processing as described above, even when a plurality of superimposed images S are superimposed on the predetermined region T, the correction unit 584 automatically selects a target target from the plurality of superimposed images S, and the target target is bright. A correction image of the equirectangular projection image and the planar image P with closeness values and color values close to each other can be created, and an omnidirectional image and a superimposed image can be created. The predetermined area T can be displayed with the optimum brightness value and color value, and a display in which a plurality of superimposed images S are well mixed with the omnidirectional image can be performed.

<Modification>
[Third embodiment]
In the present embodiment, a case will be described in which, in the second embodiment, there is no planar image in the predetermined region T according to the viewing direction (center point) and the angle of view, that is, there is no target object.

  FIG. 48 is a conceptual diagram of an image in the process of correcting the equirectangular projection image EC in the present embodiment. In the description of FIG. 48, differences from FIG. 31 are mainly described. 48, unlike FIG. 31, the superimposed image S does not exist in the predetermined area image Q.

  FIG. 49 is an example of a diagram for explaining the correction processing (step 500) of the equirectangular projection image EC shown in FIG. In the present embodiment, as in the second embodiment, a plurality of equirectangular projection images (EC1 and EC2 in the figure) captured at different exposures are required separately from the equirectangular projection image EC.

  The special imaging apparatus 1 preferably captures a plurality of equirectangular projection images EC1 and equirectangular projection images EC2 by changing only the exposure conditions with the same mechanism as that of the equirectangular projection image EC. At least the equirectangular projection image EC1 that is overexposed and the underexposed equirectangular projection image EC2 is required as compared with the exposure of the equirectangular projection image EC. FIG. 49 shows an overexposed equirectangular projection image EC1 and an underexposed equirectangular projection image EC2 based on the exposure of the equirectangular projection image EC. May be three or more (at least one of overexposed equirectangular projection image EC1 and underexposed equirectangular projection image EC2 is two or more).

  In FIG. 49, after the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC on the solid sphere CS, the projection method conversion unit 590 performs the reference method image R with the projection method conversion. However, an area corresponding to the predetermined area T on the equirectangular projection image EC may be obtained.

  In step S510 of FIG. 49, the omnidirectional image CE is created by pasting the equirectangular projection image EC that has not been corrected by the image creation unit 586 to the solid sphere CS in the same manner as in step S350.

  In step S520, similarly to step S370, the projection method conversion unit 590 performs the projection method conversion so that the predetermined area T specified by the line-of-sight direction (center point), the angle of view, etc. can be seen on the display. At this time, the created image is set as a reference area image R. In the present embodiment, a case will be described in which a part of an equirectangular projection image EC having no planar image P is created as the reference region image R.

  In step S530, the equirectangular projection image EC1 captured with a different exposure that is combined with the equirectangular projection image EC according to the average value of the overall brightness values of the reference region image R, or the regularity projection image EC1. The distance cylindrical projection image EC2 is selected.

  The selection of the equirectangular projection image EC1 or the equirectangular projection image EC2 will be described with reference to FIG. FIG. 50 is a flowchart illustrating an example of a procedure for the correction unit 584 to select an equirectangular projection image captured at a different exposure for composition with the reference region image R.

  For example, the correction unit 584 determines the target brightness value of the reference area image R, and compares it with the average (luminance information) of the entire brightness values of the reference area image R (S531). The target brightness value is determined in advance as an example. For example, when each color of RGB is 8 bits, there are 256 gradations, so the middle is 128. The correction unit 584 compares 128 with the average brightness value of the reference region image R. In this embodiment, since the pixel value is normalized to a value of 0 to 1, 0.5 is a target brightness value. Instead of comparing with the intermediate value of the brightness that can be taken, the target brightness value may be 100 or 150 in 256 stages.

  If the average of the entire brightness values of the reference area image R is larger, the process proceeds to step S532. In this case, since the reference area image R is overexposed than the target brightness value, the correcting unit 584 selects the underexposed image to be combined with the underexposed image (S532).

  If the average brightness value of the entire reference area image R is smaller, the correction area 584 selects the overexposed image because the reference area image R is underexposed than the target brightness value (S533). If the brightness value of the reference area image R is equal to the target brightness value, either one may be selected.

  Here, as an example, the average of the entire brightness values of the reference region image R is used, but only a part of the brightness values such as the central portion may be used. Further, brightness characteristic values such as a histogram may be used instead of the average brightness value.

Returning to FIG. In step S540, the correcting unit 584 combines the equirectangular projection image EC1 or equirectangular projection image EC2 selected in step S530 and the equirectangular projection image EC. If the average brightness value of the reference area image R is ref (ref takes a value of 0.0 to 1.0) and the target brightness value is aim (aim is 0.0 to 1.0). The correction value adj is obtained by the following (formula 18).
adj = | (aim−ref) × correction coefficient |
However, clip to 0.0 ≦ adj ≦ 1.0 (Formula 18)
Clip means replacing the lower limit value of the range if the value is less than the range, and replacing the upper limit value of the range if the value exceeds the range. The correction coefficient is a value that determines how much the overexposure or underexposure image is approached in accordance with how far the target brightness value is from the average brightness value of the reference region image R. The image may be specified by the user by looking at the image, or automatically calculated from the exposure change value at the time of imaging. If it is determined in advance, the correction coefficient is set to 3.0, for example.

  When the equirectangular projection image selected in step S530 is ECS, each pixel value D (u, v) of the corrected image D is obtained by (Equation 19) below.

D (u, v) = ECS (u, v) × adj + EC (u, v) × (1.0−adj)
(Formula 19)
Thus, the corrected image D shown in FIG. 48 is created. The subsequent processing is the same as in FIG. In step S360, the superimposed image S is superimposed on the omnidirectional image CE, and the projection method is converted according to the line-of-sight direction (center point) and angle of view, and the predetermined area image Q is displayed on the display 517. In the present embodiment, since the planar image P is not included in the predetermined area image Q, the planar image P exists in the area 3 in FIG. That is, the composition ratio of the superimposed image S and the corrected image C is 1.0, and the corrected image C2 is the corrected image C1 (brightness / color value of the equirectangular projection image EC). Conversely, the composition ratio of the equirectangular projection image EC and the corrected image D1 is 0.0, and the corrected image D2 is the corrected image D1 (brightness / color value of the equirectangular projected image EC). Since the brightness value and the color value are corrected to appropriate values by the corrected image D, the predetermined area T can be displayed with the optimum brightness value and color value.

<If there are multiple images on the over or under side>
As an example, a case where there are two over images and two under images will be described. Including the equirectangular projection image EC (reference region image R), there are five images. The exposure of each image is set to −2.0, −1.0, 0.0, +1.0, and +2.0. This exposure value is an EV value, with +1.0 and +2.0 indicating the over side, and −1.0 and −2.0 indicating the under side. When the exposure of the equirectangular projection image EC is Ev0.0, an image captured in an exposure range of ± 2.0 is used.

  The method of obtaining the composite value blend may be (Equation 19). However, the range of clipped values is different.

Clip to 0.0 ≦ blend ≦ 2.0 (Formula 20)
When there are a plurality of exposure change images on the over side or the under side, the correction unit 584 switches the image to be synthesized in accordance with the synthesis value blend by changing the clip range. Here, for explanation, a case where the reference area image R is darkened will be described.
(i) When 0.0 ≤ blend ≤ 1.0
adj ＝ blend
I1 = Image with exposure -1.0
I2 = image with exposure 0.0 (reference area image R)
(ii) 1.0 <blend ≤ 2.0
adj ＝ blend-1.0
I1 = Image with -2.0 exposure
I2 = Image with exposure of −1.0 Each pixel value D (u, v) of the corrected image D is obtained by (Equation 21).
D (u, v) = I1 (u, v) × adj + I2 (u, v) × (1.0−adj) (Formula 21)
In this way, when the composite value blend is large, the reference region image R and the next dark image are composited, and when the composite value blend is small, two images that are darker than the reference region image R are composited. Thereby, when it is desired to darken the reference area image R, two images are selected from the reference area image R and the two images on the under side according to the degree to which the reference area image R is desired to be dark (composite value blend). Can be synthesized.

<Synthesis example>
A specific example of image composition will be described with reference to FIG. FIG. 51A shows an equirectangular projection image EC that is captured with a reference exposure. Since the difference between brightness and darkness is large, the central portion of the predetermined region T in FIG.

  FIG. 51B shows a predetermined region T of the equirectangular projection image EC1 in which the same location as that in FIG. The exposure at the center of FIG. 51 (b) is almost appropriate. FIG. 51C shows a corrected image D obtained by combining the equirectangular projection image EC of FIG. 51A and the equirectangular projection image EC1 of FIG. 51B using the third embodiment.

  As described above, even if there is no target object, the underexposure of the predetermined area T can be reduced by combining the overexposed image according to the brightness of the predetermined area T.

  FIG. 52A shows an equirectangular projection image EC that is captured with a reference exposure. Since the difference between brightness and darkness is large, the central portion of the predetermined area T in FIG.

  FIG. 52B shows a predetermined region T of the equirectangular projection image EC2 in which the same place as that in FIG. 52A is captured with underexposure. The exposure at the center of FIG. 52 (b) is almost appropriate. FIG. 52C shows a corrected image D in which the equirectangular projection image EC of FIG. 52A and the equirectangular projection image EC2 of FIG. 52B are synthesized using the third embodiment. .

  As described above, even if there is no target object, the overexposure of the predetermined area T can be reduced by combining the underexposed image according to the brightness of the predetermined area T.

  In this embodiment, among the methods that can appropriately correct the planar image superimposed on the wide-angle image, the brightness of the omnidirectional image when the superimposed image is not within the predetermined region depending on the viewing direction (center position) and the angle of view. Although the correction of the depth value has been described, the present invention can also be applied to a case where there is no flat image and there is only a wide-angle image.

[Fourth embodiment]
The fourth embodiment will be described with reference to FIGS. 53 and 54. FIG. 53 is a conceptual diagram of an image in the process of correcting the equirectangular projection image EC in the fourth embodiment. In the description of FIG. 53, differences from FIG. 48 will be mainly described.

  The correction processing (step S500) of the equirectangular projection image EC shown in FIG. 53 is the same as the method described in the third embodiment, and a plurality of equirectangular projection images EC1, EC2 having different exposures. Thus, a corrected image D that is close to the target brightness value (or color value) of the entire image viewed by the user is created.

  In the third embodiment, the embodiment has been described in which the exposure can be corrected to an appropriate exposure even when the planar image does not exist in the predetermined region T. In the fourth embodiment, the planar image P is converted into the brightness value of the equirectangular projection image and Correct the color value so that it approaches. Therefore, even if the planar image P exists in the predetermined region T, the planar image P can be corrected to an appropriate brightness value and color value.

  The fourth embodiment includes step 600, and the correction unit 584 creates a corrected image C in which the planar image P is corrected so as to approach the brightness value (or color value) of the corrected image D.

  FIG. 54 is a diagram illustrating the relationship between the corrected image D and the corrected image C. FIG. 54 shows that the signal value is large in the upward direction and the signal value is small in the downward direction. In the correction processing of the equirectangular projection image EC (step 500), a target is obtained from the equirectangular projection image EC and the equirectangular projection image EC1 or the equirectangular projection image EC2 (the equirectangular projection image EC1 in the figure). A corrected image D close to the brightness value (or color value) was created.

  In the correction process (step 600) of the planar image P, the correction unit 584 creates a corrected image C in which the planar image P is corrected so as to approach the brightness value (or color value) of the corrected image D.

  A specific example of a method for creating the corrected image C will be described. Since the corrected image D is an image obtained by correcting the brightness value (or color value) of the equirectangular projection image EC, the brightness of the equirectangular projection image EC and the correction image D in the third corresponding area CA3 is corrected by the correcting unit 584. The ratio of the value (or color value) is calculated.

  For example, if the brightness value of the third corresponding area CA3 of the equirectangular projection image EC is Y and the brightness value of the correction image D is YD, the ratio of the brightness value of the correction image D to the equirectangular projection image EC Becomes YD / Y. The correction parameter in the superimposed display metadata stores gain data that matches the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC. A correction parameter for correcting the planar image P to the brightness value of the corrected image D can be calculated by multiplying the correction parameter in the data by this ratio. If the ratio is calculated in the same manner for the color value, the correction parameter for correcting the planar image P to the color value of the corrected image D can be calculated. Using the correction parameter calculated in this way, the correction image C corrected to be close to the brightness value (or color value) of the correction image D is created by multiplying the plane image P by the correction parameter. Can do.

  According to FIG. 54, the equirectangular projection image EC is combined with the overexposed equirectangular projection image EC1, thereby obtaining a corrected image D in which the brightness value and the color value are adjusted. In addition, the brightness value and the color value of the planar image P are adjusted, and the corrected image C is obtained.

  Since the corrected image C is corrected to the brightness value and the color value of the corrected image D, the smartphone 5 can detect the brightness value and the color of the corrected image D regardless of whether the predetermined area includes the planar image P or not. The predetermined area image Q can be displayed by the value.

  By using the corrected image D and the corrected image C created as described above, superimposed display can be performed with optimal exposure and color in the area displayed on the display or the like.

  In the fourth embodiment, the degree of freedom of the corrected brightness value and color value is higher than those in the first and second embodiments. That is, in the first and second embodiments, the brightness value and the color value that are finally displayed are the brightness value generated between the equirectangular projection image EC and the planar image P as shown in FIG. Limited to color values. In the fourth embodiment, the equirectangular projection image EC is converted into target brightness values and color values, and the planar image P is brought close to the target brightness values and color values. There is no limitation as in the second embodiment, and the brightness value and color value appropriate for displaying the brightness value and color value of the finally displayed image can be obtained.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, the angle β is exemplified as the displacement amount, but the displacement amount may be represented by a distance between the line-of-sight direction and the planar image P.

  Moreover, the display of the omnidirectional image may be performed by browser software or may be performed by application software for displaying the omnidirectional image.

  In addition, the server may perform the processing that the smartphone 5 has performed, and the smartphone 5 may mainly accept operations or display the omnidirectional image CE. In this case, the smartphone 5 transmits image data to the server via the network, and the server creates a predetermined area image and transmits it to the smartphone 5. The server has some or all of the functions of the smartphone 5 described in the present embodiment.

  Further, the omnidirectional image of the present embodiment may be image data having an angle of view that cannot be displayed in the predetermined area T. For example, a wide-angle image having an angle of view of 180 degrees to 360 degrees only in the horizontal direction may be used. That is, it is not limited to the omnidirectional image.

  In addition, “superimpose” in the scope of claims means that the other image can be seen with priority over one image. The two images may exist independently or may be integrated by replacing a part of the image with one image. “Superimpose” may be expressed as pasting, combining, or arranging.

  Also, the number of cameras that the smartphone 5 has or is externally connected to the smartphone 5 may be three or more.

  In addition, the configuration examples such as FIGS. 14 and 16 are divided according to main functions in order to facilitate understanding of processing by the special imaging device 1, the general imaging device 3, and the smartphone 5. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the special imaging device 1, the general imaging device 3, and the smartphone can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  The special imaging device 1 is an example of a first imaging device, the smartphone 5 is an example of a second imaging device, the short-range communication unit 58 is an example of an acquisition unit, and the correction unit 584 is a correction unit. The projection method reverse conversion unit 562 is an example of a projection method reverse conversion unit, the image creation unit 586 is an example of an image creation unit, and the projection method conversion unit 590 is an example of a projection method conversion unit. The unit 5000 is an example of a storage unit.

The correction image C1 is an example of the first correction image, the correction image D1 is an example of the second correction image, the composition ratio of the superimposed image S in FIG. 34 is an example of the first composition ratio, and FIG. The equirectangular projection image EC is an example of the second synthesis proportion, the equirectangular projection image EC1 in FIG. 39 is an example of the second wide-angle image, and the equirectangular projection image EC2 is the third composition ratio. 34 is an example of the first threshold value, and the threshold value th2 of FIG. 34 is an example of the second threshold value. The equirectangular projection image EC of the third embodiment is an example of a fourth wide-angle image, the equirectangular projection image EC1 or EC2 is an example of a fifth wide-angle image, and the correction image D is a third correction image. It is an example of an image, and the correction image C of the fourth embodiment is an example of a fourth correction image.
th1 is an example of a first threshold, and th2 is an example of a second threshold.

1 Special imaging device (an example of a first imaging device)
5 Smartphone (example of second imaging device)
52 reception unit 55 image / sound processing unit 56 display control unit 58 near field communication unit (an example of an acquisition unit)
550 Extraction unit 584 Correction unit (an example of correction means)
562 Projection method reverse conversion unit (an example of projection method reverse conversion means)
586 Image creation unit (an example of image creation means)
588 Image superimposing unit 590 Projection method conversion unit (an example of projection method conversion means)
5000 storage unit (an example of storage means)

Japanese Patent No. 5745134

Claims (15)

An information processing apparatus that superimposes a planar image obtained by the second projection method on a wide-angle image obtained by the first projection method,
A projection method inverse transform means for calculating a corresponding area corresponding to the planar image in the wide-angle image;
Correction means for correcting at least one of the wide-angle image and the planar image according to the amount of displacement between the line-of-sight direction of the wide-angle image and the center of the planar image;
Image creating means for superimposing the planar image corrected by the correcting means on the corresponding area of the wide-angle image corrected by the correcting means;
A program characterized by functioning as The correction unit reduces the correction amount of the pixel value of the planar image as the displacement amount between the line-of-sight direction of the wide-angle image and the center of the planar image decreases.
The program according to claim 1, wherein the pixel value of the wide-angle image is corrected so that the pixel value of the wide-angle image approaches the pixel value of the planar image. The correction unit corrects the pixel value of the planar image so that the pixel value of the planar image approaches the pixel value of the wide-angle image as the amount of displacement between the line-of-sight direction of the wide-angle image and the center of the planar image increases. And
The program according to claim 1, wherein a correction amount of a pixel value of the wide-angle image is reduced. The correction unit does not correct the pixel value of the planar image when the displacement amount between the line-of-sight direction of the wide-angle image and the center of the planar image is smaller than a first threshold,
The program according to claim 1, wherein the pixel value of the wide-angle image is corrected so that the pixel value of the wide-angle image approaches the pixel value of the planar image. When the displacement between the line-of-sight direction of the wide-angle image and the center of the planar image exceeds a second threshold value that is larger than the first threshold value, the correcting unit changes the pixel value of the planar image to the pixel value of the wide-angle image. Correct the pixel value of the planar image to approach,
The program according to any one of claims 1 to 4, wherein a pixel value of the wide-angle image is not corrected. The correction unit corrects the pixel value of the planar image by a correction parameter for correcting the pixel value of the planar image so as to approach the pixel value of the wide-angle image, and creates a first corrected image.
With reference to the first composition ratio of the pixel value of the planar image and the pixel value of the first corrected image that increase as the displacement amount increases,
The pixel value of the planar image is corrected by combining the pixel value of the planar image and the pixel value of the first correction image at the first synthesis ratio corresponding to the displacement amount. The program according to any one of the above items. The correction means corrects the pixel value of the wide-angle image by the reciprocal of the correction parameter to create a second corrected image,
With reference to a second composition ratio of the pixel value of the wide-angle image and the pixel value of the second corrected image that increases as the displacement amount decreases, the wide-angle image is obtained at the second composition ratio corresponding to the displacement amount. The program according to claim 6, wherein the pixel value of the wide-angle image is corrected by combining the pixel value of the image and the pixel value of the second corrected image. The correction means selects one of a second wide-angle image and a third wide-angle image having the same composition as the wide-angle image and different exposures based on the correction parameter,
8. The second corrected image is created by synthesizing the selected second wide-angle image or the third wide-angle image and the wide-angle image based on a composition ratio calculated by the correction parameter. Program. When a plurality of the planar images are superimposed on a predetermined area when the wide-angle image is displayed,
The correcting means corrects the pixel value of the wide-angle image according to the displacement amount of the planar image with the smallest displacement amount of the line-of-sight direction and the center point of the planar image, or
The program according to any one of claims 1 to 8, wherein a pixel value of the wide-angle image is corrected according to the amount of displacement from the planar image having the largest area in the predetermined region. When a plurality of the planar images are superimposed on a predetermined area when the wide-angle image is displayed,
The correction means calculates a ratio from a weighted average of displacement amounts between the line-of-sight direction and the center points of the plurality of planar images, and reflects the pixel values of the plurality of planar images in the pixel values of the wide-angle image at the ratio. To correct the pixel value of the wide-angle image, or
A ratio is calculated from a weighted average of the areas of the plurality of planar images in the predetermined region, and the pixel values of the plurality of planar images are reflected in the pixel values of the wide-angle image at the ratios, thereby obtaining the pixel values of the wide-angle image. The program according to any one of claims 1 to 8, which is corrected. The correction means includes a brightness value or a color value of a fourth wide-angle image captured by the first projection method, and a brightness value of a fifth wide-angle image having an exposure condition different from that of the fourth wide-angle image. Or a color value is synthesized at a ratio according to the luminance information of the predetermined area of the fourth wide-angle image to generate a third corrected image,
If the displacement amount exceeds the second threshold, the planar image is not corrected,
The program according to claim 1, wherein the image creating unit creates an image for display using the third corrected image. The correction means includes a brightness value or a color value of a fourth wide-angle image captured by the first projection method, and a brightness value of a fifth wide-angle image having an exposure condition different from that of the fourth wide-angle image. Alternatively, a third correction image is generated by combining the color value with a ratio according to the luminance information of the predetermined area of the fourth wide-angle image,
Generating a fourth corrected image in which the planar image is close to the brightness value or color value of the third corrected image;
The program according to claim 1, wherein the image creating unit superimposes the fourth corrected image on the corresponding area of the third corrected image. The correction means is configured to determine whether the fifth wide-angle image or the fifth bright-angle image is brighter than the predetermined area of the fourth wide-angle image based on luminance information of the predetermined area of the fourth wide-angle image specified by the viewing direction and the angle of view. Selecting the fifth wide-angle image darker than a predetermined region of the four wide-angle images;
The program according to claim 11 or 12, wherein the fourth wide-angle image and the selected fifth wide-angle image are synthesized. An information processing apparatus that superimposes a planar image obtained by the second projection method on a wide-angle image obtained by the first projection method,
A projection method inverse transform means for calculating a corresponding area corresponding to the planar image in the wide-angle image;
Correction means for correcting at least one of the wide-angle image and the planar image according to the amount of displacement between the line-of-sight direction of the wide-angle image and the center of the planar image;
Image creating means for superimposing the planar image corrected by the correcting means on the corresponding area of the wide-angle image corrected by the correcting means;
A program characterized by functioning as
An acquisition unit that acquires the wide-angle image from a first imaging device and acquires the planar image from a second imaging device;
An imaging system. An information processing apparatus that superimposes a planar image obtained by a second projection method on a wide-angle image obtained by a first projection method,
A projection method inverse transform means for calculating a corresponding area corresponding to the planar image in the wide-angle image;
Correction means for correcting at least one of the wide-angle image and the planar image according to the amount of displacement between the line-of-sight direction of the wide-angle image and the center of the planar image;
Image creating means for superimposing the planar image corrected by the correcting means on the corresponding area of the wide-angle image corrected by the correcting means;
An information processing apparatus.