JP2020109955A - Photographing device, photographing system, image processing method, and program - Google Patents

Abstract

To solve the problem in which: in a conventional technology, definition is made different between an entire image and a partial image, but both images use the same projection method, which causes the problem of low versatility.SOLUTION: An entire celestial sphere photographing device 1 creates a low-definition entire image (including a low-definition partial image) from an entire celestial sphere image that is a high-definition wide-angle image (S140), and creates a high-definition partial image using different projection method from the same high-definition entire celestial sphere image (S150). The entire celestial sphere photographing device 1 transmits data of the low-definition entire image and the high-definition partial image, and thereby a smartphone 5 at a reception side cannot only composite the partial image with the entire image, but display the composite image even if the low-definition entire image and the high-definition partial image use different projection methods, which leads to an effect of high versatility on the projection method.SELECTED DRAWING: Figure 13

Description

The present invention relates to a photographing device, a photographing system, an image processing method, and a program.

Wide-angle image data obtained by shooting a subject using a fisheye lens instead of a camera system consisting of a pan/tilt and zoom lens using a turntable is distributed, and the image is received by a viewer on the receiving side. Electronically pan/tilt/zoom technology for generating and displaying partial images is already known.

However, in the technique for distributing the data of the wide-angle image obtained by photographing the subject, the data of the image of the area not being noticed is also distributed, which causes a problem that the communication amount increases.

On the other hand, in order to reduce the amount of communication, the image capturing device distributes a low-definition entire image in which the entire wide-angle image is reduced, and high-definition partial image data that is a region of interest in the wide-angle image, A technique has been disclosed in which a receiving side inserts a partial image into an entire image (see Patent Document 1).

However, in the conventional technique, although the fineness is made different between the whole image and the partial image, the same projection method causes a problem of low versatility.

The invention according to claim 1 is a photographing apparatus for photographing a subject to obtain image data of a predetermined definition, and a fine definition of a wide-angle image which is all or part of an image related to the image data of the predetermined definition. And a projection method conversion means for converting a projection method for a narrow-angle image, which is a partial region of the wide-angle image, with a definition different from that of the wide-angle image. It is a device.

As described above, according to the present invention, not only the communication amount can be reduced, but also the versatility of the projection method can be enhanced.

(A) is a right side view of the spherical imaging device, (b) is a rear view of the spherical imaging device, (c) is a plan view of the spherical imaging device, (d) is. It is a bottom view of a spherical imaging device. It is a usage image figure of a spherical imaging device. (A) is a hemispherical image taken by the omnidirectional imager (front), (b) is a hemispherical image taken by the omnidirectional imager (rear), and (c) is an equirectangular projection. It is the figure which showed the image. (A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. It is a figure showing the position of a virtual camera and a predetermined field when a spherical image is made into a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a state in which an image of a predetermined area is displayed on the display of the communication terminal. It is a figure explaining the outline of a partial image parameter. It is a schematic diagram of a configuration of an imaging system according to the first embodiment. It is a hardware block diagram of the omnidirectional imaging device 1. 3 is a hardware configuration diagram of the smartphone 5. FIG. It is a functional block diagram of the spherical imaging device which concerns on 1st Embodiment. It is a functional block diagram of the smart phone concerning a 1st embodiment. It is a conceptual diagram of the image in the process of the image processing which the spherical imaging device 1 which concerns on 1st Embodiment performs. It is a figure explaining a partial image parameter. It is a conceptual diagram of the image data transmitted from the omnidirectional imaging device 1 to the smartphone 5. It is a conceptual diagram of the image in the process of the image processing which the smart phone 5 which concerns on 1st Embodiment performs. It is a figure explaining creation of a partial solid sphere from a partial plane. It is a two-dimensional conceptual diagram at the time of superposing a partial image on a omnidirectional image, without performing partial solid-sphere creation of this embodiment. It is a two-dimensional conceptual diagram at the time of performing partial solid sphere creation of this embodiment, and superimposing a partial image on a spherical image. (A) Wide image display example without superimposed display, (b) Tele image display example without superimposed display, (c) Wide image display example with superimposed display, (d) Superimposed display It is a conceptual diagram which showed the example of a display of a tele image. It is a schematic diagram of a configuration of an imaging system according to a second embodiment. It is a functional block diagram of the omnidirectional imaging device 1a which concerns on 2nd Embodiment. It is a functional block diagram of smart phones 5a and 5b concerning a 2nd embodiment. It is a functional block diagram of the smart phone 5c which concerns on 2nd Embodiment. (A) It is a conceptual diagram of an instruction data restriction table, and (b) is a schematic diagram of an equirectangular projection image. (A) A conceptual diagram of image data transmitted from the omnidirectional imaging device to the smartphone when the instruction data is not limited, and (b) Sent from the omnidirectional imaging device when the instruction data is limited to the smartphone. It is a conceptual diagram of the image data. It is a conceptual diagram of the image in the process of the image processing which the omnidirectional imaging device 1a which concerns on 2nd Embodiment performs. (A) is a display example of the partial image P11 when no restriction is applied, (b) is a display example of an area other than the partial image P11 in the entire image when no restriction is applied, and (c) is a display example of when the restriction is applied. A display example of the partial image P21 and a conceptual diagram of the transmitted partial image P22, (d) is a display example of an area other than the partial image P21 in the partial image P22 when restriction is applied.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Outline of Embodiment]
The outline of this embodiment will be described below.

A method of generating a spherical image will be described with reference to FIGS. 1 to 6.

First, the external appearance of the spherical imaging device will be described with reference to FIG. The omnidirectional photographing device is a digital camera for obtaining a photographed image which is a source of a omnidirectional (360°) panoramic image. Note that FIG. 1A is a right side view of the spherical imaging device, FIG. 1B is a front view of the spherical imaging device, and FIG. 1C is a plan view of the spherical imaging device. FIG. 1D is a bottom view of the spherical imaging device.

As shown in FIGS. 1(a), 1(b), 1(c), and 1(d), the front side (front side) of the omnidirectional imaging device has a fish-eye shape on the front side. The image pickup elements (image sensors) 103a and 103b, which will be described later, are provided inside the celestial sphere imaging device in which the lens 102a of FIG. A hemispherical image (angle of view of 180° or more) can be obtained by photographing a subject or a landscape through the lenses 102a and 102b. A shutter button 115a is provided on the surface opposite to the front side of the omnidirectional photographing device. A power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the omnidirectional shooting device. The shutter button 115a, the power button 115b, and the Wi-Fi button 115c are switched on and off each time they are pressed. The shooting mode switching button 115d switches between a still image shooting mode, a moving image shooting mode, and a moving image distribution mode each time the button is pressed. The shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are types of the operation unit 115, and the operation unit 115 is not limited to these buttons.

Further, a tripod screw hole 151 for attaching the omnidirectional photographing device to the tripod for camera is provided at the center of the bottom 150 of the omnidirectional photographing device. A Micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom portion 150. An HDMI (High-Definition Multimedia Interface) terminal is provided on the right end side of the bottom portion 150. HDMI is a registered trademark.

Next, the use situation of the omnidirectional imaging device will be described with reference to FIG. Note that FIG. 2 is a conceptual diagram of the use of the spherical imaging device. As shown in FIG. 2, the omnidirectional photographing device is used, for example, for a user to hold in his/her hand and photograph a subject around the user. In this case, two hemispherical images can be obtained by imaging the subject around the user by the image sensor 103a and the image sensor 103b shown in FIG. 1, respectively.

Next, with reference to FIGS. 3 and 4, the outline of the processing from the images captured by the omnidirectional imaging device to the equirectangular cylindrical projection image EC and the omnidirectional image CE will be described. It should be noted that FIG. 3A is a hemispherical image captured by the omnidirectional imaging device (front side), and FIG. 3B is a hemispherical image captured by the omnidirectional imaging device (rear side), FIG. 3C. FIG. 4 is a diagram showing an image represented by an equirectangular projection (hereinafter, referred to as “equidistant cylinder projected image”). FIG. 4A is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing a spherical image.

As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fisheye lens 102a described later. Further, as shown in FIG. 3B, the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by the fisheye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) are combined by the omnidirectional imaging device, and as shown in FIG. An EC is created.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular cylindrical projected image is pasted so as to cover the spherical surface as shown in FIG. A spherical image CE as shown in b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. The spherical image CE may be a still image or a moving image.

As described above, since the omnidirectional image CE is an image attached so as to cover the spherical surface, it feels strange to a human being. Therefore, by displaying a part of the predetermined area of the celestial sphere image CE (hereinafter, referred to as a “predetermined area image”) as a flat image with less curvature, it is possible to provide a display that does not make a person feel strange. This will be described with reference to FIGS. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the spherical image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. Further, FIG. 6A is a stereoscopic perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. Further, in FIG. 6A, the spherical image shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the spherical image CE generated in this manner is the solid sphere CS, the virtual camera IC is located inside the spherical image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC, and the position coordinates (x(rH), y including the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE are included. (rV), angle of view α (angle)) is specified by the predetermined area information. The zoom of the predetermined area T can be expressed by expanding or contracting the range (arc) of the angle of view α. The zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or further from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE.

Then, the predetermined area image Q shown in FIG. 6A is displayed as an image of the shooting area of the virtual camera IC on a predetermined display, as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined area information that is initially set (default). The predetermined area information and the position coordinates of the virtual camera IC may be indicated by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T.

FIG. 7 is a diagram for explaining the outline of partial image parameters. Here, a method of designating a part of the spherical image will be described. In the spherical image, the center point CP of the partial image can be represented by the direction of the camera that captures the image. The azimuth angle is "aa" and the elevation angle is "ea" with the center of the whole sphere image as the front of the whole image. Further, in order to represent the range of the partial image, for example, the horizontal view angle α is used. Further, in order to represent the range in the vertical and horizontal directions, it is possible to represent the aspect ratio (width w/height h) of the image. Although the horizontal angle of view and the aspect ratio are used here to represent the range, the diagonal angle of view and the aspect ratio may be used. In addition to the azimuth angle and the elevation angle, the rotation angle may be used.

First Embodiment Hereinafter, the first embodiment of the present invention will be described.

[Outline of shooting system]
First, the outline of the imaging system according to the first embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the imaging system according to the first embodiment.

As shown in FIG. 8, the imaging system of this embodiment is configured by the omnidirectional imaging device 1 and the smartphone 5. The user Y operates both the spherical imaging device 1 and the smartphone 5. In this case, the user Y is also a viewer who browses the image displayed on the smartphone 5.

Among these, the omnidirectional photographing device 1 is a special digital camera for photographing two objects such as a subject or a landscape to obtain two hemispherical images which are the basis of the omnidirectional (panoramic) image, as described above. ..

The smartphone 5 can perform wireless communication with the omnidirectional imaging device 1 by using a short-range wireless communication technology such as Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication). Further, in the smartphone 5, the image acquired from the omnidirectional imaging device 1 can be displayed on a display 517, which will be described later, provided in the device itself.

Note that the smartphone 5 may communicate with the celestial sphere imaging device 1 by a wired cable without using the short-range wireless communication technology.

[Hardware Configuration of Embodiment]
Next, the hardware configurations of the omnidirectional imaging device 1 and the smartphone 5 according to the present embodiment will be described in detail with reference to FIGS. 9 and 10.

<Hardware configuration of spherical camera 1>
First, the hardware configuration of the spherical imaging device 1 will be described with reference to FIG. 9. FIG. 9 is a hardware configuration diagram of the spherical imaging device 1. In the following, the omnidirectional imaging device 1 is an omnidirectional (omnidirectional) omnidirectional imaging device using two image pickup elements, but any number of two or more image pickup elements may be used. Further, it is not necessarily required to be a device exclusively for omnidirectional shooting, and by attaching an omnidirectional image pickup unit that is attached later to a normal digital camera or smartphone, it is possible to have substantially the same function as the omnidirectional image pickup device 1. You may

As shown in FIG. 9, the omnidirectional imaging device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM ( Read Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I/F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, acceleration sensor. 120 and a terminal 121.

Of these, the image pickup unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having a field angle of 180° or more for forming a hemispherical image, and two image pickups provided corresponding to the respective wide-angle lenses. It is provided with elements 103a and 103b. The image sensor 103a, 103b is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image of the fisheye lens 102a, 102b into image data of an electric signal and outputs the image data. It has a timing generation circuit for generating a horizontal or vertical synchronization signal, a pixel clock, etc., a register group for setting various commands and parameters necessary for the operation of the image sensor, and the like.

The image pickup devices 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I/F bus. On the other hand, the image pickup devices 103a and 103b of the image pickup unit 101 are connected to the image pickup control unit 105 by a serial I/F bus (I2C bus or the like). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Further, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, network I/F 116, communication unit 117, electronic compass 118, and the like.

The image processing unit 104 takes in the image data output from the image pickup devices 103a and 103b through a parallel I/F bus, performs a predetermined process on each image data, and then performs a combining process on these image data. , The data of the equirectangular cylindrical projected image as shown in FIG.

The imaging control unit 105 generally sets the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices, and sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to fetch status data and the like of the register group of the image pickup elements 103a and 103b and send them to the CPU 111.

Further, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the omnidirectional imaging device 1, there is a case where the display (for example, the display 517 of the smartphone 5) has a preview display function or a function corresponding to the moving image display. In this case, the output of the image data from the image pickup devices 103a and 103b is continuously performed at a predetermined frame rate (frame/second).

The image pickup control unit 105 also functions as a synchronization control unit that synchronizes the output timing of the image data of the image pickup devices 103a and 103b in cooperation with the CPU 111, as described later. In addition, in the present embodiment, the omnidirectional imaging device 1 is not provided with a display, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I/F bus and performs a predetermined process on the sound data.

The CPU 111 controls the entire operation of the celestial sphere imaging device 1 and executes necessary processing. The CPU 111 may be single or plural. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data in the process of being processed by the image processing unit 104 and data of the processed equidistant cylindrical projection image.

The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various shooting modes and shooting conditions.

The network I/F 116 is a general term for interface circuits (USB I/F, etc.) with external media such as SD cards and personal computers. The network I/F 116 may be wireless or wired. The data of the equidistant cylindrical projection image stored in the DRAM 114 is recorded on an external medium via the network I/F 116, and if necessary, an external terminal (device) such as the smartphone 5 via the network I/F 116. ) Is sent to.

The communication unit 117 communicates with an external terminal (device) such as the smartphone 5 via a near-field wireless communication technology such as Wi-Fi, NFC, or Bluetooth via the antenna 117a provided in the omnidirectional imaging device 1. .. The communication unit 117 can also transmit the data of the equidistant cylindrical projected image to the external terminal (device) such as the smartphone 5.

The electronic compass 118 calculates the azimuth of the celestial sphere imaging device 1 from the magnetism of the earth and outputs the azimuth information. This azimuth information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image. It should be noted that the related information also includes each data of the image capturing date and time and the data capacity of the image data.

The gyro sensor 119 is a sensor that detects a change in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the celestial sphere imaging device 1. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

The acceleration sensor 120 is a sensor that detects acceleration in three axis directions. The omnidirectional imaging device 1 calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device 1) based on the acceleration detected by the acceleration sensor 120. By providing both the gyro sensor 119 and the acceleration sensor 120 in the spherical imaging device 1, the accuracy of image correction is improved.

The terminal 121 is a concave terminal for Micro USB.

<Hardware configuration of smartphone 5>
Next, the hardware of the smartphone 5 will be described with reference to FIG. FIG. 10 is a hardware configuration diagram of the smartphone 5. As shown in FIG. 10, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I/F 513a, an acceleration/direction sensor 506, a media I/F 508, and a GPS receiving unit 509. There is.

Of these, the CPU 501 controls the operation of the entire smartphone 5. The CPU 501 may be single or plural. The ROM 502 stores programs used for driving the CPU 501 such as the CPU 501 and IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as a smartphone program under the control of the CPU 501. The CMOS sensor 505 images a subject (mainly a self-portrait) under the control of the CPU 501 and obtains image data. The image sensor I/F 513a is a circuit that controls driving of the CMOS sensor 512. The acceleration/direction sensor 506 is various sensors such as an electronic magnetic compass that detects geomagnetism, a gyro compass, and an acceleration sensor. The media I/F 508 controls reading or writing (storage) of data with respect to the recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

In addition, the smartphone 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I/F 513b, a microphone 514, a speaker 515, a sound input/output I/F 516, a display 517, an external device connection I/F 518, and a short distance. A communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521 are provided.

Of these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network such as the Internet. The CMOS sensor 512 is a kind of a built-in image pickup unit that obtains image data by capturing an image of a subject under the control of the CPU 501. The image sensor I/F 513b is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting voice. The sound input/output I/F 516 is a circuit that processes input/output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons and the like. The external device connection I/F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, and Bluetooth. The touch panel 521 is a type of input means for operating the smartphone 5 when the user presses the display 517.

The smartphone 5 also includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

[Functional Configuration of Embodiment]
Next, the functional configuration of this embodiment will be described with reference to FIGS. 11 to 12. FIG. 11 is a functional block diagram of the celestial sphere imaging device 1 according to the first embodiment. FIG. 12 is a functional block diagram of the smartphone 5 according to the first embodiment.

<Functional configuration of spherical imaging device 1>
The omnidirectional imaging device 1 includes a transmission/reception unit 11, a partial image parameter creation unit 12, an imaging control unit 13, imaging units 14a and 14b, an image processing unit 15, a temporary storage unit 16, a low definition changing unit 17, and a projection method changing unit. 18 and an encoding unit 19. Each of these units is realized by any one of the components shown in FIG. 9 being operated by an instruction from the CPU 111 according to a program for the omnidirectional image capturing device expanded from the SRAM 113 to the DRAM 114. Function or means.

The transmission/reception unit 11 transmits/receives data to/from the outside of the omnidirectional imaging device 1. For example, the transmission/reception unit 11 receives instruction data from the transmission/reception unit 51 of the smartphone 5, or transmits image data to the transmission/reception unit 51 of the smartphone 5. Further, the transmission/reception unit 11 collectively transmits the image data and the partial image parameters to the transmission/reception unit 51 of the smartphone 5.

The partial image parameter creation unit 12 creates a partial image parameter based on the instruction data sent from the smartphone 5 via the transmission/reception unit 11. This “instruction data” is accepted by the user's operation in the accepting unit 52 of the smartphone 5, and is used by the omnidirectional imaging device 1 to specify a cutout area CA that is a partial area in an overall image described later. This is data indicating an instruction. In addition, the “partial image parameter” is a parameter for specifying a later-described overlapping area for the entire image in the smartphone 5.

The imaging control unit 13 outputs an instruction to synchronize the output timing of the image data of the imaging units 14a and 14b.

The image capturing units 14a and 14b respectively capture an image of a subject or the like in accordance with an instruction from the image capturing control unit 13, and, for example, as shown in FIGS. 3(a) and 3(b), based on the spherical image data. Output hemispherical image data.

The image processing unit 15 synthesizes (joins) the two hemispherical image data obtained by the image capturing units 14a and 14b into the data of the equidistant cylindrical projection image which is the image of the equidistant cylindrical projection method.

The temporary storage unit 16 plays a role of a buffer for temporarily storing the data of the equirectangular cylindrical projected image which is synthesized and converted by the image processing unit 15. Although the equirectangular cylindrical projected image in this state is synthesized and converted, the equidistant cylindrical projected image has relatively high definition because it is an image obtained by the photographing units 14a and 14b.

The low-definition changing unit 17 changes the equidistant cylindrical projected image from the high-definition image to the low-definition image by reducing the image or the like in accordance with the instruction data from the smartphone 5 received by the transmitting/receiving unit 11. As a result, a low-definition equidistant cylindrical projection image (entire image) is generated.

The projection method conversion unit 18 follows the instruction data received by the transmission/reception unit 11, that is, the direction of the partial image which is a partial image of the entire image, the angle of view and the aspect ratio, and the image for transmission when transmitting to the smartphone 5. According to the size (see FIG. 15), the equirectangular cylindrical projection method is converted into the perspective projection method. As a result, the partial image is generated in the high definition state. In FIG. 15, the aspect ratio of the entire image is 16:9, and the aspect ratio of the partial image is 16:9. Therefore, the aspect ratio of the entire transmission image is 16:18.
As described above, the whole image data output from the low definition changing unit 17 has a lower definition (or resolution) than the partial image data output from the projection method converting unit 18. That is, the partial image data output from the projection method conversion unit 18 has a higher definition (resolution) than the entire image data output from the low definition changing unit 17.

The encoding unit 19 encodes each data of the whole image and the partial image and temporarily stores the data. At this time, the entire toothpick image and the partial image shown in FIG. 15 are associated with each other.

Here, a case where, for example, an equidistant cylindrical projection image of 2K, 4K, or 8K is output as the high-definition image will be described. Based on these high-definition equidistant cylindrical projection image data, the partial image data output from the projection method conversion unit 18 remains at the equidistant cylindrical projection image resolution (either 2K, 4K, or 8K). , Data converted into a predetermined projection method. On the other hand, the entire image data output from the low-definition conversion unit 17 is data of lower definition (lower resolution) than the equirectangular cylindrical projected image, such as 1K, 2K, and 4K. Thus, low definition means that the resolution is lower than the original image data, and that the resolution is relatively low when compared relatively. The low definition changing unit 17 is an example of a changing unit.

<Functional configuration of smartphone 5>
The smartphone 5 includes a transmission/reception unit 51, a reception unit 52, a decoding unit 53, a superimposition region creation unit 54, an image creation unit 55, an image superposition unit 56, a projection method conversion unit 57, and a display control unit 58. Each of these units is a function or means realized by any one of the components shown in FIG. 10 being operated by an instruction from the CPU 501 according to a smartphone program expanded from the EEPROM 504 to the RAM 503. ..

(Functional configuration of smartphone 5)
Next, each functional configuration of the smartphone 5 will be described in more detail with reference to FIG.

The transmission/reception unit 51 transmits data to the outside of the smartphone 5 and receives data from the outside. For example, the transmission/reception unit 51 receives image data from the transmission/reception unit 11 of the omnidirectional imaging device 1, or transmits instruction data to the transmission/reception unit 11 of the omnidirectional imaging device 1. Further, the transmission/reception unit 51 separates the image data (entire image data and partial image data) sent from the transmission/reception unit 11 of the omnidirectional imaging device 1 from the partial image parameter.

The accepting unit 52 accepts from the user an operation of designating the direction, the angle of view and the aspect of the partial image, and the size of the image data received by the smartphone 5.

The decoding unit 53 decodes each data of the whole image and the partial image encoded by the encoding unit 19.

The overlapping area creating unit 54 creates an overlapping area specified by the partial image parameter. This superposition area indicates the superposition position and superposition range of the superposition image S and the mask image M on the omnidirectional image CE.

The image creating unit 55 creates the superimposed image S and the mask image according to the overlapping region, and creates the celestial sphere image CE from the low-definition entire image.

The image superimposing unit 56 superimposes the superimposition image S and the mask image M on the superimposition region on the celestial sphere image CE to create a final celestial sphere image CE.

The projection method conversion unit 57 converts the final spherical image CE into an image of the perspective projection method according to the user's instruction received by the reception unit 52.

The display control unit 58 controls the display 517 or the like to display the image converted into the perspective projection method.

[Process or Operation of Embodiment]
Next, the processing or operation of this embodiment will be described with reference to FIGS. 13 to 20.

<Processing or operation of spherical imaging device 1>
First, the processing of the omnidirectional imaging device 1 will be described with reference to FIG. FIG. 13 is a conceptual diagram of an image in the process of image processing performed by the celestial sphere imaging device 1 according to the first embodiment.

The image processing unit 15 combines the two hemispherical image data obtained by the imaging units 14a and 14b with the data of the equidistant cylinder projection image which is the image of the equidistant cylinder projection method (joining process) (S120). The data after the image combination is temporarily stored in the temporary storage unit 16 in the state of the high definition image.

Next, the partial image parameter creation unit 12 creates a partial image parameter based on the instruction data sent from the smartphone 5 (S130).

Next, the low-definition changing unit 17 changes the high-definition image to the low-definition image according to the instruction data from the smartphone 5 received by the transmitting/receiving unit 11 (S140). That is, the low-definition changing unit 17 reduces the equidistant cylindrical projected image according to the size of the image data transmitted to the smartphone 5. As a result, a low-definition equidistant cylindrical projection image (entire image) is generated.

Further, the projection method changing unit 18 follows the instruction data received by the transmitting/receiving unit 11, that is, the direction, the angle of view and the aspect of the partial image which is a partial area of the entire image, and the image data to be transmitted to the smartphone 5. According to the size, the equirectangular cylindrical projection method is converted into the perspective projection method (S150). As a result, the partial image is generated in the high definition state. After that, the transmission/reception unit 11 transmits the whole image data and the partial image data encoded by the encoding unit 19 to the transmission/reception unit 51 of the smartphone 5.

Here, the processing shown in FIG. 13 will be described in more detail with reference to FIGS. 14 and 15. FIG. 14 is a diagram illustrating partial image parameters.

(Partial image parameter)
Here, the partial image parameters will be described in detail with reference to FIG. FIG. 14A shows the entire image after the image combination in S120 and the cutout area CA of the partial image on the entire image. FIG. 14B is a diagram showing an example of partial image parameters. FIG. 14C shows a partial image after the projection method has been converted in S150.

The azimuth angle (aa) described in FIG. 7 is the lateral direction (latitude λ) in the equidistant azimuth projection method illustrated in FIG. 14A, and the elevation angle (ea) described in FIG. Longitude φ). The partial image parameter is a parameter for specifying the area of the partial image in the equidistant cylindrical projection image. Specifically, the partial image parameters are, as shown in FIG. 14B, the gazing point CP(aa,ea) as the center point, the angle of view α, and the horizontal (w) and vertical (h). This is indicated by the information indicating the aspect ratio. FIG. 14C is an example of a partial image in which the portion surrounded by the frame in the equirectangular projection of FIG. 14A is cut out by the partial image parameter.

Here, the conversion of the projection method will be described. As shown in FIG. 4A, a spherical image is created by covering a solid sphere with an equirectangular projection image. Therefore, each pixel data of the equidistant cylinder projection image can be made to correspond to each pixel data on the surface of the three-dimensional spherical image of the three-dimensional spherical image. Therefore, the conversion method by the projection method conversion unit expresses the coordinates in the equirectangular cylindrical projected image as (latitude, longitude)=(e, a), and the coordinates on the three-dimensional solid sphere as orthogonal coordinates (x, y, When expressed by z), it can be expressed by the following (formula 1).
(x, y, z) = (cos(e) × cos(a), cos(e) × sin(a), sin(e)) (Equation 1)
However, the radius of the solid sphere at this time is 1.

On the other hand, the partial image which is a perspective projection image is a two-dimensional image, but if this is expressed by two-dimensional polar coordinates (radial radius, declination angle)=(r, a), the radial radius becomes the diagonal angle of view. Correspondingly, the possible range is 0 ≤ r ≤ tan (diagonal angle of view/2). When the partial image is represented by a two-dimensional orthogonal coordinate system (u, v), the conversion relationship with polar coordinates (radial radius, argument)=(r, a) can be represented by the following (Equation 2). it can.
u = r × cos(a), v = r × sin(a) (Equation 2)
Next, it is considered that this (Equation 1) is associated with three-dimensional coordinates (radius, polar angle, azimuth angle). Since only the surface of the solid sphere CS is considered now, the radius vector in three-dimensional polar coordinates is "1". In addition, the projection for perspective perspective transformation of the equirectangular cylindrical projection image attached to the surface of the solid sphere CS is considered to have a virtual camera at the center of the solid sphere, and the above-mentioned two-dimensional polar coordinates (radius and declination)=( Using r and a), they can be expressed by the following (formula 3) and (formula 4).
r = tan (polar angle) (Equation 3)
a = azimuth angle (Equation 4)
Here, if the polar angle is t, then t = arctan(r), so the three-dimensional polar coordinates (radial, polar, azimuth) are (radial, polar, azimuth) = (1, arctan( It can be expressed as r), a).

A conversion formula for converting the three-dimensional polar coordinates into the orthogonal coordinate system (x, y, z) can be expressed by the following (Formula 5).
(x, y, z) = (sin(t) × cos(a), sin(t) × sin(a), cos(t)) (Equation 5)
By the above (formula 5), it becomes possible to perform mutual conversion between the whole image by the equidistant cylindrical projection method and the partial image by the perspective projection method. That is, by using the radius vector r corresponding to the diagonal angle of view of the partial image to be created, it is possible to calculate the conversion map coordinate indicating which coordinate in the equidistant cylindrical projection image each pixel of the partial image corresponds to. A partial image that is a perspective projection image can be created from the equirectangular projection image based on the conversion map coordinates.

By the way, the conversion of the projection method is a conversion in which the position where (latitude, longitude) of the equirectangular cylindrical projection image is (90°, 0°) is the center point of the partial image which is the perspective projection image. ing. Therefore, when performing perspective projection conversion with an arbitrary point of the equidistant cylindrical projection image as the gazing point, the coordinates (latitude, longitude) of the gazing point can be changed by rotating the solid sphere to which the equidistant cylindrical projection image is pasted. The coordinates may be rotated so that the coordinates are arranged at the position of (90°, 0°).

Since the conversion formula regarding the rotation of the solid sphere is a general coordinate rotation formula, its explanation is omitted.

(Image data to be sent)
Next, the image data transmitted from the omnidirectional imaging device 1 to the smartphone 5 will be described in detail with reference to FIG. FIG. 15 is a conceptual diagram of image data transmitted from the omnidirectional imaging device 1 to the smartphone 5. As shown in FIG. 15, according to a predetermined image size for transmission, the whole image is arranged at the upper part of one image and the partial image is arranged at the lower part thereof. The image thus associated is stored in the SRAM 113, the DRAM 114, or the like before transmission, if necessary. In the present embodiment, it is generally arranged in accordance with 16:9 which is an aspect of HD or the like, but the aspect does not matter. Further, the arrangement is not limited to the top and bottom, and may be the left and right. When there are a plurality of partial images, for example, the upper half may be divided into the entire image and the lower half may be divided into the partial images. By combining the whole image and the partial image into one, it is possible to guarantee the synchronization between the images. Further, the spherical image capturing apparatus 1 may separately transmit the whole image data and the partial image data to the smartphone 5, as long as the smartphone 5 can display the images in a superimposed manner.

<Processing or operation of smartphone 5>
Next, the processing of the smartphone 5 will be described with reference to FIG. FIG. 16 is a conceptual diagram of an image in the process of image processing performed by the smartphone 5 according to the first embodiment. The superimposition region creation unit 54 shown in FIG. 12 creates the partial solid sphere PS designated by the partial image parameter as shown in FIG. 16 (S320).

Next, the image creating unit 55 creates the superimposed image S by superimposing the partial image of the perspective projection method on the partial solid sphere PS (S330). Further, the image creating unit 55 creates the mask image M based on the partial solid sphere PS (S340). Further, the image creating unit 55 creates the omnidirectional image CE by pasting the whole image of the equidistant cylindrical projection method on the solid sphere CS (S350). Then, the image superimposing unit 56 superimposes the superimposition S and the mask image M on the omnidirectional image CE (S360). As a result, the low-definition celestial sphere image CE on which the high-definition superimposed image S is superimposed so that the boundary is inconspicuous is completed.

Next, the projection method conversion unit 57 causes the display 517 to display a predetermined region in the celestial sphere image CE in the state in which the superimposed image S is superimposed on the basis of the line-of-sight direction and the angle of view of the virtual camera specified by the viewer. Projective transformation is performed so that the user can browse (S370). Accordingly, the display control unit 58 causes the display 517 to display the predetermined area image Q which is the predetermined area in the omnidirectional image CE (S380).

FIG. 17 is a diagram illustrating the creation of a partial solid sphere from a partial plane. Normally, in the perspective projection method, projection is performed on a plane, so that it is often represented by a plane in a three-dimensional space as shown in FIG. In the present embodiment, as shown in FIG. 17B, a partial solid sphere that is a part of the sphere is used in accordance with the spherical image. Here, conversion from a plane to a partial solid sphere will be described.

Consider the projection of each point (X, Y, Z) on a plane arranged with an appropriate size (angle of view) onto a sphere as shown in FIG. The projection on the sphere is the intersection of the sphere and the line passing from each point (X, Y, Z) from the origin of the sphere. Each point on the sphere is a point whose distance from the origin is equal to the radius of the sphere. Therefore, assuming that the radius of the sphere is 1, the point (X', Y', Z') on the spherical surface shown in FIG. 17B is represented by the following (Equation 6).

(X′,Y′,Z′)=(X,Y,Z)×1/√(X ² +Y ² +Z ² )... (Equation 6)
FIG. 18 is a two-dimensional conceptual diagram in which the partial image is superimposed on the omnidirectional image without creating the partial solid sphere of the present embodiment. FIG. 19 is a two-dimensional conceptual diagram when the partial stereoscopic sphere of this embodiment is created and the partial image is superimposed on the omnidirectional image.

As shown in FIG. 18A, when the virtual camera IC is located at the center point of the solid sphere CS, the subject P1 is represented as an image P2 on the omnidirectional image CE. , Is shown as an image P3 on the superimposed image S. As shown in FIG. 18A, since the images P2 and P3 are located on the straight line connecting the virtual camera IC and the subject P1, the superimposed image S is superimposed on the celestial sphere image CE. Even when displayed in the open state, the spherical image CE and the superimposed image S are not displaced. However, as shown in FIG. 18B, when the virtual camera IC is separated from the center point of the solid sphere CS, the image P2 is located on the straight line connecting the virtual camera IC and the subject P1. , The image P3 is located slightly inside. Therefore, assuming that the image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is an image P3′, the celestial sphere image CE and the superimposed image S are misaligned between the images P3 and P3′. A gap of the amount g occurs. As a result, the superimposed image S is displayed with a deviation from the omnidirectional image CE. In this way, the display may be superimposed.

On the other hand, further, in the present embodiment, since the partial solid sphere is created, the superimposed image S is converted into the spherical image CE as shown in FIGS. 19(a) and 19(b). Can be superimposed along. Accordingly, not only when the virtual camera IC is located at the center point of the solid sphere CS as shown in FIG. 19A, but as shown in FIG. Even if the image P2 and the image P3 are separated from the center point of the solid sphere CS, the images P2 and P3 are located on the straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superimposed image S is displayed in a state of being superimposed on the omnidirectional image CE, no deviation occurs between the omnidirectional image CE and the superimposed image S.

20A is a display example of a wide image without overlay display, FIG. 20B is a display example of a tele image without overlay display, and FIG. 20C is a display example of a wide image with overlay display. 20(d) is a conceptual diagram showing a display example of a tele image in the case of superimposing display. Note that the wavy line in the figure is shown only for convenience of description, and may or may not be actually displayed on the display 517.

As shown in FIG. 20A, when the partial image P is not displayed by being superimposed on the omnidirectional image CE, when the area shown by the broken line in FIG. As shown in (b), the low-definition image remains, and the user sees an unclear image. On the other hand, as shown in FIG. 20(c), when the partial image P is superimposed and displayed on the omnidirectional image CE, it is enlarged to the area indicated by the dotted line in FIG. 20(c). When displayed, a high-definition image is displayed as shown in FIG. 20D, and the user can see a clear image. In particular, when a signboard or the like in which characters are drawn is displayed in the area indicated by the wavy line, the characters will be blurred even if they are enlarged and displayed unless the high-definition partial image P is superimposed and displayed. I don't know if is written. However, when the high-definition partial image P is superimposed and displayed, the characters can be seen clearly even when enlarged and displayed, so that the user can understand what is written.

[Main effects of the embodiment]
As described above, according to the present embodiment, the omnidirectional imaging device 1 creates a low-definition entire image from a high-definition omnidirectional image (S140), and projects from the same high-definition omnidirectional image. High-definition partial images of different methods are created (S150). Then, the omnidirectional imaging device 1 transmits each data of the low-definition whole image and the high-definition partial image to the smartphone 5. On the other hand, the smartphone 5 superimposes the high-definition partial image on the low-definition entire image (S360) and converts the projection method according to the line-of-sight direction and the angle of view specified by the user (viewer). Yes (S370). As described above, the omnidirectional image capturing apparatus 1 transmits the high-definition omnidirectional image obtained by capturing an image of a subject or the like as a high-definition partial image that is a region of interest. The whole image for grasping the whole celestial sphere image is converted to low definition before being transmitted, and the partial image of high definition is transmitted after conversion of the projection method. As a result, in the smartphone 5 on the receiving side, the amount of data can be reduced as compared with the related art, and therefore, it is possible to quickly display the omnidirectional image in which the partial image is superimposed on the entire image.

Therefore, according to the present embodiment, in order to reduce the communication amount, the omnidirectional image capturing apparatus 1 reduces the entire omnidirectional image and a low-definition entire image and the attention in the omnidirectional image (overall image). The data of the high-definition partial image that is the area to be distributed is distributed, and not only the partial image is combined with the whole image by the smartphone 5 on the receiving side, but also the low-definition whole image and the high-definition partial image are projected differently. Even if there is, it can be combined and displayed on the smartphone 5, so that the versatility of the projection method is high.

Second Embodiment Hereinafter, a second embodiment of the present invention will be described.

[Outline of shooting system]
First, the outline of the imaging system according to the second embodiment will be described with reference to FIG. FIG. 21 is a schematic diagram of the configuration of an imaging system according to the second embodiment.

As shown in FIG. 21, the imaging system of this embodiment is configured by the omnidirectional imaging device 1a and smartphones 5a, 5b, 5c. User A operates the smartphone 5a. In this case, the user A is also a viewer who browses the image displayed on the smartphone 5a. User B operates the smartphone 5b. In this case, the user B is also a viewer who browses the image displayed on the smartphone 5b. User C operates the smartphone 5c. In this case, the user C is also a viewer who browses the image displayed on the smartphone 5c. The above-mentioned instruction data can be transmitted from the smartphones 5a and 5b to the omnidirectional image capturing apparatus 1a, but the above-mentioned instruction data cannot be transmitted from the smartphone 5c to the omnidirectional image capturing apparatus 1a. ..

The omnidirectional imaging device 1a has the same hardware configuration as the omnidirectional imaging device 1 according to the first embodiment, but a new functional configuration described later is added. The smartphones 5a and 5b have the same hardware configuration as the smartphone 5 according to the first embodiment, but a new functional configuration described later is added. The smartphone 5c has a hardware configuration similar to that of the smartphone 5 according to the first embodiment, but conversely, a part of the function is deleted. Note that FIG. 21 shows two smartphones 5a and 5b having the same function and one smartphone 5c having a different function, but this is just an example. Therefore, there may be three or more smartphones having the same function as the smartphone 5a and two or more smartphones having the same function as the smartphone 5c.

Further, the omnidirectional imaging device 1a has the same hardware configuration as the omnidirectional imaging device 1 (see FIG. 9) of the first embodiment, and thus the description thereof will be omitted. Further, since the smartphones 5a, 5b, 5c have the same hardware configuration as the smartphone 5 (see FIG. 10), the description will be omitted.

[Functional Configuration of Embodiment]
Next, the functional configuration of this embodiment will be described with reference to FIGS. 22 to 24. FIG. 22 is a functional block diagram of the celestial sphere imaging device 1a according to the second embodiment. FIG. 23 is a functional block diagram of the smartphones 5a and 5b according to the second embodiment. FIG. 24 is a functional block diagram of the smartphone 5c according to the second embodiment.

<Functional configuration of spherical imaging device 1a>
As shown in FIG. 22, the omnidirectional imaging device 1a further includes an instruction selection unit 21 and an instruction data restriction unit 22 in addition to the omnidirectional imaging device 1 according to the first embodiment. ing. Each of these units is realized by any one of the components shown in FIG. 9 being operated by an instruction from the CPU 111 according to a program for the omnidirectional image capturing device expanded from the SRAM 113 to the DRAM 114. Function or means.

(Each functional configuration of the spherical imaging device 1a)
Next, with reference to FIG. 22, the instruction selection unit 21 and the instruction data restriction unit 22 of the omnidirectional imaging device 1a will be described in more detail.

The instruction selection unit 21 determines whether or not the omnidirectional imaging device 1a uses the instruction data based on the instruction right request data added to the instruction data transmitted from the smartphones 5a and 5b via the transmission/reception unit 11. Make a choice. The instruction right request data is data indicating a request for prioritizing the instruction data from the transmission source smartphone to be used by the omnidirectional imaging device 1a, such as a terminal ID for identifying the smartphones 5a and 5b. It is included. For example, when the instruction selection unit 21 sends the instruction data from the smartphone 5b to the omnidirectional imaging device 1a immediately after the instruction data is sent from the smartphone 5a, the omnidirectional shooting is performed. The device 1a uses the instruction data sent from the smartphone 5a to perform the projection method conversion and the like, and does not use the instruction data sent from the smartphone 5b. In addition, the instruction selection unit 21 temporarily (for example, for 30 seconds) all or part (for example, the terminal ID) of the instruction right request data added to the instruction data first sent from a predetermined smartphone. ) Hold by yourself, and while holding this temporarily, check whether it is the same as the terminal ID of the instruction request data added to the instruction data sent from another smartphone. If it is determined that they are not the same, the instruction data restriction unit 22 is controlled not to use the instruction data sent from another smartphone.

When the angle of view of the cutout area CA of the partial image indicated by the instruction data is less than the threshold (horizontal angle of view threshold, vertical angle of view threshold) for each image size for transmission, which will be described later, the instruction data restriction unit 22 determines the partial image. The instruction data is restricted so that the partial images of the threshold value are browsed on the smartphone 5a, 5b, 5c side so that the cutout area CA of is not too small.

The partial image parameter creation unit 12 of the second embodiment uses the partial image parameters based on the instruction data after the instruction data sent from the smartphone 5 via the transmission/reception unit 11 is restricted by the instruction data restriction unit. To create.

(Limitation of reduction of cutout area)
Here, with reference to FIG. 25 and FIG. 26, a description will be given of limiting the cutout area CA of the partial image from becoming too small. The purpose and effect of this restriction will be described later using an example. FIG. 25A is a conceptual diagram of the instruction data restriction table, and FIG. 25B is a schematic diagram of the equidistant cylindrical projection image. The instruction data restriction table is stored in the SRAM 113 and managed by the instruction data restriction unit 22.

As shown in FIG. 25A, the instruction data restriction table manages a plurality of setting patterns of setting 1 to setting 4. For each set pattern, the number of horizontal pixels and vertical pixels of the high-definition image, the number of horizontal pixels and vertical pixels of the partial image, and the size of the high-definition image and the partial image size in the image size for transmission (see the lower part of FIG. 15). ), the threshold value of the angle of view (horizontal view angle threshold value and vertical view angle threshold value) of the cutout area CA of each partial image is managed in association with each other. The angle of view of the instruction data may be either the horizontal angle of view threshold or the vertical angle of view threshold. In FIG. 25A, four setting patterns are shown, but this is an example, and one or more and three or less or five or more may be set. Further, a calculation formula may be used without using the instruction data restriction table. Here, the setting pattern “setting 1” will be specifically described.

For example, the data of the high-definition equirectangular cylindrical projection image stored in the temporary storage unit 16 is a 2:1 image of horizontal 360 degrees and vertical 180 degrees as shown in FIG. When the horizontal size of the image is 4000 pixels (pix) and the vertical size is 2000 pixels (pix), the angle per pixel is 0.09 degrees (=360/4000). When a partial image is a perspective projection image obtained by conversion by the perspective projection method, the ratio of the angle of view and the pixel is not constant depending on the area in the perspective projection image, but as a guideline, for example, When the number of pixels is 1920, 1920×0.09=172.8 degrees, and the threshold of the horizontal field angle of the image size for transmission is managed as 172.8 degrees. Similarly, when the vertical pixel of the partial image is 1080 pixels, 1080×0.09=97.2 degrees, and the threshold of the vertical angle of view of the image size for transmission is managed as 97.2 degrees. As a result, even when the cutout area CA of the partial image smaller than the horizontal angle of view of 172.8 degrees is specified by the instruction data sent from the smartphone 5a to the celestial sphere imaging device 1a, the instruction data The 22 restriction unit restricts the angle of view so that the 172.8 degree region is converted into a partial image.

Further, processing for limiting the cutout area CA of the partial image from becoming too small will be described in detail with reference to FIG. FIG. 26A is a conceptual diagram of image data transmitted from the omnidirectional imaging device to the smartphone when the instruction data is not restricted, and FIG. 26B is a omnidirectional image capturing when the instruction data is restricted. It is a conceptual diagram of the image data transmitted from the apparatus 1a to a smart phone.

In the first embodiment, as shown in FIG. 26A, the low-definition changing unit 17 adjusts to the image size for transmission when changing the entire image to low-definition based on the instruction data. To reduce. Further, when the cutout area CA of the partial image is specified in the equidistant cylindrical projection image EC1 as the entire image, the projection method conversion unit 18 extracts the partial image P11 from the entire image and converts the projection method. Then, processing corresponding to enlargement is performed according to the image size for transmission.

On the other hand, in the second embodiment, as shown in FIG. 26B, when the partial image P21 is specified in the equidistant cylindrical projection image EC1 as the entire image by the instruction data, The low-definition changing unit 17 reduces the entire image according to the image size for transmission when changing the entire image to low-definition, as in the first embodiment. However, the projection method conversion unit 18 converts the projection method by extracting the partial image P22 having the same size as the transmission image size in order to extract the image according to the transmission image size.

<Functional configuration of smartphones 5a and 5b>
As shown in FIG. 23, the smartphones 5a and 5b further include an instruction right claiming unit 59 with respect to the smartphone 5 according to the first embodiment. The instruction right claiming unit 59 is a function realized by any of the components shown in FIG. 10 being operated by an instruction from the CPU 501 according to a smartphone program expanded from the EEPROM 504 to the RAM 503, or It is a means.

(Each functional configuration of the smartphones 5a and 5b)
Hereinafter, only the functional configuration of the instruction right claiming unit 59 not described in the first embodiment will be described.

The instruction right requesting unit 59 issues an instruction right (also referred to as instruction request right data) for preferentially causing the omnidirectional imaging device 1a to use the instruction data when the receiving unit 52 receives the instruction from the user. To do. The issued instruction right is transmitted together with instruction data that the transmitting/receiving unit 51 transmits to the celestial sphere imaging device 1a. As shown in FIG. 21, in the case of a system in which a plurality of smartphones 5a and 5b transmit instruction data to the celestial sphere imaging device 1a, instructions may conflict. Therefore, in the second embodiment, the idea of the instruction right is introduced.

<Functional configuration of smartphone 5c>
As shown in FIG. 24, the smartphone 5c has the same functional configuration except that the receiving unit 52 cannot receive the user's instruction from the smartphone 5 of the first embodiment shown in FIG. Therefore, the description is omitted. That is, unlike the smartphones 5a and 5b, the smartphone 5c does not have a function of transmitting instruction data to the omnidirectional imaging device 1a. Therefore, the smartphone 5c may not conflict with the instructions of the smartphones 5a and 5b.

[Process or Operation of Embodiment]
Subsequently, the processing or operation of the second embodiment will be described using FIGS. 27 and 28. In addition, in this embodiment, only a process or an operation different from that of the first embodiment will be described.

<Processing or operation of spherical imaging device 1a>
First, the processing or operation of the spherical imaging device 1a will be described with reference to FIG. FIG. 27 is a conceptual diagram of an image in the process of image processing performed by the omnidirectional imaging device 1a according to the second embodiment.

As shown in FIG. 27, the instruction selecting unit 21 of the celestial sphere imaging device 1a receives the first instruction data from the smartphone 5a and the second instruction data from the smartphone 5b. When it comes, the instruction data to be used is selected (S110). In this case, the instruction selection unit 21 selects the instruction data to be used based on the instruction request data sent in addition to the instruction data. For example, the instruction selection unit 21 selects the instruction data previously received by the transmission/reception unit 11 for preferential use. The instruction right request data is issued by each instruction right requesting unit 59 of the smartphones 5a and 5b.

Next, when the instruction data is less than the threshold of the image size for transmission, the instruction data restriction unit 22 limits the cutout area CA of the partial image from becoming too small, and the instruction data restriction unit 22 restricts the smartphone 5a. , 5b, 5c are limited to the instruction data so that the partial image of the threshold value is displayed (S110).

After that, instead of the instruction data in the first embodiment, the instruction data after the limitation in the processing in step S110 is used in the processing in step S130 and subsequent steps. Here, the difference in display on the smartphone between the first embodiment and the second embodiment will be described with reference to FIG. 28. In FIG. 28, (a) is a display example of the partial image P11 when no restriction is applied, (b) is a display example of an area other than the partial image P11 in the entire image when no restriction is applied, and (c) is a restriction. A display example of the partial image P21 when applied and a conceptual diagram of the transmitted partial image P22, (d) is a display example of an area other than the partial image P21 in the partial image P22 when restricted. 28(a) and 28(b) are images in the case of the process of FIG. 26(a), and FIGS. 28(c) and 28(d) are images in the case of the process of FIG. 26(b).

As shown in FIG. 26A, when the user Y specifies the partial image P11 on the smartphone 5, the display control unit 58 of the smartphone 5 is sent from the omnidirectional camera 1. As the image, first, a predetermined area image Q11 as shown in FIG. 28A is displayed. In the first predetermined area image Q11, the central portion of the castle tower corresponds to the partial image P11 in FIG. Here, when the user Y performs an operation for displaying the upper part of the castle tower in the predetermined area image shown in FIG. 28A, the display control unit 58 is shown in FIG. 28B. Such a predetermined area image Q12 is displayed. As shown in FIG. 28( b ), except for the central portion of the castle tower, the image quality is a partial enlargement of the low-definition entire image (corresponding to the equirectangular cylindrical projected image EC 1 of FIG. 26( a )). Is bad. If the image shown in FIG. 28( b) is to be displayed in high definition, the smartphone 5 designates the area shown in FIG. 28( b) for the omnidirectional camera 1. The designated data for will be transmitted again. However, even if the smartphone 5 retransmits the instruction data to the omnidirectional imaging device 1, the updated partial image data is not immediately sent from the omnidirectional imaging device 1 due to a delay or the like. A low-definition image as shown in FIG. 28B is temporarily displayed. In addition, when a plurality of users browse images on their respective smartphones, even if a user who does not have an instruction right tries to browse an area other than the area indicated by the instruction data, the celestial sphere image capturing device 1 can be used. Since fine image data cannot be obtained, the image displayed on the smartphone becomes low-definition as shown in FIG.

On the other hand, as shown in FIG. 26B, when the user A obtains the instruction right and specifies the partial image P21 on the smartphone 5a, it is sent from the omnidirectional imaging device 1a. The image is the partial image P22 showing the whole of FIG. 28(c), but the display control unit 58 of the smartphone 5a displays the partial image P21 of the area within the broken line of FIG. 28(c) as the first predetermined area image Q21. Let The image sent from the omnidirectional imaging device 1a corresponds to the partial image P22 in FIG. 26(b) which is the entire part of the castle tower. The first predetermined area image Q21 corresponds to the partial image P21 in FIG. 26(b), which is the central portion of the castle tower.

Here, even if the user A performs an operation to display the upper part of the castle tower while the smartphone 5a is displaying the central part of the castle tower in the predetermined area image Q21 shown in FIG. Until the data of the high-definition partial image of the upper part of the castle tower arrives from the device 1a, the area of the upper part of the castle tower in the image already shown in FIG. 28(c) is shown in FIG. 28(d). By displaying the predetermined area image Q22 as described above, the image quality does not deteriorate extremely. Further, when one of the users B and C performs an operation of displaying the upper part of the castle tower in the predetermined area image shown in FIG. 28C with the smartphone 5b or 5c of the user, the display control unit 58 The predetermined area image Q22 as shown in FIG. 28D is displayed. As shown in FIG. 28(c), since the entire image of the castle tower is a high-definition partial image, the display control unit 58 causes the display controller 58 to display an image of the upper part of the castle tower as shown in FIG. 28(d). Even when is displayed, the image quality does not deteriorate significantly.

Further, since the user C is using the smartphone 5c that cannot acquire the instruction right, he/she cannot send the designated data again with his/her own smartphone 5c, but the partial image already received by his/her own smartphone 5c cannot be displayed. High-definition images can be viewed by simply enlarging a part.

[Main effects of the embodiment]
As described above, according to the present embodiment, when the instruction data is less than the threshold of the transmission image size, the instruction data restriction unit 22 restricts the cutout area CA of the partial image from becoming too small, The instruction data is restricted so that the partial images of the threshold value are displayed on the smartphone 5a, 5b, 5c side. As a result, in addition to the effect of the first embodiment, the user A who has the operation right is viewing the partial image P21 indicated by the instruction data as the predetermined area image Q21 while surrounding the partial image P21 in the partial image 22. When the area (predetermined area image Q22) is displayed, the image quality of the predetermined area image to be displayed is extremely poor even while the data of the high-definition partial image is sent again from the omnidirectional imaging device 1a. Has the effect of not becoming. Moreover,
Since the size of the partial image transmitted from the omnidirectional imaging device 1a to the smartphone 5a is not increased (the amount of data is increased) to expand the high-quality image area, the amount of data to be transmitted is not changed, and There is an effect that the image quality of the predetermined area image Q21 does not deteriorate. Further, when users B and C who do not have the operation right display the area (predetermined area image Q22) around the partial image P21 in the partial image 22 while browsing the desired area in the entire image, , The effect that the image quality is not extremely deteriorated. Thereby, the readability of the user can be improved.

[Supplement] The functional configuration other than a part (the image capturing unit 14a, the image capturing unit 14b, the image capturing control unit 13, and the image processing unit 15) of the spherical image capturing apparatus 1 illustrated in FIG. The image processing server, which is another device, may be realized instead of being realized in 1. In this case, the image processing server performs data communication with the omnidirectional imaging device 1 and the smartphone 5 via a communication network such as the Internet. Similarly, the functional configuration other than a part (the image capturing unit 14a, the image capturing unit 14b, the image capturing control unit 13, and the image processing unit 15) of the omnidirectional image capturing apparatus 1a illustrated in FIG. The image processing server, which is a separate device, may be realized instead of being realized within 1a. In this case also, the image processing server performs data communication with the omnidirectional imaging device 1a and the smartphones 5a, 5b, 5c via a communication network such as the Internet.

Further, the omnidirectional imaging device 1, 1a is an example of the imaging device, and the imaging device also includes a digital camera, a smartphone, or the like that obtains a normal plane image. In the case of a digital camera or a smartphone that obtains a normal plane image, it is possible to obtain a relatively wide-angle image (wide-angle image) instead of obtaining a spherical image.

The smartphones 5, 5a, 5b, 5c are examples of communication terminals or image processing devices, and the communication terminals or image processing devices include tablet PCs (Personal Computers), notebook PCs, desktop PCs, and the like. PC is also included. The communication terminal or the image processing device includes a smart watch, a game machine, a car navigation terminal mounted on a vehicle, or the like.

In the above embodiment, a low-definition image is described as a whole image that is the entire region of the image of the image data obtained from the imaging units 14a and 14b, and a high-definition image is a partial image that is a partial region of the whole region. However, it is not limited to this. The low-definition image may be an image of a partial area A1 of the entire area of the image of the image data obtained from the imaging units 14a and 14b. In this case, the high-definition image is an image of a further partial area A2 of the partial area A1. That is, regarding the relationship between the low-definition image and the high-definition image, the former is a wide-angle image, while the latter is a narrow-angle image.

In the above embodiment, the case where the plane image is superimposed on the omnidirectional image has been described, but the superposition is an example of composition. In addition to superimposing, combining includes pasting, fitting, superimposing, and the like. The superimposed image is an example of a composite image. The overlapping area is an example of a composite area. In addition to the superimposed image, the composite image also includes a pasted image, a fitting image, a superimposed image, and the like. Further, the image superimposing unit 56 is an example of an image synthesizing unit.

Further, both the equirectangular cylindrical projected image EC and the plane image P may be either still images, both moving image frames, one still image and the other moving image frame.

Further, in the above embodiment, the projection method conversion unit 18 converts the projection method of the high definition image data acquired from the temporary storage unit 16 with the same definition, but the present invention is not limited to this. For example, if the definition is higher than the overall image data output from the low definition changing unit 17, the projection method conversion unit 18 sets the definition of the image data acquired from the temporary storage unit 16 when converting the projection method. You can lower it.

Each functional configuration shown in FIGS. 11 and 12 is realized in the form of a software functional unit, and can be stored in a computer-readable storage medium when sold or used as an independent product. In this case, in the technical solution of the present embodiment, the essential or the part that contributes to the conventional technology or the part of the technical solution is expressed in the form of a software product. The computer software product is stored in a storage medium, and stores a plurality of instructions for causing a computer device (personal computer, server, network device, etc.) to execute all or some of the steps of the method according to each of the embodiments. Including. The above-mentioned storage medium includes various media such as a USB memory, a removable disk, a ROM, a RAM, a magnetic disk, or an optical disk, which can store the program code.

Further, the method according to the above embodiments is applied to or realized by the processor. A processor is an integrated circuit board capable of processing signals. Each step of the method of each of the above-described embodiments is realized by an instruction in the form of an integrated logic circuit which is hardware in a processor or software. The processor is a general purpose processor, a digital signal processor (DSP), a dedicated integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, each of the above The methods, steps, and logic boxes disclosed in the embodiments can be implemented or executed. A general purpose processor may be a microprocessor or any general purpose processor or the like. Each step of the method according to each of the above embodiments may be realized by being executed by a decoder that is hardware, or may be realized by a combination of hardware and software in the decoder. The software modules are stored in storage media mature in the art, such as random memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers. From a memory with a storage medium in which this software is stored, the processor reads the information and, according to the hardware, implements the method steps.

The embodiments described above are realized by hardware, software, firmware, middleware, microcode, or a combination thereof. Among them, in terms of hardware implementation, the processing unit is one or more dedicated integrated circuits (ASIC), digital signal processor (DSP), digital signal processor (DSPD), programmable logic circuit (PLD), field programmable. It may be implemented by a gate array (FPGA), a general purpose processor, a controller, a microcontroller, a microprocessor, another electronic unit that performs the functions of the present invention, or a combination thereof. As for software implementation, the above technology is implemented by modules (for example, processes, functions, etc.) that implement the above-mentioned functions. The software code is stored in memory and executed by the processor. The memory is realized inside or outside the processor.

1 Spherical photographing device (an example of photographing device)
5 Smartphones (one example of communication terminal, one example of image processing device)
13 imaging control unit 14a imaging unit 14b imaging unit 15 image processing unit 16 temporary storage unit 17 low-definition changing unit (an example of changing means)
18 Projection method conversion unit (an example of projection method conversion means)
19 encoding unit 21 instruction selection unit 22 instruction data restriction unit 51 transmission/reception unit 52 reception unit 53 decoding unit 54 superimposition region creation unit 55 image creation unit 56 image superposition unit 57 projection method conversion unit 58 display control unit 59 instruction right requesting unit

JP, 2006-340091, A

Claims (13)

A photographing device for photographing a subject to obtain image data of a predetermined definition,
Changing means for changing the definition of a wide-angle image, which is all or a part of the image relating to the image data of the predetermined definition,
Projection method conversion means for converting a narrow angle image, which is a partial region of the wide angle image, with a definition different from that of the wide angle image,
An image pickup apparatus comprising: The photographing apparatus according to claim 1, wherein the changing unit changes the wide-angle image with a definition lower than the predetermined definition. The image capturing apparatus according to claim 1, wherein the wide-angle image is an entire image that is all the images related to the image data of the predetermined definition obtained by capturing the subject. 4. The photographing apparatus according to claim 1, wherein the projection method conversion unit converts the projection method with a predetermined definition higher than that of the wide-angle image. .. The photographing apparatus according to claim 1, wherein the photographing apparatus is a celestial sphere photographing apparatus that photographs a subject to obtain celestial sphere image data. The photographing device is
The image capturing apparatus according to claim 1, further comprising a storage unit that stores the data of the wide-angle image and the data of the narrow-angle image in association with each other. An image capturing apparatus according to claim 1,
A communication terminal that acquires the data of the wide-angle image and the data of the narrow-angle image from the photographing device, and synthesizes and displays the narrow-angle image in the partial area on the wide-angle image,
An imaging system having: The photographing device is
A receiving unit that receives instruction data indicating an instruction for specifying a composite area, which is the partial area in the wide-angle image, from the communication terminal,
When the size of the synthesis area indicated by the instruction data is less than a threshold value, the projection method conversion means performs the projection method conversion on the narrow-angle image that is the combination area indicated by the threshold value. Item 6. The imaging system according to Item 6. The photographing device is
9. The imaging system according to claim 7, further comprising a transmission unit that transmits the data of the wide-angle image and the data of the narrow-angle image in association with the communication terminal. The photographing device is
The image capturing system according to claim 9, wherein the transmitting unit transmits the image in accordance with a predetermined image size for transmission. The imaging system according to claim 7, wherein the communication terminal is a smartphone, a smart watch, a PC, or a car navigation terminal. A photographing method for photographing a subject to obtain image data of a predetermined definition,
A changing step of changing the definition of a wide-angle image, which is all or a part of the image related to the image data of the predetermined definition,
A projection method conversion step of performing a projection method conversion of a narrow-angle image, which is a partial region of the wide-angle image, with a different definition from the wide-angle image;
A shooting method characterized by executing. A program that causes a computer to execute the method according to claim 12.