JP6816403B2 - Image management system, image communication system, image management method, and program - Google Patents

Description

The present invention relates to an image management system, an image communication system, an image management method, and a program.

In recent years, a special digital camera that obtains a 360 ° spherical panoramic image with a single shooting has been provided (see Patent Document 1). However, although this special digital camera obtains an omnidirectional panoramic image, the image is curved and difficult for the user to see as it is. Therefore, a smartphone or the like is a part of the spherical panoramic image, and by displaying a predetermined area image in a predetermined area that enters the field of view from a predetermined viewpoint, the user can take an image taken by a conventional digital camera. You can browse the predetermined area image as if you were browsing the same plane image as.

On the other hand, conventionally, it is known that an attached image including this drawn image is created based on a drawn image drawn on a flat image by a user, and the attached image is attached to the flat image.

However, when the attached image is attached to the predetermined region image which is a part of the spherical panoramic image, the attached image is actually attached in a curved state according to the surface of the three-dimensional sphere. Therefore, if the position of the viewpoint is changed, a drawn image having a shape different from the shape of the drawn image at the time of drawing is displayed, which causes a problem that the user is confused.

The invention according to claim 1 is an image management system that manages captured images, and includes the captured image in the captured image based on the captured image drawn on the captured image which is an all-sky panoramic image. The image management system is characterized by having a creating means for creating an attached image for attachment and an attaching means for attaching the created attached image to the captured image as a flat image.

As described above, according to the present invention, since the attached image is attached to the captured image as a flat image without being curved, it is possible to eliminate the confusion of the user.

(A) is a left side view of the photographing device, (b) is a front view of the photographing device, and (c) is a plan view of the photographing device. It is a use image figure of a photographing apparatus. (A) is a hemispherical image (front) taken by the photographing device, (b) is a hemispherical image (rear) taken by the photographing device, and (c) is a diagram showing an image represented by the Mercator projection. (A) is a conceptual diagram showing a state of covering a sphere in a Mercator image, and (b) is a diagram showing a panoramic image of an omnidirectional sphere. It is a figure which showed the position of the virtual camera and a predetermined area when the whole celestial sphere panoramic image is made into a three-dimensional three-dimensional sphere. (A) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a communication terminal in which an image of a predetermined area is displayed on a display. It is a figure which showed the relationship between the predetermined area information and a predetermined area image. It is the schematic of the image communication system which concerns on embodiment of this invention. It is a hardware block diagram of a photographing apparatus. It is a hardware configuration diagram of a communication terminal 3. It is a hardware block diagram of an image management system and a communication terminal 7. It is a figure which showed the outline of the process of registration and acquisition of image data. It is a conceptual diagram which shows the photographed image management table. It is a conceptual diagram which shows the attached image management table. It is a flowchart which showed the process of deriving the attachment position of the attachment image in the photographed image. It is a flowchart which showed the process of attaching an attached image to a photographed image. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a two-dimensional drawing area in the image management system 5 and the communication terminal 7, and (b) is a conceptual diagram showing a three-dimensional virtual space in the image management system 5. (A) is a conceptual diagram showing a drawn image g1 in a predetermined area image with transparency processing and distortion correction, and (b) is a drawn image g9 in a predetermined area image with transparency processing and no distortion correction. It is a conceptual diagram which showed. (A) is a conceptual diagram showing a drawn image g9 in an omnidirectional panoramic image with transparency processing and no distortion correction, and (b) is an omnidirectional panoramic image without distortion correction without transmission processing. It is a conceptual diagram which showed the drawing image g9 in.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< Outline of Embodiment >>
<How to generate spherical panoramic images>
A method of generating a spherical panoramic image will be described with reference to FIGS. 1 to 7.

First, the appearance of the photographing apparatus 1 will be described with reference to FIG. The photographing device 1 is a digital camera for obtaining a photographed image that is the basis of the spherical panoramic image. Note that FIG. 1A is a left side view of the photographing device, FIG. 1B is a front view of the photographing device, and FIG. 1C is a plan view of the photographing device.

As shown in FIG. 1A, the photographing device 1 has a size that a human can hold with one hand. Further, as shown in FIGS. 1A, 1B, and 1C, the image sensor 103a is on the front side (front side) and the image sensor is on the back side (rear side) on the upper part of the photographing device 1. 103b is provided. These image pickup devices 103a and 103b are used in combination with an optical member (for example, a fisheye lens 102a and 102b described later) capable of capturing a hemispherical image (angle of view of 180 ° or more). Further, as shown in FIG. 1B, an operation unit 115 such as a shutter button is provided on the back side (rear side) of the photographing device 1.

Next, the usage state of the photographing apparatus 1 will be described with reference to FIG. Note that FIG. 2 is an image diagram of the use of the photographing apparatus. As shown in FIG. 2, the photographing device 1 is used, for example, for the user to hold in his / her hand and photograph a subject around the user. In this case, two hemispherical images can be obtained by taking an image of a subject around the user by the image sensor 103a and the image sensor 103b shown in FIG.

Next, the outline of the process from the image taken by the photographing apparatus 1 to the creation of the spherical panoramic image will be described with reference to FIGS. 3 and 4. Note that FIG. 3 (a) is a hemispherical image (front side) taken by the photographing device, FIG. 3 (b) is a hemispherical image (rear side) taken by the photographing device, and FIG. 3 (c) is represented by the Mercator projection. It is a figure which showed the image (hereinafter, referred to as "Mercator image"). FIG. 4A is a conceptual diagram showing a state of covering a sphere in a Mercator image, and FIG. 4B is a diagram showing a panoramic image of a spherical image.

As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by the fisheye lens 102a described later. Further, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by the fisheye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image (rear side) inverted 180 degrees are combined by the photographing apparatus 1, and a Mercator image is created as shown in FIG. 3 (c).

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4 (a), the Mercator image is pasted so as to cover the spherical surface, and in FIG. 4 (b). An omnidirectional panoramic image as shown is created. In this way, the spherical panoramic image is represented as an image in which the Mercator image faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The spherical panoramic image may be a still image or a moving image.

As described above, since the spherical panoramic image is an image pasted so as to cover the spherical surface, it gives a sense of discomfort to humans. Therefore, by displaying a part of a predetermined region of the spherical panoramic image (hereinafter, referred to as “predetermined region image”) as a flat image with less curvature, it is possible to display the image without giving a sense of discomfort to humans. This will be described with reference to FIGS. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the spherical panoramic image is a three-dimensional three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who sees the panoramic image of the spherical image displayed as a three-dimensional three-dimensional sphere. Further, FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on a display. Further, in FIG. 6A, the spherical panoramic image shown in FIG. 4 is represented by a three-dimensional three-dimensional sphere CS. Assuming that the spherical panoramic image generated in this way is a three-dimensional sphere CS, the virtual camera IC is located outside the spherical panoramic image as shown in FIG. The predetermined area T in the spherical panoramic image is specified by the predetermined area information of the position of the virtual camera IC in the spherical panoramic image. This predetermined area information is indicated by, for example, coordinates (x (rH), y (rV), and angle of view α (angle)) or coordinates (X, Y, Z). The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. Further, the zoom of the predetermined area T can be expressed by moving the virtual camera IC closer to or further away from the spherical panoramic image.

Then, as shown in FIG. 6 (a), the image of the predetermined region T in the spherical panoramic image is displayed as a predetermined region image on the predetermined display as shown in FIG. 6 (b). Is displayed. The image shown in FIG. 6B is an image represented by the predetermined area information (x, y, α) that has been initialized (default).

Here, the relationship between the predetermined area information and the predetermined area image will be described with reference to FIG. 7. Note that FIG. 7 is a diagram showing the relationship between the predetermined area information and the predetermined area image. As shown in FIG. 7, the center point CP when the diagonal angle of view 2L of the predetermined area T represented by the angle of view α of the virtual camera IC is set is the (x, y) parameter of the predetermined area information. .. f is the distance from the virtual camera IC to the center point CP. Then, in FIG. 7, the trigonometric function generally represented by the following equation (1) holds.

Lf = tan (α / 2) ... (Equation 1)
<Outline of image communication system>
Subsequently, the outline of the configuration of the image communication system of the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the image communication system of the present embodiment.

As shown in FIG. 8, the image communication system of the present embodiment is composed of a photographing device 1, a communication terminal 3, an image management system 5, and a communication terminal 7.

Of these, the photographing device 1 is a digital camera for obtaining a spherical panoramic image as described above. The photographing device 1 may be a general digital camera, and when the communication terminal 3 is equipped with a camera, the communication terminal 3 can be a digital camera. In the present embodiment, in order to make the explanation easy to understand, the description will be given as a digital camera for obtaining a spherical panoramic image. The communication terminal 3 can perform data communication with the photographing device 1 and data communication with the image management system 5 via the communication network 9 by using a wireless communication technology such as WiFi. The communication network 9 is, for example, the Internet.

Further, the image management system 5 is, for example, a server computer, and can perform data communication with the communication terminals 3 and 7 via the communication network 9. OpenGL ES is installed in the image management system 5, and a spherical panoramic image is created. Further, the image management system 5 creates predetermined area information (or a predetermined area image which is an image indicated by the predetermined area information) indicating a part of the panoramic image of the whole sky, and captures image data on the communication terminal 7. And provide predetermined area information (or predetermined area image).

Further, the communication terminal 7 is, for example, a notebook PC (Personal Computer), and can perform data communication with the image management system 5 via the communication network 9. The image management system 5 may be configured by a single server computer or may be configured by a plurality of server computers.

Further, the photographing device 1 and the communication terminal 3 are used by the photographer X. The communication terminal 7 is used by the viewer Y. The image management system 5 is installed in a service company or the like that provides a service of providing captured image data sent from the communication terminal 3 of each base to the communication terminal 7.

<Hardware configuration of the embodiment>
Next, the hardware configurations of the photographing device 1, the communication terminals 3 and 7, and the image management system 5 of the present embodiment will be described in detail with reference to FIGS. 9 to 11.

First, the hardware configuration of the photographing device 1 will be described with reference to FIG. Note that FIG. 9 is a hardware configuration diagram of the photographing device. In the following, the photographing device 1 is an omnidirectional image pickup device using two image pickup elements, but the number of image pickup devices may be three or more. Further, the device does not necessarily have to be a device dedicated to omnidirectional shooting, and by attaching a retrofitted omnidirectional shooting unit to a normal digital camera, smartphone, or the like, it may have substantially the same function as the shooting device 1. ..

As shown in FIG. 9, the photographing device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, and antenna 117a.

Of these, the image pickup unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two image pickups provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images from fisheye lenses into image data of electrical signals and output them, or the horizontal or horizontal of these image sensors. It has a timing generation circuit that generates a vertical synchronization signal, a pixel clock, and the like, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The image sensors 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I / F bus. On the other hand, the image pickup elements 103a and 103b of the image pickup unit 101 are connected by a serial I / F bus (I2C bus or the like) separately from the image pickup control unit 105. The image processing unit 104 and the image pickup control unit 105 are connected to the CPU 111 via the bus 110. Further, a ROM 112, a SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image pickup elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data. , Create data for a Mercator image as shown in FIG. 3 (c).

The image pickup control unit 105 generally uses the image pickup control unit 105 as a master device and the image pickup elements 103a and 103b as slave devices, and sets commands and the like in the register group of the image pickup elements 103a and 103b by using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to take in the status data of the register group of the image pickup elements 103a and 103b and send it to the CPU 111.

Further, the image pickup control unit 105 instructs the image pickup devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the shooting device, it may have a preview display function on a display or a function corresponding to a moving image display. In this case, the output of the image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frames / minute).

Further, as will be described later, the image pickup control unit 105 also functions as a synchronization control means for synchronizing the output timings of the image data of the image pickup elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the photographing device is not provided with a display unit, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data during processing by the image processing unit 104 and data of the processed Mercator image.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. By operating the operation buttons, the user inputs various shooting modes and shooting conditions.

Network I / F116 is a general term for interface circuits (USB I / F, etc.) with external media such as SD cards and personal computers. Further, the network I / F116 may be a network interface regardless of whether it is wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or if necessary, the communication terminal 3 or the like via the network I / F 116 which becomes the network I / F. It may be sent to an external device.

The communication unit 117 communicates with an external device such as a communication terminal 3 by a short-range wireless technology such as WiFi (wireless fidelity) or NFC via an antenna 117a provided in the photographing device 1. The communication unit 117 can also transmit the Mercator image data to the external device of the communication terminal 3.

The electronic compass 118 calculates the direction and tilt (Roll rotation angle) of the photographing device 1 from the magnetism of the earth, and outputs the direction / tilt information. This orientation / tilt information is an example of related information (metadata) along with Exif, and is used for image processing such as image correction of a captured image. The related information also includes each data of the shooting date and time of the image and the data capacity of the image data.

Next, the hardware configuration of the communication terminal 3 will be described with reference to FIG. FIG. 10 is a hardware configuration diagram of the communication terminal 3. As shown in FIG. 10, the communication terminal 3 includes a CPU 301 that controls the operation of the entire communication terminal 3, a ROM 302 that stores basic input / output programs, and a RAM (Random Access Memory) 303 that is used as a work area of the CPU 301. , EEPROM (Electrically Erasable and Programmable ROM) 304 that reads or writes data under the control of CPU 301, CMOS sensor 305 as an imaging element that images a subject and obtains image data under the control of CPU 301, and an electronic magnetic compass that detects geomagnetism. It is equipped with various acceleration / orientation sensors 306 such as a gyro compass and an acceleration sensor, and a media drive 308 that controls reading or writing (storage) of data to a recording medium 307 such as a flash memory. Then, according to the control of the media drive 308, the recording medium 307 in which the already recorded data is read out or the data is newly written and stored is detachable.

The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data. Further, a CCD sensor may be used instead of the CMOS sensor 305.

Further, the communication terminal 3 uses a voice input unit 311 that converts voice into a voice signal, a voice output unit 312 that converts voice signals into voice, an antenna 313a, and a wireless communication signal using the antenna 313a to obtain a nearest base. Receives GPS signals including position information (latitude, longitude, and altitude) of the communication terminal 3 by the communication unit 313 that communicates with stations, GPS (Global Positioning Systems) satellites, or IMES (Indoor MEssaging System) as indoor GPS. GPS receiver 314, display 315 such as liquid crystal or organic EL that displays images of subjects and various icons, mounted on this display 315, composed of pressure-sensitive or electrostatic panels, with fingers, touch pens, etc. It includes a touch panel 316 that detects a touch position on the display 315 by touch, and a bus line 310 such as an address bus or a data bus for electrically connecting each of the above parts.

The voice input unit 311 includes a microphone for inputting voice, and the voice output unit 312 includes a speaker for outputting voice.

Next, the hardware configuration of the communication terminal 7 in the case of the image management system 5 and the notebook PC will be described with reference to FIG. Note that FIG. 11 is a hardware configuration diagram of the image management system 5 and the communication terminal 7. Since both the image management system 5 and the communication terminal 7 are computers, the configuration of the image management system 5 will be described below, and the description of the configuration of the communication terminal 7 will be omitted.

The image management system 5 includes a CPU 501 that controls the operation of the entire image management system 5, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL, a RAM 503 that is used as a work area of the CPU 501, and a program for the image management system 5. HD504 that stores various data such as, HDD (Hard Disk Drive) 505 that controls reading or writing of various data to HD504 according to the control of CPU501, reading or writing (storage) of data to recording media 506 such as flash memory. Media drive 507 to control, display 508 that displays various information such as cursors, menus, windows, characters, or images, network I / F 509 for data communication using communication network 9, characters, numerical values, various instructions, etc. A keyboard 511 equipped with multiple keys for inputting data, a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc., and a CD-ROM (Compact Disc) as an example of a removable recording medium. Read Only Memory) A CD-ROM drive 514 that controls reading or writing of various data to 513, and an address bus, data bus, etc. for electrically connecting each of the above components as shown in FIG. It is equipped with a bus line 510.

<< Functional configuration of the embodiment >>
Next, the functional configuration of the present embodiment will be described with reference to FIGS. 9 to 14. FIG. 12 is a functional block diagram of the photographing device 1, the communication terminal 3, the image management system 5, and the communication terminal 7, which form a part of the image communication system of the present embodiment. In FIG. 12, the image management system 5 can perform data communication with the communication terminal 3 and the communication terminal 7 via the communication network 9.

<Functional configuration of photographing device 1>
As shown in FIG. 12, the photographing apparatus 1 includes a reception unit 12, an imaging unit 13, a sound collecting unit 14, a communication unit 18, and a storage / reading unit 19. Each of these parts is a function or means realized by operating any of the components shown in FIG. 9 by a command from the CPU 111 according to a program for photographing and storing developed on the DRAM 113 from the SRAM 113. Is.

Further, the photographing apparatus 1 has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG.

(Each function configuration of the photographing device 1)
Next, each functional configuration of the photographing apparatus 1 will be described in more detail with reference to FIGS. 9 and 12.

The reception unit 12 of the photographing device 1 is realized mainly by the processing of the operation unit 115 and the CPU 111 shown in FIG. 9, and receives the operation input from the user (photographer X in FIG. 8).

The imaging unit 13 is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. 9, and images a landscape or the like to obtain captured image data.

The sound collecting unit 14 is realized by the processing of the 108 shown in FIG. 9, the sound processing unit 109, and the CPU 111, and collects the sound around the photographing device 1.

The communication unit 18 is mainly realized by the processing of the CPU 111, and is realized by the communication unit 38 of the communication terminal 3 and the short-range wireless technology by NFC (Near Field Communication) standard, Bluetooth (registered trademark), WiFi (Wireless Fidelity) and the like. Can communicate.

The storage / reading unit 19 is mainly realized by the processing of the CPU 111 shown in FIG. 9, and stores various data (or information) in the storage unit 1000 or stores various data (or information) from the storage unit 1000. Read it out.

<Functional configuration of communication terminal 3>
As shown in FIG. 12, the communication terminal 3 has a transmission / reception unit 31, a reception unit 32, a display control unit 33, a determination unit 34, a communication unit 38, and a storage / reading unit 39. Each of these parts has a function or means realized by operating any of the components shown in FIG. 10 by an instruction from the CPU 301 according to a program for the communication terminal 3 developed on the RAM 303 from the EEPROM 304. Is.

Further, the communication terminal 3 has a storage unit 3000 constructed by the ROM 302, the RAM 303, and the EEPROM 304 shown in FIG.

(Each function configuration of communication terminal 3)
Next, each functional configuration of the communication terminal 3 will be described in more detail with reference to FIGS. 10 and 12.

The transmission / reception unit 31 of the communication terminal 3 is realized mainly by the processing of the communication unit 313 and the CPU 301 shown in FIG. 10, and transmits / receives various data (or information) to / from the image management system 5 via the communication network 9. I do.

The reception unit 32 is realized mainly by processing by the touch panel 316 and the CPU 301, and receives various selections or inputs from the user.

The display control unit 33 is realized mainly by the processing of the CPU 301, and controls the display 315 to display various images, characters, and the like.

The determination unit 34 is realized mainly by the processing of the CPU 301, and makes various determinations.

The communication unit 38 is mainly realized by the processing of the CPU 301, and can communicate with the photographing device 1 communication unit 18 by a short-range wireless technology based on the NFC standard, Bluetooth, WiFi, or the like.

The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 10, and stores various data (or information) in the storage unit 3000 or stores various data (or information) from the storage unit 3000. Read it out.

<Functional configuration of image management system>
Next, each functional configuration of the image management system 5 will be described in detail with reference to FIGS. 11 and 12. The image management system 5 includes a transmission / reception unit 51, a specific unit 52, a creation unit 53, a conversion unit 54, a derivation unit 55, an attachment unit 56, a transmission unit 57, and a storage / reading unit 59. Each of these parts has a function realized by operating any of the components shown in FIG. 11 by a command from the CPU 501 according to a program for the image management system 5 developed on the RAM 503 from the HD 504. It is a means.

Further, the image management system 5 has a RAM 503 shown in FIG. 11 and a storage unit 5000 constructed by the HD 504. A photographed image management DB 5001 and an attached image management DB 5002 are constructed in the storage unit 5000. Of these, the captured image management DB 5001 is configured by the captured image management table shown in FIG. The attached image management DB 5002 is configured by the attached image management table shown in FIG.

(Captured image management table)
FIG. 13 is a conceptual diagram showing a captured image management table. In this captured image management table, a user ID, a captured image ID, a file name of captured image data, and a captured date and time are stored and managed in association with each other. Of these, the user ID is an example of user identification information for identifying each photographer. The captured image ID is an example of captured image identification information for identifying each captured image (omnidirectional panoramic image). The shooting date and time indicates the date and time when the photographer took a picture and obtained the shot image.

(Attached image management table)
FIG. 14 is a conceptual diagram showing an attached image management table. In the following description, coordinates (X, Y, Z) in a three-dimensional virtual space will be used. In this attached image management table, the captured image ID, the attached image ID, the file name of the attached image data, the attached position, the reference position, and the size of the attached image are stored and managed in association with each other. Of these, the attached image ID is an example of attached image identification information for identifying each attached image. The attachment position indicates the attachment position of the attached image in the captured image (omnidirectional panoramic image) in the three-dimensional virtual space. The reference position indicates the initial position (position of the initial viewpoint) of the virtual camera IC in the three-dimensional virtual space.

(Each function configuration of the image management system)
Next, each functional configuration of the image management system 5 will be described in detail with reference to FIGS. 11 and 12.

The transmission / reception unit 51 of the image management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and is realized by the communication terminal 3 or the communication terminal 7 and various data (various data) via the communication network 9. Or information) is sent and received.

The identification unit 52 is mainly realized by the processing of the CPU 501 shown in FIG. 11, and specifies, for example, the center point p1 (x1, y1) of the drawn image g1 in the two-dimensional drawing area.

The creation unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. 11, and includes, for example, a drawing image g1 having the center point in the two-dimensional drawing area specified by the specific unit 52 as p1. Image a1 is created.

The conversion unit 54 is mainly realized by the processing of the CPU 501 shown in FIG. 11, and for example, the center point p1 in the two-dimensional drawing area specified by the specific unit 52 is specified in the three-dimensional virtual area. Convert to a position (eg, point P2 (X2, Y2, Z2)).

The derivation unit 55 is mainly realized by the processing of the CPU 501 shown in FIG. 11, for example, in a three-dimensional virtual space, a specific position (for example, a point P2) converted by the conversion unit 54 and a virtual camera. A straight line D1 passing through the position P01 of the IC is derived.

The attachment portion 56 is mainly realized by the processing of the CPU 501 shown in FIG. 11, for example, at a specific point on a straight line in the three-dimensional virtual space, the attached image is formed along the normal of this straight line. Attach.

The transparent portion 57 is mainly realized by the processing of the CPU 501 shown in FIG. 11, and for example, a portion of a predetermined attached image other than the drawn image is transparent.

The storage / reading unit 59 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. 11, and stores various data (or information) in the storage unit 5000 or various data (or information) from the storage unit 5000. Information) is read out.

<Functional configuration of communication terminal 7>
Next, the functional configuration of the communication terminal 7 will be described in detail with reference to FIGS. 11 and 13. The communication terminal 7 has a transmission / reception unit 71, a reception unit 72, a determination unit 74, a display control unit 73, and a storage / reading unit 79. Each of these parts has a function or means realized by operating any of the components shown in FIG. 11 by an instruction from the CPU 501 according to a program for the communication terminal 7 developed on the RAM 503 from the HD 504. Is.

Further, the communication terminal 7 has a RAM 503 shown in FIG. 11 and a storage unit 7000 constructed by the HD 504.

(Each function configuration of communication terminal 7)
Next, each functional configuration of the communication terminal 7 will be described in detail with reference to FIG.

The transmission / reception unit 71 of the communication terminal 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits / receives various data (or information) to / from the image management system 5 via the communication network 9. I do.

The reception unit 72 is realized mainly by the processing of the keyboard 511 and the mouse 512 shown in FIG. 11 and the CPU 501, and receives an operation input from the user (viewer Y in FIG. 8).

The display control unit 73 is mainly realized by the processing of the CPU 501 shown in FIG. 11, and controls the display 508 of the communication terminal 7 to display various images.

The determination unit 74 is mainly realized by the processing of the CPU 501 shown in FIG. 11 and makes various determinations.

The storage / reading unit 79 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. 11, and stores various data (or information) in the storage unit 7000 or various data (or information) from the storage unit 7000. Information) is read out.

<< Processing or operation of the embodiment >>
Subsequently, the process or operation of the present embodiment will be described with reference to FIGS. 15 to 27. FIG. 15 is a flowchart showing a process of deriving the attachment position of the attached image in the captured image. FIG. 16 is a flowchart showing a process of attaching an attached image to a captured image. 17 to 25 are conceptual diagrams in which (a) shows a two-dimensional (x, y) drawing area in the image management system 5 and the communication terminal 7, respectively, and (b) is a three-dimensional (X) in the image management system 5. , Y, Z) is a conceptual diagram showing a virtual space.

In each of FIGS. 17 to 25 (a), in order to make the drawn attached image a1 stand out, the predetermined area image such as the building shown in FIG. 26 (a) is omitted, and the drawn area of the attached image is omitted. Show only. Further, the display control unit 73 of the communication terminal 7 has a browser function. As a result, in the drawings of FIGS. 17 to 25 (a), a two-dimensional drawing area in both the image management system 5 and the communication terminal 7 is realized. That is, when the input data to the drawing area by the viewer Y on the communication terminal 7 side is sent to the image management system 5, processing is performed on the image management system 5 side, and the processed image data is sent to the communication terminal 7. By doing so, the communication terminal 7 can draw and display the drawn image g1 as shown in FIG. 26A on the display 508. Further, in each (b) of FIGS. 17 to 25, the case where the virtual camera IC is outside the captured image is shown.

First, when the viewer Y draws the drawn image g1 on the predetermined region image as shown in FIG. 26 (a) using FIGS. 15 and 17 to 22, the photographed image (omnidirectional image) The process of deriving the attachment position P4'of the attachment image a1 in the panorama image) will be described.

As shown in FIG. 17A, when the viewer Y draws a predetermined drawing image (here, an equilateral triangle) g1 in the two-dimensional drawing area of the communication terminal 7, the reception unit 72 draws. The input is received, and the transmission / reception unit 71 transmits a drawing command to the transmission / reception unit of the image management system 5 (step S101). At this point, the specific unit 52 of the image management system 5 determines the position of the captured image A1 which is a spherical panoramic image and the virtual camera in the three-dimensional virtual space as shown in FIG. 17 (b). The reference position P01 (X01, Y01, Z01) of the IC and the predetermined area T are specified. The viewpoint when the predetermined area T is in the field of view is the position P01 of the virtual camera IC. Further, since FIG. 18 and thereafter, the drawing becomes gradually complicated, the description of the predetermined area T is omitted.

Next, the creating unit 53 of the image management system 5 identifies the center point p1 (x1, y1) of the drawn image g1 as shown in FIG. 18A (step S102). However, the center point p1 is not visually displayed, and only the drawn image g1 is visually displayed. Further, the creating unit 53 creates an attached image a1 including the drawn image g1 having the center point as p1 (step S103). However, the attached image is not visually displayed, and only the drawn image g1 is visually displayed. At this point, as shown in FIG. 18B, the state in the three-dimensional virtual space does not change.

Next, the conversion unit 54 sets the center point p1 in the two-dimensional drawing area shown in FIG. 19A at a specific position in the three-dimensional virtual area in FIG. 19B (here, the point P2). (X2, Y2, Z2)) (step S104).

Next, the out-licensing unit 55 derives a straight line D1 passing through the point P2 and the position P01 of the virtual camera IC as shown in FIG. 20 (b) (step S105). At this point, as shown in FIG. 20 (a), the state in the two-dimensional drawing area does not change.

Next, as shown in FIG. 21B, the derivation unit 55 derives the intersections P4 (X4, Y4, Z4) and P5 (X5, Y5, Z5) between the straight line D1 and the captured image A1. (Step S106). Here, the intersection P4 is a point far from the virtual camera IC, and the intersection P5 is a point close to the virtual camera IC. Further, the identification unit 52 identifies the intersection P4 far from the virtual camera IC among the two intersections P4 and P5 (step S107). At this point, as shown in FIG. 21 (a), the state in the two-dimensional drawing area does not change.

Next, as shown in FIG. 22B, the derivation unit 55 is inside the three-dimensional sphere of the spherical panoramic image by a distance d (= βφ / q) from the intersection P4 along the straight line D1. The point P4'(X4', Y4', Z4'), which is the moved position, is derived (step S108). Β is a constant, φ is the radius of the three-dimensional sphere, and q is the linear distance between the position P01 of the virtual camera IC and the intersection P4. At this point, as shown in FIG. 22 (a), the state in the two-dimensional drawing area does not change. In this way, the attached image a1 is attached to the inside of the three-dimensional sphere by d. As shown in FIG. 26 (a), the drawn image g11 of the attached image a1 is a drawn image at the time of drawing. This is because it has been found that it is possible to display a clear image close to g1. That is, as shown in FIG. 26A, the drawn image g1 is once drawn, each data related to the attached image a1 is stored in step S109 described later, and then attached in the processes after step S201 described later. When each data related to the image a1 is read and the attached image a1 is attached to the captured image A1, the drawn image g11 of the attached image a1 is displayed as a clear image close to the drawn image g1 at the time of drawing. Is.

Next, the storage / reading unit 59 stores each data related to the attached image in the attached image management table (see FIG. 14) (step S109). Specifically, the storage / reading unit 59 associates the X, Y, and Z parameters of the point P4'with the field of the attachment position with respect to the captured image ID, the attached image ID, and the file name of the attached image data. , The X, Y, and Z parameters of the point P01 are associated with the reference position field, and the image size parameter of the attached image a1 is associated with the attached image size field for management. In this way, not only the attached position but also the reference position is managed in association with each other, as shown in FIG. 26A, once the drawn image g1 is drawn, and each of the attached images a1 is managed by step S109. After the data is stored, each data related to the attached image a1 is read out in the processing after step S201 described later, and when the attached image a1 is attached to the captured image A1, the shape is the same as the drawn image g1 at the time of drawing. This is because the drawn image g11 of is displayed. On the contrary, when the attached position is memorized and the reference position is not memorized, even if the point P4'in FIG. 22B becomes clear, the normal line of the straight line D1 is unclear. ), The drawn image g11 after reading may not be displayed as the same shape as the drawn image g1 at the time of drawing.

After the process of step S109, the viewer Y ends the display of the predetermined area image (photographed image) in the state in which the drawn image g1 is drawn, which is shown in FIG. 26 (a).

Subsequently, using FIGS. 16 and 23, when the viewer Y uses the communication terminal 7 to display the predetermined area image (photographed image) again as shown in FIG. 26A, the image is displayed. The process in which the management system 5 displays the attached image a1 in a state of being attached to the captured image A1 will be described.

First, when the viewer Y inputs for redisplaying the captured image on the communication terminal 7, the reception unit 72 accepts the redisplay of the captured image and redisplays it from the transmission / reception unit 71 to the transmission / reception unit of the image management system 5. Is transmitted (step S201). In this case, the captured image ID of the captured image to be redisplayed is also transmitted.

Next, the storage / reading unit 59 of the image management system 5 uses the captured image ID as a search key to search the attached image management table (see FIG. 14) to search each corresponding data (attached image ID, attached image data). The file name, attachment position, reference position, and attachment are the size of the image) are read (step S202).

Next, based on the data read in step S202, the attachment portion 56 is located along the normal line of the straight line D1 at the point P4'on the straight line D1 as shown in FIG. 23 (b). The attached image a1 is attached (step S203). As a result, as shown in FIG. 23 (a), the attached image a1 in which the drawn image g11 has the same shape as the drawn image g1 at the time of drawing shown in FIG. 22 (a) becomes the captured image A1. It will be attached.

Next, the transparent portion 57 transmits a portion of the attached image a1 other than the drawn image g11 (step S204). Then, the transmission / reception unit 51 transmits the data of the captured image A1 in the state where the attached image a1 is attached by the processing of steps S201 to S204 to the transmission / reception unit 71 of the communication terminal 7, so that the display control unit 73 of the communication terminal 7 is transmitted. Displays a predetermined area image including the drawn image g11 as shown in FIG. 26A on the display 508 of the communication terminal 7 (step S205).

As described above, when the captured image is redisplayed, the captured image including the drawn image g11 having the same shape as the drawn image g1 at the time of drawing can be redisplayed.

Note that FIG. 26 (a) is an example of a predetermined area image in the case where the distortion is corrected by step S203 and the transmission process is performed in step S204, but FIG. 26 (b) is the case where the transparency process is performed without distortion correction. This is an example of a predetermined area image of. Further, FIG. 27A is an example of an omnidirectional panoramic image in the case where there is transmission processing without distortion correction. Further, FIG. 27B is an example of an omnidirectional panoramic image in the case where there is no distortion correction and no transmission processing.

By the way, if the viewer Y moves the predetermined area image in the direction of the arrow (left side) on the communication terminal 7 side as shown in FIG. 24 (a), it is shown in FIG. 24 (b). As shown above, the position of the virtual camera IC moves from P01 to the position of P02 (X02, Y02, Z02) in the three-dimensional virtual space. As a result, the attached image a1 becomes an image viewed from the viewpoint P02, so that the drawn image g1 becomes a drawn image g1'in which the shape is deformed as shown in FIG. 24 (a).

Further, as shown in FIG. 25 (a), when a newly drawn image g2 is drawn, as shown in FIG. 25 (b), at the position of the point P14'on the new straight line D2. , Attached image a2 is attached along the normal line of the straight line D2. In addition, in FIG. 25 (b), the straight line D2, the point P02, the point P12, the point P14 and the point P14'are the straight line D1, the point P01, the point P2, the point P4 and the point P4', respectively. It corresponds to.

<< Main effects of this embodiment >>
As described above, according to the present embodiment, since the attached image a1 is attached to the captured image A1 as a flat image without being curved, when the viewer Y browses the captured image (predetermined area image) again. , As shown in FIG. 26A, the drawn image g11 is displayed in the same shape as the drawn image g1 at the time of drawing. That is, it is not displayed in a curved state as in the drawn image g9 shown in FIG. 26 (b). As a result, it is possible to eliminate the confusion that the viewer Y (user) browses the drawn image g11.

[Supplement to Embodiment]
The image management system 5 in the above embodiment may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each unit (function, means, or storage unit). Good.

In addition, a recording medium such as a CD-ROM in which each program of the above-described embodiment is stored, and an HD504 in which these programs are stored may be provided as a program product in Japan or overseas. it can.

1 Imaging device 3 Communication terminal 5 Image management system 7 Communication terminal 9 Communication network 51 Transmission / reception unit (example of transmission means, example of reception means)
52 Specific part (example of specific means)
53 Creation Department (Example of Creation Method)
54 Conversion unit (example of conversion means)
55 Derivation unit (example of derivation means)
56 Attachment (an example of attachment means)
57 Transmissive part (an example of transmissive means)

Japanese Unexamined Patent Publication No. 2014-131215

Claims (7)

An image management system that manages captured images
A creation means for creating an attached image including the drawn image and attaching it to the captured image based on the drawn image drawn on the captured image which is a spherical panoramic image.
And attach attachment means as a planar image on the shooting image attached image created in the above,
An image management system characterized by having. The image management system according to claim 1, further comprising a transparent means for transmitting a portion of the attached image other than the drawn image. Said attachment means, according to claim 1, characterized in that to attach the attached image along a normal line of a straight line portion of the predetermined region of the captured image passes through the center point of viewpoint and the drawing image into the field of view Image management system described in . It has a specific means for specifying the normal to a position moved by a predetermined distance in the spherical panoramic image along the straight line from a specific intersection far from the viewpoint among the intersections of the straight line and the captured image. The image management system according to claim 3, wherein the image management system is characterized by the above. The image management system according to any one of claims 1 to 4.
A communication terminal that displays the captured image to which the attached image is attached by the attachment means, and
An image communication system characterized by having. It is an image management method executed by an image management system that manages captured images.
The image management system
Based on the drawn image drawn on the captured image which is a spherical panoramic image, a creation step of creating an attached image including the drawn image and attaching to the captured image, and
And attaching attachment step as a planar image on the shooting image attached image created in the above,
An image management method characterized by executing. A program for causing a computer to perform the method according to claim 6.

Images