JP6714819B2 - Image display system, information processing device, image display method, and image display program - Google Patents

Description

The present invention relates to an image display system, an information processing device, an image display method, and an image display program.

There is known a technique of creating a 360-degree panoramic image using a spherical camera.

Further, in Patent Document 1, a still image and a moving image to be attached to the still image are read from a storage device, and alignment of the moving image with respect to the still image is adjusted based on a control point for associating the moving image with a region of the still image. However, there is disclosed an image display method of starting reproduction of a moving image instead of the still image frame of the moving image when an instruction to zoom in on a region including the moving image in the still image is received.

However, the conventional omnidirectional camera can take a 360-degree omnidirectional image with the same resolution, but there is a problem that the image becomes rough when a certain part is zoomed.

The present invention has been made in view of the above, and provides an image display system, an information processing device, an image display method, and an image display program capable of easily associating a high-definition image with a part of the omnidirectional image. The purpose is to provide.

In order to solve the above-mentioned problems and achieve the object, the present invention provides a spherical image with azimuth information and information indicating up and down directions, and the spherical image at the shooting position of the spherical image. A storage unit that stores a high-definition image that has been captured with a higher image quality than an image and that has range information indicating a shooting range, based on the azimuth information , the information indicating the vertical direction, and the range information, An associating unit that associates a high-definition image with a corresponding region in the omnidirectional image, and displays at least one of the omnidirectional image and the high-definition image at a magnification according to an instruction from the outside. When the magnification of the display unit and the corresponding region displayed by the display unit is changed to a predetermined value or more, the corresponding region displayed by the display unit is switched to the high-definition image and the display unit And a switching unit for displaying.

According to the present invention, it is possible to easily associate a high-definition image with a part of the omnidirectional image.

FIG. 1 is a diagram showing an outline of an image display system according to an embodiment. FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. FIG. 3 is a diagram illustrating a spherical image. FIG. 4 is a diagram illustrating a hardware configuration of the mobile terminal. FIG. 5 is a diagram illustrating a high definition image. FIG. 6 is a diagram illustrating a hardware configuration of the server. FIG. 7 is a diagram showing an outline of the function of the image display system that operates when the content stored in the server is displayed on the display device. FIG. 8 is a diagram conceptually showing the operation of the association unit creating association data. FIG. 9 is a diagram showing a state in which a spherical image and a high-definition image are associated with each other. FIG. 10 is a diagram showing the radius of the sphere, the angle of view, and the length of the chord of the spherical image. FIG. 11 is a diagram showing the relationship between the display unit of the display device, the spherical image and the high-definition image. FIG. 12 is a flowchart showing an example of processing performed when the association unit generates association data. FIG. 13 is a flowchart showing an example of processing performed by the switching unit.

First, the outline of the embodiment of the image display system will be described. FIG. 1 is a diagram showing an outline of an image display system 1 according to an embodiment of the present invention. In the image display system 1, the imaging device 2 and the mobile terminal 3 are connected to the server 4 via the network 10. The server 4 is also connected to the display device 14 via the network 10 and the access point 12.

The imaging device 2 is a omnidirectional camera and is capable of taking a omnidirectional image (360-degree panoramic image). The mobile terminal 3 is a terminal equipped with a camera (imaging element), and is capable of capturing an image with a resolution higher than that of the imaging device 2.

In the image display system 1, the imaging device 2 is set to capture an image of the surroundings, for example, periodically or at regular intervals, and associates the captured image (a celestial sphere image described below) with the imaging position and the shooting date and time. And sends it to the server 4. Further, the photographer takes a photograph in a direction of particular interest by using the mobile terminal 3 equipped with a position sensor (GPS sensor) and a direction sensor at the timing of taking an image with the imaging device 2. At this time, in the image (photograph) taken by the mobile terminal 3, information on the shooting position, the shooting direction, and the shooting date and time is simultaneously saved by the GPS sensor, the direction sensor, and the like of the mobile terminal 3.

For example, the photographer carries the imaging device 2 and the mobile terminal 3, walks through points A, B, and C, and performs imaging with the imaging device 2 at points A, B, and C. Photographing is performed by the mobile terminal 3 at points C and C.

When the photographer walks to the destination and finishes photographing on the route, the photographed spherical image and the image photographed by the mobile terminal 3 are simultaneously transmitted to the server 4. Then, the image captured by the mobile terminal 3 at the same position as the celestial sphere image is associated with the celestial sphere image using information such as the photographing date and time.

For example, the server 4 associates (combines, for example) the omnidirectional image with the image captured by the mobile terminal 3 by using the shooting date and time, the orientation information embedded in the image, the GPS information, and the like. This association may be performed by the mobile terminal 3 or the like.

The display device 14 is, for example, a smartphone or a PC (Personal Computer), and receives, for example, a spherical image associated with the server 4 and an image captured by the mobile terminal 3 from the server 4 and displays the image. For example, the display device 14 displays an image using a browser, an application, or the like. When the display device 14 is, for example, a smartphone, the display device 14 may be referred to as a mobile terminal because it is easy to carry and move.

Next, the details of each unit constituting the image display system 1 will be described. FIG. 2 is a diagram illustrating a hardware configuration of the imaging device 2. As shown in FIG. 2, the imaging device 2 includes, for example, the photographing unit 20, the photographing control unit 204, the image processing unit 206, the CPU 208, the ROM 210, the SRAM 212, the DRAM 214, the operation unit 216, the network interface (I/F) 218, and the communication unit. It has 220 and an electronic compass 224.

The imaging unit 20 includes, for example, two fisheye lenses 200 having an angle of view of about 180 degrees and two image pickup elements 202 such as CMOS sensors. The shooting control unit 204 controls the two image pickup devices 202 to shoot images at the same time. The image processing unit 206 performs a process of converting the images captured by the two image pickup elements 202 into a spherical image as illustrated in FIG.

The CPU 208 controls each unit that constitutes the imaging device 2. The ROM 210 stores a program for operating the imaging device 2 and the like. The SRAM 212 is used as a processing area when the program is executed. The DRAM 214 is used for storing, processing, developing, and the like of image data having a large data size (all-sky image). The operation unit 216 receives a user operation on the imaging device 2. The network I/F 218 is an interface that can be connected to the network 10 or the like. The communication unit 220 is capable of communicating with the server 4 via the antenna 222 or the like.

The electronic compass 224 detects the direction and the direction in which the photographing unit 20 photographs. For example, the electronic compass 224 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 2 from the magnetism of the earth, and outputs azimuth information/tilt information. This azimuth information/tilt information is attached to the image captured by the image sensor 202 according to the Exif (metadata) format. The azimuth/tilt information is used for image processing such as image correction. The Exif also includes the date and time when the image was captured, the thumbnail of the image data, the data capacity of the image data, and the like. The image pickup apparatus 2 is also configured to detect the position (position information: shooting position) by GPS by the electronic compass 224. Table 1 exemplifies information embedded in the omnidirectional image captured by the imaging device 2.

FIG. 4 is a diagram illustrating a hardware configuration of the mobile terminal 3. The mobile terminal 3 is, for example, a smart phone or the like enabled to communicate via a mobile phone network, and has a voice input unit 300, a voice output unit 302, a communication unit 304, a GPS receiving unit 308, a display 310, a touch panel 312, an image sensor. 314, CPU 316, ROM 318, RAM 320, EEPROM 322, CMOS 324, sensor 326, and media drive 328, and each unit is connected via a bus 330.

The voice input unit 300 is, for example, a microphone and receives the voice input of the user. The voice output unit 302 is, for example, a speaker and outputs voice. The communication unit 304 performs network communication using, for example, a mobile phone network via the antenna 306. The GPS receiving unit 308 is a position sensor (GPS sensor) that detects the position (for example, the shooting position) of the mobile terminal 3 using GPS. The display 310 displays the operation of the mobile terminal 3. The touch panel 312 receives a user's operation input to the mobile terminal 3. The image sensor 314 is, for example, a CMOS sensor or the like, and captures a high-definition image (high-definition image) having a higher resolution than the image pickup device 2 illustrated in FIG.

The CPU 316 controls each unit that makes up the mobile terminal 3. The ROM 318 stores a program for operating the mobile terminal 3 and the like. The RAM 320 is used for a processing area during program execution, storage of high-definition images, and the like. The EEPROM 322 holds data so that it can be rewritten. The CMOS 324 performs digital processing including image processing. The sensor 326 is an acceleration/direction sensor having a function as an electronic magnetic compass that detects geomagnetism and a function such as a gyro sensor/acceleration sensor. For example, the sensor 326 has a function as a direction sensor that detects a shooting direction of the mobile terminal 3. .. The media drive 328 reads/writes data from/to the medium 332 such as a memory card. The medium 332 stores, for example, a high definition image.

It should be noted that information such as GPS information and shooting time is added to the image (high-definition image) taken by the mobile terminal 3. Table 2 exemplifies information embedded in the high-definition image captured by the mobile terminal 3.

Note that the imaging positions of the imaging device 2 and the mobile terminal 3 are considered to be the same position if there is the same information up to the integer part and up to 4 digits after the decimal point, if there is GPS latitude and longitude information as shown in the following formula 1. ..

Further, when associating a high-definition image with a part of the omnidirectional image, information on the shooting time is also used. In this case, the high-definition images taken within a range of, for example, about ±5 minutes of the omnidirectional image are considered to be photographs taken at the same place. The sensor size of the camera (sensor size of the mobile terminal 3) is the size of the image sensor 314. Further, the direction/direction in which the mobile terminal 3 is capturing an image is detected by the sensor 326.

In addition, the azimuth information of the high-definition image captured and the inclination from the horizontal direction, and the information including the focal length and the sensor size of the mobile terminal 3 capturing the high-definition image are used to calculate the imaging range captured by the mobile terminal 3. Since it is possible, it may be referred to as range information.

FIG. 6 is a diagram illustrating a hardware configuration of the server 4. The server 4 includes a CPU 400, a ROM 402, a RAM 404, a HDD control unit 406, a HDD 408, a display 410, a network interface (I/F) 412, a keyboard 414, a mouse 416, a media drive 418, and an optical drive 420 (an information processing apparatus). Is.

The CPU 400 controls each unit that constitutes the server 4. The ROM 402 stores the boot program of the server 4 and the like. The RAM 404 is used as a processing area. The HDD control unit 406 controls the HDD 408 that stores the program 430 and the like. The display 410, keyboard 414 and mouse 416 are used as a user interface. The network interface (I/F) 412 is a communication unit connected to the network 10. The media drive 418 reads/writes data from/to the memory card 424 or the like. The optical drive 420 reads data from and writes data to a medium 426 such as a disk.

The server 4 also has a function as a content creation server having images as contents. For example, the server 4 includes an image correction unit, a route creation unit, a map acquisition unit, an image reception unit with data, a URI transmission unit, a content transmission unit, an image embedding unit, and the like, and images captured by the imaging device 2 and the mobile terminal 3 are included. The content is displayed as an image on the display device 14.

FIG. 7 is a diagram showing an outline of the functions of the image display system 1 that operates when the content stored in the server 4 is displayed on the display device 14. The server 4 has a storage unit 50, an associating unit 52, a display image creating unit 54, and a switching unit 56 as a function for displaying an image on the display device 14. The display device 14 also includes a display unit 140 corresponding to a display, and a region information acquisition unit 142 that acquires region information indicating a predetermined region in the image displayed by the display unit 140.

The storage unit 50 has a first storage area 500 that stores image data of each of a celestial sphere image and a high-definition image, and a second storage area 502 that stores association data described below. The associating unit 52 receives the omnidirectional image captured by the imaging device 2 and the high-definition image captured by the mobile terminal 3, and generates association data that associates the omnidirectional image with the high-definition image. Specifically, the associating unit 52 associates the high-definition image with the corresponding region in the omnidirectional image using the azimuth information and the range information. Then, the associating unit 52 stores the image data of each of the spherical image and the high-definition image in the first storage area 500, and stores the association data in the second storage area 502.

The display image creation unit 54 uses the omnidirectional image and the high-definition image and the association data generated by the association unit 52 for the image displayed by the display unit 140 in accordance with the instruction from the display device 14. Create (composite may be used). The display unit 140 displays at least one of the spherical image and the high-definition image at a magnification according to an instruction from the outside. The area information acquisition unit 142 acquires area information indicating a predetermined area in the image displayed by the display unit 140, and outputs the area information to the switching unit 56 of the server 4.

The switching unit 56 switches the image using the omnidirectional image and the high-definition image created by the display image creating unit 54 according to the area information received from the area information acquisition unit 142. For example, the switching unit 56 changes the corresponding area displayed on the display unit 140 when the magnification of the corresponding area in the spherical image displayed on the display unit 140 is changed to a predetermined value (threshold value) or more. The high-definition image is switched to and displayed on the display unit 140 via the display image creation unit 54.

Next, the association data generated by the association unit 52 will be described. 8 to 11 are diagrams conceptually showing the operation of the association unit 52 to create association data. As shown in Table 2, the high-definition image is provided with information indicating the direction (azimuth information and tilt) taken by the mobile terminal 3. The celestial sphere image is provided with the information directly below (gravitational direction) and the azimuth information, so that the vertical direction and azimuth can be uniquely determined.

The associating unit 52 uses the azimuth information and the inclination of the high-definition image, and the information indicating the vertical direction and the azimuth of the omnidirectional image to determine that the endpoints of the high-definition image (see FIGS. 8 and 9) are omnidirectional. Calculate which coordinate in the sphere image corresponds to.

First, the associating unit 52 calculates the angle of view of the high-definition image by the following equations 2 and 3.

Note that θ is the angle of view of the mobile terminal 3, f is the focal length of the lens of the mobile terminal 3, h is the size of the image sensor 314 in the vertical direction (height), and w is the size of the image sensor 314 in the horizontal direction (width). , D indicate the size of the diagonal line of the image sensor 314.

Further, as shown in FIG. 10, the chord length c when the radius of the sphere of the spherical image is r and the angle of view is θ is calculated by the following expression 4.

When the horizontal angle of view and the vertical angle of view of the mobile terminal 3 that has captured the high-definition image are θh and θv, respectively, the image is perpendicular to the x-axis and the x-axis passes through the center of the image. The coordinates of the four corners of the image in the horizontal state (horizontal with respect to the plane formed by the xy axes) are shown by the following equations 5 to 8. The upper left of the image is LT, the upper right is RT, the lower right is RB, and the lower left is LB.

Further, when transforming the coordinates in consideration of the orientation and rotation of the image, the following equation 9 is used. Here, the rotation angles of the x axis, the y axis, and the z axis are α, β, and γ, respectively.

Then, the display image creation unit 54 can replace the area within the frame in the spherical image shown in FIG. 9 with a high-definition image. For example, when the area within the frame in the omnidirectional image is enlarged to a size equal to or larger than a predetermined threshold value, the display image creation unit 54 replaces the area within the frame with a high-definition image.

At this time, the display image creation unit 54 replaces the image using the association data stored in the second storage area 502 shown in Table 3.

Each region of the omnidirectional image associated with the high-definition image has threshold information so that the image can be switched to the high-definition image when the magnification of the image displayed by the display unit 140 is equal to or higher than a predetermined threshold. Is set. This threshold information may be changed when the area of the omnidirectional image becomes a target to be switched to the high definition image.

The display device 14 has a range (display area) in which the display unit 140 can display. As shown in FIG. 11, for example, when the display unit 140 displays a part of the frame in the spherical image, the user performs an operation of enlarging the displayed image, and the area information acquisition unit 142. When the magnification indicated by the area information acquired by the above becomes equal to or larger than the threshold value, the switching unit 56 switches the area where the omnidirectional image is displayed to the corresponding high-definition image.

When the image displayed by the display unit 140 is enlarged or reduced, the displayed area in the spherical image changes. For example, the area information acquisition unit 142 uses the size of the area in the omnidirectional image displayed by the display unit 140 and the size (VSize_x, VSize_y) of the display unit 140 to display the current display unit according to Expression 10 below. The magnification of the image displayed on 140 is calculated. The switching unit 56 may calculate the magnification of the image.

When this magnification becomes, for example, 100% or more (calculation result is 1.0 or more), the switching unit 56 switches the area of the omnidirectional image to a high-definition image. The coordinates of the four corners of the high-definition image are associated with the coordinates on the omnidirectional image, and the display image creation unit 54 calculates the corresponding region of the high-definition image from the region where the omnidirectional image is displayed. To do. Then, the display image creation unit 54 reduces the calculated area of the high-definition image so as to fit in the display area of the display unit 140 and causes the display unit 140 to display.

Further, when the display unit 140 is displaying a portion of the omnidirectional image, the switching unit 56 selects the image displayed by the display unit 140 if a high-definition image is associated with all of the region. It may be configured to switch to a high definition image. That is, the switching unit 56 switches the corresponding region of the omnidirectional image displayed by the display unit 140 to the high-definition image when the entire image displayed by the display unit 140 can be displayed as the high-definition image. Good.

The associating unit 52, the display image creating unit 54, the switching unit 56, and the area information acquiring unit 142 may be configured by hardware or software.

FIG. 12 is a flowchart showing an example of processing performed by the associating unit 52 when generating association data. The associating unit 52 first acquires omnidirectional image data from the imaging device 2 (S100), and acquires sensor/image information of the camera (imaging device 2) that captured the omnidirectional image (S102). Next, the associating unit 52 acquires high-definition image data from the mobile terminal 3 (S104), and acquires sensor/image information of the camera (mobile terminal 3) that captured the high-definition image (S106).

Then, the associating unit 52 calculates the corresponding area of the spherical image from the image information and the sensor information (S108). That is, the associating unit 52 calculates the angle of view of the high-definition image by using the above expressions 2 and 3, calculates the length of the chord corresponding to the angle of view by using the above expression 4, and calculates the coordinates of the four corners of the image. Is calculated by the above equations 5 to 8 to calculate the corresponding region of the spherical image. When transforming the coordinates in consideration of the orientation and rotation of the image, the above equation 9 is used.

Next, the associating unit 52 allocates the high-definition image to the calculated area and generates the association data (S110). The associating unit 52 determines whether there is another high-definition image (S112). If there is a high-definition image (S112: Yes), the process returns to S104, and if there is no high-definition image (S112). S112: No), the process ends.

In this way, in the image display system 1, since the associating unit 52 generates the associative data, the omnidirectional image and the high definition image can be obtained even if the user does not specify the position of the high definition image with respect to the omnidirectional image. It becomes possible to associate the positions of the images.

FIG. 13 is a flowchart showing a processing example performed by the switching unit 56. Here, the coordinates of the area currently displayed by the display unit 140 in the spherical image are the upper left (Vx1, Vy1), upper right (Vx2, Vy2), lower right (Vx3, Vy3), lower left (Vx4). , Vy4). Further, the upper left, upper right, lower right, and lower left coordinates of the area associated with the high-definition image are (Ax1, Ay1) to (Ax4, Ay4), respectively.

First, the switching unit 56 acquires the values of the information (Vx1, Vy1) to (Vx4, Vy4) of the area currently displayed on the display unit 140 from the area information acquisition unit 142 (S200).

The switching unit 56 determines whether or not there is an area to which the high-definition image is allocated in the acquired area (S202). If there is an area (S202: Yes), the process proceeds to S204, and if there is no area. If (S202: No), the process proceeds to S214.

The switching unit 56 acquires the coordinates (Ax1, Ay1) to (Ax4, Ay4) of the area to which the high-definition image is assigned from the second storage area 502 (S204).

Then, the switching unit 56 determines whether the area of (Vx1, Vy1) to (Vx4, Vy4) is in the area of (Ax1, Ay1) to (Ax4, Ay4) (S206), and If it is inside (S206: Yes), the process proceeds to S208. If it is not inside the region (S206: No), the process returns to S202.

The switching unit 56 calculates the current display magnification of the display unit 140 (S208) and determines whether the display magnification is 100% or more (S210). The switching unit 56 proceeds to the processing of S212 when the display magnification is 100% or more (S210: Yes), and proceeds to the processing of S214 when the display magnification is not 100% or more (S210: No).

In the process of S212, the switching unit 56 displays the high-definition image via the display image creation unit 54 and the display unit 140. Further, in the process of S214, the switching unit 56 displays the spherical image through the display image creation unit 54 and the display unit 140.

As described above, according to the image display system of the embodiment, the high-definition image is associated with the corresponding area in the omnidirectional image based on the azimuth information and the range information. When changed to the above, the corresponding area displayed on the display unit can be switched to a high-definition image and displayed on the display unit. Further, the image display system, when the entire image displayed by the display unit can be displayed by the high-definition image, the switching unit switches the corresponding area displayed by the display unit to the high-definition image, and the omnidirectional image is displayed. A high-definition image can be easily associated with a part of the image.

DESCRIPTION OF SYMBOLS 1 image display system 2 imaging device 3 mobile terminal 4 server 10 network 14 display device 20 imaging unit 50 storage unit 52 association unit 54 display image creation unit 56 switching unit 140 display unit 142 area information acquisition unit 220 communication unit 224 electronic compass 304 Communication unit 308 GPS receiving unit 314 Image pickup device 326 Sensor 412 Network I/F
500 First storage area 502 Second storage area

JP, 2014-082764, A

Claims (8)

The omnidirectional image to which the azimuth information and the information indicating the up and down direction are attached, and the range information indicating the imaging range in which the image quality is finer than that of the omnidirectional image at the photographing position of the omnidirectional image. A storage unit that stores the high-definition image that has been created,
An associating unit that associates the high-definition image with a corresponding region in the omnidirectional image based on the azimuth information , information indicating the vertical direction, and the range information,
At least one of the spherical image and the high-definition image, a display unit that displays at a magnification according to an instruction from the outside,
When the magnification of the corresponding area displayed on the display unit is changed to a predetermined value or more, the corresponding area displayed on the display unit is switched to the high-definition image and displayed on the display unit. An image display system comprising: Before Symbol mapping unit, on the basis of the omnidirectional image and the information indicating the photographing time of the high-resolution image has the the omnidirectional image and the high resolution image is a captured image at the same location To judge,
The image display system according to claim 1. The range information is
Inclination from captured orientation and horizontal direction of the high-definition image, as well as, according to claim 1 or 2, characterized in that the information including the focal distance and the sensor size of the cameras taking the high-definition image Image display system. The switching unit,
The whole image in which the display unit is displayed when enabled display by the high-definition image, according to claim 1 to 3, characterized in that switching the corresponding area in which the display unit is displaying the high-definition image The image display system according to any one of 1 . The display unit is
Provided in the mobile terminal,
The storage unit, the associating unit, and the switching unit,
The image display system according to any one of claims 1 to 4 , wherein the image display system is provided in a server that transmits the omnidirectional image and the high-definition image to the mobile terminal. The omnidirectional image to which the azimuth information and the information indicating the up and down direction are attached, and the range information indicating the imaging range in which the image quality is finer than that of the omnidirectional image at the photographing position of the omnidirectional image. A storage unit that stores the high-definition image that has been created,
An associating unit that associates the high-definition image with a corresponding region in the omnidirectional image based on the azimuth information , information indicating the vertical direction, and the range information,
A communication unit that transmits at least one of the spherical image and the high-definition image to a display device that displays at a magnification according to an instruction from the outside,
When the magnification of the corresponding area displayed by the display device is changed to a predetermined value or more, the corresponding area displayed by the display device is switched to the high-definition image and displayed on the display device. An information processing device comprising: The omnidirectional image to which the azimuth information and the information indicating the up and down direction are attached, and the range information indicating the imaging range in which the image quality is finer than that of the omnidirectional image at the photographing position of the omnidirectional image. A step of storing the high-definition image thus obtained,
A step of associating the high-definition image with a corresponding area in the omnidirectional image based on the azimuth information , the information indicating the vertical direction, and the range information;
At least one of the omnidirectional image and the high-definition image, when the magnification of the corresponding region displayed by the display unit for displaying at a magnification according to an instruction from the outside is changed to a predetermined value or more, Switching the corresponding area displayed by the display unit to the high-definition image and displaying the image on the display unit,
Image display method including. The omnidirectional image to which the azimuth information and the information indicating the up and down direction are attached, and the range information indicating the imaging range in which the image quality is finer than that of the omnidirectional image at the photographing position of the omnidirectional image. Storing the created high-definition image,
A step of associating the high-definition image with a corresponding area in the omnidirectional image, based on the azimuth information , information indicating the vertical direction, and the range information;
A step of transmitting at least one of the spherical image and the high-definition image to a display device that displays at a magnification according to an instruction from the outside;
Switching the corresponding area displayed by the display device to the high-definition image and displaying the corresponding area on the display device when the magnification of the corresponding area displayed by the display device is changed to a predetermined value or more. When,
An image display program that causes a computer to execute.

Images