US20190391778A1 - Apparatus, system, and method for controlling display, and recording medium - Google Patents
Apparatus, system, and method for controlling display, and recording medium Download PDFInfo
- Publication number
- US20190391778A1 US20190391778A1 US16/562,437 US201916562437A US2019391778A1 US 20190391778 A1 US20190391778 A1 US 20190391778A1 US 201916562437 A US201916562437 A US 201916562437A US 2019391778 A1 US2019391778 A1 US 2019391778A1
- Authority
- US
- United States
- Prior art keywords
- image
- display
- captured image
- planar
- spherical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000010365 information processing Effects 0.000 claims description 22
- 238000003672 processing method Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 description 61
- 238000010586 diagram Methods 0.000 description 58
- 238000006243 chemical reaction Methods 0.000 description 33
- 230000006870 function Effects 0.000 description 32
- 238000003384 imaging method Methods 0.000 description 32
- 230000008569 process Effects 0.000 description 23
- 238000004891 communication Methods 0.000 description 6
- 239000000470 constituent Substances 0.000 description 6
- 230000014509 gene expression Effects 0.000 description 6
- 230000002441 reversible effect Effects 0.000 description 6
- 230000001133 acceleration Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 230000009467 reduction Effects 0.000 description 5
- 230000007423 decrease Effects 0.000 description 4
- 238000009877 rendering Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000003702 image correction Methods 0.000 description 3
- 230000015654 memory Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000005389 magnetism Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G2340/125—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels wherein one of the images is motion video
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2354/00—Aspects of interface with display user
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/20—Details of the management of multiple sources of image data
Definitions
- the present invention relates to an apparatus, system, and method of controlling display, and a recording medium.
- Image capturing devices capable of capturing wider-angle images than those captured by general-purpose (generic) digital cameras are widely used.
- a spherical camera provided with a plurality of, for example, fish-eye lenses generates a spherical image of a 360-degree surrounding view captured with a single image capturing operation.
- a user can determine the line-of-sight direction as desired for the spherical image and view the spherical image in various directions.
- image processing of a spherical image for a wide image capturing range is likely to be restricted.
- the image quality of a spherical image tends to be lower (for example, the resolution is lower, whiteout is more likely to occur, or color reproducibility is lower) than that of an image captured by the generic digital camera. Accordingly, in a case where a user enlarges a portion of a spherical image and views the spherical image, a decrease in the image quality may be unallowable to the user or the user may have difficulty in obtaining detailed information, which may be inconvenient to the user.
- Embodiments of the present invention include an apparatus, system, and method for controlling a display, each of which: controls the display to display an image of a predetermined area of a first image, the first image being superimposed with a second image; determines whether the second image is viewed by a user; and switches a display of the second image between a first display in which the second image is displayed as a still image and a second display in which the second image is displayed as a moving image, based on a determination result indicating whether the second image is viewed by the user.
- FIGS. 1A and 1B are example diagrams illustrating a spherical image
- FIGS. 2A to 2C are example diagrams illustrating a display method for displaying a moving image superimposed on a spherical image
- FIGS. 3A to 3C are diagrams illustrating example system configurations of an image processing system including a spherical camera and a display control apparatus;
- FIG. 4 is a schematic diagram illustrating a hardware configuration of the spherical camera, according to an embodiment
- FIG. 5 is a schematic diagram illustrating a hardware configuration of the display control apparatus, according to an embodiment
- FIG. 6 is a conceptual diagram illustrating example use of the spherical camera
- FIGS. 7A to 7C are diagrams schematically illustrating example operation of generating a spherical image from images captured by the spherical camera
- FIGS. 8A and 8B are diagrams schematically illustrating example operation of generating a spherical image from images captured by the spherical camera
- FIGS. 9A and 9B are example diagrams illustrating placement of a spherical image on a sphere in equirectangular projection
- FIG. 10 is an example diagram illustrating the line of sight of a user
- FIGS. 11A to 11D are example diagrams schematically illustrating enlargement and reduction of a view area in a spherical image
- FIG. 12 is a schematic block diagram illustrating a functional configuration of a digital camera, the spherical camera, and the display control apparatus, according to an embodiment
- FIGS. 13A to 13C are diagrams illustrating projection conversion of a planar image in a match area and an example of a location parameter for associating the planar image with the match area;
- FIGS. 14A and 14B are example diagrams illustrating a projection-converted image and mask data generated by a projection converter
- FIG. 15 is an example diagram schematically illustrating superimposition of a planar image on a spherical image
- FIG. 16 is a diagram schematically illustrating a planar image superimposed on a spherical image
- FIG. 17 is a flowchart illustrating operation of superimposing a planar image on a spherical image, performed by the display control apparatus, according to an embodiment
- FIG. 18 is a schematic block diagram illustrating a functional configuration of an to image superimposing unit, according to an embodiment
- FIGS. 19A to 19D are example diagrams illustrating the relative position of a planar image relative to a view area
- FIG. 20 is an example diagram illustrating a. way of calculating an angle of view a and an angle of view b; is FIG. 21 is an example diagram illustrating relationships between the vertexes of a view area and the diagonally opposite vertexes of a planar image;
- FIGS. 22A to 22E are diagrams illustrating an example planar image that starts being replayed when a user enlarges a spherical image
- FIG. 23 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment
- FIG. 24 is a diagram illustrating an example menu for a user to control replay of a moving image
- FIG. 25 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment
- FIG. 26 is an example diagram illustrating a process for automatic enlargement of a planar image in a view area
- FIG. 27 is an example diagram illustrating a process for automatic enlargement of a planar image in a case where the entire planar image is not included in a view area;
- FIG. 28 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment
- FIGS. 29A to 29C are example diagrams illustrating metadata for replay of a moving image
- FIG. 30 is an example diagram schematically illustrating an outline obtained as a result of projection conversion
- FIG. 31 is an example diagram schematically illustrating superimposition of an outline
- FIG. 32 is a diagram illustrating examples of an outline displayed in different forms for a moving image and a still image.
- FIG. 33 is a flowchart illustrating operation of displaying an outline of a planar image on a spherical image, performed by the display control apparatus, according to an embodiment.
- a first image is an image superimposed with a second image
- a second image is an image to be superimposed on the first image.
- the first image is an image covering an area larger than that of the second image.
- the first image and the second image are images expressed in different projections.
- the second image is an image with image quality higher than that of the first image, for example, in terms of image resolution.
- “high image quality” differs depending on the image, and therefore, may vary depending on the user's purpose of viewing.
- “high image quality” means an image faithfully representing a scene.
- “high image quality” means an image having a high resolution, a wide dynamic range, high color reproducibility, or less noise.
- the first image include a spherical image, an equirectangular projection image, and a low-definition image.
- the second image include a planar image, a perspective projection image, and a high-definition image.
- the spherical image does not have to be the full-view spherical image.
- the spherical image may be the wide-angle view image having an angle of about 180 to 360 degrees in the horizontal direction.
- it is desirable that the spherical image is image data having at least a part that is not entirely displayed in the predetermined area T.
- FIG. 1A illustrates a spherical image CE mapped on a virtual, three-dimensional (3D) sphere CS.
- FIG. 1B illustrates the equirectangular projection image CE.
- the spherical image CE generated by a spherical camera has a 3D form that is obtained by mapping the to equirectangular projection image CE as illustrated in FIG. 1B on the sphere CS illustrated in FIG. 1A .
- the virtual camera IC corresponds to the point of view of a user. In FIG. 1A , the point of view is present at the center of the spherical image CE.
- the user can display a predetermined area T, such as a view area T (an area on a display or an area displayed by display software), in the spherical image CE as desired. Further, the user can enlarge or reduce the view area T.
- the predetermined area T is referred to as the view area T.
- FIGS. 2A to 2C are example diagrams illustrating a display method for displaying a moving image superimposed on a spherical image.
- a planar image P that is a moving image is superimposed on the spherical image CE illustrated in FIG. 2A .
- a user enlarges the spherical image CE in order to enlarge and view the planar image P as illustrated in FIG. 2A to FIG. 2B .
- the user may or may not be aware of the moving image.
- the display control apparatus may indicate the presence of the planar image P (for example, display an outline described below).
- the moving image means an active image obtained by successively displaying, at short intervals, a series of images (frames) captured at substantially regular intervals.
- the user gradually enlarges the spherical image CE as illustrate in FIG. 2A to FIG. 2B .
- the planar image P illustrated in FIG. 2A corresponds to the entire planar image P illustrated in FIGS. 2B and 2C .
- FIG. 2B when the user enlarges the spherical image CE to some extent, it is inferred that the user wants to view the planar image P, and the display control apparatus starts replaying the moving image.
- FIG. 2C the moving image in which a monorail train 351 is moving leftward from the right is displayed.
- the moving image is replayed. Therefore, the moving image can be replayed in a case where the user wants to view the planar image P, and unwanted replay of the moving image can be suppressed. Further, the user can view the moving image from the beginning. If the moving image is replayed at all times, when the user starts viewing the moving image, the user is unable to know the time elapsed since the start of replay. Further, when the user enlarges the planar image P, replay of the moving image starts. Therefore, motion seems to start suddenly on the screen, which can attract the user's attention. In this disclosure, it is assumed that the second image is being viewed by the user, when there is an event that can be determined that the user is consciously looking at the second image.
- FIGS. 3A to 3C are diagrams illustrating example system configurations of an image processing system 100 including a spherical camera 20 and a display control apparatus 30 .
- FIG. 3A illustrates the image processing system 100 that includes the spherical camera 20 and a general-purpose (generic) digital camera 9 .
- the generic digital camera 9 is typically an is image capturing device (camera) that captures images with a focal length of 35 mm or more on a 35-mm-film basis and may be an image capturing device capable of capturing wide-angle images corresponding to about 24 mm to 35 mm.
- the digital camera 9 is not the spherical camera 20 but an image capturing device that captures the planar image P.
- the digital camera 9 is an image capturing device capturing an image that can be entirely displayed in the view area T.
- the digital camera 9 includes an imaging element having at least a resolution higher than that of the spherical image CE (having a large number of pixels relative to the angle of view for image capturing). Further, in the digital camera 9 , image capturing conditions (for example, the exposure time, the shutter speed, and the white balance) are optimized for an image capturing range narrower than that of the spherical camera 20 , and therefore, it is often the case that whiteout is less likely to occur, the color reproducibility is high, and less noise is produced. As described above, the digital camera 9 generates planar images of high image quality.
- the digital camera 9 also functions as the display control apparatus 30 .
- Specific examples of the digital camera 9 include a digital still camera and a digital video camera.
- the digital camera 9 may be replaced by another device that includes an image capturing device.
- the digital camera 9 may be replaced by a smartphone, a tablet terminal, a personal digital assistant (PDA), or a wearable personal computer (PC).
- the spherical camera 20 may be externally attached to, for example, the smartphone.
- the spherical camera 20 and the digital camera 9 can communicate with each other using, for example, Universal Serial Bus (USB), Bluetooth (registered trademark), or a wireless local area network (LAN).
- USB Universal Serial Bus
- LAN wireless local area network
- the spherical image CE captured by the spherical camera 20 is transmitted to the digital camera 9 .
- the digital camera 9 captures the planar image P.
- the planar image P corresponds to a part of the spherical image CE.
- the digital camera 9 functions as the display control apparatus 30 and superimposes the planar image P on the spherical image CE. Specifically, for example, a user U selects the planar image P and the spherical image CE and instructs the digital camera 9 to perform superimposition. Alternatively, the user U sets the digital camera 9 to a superimposition mode and captures the planar image P. Then, the digital camera 9 automatically superimposes the planar image P on the spherical image CE that matches the planar image P. If the degree of image matching is low, the digital camera 9 does not perform superimposition, which causes less inconvenience. Further, the digital camera 9 that functions as the display control apparatus 30 displays a moving image on the display of the digital camera 9 while switching between replay and stop of the moving image.
- the digital camera 9 functions as the display control apparatus 30
- the planar image P can be superimposed on the spherical image CE, and switching between replay and stop of the moving image can be performed.
- FIG. 3B illustrates the image processing system 100 that includes the spherical camera 20 , the generic digital camera 9 , and the display control apparatus 30 .
- the spherical camera 20 and the display control apparatus 30 can communicate with each other using, for example, USB, Bluetooth (registered trademark), or a wireless LAN, and the digital camera 9 and the display control apparatus 30 can communicate with each other using, for example, USB. Bluetooth (registered trademark), or a wireless LAN.
- One or more spherical images CE captured by the spherical camera 20 owe transmitted to the display control apparatus 30 .
- One or more planar images P captured by the digital camera 9 are also transmitted to the display control apparatus 30 . Instead of transmission, the spherical images CE or the planar images P stored in a storage medium may be read by the display control apparatus 3 C) from the storage medium.
- the display control apparatus 30 is an information processing apparatus, such as a PC, a smartphone, a tablet terminal, or a PDA.
- the display control apparatus 30 may be a multifunction peripheral, a projector, or a video conference terminal having functions of an information processing apparatus.
- the display control apparatus 30 superimposes at least one of the planar images P on one of the spherical images CE.
- the display control apparatus 30 displays a moving image on the display of the display control apparatus 30 while switching between replay and stop of the moving image. More specifically, in the case of superimposing one planar image P on the spherical image CE, the to user U compares the planar images P and the spherical images CE and knows which planar image P constitutes part of which spherical image CE. Then, the user U selects one of the spherical images CE and the planar image P that is to be placed on the spherical image CE, and instructs the display control apparatus 30 to perform placement.
- the display control apparatus 30 may automatically perform superimposition. As described above, the display control apparatus 30 can obtain the spherical image CE and the planar image P, superimpose the planar image P on the spherical image CE, and switch between replay and stop of the moving image.
- the description of FIG. 4 and subsequent figures is given while assuming the configuration illustrated in FIG. 3B unless otherwise specified.
- FIG. 3C illustrates the image processing system 100 in which the spherical camera 20 , the generic digital camera 9 , the display control apparatus 30 , and an image processing server 70 are connected to one another via a network N.
- the network N is formed of, for example, a LAN formed in the user U's environment, a provider network of a provider that connects the LAN to the Internet, and a line provided by a carrier.
- the network N includes a plurality of LANs
- the network N is called a wide area network (WAN) or the Internet.
- the network N may be formed of wired or wireless networks or may be formed of a combination of wired and wireless networks.
- the spherical camera 20 and the digital camera 9 can be connected to the provider network without using the LAN.
- the image processing server 70 may be a general information processing apparatus or a server apparatus. Specific examples of the display control apparatus 30 illustrated in FIG. 3C may be similar to the specific examples of the display control apparatus 30 illustrated in FIG. 3B . It is preferable that the image processing server 70 support cloud computing. Cloud computing is a use form in which a user uses resources in a network without awareness of a specific hardware resource.
- One or more spherical images CE captured by the spherical camera 20 are transmitted to the image processing server 70 .
- One or more planar images P captured by the digital camera 9 are also transmitted to the image processing server 70 .
- the image processing server 70 superimposes at least one of the planar images P on one of the spherical images CE. More specifically, the display control apparatus 30 accesses the image processing server 70 in response to an operation by the user U and displays a list of the spherical images CE and a list of the planar images P (for example, thumbnail images).
- the user U compares the planar images P and the spherical images CE and knows which planar image P constitutes part of which spherical image CE. Then, the user U selects one of the spherical images CE and the planar image P that is to be placed on the spherical image CE, and transmits the selected spherical image CE and planar image P to the image processing server 70 .
- the image processing server 70 may automatically perform superimposition.
- the image processing server 70 superimposes the planar image P on the spherical image CE and transmits image data of the view area T to the display control apparatus 30 . It is assumed that the line-of-sight direction and the angle of view in the initial state are determined in advance.
- the image processing server 70 which functions as a Web server, transmits image data of the view area T and a display program (described in a script language) for accepting an operation performed by the user U to the display control apparatus 30 .
- the display control apparatus 30 executes the display program to accept an operation performed by the user U and transmits the line-of-sight direction and the angle of view to the image processing server 70 .
- the image processing server 70 updates the view area.
- the image processing server 70 switches between replay and stop of a moving image, and therefore, the display control apparatus 30 can display an image in which a moving image is replayed and an image in which a moving image is not replayed.
- the image processing server 70 may transmit the spherical image CE, the planar image P, and the display program to the display control apparatus 30 .
- the display control apparatus 30 determines the view area T in accordance with an operation performed by the user U and switches between replay and stop of a moving image.
- the image processing server 70 may calculate a location parameter (described below) from the spherical image CE and the planar image P, and the display control apparatus 30 may download the spherical image CE, the planar image P, and the location parameter. Although the user changes the line-of-sight direction as desired, the display control apparatus 30 can superimpose the planar image P on the spherical image CE by using the same location parameter.
- the image processing server 70 includes a location parameter generator 8 illustrated in FIG. 12 described below, and the display control apparatus 30 includes a conversion/display unit 7 illustrated in FIG. 12 .
- FIG. 4 is a hardware configuration diagram of the spherical camera 20 .
- the spherical camera 20 is an omnidirectional image capturing device including two imaging elements: however, the number of imaging elements may be three or more.
- the spherical camera 20 need not be a device dedicated to omnidirectional is image capturing.
- an omnidirectional image capturing unit may be attached to, for example, the generic digital camera 9 or a smartphone to thereby implement substantially the same functions as those of the spherical camera 20 .
- the spherical camera 20 includes an imaging unit 101 , an image processor 104 , an imaging controller 105 , a microphone 108 , an audio processor 109 , a central processing unit (CPU) 111 , a read-only memory (ROM) 112 , a static random access memory (SRAM) 113 , a dynamic random access memory (DRAM) 114 , an operation unit 115 , a network interface (UF) 116 , a communication circuit 117 , an antenna 117 a, an electronic compass 118 , a gyro sensor 119 , and an acceleration sensor 120 .
- CPU central processing unit
- ROM read-only memory
- SRAM static random access memory
- DRAM dynamic random access memory
- the imaging unit 101 includes wide-angle lenses (fish-eye lenses) 102 a and 102 b each having an angle of view of 180° or more so as to form a hemispherical image and two imaging elements 103 a and 103 b corresponding to the wide-angle lenses 102 a and 102 h respectively.
- wide-angle lenses fish-eye lenses
- the imaging elements 103 a and 103 b each include an image sensor, such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor, that converts an optical image captured by the fish-eye lens to an electric signal and outputs image data, a timing generation circuit that generates, for example, a horizontal or vertical synchronization signal and a pixel clock for the image sensor, and a group of registers in which, for example, various commands and parameters used in an operation of the imaging element are set.
- CMOS complementary metal oxide semiconductor
- CCD charge coupled device
- the imaging elements 103 a and 103 b of the imaging unit 101 are each connected to the image processor 104 via a parallel I/F bus. Further, the imaging elements 103 a and 103 b of the imaging unit 101 are each connected to the imaging controller 105 via a serial I/F bus (for example, an I2C bus).
- the image processor 104 and the imaging controller 105 are connected to the CPU 111 via a bus 110 .
- the ROM 112 , the SRAM 113 , the DRAM 114 , the operation unit 115 , the network OF 116 , the communication circuit 117 , and the electronic compass 118 are connected to the bus 110 .
- the image processor 104 acquires pieces of image data output from the respective imaging elements 103 a and 103 b via the parallel I/F buses, performs a predetermined process for the pieces of image data, and performs a superimposition process for the pieces of image data to generate data of an image in equirectangular projection as illustrated in FIG. 1B .
- the imaging controller 105 functions as a master device, and the imaging elements 103 a and 103 b function as slave devices. Then, the imaging controller 105 uses the 12 C buses to set commands and so on in the group of registers of each of the imaging elements 103 a and 103 b. The commands and so on that are set are received from the CPU 111 . Furthermore, the imaging controller 105 also uses the I2C buses to acquire status data and so on of the group of registers of each of the imaging elements 103 a and 103 b and transmits the status data and so on to the CPU 111 .
- the imaging controller 105 instructs the imaging elements 103 a and 103 b to output image data at the timing at which the shutter button of the operation unit 115 is pressed.
- the spherical camera 20 may have a preview display function using a display or a function of enabling moving image display. In the case of moving image display, pieces of image data are output from the imaging elements 103 a and 103 b successively at a predetermined frame rate (frames per minute).
- the imaging controller 105 also functions as a synchronization controller that cooperates with the CPU 111 to synchronize the timings at which piece of image data are output from the imaging elements 103 a and 103 b , as described below.
- the spherical camera 20 does not include a display in this embodiment, the spherical camera 20 may include a display.
- the microphone 108 converts sounds to audio data (signal).
- the audio processor 109 acquires the audio data output from the microphone 108 via an I/F bus and performs a predetermined process for the audio data.
- the CPU 111 controls the entire operation of the spherical camera 20 and performs processing as appropriate.
- the ROM 112 stores various programs for the CPU 111 .
- the SRAM 113 and the DRAM 114 are work memories and store, for example, programs to be executed by the CPU 111 and data that is being processed.
- the DRAM 114 stores image data that is being processed by the image processor 104 and data of an image in equirectangular projection that has been processed.
- the operation unit 115 generally refers to various operation buttons, a power switch, the shutter button, and a touch panel having both display and operation functions.
- the user U operates the operation buttons to input various image capturing modes and image capturing conditions.
- the network I/F 116 generally refers to interface circuits (including a USB I/F) providing interfaces with, for example, an external medium, such as an SD card, and a PC.
- the network I/F 116 may function as a wireless or a wired network interface. Data of an image in equirectangular projection stored in the DRAM 114 is recorded to an external medium via the network I/F 116 or transmitted to an external device, such as the display control apparatus 30 , via the network I/F 116 , which functions as a network interface as appropriate.
- the communication circuit 117 communicates with an external device, such as the display control apparatus 30 , via the antenna 117 a included in the spherical camera 20 by using a short-range wireless technology, such as Wireless Fidelity (Wi-Fi) or the near-field communication (NFC). Data of an image in equirectangular projection can be transmitted to an external device, such as the display control apparatus 30 , also via the communication circuit 117 .
- a short-range wireless technology such as Wireless Fidelity (Wi-Fi) or the near-field communication (NFC).
- Wi-Fi Wireless Fidelity
- NFC near-field communication
- the electronic compass 118 calculates the azimuth of the spherical camera 20 from the magnetism of the earth and outputs azimuth information.
- the azimuth information is an example of related information (metadata) compliant with Exchangeable image file format (Exif) and used in image processing, such as image correction of a captured image.
- the related information also includes other data, such as the date and time when the image is captured and the size of the image data.
- the gyro sensor 119 detects changes in angles (roll, pitch, and yaw angles) associated with a tilt of the spherical camera 20 .
- the changes in the angles are examples of the related data (metadata) compliant with Exif and used in image processing, such as image correction of a captured image.
- the acceleration sensor 120 detects accelerations in the directions of three axes. On the basis of the detected accelerations, the attitude of the spherical camera 20 (the angle relative to the gravity direction) is detected. Both the gyro sensor 119 and the acceleration sensor 120 are provided to thereby increase the accuracy of image correction.
- the imaging unit of the digital camera 9 may include a lens unit having a plurality of lenses, a mechanical shutter button, and an imaging element.
- FIG. 5 is a hardware configuration diagram of the display control apparatus 30 .
- the display control apparatus 30 includes a CPU 501 , which controls the entire operation of the display control apparatus 30 , a ROM 502 , which stores programs including an initial program loader (IPL) that are used to drive the CPU 501 , a RAM 503 , which is used as a work area of the CPU 501 , and a hard disk (HD) 504 , which stores various types of data including a display control apparatus program.
- IPL initial program loader
- HD hard disk
- the display control apparatus 30 further includes a hard disk drive (HDD) 505 , which controls reading/writing of various types of data from/to the HD 504 in accordance with control by the CPU 501 , and a medium drive (media drive) 507 , which controls reading/writing (storing) of data from/to (in) a recording medium 506 , which is, for example, a flash memory.
- HDD hard disk drive
- media drive medium drive
- the display control apparatus 30 further includes a display 508 , which displays various types of information, such as cursors, menus, windows, text, and images, a network I/F 509 , which is used to perform data communication using the network N, a keyboard 511 , which includes a plurality of keys for inputting characters, numbers, and various instructions, and a mouse 512 , which is used to, for example, select and execute various instructions, select a processing target, and move a cursor.
- a display 508 which displays various types of information, such as cursors, menus, windows, text, and images
- a network I/F 509 which is used to perform data communication using the network N
- a keyboard 511 which includes a plurality of keys for inputting characters, numbers, and various instructions
- a mouse 512 which is used to, for example, select and execute various instructions, select a processing target, and move a cursor.
- the display control apparatus 30 further includes an optical medium drive (such as CD-RW drive) 514 , which controls reading/writing of various types of data from/to an optical recording medium 513 , which is an example of an attachable/detachable recording medium, and a bus line 510 , which is constituted by, for example, an address bus and a data bus used to electrically connect the above-described constituent elements, as illustrated in FIG. 5 .
- an optical medium drive such as CD-RW drive
- a bus line 510 which is constituted by, for example, an address bus and a data bus used to electrically connect the above-described constituent elements, as illustrated in FIG. 5 .
- FIG. 6 is a conceptual diagram illustrating use of the spherical camera 20 .
- the user U holds the spherical camera 20 in their hand and uses the spherical camera 20 to capture images of objects around the user U.
- the spherical camera 20 includes the two imaging elements that are arranged such that the back surfaces thereof face each other. Each of the two imaging elements captures an image of objects around the user U to thereby obtain two hemispherical images.
- FIG. 7A is a diagram illustrating a hemispherical image (front side) captured by the spherical camera 20
- FIG. 7B is a diagram illustrating a hemispherical image (back side) captured by the spherical camera 20
- FIG. 7C is a diagram illustrating a spherical image in equirectangular projection
- FIG. 8A is a diagram schematically illustrating a state where a sphere is covered by an image in equirectangular projection
- FIG. 8B is a diagram illustrating the spherical image CE.
- an image captured by the spherical camera 20 is a hemispherical image (front side) that curves due to the fish-eye lens.
- an image captured by the spherical camera 20 is a hemispherical image (back side) that curves due to the fish-eye lens.
- the hemispherical image (front side) and the hemispherical image (back side), which is flipped by 180 degrees, are combined by the spherical camera 20 , and an image in equirectangular projection is generated accordingly, as illustrated in FIG. 7C .
- the image in equirectangular projection is placed so as to cover the spherical surface, as illustrated in FIG. 8A , by using Open Graphics Library for Embedded Systems (OpenGL ES), and the spherical image CE as illustrated in FIG. 8B is generated. Accordingly, the spherical image CE is represented as an image in equirectangular projection that faces the center of the sphere.
- OpenGL ES is a graphics library that is used to visualize two-dimensional (2D) and 3D data.
- the spherical image CE may be a still image or a moving image.
- the image in equirectangular projection illustrated in FIG. 8A is converted to the 3D spherical image illustrated in FIG. 8B by using expressions (1) described with reference to FIGS. 9A and 9B .
- FIG. 9A is a diagram illustrating the spherical image CE in equirectangular projection
- FIG. 9B illustrates the spherical image CE that is obtained by placing the spherical image CE in equirectangular projection on a full sphere.
- Conversion from the image illustrated in FIG. 9A to the image illustrated in FIG. 9B is a process that is performed by the spherical camera 20 illustrated in FIG. 1 at the time when the spherical camera 20 performs image capturing.
- the coordinates of a point (x, y, z) in a 3D space corresponding to a point ( ⁇ , ⁇ ) in equirectangular projection can be calculated by using the following expression.
- the radius of the sphere is equal to 1.0.
- the spherical image CE that is a 3D image is obtained from the spherical image CE in equirectangular projection.
- the spherical image CE is an image placed so as to cover the spherical surface and is curved, Accordingly, the spherical image
- the display control apparatus 30 displays the planar image P that curves to a small degree in the view area T, which is part of the spherical image CE, so that the image does not feel unnatural to the person.
- the view area T is represented by coordinates (x, v, z) in a 3D virtual space.
- the display 508 is a 2D plane, and therefore, the display control apparatus 30 is unable to display the view area T as is,
- the display control apparatus 30 performs perspective projection conversion in which a 3D object is projected onto a 2D plane by using a 3D computer graphics technique to thereby obtain an image that curves to a smaller degree.
- the view area T of the spherical image CE is displayed on the display 508 .
- FIG. 10 is an example diagram illustrating the line of sight of the user U.
- the spherical image CE has a 3D coordinate system, and therefore, the line-of-sight direction is identified by using information, such as 3D coordinates or a latitude and longitude, for identifying a position on the sphere.
- the center cp of the view area T is assumed to correspond to the line-of-sight direction.
- the user U can change the line-of-sight direction using the keyboard 511 or the mouse 512 . If it is assumed that the virtual camera IC does not allow translation, the virtual camera IC allows three types of rotation, namely, roll (rotation about the Z axis), yaw (rotation about the Y axis), and pitch (rotation about the X axis). When any of the three types of rotation occurs, the line-of-sight direction changes. For example, when the user U rotates the spherical image CE in the horizontal direction, the yaw angle changes. When the user U rotates the spherical image CE in the vertical direction, the pitch angle changes.
- roll rotation about the Z axis
- yaw rotation about the Y axis
- pitch rotation about the X axis
- the roll angle changes.
- an operation performed by the user U on a Web page is reflected to, for example, the line-of-sight direction (the roll angle, the yaw angle, and the pitch angle).
- the way of reflection is described in advance in a program that is executed by the display control apparatus 30 .
- FIGS. 11A to 11D are example diagrams schematically illustrating enlargement and reduction of the view area T in the spherical image CE.
- FIG. 11A illustrates the view area in the initial state.
- the view area T is determined in accordance with the angle of view ⁇ .
- the angle of view ⁇ is an angle that is determined when, for example, diagonally opposite vertexes of the view area T are viewed from the center of the sphere CS.
- the angle of view ⁇ in the initial state is represented by ⁇ 0 .
- ⁇ 0 The angle of view ⁇ in the initial state
- the display control apparatus 30 decreases or increases the angle of view in accordance with the amount of operation.
- the display control apparatus 30 is unable to display an image behind the virtual camera IC as long as the virtual camera IC is present at the center. Accordingly, when the user U performs an operation for is further reduction in the state illustrated in FIG. 11C , the display control apparatus 30 moves the virtual camera IC backward. In FIG. 11D , the angle of view ⁇ 2 remains the same as that in FIG. 11C ; however, the virtual camera IC moves backward, and the view area T further becomes wider accordingly. As a consequence, the image can be reduced further than that in FIG. 11C .
- FIG. 12 is an example functional block diagram illustrating functions of the digital camera 9 , the spherical camera 20 , and the display control apparatus 30 as blocks.
- the digital camera 9 includes a planar image obtainer 11 .
- the planar image obtainer 11 is implemented as an image capturing function of the digital camera 9 and generates image data (planar image P) in which each of the pixels according to a predetermined resolution represents gradation.
- the planar image P may include a single still image, a plurality of still images, or a moving image. In this embodiment, it is assumed that the planar image P is a moving image unless otherwise specified.
- the spherical camera 20 includes a spherical image obtainer 12 .
- the spherical image obtainer 12 is implemented as an image capturing function of the spherical camera 20 and generates image data (spherical image CE) of a 360-degree view around the spherical camera 20 .
- the spherical image CE may include a single still image, a plurality of still images, or a moving image. In this embodiment, it is assumed that the spherical image CE is a still image unless otherwise specified.
- the display control apparatus 30 mainly includes the location parameter generator 8 and the conversion/display unit 7 .
- the functions of the location parameter generator 8 and the conversion/display unit 7 included in the display control apparatus 30 are functions or units implemented by the CPU 501 of the display control apparatus 30 illustrated in FIG. 5 executing a program loaded to the RAM 503 from the HD 504 .
- the location parameter generator 8 includes a spherical image reader 21 , a planar image reader 22 , a location parameter calculator 23 , and a location parameter writer 24 .
- the spherical image reader 21 reads the spherical image CE from the spherical image obtainer 12 .
- the planar image reader 22 reads the planar image P from the planar image obtainer 11 .
- the term “read” means to obtain, receive, read from a storage medium, or accept input.
- the location parameter calculator 23 identifies a match area in the spherical image CE that matches the planar image P, and calculates a location parameter PP for identifying the match area.
- the details of the location parameter PP are described with reference to FIGS. 13A to 13C .
- the spherical image CE is distorted due to equirectangular projection, and therefore, it is preferable to perform similar conversion for the planar image P to make the planar image P distorted. Then, features points in both the spherical image CE and the planar image P are extracted and matched to thereby obtain the location parameter PP.
- the spherical image CE is a still image
- the planar image P is a moving image, and therefore, only a frame (a still image included in the moving image) at a specific time matches the spherical image CE.
- the location parameter calculator 23 determines whether each of the frames of the planar image P matches the spherical image CE sequentially from the first frame. Then, a frame for which the matching degree is equal to or larger than a threshold is used to calculate the location parameter PP. Alternatively, the user U may communicate the matching frame to the display control apparatus 30 .
- the location parameter writer 24 transmits the frame identification (ID) 6 of the frame that matches the spherical image CE to the conversion/display unit 7 .
- ID means an identifier or identification information.
- the ID is a name, a code, a character string, a numerical value, or any combination thereof used to uniquely distinguish a specific target from the others. The same applies to any ID other than the frame ID.
- planar image P is divided in to mesh-like areas and it is determined whether each of the areas matches the spherical image CE, a shift due to, for example, a distortion in a lens can be additionally corrected.
- the location parameter calculator 23 calculates the location parameter PP for each of the planar images P.
- the conversion/display unit 7 includes a planar image reader 25 , a line-of-sight direction/angle-of-view specifier 26 , a spherical image reader 27 , a first projection converter 28 , a second projection converter 29 , an image superimposing unit 33 , a location parameter reader 32 , and an image display unit 31 .
- the functions of the spherical image reader 27 are the same as the functions of the spherical image reader 21 of the location parameter generator 8 .
- the location parameter reader 32 obtains the location parameter PP and the frame ID from the location parameter generator 8 .
- the planar image reader 25 reads each frame of the planar image P.
- the planar image reader 25 reads a frame having the frame ID specified by the location parameter calculator 23 until the image superimposing unit 33 described below sends a notification indicating that moving image reading is ON to the planar image reader 25 .
- the planar image reader 25 reads chronologically subsequent frames (or succeeding frames in a case of reverse replay) in accordance with the replay speed of the moving image.
- the planar image reader 25 repeatedly reads a frame having the frame ID or the last read frame.
- the line-of-sight direction/angle-of-view specifier 26 accepts the line-of-sight direction and the angle of view (enlargement or reduction) controlled by the user U. On the basis of the line-of-sight direction and the angle of view that are controlled and input, the view area T is determined.
- the user U controls the line-of-sight direction and the angle of view (enlargement or reduction) as desired. Therefore, a part of the planar image P subjected to projection conversion may fit in the view area T or the planar image P subjected to projection conversion might not fit in the view area T at all. Accordingly, the first projection converter 28 generates mask data that indicates an area, in the projection-converted image, that is included in the view area T. The details are described with reference to FIGS. 14A and 14B .
- the second projection converter 29 determines the view area T that corresponds to the line-of-sight direction and the angle of view accepted by the line-of-sight direction/angle-of-view specifier 26 , performs, on the basis of the display size of the image display unit 31 , to perspective projection conversion for the view area T of the spherical image CE that has been read, and generates a display image. Therefore, the display image is a 2D planar image.
- the image superimposing unit 33 superimposes the projection-converted image on the display image by using the mask data to generate a superimposed image. The details are described with reference to FIG. 15 .
- the image display unit 31 displays the superimposed image on the display 508 .
- FIGS. 13A to 13C are diagrams illustrating projection conversion of the planar image P in a match area and an example of the location parameter PP for associating the planar image P with the match area. It is assumed that the correspondence between the planar image P and the spherical image CE has been determined via matching.
- the location parameter PP includes the latitude and longitude of a position in the spherical image CE at which the planar image P is present when the feature points match to the highest degree.
- FIG. 13A illustrates the planar image P
- FIG. 13B illustrates the spherical image CE in equirectangular projection
- a match area 301 is illustrated.
- the location parameter calculator 23 divides the planar image P into grids, and obtains, for each intersection point of the grids represented by coordinates (x, y), a corresponding point ( ⁇ , ⁇ ) in the match area 301 of the spherical image CE as the location parameter PP.
- ⁇ represents the longitude
- ⁇ represents the latitude
- FIG. 13C illustrates an example of the location parameter PP.
- the coordinates of each grid in the planar image P is associated with the latitude and longitude of a corresponding point in the match area 301 of the spherical image CE.
- FIGS. 14A and 14B are example diagrams illustrating a projection-converted image 302 and mask data 303 generated by the first projection converter 28 .
- FIG. 14A illustrates the projection-converted image 302 .
- the projection-converted image 302 has an area having the same size as the size of the view area T in the spherical image CE in equirectangular projection.
- the planar image P after projection conversion is placed on the match area 301 .
- the planar image P is subjected to projection conversion and placed on the match area 301 , and therefore, the planar image P is in a state of trapezoidal distortion.
- a portion not covered by the planar image P is a uniform gray image 307 .
- the uniform image 307 is not used in superimposition, and therefore, may have any pixel value.
- FIG. 14B illustrates the mask data 303 that corresponds to the planar image P to illustrated in FIG. 14A .
- the mask data 303 is data used to take the planar image P included in the view area T from the projection-converted image 302 illustrated in FIG. 14A .
- the area of white pixels in the mask data 303 corresponds to an area that is the area of the planar image P and the area of the view area I Therefore, the area of white pixels in the mask data 303 is equal to or smaller than the area of the planar image P.
- the mask data 303 illustrated in FIG. 14B is mask data in a case where the view area view area T is larger than the planar image P. Therefore, in the example illustrated in FIG.
- the match area 301 is substantially equal to the area of white pixels in the mask data 303 .
- the entire area of the mask data 303 is constituted by black pixels.
- the mask data 303 is constituted by only white pixels.
- the area of white pixels in the mask data 303 and the match area 301 have the same size and are present at the same position.
- the image superimposing unit 33 performs a mask process using the projection-converted image 302 and the mask data 303 .
- the mask process is a process for taking a pixel corresponding to a white pixel in the mask data 303 from the projection-converted image 302 .
- the image superimposing unit 33 acquires a pixel value corresponding to the position of a white pixel from the projection-converted image 302 and superimposes the pixel value on the display image at the same position as the position in the projection-converted image 302 .
- the boundary between the black pixels and the white pixels of the mask data 303 gradations in which pixels gradually change from the white pixels to the black pixels. Accordingly, the boundary between the spherical image CE having a low resolution and the planar image P having a high resolution can be less noticeable.
- FIG. 15 is an example diagram schematically illustrating superimposition of the planar image P on the spherical image CE.
- the projection-converted image 302 ( FIG. 14A ), and the mask data 303 ( FIG. 14B ) are input to the image superimposing unit 33 .
- the image superimposing unit 33 superimposes the projection-converted image 302 , which is a wider-angle image, on the display image 304 .
- the image superimposing unit 33 to performs a mask process using the mask data 303 to take the projection-converted image 302 corresponding to the pixel positions of white pixels in the mask data 303 and overlay the taken image on the display image 304 . In this case, corresponding pixels in the display image 304 are lost at this time. Accordingly, a superimposed image 305 is generated in which the planar image P having a high resolution is placed on the spherical image CE having a low resolution.
- a transparent layer may be used, and the projection-converted image 302 corresponding to the pixel positions of white pixels in the mask data 303 may be taken and placed on the transparent layer. In this case, no pixels are lost form the display image 304 and, for example, the user U can switch between display and non-display of the planar image P.
- the image superimposing unit 33 outputs the superimposed image 305 .
- the planar image P having a high resolution is placed on the spherical image CE having a low resolution.
- the display control apparatus 30 performs perspective projection conversion of the spherical image CE and superimposition of the planar image P in parallel. If the planar image P is placed on the spherical image CE, and thereafter, perspective projection conversion is performed, the resolution of the spherical image CE needs to be the same as the resolution of the planar image P, resulting in an increase in the amount of data. In this embodiment, the display image 304 is generated, and thereafter, the planar image P is superimposed, which can suppress an increase in the amount of data.
- the display control apparatus 30 repeatedly superimposes the planar image P on the spherical image CE in accordance with the display cycle of the display 50 l (for example, 30 to 60 times per second). Accordingly, the display control apparatus 30 retains one planar image P and one spherical image CE and generates the superimposed image 305 in real time from the point of view according to an operation by the user U. Furthermore, the display control apparatus 30 can replay a moving image.
- FIG. 16 is a diagram schematically illustrating the planar image P superimposed on the spherical image CE.
- the planar image P which is a plane, is superimposed on the display image 304 as is, and is not placed on the sphere CS as illustrated in FIG. 16 .
- the planar image P seems to be superimposed on part of the sphere having the same center position and the same radius as those of the spherical image CE when viewed by the user U, as illustrated in FIG. 16 .
- the planar image P may be superimposed on the spherical image CE that is a 3D image.
- the longitude and latitude corresponding to the planar image P are known via matching (using the location parameter PP), and therefore, the display control apparatus 30 can place the planar image P on the spherical image CE by using expressions is (1).
- hidden surface removal (Z-buffer algorithm) using depth information is not used, and rendering needs to be performed on a later-priority basis.
- Superimposition of a plurality of planar images P can be similarly performed by the location parameter calculator 23 calculating the location parameter PP for each of the planar images P.
- rendering of a wide-angle image is performed first, and rendering of a telephoto image is performed later. Accordingly, in a case where images are captured by the digital camera 9 having the same number of pixels, the planar image P having a low resolution is not overlaid on the planar image P having a high resolution.
- FIG. 17 is an example flowchart illustrating a procedure for the display control apparatus 30 to superimpose the planar image P on the spherical image CE.
- the process in FIG. 17 starts when, for example, a user gives an instruction for superimposition. Steps S 10 to S 30 are performed in parallel with steps S 40 to S 60 : however, steps S 10 to S 60 may be performed sequentially.
- the planar image reader 22 reads the planar image P (step S 10 ).
- the spherical image reader 21 reads the spherical image CE (step S 20 ).
- the location parameter calculator 23 calculates the location parameter PP (step S 30 ).
- the planar image reader 25 reads the planar image P (step S 40 ).
- the spherical image reader 27 reads the spherical image CE (step S 50 ),
- the line-of-sight direction/angle-of-view specifier 26 accepts a line-of-sight direction and an angle of view (step S 60 ).
- Step S 60 is performed as appropriate, for example, at every predetermined time or according to detection of a user operation,
- the first projection converter 28 performs projection conversion of the planar image P using the location parameter PP to generate the projection-converted image 302 (step S 70 ).
- the first projection converter 28 generates the mask data 303 in accordance with the view area I that is determined on the basis of the line-of-sight direction and the angle of view and in accordance with the planar image P (match area) subjected to projection conversion (step S 80 ).
- the second projection converter 29 performs perspective projection conversion of the spherical image CE in accordance with the view area T to generate the display image 304 (step S 90 ).
- the image superimposing unit 33 performs superimposition of the is projection-converted image 302 and the display image 304 using the mask data 303 to generate the superimposed image 305 (step S 100 ).
- the image display unit 31 displays the superimposed image 305 (step S 110 ).
- the display control apparatus 30 repeats the process from step S 60 to step 5110 in FIG. 17 .
- FIG. 18 is an example functional block diagram illustrating functions of the image superimposing unit 33 as blocks.
- the image superimposing unit 33 includes a determiner 35 and an image switching unit 36 .
- the determiner 35 determines whether it is inferred that the user U views the planar image P superimposed on the superimposed image 305 and transmits the result of determination to the image switching unit 36 .
- the image switching unit 36 sends a notification indicating that moving image reading is ON or moving image reading is OFF to the planar image reader 25 in accordance with the result of determination.
- the display control apparatus 30 starts replaying a moving image.
- Specific examples of when it is inferred that the user U views the planar image P include cases when any one of the following conditions (i) to (iii) is satisfied. In such case, a moving image is replayed.
- the angle of view of the planar image P in the current view area T relative to the angle of view of the view area T is equal to or larger than a predetermined value.
- the area of the planar image P has an angle of view equal to or larger than a predetermined value.
- the case where it is inferred that the user U views the planar image P includes a case where the user U is actually viewing the planar image P and a case where the user U is likely to view the planar image P.
- the condition (iii) may be referred to as replay of a moving image in response to an operation by the user U for viewing.
- FIGS. 19A to 19D are example diagrams illustrating the relative position of the planar image P relative to the view area T.
- the view area T is an area in the spherical image CE that is displayed on the display 508 , and therefore, the user U views the planar image P from substantially the front.
- the angle of view of the view area T is represented by a
- the angle of view of the planar image P is represented by b
- angle of view a> angle of view b is satisfied, the entire planar image P and part of the spherical image CE are visible to the user U.
- the determiner 35 need not determine that the user U wants to view the planar image P.
- the determiner 35 determines, for example, whether the angle of view a is 1.2 times the angle of view b or less. That is, in a case where the angle of view a is 1.2 times the angle of view b or less, the determiner 35 determines that the ratio of the angle of view of the planar image P relative to the current angle of view is equal to or larger than a predetermined value. Then, the image switching unit 36 transmits moving image replay request ON to the planar image reader 25 . In a case where the determiner 35 determines that the angle of view a is more than 1.2 times the angle of view b, the image switching unit 36 transmits moving image replay request OFF to the planar image reader 25 .
- the view area T is known.
- the latitude and longitude of the planar image P are registered as the location parameter PP, and therefore, the angle of view b is calculated from the location parameter PP.
- the details are described with reference to FIG. 20 and FIG. 21 .
- the view area is narrower than the planar image P. That is, the user U views only the planar image P.
- the angle of view a is 1.2 times the angle of view b or less, and therefore, a moving image is replayed, as a matter of course.
- FIG. 19C only an upper portion of the planar image P is included in the view area T. That is, the user U views an upper portion of the planar image P. However, the view area T is fully included in the planar image P (the angle of view a is 1.2 times the angle of view b or less), and therefore, the image switching unit 36 transmits moving image replay request ON.
- the shift (angle) between the center of the view area T and the center of in the planar image P is large, and part of the view area T lies outside the planar image P.
- the planar image P in the view area T is small relative to the view area T (the angle of view a is more than 1.2 times the angle of view b), and therefore, the image switching unit 36 transmits moving image replay request OFF.
- the threshold used to determine whether a moving image is replayed is set to, for example, 1.2 and may be set to any value larger than 1.0.
- the threshold may be set to a value from 1.1 to 1.5.
- the user U may set the threshold.
- FIG. 20 is an example diagram illustrating a way of calculating the angle of view a and the angle of view b.
- the angle of view a is calculated from the view area T, which is determined by the user U performing an operation.
- the distance from the virtual camera IC to the view area T is represented by f
- the length of a diagonal line of the view area T is represented by 2 L
- the angle of view ⁇ , the distance f, and the length L has a relationship expressed by expression (2).
- the coordinates of diagonally opposite vertexes of the view area T are known from the operation performed by the user U, and therefore, the length of the diagonal line 2 L can be easily calculated.
- the distance f is known from the radius of the full sphere or the operation performed by the user U. Therefore, the angle of view a can be calculated by using expression (2).
- the angle of view b can be similarly calculated.
- the angle of view b is calculated in a case where at least part of the planar image P is included in the view area T.
- FIG. 21 is a diagram illustrating a relationship between the view area T and the planar image P.
- the determiner 35 determines whether the planar image P is included in the view area T based on each of the four vertexes.
- the four vertexes of the view area T are represented by A 1 to A 4
- the four vertexes of the planar image P are represented by B 1 to B 4 .
- the vertexes B 1 to B 4 are identified on the basis of, for example, the longitudes and latitudes of corresponding points on the full sphere or may be identified on the basis of the coordinates of corresponding points on the display 508 .
- the vertex B 1 is determined as follows.
- the vertexes B 2 , B 3 , and B 4 are similarly determined. If diagonally opposite vertexes (B 1 and B 3 , and 132 and B 4 ) are included the view area T, the angle of view b can be calculated from the location parameter PP by using expression (2). If not all of the vertexes to B 1 to B 4 of the planar image P are included in the view area T, as illustrated in FIG. 21 , the angle of view h is calculated from the vertex B 1 included in the view area T and the vertex A 3 of the view area T.
- the display control apparatus 30 includes settings indicating that a diagonally opposite vertex corresponding to the vertex B 1 is the vertex A 3 , a diagonally opposite vertex corresponding to the vertex B 2 is the vertex A 4 , a diagonally opposite vertex corresponding to the vertex B 3 is the vertex A 1 , and a diagonally opposite vertex corresponding to the vertex B 4 is the vertex A 2 .
- FIGS. 22A to 22E are diagrams illustrating an example of the planar image P that starts being replayed when the user U enlarges the spherical image CE.
- the planar image P that is small is included in the view area T.
- the angle of view a is more than 1.2 times the angle of view b, and therefore, a moving image is not replayed.
- the angle of view a is 1.2 times the angle of view b or less, and therefore, replay of a moving image starts.
- the spherical image CE is not enlarged, and therefore, the composition (angle of view) remains the same. However, the moving image is replayed, and therefore, the monorail train 351 gradually moves from the right to the left.
- FIG. 23 is an example flowchart illustrating a procedure for the display control apparatus 30 to control whether to replay a moving image.
- the procedure illustrated in FIG. 23 starts when the display control apparatus 30 displays the spherical image CE. It is assumed that the planar image P has been superimposed on the spherical image CE.
- the determiner 35 determines whether the view area T is changed by the user U or a mouse operation is performed by the user U (step S 210 ).
- the user U changing the view area T means the user U changing the line-of-sight direction or performing an enlarging or reducing operation.
- the line-of-sight direction/angle-of-view specifier 26 converts an operation performed by the user U to a line-of-sight direction and an angle of view and transmits the line-of-sight direction and the angle of view to the image superimposing unit 33 .
- the determiner 35 may simply determine whether a mouse operation is performed. Accordingly, the flow proceeds to step S 220 upon a specific mouse event even if the view area T is not changed.
- the image superimposing unit 33 identifies the view area T on the basis of the line-of-sight direction and the angle of view (step S 220 ). That is, the latitude range and the longitude range that correspond to the view area T in the spherical image CE are identified.
- the determiner 35 determines whether at least part of the planar image P is included in the view area T (step S 230 ). If the planar image P is not included in the view area T at all, a moving image need not be replayed, and therefore, the flow proceeds to step S 290 .
- the determiner 35 calculates the angle of view b of the planar image P (step S 240 ). The determiner 35 further calculates the angle of view a of the view area T (step S 250 ).
- the determiner 35 compares the ratio a/b between the angle of view a and the angle of view b with a threshold 1.2 and determines whether the ratio a/b is larger than the threshold 1.2(step S 260 ).
- step S 260 If determination in step S 260 results in Yes (if the ratio a/b is larger than the threshold 1.2), it is not inferred that the user U views the planar image P, and therefore, the image switching unit 36 sends a notification indicating that moving image reading is OFF to the planar image reader 25 (step S 270 ).
- step S 260 If determination in step S 260 results in No (if the ratio a/b is equal to or smaller than the threshold 1.2), it is inferred that the user U views the planar image P, and therefore, the image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S 280 ).
- the image display unit 31 displays, on the display 508 , the superimposed image 305 generated by superimposing the planar image P, which is a still image or a moving image, on the spherical image CE (step S 290 ), If determination in step S 230 results in No, the planar image P is not superimposed.
- a moving image can be replayed.
- replay of the moving image can be stopped.
- the image display unit 31 may display a menu for the user U to control replay of the moving image and accept, for example, stop, replay, rewinding, or fast forward in accordance with an operation by the user U.
- FIG. 24 is a diagram illustrating an example menu (user interface (UI)) for a user to control replay of a moving image.
- FIG. 24 illustrates a menu 20 ( )that includes a replay button 201 , a pause button 202 , a stop button 203 , a fast-forward button 206 , a fast-rewind button 205 , a subsequent-moving-image button 207 , and a previous-moving-image button 204 .
- the menu 200 may further include, for example, a replay repeat button.
- the subsequent-moving-image button 207 and the previous-moving-image button 204 are activated in a case where the planar image P includes a plurality of moving images.
- the designs of the buttons illustrated in FIG. 24 are merely examples.
- the menu 200 as illustrated in FIG. 24 is to be displayed, for example, in a case where a moving image is replayed and the user clicks the moving image replay area or in a case where replay of a moving image ends.
- the planar image reader 25 may repeat reading of the planar image P from the first frame or stop reading the planar image P.
- the method for determination in “(ii) the case where the area of the planar image P has an angle of view equal to or larger than a predetermined value” is described. Determination in this case is substantially the same as that in (i) and is based on the assumption that, in a case where the user U enlarges and displays the planar image P, it is inferred that the user U wants to view the planar image P.
- the coordinates of the vertexes B 1 to B 4 of the planar image P are used as described with reference to FIG. 21 .
- the coordinates of corresponding points on the display 508 obtained as a result of perspective projection conversion of the coordinates of the vertexes B 1 to 34 of the planar image P may be used to perform determination.
- the user U wants to view the planar image P when the distance between the center of the planar image P and the center of the view area T is smaller than a threshold without using the angle-of-view ratio.
- the threshold may be set to a value that corresponds to a fraction of the angle of view of the view area T, such as a quarter of the angle of view of the view area T.
- FIG. 25 is an example flowchart illustrating a procedure for the display control apparatus 30 to control whether to replay a moving image.
- the determiner 35 need not calculate the angle of view a of the view area T, and therefore, step S 250 is omitted.
- step S 260 - 2 the determiner 35 determines whether the angle of view h is larger than a threshold Th. If determination in step S 260 - 2 results in No (if the angle of view b is equal to or smaller than the threshold), it is not inferred that the user U views the planar image P, and therefore, the image switching unit 36 sends a notification indicating that moving image reading is OFF to the planar image reader 25 (step S 270 ).
- step S 260 - 2 If determination in step S 260 - 2 results in Yes (if the angle of view b is larger than the threshold), it is inferred that the user U views the planar image P, and therefore, the image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S 280 ).
- the threshold Th is, for example, an angle of view that corresponds to a specific number of pixels with which the user U can recognize an object.
- a typical number of pixels of the display 508 is assumed to be 768 ⁇ 1024.
- the threshold Th can be set to, for example, an angle of view that corresponds to a number of pixels 500. Accordingly, in a case where the planar image P is enlarged to some extent and displayed, a moving image can be replayed.
- this display method is a method in which, when the user U clicks the planar image P illustrated in FIG. 22A , the view area T illustrated in FIG. 22A changes to the view area T as illustrated in FIG. 22B .
- Such a display method is referred to as automatic enlargement of the planar image P and described.
- FIG. 26 is an example diagram illustrating a process for automatic enlargement of the planar image P in the view area T. It is assumed that the user U performs a click operation in a state where a mouse cursor 311 is placed over the planar image P. In a case of a touch panel, this operation corresponds to an operation in which the user U taps the planar image P with their to finger.
- a mouse cursor and a touch panel are called pointing devices. To simplify description, the following description is given under the assumption that the pointing device is, for example, the mouse cursor 311 .
- the line-of-sight direction/angle-of-view specifier 26 converts the coordinates of a position on the display 508 at which the mouse cursor 311 is placed and at which a click operation is performed to 3D coordinates of a corresponding position in the spherical image CE. This conversion corresponds to the reverse of perspective projection conversion.
- the determiner 35 converts the 3D coordinates corresponding to the position of the mouse cursor 311 to a longitude and latitude and determines whether the position of the mouse cursor 311 indicated by the coordinates is within the planar image P. This determination is performed as follows.
- the determiner 35 determines that the planar image P is clicked.
- the image display unit 31 automatically enlarges the planar image P.
- the second projection converter 29 performs image processing for gradually enlarging the planar image P so as to fit in the view area T. Specifically, a line segment 312 connecting the vertex A 1 and the vertex B 1 , a line segment 312 connecting the vertex A 2 and the vertex B 2 , a line segment 312 connecting the vertex A 3 and the vertex B 3 , and a line segment 312 connecting the vertex A 4 and the vertex B 4 are each divided at equal intervals. In FIG. 26 , each of the line segments 312 is divided into four line segment. The interpolation points on the line segments 312 obtained as a result of the division are referred to as P 1i to P 4i (i is an integer from 1 to 3).
- the process for automatically enlarging the planar image P is a process for decreasing the angle of view a of the view area T.
- the second projection converter 29 reduces the view area ‘T’ so as to fit in an area formed by connecting the interpolation points P 13 , P 23 , P 33 , and P 43 (decreases the angle of view a).
- the second projection converter 29 reduces the view area T so as to fit in an area formed by connecting the interpolation points P 12 , P 22 , P 32 , and P 42 .
- the second projection converter 29 reduces the view area T so as to fit in an area formed by connecting the interpolation points P 11 , P 21 , P 31 , and P 41 .
- the second projection converter 29 reduces the view area T so as to fit in an area formed by connecting the vertexes B 1 , B 2 , B 3 , and B 4 of the planar image P.
- planar image P appears to gradually enlarge to the user U.
- the planar image P is subjected to perspective projection conversion in the view area T and displayed across the display 508 (across an area displayed by display software), and therefore, the planar image P appears to be large to the user U.
- FIG. 27 is an example diagram illustrating a process for automatic enlargement of the planar image P in the case where the entire planar image P is not included in the view area T.
- FIG. 27 only the vertex B 4 of the planar image P is within the view area T.
- the vertexes B 1 to B 3 of the planar image P are not displayed in the view area T but the coordinates of the vertexes B 1 to B 3 are known, and therefore, the planar image P can be automatically enlarged in a similar manner to FIG. 26 .
- FIG. 28 is an example flowchart illustrating a procedure for the display control apparatus 30 to determine whether to replay a moving image. In the description of FIG. 28 , differences from FIG. 23 are described.
- steps S 210 and S 220 is performed in a substantially similar as described above referring to FIG. 23 . Subsequently, the determiner 35 determines whether the planar image P is clicked (step S 222 ). If determination in step S 222 results in No, the subsequent process is similar to that in FIG. 23 .
- step S 222 If determination in step S 222 results in Yes, the image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S 224 ). Accordingly, it is determined that a moving image is to be replayed.
- the second projection converter 29 automatically enlarges the planar image P (step S 226 ). With the process described above, the user U need not perform an operation so as to place the planar image P in front of the user U.
- the user U be allowed to set whether a moving image is to be replayed when the user U clicks the planar image P. After replay of the moving image, when the user U performs an operation for displaying a portion other than the planar image P and the condition in step S 260 is satisfied, the moving image can be stopped. Also in this case, an operation menu may be displayed when the moving image is displayed.
- the description has been given under the assumption that the planar image P is a moving image and the spherical image CE is a still image. Even in a case where the planar image P is a still image and the spherical image CE is a moving image, the above-described display can be implemented with similar processing.
- the spherical image reader 27 needs to read frames of the spherical image CE that is a moving image.
- the display control apparatus 30 displays the spherical image CE of the inside of an art museum
- pictures are represented by the planar images P
- the spherical image CE obtained by capturing an image of the inside of the entire art museum is provided as a moving image.
- the planar images P no visitors are present. Therefore, the user U can enlarge and view the planar images P and easily know the situation from a state represented by the spherical image CE in which, for example, visitors move around the art museum.
- the spherical image CE is displayed as a moving image until one of the conditions (i) to (iii) described above is satisfied.
- the spherical image CE is displayed as a still image. Accordingly, when the user U performs an operation for enlarging and displaying the planar image P, the spherical image CE becomes a still image, resulting in increased viewability.
- both the planar image P and the spherical image CE may be moving images.
- the spherical image CE and the planar image P are displayed as moving images until one of the conditions (i) to (iii) described above is satisfied.
- the spherical image CE and the planar image P are displayed as still images. Accordingly, when the user U performs an operation for enlarging and displaying the planar image P, the spherical image CE becomes a still image, resulting in increased viewability.
- Metadata for replay of a moving image is data in which how the moving image is to be replayed is specified.
- FIGS. 29A to 29C are example diagrams illustrating metadata for replay of a moving image.
- Metadata for replay of a moving image includes a replay start time, a replay end time, a replay direction, a replay speed, a repetition setting, the number of repetitions, a repetition method, and a state after stop, which are described with reference to FIGS. 29A to to 29 C.
- the replay start time is a parameter for replaying the moving image from a midpoint instead of the beginning of the moving image and is specified as a time or a frame number.
- the initial is value is set to 0.
- the replay end time is a parameter for stopping replay of the moving image at a midpoint without replaying the moving image to the end and is specified as a time or a frame number.
- the initial value is set to the length of the moving image.
- the replay start time and the replay end time are specified in a case where the user wants to replay only part of the moving image.
- the moving image that is actually replayed is a portion from the replay start time to the replay end time.
- the replay direction indicates the replay direction when replay of the moving image starts and is set to the forward direction or the reverse direction.
- the replay speed is a parameter for slow replay or fast replay.
- the usual replay speed is 1.0.
- the replay speed of less than 1 (for example, 0.5) indicates slow replay, and the replay speed of more than 1 (for example, 2.0) indicates fast replay.
- the repetition setting is a parameter for specifying whether the moving image is to be repeatedly replayed in a case of replaying the moving image.
- FIG. 29B illustrates a state where the moving image is repeatedly replayed from the beginning to the end.
- the replay direction is the reverse direction and non-repetition is specified, after the moving image has been replayed from the end to the beginning, the replay of the moving image ends.
- the number of repetitions is the number of times the moving image is repeatedly replayed. After the moving image has been repeatedly replayed a specified number of times, the replay of the moving image ends.
- the number of repetitions can be set to a number that corresponds to an infinite number. In this case, the moving image is repeatedly replayed until a stop condition is satisfied.
- the repetition method is a parameter for specifying a replay method in a case of repeatedly replaying the moving image and is set to loop replay or alternate replay.
- the loop replay is replay repeated in the same direction, and the alternate replay is replay repeatedly performed in which the replay direction is flipped each time replay ends, as illustrated in FIG. 29C .
- the replay direction is flipped.
- the state after stop is a parameter for specifying a state after replay of the moving image has stopped.
- the state after stop is set to “unchanged”, “to the beginning” (or “to the end” in the case of replay in the reverse direction), or “transparent”.
- the image processing server 70 can control replay of a moving image in the display control apparatus 30 .
- At least one of the image processing server 70 and the display control apparatus 30 can save metadata in association with the location parameter.
- the display control apparatus 30 when it is inferred that the user views the planar image P, a moving image is replayed. Therefore, the moving image can be replayed in a case where the user wants to view the planar image P, and unwanted replay of the moving image can be suppressed. Further, the user can view the moving image from the beginning. When the user enlarges the planar image P, replay of the moving image starts. Therefore, motion seems to start suddenly on the screen, which can attract the user's attention.
- the display control apparatus 30 that displays the planar image P on the spherical image CE together with an outline indicating the presence of the planar image P is described.
- a constituent element assigned the same reference numeral as that of a corresponding constituent element described in the first embodiment has the same functions as those of the corresponding constituent element. Therefore, description of a constituent element that has been described may be omitted, or only differences may be described.
- planar image P It may be difficult to know a position in the spherical image CE at which the planar image P is superimposed.
- the user U may have difficulty in determining whether the planar image P is a moving image or a still image.
- an outline 40 indicating the presence of the planar image P and displayed by the image superimposing unit 33 is effective. If the form of the outline 40 is made different for a still image and a moving image, the user U looking at the outline 40 can know whether the planar image P is a moving image or a still image.
- the planar image P that is a moving image or a still image has higher image quality than that of the spherical image CE.
- the user U enlarges the spherical image CE to thereby display the enlarged planar image P, the user U can view the image of higher image quality (for example, a higher resolution).
- the outline 40 can be displayed in a similar manner to projection conversion of the planar image P.
- the outline 40 is obtained.
- the first projection converter 28 performs projection conversion of the rectangular indicating the outer edge similarly to a case of projection conversion of the planar image P using the location parameter PP.
- FIG. 30 is an example diagram schematically illustrating the outline 40 obtained as a result of projection conversion.
- the first projection converter 28 makes the form of the outline 40 different depending on whether the planar image P is a moving image or a still image. For example, the first projection converter 28 changes the color of the outline 40 to red in a case of a moving image and to blue in a case of a still image. The first projection converter 28 may change the line type of the outline 40 to a solid line in the case of a moving image and to a dotted line in a case of a still image. Alternatively, the first projection converter 28 may change the line thickness of the outline 40 . The first projection converter 28 may display text indicating whether the planar image P is a moving image or a still image near the outline 40 .
- the planar image reader 25 transmits a notification to the first projection converter 28 whether the planar image P is a moving image or a still image.
- a moving image and a still image may be distinguished from each other on the basis of, for example, the extension of the to file (jpeg indicates a sill image, and mpeg indicates a moving image) or on the basis of the file name of the planar image P.
- FIG. 31 is an example diagram schematically illustrating superimposition of the outline 40 .
- the display image 304 , the projection-converted image 302 , the mask data 303 , and the transparent layer 308 are input to the image superimposing unit 33 .
- the method for superimposing the projection-converted image 302 on the display image 304 has been described with reference to FIG. 15 .
- the image superimposing unit 33 uses the mask data 303 to take, from the transparent layer 308 , an area corresponding to white pixels in the mask data 303 .
- the image superimposing unit 33 superimposes the area taken from the transparent layer 308 on the superimposed image 305 to thereby display the outline 40 indicating the outer edge of the planar image P.
- the user U may perform an operation to switch between display and non-display of the outline 40 .
- the image superimposing unit 33 switches between display and non-display of the transparent layer 308 in accordance with the operation by the user U.
- the image superimposing unit 33 may display the outline 40 of the planar image P over which the mouse cursor is placed.
- FIG. 32 is a diagram illustrating examples of the outline 40 displayed in different forms for a moving image and a still image.
- the moving image is outlined by an outline 40 a represented by a dotted line
- the still image is outlined by an outline 40 b represented by a solid line. Therefore, the user can easily determine whether the planar image P is a moving image or a still image. Further, the user can know the position and size of the planar image P.
- FIG. 33 is an example flowchart illustrating a procedure for the display control apparatus 30 to display the outline 40 of the planar image P on the spherical image CE.
- the process from step S 10 to step S 30 is the same as that in FIG. 17 .
- the location parameter calculator 23 calculates the location parameter PP for all of the planar images P.
- the subsequent steps S 40 to S 90 are performed in a substantially similar manner as described above referring to FIG. 17 .
- the first projection converter 28 obtains the rectangular data 321 that indicates the outer edge of the planar image P, the rectangular data 321 having a different form depending on whether the planar image P is a moving image or a still image.
- the first projection converter 28 performs projection to conversion of the rectangular data 321 indicating the outer edge on the transparent layer 308 using the location parameter PP (step S 92 ), As a result, the outline 40 that differs depending on whether the planar image P is a moving image or a still image is projected onto the transparent layer 308 .
- the image superimposing unit 33 generates the superimposed image 305 (step S 100 ), and uses the mask data 303 to extract the outline 40 from the transparent layer 308 (step S 102 ).
- the image superimposing unit 33 determines whether all of the planar images P have been superimposed (step S 104 ). Until all of the planar images P are superimposed, the flow returns to step S 70 , and the process from step S 70 to step S 102 is performed.
- the display control apparatus 30 can superimpose the planar image P on the spherical image CE and display the outline 40 having a different form depending on whether the planar image P is a moving image or a still image. It is preferable to stop displaying the outline 40 in a case where one of the conditions (i) to (iii) described above is satisfied. If the outline 40 is not displayed, the outline 40 does not hinder the user U from viewing the planar image P any more.
- the image superimposing unit 33 may display the outline 40 only in a case where the user U places the mouse cursor over the planar image P or may display the outline 40 only in a case where the user U does not place the mouse cursor over the planar image P.
- the user U can move the mouse cursor to switch between display and non-display of the outline 40 .
- the image superimposing unit 33 may display the outline 40 until the time elapsed since a change of the view area T reaches a predetermined time, and thereafter, may stop displaying the outline 40 .
- the outline 40 may be displayed with the focal length.
- the outline 40 and focal length information are displayed near the planar image P, and therefore, the user U can know the focal length of the planar image P that is superimposed.
- the position where the planar image P is superimposed can be indicated by, for example, the outline 40 . Further, the user can determine whether the planar image P is a moving image or a still image.
- switching between replay and stop of a moving image is automatically performed; however, the user U may intentionally specify replay or stop of a moving image.
- the spherical image CE may be displayed by using browser software or by using application software for displaying the spherical image CE.
- the image processing server 70 may perform processing up to generation of the location parameter for display and return the generated location parameter for display and the frame ID to the display control apparatus 30 as metadata. Then, the display control apparatus 30 may perform display using the metadata, the spherical image, and the planar image (that may include a moving image).
- FIG. 12 and FIG. 18 are divided into blocks in accordance with main functions in order to facilitate understanding of processing by the spherical camera 20 , the display control apparatus 30 , and the digital camera 9 .
- the present invention is not limited by the way of division into processing units or the names of the processing units. Processing by the spherical camera 20 , the display control apparatus 30 , and the digital camera 9 can be further divided into a larger number of processing units depending on the details of processing. Further, the processing can be divided such that one processing unit includes a larger number of processes. Regarding the processing performed in the configurations illustrated in FIG. 12 and FIG. 18 , either the display control apparatus 30 or the image processing server 70 may perform part or all of the processing.
- the image processing server 70 may perform processing up to processing performed by the image superimposing unit 33 , and the image display unit 31 may be included in the display control apparatus 30 , which is separate from the image processing server 70 .
- a ratio of an area of the view area T in the spherical image CE with respect to an area of the planar image P in the view area T is less than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image.
- a ratio of an area of the view area T in the spherical image CE with respect to an area of the planar image P in the view area T is equal to or greater than the predetermined value, it is inferred that the planar image P is not to viewed by the user, and the planar image P is displayed as a still image.
- a size of the planar image P in the view area T is equal to or larger than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image.
- a size of the planar image P in the view area is less than the predetermined value, it is inferred that the planar image P is not viewed by the user, and the planar image P is displayed as a still image.
- planar image P when a difference between the center of the planar image P and center of the view area T is equal to or less than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image.
- a difference between the center of the planar image P and center of the view area T is larger than the predetermined value, it is inferred that the planar image P is not viewed by the user, and the planar image P is displayed as a still image.
- the view area T may be defined in various ways, determining whether the user is viewing the planar image P may be performed in various ways as described above. Further, the threshold for comparison may be determined, for example, according to the user preference as described above.
- examples of superimposition of images include, but not limited to, placement of one image on top of other image entirely or partly, laying one image over other image entirely or partly, mapping one image on other image entirely or partly, pasting one image on other image entirely or partly, combining one image with other image, and integrating one image with other image. That is, as long as the user can perceive a plurality of images (such as the spherical image and the planar image) being displayed on a display as they were one image, processing to be performed on those images for display is not limited to the above-described examples.
- Processing circuitry includes a programmed processor, as a processor includes circuitry.
- a processing circuit also includes devices such as an application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA), and conventional circuit components arranged to perform the recited functions.
- ASIC application specific integrated circuit
- DSP digital signal processor
- FPGA field programmable gate array
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
An apparatus, system, and method for controlling a display, each of which: controls the display to display an image of a predetermined area of a first image, the first image being superimposed with a second image; determines whether the second image is viewed by a user; and switches a display of the second image between a first display in which the second image is displayed as a still image and a second display in which the second image is displayed as a moving image, based on a determination result indicating whether the second image is viewed by the user.
Description
- This patent application is a continuation of U.S. patent application Ser. No. 15/855,443, filed Dec. 27, 2017, which is based on and claims priority pursuant to 35 U.S.C. § 119(a) to Japanese Patent Application No. 2016-256383, filed on Dec. 28, 2016 and Japanese Patent Application No. 2017-208675, filed on Oct. 27, 2017, in the Japan Patent Office, the entire disclosure of each is hereby incorporated by reference herein.
- The present invention relates to an apparatus, system, and method of controlling display, and a recording medium.
- Image capturing devices capable of capturing wider-angle images than those captured by general-purpose (generic) digital cameras are widely used. For example, a spherical camera provided with a plurality of, for example, fish-eye lenses generates a spherical image of a 360-degree surrounding view captured with a single image capturing operation. A user can determine the line-of-sight direction as desired for the spherical image and view the spherical image in various directions.
- However, taking into consideration an image processing load, image processing of a spherical image for a wide image capturing range is likely to be restricted. The image quality of a spherical image tends to be lower (for example, the resolution is lower, whiteout is more likely to occur, or color reproducibility is lower) than that of an image captured by the generic digital camera. Accordingly, in a case where a user enlarges a portion of a spherical image and views the spherical image, a decrease in the image quality may be unallowable to the user or the user may have difficulty in obtaining detailed information, which may be inconvenient to the user.
- Embodiments of the present invention include an apparatus, system, and method for controlling a display, each of which: controls the display to display an image of a predetermined area of a first image, the first image being superimposed with a second image; determines whether the second image is viewed by a user; and switches a display of the second image between a first display in which the second image is displayed as a still image and a second display in which the second image is displayed as a moving image, based on a determination result indicating whether the second image is viewed by the user.
- A more complete appreciation of the disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings, wherein:
-
FIGS. 1A and 1B are example diagrams illustrating a spherical image; -
FIGS. 2A to 2C are example diagrams illustrating a display method for displaying a moving image superimposed on a spherical image; -
FIGS. 3A to 3C are diagrams illustrating example system configurations of an image processing system including a spherical camera and a display control apparatus; -
FIG. 4 is a schematic diagram illustrating a hardware configuration of the spherical camera, according to an embodiment; -
FIG. 5 is a schematic diagram illustrating a hardware configuration of the display control apparatus, according to an embodiment; -
FIG. 6 is a conceptual diagram illustrating example use of the spherical camera; -
FIGS. 7A to 7C are diagrams schematically illustrating example operation of generating a spherical image from images captured by the spherical camera; -
FIGS. 8A and 8B are diagrams schematically illustrating example operation of generating a spherical image from images captured by the spherical camera; -
FIGS. 9A and 9B are example diagrams illustrating placement of a spherical image on a sphere in equirectangular projection; -
FIG. 10 is an example diagram illustrating the line of sight of a user; -
FIGS. 11A to 11D are example diagrams schematically illustrating enlargement and reduction of a view area in a spherical image; -
FIG. 12 is a schematic block diagram illustrating a functional configuration of a digital camera, the spherical camera, and the display control apparatus, according to an embodiment; -
FIGS. 13A to 13C are diagrams illustrating projection conversion of a planar image in a match area and an example of a location parameter for associating the planar image with the match area; -
FIGS. 14A and 14B are example diagrams illustrating a projection-converted image and mask data generated by a projection converter; -
FIG. 15 is an example diagram schematically illustrating superimposition of a planar image on a spherical image; -
FIG. 16 is a diagram schematically illustrating a planar image superimposed on a spherical image; -
FIG. 17 is a flowchart illustrating operation of superimposing a planar image on a spherical image, performed by the display control apparatus, according to an embodiment; -
FIG. 18 is a schematic block diagram illustrating a functional configuration of an to image superimposing unit, according to an embodiment; -
FIGS. 19A to 19D are example diagrams illustrating the relative position of a planar image relative to a view area; -
FIG. 20 is an example diagram illustrating a. way of calculating an angle of view a and an angle of view b; isFIG. 21 is an example diagram illustrating relationships between the vertexes of a view area and the diagonally opposite vertexes of a planar image; -
FIGS. 22A to 22E are diagrams illustrating an example planar image that starts being replayed when a user enlarges a spherical image; -
FIG. 23 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment; -
FIG. 24 is a diagram illustrating an example menu for a user to control replay of a moving image; -
FIG. 25 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment; -
FIG. 26 is an example diagram illustrating a process for automatic enlargement of a planar image in a view area; -
FIG. 27 is an example diagram illustrating a process for automatic enlargement of a planar image in a case where the entire planar image is not included in a view area; -
FIG. 28 is a flowchart illustrating operation of controlling replay of a moving image, performed by the display control apparatus, according to an embodiment; -
FIGS. 29A to 29C are example diagrams illustrating metadata for replay of a moving image; -
FIG. 30 is an example diagram schematically illustrating an outline obtained as a result of projection conversion; -
FIG. 31 is an example diagram schematically illustrating superimposition of an outline; -
FIG. 32 is a diagram illustrating examples of an outline displayed in different forms for a moving image and a still image; and -
FIG. 33 is a flowchart illustrating operation of displaying an outline of a planar image on a spherical image, performed by the display control apparatus, according to an embodiment. - The accompanying drawings are intended to depict embodiments of the present invention and should not be interpreted to limit the scope thereof. The accompanying to drawings are not to be considered as drawn to scale unless explicitly noted.
- The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.
- In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.
- Hereinafter, embodiments of the present invention will be described with reference to the drawings.
- In this disclosure, a first image is an image superimposed with a second image, and a second image is an image to be superimposed on the first image. For example, the first image is an image covering an area larger than that of the second image. In another example, the first image and the second image are images expressed in different projections. In another example, the second image is an image with image quality higher than that of the first image, for example, in terms of image resolution. However, the definition of “high image quality” differs depending on the image, and therefore, may vary depending on the user's purpose of viewing. In general, “high image quality” means an image faithfully representing a scene. For example, “high image quality” means an image having a high resolution, a wide dynamic range, high color reproducibility, or less noise. Examples of the first image include a spherical image, an equirectangular projection image, and a low-definition image. Examples of the second image include a planar image, a perspective projection image, and a high-definition image.
- Further, in this disclosure, the spherical image does not have to be the full-view spherical image. For example, the spherical image may be the wide-angle view image having an angle of about 180 to 360 degrees in the horizontal direction. As described below, it is desirable that the spherical image is image data having at least a part that is not entirely displayed in the predetermined area T.
-
FIG. 1A illustrates a spherical image CE mapped on a virtual, three-dimensional (3D) sphere CS.FIG. 1B illustrates the equirectangular projection image CE. The spherical image CE generated by a spherical camera has a 3D form that is obtained by mapping the to equirectangular projection image CE as illustrated inFIG. 1B on the sphere CS illustrated inFIG. 1A . The virtual camera IC corresponds to the point of view of a user. InFIG. 1A , the point of view is present at the center of the spherical image CE. When the user makes the virtual camera IC rotate about three axes, namely, an X axis, a Y axis, and a Z axis, that pass through the virtual camera IC, the user can display a predetermined area T, such as a view area T (an area on a display or an area displayed by display software), in the spherical image CE as desired. Further, the user can enlarge or reduce the view area T. In this disclosure, the predetermined area T is referred to as the view area T. -
FIGS. 2A to 2C are example diagrams illustrating a display method for displaying a moving image superimposed on a spherical image. On the spherical image CE illustrated inFIG. 2A , a planar image P that is a moving image is superimposed. A user enlarges the spherical image CE in order to enlarge and view the planar image P as illustrated inFIG. 2A toFIG. 2B . The user may or may not be aware of the moving image. In order to make the user be aware of the moving image, the display control apparatus may indicate the presence of the planar image P (for example, display an outline described below). In this disclosure, the moving image means an active image obtained by successively displaying, at short intervals, a series of images (frames) captured at substantially regular intervals. - The user gradually enlarges the spherical image CE as illustrate in
FIG. 2A toFIG. 2B . The planar image P illustrated inFIG. 2A corresponds to the entire planar image P illustrated inFIGS. 2B and 2C . As illustrated inFIG. 2B , when the user enlarges the spherical image CE to some extent, it is inferred that the user wants to view the planar image P, and the display control apparatus starts replaying the moving image. InFIG. 2C , the moving image in which amonorail train 351 is moving leftward from the right is displayed. - As described above, when it is inferred that the user views the planar image P, the moving image is replayed. Therefore, the moving image can be replayed in a case where the user wants to view the planar image P, and unwanted replay of the moving image can be suppressed. Further, the user can view the moving image from the beginning. If the moving image is replayed at all times, when the user starts viewing the moving image, the user is unable to know the time elapsed since the start of replay. Further, when the user enlarges the planar image P, replay of the moving image starts. Therefore, motion seems to start suddenly on the screen, which can attract the user's attention. In this disclosure, it is assumed that the second image is being viewed by the user, when there is an event that can be determined that the user is consciously looking at the second image.
-
FIGS. 3A to 3C are diagrams illustrating example system configurations of animage processing system 100 including aspherical camera 20 and adisplay control apparatus 30.FIG. 3A illustrates theimage processing system 100 that includes thespherical camera 20 and a general-purpose (generic)digital camera 9. The genericdigital camera 9 is typically an is image capturing device (camera) that captures images with a focal length of 35 mm or more on a 35-mm-film basis and may be an image capturing device capable of capturing wide-angle images corresponding to about 24 mm to 35 mm. Simply speaking, thedigital camera 9 is not thespherical camera 20 but an image capturing device that captures the planar image P. In other words, thedigital camera 9 is an image capturing device capturing an image that can be entirely displayed in the view area T. - The
digital camera 9 includes an imaging element having at least a resolution higher than that of the spherical image CE (having a large number of pixels relative to the angle of view for image capturing). Further, in thedigital camera 9, image capturing conditions (for example, the exposure time, the shutter speed, and the white balance) are optimized for an image capturing range narrower than that of thespherical camera 20, and therefore, it is often the case that whiteout is less likely to occur, the color reproducibility is high, and less noise is produced. As described above, thedigital camera 9 generates planar images of high image quality. - Accordingly, the spherical image CE, which is an image blurred when enlarged, can be complemented by the planar image P, which is an image of high image quality. In the configuration illustrated in
FIG. 3A , thedigital camera 9 also functions as thedisplay control apparatus 30. Specific examples of thedigital camera 9 include a digital still camera and a digital video camera. Thedigital camera 9 may be replaced by another device that includes an image capturing device. For example, thedigital camera 9 may be replaced by a smartphone, a tablet terminal, a personal digital assistant (PDA), or a wearable personal computer (PC). In this case, thespherical camera 20 may be externally attached to, for example, the smartphone. - The
spherical camera 20 and thedigital camera 9 can communicate with each other using, for example, Universal Serial Bus (USB), Bluetooth (registered trademark), or a wireless local area network (LAN). The spherical image CE captured by thespherical camera 20 is transmitted to thedigital camera 9. Similarly, thedigital camera 9 captures the planar image P. In the spherical image CE, a 360-degree surrounding view is present, and therefore, the planar image P corresponds to a part of the spherical image CE. - The
digital camera 9 functions as thedisplay control apparatus 30 and superimposes the planar image P on the spherical image CE. Specifically, for example, a user U selects the planar image P and the spherical image CE and instructs thedigital camera 9 to perform superimposition. Alternatively, the user U sets thedigital camera 9 to a superimposition mode and captures the planar image P. Then, thedigital camera 9 automatically superimposes the planar image P on the spherical image CE that matches the planar image P. If the degree of image matching is low, thedigital camera 9 does not perform superimposition, which causes less inconvenience. Further, thedigital camera 9 that functions as thedisplay control apparatus 30 displays a moving image on the display of thedigital camera 9 while switching between replay and stop of the moving image. As described above, in the case where thedigital camera 9 functions as thedisplay control apparatus 30, in the configuration constituted by thespherical camera 20 and thedigital camera 9, the planar image P can be superimposed on the spherical image CE, and switching between replay and stop of the moving image can be performed. -
FIG. 3B illustrates theimage processing system 100 that includes thespherical camera 20, the genericdigital camera 9, and thedisplay control apparatus 30. Thespherical camera 20 and thedisplay control apparatus 30 can communicate with each other using, for example, USB, Bluetooth (registered trademark), or a wireless LAN, and thedigital camera 9 and thedisplay control apparatus 30 can communicate with each other using, for example, USB. Bluetooth (registered trademark), or a wireless LAN. One or more spherical images CE captured by thespherical camera 20 owe transmitted to thedisplay control apparatus 30. One or more planar images P captured by thedigital camera 9 are also transmitted to thedisplay control apparatus 30. Instead of transmission, the spherical images CE or the planar images P stored in a storage medium may be read by the display control apparatus 3C) from the storage medium. - The
display control apparatus 30 is an information processing apparatus, such as a PC, a smartphone, a tablet terminal, or a PDA. Alternatively, thedisplay control apparatus 30 may be a multifunction peripheral, a projector, or a video conference terminal having functions of an information processing apparatus. - In the configuration illustrated in
FIG. 3B , thedisplay control apparatus 30 superimposes at least one of the planar images P on one of the spherical images CE. Thedisplay control apparatus 30 displays a moving image on the display of thedisplay control apparatus 30 while switching between replay and stop of the moving image. More specifically, in the case of superimposing one planar image P on the spherical image CE, the to user U compares the planar images P and the spherical images CE and knows which planar image P constitutes part of which spherical image CE. Then, the user U selects one of the spherical images CE and the planar image P that is to be placed on the spherical image CE, and instructs thedisplay control apparatus 30 to perform placement. Similarly to thedigital camera 9, thedisplay control apparatus 30 may automatically perform superimposition. As described above, thedisplay control apparatus 30 can obtain the spherical image CE and the planar image P, superimpose the planar image P on the spherical image CE, and switch between replay and stop of the moving image. The description ofFIG. 4 and subsequent figures is given while assuming the configuration illustrated inFIG. 3B unless otherwise specified. -
FIG. 3C illustrates theimage processing system 100 in which thespherical camera 20, the genericdigital camera 9, thedisplay control apparatus 30, and animage processing server 70 are connected to one another via a network N. The network N is formed of, for example, a LAN formed in the user U's environment, a provider network of a provider that connects the LAN to the Internet, and a line provided by a carrier. In a case where the network N includes a plurality of LANs, the network N is called a wide area network (WAN) or the Internet. The network N may be formed of wired or wireless networks or may be formed of a combination of wired and wireless networks. In a case where thespherical camera 20 and thedigital camera 9 are directly connected to a public line network, thespherical camera 20 and thedigital camera 9 can be connected to the provider network without using the LAN. - The
image processing server 70 may be a general information processing apparatus or a server apparatus. Specific examples of thedisplay control apparatus 30 illustrated inFIG. 3C may be similar to the specific examples of thedisplay control apparatus 30 illustrated inFIG. 3B . It is preferable that theimage processing server 70 support cloud computing. Cloud computing is a use form in which a user uses resources in a network without awareness of a specific hardware resource. - One or more spherical images CE captured by the
spherical camera 20 are transmitted to theimage processing server 70. One or more planar images P captured by thedigital camera 9 are also transmitted to theimage processing server 70. In the configuration illustrated inFIG. 3C , theimage processing server 70 superimposes at least one of the planar images P on one of the spherical images CE. More specifically, thedisplay control apparatus 30 accesses theimage processing server 70 in response to an operation by the user U and displays a list of the spherical images CE and a list of the planar images P (for example, thumbnail images). For example, in the case of superimposing one planar image P on the spherical image CE, the user U compares the planar images P and the spherical images CE and knows which planar image P constitutes part of which spherical image CE. Then, the user U selects one of the spherical images CE and the planar image P that is to be placed on the spherical image CE, and transmits the selected spherical image CE and planar image P to theimage processing server 70. Theimage processing server 70 may automatically perform superimposition. - The
image processing server 70 superimposes the planar image P on the spherical image CE and transmits image data of the view area T to thedisplay control apparatus 30. It is assumed that the line-of-sight direction and the angle of view in the initial state are determined in advance. Theimage processing server 70, which functions as a Web server, transmits image data of the view area T and a display program (described in a script language) for accepting an operation performed by the user U to thedisplay control apparatus 30. Thedisplay control apparatus 30 executes the display program to accept an operation performed by the user U and transmits the line-of-sight direction and the angle of view to theimage processing server 70. Theimage processing server 70 updates the view area. T in accordance with the line-of-sight direction and the angle of view and transmits image data of the view area T to thedisplay control apparatus 30. Theimage processing server 70 switches between replay and stop of a moving image, and therefore, thedisplay control apparatus 30 can display an image in which a moving image is replayed and an image in which a moving image is not replayed. - The
image processing server 70 may transmit the spherical image CE, the planar image P, and the display program to thedisplay control apparatus 30. In this case, thedisplay control apparatus 30 determines the view area T in accordance with an operation performed by the user U and switches between replay and stop of a moving image. - The
image processing server 70 may calculate a location parameter (described below) from the spherical image CE and the planar image P, and thedisplay control apparatus 30 may download the spherical image CE, the planar image P, and the location parameter. Although the user changes the line-of-sight direction as desired, thedisplay control apparatus 30 can superimpose the planar image P on the spherical image CE by using the same location parameter. In this case, theimage processing server 70 includes alocation parameter generator 8 illustrated inFIG. 12 described below, and thedisplay control apparatus 30 includes a conversion/display unit 7 illustrated inFIG. 12 . -
FIG. 4 is a hardware configuration diagram of thespherical camera 20. It is assumed hereinafter that thespherical camera 20 is an omnidirectional image capturing device including two imaging elements: however, the number of imaging elements may be three or more. Further, thespherical camera 20 need not be a device dedicated to omnidirectional is image capturing. For example, an omnidirectional image capturing unit may be attached to, for example, the genericdigital camera 9 or a smartphone to thereby implement substantially the same functions as those of thespherical camera 20. - As illustrated in
FIG. 4 , thespherical camera 20 includes animaging unit 101, animage processor 104, animaging controller 105, amicrophone 108, anaudio processor 109, a central processing unit (CPU) 111, a read-only memory (ROM) 112, a static random access memory (SRAM) 113, a dynamic random access memory (DRAM) 114, anoperation unit 115, a network interface (UF) 116, acommunication circuit 117, anantenna 117 a, anelectronic compass 118, agyro sensor 119, and anacceleration sensor 120. - Among these constituent elements, the
imaging unit 101 includes wide-angle lenses (fish-eye lenses) 102 a and 102 b each having an angle of view of 180° or more so as to form a hemispherical image and twoimaging elements angle lenses 102 a and 102 h respectively. Theimaging elements - The
imaging elements imaging unit 101 are each connected to theimage processor 104 via a parallel I/F bus. Further, theimaging elements imaging unit 101 are each connected to theimaging controller 105 via a serial I/F bus (for example, an I2C bus). Theimage processor 104 and theimaging controller 105 are connected to theCPU 111 via abus 110. Furthermore, for example, theROM 112, theSRAM 113, theDRAM 114, theoperation unit 115, the network OF 116, thecommunication circuit 117, and theelectronic compass 118 are connected to thebus 110. - The
image processor 104 acquires pieces of image data output from therespective imaging elements FIG. 1B . - In general, the
imaging controller 105 functions as a master device, and theimaging elements imaging controller 105 uses the 12C buses to set commands and so on in the group of registers of each of theimaging elements CPU 111. Furthermore, theimaging controller 105 also uses the I2C buses to acquire status data and so on of the group of registers of each of theimaging elements CPU 111. - The
imaging controller 105 instructs theimaging elements operation unit 115 is pressed. Thespherical camera 20 may have a preview display function using a display or a function of enabling moving image display. In the case of moving image display, pieces of image data are output from theimaging elements - The
imaging controller 105 also functions as a synchronization controller that cooperates with theCPU 111 to synchronize the timings at which piece of image data are output from theimaging elements spherical camera 20 does not include a display in this embodiment, thespherical camera 20 may include a display. - The
microphone 108 converts sounds to audio data (signal). Theaudio processor 109 acquires the audio data output from themicrophone 108 via an I/F bus and performs a predetermined process for the audio data. - The
CPU 111 controls the entire operation of thespherical camera 20 and performs processing as appropriate. TheROM 112 stores various programs for theCPU 111. TheSRAM 113 and theDRAM 114 are work memories and store, for example, programs to be executed by theCPU 111 and data that is being processed. Specifically, theDRAM 114 stores image data that is being processed by theimage processor 104 and data of an image in equirectangular projection that has been processed. - The
operation unit 115 generally refers to various operation buttons, a power switch, the shutter button, and a touch panel having both display and operation functions. The user U operates the operation buttons to input various image capturing modes and image capturing conditions. - The network I/
F 116 generally refers to interface circuits (including a USB I/F) providing interfaces with, for example, an external medium, such as an SD card, and a PC. The network I/F 116 may function as a wireless or a wired network interface. Data of an image in equirectangular projection stored in theDRAM 114 is recorded to an external medium via the network I/F 116 or transmitted to an external device, such as thedisplay control apparatus 30, via the network I/F 116, which functions as a network interface as appropriate. - The
communication circuit 117 communicates with an external device, such as thedisplay control apparatus 30, via theantenna 117 a included in thespherical camera 20 by using a short-range wireless technology, such as Wireless Fidelity (Wi-Fi) or the near-field communication (NFC). Data of an image in equirectangular projection can be transmitted to an external device, such as thedisplay control apparatus 30, also via thecommunication circuit 117. - The
electronic compass 118 calculates the azimuth of thespherical camera 20 from the magnetism of the earth and outputs azimuth information. The azimuth information is an example of related information (metadata) compliant with Exchangeable image file format (Exif) and used in image processing, such as image correction of a captured image. The related information also includes other data, such as the date and time when the image is captured and the size of the image data. - The
gyro sensor 119 detects changes in angles (roll, pitch, and yaw angles) associated with a tilt of thespherical camera 20. The changes in the angles are examples of the related data (metadata) compliant with Exif and used in image processing, such as image correction of a captured image. - The
acceleration sensor 120 detects accelerations in the directions of three axes. On the basis of the detected accelerations, the attitude of the spherical camera 20 (the angle relative to the gravity direction) is detected. Both thegyro sensor 119 and theacceleration sensor 120 are provided to thereby increase the accuracy of image correction. - It is assumed that the hardware configuration of the
digital camera 9 is similar to that of thespherical camera 20 illustrated inFIG. 4 . However, the imaging unit may be different from that of thedigital camera 9. The imaging unit of thedigital camera 9 may include a lens unit having a plurality of lenses, a mechanical shutter button, and an imaging element. -
FIG. 5 is a hardware configuration diagram of thedisplay control apparatus 30. Thedisplay control apparatus 30 includes aCPU 501, which controls the entire operation of thedisplay control apparatus 30, aROM 502, which stores programs including an initial program loader (IPL) that are used to drive theCPU 501, aRAM 503, which is used as a work area of theCPU 501, and a hard disk (HD) 504, which stores various types of data including a display control apparatus program. Thedisplay control apparatus 30 further includes a hard disk drive (HDD) 505, which controls reading/writing of various types of data from/to theHD 504 in accordance with control by theCPU 501, and a medium drive (media drive) 507, which controls reading/writing (storing) of data from/to (in) arecording medium 506, which is, for example, a flash memory. Thedisplay control apparatus 30 further includes adisplay 508, which displays various types of information, such as cursors, menus, windows, text, and images, a network I/F 509, which is used to perform data communication using the network N, akeyboard 511, which includes a plurality of keys for inputting characters, numbers, and various instructions, and amouse 512, which is used to, for example, select and execute various instructions, select a processing target, and move a cursor. Thedisplay control apparatus 30 further includes an optical medium drive (such as CD-RW drive) 514, which controls reading/writing of various types of data from/to anoptical recording medium 513, which is an example of an attachable/detachable recording medium, and abus line 510, which is constituted by, for example, an address bus and a data bus used to electrically connect the above-described constituent elements, as illustrated inFIG. 5 . - It is assumed that the hardware configuration of the
image processing server 70 is similar to that illustrated inFIG. 5 . - The spherical image CE is described with reference to
FIG. 6 toFIG. 10 .FIG. 6 is a conceptual diagram illustrating use of thespherical camera 20. As illustrated inFIG. 6 , the user U holds thespherical camera 20 in their hand and uses thespherical camera 20 to capture images of objects around the user U. Thespherical camera 20 includes the two imaging elements that are arranged such that the back surfaces thereof face each other. Each of the two imaging elements captures an image of objects around the user U to thereby obtain two hemispherical images. - Now, the overview of processing up to generation of the spherical image CE from images captured by the
spherical camera 20 is described with reference toFIG. 7A toFIG. 99 .FIG. 7A is a diagram illustrating a hemispherical image (front side) captured by thespherical camera 20,FIG. 7B is a diagram illustrating a hemispherical image (back side) captured by thespherical camera 20, andFIG. 7C is a diagram illustrating a spherical image in equirectangular projection.FIG. 8A is a diagram schematically illustrating a state where a sphere is covered by an image in equirectangular projection, andFIG. 8B is a diagram illustrating the spherical image CE. - As illustrated in
FIG. 7A , an image captured by thespherical camera 20 is a hemispherical image (front side) that curves due to the fish-eye lens. As illustrated inFIG. 7B , an image captured by thespherical camera 20 is a hemispherical image (back side) that curves due to the fish-eye lens. The hemispherical image (front side) and the hemispherical image (back side), which is flipped by 180 degrees, are combined by thespherical camera 20, and an image in equirectangular projection is generated accordingly, as illustrated inFIG. 7C . - Then, the image in equirectangular projection is placed so as to cover the spherical surface, as illustrated in
FIG. 8A , by using Open Graphics Library for Embedded Systems (OpenGL ES), and the spherical image CE as illustrated inFIG. 8B is generated. Accordingly, the spherical image CE is represented as an image in equirectangular projection that faces the center of the sphere. Note that OpenGL ES is a graphics library that is used to visualize two-dimensional (2D) and 3D data. The spherical image CE may be a still image or a moving image. Specifically, the image in equirectangular projection illustrated inFIG. 8A is converted to the 3D spherical image illustrated inFIG. 8B by using expressions (1) described with reference toFIGS. 9A and 9B . -
FIG. 9A is a diagram illustrating the spherical image CE in equirectangular projection, andFIG. 9B illustrates the spherical image CE that is obtained by placing the spherical image CE in equirectangular projection on a full sphere. Conversion from the image illustrated inFIG. 9A to the image illustrated inFIG. 9B is a process that is performed by thespherical camera 20 illustrated inFIG. 1 at the time when thespherical camera 20 performs image capturing. - The coordinates of a point (x, y, z) in a 3D space corresponding to a point (λ, φ) in equirectangular projection can be calculated by using the following expression. Here, the radius of the sphere is equal to 1.0.
-
x=cos(φ)*cos(λ) -
y=sin(φ) -
z=cos(φ)*sin(λ) . . . (1) - Accordingly, the spherical image CE that is a 3D image is obtained from the spherical image CE in equirectangular projection. The spherical image CE is an image placed so as to cover the spherical surface and is curved, Accordingly, the spherical image
- CE may feel unnatural to a person who views the image. The
display control apparatus 30 displays the planar image P that curves to a small degree in the view area T, which is part of the spherical image CE, so that the image does not feel unnatural to the person. The view area T is represented by coordinates (x, v, z) in a 3D virtual space. However, thedisplay 508 is a 2D plane, and therefore, thedisplay control apparatus 30 is unable to display the view area T as is, - Accordingly, the
display control apparatus 30 performs perspective projection conversion in which a 3D object is projected onto a 2D plane by using a 3D computer graphics technique to thereby obtain an image that curves to a smaller degree. As a is consequence, the view area T of the spherical image CE is displayed on thedisplay 508. -
FIG. 10 is an example diagram illustrating the line of sight of the user U. The spherical image CE has a 3D coordinate system, and therefore, the line-of-sight direction is identified by using information, such as 3D coordinates or a latitude and longitude, for identifying a position on the sphere. In this embodiment, the center cp of the view area T is assumed to correspond to the line-of-sight direction. - The user U can change the line-of-sight direction using the
keyboard 511 or themouse 512. If it is assumed that the virtual camera IC does not allow translation, the virtual camera IC allows three types of rotation, namely, roll (rotation about the Z axis), yaw (rotation about the Y axis), and pitch (rotation about the X axis). When any of the three types of rotation occurs, the line-of-sight direction changes. For example, when the user U rotates the spherical image CE in the horizontal direction, the yaw angle changes. When the user U rotates the spherical image CE in the vertical direction, the pitch angle changes. When the user U rotates the spherical image CE about an axis that passes through the center of thedisplay 508, the roll angle changes. In this embodiment, an operation performed by the user U on a Web page is reflected to, for example, the line-of-sight direction (the roll angle, the yaw angle, and the pitch angle). The way of reflection is described in advance in a program that is executed by thedisplay control apparatus 30. -
FIGS. 11A to 11D are example diagrams schematically illustrating enlargement and reduction of the view area T in the spherical image CE.FIG. 11A illustrates the view area in the initial state. In a case where the virtual camera IC is present at the center of the sphere CS, the view area T is determined in accordance with the angle of view α. The angle of view α is an angle that is determined when, for example, diagonally opposite vertexes of the view area T are viewed from the center of the sphere CS. - The angle of view α in the initial state is represented by α0. As illustrated in
FIG. 11B , as the angle of view decreases (α1≤α0), the view area T becomes narrower, and the image is enlarged on thedisplay 508 accordingly. As illustrated inFIG. 11C , as the angle of view increases (α2>α0), the view area T becomes wider, and the image is reduced on thedisplay 508 accordingly. When the user U performs an operation for enlarging or reducing to the spherical image CE, thedisplay control apparatus 30 decreases or increases the angle of view in accordance with the amount of operation. - Even if the angle of view increases, as illustrated in
FIG. 11C , thedisplay control apparatus 30 is unable to display an image behind the virtual camera IC as long as the virtual camera IC is present at the center. Accordingly, when the user U performs an operation for is further reduction in the state illustrated inFIG. 11C , thedisplay control apparatus 30 moves the virtual camera IC backward. InFIG. 11D , the angle of view α2 remains the same as that inFIG. 11C ; however, the virtual camera IC moves backward, and the view area T further becomes wider accordingly. As a consequence, the image can be reduced further than that inFIG. 11C . - Now, functions related to superimposition of the planar image P on the spherical image CE are described with reference to
FIG. 12 toFIG. 17 . As described above, the following description is given on the basis of the configuration illustrated inFIG. 3B . -
FIG. 12 is an example functional block diagram illustrating functions of thedigital camera 9, thespherical camera 20, and thedisplay control apparatus 30 as blocks. Thedigital camera 9 includes aplanar image obtainer 11. Theplanar image obtainer 11 is implemented as an image capturing function of thedigital camera 9 and generates image data (planar image P) in which each of the pixels according to a predetermined resolution represents gradation. The planar image P may include a single still image, a plurality of still images, or a moving image. In this embodiment, it is assumed that the planar image P is a moving image unless otherwise specified. - The
spherical camera 20 includes aspherical image obtainer 12. Thespherical image obtainer 12 is implemented as an image capturing function of thespherical camera 20 and generates image data (spherical image CE) of a 360-degree view around thespherical camera 20. The spherical image CE may include a single still image, a plurality of still images, or a moving image. In this embodiment, it is assumed that the spherical image CE is a still image unless otherwise specified. - The
display control apparatus 30 mainly includes thelocation parameter generator 8 and the conversion/display unit 7. The functions of thelocation parameter generator 8 and the conversion/display unit 7 included in thedisplay control apparatus 30 are functions or units implemented by theCPU 501 of thedisplay control apparatus 30 illustrated inFIG. 5 executing a program loaded to theRAM 503 from theHD 504. - The
location parameter generator 8 includes aspherical image reader 21, aplanar image reader 22, alocation parameter calculator 23, and alocation parameter writer 24. Thespherical image reader 21 reads the spherical image CE from thespherical image obtainer 12. Theplanar image reader 22 reads the planar image P from theplanar image obtainer 11. The term “read” means to obtain, receive, read from a storage medium, or accept input. - The
location parameter calculator 23 identifies a match area in the spherical image CE that matches the planar image P, and calculates a location parameter PP for identifying the match area. The details of the location parameter PP are described with reference toFIGS. 13A to 13C . The spherical image CE is distorted due to equirectangular projection, and therefore, it is preferable to perform similar conversion for the planar image P to make the planar image P distorted. Then, features points in both the spherical image CE and the planar image P are extracted and matched to thereby obtain the location parameter PP. For detection of feature points, various methods using, for example, edge detection, corner detection, Scale-Invariant Feature Transform (SIFT) feature values or Speeded-Up Robust Features (SURF) feature values, or a center point of the same continuous color are available. Alternatively, the sum of the absolute values of the differences between pixel values of the spherical image CE and the pixel values of the planar image P or the square sum of the differences is calculated while the target pixels are shifted by one pixel each time the calculation is performed, and the position of the planar image P when the sum of the absolute values or the square sum is minimized may be regarded as the match area. - In this embodiment, the spherical image CE is a still image, and the planar image P is a moving image, and therefore, only a frame (a still image included in the moving image) at a specific time matches the spherical image CE. The
location parameter calculator 23 determines whether each of the frames of the planar image P matches the spherical image CE sequentially from the first frame. Then, a frame for which the matching degree is equal to or larger than a threshold is used to calculate the location parameter PP. Alternatively, the user U may communicate the matching frame to thedisplay control apparatus 30. Thelocation parameter writer 24 transmits the frame identification (ID) 6 of the frame that matches the spherical image CE to the conversion/display unit 7. Note that the term “ID” means an identifier or identification information. The ID is a name, a code, a character string, a numerical value, or any combination thereof used to uniquely distinguish a specific target from the others. The same applies to any ID other than the frame ID. - When the planar image P is divided in to mesh-like areas and it is determined whether each of the areas matches the spherical image CE, a shift due to, for example, a distortion in a lens can be additionally corrected.
- The correspondence between the spherical image CE and the planar image P can be uniquely determined, and therefore, the location parameter PP needs to be calculated once. In a case where the
planar image reader 22 reads a plurality of planar images P, thelocation parameter calculator 23 calculates the location parameter PP for each of the planar images P. - The conversion/
display unit 7 includes aplanar image reader 25, a line-of-sight direction/angle-of-view specifier 26, aspherical image reader 27, afirst projection converter 28, asecond projection converter 29, animage superimposing unit 33, alocation parameter reader 32, and animage display unit 31. The functions of thespherical image reader 27 are the same as the functions of thespherical image reader 21 of thelocation parameter generator 8. Thelocation parameter reader 32 obtains the location parameter PP and the frame ID from thelocation parameter generator 8. - The
planar image reader 25 reads each frame of the planar image P. Theplanar image reader 25 reads a frame having the frame ID specified by thelocation parameter calculator 23 until theimage superimposing unit 33 described below sends a notification indicating that moving image reading is ON to theplanar image reader 25. After theimage superimposing unit 33 has communicated moving image reading ON to theplanar image reader 25, theplanar image reader 25 reads chronologically subsequent frames (or succeeding frames in a case of reverse replay) in accordance with the replay speed of the moving image. In a case where theimage superimposing unit 33 sends a notification indicating that moving image reading is OFF to theplanar image reader 25, theplanar image reader 25 repeatedly reads a frame having the frame ID or the last read frame. - The line-of-sight direction/angle-of-
view specifier 26 accepts the line-of-sight direction and the angle of view (enlargement or reduction) controlled by the user U. On the basis of the line-of-sight direction and the angle of view that are controlled and input, the view area T is determined. - The user U controls the line-of-sight direction and the angle of view (enlargement or reduction) as desired. Therefore, a part of the planar image P subjected to projection conversion may fit in the view area T or the planar image P subjected to projection conversion might not fit in the view area T at all. Accordingly, the
first projection converter 28 generates mask data that indicates an area, in the projection-converted image, that is included in the view area T. The details are described with reference toFIGS. 14A and 14B . - The
second projection converter 29 determines the view area T that corresponds to the line-of-sight direction and the angle of view accepted by the line-of-sight direction/angle-of-view specifier 26, performs, on the basis of the display size of theimage display unit 31, to perspective projection conversion for the view area T of the spherical image CE that has been read, and generates a display image. Therefore, the display image is a 2D planar image. - The
image superimposing unit 33 superimposes the projection-converted image on the display image by using the mask data to generate a superimposed image. The details are described with reference toFIG. 15 . Theimage display unit 31 displays the superimposed image on thedisplay 508. -
FIGS. 13A to 13C are diagrams illustrating projection conversion of the planar image P in a match area and an example of the location parameter PP for associating the planar image P with the match area. It is assumed that the correspondence between the planar image P and the spherical image CE has been determined via matching. The location parameter PP includes the latitude and longitude of a position in the spherical image CE at which the planar image P is present when the feature points match to the highest degree. -
FIG. 13A illustrates the planar image P, andFIG. 13B illustrates the spherical image CE in equirectangular projection, InFIG. 13B , amatch area 301 is illustrated. Thelocation parameter calculator 23 divides the planar image P into grids, and obtains, for each intersection point of the grids represented by coordinates (x, y), a corresponding point (λ, φ) in thematch area 301 of the spherical image CE as the location parameter PP. Here, λ represents the longitude, and φ represents the latitude.FIG. 13C illustrates an example of the location parameter PP. The coordinates of each grid in the planar image P is associated with the latitude and longitude of a corresponding point in thematch area 301 of the spherical image CE. -
FIGS. 14A and 14B are example diagrams illustrating a projection-convertedimage 302 andmask data 303 generated by thefirst projection converter 28.FIG. 14A illustrates the projection-convertedimage 302. The projection-convertedimage 302 has an area having the same size as the size of the view area T in the spherical image CE in equirectangular projection. The planar image P after projection conversion is placed on thematch area 301. The planar image P is subjected to projection conversion and placed on thematch area 301, and therefore, the planar image P is in a state of trapezoidal distortion. In the projection-convertedimage 302, a portion not covered by the planar image P is a uniformgray image 307. Theuniform image 307 is not used in superimposition, and therefore, may have any pixel value. -
FIG. 14B illustrates themask data 303 that corresponds to the planar image P to illustrated inFIG. 14A . Themask data 303 is data used to take the planar image P included in the view area T from the projection-convertedimage 302 illustrated inFIG. 14A . The area of white pixels in themask data 303 corresponds to an area that is the area of the planar image P and the area of the view area I Therefore, the area of white pixels in themask data 303 is equal to or smaller than the area of the planar image P. Themask data 303 illustrated inFIG. 14B is mask data in a case where the view area view area T is larger than the planar image P. Therefore, in the example illustrated inFIG. 14A , thematch area 301 is substantially equal to the area of white pixels in themask data 303. In a case where the view area I and the planar image P do not overlap at all, the entire area of themask data 303 is constituted by black pixels. In a case where the view area T is smaller than the planar image P, themask data 303 is constituted by only white pixels. As described above, the area of white pixels in themask data 303 and thematch area 301 have the same size and are present at the same position. - The
image superimposing unit 33 performs a mask process using the projection-convertedimage 302 and themask data 303. The mask process is a process for taking a pixel corresponding to a white pixel in themask data 303 from the projection-convertedimage 302. Theimage superimposing unit 33 acquires a pixel value corresponding to the position of a white pixel from the projection-convertedimage 302 and superimposes the pixel value on the display image at the same position as the position in the projection-convertedimage 302. - It is preferable to provide, at the boundary between the black pixels and the white pixels of the
mask data 303, gradations in which pixels gradually change from the white pixels to the black pixels. Accordingly, the boundary between the spherical image CE having a low resolution and the planar image P having a high resolution can be less noticeable. - Processing performed by the
image superimposing unit 33 is described with reference toFIG. 15 .FIG. 15 is an example diagram schematically illustrating superimposition of the planar image P on the spherical image CE. As illustrated inFIG. 15 , adisplay image 304 obtained by thesecond projection converter 29 performing perspective projection conversion for the view area T. the projection-converted image 302 (FIG. 14A ), and the mask data 303 (FIG. 14B ) are input to theimage superimposing unit 33. - The
image superimposing unit 33 superimposes the projection-convertedimage 302, which is a wider-angle image, on thedisplay image 304. Theimage superimposing unit 33 to performs a mask process using themask data 303 to take the projection-convertedimage 302 corresponding to the pixel positions of white pixels in themask data 303 and overlay the taken image on thedisplay image 304. In this case, corresponding pixels in thedisplay image 304 are lost at this time. Accordingly, asuperimposed image 305 is generated in which the planar image P having a high resolution is placed on the spherical image CE having a low resolution. - Alternatively, a transparent layer may be used, and the projection-converted
image 302 corresponding to the pixel positions of white pixels in themask data 303 may be taken and placed on the transparent layer. In this case, no pixels are lost form thedisplay image 304 and, for example, the user U can switch between display and non-display of the planar image P. - The
image superimposing unit 33 outputs thesuperimposed image 305. In thesuperimposed image 305, the planar image P having a high resolution is placed on the spherical image CE having a low resolution. - In this embodiment, as described above, the
display control apparatus 30 performs perspective projection conversion of the spherical image CE and superimposition of the planar image P in parallel. If the planar image P is placed on the spherical image CE, and thereafter, perspective projection conversion is performed, the resolution of the spherical image CE needs to be the same as the resolution of the planar image P, resulting in an increase in the amount of data. In this embodiment, thedisplay image 304 is generated, and thereafter, the planar image P is superimposed, which can suppress an increase in the amount of data. - The
display control apparatus 30 repeatedly superimposes the planar image P on the spherical image CE in accordance with the display cycle of the display 50 l (for example, 30 to 60 times per second). Accordingly, thedisplay control apparatus 30 retains one planar image P and one spherical image CE and generates thesuperimposed image 305 in real time from the point of view according to an operation by the user U. Furthermore, thedisplay control apparatus 30 can replay a moving image. -
FIG. 16 is a diagram schematically illustrating the planar image P superimposed on the spherical image CE. As described above, the planar image P, which is a plane, is superimposed on thedisplay image 304 as is, and is not placed on the sphere CS as illustrated inFIG. 16 . However, the planar image P seems to be superimposed on part of the sphere having the same center position and the same radius as those of the spherical image CE when viewed by the user U, as illustrated inFIG. 16 . - As illustrated in
FIG. 16 , the planar image P may be superimposed on the spherical image CE that is a 3D image. The longitude and latitude corresponding to the planar image P are known via matching (using the location parameter PP), and therefore, thedisplay control apparatus 30 can place the planar image P on the spherical image CE by using expressions is (1). In this case, for example, hidden surface removal (Z-buffer algorithm) using depth information is not used, and rendering needs to be performed on a later-priority basis. - Superimposition of a plurality of planar images P can be similarly performed by the
location parameter calculator 23 calculating the location parameter PP for each of the planar images P. In this case, regarding the order of rendering, rendering of a wide-angle image is performed first, and rendering of a telephoto image is performed later. Accordingly, in a case where images are captured by thedigital camera 9 having the same number of pixels, the planar image P having a low resolution is not overlaid on the planar image P having a high resolution. -
FIG. 17 is an example flowchart illustrating a procedure for thedisplay control apparatus 30 to superimpose the planar image P on the spherical image CE. The process inFIG. 17 starts when, for example, a user gives an instruction for superimposition. Steps S10 to S30 are performed in parallel with steps S40 to S60: however, steps S10 to S60 may be performed sequentially. - The
planar image reader 22 reads the planar image P (step S10). Thespherical image reader 21 reads the spherical image CE (step S20). Thelocation parameter calculator 23 calculates the location parameter PP (step S30). - The
planar image reader 25 reads the planar image P (step S40). Thespherical image reader 27 reads the spherical image CE (step S50), The line-of-sight direction/angle-of-view specifier 26 accepts a line-of-sight direction and an angle of view (step S60). Step S60 is performed as appropriate, for example, at every predetermined time or according to detection of a user operation, - Subsequently, the
first projection converter 28 performs projection conversion of the planar image P using the location parameter PP to generate the projection-converted image 302 (step S70). - Subsequently, the
first projection converter 28 generates themask data 303 in accordance with the view area I that is determined on the basis of the line-of-sight direction and the angle of view and in accordance with the planar image P (match area) subjected to projection conversion (step S80). - Subsequently, the
second projection converter 29 performs perspective projection conversion of the spherical image CE in accordance with the view area T to generate the display image 304 (step S90). - Subsequently, the
image superimposing unit 33 performs superimposition of the is projection-convertedimage 302 and thedisplay image 304 using themask data 303 to generate the superimposed image 305 (step S100). - Subsequently, the
image display unit 31 displays the superimposed image 305 (step S110). Thedisplay control apparatus 30 repeats the process from step S60 to step 5110 inFIG. 17 . - Now, functions related to replay of the planar image P superimposed on the spherical image CE are described with reference to
FIG. 18 toFIG. 23 .FIG. 18 is an example functional block diagram illustrating functions of theimage superimposing unit 33 as blocks. Theimage superimposing unit 33 includes adeterminer 35 and animage switching unit 36. Thedeterminer 35 determines whether it is inferred that the user U views the planar image P superimposed on thesuperimposed image 305 and transmits the result of determination to theimage switching unit 36. Theimage switching unit 36 sends a notification indicating that moving image reading is ON or moving image reading is OFF to theplanar image reader 25 in accordance with the result of determination. - In a case where it is inferred that the user U views the planar image P superimposed on the
superimposed image 305 as described above, thedisplay control apparatus 30 starts replaying a moving image. Specific examples of when it is inferred that the user U views the planar image P include cases when any one of the following conditions (i) to (iii) is satisfied. In such case, a moving image is replayed. - (i) The angle of view of the planar image P in the current view area T relative to the angle of view of the view area T is equal to or larger than a predetermined value.
- (ii) The area of the planar image P has an angle of view equal to or larger than a predetermined value.
- (iii) The user U clicks the planar image P.
- Note that the case where it is inferred that the user U views the planar image P includes a case where the user U is actually viewing the planar image P and a case where the user U is likely to view the planar image P. The condition (iii) may be referred to as replay of a moving image in response to an operation by the user U for viewing.
- First, the method for determination in “(i) the case where the angle of view of the planar image P in the current view area T relative to the angle of view of the view area T is equal to or larger than a predetermined value” is described.
-
FIGS. 19A to 19D ) are example diagrams illustrating the relative position of the planar image P relative to the view area T. InFIG. 19A , the entire planar image P is included in the view area T. The view area T is an area in the spherical image CE that is displayed on thedisplay 508, and therefore, the user U views the planar image P from substantially the front. When the angle of view of the view area T is represented by a, and the angle of view of the planar image P is represented by b, if angle of view a> angle of view b is satisfied, the entire planar image P and part of the spherical image CE are visible to the user U. In the state as illustrated inFIG. 19A , it is inferred that the user U wants to view the planar image P. - In a case where the spherical image CE is larger than the planar image P to some extent, the
determiner 35 need not determine that the user U wants to view the planar image P. Here, thedeterminer 35 determines, for example, whether the angle of view a is 1.2 times the angle of view b or less. That is, in a case where the angle of view a is 1.2 times the angle of view b or less, thedeterminer 35 determines that the ratio of the angle of view of the planar image P relative to the current angle of view is equal to or larger than a predetermined value. Then, theimage switching unit 36 transmits moving image replay request ON to theplanar image reader 25. In a case where thedeterminer 35 determines that the angle of view a is more than 1.2 times the angle of view b, theimage switching unit 36 transmits moving image replay request OFF to theplanar image reader 25. - The view area T is known. The latitude and longitude of the planar image P are registered as the location parameter PP, and therefore, the angle of view b is calculated from the location parameter PP. The details are described with reference to
FIG. 20 andFIG. 21 . - In
FIG. 19B , the view area is narrower than the planar image P. That is, the user U views only the planar image P. In this case, the angle of view a is 1.2 times the angle of view b or less, and therefore, a moving image is replayed, as a matter of course. - In
FIG. 19C , only an upper portion of the planar image P is included in the view area T. That is, the user U views an upper portion of the planar image P. However, the view area T is fully included in the planar image P (the angle of view a is 1.2 times the angle of view b or less), and therefore, theimage switching unit 36 transmits moving image replay request ON. - In
FIG. 19D , the shift (angle) between the center of the view area T and the center of in the planar image P is large, and part of the view area T lies outside the planar image P. In this case, the planar image P in the view area T is small relative to the view area T (the angle of view a is more than 1.2 times the angle of view b), and therefore, theimage switching unit 36 transmits moving image replay request OFF. - The threshold used to determine whether a moving image is replayed is set to, for example, 1.2 and may be set to any value larger than 1.0. For example, the threshold may be set to a value from 1.1 to 1.5. The user U may set the threshold.
-
FIG. 20 is an example diagram illustrating a way of calculating the angle of view a and the angle of view b. The angle of view a is calculated from the view area T, which is determined by the user U performing an operation. When the distance from the virtual camera IC to the view area T is represented by f, and the length of a diagonal line of the view area T is represented by 2L, the angle of view α, the distance f, and the length L has a relationship expressed by expression (2). -
L/f=tan(α/2) . . . (2) - The coordinates of diagonally opposite vertexes of the view area T are known from the operation performed by the user U, and therefore, the length of the
diagonal line 2L can be easily calculated. The distance f is known from the radius of the full sphere or the operation performed by the user U. Therefore, the angle of view a can be calculated by using expression (2). - The angle of view b can be similarly calculated. The angle of view b is calculated in a case where at least part of the planar image P is included in the view area T.
FIG. 21 is a diagram illustrating a relationship between the view area T and the planar image P. Thedeterminer 35 determines whether the planar image P is included in the view area T based on each of the four vertexes. The four vertexes of the view area T are represented by A1 to A4, and the four vertexes of the planar image P are represented by B1 to B4. The vertexes B1 to B4 are identified on the basis of, for example, the longitudes and latitudes of corresponding points on the full sphere or may be identified on the basis of the coordinates of corresponding points on thedisplay 508. - For example, the vertex B1 is determined as follows.
- Longitude of vertex A1≤Longitude of vertex B1≤Longitude of vertex d
- Latitude of vertex A1≥Latitude of vertex B1≥Latitude of vertex A4
- The vertexes B2, B3, and B4 are similarly determined. If diagonally opposite vertexes (B1 and B3, and 132 and B4) are included the view area T, the angle of view b can be calculated from the location parameter PP by using expression (2). If not all of the vertexes to B1 to B4 of the planar image P are included in the view area T, as illustrated in
FIG. 21 , the angle of view h is calculated from the vertex B1 included in the view area T and the vertex A3 of the view area T. Thedisplay control apparatus 30 includes settings indicating that a diagonally opposite vertex corresponding to the vertex B1 is the vertex A3, a diagonally opposite vertex corresponding to the vertex B2 is the vertex A4, a diagonally opposite vertex corresponding to the vertex B3 is the vertex A1, and a diagonally opposite vertex corresponding to the vertex B4 is the vertex A2. -
FIGS. 22A to 22E are diagrams illustrating an example of the planar image P that starts being replayed when the user U enlarges the spherical image CE. InFIG. 22A , the planar image P that is small is included in the view area T. In the state illustrated inFIG. 22A , the angle of view a is more than 1.2 times the angle of view b, and therefore, a moving image is not replayed. - In
FIG. 22B , the angle of view a is 1.2 times the angle of view b or less, and therefore, replay of a moving image starts. InFIG. 22C toFIG. 22E , the spherical image CE is not enlarged, and therefore, the composition (angle of view) remains the same. However, the moving image is replayed, and therefore, themonorail train 351 gradually moves from the right to the left. - That is, when the user U enlarges the planar image P such that the ratio of the planar image P to the spherical image CE is equal to or larger than a threshold, replay of a moving image automatically starts, and the user U can easily know the situation from the moving image (can enjoy the active image). Further, when the user U enlarges the planar image P, replay of a moving image starts. Therefore, motion seems to start suddenly on the screen, which can attract the user U's attention.
-
FIG. 23 is an example flowchart illustrating a procedure for thedisplay control apparatus 30 to control whether to replay a moving image. The procedure illustrated inFIG. 23 starts when thedisplay control apparatus 30 displays the spherical image CE. It is assumed that the planar image P has been superimposed on the spherical image CE. - The
determiner 35 determines whether the view area T is changed by the user U or a mouse operation is performed by the user U (step S210). The user U changing the view area T means the user U changing the line-of-sight direction or performing an enlarging or reducing operation. Specifically, the line-of-sight direction/angle-of-view specifier 26 converts an operation performed by the user U to a line-of-sight direction and an angle of view and transmits the line-of-sight direction and the angle of view to theimage superimposing unit 33. Alternatively, thedeterminer 35 may simply determine whether a mouse operation is performed. Accordingly, the flow proceeds to step S220 upon a specific mouse event even if the view area T is not changed. - The
image superimposing unit 33 identifies the view area T on the basis of the line-of-sight direction and the angle of view (step S220). That is, the latitude range and the longitude range that correspond to the view area T in the spherical image CE are identified. - Subsequently, the
determiner 35 determines whether at least part of the planar image P is included in the view area T (step S230). If the planar image P is not included in the view area T at all, a moving image need not be replayed, and therefore, the flow proceeds to step S290. - If at least part of the planar image P is included in the view area T (Yes in step S230), the
determiner 35 calculates the angle of view b of the planar image P (step S240). Thedeterminer 35 further calculates the angle of view a of the view area T (step S250). - The
determiner 35 compares the ratio a/b between the angle of view a and the angle of view b with a threshold 1.2 and determines whether the ratio a/b is larger than the threshold 1.2(step S260). - If determination in step S260 results in Yes (if the ratio a/b is larger than the threshold 1.2), it is not inferred that the user U views the planar image P, and therefore, the
image switching unit 36 sends a notification indicating that moving image reading is OFF to the planar image reader 25 (step S270). - If determination in step S260 results in No (if the ratio a/b is equal to or smaller than the threshold 1.2), it is inferred that the user U views the planar image P, and therefore, the
image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S280). - The
image display unit 31 displays, on thedisplay 508, thesuperimposed image 305 generated by superimposing the planar image P, which is a still image or a moving image, on the spherical image CE (step S290), If determination in step S230 results in No, the planar image P is not superimposed. - With the process described above, when the user U enlarges and displays the planar image P, a moving image can be replayed. When the user U views a portion other than the planar image P, replay of the moving image can be stopped. While a moving image is replayed, the
image display unit 31 may display a menu for the user U to control replay of the moving image and accept, for example, stop, replay, rewinding, or fast forward in accordance with an operation by the user U. -
FIG. 24 is a diagram illustrating an example menu (user interface (UI)) for a user to control replay of a moving image.FIG. 24 illustrates a menu 20( )that includes areplay button 201, apause button 202, astop button 203, a fast-forward button 206, a fast-rewind button 205, a subsequent-moving-image button 207, and a previous-moving-image button 204. Themenu 200 may further include, for example, a replay repeat button. The subsequent-moving-image button 207 and the previous-moving-image button 204 are activated in a case where the planar image P includes a plurality of moving images. The designs of the buttons illustrated inFIG. 24 are merely examples. - The
menu 200 as illustrated inFIG. 24 is to be displayed, for example, in a case where a moving image is replayed and the user clicks the moving image replay area or in a case where replay of a moving image ends. - When the entire moving image of the planar image P has been replayed, the
planar image reader 25 may repeat reading of the planar image P from the first frame or stop reading the planar image P. - Next, the method for determination in “(ii) the case where the area of the planar image P has an angle of view equal to or larger than a predetermined value” is described. Determination in this case is substantially the same as that in (i) and is based on the assumption that, in a case where the user U enlarges and displays the planar image P, it is inferred that the user U wants to view the planar image P. In the method for determination, the coordinates of the vertexes B1 to B4 of the planar image P are used as described with reference to
FIG. 21 . Alternatively, the coordinates of corresponding points on thedisplay 508 obtained as a result of perspective projection conversion of the coordinates of the vertexes B1 to 34 of the planar image P may be used to perform determination. - Alternatively, it may be inferred that the user U wants to view the planar image P when the distance between the center of the planar image P and the center of the view area T is smaller than a threshold without using the angle-of-view ratio. The threshold may be set to a value that corresponds to a fraction of the angle of view of the view area T, such as a quarter of the angle of view of the view area T.
-
FIG. 25 is an example flowchart illustrating a procedure for thedisplay control apparatus 30 to control whether to replay a moving image. In the description ofFIG. 25 , differences fromFIG. 23 are described. InFIG. 25 , thedeterminer 35 need not calculate the angle of view a of the view area T, and therefore, step S250 is omitted. - In step S260-2, the
determiner 35 determines whether the angle of view h is larger than a threshold Th. If determination in step S260-2 results in No (if the angle of view b is equal to or smaller than the threshold), it is not inferred that the user U views the planar image P, and therefore, theimage switching unit 36 sends a notification indicating that moving image reading is OFF to the planar image reader 25 (step S270). - If determination in step S260-2 results in Yes (if the angle of view b is larger than the threshold), it is inferred that the user U views the planar image P, and therefore, the
image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S280). - The threshold Th is, for example, an angle of view that corresponds to a specific number of pixels with which the user U can recognize an object. When a typical number of pixels of the
display 508 is assumed to be 768×1024. the threshold Th can be set to, for example, an angle of view that corresponds to a number of pixels 500. Accordingly, in a case where the planar image P is enlarged to some extent and displayed, a moving image can be replayed. - Next, the method for determination in “(iii) the case where the user U clicks the planar image P” is described. In the case where the user U clicks the planar image P, it is obvious that the user U wants to view the planar image P. Therefore, even if any of the conditions (i) and (ii) is not satisfied, it is determined that a moving image is to be replayed. In a case of display on a device including a. touch panel, a similar Operation is performed when the planar image P is touched. It is obvious that the user U wants to view the planar image P, and therefore, if the
second projection converter 29 displays the planar image P across the view area T, the user U need not perform an operation so as to place the planar image P in front of the user U, which is convenient to the user U. Specifically, this display method is a method in which, when the user U clicks the planar image P illustrated inFIG. 22A , the view area T illustrated inFIG. 22A changes to the view area T as illustrated inFIG. 22B . Such a display method is referred to as automatic enlargement of the planar image P and described. - Automatic enlargement of the planar image P is described with reference to
FIG. 26 .FIG. 26 is an example diagram illustrating a process for automatic enlargement of the planar image P in the view area T. It is assumed that the user U performs a click operation in a state where amouse cursor 311 is placed over the planar image P. In a case of a touch panel, this operation corresponds to an operation in which the user U taps the planar image P with their to finger. A mouse cursor and a touch panel are called pointing devices. To simplify description, the following description is given under the assumption that the pointing device is, for example, themouse cursor 311. - The line-of-sight direction/angle-of-
view specifier 26 converts the coordinates of a position on thedisplay 508 at which themouse cursor 311 is placed and at which a click operation is performed to 3D coordinates of a corresponding position in the spherical image CE. This conversion corresponds to the reverse of perspective projection conversion. Thedeterminer 35 converts the 3D coordinates corresponding to the position of themouse cursor 311 to a longitude and latitude and determines whether the position of themouse cursor 311 indicated by the coordinates is within the planar image P. This determination is performed as follows. - Longitude of vertex B1≤Longitude corresponding to coordinates of mouse cursor position≤Longitude of vertex B2 and
- Latitude of vertex B1≥Latitude corresponding to coordinates of mouse cursor position≥Latitude of vertex B4
- In a case where the position of the
mouse cursor 311 indicated by the coordinates is within the planar image P, thedeterminer 35 determines that the planar image P is clicked. - In this case, the
image display unit 31 automatically enlarges the planar image P. Thesecond projection converter 29 performs image processing for gradually enlarging the planar image P so as to fit in the view area T. Specifically, aline segment 312 connecting the vertex A1 and the vertex B1, aline segment 312 connecting the vertex A2 and the vertex B2, aline segment 312 connecting the vertex A3 and the vertex B3, and aline segment 312 connecting the vertex A4 and the vertex B4 are each divided at equal intervals. InFIG. 26 , each of theline segments 312 is divided into four line segment. The interpolation points on theline segments 312 obtained as a result of the division are referred to as P1i to P4i (i is an integer from 1 to 3). - The process for automatically enlarging the planar image P is a process for decreasing the angle of view a of the view area T. The
second projection converter 29 reduces the view area ‘T’ so as to fit in an area formed by connecting the interpolation points P13, P23, P33, and P43 (decreases the angle of view a). Subsequently, thesecond projection converter 29 reduces the view area T so as to fit in an area formed by connecting the interpolation points P12, P22, P32, and P42. Subsequently, thesecond projection converter 29 reduces the view area T so as to fit in an area formed by connecting the interpolation points P11, P21, P31, and P41. Subsequently, thesecond projection converter 29 reduces the view area T so as to fit in an area formed by connecting the vertexes B1, B2, B3, and B4 of the planar image P. - With such image processing, the planar image P appears to gradually enlarge to the user U. The planar image P is subjected to perspective projection conversion in the view area T and displayed across the display 508 (across an area displayed by display software), and therefore, the planar image P appears to be large to the user U.
- In a case where the entire planar image P is not included in the view area T, automatic enlargement can be similarly performed.
FIG. 27 is an example diagram illustrating a process for automatic enlargement of the planar image P in the case where the entire planar image P is not included in the view area T. InFIG. 27 , only the vertex B4 of the planar image P is within the view area T. The vertexes B1 to B3 of the planar image P are not displayed in the view area T but the coordinates of the vertexes B1 to B3 are known, and therefore, the planar image P can be automatically enlarged in a similar manner toFIG. 26 . -
FIG. 28 is an example flowchart illustrating a procedure for thedisplay control apparatus 30 to determine whether to replay a moving image. In the description ofFIG. 28 , differences fromFIG. 23 are described. - The process in steps S210 and S220 is performed in a substantially similar as described above referring to
FIG. 23 . Subsequently, thedeterminer 35 determines whether the planar image P is clicked (step S222). If determination in step S222 results in No, the subsequent process is similar to that inFIG. 23 . - If determination in step S222 results in Yes, the
image switching unit 36 sends a notification indicating that moving image reading is ON to the planar image reader 25 (step S224). Accordingly, it is determined that a moving image is to be replayed. - The
second projection converter 29 automatically enlarges the planar image P (step S226). With the process described above, the user U need not perform an operation so as to place the planar image P in front of the user U. - It is preferable that the user U be allowed to set whether a moving image is to be replayed when the user U clicks the planar image P. After replay of the moving image, when the user U performs an operation for displaying a portion other than the planar image P and the condition in step S260 is satisfied, the moving image can be stopped. Also in this case, an operation menu may be displayed when the moving image is displayed.
- In this embodiment, the description has been given under the assumption that the planar image P is a moving image and the spherical image CE is a still image. Even in a case where the planar image P is a still image and the spherical image CE is a moving image, the above-described display can be implemented with similar processing. In this case, the
spherical image reader 27 needs to read frames of the spherical image CE that is a moving image. For example, in a case where thedisplay control apparatus 30 displays the spherical image CE of the inside of an art museum, pictures are represented by the planar images P, and the spherical image CE obtained by capturing an image of the inside of the entire art museum is provided as a moving image. In the planar images P, no visitors are present. Therefore, the user U can enlarge and view the planar images P and easily know the situation from a state represented by the spherical image CE in which, for example, visitors move around the art museum. - In this case, the spherical image CE is displayed as a moving image until one of the conditions (i) to (iii) described above is satisfied. When one of the conditions (i) to (iii) is satisfied, the spherical image CE is displayed as a still image. Accordingly, when the user U performs an operation for enlarging and displaying the planar image P, the spherical image CE becomes a still image, resulting in increased viewability.
- Alternatively, both the planar image P and the spherical image CE may be moving images. In this case, the spherical image CE and the planar image P are displayed as moving images until one of the conditions (i) to (iii) described above is satisfied. When one of the conditions (i) to (iii) is satisfied, the spherical image CE and the planar image P are displayed as still images. Accordingly, when the user U performs an operation for enlarging and displaying the planar image P, the spherical image CE becomes a still image, resulting in increased viewability.
- The description has been given with reference to
FIG. 24 indicating that the user can control replay of a moving image using themenu 200. If metadata related to replay of a moving image is available, thedisplay control apparatus 30 can replay the moving image in accordance with the metadata. Metadata for replay of a moving image is data in which how the moving image is to be replayed is specified. -
FIGS. 29A to 29C are example diagrams illustrating metadata for replay of a moving image. Metadata for replay of a moving image includes a replay start time, a replay end time, a replay direction, a replay speed, a repetition setting, the number of repetitions, a repetition method, and a state after stop, which are described with reference toFIGS. 29A to to 29C. - As illustrated in
FIG. 29A , the replay start time is a parameter for replaying the moving image from a midpoint instead of the beginning of the moving image and is specified as a time or a frame number. In a case where the replay start time is not specified, the initial is value is set to 0. - The replay end time is a parameter for stopping replay of the moving image at a midpoint without replaying the moving image to the end and is specified as a time or a frame number. In a case where the replay end time is not specified, the initial value is set to the length of the moving image. The replay start time and the replay end time are specified in a case where the user wants to replay only part of the moving image. The moving image that is actually replayed is a portion from the replay start time to the replay end time.
- The replay direction indicates the replay direction when replay of the moving image starts and is set to the forward direction or the reverse direction.
- The replay speed is a parameter for slow replay or fast replay. The usual replay speed is 1.0. The replay speed of less than 1 (for example, 0.5) indicates slow replay, and the replay speed of more than 1 (for example, 2.0) indicates fast replay.
- The repetition setting is a parameter for specifying whether the moving image is to be repeatedly replayed in a case of replaying the moving image.
FIG. 29B illustrates a state where the moving image is repeatedly replayed from the beginning to the end. In a case of non-repetition, after the moving image has been replayed from the beginning to the end, the replay of the moving image ends. In a case where the replay direction is the reverse direction and non-repetition is specified, after the moving image has been replayed from the end to the beginning, the replay of the moving image ends. - The number of repetitions is the number of times the moving image is repeatedly replayed. After the moving image has been repeatedly replayed a specified number of times, the replay of the moving image ends. The number of repetitions can be set to a number that corresponds to an infinite number. In this case, the moving image is repeatedly replayed until a stop condition is satisfied.
- The repetition method is a parameter for specifying a replay method in a case of repeatedly replaying the moving image and is set to loop replay or alternate replay. The loop replay is replay repeated in the same direction, and the alternate replay is replay repeatedly performed in which the replay direction is flipped each time replay ends, as illustrated in
FIG. 29C . In both the loop replay and the alternate replay, in a case where the replay direction parameter is set to the reverse direction, the replay direction is flipped. - The state after stop is a parameter for specifying a state after replay of the moving image has stopped. The state after stop is set to “unchanged”, “to the beginning” (or “to the end” in the case of replay in the reverse direction), or “transparent”.
- Accordingly, when metadata is attached to the spherical image CE, the
image processing server 70 can control replay of a moving image in thedisplay control apparatus 30. - With the configuration in which the
image processing server 70 and thedisplay control apparatus 30 communicate with each other, as illustrated inFIG. 3C , at least one of theimage processing server 70 and thedisplay control apparatus 30 can save metadata in association with the location parameter. - As described above, with the
display control apparatus 30 according to this embodiment, when it is inferred that the user views the planar image P, a moving image is replayed. Therefore, the moving image can be replayed in a case where the user wants to view the planar image P, and unwanted replay of the moving image can be suppressed. Further, the user can view the moving image from the beginning. When the user enlarges the planar image P, replay of the moving image starts. Therefore, motion seems to start suddenly on the screen, which can attract the user's attention. - In this embodiment, the
display control apparatus 30 that displays the planar image P on the spherical image CE together with an outline indicating the presence of the planar image P is described. In this embodiment, a constituent element assigned the same reference numeral as that of a corresponding constituent element described in the first embodiment has the same functions as those of the corresponding constituent element. Therefore, description of a constituent element that has been described may be omitted, or only differences may be described. - It may be difficult to know a position in the spherical image CE at which the planar image P is superimposed. In a case where the planar image P that is a still image is also to superimposed, the user U may have difficulty in determining whether the planar image P is a moving image or a still image. In such cases, an
outline 40 indicating the presence of the planar image P and displayed by theimage superimposing unit 33 is effective. If the form of theoutline 40 is made different for a still image and a moving image, the user U looking at theoutline 40 can know whether the planar image P is a moving image or a still image. - The planar image P that is a moving image or a still image has higher image quality than that of the spherical image CE. When the user U enlarges the spherical image CE to thereby display the enlarged planar image P, the user U can view the image of higher image quality (for example, a higher resolution).
- The
outline 40 can be displayed in a similar manner to projection conversion of the planar image P. When thefirst projection converter 28 performs projection conversion of a rectangular indicating the outer edge of the planar image P, theoutline 40 is obtained. Thefirst projection converter 28 performs projection conversion of the rectangular indicating the outer edge similarly to a case of projection conversion of the planar image P using the location parameter PP. -
FIG. 30 is an example diagram schematically illustrating theoutline 40 obtained as a result of projection conversion. Thefirst projection converter 28 obtains atransparent layer 308 having the same size (the same number of pixels as that of the spherical image CE in equirectangular projection. On thetransparent layer 308, thefirst projection converter 28 performs projection conversion ofrectangular data 321 that indicates the outer edge of the planar image P. Alternatively, thefirst projection converter 28 may draw a dotted line that constitutes theoutline 40 in accordance with the latitude and longitude of a grid defined by the location parameter PP having values x=0.5, −0.5 or y=0.5, −0.5. - The
first projection converter 28 makes the form of theoutline 40 different depending on whether the planar image P is a moving image or a still image. For example, thefirst projection converter 28 changes the color of theoutline 40 to red in a case of a moving image and to blue in a case of a still image. Thefirst projection converter 28 may change the line type of theoutline 40 to a solid line in the case of a moving image and to a dotted line in a case of a still image. Alternatively, thefirst projection converter 28 may change the line thickness of theoutline 40. Thefirst projection converter 28 may display text indicating whether the planar image P is a moving image or a still image near theoutline 40. Theplanar image reader 25 transmits a notification to thefirst projection converter 28 whether the planar image P is a moving image or a still image. A moving image and a still image may be distinguished from each other on the basis of, for example, the extension of the to file (jpeg indicates a sill image, and mpeg indicates a moving image) or on the basis of the file name of the planar image P. -
FIG. 31 is an example diagram schematically illustrating superimposition of theoutline 40. Thedisplay image 304, the projection-convertedimage 302, themask data 303, and thetransparent layer 308 are input to theimage superimposing unit 33. The method for superimposing the projection-convertedimage 302 on thedisplay image 304 has been described with reference toFIG. 15 . - The
image superimposing unit 33 uses themask data 303 to take, from thetransparent layer 308, an area corresponding to white pixels in themask data 303. Theimage superimposing unit 33 superimposes the area taken from thetransparent layer 308 on thesuperimposed image 305 to thereby display theoutline 40 indicating the outer edge of the planar image P. - For example, the user U may perform an operation to switch between display and non-display of the
outline 40. In this case, theimage superimposing unit 33 switches between display and non-display of thetransparent layer 308 in accordance with the operation by the user U. In a case where the mouse cursor is placed over the planar image P, theimage superimposing unit 33 may display theoutline 40 of the planar image P over which the mouse cursor is placed. -
FIG. 32 is a diagram illustrating examples of theoutline 40 displayed in different forms for a moving image and a still image. The moving image is outlined by anoutline 40 a represented by a dotted line, and the still image is outlined by anoutline 40 b represented by a solid line. Therefore, the user can easily determine whether the planar image P is a moving image or a still image. Further, the user can know the position and size of the planar image P. -
FIG. 33 is an example flowchart illustrating a procedure for thedisplay control apparatus 30 to display theoutline 40 of the planar image P on the spherical image CE. In the description ofFIG. 33 , differences fromFIG. 17 are described. The process from step S10 to step S30 is the same as that inFIG. 17 . However, in step S30, thelocation parameter calculator 23 calculates the location parameter PP for all of the planar images P. - The subsequent steps S40 to S90 are performed in a substantially similar manner as described above referring to
FIG. 17 . After step S90, thefirst projection converter 28 obtains therectangular data 321 that indicates the outer edge of the planar image P, therectangular data 321 having a different form depending on whether the planar image P is a moving image or a still image. Thefirst projection converter 28 performs projection to conversion of therectangular data 321 indicating the outer edge on thetransparent layer 308 using the location parameter PP (step S92), As a result, theoutline 40 that differs depending on whether the planar image P is a moving image or a still image is projected onto thetransparent layer 308. - Subsequently, the
image superimposing unit 33 generates the superimposed image 305 (step S100), and uses themask data 303 to extract theoutline 40 from the transparent layer 308 (step S102). - The
image superimposing unit 33 determines whether all of the planar images P have been superimposed (step S104). Until all of the planar images P are superimposed, the flow returns to step S70, and the process from step S70 to step S102 is performed. - As described above, the
display control apparatus 30 can superimpose the planar image P on the spherical image CE and display theoutline 40 having a different form depending on whether the planar image P is a moving image or a still image. It is preferable to stop displaying theoutline 40 in a case where one of the conditions (i) to (iii) described above is satisfied. If theoutline 40 is not displayed, theoutline 40 does not hinder the user U from viewing the planar image P any more. - The
image superimposing unit 33 may display theoutline 40 only in a case where the user U places the mouse cursor over the planar image P or may display theoutline 40 only in a case where the user U does not place the mouse cursor over the planar image P. The user U can move the mouse cursor to switch between display and non-display of theoutline 40. Theimage superimposing unit 33 may display theoutline 40 until the time elapsed since a change of the view area T reaches a predetermined time, and thereafter, may stop displaying theoutline 40. - The
outline 40 may be displayed with the focal length. Theoutline 40 and focal length information are displayed near the planar image P, and therefore, the user U can know the focal length of the planar image P that is superimposed. - As described above, with the
display control apparatus 30 according to this embodiment, the position where the planar image P is superimposed can be indicated by, for example, theoutline 40. Further, the user can determine whether the planar image P is a moving image or a still image. - The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and/or features of different illustrative embodiments may be combined with each other and/or substituted for each other within the scope of the present invention.
- For example, in the above-described embodiments, switching between replay and stop of a moving image is automatically performed; however, the user U may intentionally specify replay or stop of a moving image.
- The spherical image CE may be displayed by using browser software or by using application software for displaying the spherical image CE.
- In the configuration illustrated in
FIG. 3C , theimage processing server 70 may perform processing up to generation of the location parameter for display and return the generated location parameter for display and the frame ID to thedisplay control apparatus 30 as metadata. Then, thedisplay control apparatus 30 may perform display using the metadata, the spherical image, and the planar image (that may include a moving image). - The example configurations illustrated in, for example,
FIG. 12 andFIG. 18 are divided into blocks in accordance with main functions in order to facilitate understanding of processing by thespherical camera 20, thedisplay control apparatus 30, and thedigital camera 9. The present invention is not limited by the way of division into processing units or the names of the processing units. Processing by thespherical camera 20, thedisplay control apparatus 30, and thedigital camera 9 can be further divided into a larger number of processing units depending on the details of processing. Further, the processing can be divided such that one processing unit includes a larger number of processes. Regarding the processing performed in the configurations illustrated inFIG. 12 andFIG. 18 , either thedisplay control apparatus 30 or theimage processing server 70 may perform part or all of the processing. - For example, the
image processing server 70 may perform processing up to processing performed by theimage superimposing unit 33, and theimage display unit 31 may be included in thedisplay control apparatus 30, which is separate from theimage processing server 70. - Further, in addition to the above-described specific examples in which it is inferred that the second image is viewed by the user, there are many other example such cases.
- For example, when a ratio of an area of the view area T in the spherical image CE with respect to an area of the planar image P in the view area T is less than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image. On the other hand, when a ratio of an area of the view area T in the spherical image CE with respect to an area of the planar image P in the view area T is equal to or greater than the predetermined value, it is inferred that the planar image P is not to viewed by the user, and the planar image P is displayed as a still image.
- In another example, when a size of the planar image P in the view area T is equal to or larger than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image. On the other hand, when a size of the planar image P in the view area is less than the predetermined value, it is inferred that the planar image P is not viewed by the user, and the planar image P is displayed as a still image.
- In another example, when a difference between the center of the planar image P and center of the view area T is equal to or less than a predetermined value, it is inferred that the planar image P is viewed by the user, and the planar image P is displayed as a moving image. when a difference between the center of the planar image P and center of the view area T is larger than the predetermined value, it is inferred that the planar image P is not viewed by the user, and the planar image P is displayed as a still image.
- In any one of the above-described embodiments, when an area of the planar image P in the view area T is large, it is assumed that the user is viewing the planar image P. On the other hand, when an area of the planar image P in the view area T is small, it is assumed that the user is not paying attention to the planar image P. Since the area of the planar image Pin the view area T may be defined in various ways, determining whether the user is viewing the planar image P may be performed in various ways as described above. Further, the threshold for comparison may be determined, for example, according to the user preference as described above.
- In this disclosure, examples of superimposition of images include, but not limited to, placement of one image on top of other image entirely or partly, laying one image over other image entirely or partly, mapping one image on other image entirely or partly, pasting one image on other image entirely or partly, combining one image with other image, and integrating one image with other image. That is, as long as the user can perceive a plurality of images (such as the spherical image and the planar image) being displayed on a display as they were one image, processing to be performed on those images for display is not limited to the above-described examples.
- Each of the functions of the described embodiments may be implemented by one or more processing circuits or circuitry. Processing circuitry includes a programmed processor, as a processor includes circuitry. A processing circuit also includes devices such as an application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA), and conventional circuit components arranged to perform the recited functions.
Claims (17)
1. An information processing apparatus, comprising:
circuitry configured to
control a display to display a first captured image; and
control the display to display a second captured image being superimposed on a view area of the first captured image, wherein
the circuitry controls the display to display the second captured image as a moving image when the second captured image is displayed by the display.
2. , The information processing apparatus of claim 1 , wherein the circuitry is further configured to control the display to display the second captured image as a still mage.
3. The information processing apparatus of claim 2 , wherein the circuitry is further configured to control the display to switch display of the second captured image from the still image to the moving image.
4. The information processing apparatus of claim 2 , wherein the still image is a particular frame of the moving image.
5. The information processing apparatus of claim I, wherein the circuitry is further configured to
determine whether the second captured image is being displayed on the display,
based on a determination that the second captured image is being displayed on the display, control the display to display the second captured image as the moving image, and
based on a determination that the second captured image is not being displayed on the display, control the display to display the second captured image as a still image.
6. The information processing apparatus of claim I, wherein the circuitry is further configured to
compare a ratio, of the view area of the first captured image with respect to a view area of the second captured image, with a threshold to generate a result,
based on the result indicating that the ratio is greater than the threshold, display the second captured image as a still image, and
based on the result indicating that the ratio of view area is equal to or less than the threshold, display the second captured image as the moving image.
7. The information processing apparatus of claim I, wherein the circuitry is further configured to
compare a size of the second captured image with a threshold to generate a result,
based on the result indicating that the size of the second captured image is greater than the threshold, display the second captured image as the moving image, and
based on the result indicating that the size of the second captured image is equal to or less than the threshold, display the second captured image as a still image.
8. The information processing apparatus of claim 1 , wherein the circuitry is further configured to
receive an operation to the first captured image, the operation including input of an input coordinate,
determine whether the input coordinate matches any coordinate in the second captured image, and
based on a determination that the input coordinate matches any coordinate in the second captured image, control the display to display the second captured image as the moving image.
9. The information processing apparatus of claim 8 , wherein when the circuitry determines that the input coordinate matches any coordinate in the second captured image, while the second captured image is being displayed on the display, the circuitry controls the display to display the second captured image such that size of the second captured image is increased to a maximum viewing area of the display.
10. The information processing apparatus of claim I, wherein
the second captured image includes a moving image and a still image, and
the circuitry is further configured to control the display to display an indication that the second captured image, displayable as the moving image and as the still image, are being superimposed.
11. The information processing apparatus of claim 1 , wherein the circuitry is further configured to set, as a control of replaying the second captured image, at least one of a replay start time, a replay end time, a replay direction, a replay speed, a repetition setting indicating whether to repeat, a number of repetitions, and a repetition method.
12. The information processing apparatus of claim I, wherein
the first captured image is a spherical image, and
the second captured image is a planar image.
13. The information processing apparatus of claim 12 , further comprising the display configured to display the first captured image and the second captured image.
14. An information processing system comprising:
the information processing apparatus of claim 1 ; and
the display configured to display the first captured image and the second captured image.
15. The information processing system of claim 14 , wherein the information processing apparatus is a server that communicates with a client provided with the display.
16. An information processing method, comprising:
controlling a display to display a first captured image;
controlling the display to display a second captured image being superimposed on a view area of the first captured image; and
when the second captured image is displayed by the display, controlling the display to display the second captured image as a moving image.
17. A non-transitory computer readable medium storing computer-executable instructions which, when executed by circuitry of an information processing apparatus, causes the circuitry to:
control a display to display a first captured image; and
control the display to display a second captured image being superimposed on a view area of the first captured image, wherein
the circuitry controls the display to display the second captured image as a moving image when the second captured image is displayed by the display.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/562,437 US20190391778A1 (en) | 2016-12-28 | 2019-09-06 | Apparatus, system, and method for controlling display, and recording medium |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016-256383 | 2016-12-28 | ||
JP2016256383 | 2016-12-28 | ||
JP2017-208675 | 2017-10-27 | ||
JP2017208675A JP2018110375A (en) | 2016-12-28 | 2017-10-27 | Display device, program, and display method |
US15/855,443 US10437545B2 (en) | 2016-12-28 | 2017-12-27 | Apparatus, system, and method for controlling display, and recording medium |
US16/562,437 US20190391778A1 (en) | 2016-12-28 | 2019-09-06 | Apparatus, system, and method for controlling display, and recording medium |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/855,443 Continuation US10437545B2 (en) | 2016-12-28 | 2017-12-27 | Apparatus, system, and method for controlling display, and recording medium |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190391778A1 true US20190391778A1 (en) | 2019-12-26 |
Family
ID=62630602
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/855,443 Expired - Fee Related US10437545B2 (en) | 2016-12-28 | 2017-12-27 | Apparatus, system, and method for controlling display, and recording medium |
US16/562,437 Abandoned US20190391778A1 (en) | 2016-12-28 | 2019-09-06 | Apparatus, system, and method for controlling display, and recording medium |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/855,443 Expired - Fee Related US10437545B2 (en) | 2016-12-28 | 2017-12-27 | Apparatus, system, and method for controlling display, and recording medium |
Country Status (1)
Country | Link |
---|---|
US (2) | US10437545B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230026625A1 (en) * | 2021-07-21 | 2023-01-26 | Tempus Ex Machina, Inc. | Adding augmented reality to a sub-view of a high resolution central video feed |
US11928775B2 (en) | 2020-11-26 | 2024-03-12 | Ricoh Company, Ltd. | Apparatus, system, method, and non-transitory medium which map two images onto a three-dimensional object to generate a virtual image |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019057264A (en) * | 2016-12-28 | 2019-04-11 | 株式会社リコー | Image processing apparatus, photographing system, image processing method, and program |
JP7135299B2 (en) | 2017-10-27 | 2022-09-13 | 株式会社リコー | Image processing device, display device, image processing method, program |
JP6413035B1 (en) * | 2018-01-18 | 2018-10-24 | 株式会社ほぼ日 | Program, information processing apparatus and information processing method |
US10855916B2 (en) | 2018-03-15 | 2020-12-01 | Ricoh Company, Ltd. | Image processing apparatus, image capturing system, image processing method, and recording medium |
US10593014B2 (en) | 2018-03-26 | 2020-03-17 | Ricoh Company, Ltd. | Image processing apparatus, image processing system, image capturing system, image processing method |
KR102435519B1 (en) * | 2018-06-20 | 2022-08-24 | 삼성전자주식회사 | Method and apparatus for processing 360 degree image |
US10805534B2 (en) * | 2018-11-01 | 2020-10-13 | Korea Advanced Institute Of Science And Technology | Image processing apparatus and method using video signal of planar coordinate system and spherical coordinate system |
US11250540B2 (en) | 2018-12-28 | 2022-02-15 | Ricoh Company, Ltd. | Image processing apparatus, image capturing system, image processing method, and recording medium |
WO2020162218A1 (en) * | 2019-02-08 | 2020-08-13 | ソニー株式会社 | Reproduction device, reproduction method, and program |
CN111726520B (en) | 2019-03-20 | 2021-12-10 | 株式会社理光 | Imaging device, imaging system, and image processing method |
US10812771B1 (en) * | 2019-06-12 | 2020-10-20 | At&T Intellectual Property I, L.P. | Methods, systems, and devices for adjusting image content for streaming panoramic video content |
US20220244726A1 (en) * | 2019-07-03 | 2022-08-04 | Sony Group Corporation | Information processing apparatus, information processing method, and program |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9128520B2 (en) * | 2011-09-30 | 2015-09-08 | Microsoft Technology Licensing, Llc | Service provision using personal audio/visual system |
US9618747B2 (en) * | 2013-03-13 | 2017-04-11 | Jason Villmer | Head mounted display for viewing and creating a media file including omnidirectional image data and corresponding audio data |
CN108600576B (en) | 2013-08-28 | 2020-12-29 | 株式会社理光 | Image processing apparatus, method and system, and computer-readable recording medium |
JP2016096487A (en) | 2014-11-17 | 2016-05-26 | 株式会社クワンズ | Imaging system |
US9824499B2 (en) * | 2015-06-23 | 2017-11-21 | Microsoft Technology Licensing, Llc | Mixed-reality image capture |
JP6555239B2 (en) | 2016-03-16 | 2019-08-07 | 株式会社リコー | Semiconductor integrated circuit and clock supply method for semiconductor integrated circuit |
US10234891B2 (en) | 2016-03-16 | 2019-03-19 | Ricoh Company, Ltd. | Semiconductor integrated circuit, and method for supplying clock signals in semiconductor integrated circuit |
-
2017
- 2017-12-27 US US15/855,443 patent/US10437545B2/en not_active Expired - Fee Related
-
2019
- 2019-09-06 US US16/562,437 patent/US20190391778A1/en not_active Abandoned
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11928775B2 (en) | 2020-11-26 | 2024-03-12 | Ricoh Company, Ltd. | Apparatus, system, method, and non-transitory medium which map two images onto a three-dimensional object to generate a virtual image |
US20230026625A1 (en) * | 2021-07-21 | 2023-01-26 | Tempus Ex Machina, Inc. | Adding augmented reality to a sub-view of a high resolution central video feed |
US11587266B2 (en) * | 2021-07-21 | 2023-02-21 | Tempus Ex Machina, Inc. | Adding augmented reality to a sub-view of a high resolution central video feed |
US11816760B2 (en) | 2021-07-21 | 2023-11-14 | Tempus Ex Machina, Inc. | Adding augmented reality to a sub-view of a high resolution central video feed |
Also Published As
Publication number | Publication date |
---|---|
US20180181358A1 (en) | 2018-06-28 |
US10437545B2 (en) | 2019-10-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10437545B2 (en) | Apparatus, system, and method for controlling display, and recording medium | |
US10789671B2 (en) | Apparatus, system, and method of controlling display, and recording medium | |
US10681271B2 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
US10593014B2 (en) | Image processing apparatus, image processing system, image capturing system, image processing method | |
US10855916B2 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
US20190340737A1 (en) | Image processing apparatus, image processing system, image capturing system, image processing method, and recording medium | |
KR20200028481A (en) | Imaging apparatus, image display system and operation method | |
US10939068B2 (en) | Image capturing device, image capturing system, image processing method, and recording medium | |
JP2018110375A (en) | Display device, program, and display method | |
JP2018129716A (en) | Image processing device, image processing system, image processing metho, and program | |
JP2019057903A (en) | Image processing apparatus, photographing system, image processing method, and program | |
US20190289206A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
US20190347775A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
US20190306420A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
JP2019164782A (en) | Image processing apparatus, image capturing system, image processing method, and program | |
US20200236277A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
US20240087157A1 (en) | Image processing method, recording medium, image processing apparatus, and image processing system | |
WO2018124267A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
JP2018109946A (en) | Display device, program, and method for display | |
US20200288083A1 (en) | Image capturing apparatus, image processing system, image processing method, and recording medium | |
JP2018157538A (en) | Program, imaging system, and information processing apparatus | |
JP2019164783A (en) | Image processing apparatus, image capturing system, image processing method, and program | |
JP2019185757A (en) | Image processing device, imaging system, image processing method, and program | |
JP2018109971A (en) | Image processing device, image processing system, photographing system, image processing method, and program | |
JP2018109740A (en) | Display device, program, and method for display |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |