JP6871458B2 - Imaging display method - Google Patents

Description

The present invention relates to a technique of an image pickup device and a display device, and relates to a screen display technique of a captured image.

Conventionally, there are digital cameras, smartphones, and the like as electronic devices such as image pickup devices having a function of capturing a plurality of images having different shooting directions and angles of view. Further, as an electronic device such as a display device having a function of reproducing and displaying a plurality of these images, there are a television receiver (hereinafter, may be referred to as "television"), a notebook PC, and the like. The above image is a still image or a moving image. The shooting direction can be defined by, for example, an azimuth angle and an elevation angle. The angle of view represents an image range as an angle and is also called a viewing angle, and can be defined by, for example, a horizontal angle of view corresponding to an azimuth angle and a vertical angle of view corresponding to an elevation angle.

Images with different angles of view include images taken at a relatively wide angle (hereinafter, may be referred to as "wide-angle image", "first image", etc.) and images taken at a relatively narrow angle. (Hereinafter, it may be described as "narrow-angle image", "second image", etc.). Examples of a wide angle include a horizontal angle of view of 120 ° and an angle of view of 180 ° or more. In the case of 180 °, it may be called a hemispherical image, and in the case of 360 °, it may be called a spherical image. An example of a narrow angle is a horizontal angle of view of 45 ° or less. Hereinafter, the unit of angle is expressed in degrees (°), but it can be converted into a unit of radians or the like.

Japanese Patent Application Laid-Open No. 2000-92439 (Patent Document 1) is mentioned as an example of prior art relating to screen display of captured images. Patent Document 1 describes the following as an electronic still camera or the like. The electronic still camera generates an image list display file that allows an external device to list and display the recorded images recorded in the recording memory when the power is turned off. A plurality of reduced images are displayed in a list on the list display screen displayed by the image search using the file.

Japanese Unexamined Patent Publication No. 2000-92439

A user who is a photographer may use an image pickup device to capture a plurality of images having different shooting directions and angles of view at the same place and date and time. For example, a user shoots a wide-angle image in a certain shooting direction and angle of view and a narrow-angle image in another shooting direction and angle of view in the same park on the same day. In such a case, it can be said that the plurality of images are related image groups having a predetermined relationship with respect to the place, the date and time, the shooting direction, the angle of view, and the like.

Conventional imaging devices and display devices have a function of displaying a plurality of images on a display screen in a list and in parallel arrangement in the order of, for example, shooting date and time and file name. For example, in the case of Patent Document 1, a plurality of reduced images are displayed in parallel on the list display screen.

However, in the prior art, it is difficult for the user to intuitively recognize a plurality of images having a predetermined relationship among all the images displayed on the display screen. For example, even when wide-angle images and narrow-angle images taken at the same place and date and time are mixed, it is difficult for the user to grasp the relationship between the images. The prior art does not have a function of associating and displaying a plurality of related images. The prior art does not have a function of referencing and displaying another image related to the image from the state of displaying the image on the display screen. In the prior art, it takes time and effort for the user to edit and manage a plurality of images.

That is, in the prior art, it is difficult to view a plurality of related images in a suitable manner, and there is room for improvement in terms of usability.

An object of the present invention is to provide a technique for easily grasping the relationship between a plurality of images and realizing better usability with respect to an imaging device and a display device.

A typical embodiment of the present invention is an imaging device or a display device, and is characterized by having the following configurations.

The imaging device of one embodiment includes an imaging unit that captures an image of a still image or a moving image to obtain image data, and a display unit that reproduces and displays the image on a display screen based on the image data. The first information including at least one of the azimuth angle and the elevation angle as the image shooting direction and at least one of the image angle or the image angle-related information from which the image angle can be calculated is acquired, and the image is captured. For a plurality of images, when the second shooting direction of the second image is included in the range of the first angle of view in the first shooting direction of the first image, the second image is associated with the first image. The first image selected based on the user's operation is displayed in the display screen, and the second image associated with the first image is displayed on the first image or the second image or It is displayed in a superposed state as the second information representing the second image.

The display device of one embodiment includes an input unit that inputs and holds image data of a still image or a moving image captured by an external imaging device, and reproduces the image on a display screen based on the image data. A third display unit including a display unit for displaying, at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of the image angle or the image angle-related information capable of calculating the image angle. When one information is acquired and the second shooting direction of the second image is included in the range of the first angle of view of the first shooting direction of the first image for the plurality of captured images, the said The second image is associated with the first image, the first image selected based on the user's operation is displayed in the display screen, and the second image associated with the first image is displayed on the first image. It is displayed in a state of being superimposed on the image as the second image or the second information representing the second image.

According to a typical embodiment of the present invention, it is easy to grasp the relationship between a plurality of images with respect to the image pickup device and the display device, and better usability can be realized.

It is a figure which shows the structure of the image pickup display system which comprises the image pickup apparatus and display apparatus of Embodiment 1 of this invention. It is a figure which shows the structure of the image pickup apparatus of Embodiment 1. It is a figure which shows the structure of the display device of Embodiment 1. FIG. It is a figure which shows the appearance of the image pickup apparatus of Embodiment 1. It is a figure which shows the structure of management information and metadata in the image pickup apparatus of Embodiment 1. FIG. It is a figure which shows the management of the relationship of a plurality of images by the image pickup apparatus of Embodiment 1. FIG. It is explanatory drawing which shows the photographing direction and the angle of view, particularly the azimuth angle and the horizontal angle of view about a plurality of images in the image pickup apparatus of Embodiment 1. It is explanatory drawing which shows the elevation angle and the vertical angle of view particularly in the image pickup apparatus of Embodiment 1. It is explanatory drawing which shows the angle-of-view related information in the image pickup apparatus of Embodiment 1. It is a figure which shows the transition of the display mode of the display screen in the image pickup apparatus of Embodiment 1. FIG. It is a figure which shows the screen example of the 2nd display mode in the image pickup apparatus of Embodiment 1. It is a figure which shows the screen example of the 1st narrow-angle image of the 3rd display mode in the image pickup apparatus of Embodiment 1. FIG. It is a figure which shows the screen example of the 2nd narrow-angle image of the 3rd display mode in the image pickup apparatus of Embodiment 1. FIG. It is a figure which shows the flow of the control process at the time of the image pickup in the image pickup apparatus of Embodiment 1. FIG. It is a figure which shows the flow of the control process at the time of reproduction display in the image pickup apparatus and display apparatus of Embodiment 1. FIG. It is explanatory drawing which shows the 1st selection method in the image pickup apparatus of Embodiment 1. It is explanatory drawing which shows the 2nd selection method in the image pickup apparatus of Embodiment 1. It is explanatory drawing which shows the 3rd selection method in the image pickup apparatus of Embodiment 1. It is a figure which shows the screen example of the 2nd display mode in the image pickup apparatus of the modification of Embodiment 1. It is a figure which shows the screen example of the 1st display mode and the 2nd display mode in the image pickup apparatus of the modification of Embodiment 1. FIG. It is a figure which shows the example of the display screen of the editing function in the image pickup apparatus of the modification of Embodiment 1. It is explanatory drawing which shows the spherical image in the image pickup apparatus and display apparatus of Embodiment 2 of this invention. It is a figure which shows the structure which concerns on the image pickup part in the image pickup apparatus of Embodiment 2. It is a figure which shows the appearance composition in the image pickup apparatus of Embodiment 2. It is a figure which shows the screen example of the 2nd display mode in the image pickup apparatus of Embodiment 2. It is explanatory drawing which shows the photographing mode in the image pickup apparatus of Embodiment 3 of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In principle, the same parts are designated by the same reference numerals in all the drawings for explaining the embodiments, and the repeated description thereof will be omitted.

(Embodiment 1)
The image pickup apparatus and the display apparatus according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 18. The imaging device and display device of the first embodiment have a function of displaying a plurality of images having different shooting directions and angles of view at the time of shooting, particularly a wide-angle image and a narrow-angle image in an associated state.

[Image capture display system]
FIG. 1 shows the configuration of an imaging display system, which is a system including the imaging device and the display device of the first embodiment. The imaging display system of FIG. 1 includes a first imaging device 1A and a second imaging device 1B, which are imaging devices 1 of the first embodiment, and a first display device 2A and a second display, which are the display devices 2 of the first embodiment. The device 2B and the server 3 are connected to each other via the communication network 4. The communication network 4 includes a wireless communication network and the Internet.

The image pickup apparatus 1 is an electronic device having at least an image pickup function and a display function. The display device 2 is an electronic device having at least a display function. A user who is a photographer has a first imaging device 1A and a second imaging device 1B, and a first display device 2A and a second display device 2B as electronic devices owned or authorized to be used by the photographer. For example, the first image pickup device 1A is a digital camera, the second image pickup device 1B is a smartphone, the first display device 2A is a television, and the second display device 2B is a notebook PC.

The first imaging device 1A and the second imaging device 1B have at least a function of holding the image data of the image captured by the own device in the storage means in the own device. Further, the first imaging device 1A and the second imaging device 1B may have a function of transmitting image data of an image captured by the own device to the server 3 and accumulating and registering the image data in the image data DB of the server 3. .. Further, the first imaging device 1A and the second imaging device 1B have a function of transmitting image data of an image captured by the own device to the first display device 2A and the second display device 2B based on a user operation. You may.

The first display device 2A and the second display device 2B have a function of receiving and inputting image data from an external device such as an image pickup device 1 or a server 3 and holding the image data in a storage means in the own device. For example, the first display device 2A receives image data from the first image pickup device 1A and the second display device 2B by communication. For example, the second display device 2B receives image data from the first image pickup device 1A and the second image pickup device 1B by communication. Further, the first display device 2A and the second display device 2B have an image data display unit. The image data display unit displays an image on the display screen based on the image data.

Further, for example, the second display device 2B has an image data editing unit. The image data editing unit edits the information and contents of the image data based on the user operation on the display screen. As will be described later, the user can make settings related to the relationship between a plurality of images by editing.

The server 3 is, for example, a server device managed by a business operator, which stores image data from a user's device and stores it in an image data DB. As a result, image data backup and centralized management can be realized. The server 3 and the image data DB may be implemented by different devices.

[Imaging device]
FIG. 2 shows the configuration of the image pickup apparatus 1. The image pickup device 1 includes a control unit 101, a storage unit 102, an image pickup unit 103, a display device 104, an operation input unit 105, a recording unit 106, a recording medium 107, a communication interface unit 108, a shooting direction measurement unit 111, an internal clock 112, and a position. It includes a detection unit 113, a sensor group 114, a microphone 115, a speaker 116, an interface circuit unit 131, a signal processing unit 132, a memory 133, and the like. Each of these units is connected to each other via a bus or the like, and cooperates at high speed based on the control of the control unit 101.

The control unit 101 controls the entire image pickup apparatus 1. The control unit 101 includes control circuits such as a processor, ROM, and RAM. The control unit 101 includes an image pickup control unit 11, a display control unit 12, a touch detection unit 13, a setting unit 14, and an association unit 15 as processing units realized by processing of a control circuit.

The control unit 101 controls each unit such as the image pickup unit 103 according to the operation mode to realize imaging and display. The operation mode includes an imaging mode, a playback display mode, and the like. The control unit 101 controls the image pickup unit 103, the display device 104, the recording unit 106, and the like via the signal processing unit 132 and the memory 133 according to the operation mode and the state.

The control unit 101 controls settings such as a shooting direction and an angle of view of imaging based on a user input operation through the operation input unit 105 and the touch sensor 42. The control unit 101 creates metadata 22 together with the image data 23 as the image is taken through the imaging unit 103, and stores or records the metadata 22 and the image data 23 as an image file 24 in the storage unit 102. Recording is performed on the recording medium 107 of 106. Further, the control unit 101 creates and manages the management information 21 regarding the image file 24.

The storage unit 102 stores various data and information including a plurality of image files 24 and management information 21 based on the control of the control unit 101. The storage unit 102 may be integrated in the control unit 101, the storage unit 102 and the recording unit 106 may be integrated into one, and the like.

The image pickup unit 103 takes an image based on the control from the image pickup control unit 11. The image pickup unit 103 includes a drive unit 31 and a lens unit 32. The drive unit 31 includes a drive circuit and drives the image sensor and the lens of the lens unit 32 based on the drive control from the image pickup control unit 11. The lens unit 32 includes an optical system with a plurality of lenses, including a lens corresponding to an electronic zoom or an optical zoom. For example, the drive unit 31 controls the position of the focus lens to perform a focusing operation for changing the focal length. The lens of the lens unit 32 may be a zoom lens system that is fixed without replacement and is compatible with electronic zoom, or an optical lens system that can be replaced by attachment / detachment.

The image pickup unit 103 includes an image pickup element such as CMOS or CCD. Photoelectric conversion elements are arranged two-dimensionally on the image pickup surface of the image pickup device. The image pickup unit 103 converts the optical image of the subject, which is incident through the lens unit 32 and formed on the image pickup surface of the image pickup device, by photoelectric conversion and converts it into an image pickup signal. The image pickup unit 103 has a built-in AD conversion circuit that converts an analog signal into a digital signal, and outputs a digitized image pickup signal. In the image pickup unit 103, an image pickup element in which pixels for phase difference autofocus are arranged may be used to speed up autofocus, or a memory may be built in to speed up input / output of an image pickup signal. May be good. An AD conversion circuit or the like may be provided outside the image pickup unit 103. The imaging method of the imaging unit 103 is not limited.

The display device 104 is, for example, a touch panel, and includes a display unit 41 and a touch sensor 42. The touch panel is both a display means and an operation input means. The display unit 41 includes a display driver and displays a display screen to the user. The display unit 41 is, for example, a liquid crystal display unit, but may be an organic EL display unit or the like. The method of the display unit 41 and the method of the touch sensor 42 are not limited. A plurality of display devices 104 may be provided, or a touch sensor 42 may not be provided. In the first embodiment, in addition to the touch panel which is the main display device 104, a display device for a shooting monitor, which will be described later, is provided.

The touch sensor 42 is arranged so as to correspond to the area of the display screen of the display unit 41, and accepts a touch input operation on the display screen. The touch sensor 42 is, for example, a capacitance type, and detects a change in capacitance due to proximity or contact of a finger or the like as an electric signal, and outputs the touch detection signal to the touch detection unit 13 of the control unit 101. The touch panel has, for example, a touch sensor 42 built in a liquid crystal panel display unit, a glass protective cover is arranged on the front side, and a backlight is arranged on the back side. The display device 104 may be controlled to be automatically turned off by turning off the backlight based on a non-use state or the like for a certain period of time. The touch panel may be a pressure sensitive type or the like.

The touch detection unit 13 detects the presence / absence of touch on the display screen, the touch position coordinates, and the like based on the touch detection signal obtained from the touch sensor 42, and also detects operations such as tapping, swiping, and pinching. The user can perform various operations such as inputting an instruction to the image pickup apparatus 1 and focusing on an arbitrary position on the display screen by touch input operation on the display screen of the touch panel.

The interface circuit unit 131 transfers the drive control signal from the control unit 101 to the image pickup unit 103 based on the control of the control unit 101 according to the operation mode. Further, the interface circuit unit 131 outputs the image pickup signal from the image pickup unit 103 to the signal processing unit 132 or the memory 133 based on the control. A plurality of imaging units 103 may be connected to the interface circuit unit 131.

The signal processing unit 132 includes an image / audio signal processing circuit and a coding / decoding circuit, and under the control of the control unit 101, various signal processing such as image signal processing, audio signal processing, coding processing and decoding processing, etc. Has a function to perform. The signal processing of the signal processing unit 132 includes signal processing for the image pickup signal from the image pickup unit 103, signal processing for the image data 23, and signal processing for displaying an image on the display device 104. The signal processing unit 132 generates a video signal for recording or display by signal processing. This video signal includes a still image or a moving image of the image data 23.

The signal processing unit 132 performs signal processing such as filtering, amplification according to the sensitivity setting, white balance correction, and the like on the image pickup signal, for example. When recording the image data 23 as the image file 24 on the recording medium 107, the signal processing unit 132 performs a predetermined format encoding process on the original data. This coding involves data compression. When reading the image data 23 from the recording medium 107, the signal processing unit 132 performs the corresponding decoding process and restores the original data. When the signal processing unit 132 reproduces and displays an image on the display device 104 based on the image data 23, the signal processing unit 132 performs signal processing for that purpose. The signal processing unit 132 performs audio signal processing on the audio input signal from the microphone 115 and the audio output signal to the speaker 116.

The memory 133 is used as a buffer memory during signal processing by the signal processing unit 132, and stores each signal and data. The memory 133 is composed of, for example, a DRAM, a flash memory, or the like.

The operation input unit 105 includes various hardware buttons and the like that enable the user to perform input operations. The user can input instructions through the operation input unit 105, and can set the angle of view-related information and the like to the setting unit 14.

The setting unit 14 accepts settings input by the user through the operation input unit 105 or the touch panel, and sets information related to shooting and display. The setting unit 14 can set the electronic zoom magnification, focal length, angle of view, etc. of the lens of the imaging unit 103 as setting information related to shooting. The image pickup control unit 11 drives and controls the image pickup unit 103 so as to automatically adjust the angle of view and the like based on the setting information of the setting unit 14.

The recording unit 106 includes a recording / reproducing circuit and the like, and a recording medium 107 is detachably provided. The recording medium 107 is, for example, an SD card. Under the control of the control unit 101, the recording unit 106 records the image file 24 including the image data 23 such as the storage unit 102 or the memory 133 on the recording medium 107. Based on the control of the control unit 101, the recording unit 106 reads the image file 24 from the recording medium 107 and stores it in the storage unit 102, the memory 133, or the like. The user can remove the recording medium 107 and carry or replace it. The image file 24 may include audio data together with the image data 23.

The communication interface unit 108 includes a wireless communication interface unit. Based on the control of the control unit 101, the communication interface unit 108 connects to the communication network 4 via a wireless access point or the like, and performs communication processing with an external device. For example, as shown in FIG. 1, the image pickup device 1 communicates with the server 3 or the display device 2 or the image pickup device 1 other than the own device via the communication network 4, and transmits the image file 24 of the own device, or is external. Receive the image file 24 from the device. At that time, the communication interface unit 108 transfers the communication data of the image file 24. The communication interface unit 108 may be provided with a telephone network communication function, a LAN communication function, a short-range communication function, and the like.

The shooting direction measuring unit 111 includes sensors such as an electronic compass, a gyro sensor, and an acceleration sensor, measures the shooting direction of the imaging device 1 using them, and outputs information on the shooting direction to the control unit 101. The shooting direction measuring unit 111 measures the azimuth angle and the elevation angle as the shooting direction. Further, the shooting direction measuring unit 111 detects the rotation angle representing the inclination of the image at the time of imaging based on the detection of the posture of the imaging device 1. The angle of rotation of an image is represented by the angle formed by the horizontal and vertical lines of the rectangle of the image with respect to the horizontal and vertical directions.

The internal clock 112 measures the current date and time. As a result, the control unit 101 can obtain the shooting date and time. The position detection unit 113 includes a GPS receiving unit and detects the position of the image pickup apparatus 1 by using a signal from the satellite. The position detection unit 113 detects {latitude, longitude, altitude (elevation above sea level)} as position information. Altitude may be omitted. The sensor group 114 is another sensor group, and examples thereof include a proximity sensor and an illuminance sensor. The microphone 115 is an audio input device, and the speaker 116 is an audio output device.

The image pickup control unit 11 controls the image pickup unit 103 and the like at the time of image pickup to capture an image according to the operation mode of the image pickup device 1, and obtains image data 23. The image pickup control unit 11 stores the image data 23 as an image file 24 in the storage unit 102 or the like. The imaging unit 103 and the imaging control unit 11 have a function of enabling imaging of both a wide-angle image and a narrow-angle image. The image pickup control unit 11 controls the zoom of the optical system of the image pickup unit 103. Zoom control is electronic zoom or optical zoom control. The image pickup control unit 11 grasps and adjusts the electronic zoom magnification, focal length, angle of view, etc. of the lens of the image pickup unit 103 based on the setting information such as the angle of view related information of the setting unit 14.

When the image pickup unit 103 can be replaced with an optical lens having a different focal length or the like, for example, when shooting a narrow-angle image, the user replaces the optical lens with a relatively long focal length and shoots a wide-angle image. In some cases, replace it with an optical lens with a relatively short focal length. When the user replaces the lens of the imaging unit 103, the setting unit 14 may automatically detect the replacement and detect the focal length of the lens after the replacement. Alternatively, after the replacement, the user may input and set the focal length of the replaced lens and the like to the setting unit 14. Then, the image pickup control unit 11 acquires the updated setting information from the setting unit 14 and grasps the focal length and the like.

The display control unit 12 controls the display unit 41 or the like of the display device 104 to display an image or information on the display screen of the display unit 41. The display control unit 12 displays the folder information and the image on the display screen based on the management information 21 and the image file 24. The display control unit 12 provides various screens that serve as a graphical user interface for the user. The display control unit 12 controls switching of the screen according to the display mode described later. The display control unit 12 also provides a menu screen, a setting screen, and the like.

The association unit 15 cooperates with the image pickup control unit 11 and the display control unit 12 to perform an association process of associating a plurality of images having a predetermined relationship. The association unit 15 performs the association process at a predetermined timing and trigger such as at the time of imaging or reproduction display according to the association method described later. As the first process in the association process, the association unit 15 makes a determination for association with a plurality of image data 23 such as the storage unit 102. In the determination of the first process, an image as a candidate for association is selected based on conditions such as a shooting location according to a selection method described later. As the second process, the association unit 15 determines the first image and the second image to be associated with each of the candidate images based on the conditions of the shooting direction and the angle of view. As a result of the determination, the association unit 15 creates the metadata 22 or the management information 21 so as to associate the first image and the second image.

The display control unit 12 has a predetermined relationship based on the determination result of the association unit 15 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24 at the time of reproducing and displaying the image. One image and a second image are specified, and a plurality of images including the first image and the second image are displayed in a state of being associated with a display screen.

The hardware configuration of each processing unit such as the control unit 101 and the signal processing unit 132 of the image pickup apparatus 1 and each function may be a form in which each processing unit and each function are implemented by circuits such as individual LSIs. It may be implemented by a circuit such as one integrated LSI.

[Display device]
FIG. 3 shows the configuration of the display device 2. The display device 2 does not include an imaging unit 103 and a processing unit related to imaging, as a configuration mainly different from that of the imaging device 1. Of the components of the display device 2, the same components as those of the image pickup device 1 will not be described. The display device 2 includes a control unit 201, a storage unit 202, a display unit 204, an operation input unit 205, a recording unit 206, a recording medium 207, a communication interface unit 208, a sensor group 214, a microphone 215, a speaker 216, and the like. Each of these units is connected to each other via a bus or the like, and cooperates at high speed based on the control of the control unit 201.

The control unit 201 controls the entire display device 2. The control unit 201 includes control circuits such as a processor, ROM, and RAM. The control unit 201 has a display control unit 212 and an association unit 215 as processing units realized by processing the control circuit. The control unit 201 controls the display on the display screen of the display unit 204 based on the user input operation through the operation input unit 205.

The storage unit 202 stores various data and information including a plurality of image files 24 and management information 21 based on the control of the control unit 201. The image file 24 and the management information 21 may be created by the own device or may be acquired from an external device.

The display unit 204 includes a display driver and displays a display screen to the user. The display unit 204 may be a touch panel.

The operation input unit 205 includes various hardware buttons and the like that enable the user to perform an input operation. The operation input unit 205 may include a remote controller and a remote controller light receiving unit. The user can input instructions and set the user through the operation input unit 205.

The recording unit 206 includes a recording / reproducing circuit and the like, and a recording medium 207 is detachably provided. Under the control of the control unit 201, the recording unit 206 records the image file 24 including the image data 23 such as the storage unit 202 or the memory on the recording medium 207. The recording unit 206 reads the image file 24 from the recording medium 207 and stores it in the storage unit 202 or a memory or the like under the control of the control unit 201. The user can remove the recording medium 207 and carry or replace it.

Based on the control of the control unit 201, the communication interface unit 208 connects to the communication network 4 and performs communication processing with an external device. For example, as shown in FIG. 1, the display device 2 communicates with the server 3 or the image pickup device 1 or the display device 2 other than the own device via the communication network 4, transmits the image file 24 of the own device, or is external. Receive the image file 24 from the device. At that time, the communication interface unit 208 transfers the communication data of the image file 24.

The display control unit 212 controls the display unit 204 and the like to display an image or information on the display screen of the display unit 204. The display control unit 212 reproduces and displays the folder information and the image on the display screen of the display unit 204 based on the image file 24 and the management information 21. The display control unit 212 provides various screens that serve as a graphical user interface for the user. The display control unit 212 controls switching of the screen according to the display mode.

The association unit 215 cooperates with the display control unit 212 to perform an association process of associating a plurality of images having a predetermined relationship. The association unit 215 can also perform an association process on the image file 24 acquired from an external device. The association process of the association unit 215 is the same as the association process of the association unit 15. When the first image and the second image are associated with each other as a result of the determination, the association unit 215 describes the association information in the metadata 22 of the image file 24 of the first image and the second image or the management information 21.

The display control unit 212 has a predetermined relationship based on the determination result of the association unit 215 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24 at the time of reproducing and displaying the image. One image and a second image are specified, and a plurality of images including the first image and the second image are displayed in a state of being associated with a display screen.

Hereinafter, the image pickup device 1 will be mainly described, but the functions and operations other than the functions and operations related to the image pickup can be similarly realized by the display device 2 as the main body.

[Imaging device-appearance]
FIG. 4 shows a configuration example of the appearance of the image pickup apparatus 1. The left side of FIG. 4 shows the configuration seen from the back surface of the housing 400 of the imaging device 1, and the right side shows the configuration seen from the right side surface of the housing 400. The back surface is the surface of the display unit 41 on which the display screen 401 is provided, and the front surface is the surface of the image pickup unit 103 on which the lens 402 is provided.

In the side configuration, the lens 402 of the lens unit 32 is arranged at the front center position of the housing 400. Further, the electronic viewfinder (EVF) 403 is provided so as to project upward from the main housing 400, for example. EVF403 is one of the display devices 104.

In the rear configuration, a rectangular display screen 401 corresponding to the above-mentioned touch panel is arranged, and information display and touch input operation are possible. The EVF 403 also has a display screen 404 and a proximity sensor 405. The EVF 403 displays the image of the subject as a shooting monitor image based on the signal input via the lens and the image sensor at the time of shooting. The user can shoot while checking the shooting monitor image of the EVF 403. Information such as the aperture value may be superimposed and displayed on the display screen 404. The proximity sensor 405 detects the proximity of an object. The proximity sensor 405 includes a capacitance type, an optical type, and the like. For example, the control unit 101 determines whether or not the EVF 403 is used by the user by detecting the presence or absence of proximity through the proximity sensor 405. The control unit 101 automatically switches between the display on the display screen 401 and the display of the captured monitor image on the display screen 404 of the EVF 403 according to the determination. The EVF 403 is not limited to the EVF 403, and an optical finder may be mounted. In addition, a mode for displaying a shooting monitor image on the display screen 401 can also be set.

The housing 400 is provided with a mode setting dial 431, a shutter button 432, a cross key 433, a playback display mode button 434, a power button 435, and the like as an operation input unit 105. The cross key 433 includes an up / down / left / right button and a center button, and can be used for inputting information such as selection, determination, and cancellation of a desired position in the display screen 401. On the display screen 401, menu display is also possible based on a predetermined input operation.

The mode setting dial 431 is used to set the operation mode. The shutter button 432 is pressed during shooting. The reproduction display mode button 434 is used when reproducing and displaying on the display screen 401, and the display mode can be switched. The power button 435 is used for power on / off and the like. The user can perform various input operations such as displaying a shooting monitor image on the display screen 401 or the display screen 404 and touching a desired position to focus.

[Imaging device-operation mode]
The operation mode of the image pickup apparatus 1 includes a shooting mode, a reproduction display mode, and the like. The outline of the operation of the shooting mode is as follows. When the user presses the power button 435 to turn on the power, the normal shooting mode is activated. The control unit 101 drives and controls the image pickup unit 103 and the like according to the normal shooting mode. In the normal shooting mode, it is possible to shoot a still image or a moving image as an image. At the time of shooting, the control unit 101 reads an imaging signal from the imaging unit 103 at a predetermined cycle, performs a predetermined signal processing by the signal processing unit 132 via the interface circuit unit 131, and converts it into a video signal in a display format. .. Then, the control unit 101 displays the captured monitor image in real time on the display screen 401 of the touch panel of the display device 104 or the display screen 404 of the EVF 403 based on the video signal. The user shoots as follows while checking the shooting monitor image on the display screen 401 or the display screen 404.

The user sets the angle of view-related information and the like as shooting conditions in the setting unit 14 through the operation input unit 105. When taking a still image, the user presses the shutter button 432. The image pickup control unit 11 detects the pressing of the shutter button 432 and determines the aperture value, shutter speed, focus, and the like according to the set shooting conditions. The shutter button 432 has, for example, a half-pressed state and a fully-pressed state. The image pickup control unit 11 sets the focus or the like when it is in the half-pressed state, and controls to start imaging when it is in the full-pressed state.

The image pickup signal of the still image captured by the image pickup unit 103 is subjected to predetermined signal processing for the still image by using the signal processing unit 132 and the memory 133. For example, JPEG is applied as the coding method at that time, but MPEG, a high-quality RAW format, or the like may be applied as another method. The image file 24 corresponding to the encoded still image data is recorded on the recording medium 107 by, for example, the recording unit 106.

When shooting a moving image, the user presses the movie shooting button to start movie shooting, and presses the movie shooting button again to stop movie shooting. The image pickup signal of the moving image is subjected to signal processing of a predetermined method for moving images. As the method, for example, MPEG such as H264 and H265, and other formats are applied. The encoded moving image data is similarly recorded on the recording medium 107, for example.

The outline of the operation of the playback display mode is as follows. When the control unit 101 detects that the playback display mode has been set by the playback display mode button 434, the control unit 101 reads the image file 24 from the recording medium 107 of the recording unit 106 and stores it in the storage unit 102. The display control unit 12 restores the original data of the image from the image data 23 by the decoding process using the signal processing unit 132, and generates video data for displaying the image or the like on the display screen. The display control unit 12 controls the display device 104 based on the video data, and displays an image or the like on the display screen.

[Management information and image files]
FIG. 5 shows a configuration example of the management information 21 and the image file 24 of the storage unit 102. The management information 21 includes image file management information and association management information. The image file management information is information managed by a known file system, and includes, for example, information such as a file name, an update date and time, a type, and a size. The association management information is unique information created for managing the relationship between a plurality of images. The association management information includes, for example, a type identifier, a reference identifier, and a setting character string.

The type identifier is an identifier for identifying the type of the image for each image file 24. There are two types of images, for example, a wide-angle image and a narrow-angle image. The reference identifier is an identifier for referencing an association between a plurality of images. The reference identifier is, for example, the file name of the image file 24 of the referenced second image as the reference identifier of the first image when there is a first image and a second image associated with the first image. Further, as the reference identifier of the second image, it is the file name of the image file 24 of the first image of the reference source. The setting character string is a character string that can be arbitrarily set by the user based on the user operation, which will be described later. When editing and managing an image, the user can set, for example, a character string representing a shooting location for the image. Although the setting character string is provided in the association management information, the setting character string may be provided in the metadata 22.

The image file 24 includes metadata 22 and image data 23. The header portion of the image file 24 has metadata 22, and the body portion has image data 23. The image data 23 is still image or moving image data, and is encoded data in the case of recording. The metadata 22 is data for managing information including various attribute values related to the image data 23, in other words, ancillary information and attribute information. The metadata 22 includes, as information items, a shooting date / time, a shooting location, a model, a resolution, a shooting direction, an angle of view, an angle of view related information, an angle of rotation, and a thumbnail.

The shooting date and time is, for example, the year, month, day, hour, minute, and second. The shooting location includes location information or a setting character string. The position information includes the position coordinates {latitude, longitude, altitude} detected by the position detection unit 113. The setting character string is a character string representing a shooting location set by the user. The model is information that uniquely represents the type of the image pickup apparatus 1. The model may include a lens type and the like. The resolution represents the resolution of the image of the image data 23.

The shooting direction is information including an azimuth angle (referred to as θ) and an elevation angle (referred to as φ). The angle of view is information including a horizontal angle of view (referred to as Ah) and a vertical angle of view (referred to as Av). The angle-of-view-related information includes information such as an electronic zoom magnification (n), a focal length (f), and the dimensions of the image pickup surface of the image sensor. The dimensions are {horizontal size sw, vertical size sh, diagonal size sd}. In the metadata 22, only one of the angle of view item and the angle of view related information item may be provided.

In other words, the rotation angle is the rotation about the optical axis of the lens at the time of imaging, and is an angle representing the rotation of the image at the time of imaging. For example, as shown in FIG. 4, the angle and direction are based on a state in which the lateral directions of the housing 400 coincide with each other with respect to the horizontal plane.

The thumbnail is a reduced image based on the original data of the image. Thumbnails are configured according to the resolution, type, etc. of the image, and are recorded with or without coding. When thumbnails are created and retained in advance, thumbnails can be displayed at high speed during playback display. As for thumbnails, a plurality of types of thumbnails may be created and retained at different resolutions and the like. The thumbnail may be omitted or may be managed separately from the metadata 22. The display control unit 12 may create and display a thumbnail each time the playback is displayed.

The metadata 22 is an extended format based on a predetermined format, for example, Exif (Exchangeable image file format), but is not limited to this. Exif is a format for photography in digital cameras. In Exif, model and shooting condition information is embedded in image data as metadata. Image display and image editing software can refer to this metadata, and the user can edit the metadata information. The metadata of the conventional Exif image file does not include information such as the angle of view. In the first embodiment, as an extended format, items such as an angle of view and angle-of-view related information are provided as specific information items in the metadata 22.

The metadata 22 is created and managed as a header portion of the image file 24, but may be created and managed as a metadata file separately from the image data file. Further, the form is not limited to the form of managing specific information using the metadata 22, and the form of collectively managing the same information in the management information 21 may be used. In the case of moving images, various information such as metadata 22 may be described for each image frame at a predetermined time interval, for example. The time interval may be constant, or only when there is a change in the angle of view or the like.

[Management of relationships between multiple images]
FIG. 6 shows association management regarding the relationship between a plurality of images. The control unit 101 manages the relationship as shown in FIG. 6 in the association management information of the management information 21 of FIG. 5, for example. In FIG. 6, the first image has a wide-angle type, and the second image has a narrow-angle type. In this example, the first image has a first wide-angle image, a second wide-angle image, and the like, and the second image has a first narrow-angle image, a second narrow-angle image, and the like. For example, a plurality of second images such as a first narrow-angle image, a second narrow-angle image, and a third narrow-angle image are associated with the first wide-angle image. As the association information, for example, an image file name such as the first narrow-angle image is provided as a reference identifier given to the image file of the first wide-angle image. Further, as a reference identifier to be given to the image file of the first narrow-angle image, the image file name of the first wide-angle image is provided. As a result, the images having a relationship can be referred to each other.

As one display mode, information representing the relationship between a plurality of images as shown in FIG. 6 may be displayed on the display screen. At that time, as shown in FIG. 6, the first image and the second image may be displayed in a tree structure, or a state in which the nodes of the second image are connected by a link line around the nodes of the first image (not shown). It may be displayed with.

In the first embodiment, a wide angle and a narrow angle regarding the angle of view are distinguished as the type of the image. The image pickup apparatus 1 distinguishes the angle of view by an absolute value based on the threshold value. The first angle of view is set as a threshold value for the angle of view. When the angle of view of the captured image is equal to or greater than the first angle of view, the control unit 101 sets it as a wide-angle image as a type, and when it is less than the first angle of view, sets it as a narrow-angle image. Not limited to this, three or more types of angles of view may be provided.

[Image shooting direction and angle of view]
FIG. 7 is an explanatory diagram that simply shows the concept of the relationship between the shooting direction and the angle of view of the wide-angle image and the narrow-angle image, which are a plurality of images used as specific examples in the image pickup apparatus 1 of the first embodiment. FIG. 7 shows, in particular, the azimuth angle and the horizontal angle of view.

It has a wide-angle image 6a which is a first image, a first narrow-angle image 6b which is a second image, and a second narrow-angle image 6c. These images are taken by the user using the image pickup apparatus 1 at a point P0, which is a shooting point at the same place. It is assumed that each image was taken in the order of the wide-angle image 6a, the first narrow-angle image 6b, and the second narrow-angle image 6c at the closest date and time. The shooting dates and times of each image are, for example, the wide-angle image 6a at 13:11:30 on January 5, 2016, the first narrow-angle image 6b at 13:14:40 on the same day, and the second narrow-angle image 6c on the same day. It is set to 13:15:50.

The wide-angle image 6a has a shooting range 601 corresponding to the shooting direction 63a indicated by the azimuth angle θ1 and the horizontal angle of view Ah1. The first narrow-angle image 6b has a photographing range 602 corresponding to a photographing direction 63b indicated by an azimuth angle θ2 and a horizontal angle of view Ah2. The second narrow-angle image 6c has a photographing range 603 corresponding to the photographing direction 63c indicated by the azimuth angle θ3 and the horizontal angle of view Ah3. In FIG. 7, among the shooting directions of the image, the shooting direction defined by the azimuth angle θ is indicated by the alternate long and short dash line.

The point P0 particularly indicates the center point of the lens 402 of the imaging unit 103. The sphere 600 represents a spherical surface of the whole celestial sphere, and is a virtual sphere obtained by abstracting the point at infinity. The spherical surface 600 particularly indicates a portion of the circumference corresponding to the azimuth angle θ and the XY plane. N indicates north, E indicates east, S indicates south, and W indicates west. The X direction corresponds to the east direction, the Y direction corresponds to the north direction, and the Z direction corresponds to the zenith direction. The azimuth angle θ is shown as + 90 ° for E, + 180 ° for S, and + 270 ° (−90 °) for W, with N as the reference 0 ° and clockwise as the positive direction.

The photographing range 601 indicated by the arc on the spherical surface 600 indicates the portion of the wide-angle image 6a corresponding to the horizontal angle of view Ah1, and corresponds to the region projected on the spherical surface 600 as seen from the point P0. In this example, a case where there are mountains 611, the sky, towers 612, buildings 613, and the like as landscapes projected on the wide-angle image 6a on the spherical surface 600 is simply shown. The first narrow-angle image 6b is an example of photographing the tower 612, and the second narrow-angle image 6c is an example of photographing the building 613.

Specific examples of the values of the azimuth angle θ, the elevation angle φ, the horizontal angle of view Ah, and the vertical angle of view Av of each image are as follows. Azimuth θ1, elevation φ1, horizontal angle of view Ah1, vertical angle of view Av1 of the wide angle image 6a, azimuth θ2, elevation φ2, horizontal angle of view Ah2, vertical angle of view Av2 of the first narrow angle image 6b, second narrow The azimuth angle θ3, the elevation angle φ3, the horizontal angle of view Ah3, and the vertical angle of view Av3 of the angular image 6c are assumed.
θ1 = + 10 °, φ1 = 0 °, Ah1 = 120 °, Av1 = 80 °
θ2 = + 45 °, φ2 = 0 °, Ah2 = 21 °, Av2 = 14 °
θ3 = -20 °, φ3 = -10 °, Ah3 = 21 °, Av3 = 14 °

FIG. 8 is an explanatory diagram showing, in particular, the elevation angle φ and the vertical angle of view Av, as in FIG. 7. In FIG. 8, another narrow-angle image 6d is shown as an example. In FIG. 8, the X direction corresponds to the horizontal direction and the Z direction corresponds to the vertical direction. The positive direction in the Z direction corresponds to the head and zenith directions, and the negative direction corresponds to the foot direction. The elevation angle φ is 0 ° with respect to the horizontal direction, + 90 ° in the zenith direction in the Z direction, and −90 ° in the foot direction. The shooting range 801 corresponding to the vertical angle of view Av4 in the narrow-angle image 6d is shown. The photographing direction 63d of the narrow-angle image 6d indicates a portion corresponding to the elevation angle φ4.

[Angle of view and angle-of-view related information]
FIG. 9 briefly shows the angle of view and the angle-of-view related information. Further, FIG. 9 shows a case where the angle of view and the angle-of-view related information are stored as the setting information in the setting unit 14. {Horizontal angle of view Ah, vertical angle of view Av} is shown as the angle of view of the image. Further, the horizontal size Sh corresponding to the horizontal angle of view Ah and the vertical size Sv corresponding to the vertical angle of view Av in the image are shown. The horizontal size Sh is the size from the right end to the left end in the horizontal direction in the image frame. The vertical size Sv is the size from the upper end to the lower end in the vertical direction in the image frame. In the first embodiment, the horizontal angle of view Ah and the vertical angle of view Av are used, but the present invention is not limited to this, and at least one angle of view including the diagonal angle of view Ad may be used.

As the angle-of-view related information, the focal length f of the lens in consideration of the optical zoom, the electronic zoom magnification n, and the dimensions of the image pickup surface of the image sensor {horizontal size sw, vertical size sh, diagonal size sd} are used. A model may be used instead of the dimensions. The focal length f is a 35 mm equivalent focal length.

The angle of view can also be calculated and obtained based on the relational expression using the angle of view-related information. If the control unit 101 has sufficient angle-of-view-related information to calculate the angle of view, the control unit 101 may calculate and obtain the angle of view from the angle-of-view-related information. The angle of view can be calculated using, for example, the dimensions of the image sensor, the focal length f, and the electronic zoom magnification n.

When calculating the angle of view from the angle of view related information, it is as follows. The image pickup control unit 11 refers to the angle of view related information regarding the image. The image pickup control unit 11 can grasp the angle-of-view related information from the setting information of the setting unit 14 described above, or the information described in the metadata 22 or the management information 21. The angle of view {horizontal angle of view Ah, vertical angle of view Av, diagonal angle of view Ad} can be calculated based on, for example, the following equation.
Ah = 2 × arctan (sw / (2 × n × f)) ・ ・ ・ (1)
Av = 2 × arctan (sh / (2 × n × f)) ・ ・ ・ (2)
Ad = 2 × arctan (sd / (2 × n × f)) ・ ・ ・ (3)

The focal length f is usually expressed in the 35 mm equivalent. This corresponds to satisfying the above equation when the size of the image sensor is 35 mm format {sw = 35 mm, sh = 24 mm}.

For example, when the electronic zoom magnification n, the focal length f, and the dimensions {horizontal size sw, vertical size sh} are given, the horizontal angle of view Ah1 and the vertical angle of view Av1 are determined. The control unit 101 describes the angle of view information in the angle of view item of the metadata 22 or the management information 21. Further, since the angle of view can be calculated from the angle of view related information, the image pickup apparatus 1 may save the angle of view related information as one of the metadata 22 and the like instead of saving the angle of view. .. Further, since the size of the image sensor of the angle of view related information can be grasped from the model, the image sensor 1 can save the model as one of the metadata 22 and the like instead of the size of the image sensor. Good. For example, the image pickup apparatus 1 holds information indicating the correspondence between the model and the angle-of-view-related information in advance. Alternatively, the imaging device 1 can refer to information representing the correspondence relationship from an external device by communication. The image pickup apparatus 1 can obtain angle-of-view-related information from the model based on the information.

[Save angle of view, etc.]
When the image pickup device 1 captures an image, it creates information including a shooting date and time, a shooting direction {azimuth θ, elevation angle φ}, an angle of view {horizontal angle of view Ah, vertical angle of view Av}, and the like. The image pickup control unit 11 obtains the shooting date and time by the internal clock 112. The image pickup control unit 11 obtains position information {latitude, longitude, altitude} by the position detection unit 113. The image pickup control unit 11 obtains the shooting direction {azimuth θ and elevation angle φ} by the shooting direction measuring unit 111. The image pickup control unit 11 obtains an angle of view {horizontal angle of view Ah and vertical angle of view Av} based on the setting information or calculation of the setting unit 14. Then, the image pickup apparatus 1 describes the information in the corresponding items of the metadata 22 of FIG. 5, and creates the image file 24 together with the image data 23.

[Display screen (1) -Display mode]
FIG. 10 shows the display mode of the display screen 401 of the display unit 41 and its transition. The image pickup apparatus 1 has at least a first display mode, a second display mode, and a third display mode as display modes that are reproduction display modes. The image pickup apparatus 1 initially displays in the first display mode, and switches the display mode based on the user operation.

The first display mode is a mode for displaying a list of information on all images, and is a mode for displaying a folder screen. On the first screen of the first display mode, information on a plurality of images in a folder is displayed in a list in an order using, for example, a file name, a date and time, and the like. In this example, information such as images G1, G2, and G3 are displayed side by side in order from the top as a plurality of images. Each image line displays an image icon or thumbnail, file name, shooting date and time, type, and other information. As other information, attribute values such as the shooting direction {azimuth θ, elevation angle φ} and the angle of view {horizontal angle of view Ah, vertical angle of view Av} may be displayed. The user can scroll and display the information that cannot be displayed on the first screen. The first screen is not limited to such a folder screen, and various known methods can be used. For example, a screen in which thumbnails of a plurality of images are arranged and displayed in parallel in a two-dimensional manner is possible. On the first screen, the user selects and operates a desired first image, for example, image G1 (corresponding to wide-angle image 6a). In that case, the image pickup apparatus 1 transitions to the second screen of the second display mode when the second image associated with the selected first image exists.

The second display mode is a unique mode for performing association display in a plurality of images including a wide-angle image and a narrow-angle image. In the second screen of the second display mode, the wide-angle image 6a, which is the selected first image 901, is displayed as a whole, and one or more second images having a predetermined relationship are displayed on the first image 901. A narrow-angle image is displayed in a state of being superposed and associated with a predetermined figure 902. For example, the figure 902 is a figure representing the first narrow-angle image 6b, and the figure 903 is a figure representing the second narrow-angle image 6c. The broken line frame indicates an area having a corresponding narrow-angle image for the sake of clarity. On the second screen, the display state can be changed by scrolling, scaling, or the like according to the user's operation.

On the second screen, the user selects and operates a desired figure of the second image. The selection operation is, for example, tapping a figure. In that case, the image pickup apparatus 1 transitions to the third screen of the third display mode for displaying the selected second image. Further, when a predetermined operation for returning to the first display mode is performed on the second screen, the image pickup apparatus 1 returns to the first screen of the first display mode. The predetermined operation is, for example, first display mode designation input, pressing the back button, or the like.

The third display mode is a mode for displaying details of a designated single image, for example, a narrow-angle image. In this example, an example of displaying a narrow-angle image which is a single second image 904 on the third screen is shown, but a single wide-angle image can be displayed in the same manner. It is also possible to change the display state by scrolling, enlarging / reducing, or the like according to the user operation on the third screen of the third display mode. When the user performs a predetermined operation on the third screen, for example, a second display mode designation input or a tap on the area of the second image 904, the image pickup apparatus 1 returns to the second screen of the second display mode.

Further, when the user wants to view only the wide-angle image, the image pickup apparatus 1 displays only the first image in detail on the display screen in accordance with a predetermined operation. For example, the image pickup apparatus 1 hides the figure of the second image in response to a predetermined operation on the second screen, thereby displaying only the first image. The image pickup apparatus 1 returns the figure of the second image to the display state again in response to a predetermined operation from that state. Alternatively, the image pickup apparatus 1 transitions to a third screen for displaying the first image in detail in response to a predetermined operation on the second screen, for example, tapping an area on the first image where there is no graphic in the second image. Let me.

[Display screen (2) -Second display mode]
FIG. 11 shows an example of the second screen of the second display mode. This example corresponds to a specific example such as FIG. 7, and shows a case where a wide-angle image 6a is selected and displayed. The case where the first narrow-angle image 6b and the second narrow-angle image 6c are associated with the wide-angle image 6a is shown. The display control unit 12 displays the second screen based on the image file 24 and the management information 21. On the second screen, the wide-angle image 6a, which is the first image, is displayed as a whole, and the figures representing the narrow-angle images, which are the second images, are superimposed and displayed on the first image.

In FIG. 11, the horizontal direction in the screen corresponding to the azimuth angle in the display screen 401 is shown as the x direction, and the vertical direction in the screen corresponding to the elevation angle is shown as the y direction. Let W be the horizontal width of the display screen 401 in the x direction and H be the vertical width in the y direction.

The point p1 indicates the central point of the display screen 401. Initially, the entire wide-angle image 6a is displayed in the display screen 401 in a state where scrolling, enlargement / reduction, or the like is not performed. First, the position coordinates (Xa, Ya) of the center point corresponding to the shooting direction 63a of the wide-angle image 6a are displayed at the point p1. The position coordinates (Xa, Ya) are points corresponding to (θ1 = + 10 °, φ1 = 0 °) in terms of (azimuth, elevation).

The horizontal angle of view Ah1 of the wide-angle image 6a is 120 °, and the vertical angle of view Av1 is 80 °. The wide-angle image 6a is displayed in a horizontal size Sh1 corresponding to a horizontal angle of view Ah1 and a vertical size Sv1 corresponding to a vertical angle of view Av1 corresponding to the horizontal width W and the vertical width H of the display screen 401. The rightmost position Xar corresponds to 70 ° azimuth, the leftmost position Xal corresponds to -50 ° azimuth, the uppermost position Yat corresponds to 40 ° elevation, and the lower end position Yab corresponds to-elevation- Corresponds to 40 °.

A mark, which is a graphic representing a narrow-angle image, is superimposed and displayed on the wide-angle image 6a. In this example, as marks, a mark 8b representing the first narrow-angle image 6b and a mark 8c representing the second narrow-angle image 6c are shown. The mark particularly indicates the case of a balloon shape. The mark is variable based on user settings. The center point p2 indicates a position where the mark 8b representing the first narrow-angle image 6b is displayed. The center point p3 indicates a position where the mark 8c representing the second narrow-angle image 6c is displayed. The position (Xb, Yb) of the center point p2 corresponds to (+ 45 °, 0 °) in (azimuth, elevation angle), and the position (Xc, Yc) of the center point p3 is (-20 °, -10 °). ) Corresponds.

The display control unit 12 determines and displays the display position of the mark of the second image on the first image based on the information such as the shooting direction and the angle of view of the second image associated with the first image.

The positions (Xb, Yb) of the center point p2 of the first narrow-angle image 6b are the difference in azimuth (35 °) and the difference in elevation (35 °) from the point p1 of the wide-angle image 6a when wide-angle distortion and distortion are ignored. 0 °) and can be obtained by the following equation.
Xb = (W / 2) × tan35 ° / tan60 ° ・ ・ ・ (4)
Yb = (H / 2) × tan 0 ° / tan 40 ° = 0 ... (5)

The position (Xc, Yc) of the center point p3 of the second narrow-angle image 6c is the difference between the azimuth angle (-30 °) and the elevation angle from the point p1 of the wide-angle image 6a when wide-angle distortion and distortion are ignored. Using (-10 °), it is obtained by the following formula.
Xc = (W / 2) × tan (-30 °) / tan60 ° ・ ・ ・ (6)
Yc = (H / 2) × tan (-10 °) / tan40 ° ・ ・ ・ (7)

The user selects, for example, taps a desired mark on the display screen 401. As a result, the second image corresponding to the mark can be selected and displayed on the third screen of the third display mode. The example of FIG. 11 shows a case where the entire wide-angle image 6a can be displayed first with respect to the size of the display screen 401. When the entire wide-angle image cannot be displayed on the display screen 401, the display control unit 12 may display a part of the wide-angle image, that is, a part of the angle of view, or reduce the size and display the entire image. You may. Further, in the display screen 401, values such as the shooting direction and the angle of view may be displayed as browsing assistance information, or icons for operations such as scrolling and enlargement / reduction may be displayed.

[Display screen (3) -3rd display mode]
FIG. 12 shows the display of the first narrow-angle image 6b when the mark 8b of the first narrow-angle image 6b is selected as a display example of the third screen in the third display mode. First, the first narrow-angle image 6b is displayed on the entire display screen 401. The center point p1 of the display screen 401 is displayed in a state where the center point p2 of the first narrow-angle image 6b and the mark 8b is arranged. The position (Xb, Yb) of the center point p2 corresponds to {θ2 = 45 °, φ2 = 0 °} in the shooting direction 63b.

The horizontal angle of view Ah2 of the first narrow angle image 6b is 21 °, and the vertical angle of view Av2 is 14 °. On the display screen 401, the rightmost position Xbr corresponds to the azimuth of + 55.5 °, the leftmost position Xbl corresponds to the azimuth of + 34.5 °, the upper end position Ybt corresponds to the elevation angle + 7 °, and the lower end position Ybb corresponds to the elevation angle. Corresponds to -7 °.

FIG. 13 shows the display of the second narrow-angle image 6c when the mark 8c of the second narrow-angle image 6c is selected as an example of the third screen of the third display mode. First, the second narrow-angle image 6c is displayed on the entire display screen 401. The center point p1 of the display screen 401 is displayed in a state where the center point p3 of the second narrow-angle image 6c and the mark 8c is arranged. The position (Xc, Yc) of the center point p3 corresponds to {θ3 = −20 °, φ3 = −10 °} in the imaging direction 63c.

The horizontal angle of view Ah3 of the second narrow-angle image 6c is 21 °, and the vertical angle of view Av3 is 14 °. On the display screen 401, the rightmost position Xcr corresponds to the azimuth of -9.5 °, the leftmost position Xcl corresponds to the azimuth of -30.5 °, the upper end position Yct corresponds to the elevation angle of -3 °, and the lower end position. Ycb corresponds to -17 ° in elevation.

It should be noted that the transition from the second display mode to the third display mode is not limited to this. An operation mode in which the first display mode is directly transitioned to the third display mode by a predetermined operation without going through the second display mode is also possible, and the operation mode can be selected according to user settings and the like.

[Association method]
As the association method, there are a first method and a second method regarding the timing of associating. The first method is a method of associating at the time of imaging, and the second method is a method of associating at the time of reproduction display. In the first embodiment, the imaging device 1 uses the first method, and the display device 2 uses the second method.

In the first method, the imaging device 1 performs the association processing using the association unit 15 immediately after capturing the image, and as a result, the association information is described in the metadata 22 or the management information 21. When the image pickup device 1 reproduces and displays the image, the relationship between the plurality of images can be grasped from the reference of the association information of the metadata 22 or the management information 21, and the image pickup device 1 displays the image on the display screen in the second display mode.

In the second method, when the display device 2 reproduces and displays the image file 24 on the display screen, the display device 2 performs the association process using the association unit 215, and as a result, displays the image file 24 on the display screen in the second display mode.

In the second method, the display device 2 may omit the description of the association information in the metadata 22 and the management information 21 by performing the association processing each time the reproduction display is performed. Alternatively, in the second method, the display device 2 performs the association processing at the first time of the reproduction display of a certain image, describes the association information in the metadata 22 and the management information 21, and then reproduces the image from the second time onward. At the time of display, the association processing may be omitted by referring to the association information.

In the image pickup apparatus 1 of another embodiment, the second method may be adopted. Which association method is adopted may be a form that is fixed in terms of implementation, or may be a form in which the association method can be selected and set according to the user setting.

[Processing flow (1)]
14 and 15 show the processing flow of the image pickup apparatus 1 and the display apparatus 2 of the first embodiment. FIG. 14 shows a flow of control processing at the time of capturing an image in the imaging device 1. FIG. 14 has steps S101-S106. The same process is repeated every time an image is captured. Hereinafter, the steps will be described in order.

(S101) The control unit 101 sets the angle of view and the like for the setting unit 14 based on the user operation. The image pickup control unit 11 grasps the angle of view and the like based on the setting information of the setting unit 14.

(S102) Based on the user operation, the image pickup control unit 11 controls the image pickup unit 103 to take an image, obtains the image data 23, and stores the image data 23 in the storage unit 102. The image pickup control unit 11 describes the angle of view or the angle-of-view-related information in the metadata 22 of the image file 24 or the management information 21.

(S103) Branching is performed depending on whether the first method is associated at the time of imaging or the second method is associated at the time of playback display. In the case of the first method, the process proceeds to S104, and in the case of the second method, the process proceeds to S106. That is, the image pickup apparatus 1 of the first embodiment proceeds to S104 in order to adopt the first method, and the image pickup apparatus 1 of the other embodiment proceeds to S106 when the second method is adopted.

(S104) The association unit 15 performs a predetermined association process. As a result, a plurality of images including the first image and the second image are associated with each other.

(S105) The association unit 15 creates the association information as a result of the association processing, and describes it in the metadata 22 of the image file 24 or the management information 21. The image pickup control unit 11 stores the image file 24 in the storage unit 102 and causes the recording medium 107 of the recording unit 106 to record the image file 24.

(S106) Since the imaging control unit 11 does not associate at this point, the metadata 22 and the management information 21 that do not include the association information are created. The control unit 101 stores the image file 24 in the storage unit 102 and causes the recording medium 107 of the recording unit 106 to record the image file 24.

[Processing flow (2)]
FIG. 15 shows a flow of control processing at the time of reproducing and displaying an image in the image pickup apparatus 1 or the display apparatus 2. The control of FIG. 15 corresponds to the control of transitioning the display mode as shown in FIG. FIG. 15 has steps S201-S210. Hereinafter, the steps will be described in order. Hereinafter, the case where the image pickup device 1 is the main body will be described, but the same applies to the case where the display device 2 is the main body.

(S201) Based on the user operation, the control unit 101 displays the first screen of the first display mode on the display screen 401 of the display unit 41 of the display device 104 by the display control unit 12. The display control unit 12 accepts selection of an image in the first screen or the like in response to a user input operation.

(S202) The display control unit 12 branches depending on whether the image selected on the first screen of the first display mode is a wide-angle image which is the first image or a narrow-angle image which is the second image. If it is the first image, the process proceeds to S203, and if it is the second image, the process proceeds to S209.

(S203) The display control unit 12 branches depending on whether it is the first method or the second method. In the case of the second method, the process proceeds to S204, and in the case of the first method, the process proceeds to S206. That is, in the case of the image pickup apparatus 1 of the first embodiment, the first method is used, so the process proceeds to S206, and in the case of the display device 2, the second method is used, so the process proceeds to S204.

(S204) Immediately before the reproduction display, the display control unit 12 performs the association processing of a plurality of images by using the association unit 15.

(S205) As a result of S204, the display control unit 12 reads out the image data 23 of the second image associated with the first image.

(S206) In the case of the first method, the association process has already been performed. The display control unit 12 confirms the association by referring to the association information in the metadata 22 or the management information 21 of the image file 24 of the first image. The display control unit 12 reads out the image data 23 of the second image associated with the first image.

(S207) The display control unit 12 displays the image data 23 as a result of S205 or S206 on the display screen in the second display mode. That is, the display control unit 12 displays the figures of the second image on the first image in a state of being superposed and associated with each other, as in the second screen of FIG. The display control unit 12 accepts the selection of the figure of the second image in the second screen in response to the user input operation. For example, the user taps the mark when he / she wants to view the second image, and taps the area of the first image when he / she wants to return to the first screen.

(S208) The display control unit 12 branches depending on whether the image selected on the second screen of the second display mode is a mark representing the second image. If it is a mark representing the second image, the process proceeds to S209. If it is another operation, for example, an operation in the area of the first image, it corresponds to the return operation, and in that case, the process returns to S201.

(S209) The display control unit 12 uses the image data 23 of the second image corresponding to the mark selected in S208 in the third display mode on the display screen, as in the third screen of FIG. 2 Display the image in detail. The display control unit 12 accepts a return operation or the like in the third screen in response to a user input operation. When the user wants to return to the second screen, for example, he / she taps the area of the second image.

(S210) The display control unit 12 returns to S207 as shown by A1 when there is an input operation corresponding to the return operation on the third screen of the third display mode, and when there is no input operation, the display control unit 12 returns to S207 when there is no return operation. Continue to display the display mode. The third display mode may be returned to the first screen of the first display mode in response to a predetermined input operation.

[Association process (1)]
The association unit 15 performs an association process for a plurality of images. The association process includes a first process and a second process. The first process is a process of selecting an image as a candidate for association based on the selection conditions of a predetermined selection method. As a predetermined selection method, there are a first selection method, a second selection method, and a third selection method shown below. In the first embodiment, the first method is used. In the modified example, the second selection method or the third selection method is used. It should be noted that which selection method is to be adopted may be a fixed form in terms of mounting, or may be a form that can be variably selected by user setting.

[First selection method-shooting location]
FIG. 16 is an explanatory diagram showing the first selection method. The first selection method uses a condition related to a shooting location as a selection condition. In the first selection method, images that are determined to have substantially the same shooting location are selected as candidates. In the first selection method, the position information {latitude, longitude, altitude} by the position detection unit 113 is used to determine the shooting location. At the time of association, the association unit 15 refers to the position information {latitude, longitude} described in the imaging location item of the metadata 22 at the time of imaging, for example. {Latitude, longitude} is represented by {LA, LO}.

As shown in FIG. 16, the association unit 15 uses the wide-angle image, which is the first image, as a reference, and takes the position 1601 representing the shooting location of the first image as a center point in space, for example, in the XY plane. Position 1601 is represented by {LA1, LO1}. The position range 1600 corresponds to the threshold value of the condition, and indicates a circular region with a predetermined radial distance centered on the position 1601. The radial distance defining the position range 1600 may be a fixed set value in design or a set value that can be changed by user setting.

The association unit 15 compares the position of each narrow-angle image, which is the second image, with respect to the position 1601 of the first image. In this example, around the wide-angle image, the position 1602 {LA2, LO2} of the first narrow-angle image, the position 1603 {LA3, LO3} of the second narrow-angle image, and the position 1604 {LA4, LO4 of the third narrow-angle image. } Indicates the case. The association unit 15 searches for an image included in the position range 1600 with respect to the position 1601 of the first image. In this example, the position 1602 of the first narrow-angle image and the position 1603 of the second narrow-angle image are included in the position range 1600, and the position 1604 of the third narrow-angle image is not included.

The association unit 15 determines that the second image whose position is included in the position range 1600 is an image captured at substantially the same shooting location as the first image, and selects it as an association candidate. An example of the same shooting location is a case where a user shoots in each direction of 360 ° while orbiting a circumference along a handrail at an observatory.

[Second selection method-shooting date and time]
FIG. 17 is an explanatory diagram showing the second selection method. In the second selection method, a condition related to the shooting date and time is used as the selection condition. In the second selection method, images that are determined to have substantially the same shooting date and time are selected as candidates. At the time of association, the association unit 15 refers to, for example, the shooting date / time (year / month / day / hour / minute / second) described in the shooting date / time item of the metadata 22.

The association unit 15 takes a time point t1 representing the shooting date and time of the first image with reference to the wide-angle image which is the first image in the time series of the shooting date and time. In this example, the time point t1 is assumed to be the date and time DT3. The date and time range 1700 corresponds to the threshold value of the condition, and indicates a range with a predetermined time including the time point t1 as the center point and before and after the time point t1. The time that defines the date and time range 1700 may be a fixed setting value by design or a setting value that can be changed by user setting. The date and time range 1700 can be set, for example, one day (24 hours), one hour, five minutes, and the like.

The association unit 15 searches for the second image included in the predetermined date and time range 1700 with respect to the time point t1 of the shooting date and time of the first image. The association unit 15 compares the shooting date and time of each narrow-angle image which is the second image with respect to the time point t1 of the shooting date and time of the first image. In this example, there is a case where there is a time point t2 of the shooting date and time of the first narrow angle image, a time point t3 of the second narrow angle image, and a time point t4 of the third narrow angle image in the vicinity of the time point t1. In this example, the time point t2 of the first narrow angle image and the time point t3 of the second narrow angle image are included in the date and time range 1700, and the time point t4 of the third narrow angle image is not included. The control unit 101 determines that the second image whose time point is included in the date and time range 1700 is an image captured at substantially the same shooting date and time as the first image, and selects it as an association candidate.

[Third selection method-setting character string]
FIG. 18 is an explanatory diagram showing the third selection method. The third selection method uses as a selection condition a condition related to a setting character string that can be arbitrarily set by the user. In the third selection method, images that are determined to have substantially the same set character strings are selected as candidates. At the time of association, the association unit 15 refers to, for example, a character string set as a setting character string of the association management information after imaging.

FIG. 18 shows an example of displaying a list of information of a plurality of images on the first screen of the first display mode. In this example, the file name, shooting date / time, tag, type, etc. are displayed as information for each image. The "tag" item corresponds to the setting character string. The user can arbitrarily set a character string in the "tag" item in the editing work. In this example, the user sets a character string representing the shooting location in the "tag" item. For example, the images G1, G2, and G3 are images taken in the same park, and the character string "A park" is set. The images G4 and G5 are images taken at the same observatory, and the character string "B observatory" is set. The "type" item displays "wide angle" or "narrow angle" as the type of the angle of view of the image.

The association unit 15 refers to the setting character string of the "tag" item of the first image with reference to the first image selected by the user, for example, the image G1. The associating unit 15 compares the setting character string of the first image with reference to the setting character string of each narrow angle image which is another second image, and for example, sets a setting character string containing the same character string. Find the image you have. For example, the setting character strings of the images G1, G2, and G3 are the same. The association unit 15 determines that these images are images captured at the same shooting location, and selects them as association candidates.

As a modification, the control unit 101 may automatically set a character string representing a shooting location at the time of imaging or the like without depending on the user operation. For example, the control unit 101 may determine a character string representing a shooting location based on the position information {latitude, longitude}. The control unit 101 may set a character string in the "file name" item or the like of the image. The control unit 101 may automatically assign a file name including a setting character string according to a predetermined rule, for example. Examples of file name rules include [shooting date and time] + [setting character string (shooting location)] + [individual sequence number].

The above selection method can also be applied in combination. For example, the condition may be a combination of the condition of the shooting location of the first selection method and the condition of the shooting date and time of the second selection method. As the condition, for example, "up to the past year at the same shooting location" can be specified.

[Association process (2)]
The second process in the association process of the association unit 15 is as follows. In the second process, the image selected as a candidate in the first process is determined to be associated with the image selected as a candidate by using the information of the shooting direction and the angle of view, and the association is performed according to the determination result. The association unit 15 refers to the information on the shooting direction and the angle of view in the metadata 22 of each image file 24 or the management information 21. As a condition, the associating unit 15 sets the shooting direction {azimuth θ, elevation φφ of the candidate image within the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av} of the wide-angle image which is the reference first image. } Is included or not. The association unit 15 determines an image satisfying this condition as an image to be associated.

In the above-mentioned example of FIG. 7, the azimuth angle θ2 = 45 ° in the shooting direction 63b of the first narrow-angle image 6b is included in the range (-50 ° to + 70 °) of the horizontal angle of view Ah1 of the wide-angle image 6a. The second narrow-angle image 6c includes an azimuth angle θ3 = −20 ° in the photographing direction 63c. Further, although not shown, the elevation angle φ2 = 0 ° of the first narrow-angle image 6b in the shooting direction 63b is included in the range (-40 ° to + 40 °) of the vertical angle of view Av1 of the wide-angle image 6a. 2 The elevation angle φ3 = −10 ° in the photographing direction 63c of the narrow-angle image 6c is included. Therefore, the wide-angle image 6a is associated with the first narrow-angle image 6b and the second narrow-angle image 6c.

When associating the first image with the second image, the association unit 15 may describe the association management information in the management information 21 as described above, or describe the same information in the metadata 22 of each image. You may.

[Video]
In the above, the case of still images has been mainly described, but basically the same can be applied to moving images. When the image pickup control unit 11 shoots a moving image, the image data 23 of the moving image is obtained by taking a picture while changing the electronic zoom magnification, the angle of view, and the like in the continuous shooting mode. The image pickup control unit 11 describes information such as the electronic zoom magnification and the angle of view according to the time point in the metadata 22 or the management information 21 for the image file 24 of the moving image. Further, when playing back and displaying a moving image, the display control unit 12 displays a mark or the like on the second screen of the second display mode, for example, in correspondence with the position of the image frame at the start of the moving image or the image frame in the paused state. May be displayed.

[Effects, etc.]
As described above, according to the imaging device 1 and the display device 2 of the first embodiment, it is easy to grasp the relationship between a plurality of images including a wide-angle image and a narrow-angle image, and better usability can be realized. On the display screen, the narrow-angle image having a relationship is automatically superimposed on the wide-angle image, so that the user can easily recognize a plurality of related images and intuitively grasp them, which is suitable. Can be viewed at. The user can recognize the narrow-angle image existing within the angle of view of the wide-angle image, and can quickly browse by referring to the associated narrow-angle image. The user can suitably browse a large number of images while switching the display mode.

The following can be mentioned as a modification of the first embodiment. As a modification, the relationship between the angles of view may be relatively determined with respect to the type of image. That is, the image pickup device 1 or the display device 2 refers to the angles of view in a plurality of images, and the image having the largest angle of view among them is set as the first image which is a wide-angle image, and images other than the first image. Is a second image which is a narrow angle image.

[Modification example (1) -second display mode]
FIG. 19 shows an example of the second screen of the second display mode in the image pickup apparatus 1 of the modified example of the first embodiment. FIG. 19 corresponds to the specific example of FIG. In the display screen 401 of FIG. 19, the figure representing the second image is not a mark but a transparent frame. For example, the transparent frame 9b represents the first narrow-angle image 6b, and the transparent frame 9c represents the second narrow-angle image 6c. In the transparent frame, the four sides of the rectangle are displayed in lines and colors, the inside thereof is a transparent area, and the portion of the first image in the background is displayed as it is. The size of the transparent frame area on the display is determined in particular according to the size of the angle of view of the second image. This makes it easier to understand the relationship between the narrow-angle images in the wide-angle image.

In the example of FIG. 19, the wide-angle image 6a is first displayed on the display screen 401 at the same reference position as that of FIG. 11, and then scrolling and enlargement / reduction are performed based on the user operation to obtain the center point on the display. It shows the state where the angle of view range has been moved or changed. In FIG. 19, a point corresponding to (+ 15 °, + 10 °) in (azimuth, elevation) as the position (Xa, Ya) of the viewpoint of the wide-angle image 6a is arranged at the point p1 in the center of the display screen. .. On the display screen 401, a 90 ° range of the 120 ° range of the horizontal angle of view of the wide-angle image 6a and a 60 ° range of the 80 ° range of the elevation angle are displayed in an enlarged manner. Position Xa is 15 ° azimuth and position Ya is 10 ° elevation. On the display screen 401, the rightmost position Xar corresponds to 60 °, the leftmost position Xal corresponds to −30 °, the upper end position Yat corresponds to 40 °, and the lower end position Yab corresponds to −20 °.

The position coordinates (Xb, Yb) of the center point p2 of the transparent frame 9b correspond to (+ 45 °, 0 °) in (azimuth, elevation). The width of the transmission frame 9b is a size corresponding to the horizontal angle of view Ah2 = 21 °, is in the range of 45 ° ± 10.5 ° in the azimuth angle, the leftmost position Xbl is + 34.5 °, and the rightmost position Xbr. Is + 55.5 °. The vertical width of the transparent frame 9b is a size corresponding to the vertical angle of view Av2 = 14 °, is in the range of 0 ° ± 7 ° in elevation angle, the upper end position Ybt is + 7 °, and the lower end position Ybb is -7 °. Is.

The position and size of the transparent frame 9c are also determined in the same manner. The position coordinates (Xc, Yc) of the center point p3 of the transparent frame 9c correspond to (-20 °, -10 °). The width of the transmission frame 9c is a size corresponding to the horizontal angle of view Ah3 = 21 °, the azimuth is in the range of -20 ° ± 10.5 °, the left end position Xcl is -30.5 °, and the right end. The position Xcr is -9.5 °. The vertical width of the transparent frame 9c is a size corresponding to the vertical angle of view Av3 = 14 °, is in the range of -10 ° ± 7 ° in elevation angle, the upper end position Yct is -3 °, and the lower end position Ycc is-. It is 17 °.

The display position {Xbr, Xbl, Ybt, Ybb} of the transmission frame 9b and the display position {Xcr, Xcl, Yct, Ycc} of the transmission frame 9c can be obtained by the following equations when wide-angle distortion and distortion are ignored. The coefficient Kw and the coefficient Kh are used.

Xbr = (W / 2) -Kw (tan50 ° -tan55.5 °) ... (8)
Xbl = (W / 2) -Kw (tan50 ° -tan34.5 °) ... (9)
Kw = W / (tan50 ° + tan40 °) ・ ・ ・ (10)
Ybt = (H / 2) -Kh (tan40 ° -tan7 °) ... (11)
Ybb = (H / 2) -Kh (tan40 ° -tan (-7 °)) ... (12)
Kh = H / (tan40 ° + tan20 °) ・ ・ ・ (13)
Xcr = (W / 2) -Kw (tan 50 ° -tan (-9.5 °)) ... (14)
Xcl = (W / 2) -Kw (tan50 ° -tan (-30.5 °)) ... (15)
Yct = (H / 2) -Kh (tan40 ° -tan (-3 °)) ... (16)
Ycb = (H / 2) -Kh (tan 40 ° -tan (-17 °)) ... (17)
Further, in the example of FIG. 19, an example relating to the rotation angle of the image is shown by the transparent frame 9e representing the narrow angle image at the point p5. This narrow angle image has an angle α with respect to the x direction.

The display control unit 12 determines the size of the transparent frame by using the information of the shooting direction and the angle of view of the image of the image data 23. The display control unit 12 determines the position and size of the transparent frame so as to correspond to the display state after the change when scrolling or enlargement / reduction corresponding to the viewpoint change in the display of the wide-angle image is performed.

This modification has the advantage that the range and positional relationship of the narrow-angle image can be easily understood by displaying the transparent frame. As another modification, thumbnails may be displayed in the same size instead of the transparent frame.

[Modification example (2) -second display mode]
FIG. 20 shows an example of a display screen in the image pickup apparatus 1 of the modified example of the first embodiment. An example of the first screen of the first display mode is shown on the upper side of FIG. 20, and on this first screen, thumbnails of wide-angle images and thumbnails of narrow-angle images are listed in a vertical and horizontal parallel arrangement. At that time, the size of the thumbnail on the display is determined to be the vertical and horizontal sizes according to the size of the angle of view of the image. The thumbnail 2001 of the wide-angle image of this example shows a case where the thumbnail of the panoramic image has a large horizontal angle of view of 180 ° or more. The thumbnail 2002 of the narrow angle image shows the case where the thumbnail of the image has a horizontal angle of view of less than 180 °. The number of images when arranged in parallel is variable, such as 2, 3, 4, or the like in one direction. The user can distinguish between wide-angle and narrow-angle by the size of the thumbnail.

The lower side of FIG. 20 shows an example of the second screen of the second display mode. In the display screen 401, it has a main area 1001 and an area other than the main area 1001, in this example, a right side area 1002 and a lower side area 1003. In the main area 1001, the marks of the second image and the like are superimposed and displayed on the first image in the second display mode. When the entire first image cannot be displayed in the area 1001, a range of a part of the angle of view is displayed first with the shooting direction as the center point. In this example, the range corresponding to a part of the horizontal angle of view of the wide-angle image selected on the first screen is displayed in the area 1001. The user can change the display state of the first image in the area 1001 by scrolling, enlarging / reducing, or the like in response to the viewpoint change or the like.

As the figure of the second image, a different figure may be displayed depending on the type of the image, the shooting date and time, and the like. In this example, circular, rhombic, and star-shaped marks are shown as figures. As the type of the image, for example, the type according to the difference in the angle of view may be set based on the user setting or the like. In addition, information may be added to the figure and displayed. In this example, an identification number is assigned to the mark. Further, as information, the graphic may display an attribute value such as a shooting date and time based on the metadata 22 or the management information 21, or may display only the attribute value.

In the area 1002 on the right side, information of all the plurality of second images associated with the first image, such as marks and attribute values, is displayed in parallel. In the area 1001, some marks of the second image are displayed according to the display state, and the marks of the other second image are not displayed. Information on the second image that is not displayed in the area 1001 is also displayed in the area 1002. When the mark of the second image is selected in the area 1001, or when the mark of the second image is selected in the area 1002, the transition to the third display mode may be performed. Further, in the lower right area, the thumbnail of the second image selected last is displayed as a preview.

In the lower area 1003, a display mode, information on the first image, operation buttons, setting information, and the like are displayed. For example, the current display mode is the second display mode, and information such as attribute values such as the shooting date and time of the first image is displayed. In addition, a button for switching between display and non-display of the mark in the second display mode is provided. In the "ON" state, the mark is displayed, and in the "OFF" state, the mark is hidden, that is, only the first image is displayed. Further, a return button or the like for the operation of returning from the second display mode to the first display mode is provided.

Further, in the second screen of the second display mode, when a plurality of associated second images are densely present in the same position and the same area on the first image, individual marks are used. Is displayed, but one representative mark is displayed. In this example, the representative mark is a star. When the user selects and operates the representative mark, information on a plurality of second images associated with the position of the representative mark is displayed. At that time, as a method of displaying the information of the plurality of second images, for example, the pop-up area 1005 is displayed on the first image, and the information of the plurality of second images is displayed in the pop-up area 1005. When the user selects a desired second image from the pop-up area 1005, the mode shifts to the third display mode in which the selected second image is displayed in detail.

As another method, the information of a plurality of second images associated with the representative mark may be displayed in the area 1002 or the like on the right side. As another method, a plurality of associated second images may be sequentially displayed in a predetermined area, that is, a slide show may be displayed, depending on the selection of the representative mark. For example, when there are a plurality of images taken at the same place, in the same shooting direction, and at different shooting dates and times, those images can be associated and continuously played back and displayed.

Further, as another method, a region in which a plurality of second images are densely displayed may be enlarged and displayed according to a representative mark selection operation so that the marks of the plurality of second images can be distinguished from each other.

[Modification example (3) -Cooperation with external devices]
The image pickup device 1 and the display device 2 may save the image file 24 in an external device, or may read the image file 24 from the external device and display it for reproduction. The image pickup device 1 and the display device 2 may perform the association processing on the image file 24 to be transmitted to the external device, and transmit the image file 24 including the association information to the external device. Further, the image pickup device 1 and the display device 2 may perform the association processing on the image file 24 received from the external device, and reproduce and display the image file 24 in the associated state based on the association information. That is, as an association method other than the above-mentioned first method and second method, a method of associating before transmitting the image file 24 to the external device and a method of associating after receiving the image file 24 from the external device. May be adopted. Further, when the image pickup device 1 and the display device 2 transmit a plurality of image files 24 to an external device, the image angle information and the association information may be described as the metadata 22 for each image file 24, or a plurality of image files 24 may be described. It may be described and provided in a management file different from the image file 24.

For example, in the system of FIG. 1, the imaging device 1 performs an association process on a plurality of captured images before transmitting them to the server 3, creates an image file 24 including an angle of view and association information, and creates a communication network 4. It is transmitted to the server 3 through the image data DB and stored in the image data DB. The image pickup device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. At that time, it is possible to collectively acquire a plurality of image files 24 by designating, for example, a shooting date and time, a shooting location, and the like as search conditions. The image pickup device 1 or the display device 2 reproduces and displays the image data 23 of the acquired plurality of image files 24 in an associated state based on the reference of the association information.

Further, for example, the image pickup apparatus 1 transmits the image file 24 including the angle of view and not including the association information to the server 3 and stores the captured images in the image data DB without performing the association processing. The image pickup device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. The image pickup device 1 or the display device 2 performs an association process on the acquired plurality of image files 24, and reproduces and displays the acquired image files 24 in an associated state based on the angle of view and the association information.

Further, the server 3 may perform the association processing on the plurality of image data 23 of the image data DB. The server 3 performs the association processing and saves it as an image file 24 including the association information. The image pickup device 1 or the display device 2 reproduces and displays a plurality of image files 24 acquired from the server 3 based on the reference of the association information.

Further, for example, the image pickup device 1 performs an association process on a plurality of captured images, and transmits an image file 24 including an angle of view and association information to the display device 2. The display device 2 acquires the image file 24, and reproduces and displays the image data 23 of the acquired image file 24 in an associated state based on the reference of the association information.

Further, for example, the image pickup device 1 transmits the image file 24 including the angle of view and not including the association information to the display device 2 without associating the plurality of captured images. The display device 2 acquires the image file 24, performs an association process on the image data 23 of the acquired image file 24, and reproduces and displays the image data 23 in the associated state based on the association information.

Further, the image pickup device 1 and the display device 2 may read each image file 24 from a plurality of devices of the same user and manage them in an integrated manner. For example, the first image pickup device 1A and the second image pickup device 1B of FIG. 1 transmit the image file 24 of the captured image to the first display device 2A without associating with each other. The first display device 2A performs an association process on the image files 24, and reproduces and displays the image files 24 in the associated state.

According to this modification, the association display function can be realized in a system in which a plurality of devices are linked, and the user can realize association and storage of a large number of images.

[Modification example (4) -Editing function]
The imaging device 1 and the display device 2 of the modified example are provided with an editing function that allows the user to determine the association of a plurality of images and manually set and edit them. The user can add, delete, or change the associated image in the display screen by using the editing function, and can change the display position of each image.

FIG. 21 shows an example of the second screen of the second display mode in the imaging device 1 and the display device 2 of the modified example. The following shows the case of the imaging device 1 having an editing function, but the same applies to the case of the display device 2 having an editing function. It is possible to switch between an "ON" state in which editing is enabled and an "OFF" state in which editing is disabled, based on user settings or input operations. When the image pickup device 1 is in the "ON" state, the association of the second image with the first image, the display position of the figure of the second image, and the like are determined on the second screen of the second display mode based on the user input operation. Set to variable. The control unit 101 saves and manages the display position of the graphic of the second image as one of the metadata 22 or the management information 21. At the time of reproduction display, the control unit 101 determines the display position of the figure with reference to the information.

In this example, a case where a transparent frame is displayed as a figure of the second image is shown. First, the image pickup apparatus 1 automatically determines and displays the display position of the second image as in the first embodiment. For example, a transparent frame 9b1 representing the first narrow-angle image 6b is displayed. The user touches, for example, the transparent frame 9b1 of the second image to move it in a desired direction and position. The control unit 101 updates the display state so as to move the display position of the figure according to the user operation. In this example, the state after the movement of the transparent frame 9b1 is shown as the transparent frame 9b2. The center point of the first narrow-angle image 6b is from the point p21 before the movement to the point p22 after the movement. The control unit 101 updates the management value so as to save the position of the figure in the second image after the movement as a new display position.

Further, the control unit 101 may display the information for confirming the change of the display position after moving the figure of the second image, and confirm with the user whether or not to change and save the information. The control unit 101 displays, for example, a pop-up area near the moved position, and displays "The display position has been moved. Save / Do not save" and the like. The user confirms it and presses, for example, a "save" button. As a result, the control unit 101 saves the display position after the movement.

Further, as an editing function, it has a function of adding a second image to be associated and a function of canceling the association of the second image. In the case of addition, the user touches a desired position in the display screen, for example, point p23, and specifies the addition. For example, the add button in the pop-up area allows the user to select the second image to be added. The control unit 101 displays a figure related to the selected second image at that position, and saves the display position of the second image in the control value.

In the case of cancellation, the user selects the figure of the second image displayed on the display screen and specifies the cancellation. For example, the user touches the transparent frame 9c of the second narrow-angle image 6c and presses the release button of the pop-up area. The control unit 101 cancels the association of the second image for which the cancellation is specified, erases the display, and updates the management value.

According to this modification, the editing function allows the user to freely edit the associated display of the plurality of images in the second display mode so as to be in a state that is easy for him / her to see or grasp.

(Embodiment 2)
The image pickup apparatus and the display apparatus according to the second embodiment of the present invention will be described with reference to FIGS. 22 to 25. Hereinafter, a portion of the configuration of the second embodiment that is different from the configuration of the first embodiment will be described. The imaging device 1 of the second embodiment has a function of capturing a spherical image as a wide-angle image, and has a function of associating and displaying the wide-angle image as a first image.

[Spherical image]
In the present specification, as shown in FIG. 7, a virtual spherical surface 600 set with the position of the image pickup apparatus and the user who is the photographer as the center point is referred to as a spherical surface 600. In the present specification, the spherical image is an image having a 360 ° angle of view at least one of a horizontal angle of view and a vertical angle of view.

FIG. 22 shows the spherical image of the second embodiment in a horizontal angle of view. The spherical surface 600 corresponding to the spherical surface has a first wide-angle image 6A, a second wide-angle image 6B, and a narrow-angle image 6C. The first wide-angle image 6A is a wide-angle image having a shooting range 2201 of 180 ° in a horizontal angle of view viewed from the front camera side of the image pickup apparatus 1 in the front shooting direction (for example, northward). The second wide-angle image 6B is a wide-angle image having a shooting range 2202 of 180 ° in a horizontal angle of view viewed from the rear camera side in a rearward shooting direction (for example, facing south). The narrow-angle image 6C is a narrow-angle image having a shooting range 2203 with a horizontal angle of view Ah in a desired shooting direction (for example, northward) from the front camera side. By combining the first wide-angle image 6A and the second wide-angle image 6B, a spherical image which is a wide-angle image having a shooting range of 360 ° at a horizontal angle of view is constructed. Although not shown, the same applies to the vertical angle of view Av. For example, the first wide-angle image 6A has a shooting range of 180 ° at a vertical angle of view.

[Imaging device]
FIG. 23 shows the configuration of a portion of the imaging device 1 related to the imaging unit 103. The imaging device 1 has a first imaging unit 103A, a second imaging unit 103B, and a third imaging unit 103C as the imaging unit 103, and these are connected to the interface circuit 131.

The first imaging unit 103A is an imaging unit corresponding to the front camera, and includes a driving unit 31a and a lens unit 32a. The lens unit 32a includes a first lens as an ultra-wide-angle lens capable of capturing a first wide-angle image 6A in a shooting range of 180 ° (that is, a hemisphere) at a horizontal angle of view and a vertical angle of view on the front surface. The second imaging unit 103B is an imaging unit corresponding to the rear camera, and includes a driving unit 31b and a lens unit 32b. The lens unit 32b includes a second lens as an ultra-wide-angle lens capable of capturing a second wide-angle image 6B in a shooting range of 180 ° (that is, a hemisphere) at a horizontal angle of view and a vertical angle of view on the back surface.

The third imaging unit 103C has the same configuration as the imaging unit 103 of the first embodiment, and includes a driving unit 31c and a lens unit 32c. The lens unit 32c includes a third lens capable of capturing a narrow-angle image 6C within a set angle of view range of less than 180 ° in the horizontal and vertical directions of the front surface. The angle of view when taking an image with the image pickup unit 103C can be set based on a user operation within a predetermined angle of view range of less than 180 °. For example, assume that a narrow angle image 6C having a horizontal angle of view Ah of 50 ° is imaged.

The control unit 101 includes a spherical image control unit 11A and an image control unit 11B. The spherical imaging control unit 11A drives and controls the first imaging unit 103A and the second imaging unit 103B to capture a spherical image. The spherical imaging control unit 11A controls the first wide-angle image 6A of the first imaging unit 103A and the second wide-angle image 6B of the second imaging unit 103B so as to be imaged at the same timing. The spherical image control unit 11A inputs and synthesizes the first wide-angle image 6A and the second wide-angle image 6B through the signal processing unit 132 and the like, so that the spherical image having an angle of view range of 360 ° is obtained. To image.

It should be noted that the configuration of the imaging unit 103, the control unit 101, and the like including the optical system capable of capturing the spherical image in this way can be realized by using known technology.

The spherical image control unit 11A stores the image data 23 of the captured spherical image in the storage unit 102 as an image file 24 together with the metadata 22 and the like including information such as the angle of view, as in the first embodiment. To do. The image pickup control unit 11B stores the image data 23 of the captured narrow-angle image in the storage unit 102 as an image file 24 together with the metadata 22 and the like including information such as the angle of view, as in the first embodiment.

In the photographing mode of the imaging device 1, it is possible to capture an image using any imaging unit according to the control of the control unit 101. Based on the control from the control unit 101, the image pickup unit 103A, the image pickup unit 103B, and the image pickup unit 103C can simultaneously perform one-time imaging. Further, based on the control from the control unit 101, the imaging unit 103A, the imaging unit 103B, and the imaging unit 103C can perform imaging at individual timings. A spherical image can be captured by simultaneous imaging and composition of the imaging unit 103A and the imaging unit 103B. Further, by photographing using one of the imaging unit 103A and the imaging unit 103B, a hemispherical image by the first wide-angle image 6A or the second wide-angle image 6B can be photographed. Further, it is possible to simultaneously take an image of one of the imaging unit 103A or the imaging unit 103B and the imaging unit 103C. In that case, the wide-angle image and the narrow-angle image 6C have the same shooting date and time, and can be automatically associated with each other. The control unit 101 may add information such as a type representing a spherical image, a hemispherical image, or a narrow-angle image 6C to the metadata 22 or the management information 21 of the captured image. The control unit 101 may add association information to the metadata 22 and the like for a plurality of images simultaneously captured.

[Imaging device-appearance]
FIG. 24 shows the appearance of an implementation example of the image pickup apparatus 1. In this example, the case where the image pickup device 1 is a smartphone is shown. The left side of FIG. 24 shows the front camera side of the housing 2400 of the imaging device 1, and the right side shows the rear camera side. On the front camera side, the first lens 2401 is provided at a predetermined position of the housing 2400, and the third lens 2403 is provided at a predetermined position. On the rear camera side, the second lens 2402 is provided at a predetermined position on the opposite side corresponding to the position of the first lens 2401 on the front camera side. Further, on the rear camera side, a display screen 2404 of the display device 104, an operation button 2405, and the like are provided.

[Display screen-2nd display mode]
FIG. 25 shows an example of the second screen of the second display mode according to the second embodiment. The first screen of the first display mode is the same as that of the first embodiment, and can display a list of information of spherical images and narrow-angle images. When the spherical image is selected and operated on the first screen of the first display mode, the display control unit 12 displays the second screen of the second display mode as shown in FIG. 25.

The display control unit 12 displays the spherical image in the second screen as shown by the circular area 2500. The display control unit 12 initially displays the area 2500 in a display state in which the photographing direction (azimuth and elevation, for example, northward in FIG. 22) of the spherical image is located at the central point p0. With respect to the area 2500, operations such as scrolling and enlargement / reduction can be performed so as to change the display state in response to the viewpoint change in response to the user's touch input operation or the like. For example, when the user swipes to the left in the area 2500, the display state in the area 2500 is updated so that the viewpoint in the spherical image moves to the right, that is, in the positive direction at the horizontal angle of view. Will be done. The control unit 101 performs image processing for changing the display state at that time. Image processing for changing the display state of the spherical image is a known technique. Further, in particular, the position of the viewpoint in the initial display state of the spherical image may be 0 °, that is, the horizontal direction as the elevation angle, or 90 °, that is, the zenith direction.

When a narrow-angle image associated with the spherical image exists, the control unit 101 displays the narrow-angle image on the region 2500 of the spherical image as a predetermined figure or a predetermined figure as in the first embodiment. Display as a reduced image. In this example, the case of displaying with a transparent frame is shown. A case where three narrow-angle images are associated with each other on the region 2500 of the spherical image is shown. Point p11 has a transparent frame 2501 representing a first narrow-angle image, point p12 has a transparent frame 2502 representing a second narrow-angle image, and point p13 represents a third narrow-angle image. It has a transmission frame 2503. With the change of the display state in the area 2500 of the spherical image, the position, shape, size, etc. of the figure of the narrow-angle image are also changed. Since the narrow-angle image region corresponding to the transparent frame is arranged in the spherical surface of the spherical surface, the shape is distorted in the circumferential direction, that is, it has an arc portion. In this example, the transparent frame is simply displayed as a rectangle.

Further, as in the first embodiment, based on a predetermined operation on the second screen of the second display mode, on / off switching of the display of the graphic of the narrow-angle image, an operation of returning to the first display mode, and the like are also performed. It is possible. When a figure representing a narrow-angle image is selected and operated on the second screen, the control unit 101 displays the narrow-angle image in detail in the third display mode, as in the first embodiment.

[Effects, etc.]
According to the second embodiment, when the whole celestial sphere image is displayed as the wide-angle image, the narrow-angle images having a predetermined relationship such as the same shooting location are associated and displayed, so that the user can grasp the relationship between the plurality of images. It is easy to browse and can be browsed appropriately.

The following is possible as a modification of the second embodiment. The second imaging unit 103B on the rear camera side may be omitted, and the first imaging unit 103A may be used to capture a hemispherical image having a front angle of view range of 180 °.

On the second screen of the second display mode, the area 2500 of the spherical image may be converted into a rectangular area instead of a circular area and displayed. An image corresponding to a part of the angle of view range of the spherical image is displayed in the rectangular area, and the display state is changed based on the user operation.

The imaging device 1 may capture a spherical image in its own device, capture a narrow-angle image captured by an external device, and display the association. On the contrary, the imaging device 1 may capture a narrow-angle image in its own device, capture a spherical image captured by an external device, and display the association. For example, the second imaging device 1B in FIG. 1 captures a spherical image, and acquires the narrow-angle image captured by the first imaging device 1A and the narrow-angle image registered in the server 3 in its own device. Then, the association display may be performed.

An image having an angle of view of 360 ° has an angle of view in all directions regardless of the shooting direction of the camera. Therefore, it is not always necessary to change the display direction in the display screen according to the shooting direction. Therefore, when the moving image of the spherical image is reproduced and displayed in the second display mode or the third display mode, the following two types of display methods may be selected. First, similar to the reproduction display of the narrow-angle image, the image that changes according to the shooting direction at the time of shooting is reproduced and displayed as it is. Alternatively, secondly, based on the information of the shooting direction recorded as metadata at the time of movie shooting, the direction of the image is fixed regardless of the shooting direction by rotating the image by an angle corresponding to, for example, the azimuth angle. , Play and display in that fixed direction.

Further, when displaying an image regardless of whether it is a moving image or a still image, the direction of the imaging device 1 (the same applies to the display device 2) is detected by using the shooting direction measuring unit 111 or the position detecting unit 113, and the detected direction. May be displayed so as to match the shooting direction of the image.

(Embodiment 3)
The image pickup apparatus and the display apparatus according to the third embodiment of the present invention will be described with reference to FIG. 26. Hereinafter, a portion of the configuration of the third embodiment that is different from the configuration of the first embodiment will be described. The image pickup apparatus 1 of the third embodiment is provided with a unique shooting mode for associating a plurality of images.

[Shooting mode]
The image pickup apparatus 1 of the third embodiment has, as a shooting mode, a first shooting mode which is a normal shooting mode and a second shooting mode for associating with the shooting mode. The image pickup apparatus 1 switches the shooting mode based on the user input operation and the user setting. The user performs a predetermined operation of switching between the first shooting mode and the second shooting mode at an arbitrary timing through the operation input unit 105 or the like. The predetermined operation may be, for example, pressing the mode switching button.

FIG. 26 shows switching of the shooting mode on the time axis in the image pickup apparatus 1 of the third embodiment. The image pickup apparatus 1 is initially set to the state of the first shooting mode, and the individual images captured in the first shooting mode are not subjected to the association processing. For example, the image captured at the time point t11 is not associated, and the association information is not described in the metadata 22 or the like.

The user performs an operation of switching from the first shooting mode to the second shooting mode at a desired timing, for example, at the time point T2. In the state of the second shooting mode, the image pickup apparatus 1 automatically associates the individual images captured during that period with each other. For example, images are taken at time points t1 to t4 in the second shooting mode. The control unit 101 performs association processing on those images by the association unit 15, and describes the association information in the metadata 22 or the management information 21. At that time, the association unit 15 associates the plurality of images as a group. In this example, each image at time points t1 to t4 is associated as a first group. The association unit 15 describes the group identification information as one of the metadata 22 or the management information 21. The group identification information may be, for example, a character string using the start date and time and the end date and time of the period of the second shooting mode, or a character string using the number of occurrences of the second shooting mode. Further, the file name and the metadata 22 of those images may be given a character string in which the group identification information and the shooting date and time and the sequence number of the individual images are combined.

The user performs an operation of switching from the second shooting mode to the first shooting mode at a desired timing, for example, at the time point T4. The image pickup apparatus 1 does not associate the individual images captured in the first photographing mode, for example, the images captured at the time point t12. The control unit 101 may execute the association process when returning to the first shooting mode.

In addition, the association unit 15 determines the types of wide-angle images and narrow-angle images for the plurality of images captured during the period of the second shooting mode. The association unit 15 does not associate when only one of the wide-angle image and the narrow-angle image exists. When both the wide-angle image and the narrow-angle image are present, the association unit 15 associates the first image, which is the wide-angle image, with the second image, which is the narrow-angle image.

At the time of playback display, the display control unit 12 confirms the association and displays the association based on the reference of the group identification information or the like in the image file 24 or the management information 21.

[Effects, etc.]
According to the third embodiment, the user can easily switch the shooting mode and make the association at his / her own discretion. The third embodiment corresponds to a mode in which the determination regarding the shooting date and time of the second selection method described above is automated by using a unique shooting mode.

The following is possible as a modification of the third embodiment. The image pickup apparatus 1 automatically switches the shooting mode according to the determination. At the time of imaging, the image pickup apparatus 1 determines the type of angle of view of the image, for example, the distinction between wide angle and narrow angle. Then, the image pickup apparatus 1 automatically switches from the first shooting mode to the second shooting mode when the captured image is a wide-angle image. The image pickup apparatus 1 determines the narrow-angle image captured in the second shooting mode, and associates the narrow-angle image with the first wide-angle image. The image pickup apparatus 1 returns from the second shooting mode to the first shooting mode when, for example, a predetermined time based on the setting elapses after entering the second shooting mode.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof. The functions and the like of the present invention may be partially or wholly realized by hardware such as integrated circuits, or may be realized by software program processing.

1 ... Imaging device, 2 ... Display device, 3 ... Server, 4 ... Communication network, 11 ... Imaging control unit, 12 ... Display control unit, 13 ... Touch detection unit, 14 ... Setting unit, 15 ... Association unit, 21 ... Management Information, 22 ... Metadata, 23 ... Image data, 24 ... Image file, 101 ... Control unit, 102 ... Storage unit, 103 ... Imaging unit, 104 ... Display device, 105 ... Operation input unit, 106 ... Recording unit, 107 ... Recording medium, 108 ... Communication interface unit, 111 ... Shooting direction measurement unit, 112 ... Internal clock, 113 ... Position detection unit, 131 ... Interface circuit unit, 132 ... Signal processing unit 133 ... Memory, 901 ... First image, 902 , 903 ... Figure, 904 ... Second image.

Claims (14)

It is an imaging display method
An imaging step that captures a still image or a moving image to obtain image data,
A display step of reproducing and displaying the image based on the image data, and
With
In the imaging step, at least one of the azimuth angle and the elevation angle as the imaging direction of the image data obtained by imaging and the image, and at least one of the angle of view of the image or the angle-of-view related information capable of calculating the angle of view. And get the first information including
When the second shooting direction of the second image is included in the range of the first angle of view of the first shooting direction of the first image, the first image is associated with the second image.
The association is
In the imaging step, when the second shooting direction of the second image is included in the range of the first angle of view of the first shooting direction of the first image, the first image is covered with the first image. The first method of associating two images and
In the display step, when the second shooting direction of the second image is included in the range of the first angle of view of the first shooting direction of the first image, the first image is described. The user can select the second method for associating the second image.
In the display step, when the second image is associated with the first image, the display is displayed in a state in which the second image or the second information representing the second image is superimposed on the first image. To do,
An imaging display method characterized by the above. In the imaging display method according to claim 1,
A storage step for storing the image data and the first information is provided.
In the storage step, the angle of view related information includes an electronic zoom magnification, a focal length, and the size or model of the image sensor.
When the angle of view related information is included in the first information, the angle of view is calculated from the angle of view related information, and the angle of view is stored in the first information.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
The second information has a reduced image of the second image, a predetermined figure, a transparent frame, or an attribute value of the second image.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
The size of the display of the second information is a size corresponding to the angle of view of the second image.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
When there are a plurality of associated second images in the area above the first image, a figure representing a representative is displayed as the second information.
In response to the selection operation of the figure representing the representative, the information regarding the plurality of associated second images is displayed so that the second image can be selected.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the imaging step, the first information is created and stored as metadata or management information of the image data, and is stored.
In the association, when the second image is associated with the first image, the association information between the first image and the second image is created and stored in the metadata or the management information.
In the display step, the association information is confirmed and displayed in the superimposed state.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the imaging step, the first information is created and stored as metadata or management information of the image data, and is stored.
In the display step, the association is performed, and when the second image is associated with the first image, the second image is displayed in the superimposed state.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
Switch between the first shooting mode and the second shooting mode,
The image captured in the first shooting mode is not associated with each other.
The plurality of images captured in the second shooting mode are associated with each other as a group.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the imaging step, position information representing a shooting location is created and stored as metadata or management information of the image data, and stored.
In the association, when the position information of the second image is included in the predetermined position range with respect to the position information of the first image, the image is selected as a candidate image of the association.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the imaging step, information including the shooting date and time is created and stored as metadata or management information of the image data, and stored.
In the association, when the shooting date and time of the second image is included in a predetermined time range with respect to the shooting date and time of the first image, the image is selected as a candidate image for the association.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the imaging step or the display step, a character string set based on the operation of the user is created and stored as metadata or management information of the image data, and stored.
In the association, when the character string of the second image includes the same character string as the character string of the first image, the image is selected as a candidate image of the association.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
The imaging step can capture a wide-angle image having an angle of view of 180 ° or more and 360 ° or less, and the wide-angle image is used as the first image for the association.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the display step, based on the user's operation
As the first display mode, a list of information related to the plurality of images is displayed.
As the second display mode, the second image or the second information associated with the first image is superimposed on the first image selected from the list display displayed in the first display mode. Display in state,
As the third display mode, the second image displayed in the second display mode or the second image corresponding to the second information is displayed in detail.
An imaging display method characterized by the above. In the imaging display method according to claim 1,
In the display step, the position information of the second image or the second information is stored as one of the items of the metadata or management information of the image data.
Based on the user's operation
The second image or the second information on the first image is moved, and the moved position information is stored in the item.
The association of the second image is added to the selected position on the first image, and the association of the second image at the selected position on the first image is released.
An imaging display method characterized by the above.

Images