JP6704301B2 - Imaging device and imaging display system - Google Patents

Description

The present invention relates to a technique of an image pickup device and a display device, and relates to a technique of displaying a captured image on a screen.

2. Description of the Related Art Conventionally, there are digital cameras, smartphones, and the like as electronic devices such as an imaging device having a function of capturing a plurality of images with different shooting directions and angles of view. Further, as an electronic device such as a display device having a function of reproducing and displaying the plurality of images, there are a television receiver (hereinafter, may be referred to as “TV”), a notebook PC, and the like. The image is a still image or a moving image. The shooting direction can be defined by an azimuth angle and an elevation angle, for example. The angle of view represents an image range by an angle and is also called a viewing angle, and can be defined by, for example, a horizontal angle of view corresponding to an azimuth angle and a vertical angle of view corresponding to an elevation angle.

The images with different angles of view are images captured with a relatively wide angle (hereinafter, sometimes referred to as “wide-angle image”, “first image”, etc.) and images captured with a relatively narrow angle. (Hereinafter, it may be described as “narrow-angle image”, “second image”, etc.). Examples of the wide angle include a horizontal angle of view of 120° and an angle of view of 180° or more. It should be noted that the case of 180° may be referred to as a hemispherical image, and the case of 360° may be referred to as a spherical image. An example of the narrow angle is a horizontal angle of view of 45° or less. Hereinafter, the unit of the angle is expressed in degrees (°), but it can be converted into a unit such as radian.

Japanese Patent Application Laid-Open No. 2000-92439 (Patent Document 1) is given as an example of a prior art relating to the screen display of a captured image. Patent Document 1 describes the following as an electronic still camera and the like. When instructed to turn off the power, the electronic still camera generates an image list display file that allows the external device to display a list of the recorded images recorded in the recording memory. A plurality of reduced images are displayed in a list on the list display screen displayed by the image search using the file.

JP-A-2000-92439

A user who is a photographer may use an image capturing device to capture a plurality of images with different shooting directions and different angles of view at the same place and time. For example, the user shoots a wide-angle image in a certain shooting direction and a certain angle of view in the same park on the same day, and shoots a narrow-angle image in another shooting direction and a certain angle of view. In such a case, it can be said that the plurality of images are related images having a predetermined relationship with respect to place, date and time, shooting direction, angle of view, and the like.

Conventional imaging devices and display devices have a function of displaying a plurality of images on a display screen in a list and in a side-by-side arrangement, for example, in the order of shooting date, file name, and the like. For example, in the case of Patent Document 1, a plurality of reduced images are displayed in parallel on the list display screen.

However, in the conventional technology, it is difficult for the user to intuitively recognize a plurality of images having a predetermined relationship among all the images displayed on the display screen. For example, even when wide-angle images and narrow-angle images captured at the same place and date and time are mixed, it is difficult for the user to grasp the relationship between the images. The related art does not have a function of reproducing and displaying a plurality of related images in association with each other. The related art does not have a function of displaying an image on the display screen and displaying it by referring to another image related to the image. In the conventional technique, it takes time and effort for the user to edit and manage a plurality of images.

That is, the related art has difficulty in suitably browsing a plurality of related images, and there is room for improvement in terms of usability.

It is an object of the present invention to provide a technique for an imaging device and a display device, which makes it easy to understand the relationship between a plurality of images and realizes better usability.

A typical embodiment of the present invention is an imaging device or a display device, which is characterized by having the following configuration.

An image pickup apparatus according to one embodiment includes an image pickup unit that obtains image data by picking up a still image or a moving image, and a display unit that reproduces and displays the image on a display screen based on the image data. The first information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view is acquired and imaged. For a plurality of images, when the second shooting direction of the second image is included within the range of the first angle of view of the first shooting direction of the first image, the second image is associated with the first image. Displaying the first image selected on the basis of a user operation in the display screen, and displaying the second image associated with the first image on the first image, the second image or The second information representing the second image is displayed in a superimposed state.

A display device according to an embodiment inputs an image data of a still image or a moving image captured by an external image capturing device and holds the input data, and reproduces the image on a display screen based on the image data. A display unit for displaying, and including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle of view related information capable of calculating the angle of view. 1 information is acquired, and in the case of the plurality of captured images, when the second capturing direction of the second image is included within the range of the first angle of view of the first capturing direction of the first image, The first image is associated with the second image, the first image selected based on a user operation is displayed in the display screen, and the second image associated with the first image is referred to as the first image. The second image or second information representing the second image is displayed on the image in a superimposed state.

According to the representative embodiment of the present invention, it is possible to easily understand the relationship between a plurality of images in the imaging device and the display device, and to realize better usability.

It is a figure which shows the structure of the imaging display system comprised including the imaging device and display of Embodiment 1 of this invention. FIG. 3 is a diagram showing a configuration of the image pickup apparatus according to the first embodiment. 1 is a diagram showing a configuration of a display device according to a first embodiment. FIG. 3 is a diagram showing an appearance of the image pickup apparatus according to the first embodiment. 3 is a diagram showing a configuration of management information and metadata in the image pickup apparatus according to the first embodiment. FIG. FIG. 6 is a diagram showing management of a relationship between a plurality of images in the image pickup apparatus according to the first embodiment. FIG. 3 is an explanatory diagram showing a shooting direction and an angle of view regarding a plurality of images, particularly an azimuth angle and a horizontal angle of view, in the image pickup apparatus of the first embodiment. FIG. 3 is an explanatory diagram particularly showing an elevation angle and a vertical angle of view in the image pickup apparatus according to the first embodiment. FIG. 4 is an explanatory diagram showing angle-of-view related information in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing a transition of display modes of a display screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of a screen in a second display mode in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing a screen example of a first narrow-angle image in a third display mode in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing a screen example of a second narrow-angle image in a third display mode in the image pickup apparatus according to the first embodiment. FIG. 4 is a diagram showing a flow of control processing at the time of image capturing in the image capturing apparatus of the first embodiment. FIG. 6 is a diagram showing a flow of control processing at the time of reproduction display in the imaging device and the display device of the first embodiment. FIG. 6 is an explanatory diagram showing a first selection method in the image pickup apparatus according to the first embodiment. FIG. 6 is an explanatory diagram showing a second selection method in the image pickup apparatus according to the first embodiment. FIG. 6 is an explanatory diagram showing a third selection method in the image pickup apparatus according to the first embodiment. FIG. 9 is a diagram showing an example of a screen in a second display mode in the image pickup apparatus of the modified example of the first embodiment. FIG. 7 is a diagram showing an example of a screen in a first display mode and a second display mode in the image pickup apparatus of the modified example of the first embodiment. FIG. 9 is a diagram showing an example of a display screen of an editing function in the image pickup apparatus of the modified example of the first embodiment. It is explanatory drawing shown about the omnidirectional image in the imaging device and display device of Embodiment 2 of this invention. FIG. 8 is a diagram showing a configuration relating to an image pickup unit in the image pickup apparatus according to the second embodiment. FIG. 9 is a diagram showing an external configuration of the image pickup apparatus according to the second embodiment. FIG. 16 is a diagram showing an example of a screen in a second display mode in the image pickup apparatus according to the second embodiment. It is explanatory drawing which shows the imaging mode in the imaging device of Embodiment 3 of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all the drawings for explaining the embodiments, the same parts are designated by the same reference numerals in principle, and the repeated description thereof will be omitted.

(Embodiment 1)
The imaging device and the display device according to the first embodiment of the present invention will be described with reference to FIGS. The imaging device and the display device according to the first embodiment have a function of displaying a plurality of images having different shooting directions and field angles at the time of shooting, in particular, a wide-angle image and a narrow-angle image in an associated state.

[Imaging display system]
FIG. 1 shows the configuration of an image pickup display system which is a system including the image pickup apparatus and the display apparatus according to the first embodiment. The image pickup display system of FIG. 1 includes a first image pickup apparatus 1A and a second image pickup apparatus 1B which are the image pickup apparatus 1 of the first embodiment, and a first display apparatus 2A and a second display which are the display apparatus 2 of the first embodiment. The device 2B and the server 3 are connected via the communication network 4. The communication network 4 includes a wireless communication network and the Internet.

The image pickup apparatus 1 is an electronic device having at least an image pickup function and a display function. The display device 2 is an electronic device having at least a display function. A user who is a photographer has a first image pickup device 1A and a second image pickup device 1B, and a first display device 2A and a second display device 2B as electronic devices that he or she owns or has authority to use. For example, the first imaging device 1A is a digital camera, the second imaging device 1B is a smartphone, the first display device 2A is a television, and the second display device 2B is a notebook PC.

The first image capturing apparatus 1A and the second image capturing apparatus 1B have at least a function of holding image data of an image captured by the own device in a storage unit in the own device. Further, the first imaging device 1A and the second imaging device 1B may have a function of transmitting the image data of the image captured by the self device to the server 3 and storing and registering the image data in the image data DB of the server 3. .. In addition, the first image pickup device 1A and the second image pickup device 1B have a function of transmitting image data of an image picked up by the device itself to the first display device 2A and the second display device 2B based on a user operation. May be.

The first display device 2A and the second display device 2B have a function of receiving and inputting image data from the external device such as the imaging device 1 or the server 3, and storing the image data in the storage unit in the device itself. For example, the first display device 2A receives image data from the first imaging device 1A and the second display device 2B by communication. For example, the second display device 2B receives image data by communication from the first imaging device 1A and the second imaging device 1B. In addition, the first display device 2A and the second display device 2B have an image data display unit. The image data display unit displays an image on the display screen based on the image data.

Further, for example, the second display device 2B has an image data editing unit. The image data editing unit edits information and contents of image data based on a user operation on the display screen. As will be described later, the user can set the relationship between a plurality of images by editing.

The server 3 is, for example, a server device managed by a business operator, stores image data from a user's device, and stores the image data in an image data DB. This realizes backup of image data, unified management, and the like. The server 3 and the image data DB may be implemented by different devices.

[Imaging device]
FIG. 2 shows the configuration of the image pickup apparatus 1. The imaging device 1 includes a control unit 101, a storage unit 102, an imaging unit 103, a display device 104, an operation input unit 105, a recording unit 106, a recording medium 107, a communication interface unit 108, a photographing direction measuring unit 111, an internal clock 112, and a position. The detection unit 113, the sensor group 114, the microphone 115, the speaker 116, the interface circuit unit 131, the signal processing unit 132, the memory 133 and the like are provided. The respective units are connected to each other via a bus or the like, and cooperate with each other at high speed under the control of the control unit 101.

The control unit 101 controls the entire imaging device 1. The control unit 101 includes a processor and control circuits such as ROM and RAM. The control unit 101 includes an imaging control unit 11, a display control unit 12, a touch detection unit 13, a setting unit 14, and an association unit 15 as processing units implemented by the processing of the control circuit.

The control unit 101 controls each unit such as the image pickup unit 103 according to the operation mode to realize image pickup and display. The operation mode includes an image pickup mode, a reproduction display mode, and the like. The control unit 101 controls the imaging unit 103, the display device 104, the recording unit 106, and the like via the signal processing unit 132 and the memory 133 according to the operation mode and state.

The control unit 101 controls settings such as a shooting direction and an angle of view of imaging based on a user input operation through the operation input unit 105 and the touch sensor 42. The control unit 101 creates the metadata 22 together with the image data 23 as the image is captured by the imaging unit 103, and stores the image data 24 including the metadata 22 and the image data 23 in the storage unit 102 or the recording unit. It records on the recording medium 107 of 106. The control unit 101 also creates and manages the management information 21 regarding the image file 24.

The storage unit 102 stores various data and information including the plurality of image files 24 and the management information 21 under the control of the control unit 101. The storage unit 102 may be integrated in the control unit 101, the storage unit 102 and the recording unit 106 may be integrated in one form, and the like.

The image capturing unit 103 captures an image under the control of the image capturing control unit 11. The imaging unit 103 includes a drive unit 31 and a lens unit 32. The drive unit 31 includes a drive circuit and drives the image pickup element and the lens of the lens unit 32 based on the drive control from the image pickup control unit 11. The lens unit 32 includes an optical system including a plurality of lenses including a lens compatible with electronic zoom or optical zoom. For example, the drive unit 31 performs a focusing operation that changes the focal length by controlling the position of the focus lens. The lens of the lens unit 32 may be a zoom lens system that is fixed without replacement and is compatible with electronic zoom, or an optical lens system that can be replaced by being attached and detached.

The image pickup unit 103 includes an image pickup device such as a CMOS or CCD. Photoelectric conversion elements are two-dimensionally arranged on the image pickup surface of the image pickup element. The image capturing unit 103 photoelectrically converts the optical image of the subject, which is incident through the lens unit 32 and formed on the image capturing surface of the image sensor, into an image capturing signal. The image pickup unit 103 includes an AD conversion circuit that converts an analog signal into a digital signal, and outputs a digitized image pickup signal. In the image pickup unit 103, an image pickup element in which pixels for phase difference autofocus are arranged may be used to speed up autofocus, or a memory may be built in to speed up input/output of an image pickup signal. Good. An AD conversion circuit or the like may be provided outside the imaging unit 103. The imaging method of the imaging unit 103 is not limited.

The display device 104 is, for example, a touch panel, and includes a display unit 41 and a touch sensor 42. The touch panel is a display unit and an operation input unit. The display unit 41 includes a display driver and displays a display screen to the user. The display unit 41 is, for example, a liquid crystal display unit, but may be an organic EL display unit or the like. The method of the display unit 41 and the method of the touch sensor 42 are not limited. A plurality of display devices 104 may be provided, or the display device 104 may not be provided. In the first embodiment, in addition to the touch panel that is the main display device 104, a display device for a photographing monitor, which will be described later, is provided.

The touch sensor 42 is arranged corresponding to the area of the display screen of the display unit 41, and receives a touch input operation on the display screen. The touch sensor 42 is, for example, an electrostatic capacitance type, detects an electrostatic capacitance change due to proximity or contact of a finger or the like as an electric signal, and outputs a touch detection signal to the touch detection unit 13 of the control unit 101. In the touch panel, for example, a touch sensor 42 is built in a liquid crystal panel display unit, a glass protective cover is arranged on the front side, and a backlight is arranged on the back side. Note that the display device 104 may be automatically controlled to be in the non-display state by turning off the backlight based on the non-use state for a certain period of time. The touch panel may be a pressure sensitive type or the like.

The touch detection unit 13 detects presence/absence of a touch on the display screen, touch position coordinates, and the like based on a touch detection signal obtained from the touch sensor 42, and also detects an operation such as tapping, swiping, or pinching. By a touch input operation on the display screen of the touch panel, the user can perform various operations such as an instruction input to the imaging device 1 and an operation of focusing on an arbitrary position on the display screen.

The interface circuit unit 131 transfers the drive control signal from the control unit 101 to the imaging unit 103 under the control of the control unit 101 according to the operation mode. Further, the interface circuit unit 131 outputs the image pickup signal from the image pickup unit 103 to the signal processing unit 132 or the memory 133 under the control. The interface circuit unit 131 may be connected to a plurality of image capturing units 103.

The signal processing unit 132 includes an image/audio signal processing circuit and an encoding/decoding circuit, and under the control of the control unit 101, various signal processing such as image signal processing, audio signal processing, encoding processing, and decoding processing. Has the function of performing. The signal processing of the signal processing unit 132 includes signal processing for the image pickup signal from the image pickup unit 103, signal processing for the image data 23, and signal processing for displaying an image on the display device 104. The signal processing unit 132 generates a video signal for recording or display by signal processing. This video signal includes a still image or a moving image of the image data 23.

The signal processing unit 132 performs signal processing such as filtering, amplification according to sensitivity setting, and white balance correction on the image pickup signal, for example. When recording the image data 23 as the image file 24 on the recording medium 107, the signal processing unit 132 performs an encoding process in a predetermined format for the original data. This encoding includes data compression. When reading the image data 23 from the recording medium 107, the signal processing unit 132 performs a corresponding decoding process to restore the original data. The signal processing unit 132 performs signal processing for reproducing and displaying an image on the display device 104 based on the image data 23. The signal processing unit 132 performs audio signal processing on an audio input signal from the microphone 115 and an audio output signal to the speaker 116.

The memory 133 is used as a buffer memory at the time of signal processing of the signal processing unit 132, and stores each signal and data. The memory 133 is composed of, for example, a DRAM or a flash memory.

The operation input unit 105 includes various hardware buttons and the like that enable an input operation by the user. The user can input an instruction through the operation input unit 105 and can set the view angle related information and the like in the setting unit 14.

The setting unit 14 accepts settings input by the user through the operation input unit 105 and the touch panel, and sets information regarding shooting and display. The setting unit 14 can set the electronic zoom magnification, the focal length, the angle of view, and the like regarding the lens of the image capturing unit 103 as the setting information related to shooting. The imaging control unit 11 drives and controls the imaging unit 103 based on the setting information of the setting unit 14 so as to automatically adjust the angle of view and the like.

The recording unit 106 includes a recording/reproducing circuit and the like, and a recording medium 107 is detachably provided. The recording medium 107 is, for example, an SD card. The recording unit 106 records the image file 24 including the image data 23 in the storage unit 102, the memory 133, or the like on the recording medium 107 under the control of the control unit 101. Under the control of the control unit 101, the recording unit 106 reads the image file 24 from the recording medium 107 and stores it in the storage unit 102, the memory 133, or the like. The user can remove the recording medium 107 and carry or replace it. The image file 24 may include audio data together with the image data 23.

The communication interface unit 108 includes a wireless communication interface unit. Under the control of the control unit 101, the communication interface unit 108 connects to the communication network 4 via a wireless access point or the like, and performs communication processing with an external device. For example, as illustrated in FIG. 1, the imaging device 1 communicates with the server 3 or the display device 2 or the imaging device 1 other than the own device via the communication network 4, and transmits the image file 24 of the own device, or an external device. The image file 24 is received from the device. At that time, the communication interface unit 108 transfers the communication data of the image file 24. The communication interface unit 108 may have a telephone network communication function, a LAN communication function, a near field communication function, and the like.

The shooting direction measurement unit 111 includes sensors such as an electronic compass, a gyro sensor, and an acceleration sensor, and uses them to measure the shooting direction of the imaging device 1 and output information on the shooting direction to the control unit 101. The shooting direction measurement unit 111 measures an azimuth angle and an elevation angle as a shooting direction. In addition, the shooting direction measurement unit 111 detects a rotation angle representing the tilt of the image at the time of shooting, based on the detection of the attitude of the imaging device 1. The rotation angle of the image is represented by the angle formed by the horizontal and vertical lines of the rectangle of the image with respect to the horizontal and vertical directions.

The internal clock 112 measures the current date and time. Accordingly, the control unit 101 can obtain the shooting date and time. The position detection unit 113 includes a GPS reception unit and detects the position of the imaging device 1 using a signal from a satellite. The position detection unit 113 detects {latitude, longitude, altitude (altitude above sea level)} as position information. The altitude may be omitted. The sensor group 114 is another sensor group, and examples thereof include a proximity sensor and an illuminance sensor. The microphone 115 is a voice input device, and the speaker 116 is a voice output device.

The image pickup control unit 11 controls the image pickup unit 103 and the like at the time of image pickup according to the operation mode of the image pickup apparatus 1 to pick up an image and obtain image data 23. The imaging control unit 11 stores the image data 23 as an image file 24 in the storage unit 102 or the like. The image capturing unit 103 and the image capturing control unit 11 have a function of capturing both wide-angle images and narrow-angle images. The imaging control unit 11 performs zoom control regarding the optical system of the imaging unit 103. The zoom control is control of electronic zoom or optical zoom. The imaging control unit 11 grasps and adjusts the electronic zoom magnification, the focal length, the angle of view, and the like regarding the lens of the imaging unit 103 based on the setting information such as the angle-of-view related information of the setting unit 14.

When it is possible to replace the optical lens with a different focal length in the imaging unit 103, for example, when capturing a narrow-angle image, the user replaces the optical lens with a relatively long focal length and captures a wide-angle image. In that case, replace it with an optical lens having a relatively short focal length. When the user replaces the lens of the imaging unit 103, the setting unit 14 may automatically detect the replacement and detect the focal length of the lens after the replacement. Alternatively, after the replacement, the user may input and set the focal length and the like of the replaced lens into the setting unit 14. Then, the imaging control unit 11 acquires the updated setting information from the setting unit 14 and grasps the focal length and the like.

The display control unit 12 controls the display unit 41 and the like of the display device 104 to display images and information on the display screen of the display unit 41. The display control unit 12 displays the folder information and the image on the display screen based on the management information 21 and the image file 24. The display control unit 12 provides various screens that serve as a graphical user interface for the user. The display control unit 12 controls screen switching according to a display mode described later. The display control unit 12 also provides a menu screen, a setting screen, and the like.

The associating unit 15 cooperates with the imaging control unit 11 and the display control unit 12 to perform an associating process of associating a plurality of images having a predetermined relationship. The associating unit 15 performs the associating process at a predetermined timing and timing, such as at the time of capturing an image or at the time of reproducing and displaying, according to an association method described later. As a first process in the association process, the association unit 15 makes a determination for association with respect to the plurality of image data 23 in the storage unit 102 or the like. In the determination of the first process, an image that is a candidate for association is selected based on conditions such as a shooting location according to a selection method described later. As the second process, the associating unit 15 determines the first image and the second image to be associated with each other in the plurality of candidate images based on the conditions of the shooting direction and the angle of view. As a result of the determination, the associating unit 15 creates the metadata 22 or the management information 21 so that the first image and the second image are associated with each other.

When the image is reproduced and displayed, the display control unit 12 has a predetermined relationship based on the determination result of the associating unit 15 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24. One image and a second image are specified, and a plurality of images including the first image and the second image are displayed in a state associated with the display screen.

The hardware configuration of each processing unit and each function such as the control unit 101 and the signal processing unit 132 of the image pickup apparatus 1 may be a configuration in which each processing unit and each function are implemented by a circuit such as an individual LSI. It may be implemented by a single integrated circuit such as an LSI.

[Display device]
FIG. 3 shows the configuration of the display device 2. The display device 2 does not include the image capturing unit 103 and a processing unit related to image capturing, which is a configuration mainly different from the image capturing device 1. Of the components of the display device 2, those similar to the components of the imaging device 1 will not be described. The display device 2 includes a control unit 201, a storage unit 202, a display unit 204, an operation input unit 205, a recording unit 206, a recording medium 207, a communication interface unit 208, a sensor group 214, a microphone 215, a speaker 216, and the like. The respective units are connected to each other via a bus or the like, and cooperate with each other at high speed under the control of the control unit 201.

The control unit 201 controls the entire display device 2. The control unit 201 includes a processor and control circuits such as ROM and RAM. The control unit 201 has a display control unit 212 and an associating unit 215 as processing units realized by the processing of the control circuit. The control unit 201 controls the display on the display screen of the display unit 204 based on the user input operation through the operation input unit 205.

The storage unit 202 stores various data and information including the plurality of image files 24 and the management information 21 under the control of the control unit 201. The image file 24 and the management information 21 may be created by the own device or acquired from an external device.

The display unit 204 includes a display driver and displays a display screen to the user. The display unit 204 may be a touch panel.

The operation input unit 205 includes various hardware buttons and the like that enable an input operation by the user. The operation input unit 205 may include a remote controller and a remote controller light receiving unit. The user can input instructions and make user settings through the operation input unit 205.

The recording unit 206 includes a recording/reproducing circuit and the like, and a recording medium 207 is detachably provided. The recording unit 206 records the image file 24 including the image data 23 in the storage unit 202 or the memory on the recording medium 207 under the control of the control unit 201. The recording unit 206 reads the image file 24 from the recording medium 207 and stores the image file 24 in the storage unit 202 or a memory under the control of the control unit 201. The user can remove the recording medium 207 and carry or replace it.

The communication interface unit 208 connects to the communication network 4 under the control of the control unit 201 and performs communication processing with an external device. For example, as shown in FIG. 1, the display device 2 communicates with the server 3 or the imaging device 1 or the display device 2 other than the own device via the communication network 4, and transmits the image file 24 of the own device, or an external device. The image file 24 is received from the device. At that time, the communication interface unit 208 transfers the communication data of the image file 24.

The display control unit 212 controls the display unit 204 and the like to display an image and information on the display screen of the display unit 204. The display control unit 212 reproduces and displays the folder information and the image on the display screen of the display unit 204 based on the image file 24 and the management information 21. The display control unit 212 provides various screens that serve as a graphical user interface for the user. The display control unit 212 controls screen switching according to the display mode.

The associating unit 215 cooperates with the display control unit 212 to perform an associating process of associating a plurality of images having a predetermined relationship. The associating unit 215 can also perform the associating process on the image file 24 acquired from the external device. The associating process of the associating unit 215 is similar to the associating process of the associating unit 15. When associating the first image and the second image as a result of the determination, the associating unit 215 describes the association information in the metadata 22 of the image file 24 of the first image and the second image or the management information 21.

When the image is reproduced and displayed, the display control unit 212 has a predetermined relationship based on the determination result of the associating unit 215 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24. One image and a second image are specified, and a plurality of images including the first image and the second image are displayed in a state associated with the display screen.

Hereinafter, the image pickup apparatus 1 will be mainly described, but the functions other than the functions and operations related to image pickup can be similarly realized mainly by the display apparatus 2.

[Imaging device-appearance]
FIG. 4 shows an example of the external configuration of the image pickup apparatus 1. The left side of FIG. 4 shows the configuration of the housing 400 of the image pickup apparatus 1 viewed from the back side, and the right side of the configuration of the imaging apparatus 1 viewed from the right side. The back surface is the surface of the display unit 41 on which the display screen 401 is provided, and the front surface is the surface of the imaging unit 103 on which the lens 402 is provided.

In the configuration of the side surface, the lens 402 of the lens unit 32 is arranged at the front center position of the housing 400. Further, an electronic viewfinder (EVF) 403 is provided, for example, in a state of projecting upward with respect to the main housing 400. The EVF 403 is one of the display devices 104.

In the rear structure, a rectangular display screen 401 corresponding to the touch panel described above is arranged, and information display and touch input operation are possible. The EVF 403 also includes a display screen 404 and a proximity sensor 405. The EVF 403 displays an image of a subject as a shooting monitor image based on a signal input via a lens and an image sensor during shooting. The user can shoot while checking the shooting monitor image of the EVF 403. Information such as the aperture value may be displayed in a superimposed manner on the display screen 404. The proximity sensor 405 detects the proximity of an object. The proximity sensor 405 is of a capacitance type, an optical type, or the like. For example, the control unit 101 determines whether or not the user is using the EVF 403 by detecting the presence or absence of proximity through the proximity sensor 405. The control unit 101 automatically switches between the display on the display screen 401 and the display of the imaging monitor image on the display screen 404 of the EVF 403 according to the determination. Note that the EVF 403 is not the only option, and an optical finder may be mounted. In addition, a mode in which a shooting monitor image is displayed on the display screen 401 can also be set.

The housing 400 is provided with a mode setting dial 431, a shutter button 432, a cross key 433, a reproduction display mode button 434, a power button 435, and the like as the operation input unit 105. The cross key 433 includes up/down/left/right buttons and a center button, and can be used for selecting information at a desired position in the display screen 401, inputting determination, cancellation, and the like. A menu can be displayed on the display screen 401 based on a predetermined input operation.

The mode setting dial 431 is used to set the operation mode. The shutter button 432 is pressed at the time of shooting. The reproduction display mode button 434 is used when reproducing and displaying on the display screen 401, and the display mode can be switched. The power button 435 is used to turn on/off the power. The user can perform various input operations such as displaying a shooting monitor image on the display screen 401 or the display screen 404 and touching a desired position to focus.

[Imaging device-operation mode]
The operation mode of the image pickup apparatus 1 includes a shooting mode, a reproduction display mode, and the like. The operation outline of the shooting mode is as follows. When the user presses the power button 435 to turn on the power, the normal shooting mode is activated. The control unit 101 drives and controls the imaging unit 103 and the like according to the normal shooting mode. In the normal shooting mode, it is possible to shoot a still image or a moving image as an image. At the time of shooting, the control unit 101 reads out an image pickup signal from the image pickup unit 103 in a predetermined cycle, performs predetermined signal processing in the signal processing unit 132 via the interface circuit unit 131, and converts it into a video signal in a display format. .. Then, the control unit 101 displays the shooting monitor image in real time on the display screen 401 of the touch panel of the display device 104 or the display screen 404 of the EVF 403 based on the video signal. The user shoots as follows while checking the shooting monitor image on the display screen 401 or the display screen 404.

The user sets the angle-of-view related information and the like as shooting conditions in the setting unit 14 through the operation input unit 105. When taking a still image, the user presses the shutter button 432. The imaging control unit 11 detects pressing of the shutter button 432 and determines the aperture value, shutter speed, focus, etc. according to the set shooting conditions. The shutter button 432 is, for example, half-pressed or fully-pressed. The imaging control unit 11 sets focus and the like in the half-pressed state, and controls to start imaging in the fully-pressed state.

A still image pickup signal picked up by the image pickup unit 103 is subjected to predetermined signal processing for a still image by using the signal processing unit 132 and the memory 133. For example, JPEG is applied as the encoding method at that time, but MPEG, high-quality RAW format, or the like may be applied as another method. The image file 24 corresponding to the encoded still image data is recorded on the recording medium 107 by the recording unit 106, for example.

When shooting a moving image, the user presses the moving image shooting button to start moving image shooting, and presses the moving image shooting button again to stop moving image shooting. The image pickup signal of the moving image is subjected to signal processing of a predetermined method for the moving image. As the method, for example, MPEG such as H264 or H265 and other formats are applied. The encoded moving image data is similarly recorded in, for example, the recording medium 107.

The outline of the operation in the reproduction display mode is as follows. When the control unit 101 detects that the reproduction display mode has been set by the reproduction display mode button 434, the control unit 101 reads the image file 24 from the recording medium 107 of the recording unit 106 and stores it in the storage unit 102. The display control unit 12 uses the signal processing unit 132 to restore the original data of the image from the image data 23 by the decoding process, and generates video data for displaying the image or the like on the display screen. The display control unit 12 controls the display device 104 based on the video data and displays an image or the like on the display screen.

[Management information and image files]
FIG. 5 shows a configuration example of the management information 21 and the image file 24 in the storage unit 102. The management information 21 includes image file management information and association management information. The image file management information is information managed by a known file system and includes, for example, file name, update date/time, type, size, and the like. The association management information is peculiar information created to manage the relationship between a plurality of images. The association management information includes, for example, a type identifier, a reference identifier, and a setting character string.

The type identifier is an identifier for identifying the type of image for each image file 24. There are two types of images, for example, a wide-angle image and a narrow-angle image. The reference identifier is an identifier for referring to the association between a plurality of images. The reference identifier is, for example, the file name of the image file 24 of the second image of the reference destination as the reference identifier held in the first image when there is the first image and the second image associated with the first image. Further, it is the file name of the image file 24 of the first image of the reference source as the reference identifier possessed by the second image. As will be described later, the setting character string is a character string that the user can arbitrarily set based on a user operation. The user can set, for example, a character string indicating a shooting location for an image when editing and managing the image. Although the setting character string is provided in the association management information, the setting character string may be provided in the metadata 22.

The image file 24 includes metadata 22 and image data 23. The header part of the image file 24 has the metadata 22, and the body part has the image data 23. The image data 23 is still image or moving image data, and is coded data for recording. The metadata 22 is data for managing information including various attribute values regarding the image data 23, in other words, attached information and attribute information. The metadata 22 includes, as information items, shooting date and time, shooting location, model, resolution, shooting direction, angle of view, angle of view related information, rotation angle, and thumbnail.

The shooting date/time is, for example, year/month/day/hour/minute/second. The shooting location includes position information or a set character string. The position information includes position coordinates {latitude, longitude, altitude} detected by the position detection unit 113. The setting character string is a character string representing the shooting location set by the user. The model is information that uniquely indicates the type of the imaging device 1. The model may include the lens type and the like. The resolution represents the resolution of the image of the image data 23.

The shooting direction is information including an azimuth angle (denoted by θ) and an elevation angle (denoted by φ). The angle of view is information including a horizontal angle of view (Ah) and a vertical angle of view (Av). The angle-of-view related information is information such as the electronic zoom magnification (n), the focal length (f), the size of the image pickup surface of the image pickup device, and the like. The dimensions are {horizontal size sw, vertical size sh, diagonal size sd}. The metadata 22 may have a format in which only one of the view angle item and the view angle related information item is provided.

In other words, the rotation angle is a rotation about the optical axis of the lens at the time of image pickup, and is an angle representing the rotation of the image at the time of image pickup. For example, it is an angle or direction with reference to a state in which the lateral direction of the housing 400 coincides with the horizontal plane as shown in FIG.

The thumbnail is a reduced image based on the original data of the image. The thumbnail is configured according to the resolution and type of the image, and is recorded with or without encoding. When the thumbnail is created and held in advance, the thumbnail can be displayed at high speed during playback display. A plurality of types of thumbnails may be created and held with different resolutions. The thumbnail may be omitted or may be managed separately from the metadata 22. The display control unit 12 may create and display a thumbnail each time reproduction display is performed.

The metadata 22 has a predetermined format, for example, a format extended based on Exif (Exchangeable image file format), but is not limited to this. Exif is a format for photographs of digital cameras. In Exif, the model and shooting condition information are embedded in the image data as metadata. Image display and image editing software can refer to this metadata, and the user can edit the metadata information. The metadata of the conventional Exif image file does not include information such as the angle of view. In the first embodiment, as an expanded format, items such as an angle of view and angle of view related information are provided as specific information items in the metadata 22.

The metadata 22 is created and managed as a header portion of the image file 24, but may be created and managed as a metadata file separately from the image data file. Further, not limited to the form of managing the specific information by using the metadata 22, the form of managing similar information in the management information 21 may be collectively managed. In the case of a moving image, various information such as the metadata 22 may be described for each image frame at a predetermined time interval. The time interval may be constant or may be set only when the angle of view or the like changes.

[Management of relationship between multiple images]
FIG. 6 shows association management regarding the relationship between a plurality of images. The control unit 101 manages the relationship illustrated in FIG. 6 in the association management information of the management information 21 illustrated in FIG. 5, for example. In FIG. 6, the first image has a wide-angle type, and the second image has a narrow-angle type. In this example, the first image includes a first wide-angle image, a second wide-angle image, and the like, and the second image includes a first narrow-angle image, a second narrow-angle image, and the like. For example, the first wide-angle image is associated with a plurality of second images such as a first narrow-angle image, a second narrow-angle image, and a third narrow-angle image. As the association information, for example, an image file name of the first narrow-angle image or the like is provided as a reference identifier to be given to the image file of the first wide-angle image. Further, it has the image file name of the first wide-angle image as a reference identifier given to the image file of the first narrow-angle image. As a result, it is possible to refer to each other among related images.

Note that, as one display mode, information indicating the relationship between a plurality of images as shown in FIG. 6 may be displayed on the display screen. At that time, as shown in FIG. 6, the first image and the second image may be displayed in a tree structure, or, although not shown, the nodes of the second image are connected by link lines around the nodes of the first image. May be displayed with.

In the first embodiment, a wide angle and a narrow angle regarding the angle of view are distinguished as the type of image. The imaging device 1 distinguishes the angle of view by an absolute value based on the threshold value. The first angle of view is set as the threshold for the angle of view. The control unit 101 sets a wide-angle image as the type when the angle of view of the captured image is equal to or larger than the first angle of view, and sets a narrow-angle image when the angle of view is less than the first angle of view. The type of view angle is not limited to this, and three or more types may be provided.

[Image shooting direction and angle of view]
FIG. 7 is an explanatory diagram that briefly shows the concept of the relationship between the shooting direction and the angle of view regarding a wide-angle image and a narrow-angle image that are a plurality of images used as a specific example in the image pickup apparatus 1 according to the first embodiment. FIG. 7 particularly shows the azimuth angle and the horizontal angle of view.

It has a wide-angle image 6a that is a first image, a first narrow-angle image 6b that is a second image, and a second narrow-angle image 6c. These images are images taken by the user using the image pickup apparatus 1 at the same point, that is, a point P0. It is assumed that the respective images were taken at a date close to the wide-angle image 6a, the first narrow-angle image 6b, and the second narrow-angle image 6c in this order. The shooting date and time of each image is, for example, the wide-angle image 6a at 13:11:30 on January 5, 2016, the first narrow-angle image 6b at 13:14:40 on the same day, and the second narrow-angle image 6c on the same day. It is 13:15:50.

The wide-angle image 6a has a shooting range 601 corresponding to the shooting direction 63a indicated by the azimuth angle θ1 and the horizontal field angle Ah1. The first narrow-angle image 6b has a shooting direction 63b indicated by an azimuth angle θ2 and a shooting range 602 corresponding to the horizontal field angle Ah2. The second narrow-angle image 6c has a shooting direction 63c indicated by the azimuth angle θ3 and a shooting range 603 corresponding to the horizontal field angle Ah3. In FIG. 7, among the image capturing directions, the image capturing direction defined by the azimuth angle θ is indicated by a one-dot chain line.

The point P0 particularly indicates the center point of the lens 402 of the imaging unit 103. The spherical surface 600 represents a spherical surface of the whole sphere, and is a virtual spherical surface that excludes the point at infinity. The spherical surface 600 shows, in particular, the portion of the circumference corresponding to the azimuth angle θ and the XY plane. N is north, E is east, S is south, and W is west. The X direction corresponds to the east direction, the Y direction corresponds to the north direction, and the Z direction corresponds to the zenith direction. The azimuth angle θ is shown with N as a reference of 0°, clockwise as a positive direction, E of +90°, S of +180°, and W of +270° (-90°).

An imaging range 601 indicated by an arc on the spherical surface 600 indicates a portion of the wide-angle image 6a corresponding to the horizontal angle of view Ah1 and corresponds to an area projected on the spherical surface 600 viewed from the point P0. In this example, a case where mountains 611, the sky, towers 612, buildings 613, and the like are simply shown on the spherical surface 600 as a landscape captured in the wide-angle image 6a. The first narrow-angle image 6b is an example of photographing the tower 612, and the second narrow-angle image 6c is an example of photographing the building 613.

Specific examples of the values of the azimuth angle θ, the elevation angle φ, the horizontal angle of view Ah, and the vertical angle of view Av of each image are as follows. The wide-angle image 6a has an azimuth angle θ1, an elevation angle φ1, a horizontal angle of view Ah1, and a vertical angle of view Av1, and the first narrow-angle image 6b has an azimuth angle θ2, an elevation angle φ2, a horizontal angle of view Ah2, and a vertical angle of view Av2. The azimuth angle θ3 of the angle image 6c, the elevation angle φ3, the horizontal angle of view Ah3, and the vertical angle of view Av3.

θ1=+10°, φ1=0°, Ah1=120°, Av1=80°
θ2=+45°, φ2=0°, Ah2=21°, Av2=14°
θ3=−20°, φ3=−10°, Ah3=21°, Av3=14°
Similar to FIG. 7, FIG. 8 is an explanatory diagram particularly showing the elevation angle φ and the vertical angle of view Av. In FIG. 8, another narrow-angle image 6d is shown as an example. In FIG. 8, the X direction corresponds to the horizontal direction and the Z direction corresponds to the vertical direction. The positive direction in the Z direction corresponds to the head and zenith directions, and the negative direction corresponds to the foot direction. The elevation angle φ is 0° with respect to the horizontal direction, +90° in the zenith direction of the Z direction, and −90° in the direction of the feet. An imaging range 801 corresponding to the vertical angle of view Av4 in the narrow-angle image 6d is shown. The shooting direction 63d of the narrow-angle image 6d indicates a portion corresponding to the elevation angle φ4.

[View angle and view angle related information]
FIG. 9 simply shows the view angle and the view angle related information. Further, FIG. 9 shows a case where the setting unit 14 stores the angle of view and the angle of view related information as the setting information. {Horizontal angle of view Ah, vertical angle of view Av} is shown as the angle of view of the image. Further, the horizontal size Sh corresponding to the horizontal angle of view Ah and the vertical size Sv corresponding to the vertical angle of view Av in the image are shown. The horizontal size Sh is a size from the right end to the left end in the horizontal direction in the image frame. The vertical size Sv is the size from the upper end to the lower end in the vertical direction in the image frame. In the first embodiment, the horizontal view angle Ah and the vertical view angle Av are used, but the present invention is not limited to this, and at least one view angle including the diagonal view angle Ad may be used.

As the angle-of-view-related information, the focal length f of the lens considering the optical zoom, the electronic zoom magnification n, and the dimensions {horizontal size sw, vertical size sh, diagonal size sd} of the imaging surface of the image sensor are used. The model may be used instead of the size. The focal length f is a 35 mm converted focal length.

The angle of view can also be calculated and obtained based on the relational expression using the angle of view related information. When there is sufficient view angle related information to calculate the view angle, the control unit 101 may calculate and obtain the view angle from the view angle related information. The angle of view can be calculated using, for example, the dimensions of the image sensor, the focal length f, and the electronic zoom magnification n.

The following is the case of calculating the angle of view from the angle of view related information. The imaging control unit 11 refers to the view angle related information regarding the image. The imaging control unit 11 can grasp the view angle related information from the setting information of the setting unit 14 described above or the information described in the metadata 22 or the management information 21. The angle of view {horizontal angle of view Ah, vertical angle of view Av, diagonal angle of view Ad} can be calculated based on the following formula, for example.

Ah=2×arctan(sw/(2×n×f)) (1)
Av=2×arctan(sh/(2×n×f)) (2)
Ad=2×arctan(sd/(2×n×f)) (3)
The focal length f is usually expressed in terms of 35 mm. This corresponds to satisfying the above formula when the size of the image sensor is 35 mm size {sw=35 mm, sh=24 mm}.

For example, when the electronic zoom magnification n, the focal length f, and the dimensions {horizontal size sw, vertical size sh} are given, the horizontal angle of view Ah1 and the vertical angle of view Av1 are determined. The control unit 101 describes the information about the view angle in the view angle item of the metadata 22 or the management information 21. Further, since the angle of view can be calculated from the angle of view related information, the imaging device 1 may store the angle of view related information as one of the metadata 22 instead of storing the angle of view. .. Further, since the size of the image sensor of the angle-of-view related information can be grasped from the model, the image capturing apparatus 1 can store the model as one of the metadata 22 etc. instead of the size of the image sensor. Good. For example, the imaging device 1 holds in advance information indicating the correspondence between the model and the view angle related information. Alternatively, the imaging device 1 can refer to the information indicating the correspondence from an external device by communication. The imaging device 1 can obtain the view angle related information from the model based on the information.

[Save angle of view]
When capturing an image, the image capturing apparatus 1 creates information including a capturing date and time, a capturing direction {azimuth θ, elevation φ}, an angle of view {horizontal angle of view Ah, vertical angle of view Av}, and the like. The imaging control unit 11 obtains the shooting date and time by the internal clock 112. The imaging control unit 11 obtains position information {latitude, longitude, altitude} by the position detection unit 113. The imaging control unit 11 obtains the imaging direction {azimuth θ and elevation φ} by the imaging direction measurement unit 111. The imaging control unit 11 obtains the angle of view {horizontal angle of view Ah and vertical angle of view Av} by the setting information or calculation of the setting unit 14. Then, the imaging device 1 describes the information in the corresponding items of the metadata 22 in FIG. 5, and creates the image file 24 together with the image data 23.

[Display screen (1)-Display mode]
FIG. 10 shows the display modes of the display screen 401 of the display unit 41 and their transitions. The imaging device 1 has at least a first display mode, a second display mode, and a third display mode as display modes that are reproduction display modes. The imaging device 1 first displays in the first display mode, for example, and switches the display mode based on a user operation.

The first display mode is a mode for displaying a list regarding information of all images, and is a mode for displaying a folder screen. On the first screen of the first display mode, information of a plurality of images in the folder is displayed in a list in the order of, for example, the file name and the date and time. In this example, information of images G1, G2, G3, etc. is displayed as a plurality of images arranged in order from the top. In each image row, an icon or thumbnail of the image, a file name, a shooting date and time, a type, and other information are displayed. As other information, attribute values such as shooting direction {azimuth θ, elevation φ} and angle of view {horizontal angle of view Ah, vertical angle of view Av} may be displayed. The user can scroll and display the information that cannot be displayed on the first screen. The first screen is not limited to such a folder screen, and various known methods can be used. For example, a screen in which thumbnails of a plurality of images are arranged in a two-dimensional array and displayed is possible. On the first screen, the user selects a desired first image, for example, the image G1 (corresponding to the wide-angle image 6a). In that case, the imaging device 1 transitions to the second screen in the second display mode when the second image associated with the selected first image exists.

The second display mode is a peculiar mode in which association display is performed on a plurality of images including a wide-angle image and a narrow-angle image. On the second screen of the second display mode, the wide-angle image 6a which is the selected first image 901 is displayed as a whole, and one or more second images having a predetermined relationship are displayed on the first image 901. A certain narrow-angle image is displayed in a state of being superimposed and associated in the state of a predetermined graphic 902. For example, the graphic 902 is a graphic representing the first narrow-angle image 6b, and the graphic 903 is a graphic representing the second narrow-angle image 6c. It should be noted that the broken line frame indicates the region in which the corresponding narrow-angle image is present, for the sake of clarity in explanation. On the second screen, the display state can be changed by operations such as scrolling and enlargement/reduction according to the user operation.

The user selects and operates the desired graphic of the second image on the second screen. The selection operation is, for example, a tap of a figure. In that case, the imaging device 1 makes a transition to the third screen in the third display mode in which the selected second image is displayed. In addition, when a predetermined operation for returning to the first display mode is performed on the second screen, the imaging device 1 returns to the first screen in the first display mode. The predetermined operation is, for example, inputting a first display mode designation, pressing a return button, or the like.

The third display mode is a mode for performing detailed display of a single designated image, for example, a narrow-angle image. In this example, an example in which a narrow-angle image, which is the single second image 904, is displayed on the third screen is shown, but it is also possible to display a single wide-angle image. In the third screen of the third display mode, it is possible to change the display state by scrolling, enlarging/reducing, etc. according to a user operation. When the user performs a predetermined operation on the third screen, for example, a second display mode designation input or a tap of the area of the second image 904, the imaging device 1 returns to the second screen in the second display mode.

Further, when the user wants to browse only the wide-angle image, the imaging device 1 displays only the first image in detail on the display screen according to a predetermined operation. For example, the imaging device 1 hides the graphic of the second image in response to a predetermined operation on the second screen, thereby displaying only the first image. The imaging device 1 returns the figure of the second image to the display state again in response to a predetermined operation from that state. Alternatively, the imaging apparatus 1 transitions to the third screen for displaying the first image in detail in response to a predetermined operation on the second screen, for example, a tap on an area of the second image on the first image where there is no figure. Let

[Display screen (2)-second display mode]
FIG. 11 shows an example of the second screen in the second display mode. This example corresponds to the specific example of FIG. 7 and the like, and shows a case where the wide-angle image 6a is selected and displayed. The case where the first narrow-angle image 6b and the second narrow-angle image 6c are associated with the wide-angle image 6a is shown. The display control unit 12 displays the second screen based on the image file 24 and the management information 21. On the second screen, the wide-angle image 6a that is the first image is displayed as a whole, and a graphic representing the narrow-angle image that is each second image is superimposed and displayed on the first image.

In FIG. 11, the in-screen horizontal direction corresponding to the azimuth angle in the display screen 401 is shown as the x direction, and the in-screen vertical direction corresponding to the elevation angle is shown as the y direction. The horizontal width of the display screen 401 in the x direction is W, and the vertical width in the y direction is H.

The point p1 indicates the center point of the display screen 401. Initially, the entire wide-angle image 6a is displayed in the display screen 401 in a state where the operation such as scrolling or scaling is not performed. First, the position coordinate (Xa, Ya) of the center point corresponding to the shooting direction 63a of the wide-angle image 6a is displayed to be arranged at the point p1. The position coordinate (Xa, Ya) is a point represented by (azimuth angle, elevation angle) and corresponding to (θ1=+10°, φ1=0°).

The horizontal angle of view Ah1 of the wide-angle image 6a is 120°, and the vertical angle of view Av1 is 80°. The wide-angle image 6a is displayed in a horizontal size Sh1 corresponding to the horizontal angle of view Ah1 and a vertical size Sv1 corresponding to the vertical angle of view Av1 in correspondence with the horizontal width W and the vertical width H of the display screen 401. The right end position Xar corresponds to an azimuth angle of 70°, the left end position Xal corresponds to an azimuth angle of −50°, the upper end position Yat corresponds to an elevation angle of 40°, and the lower end position Yab corresponds to an elevation angle of −°. Corresponds to 40°.

A mark, which is a graphic representing a narrow-angle image, is superimposed and displayed on the wide-angle image 6a. In this example, a mark 8b representing the first narrow-angle image 6b and a mark 8c representing the second narrow-angle image 6c are shown as marks. The mark indicates the case of a balloon shape in particular. The mark is variable based on user settings. The center point p2 indicates the position where the mark 8b representing the first narrow-angle image 6b is displayed. The center point p3 indicates the position where the mark 8c representing the second narrow-angle image 6c is displayed. The position (Xb, Yb) of the center point p2 corresponds to (+45°, 0°) in (azimuth angle, elevation angle), and the position (Xc, Yc) of the center point p3 is (-20°, -10°). ) Corresponds to.

The display control unit 12 determines and displays the display position of the mark of the second image on the first image based on the information such as the shooting direction and the angle of view of the second image associated with the first image.

At the position (Xb, Yb) of the center point p2 of the first narrow-angle image 6b, when the wide-angle distortion and the distortion are ignored, the difference in azimuth angle (35°) from the point p1 in the wide-angle image 6a and the difference in elevation angle ( 0°) is used to obtain the following equation.

Xb=(W/2)×tan 35°/tan 60° (4)
Yb=(H/2)×tan0°/tan40°=0 (5)
The position (Xc, Yc) of the center point p3 of the second narrow-angle image 6c is the difference between the azimuth angle (−30°) and the elevation angle from the point p1 of the wide-angle image 6a when the wide-angle distortion and the distortion are ignored. It is obtained by the following formula using (-10°).

Xc=(W/2)×tan(−30°)/tan60° (6)
Yc=(H/2)×tan(−10°)/tan40° (7)
The user selects a desired mark on the display screen 401, for example, taps it. Thereby, the second image corresponding to the mark can be selected and displayed on the third screen in the third display mode. Note that the example of FIG. 11 shows a case where the entire wide-angle image 6a can be initially displayed with respect to the size of the display screen 401. When the entire wide-angle image cannot be displayed on the display screen 401, the display control unit 12 may display the wide-angle image in a part of the wide-angle image, that is, in a part of the angle of view, or may display it as a reduced size so that the entire image fits. You may. Further, in the display screen 401, values such as a shooting direction and an angle of view may be displayed as browsing auxiliary information, or icons for operations such as scrolling and enlargement/reduction may be displayed.

[Display screen (3)-third display mode]
FIG. 12 shows a display of the first narrow-angle image 6b when the mark 8b of the first narrow-angle image 6b is selected, as a display example of the third screen in the third display mode. First, the first narrow-angle image 6b is displayed on the entire display screen 401. At the center point p1 of the display screen 401, the center point p2 of the first narrow-angle image 6b and the mark 8b is displayed. The position (Xb, Yb) of the center point p2 corresponds to {θ2=45°, φ2=0°} which is the shooting direction 63b.

The horizontal field angle Ah2 of the first narrow-angle image 6b is 21°, and the vertical field angle Av2 is 14°. In the display screen 401, the right end position Xbr corresponds to +55.5° in azimuth angle, the left end position Xbl corresponds to +34.5° in azimuth angle, the upper end position Ybt is +7° in elevation angle, and the lower end position Ybb is in elevation angle. Corresponds to -7°.

FIG. 13 shows a display of the second narrow-angle image 6c when the mark 8c of the second narrow-angle image 6c is selected, as an example of the third screen in the third display mode. First, the second narrow-angle image 6c is displayed on the entire display screen 401. At the center point p1 of the display screen 401, the center point p3 of the second narrow-angle image 6c and the mark 8c is displayed. The position (Xc, Yc) of the center point p3 corresponds to {θ3=−20°, φ3=−10°} which is the shooting direction 63c.

The horizontal field angle Ah3 of the second narrow-angle image 6c is 21°, and the vertical field angle Av3 is 14°. On the display screen 401, the right end position Xcr corresponds to an azimuth angle of −9.5°, the left end position Xcl corresponds to an azimuth angle of −30.5°, and the upper end position Yct corresponds to an elevation angle of −3° and a lower end position. Ycb corresponds to -17° in elevation.

The transition from the second display mode to the third display mode is possible without being limited thereto. An operation mode in which the first display mode is directly changed to the third display mode by a predetermined operation without passing through the second display mode is also possible, and the operation mode can be selected according to the user setting or the like.

[Association method]
As the association method, there are a first method and a second method with respect to the timing of performing the association. The first method is a method of associating at the time of image capturing, and the second method is a method of associating at the time of reproducing and displaying. In the first embodiment, the imaging device 1 uses the first method and the display device 2 uses the second method.

In the first method, the imaging device 1 performs the association process using the association unit 15 immediately after capturing an image, and as a result, the association information is described in the metadata 22 or the management information 21. When reproducing and displaying the image, the imaging apparatus 1 can grasp the relationship between the plurality of images by referring to the association information of the metadata 22 or the management information 21, and display the image on the display screen in the second display mode.

In the second method, when the display device 2 reproduces and displays the image file 24 on the display screen, the display device 2 performs the association process using the association unit 215, and as a result, displays the display screen in the second display mode.

In the second method, the display device 2 may omit the description of the association information in the metadata 22 or the management information 21 by performing the association processing each time the reproduction display is performed. Alternatively, in the second method, the display device 2 performs the association process at the first time of reproduction and display of an image, describes the association information in the metadata 22 and the management information 21, and then reproduces the image from the second time onward. At the time of display, the association process may be omitted by referring to the association information.

The image pickup apparatus 1 according to another embodiment may employ the second method. Which association method is adopted may be fixed in terms of mounting, or may be a mode in which the association method can be selected and set according to user settings.

[Processing flow (1)]
14 and 15 show processing flows of the image pickup apparatus 1 and the display apparatus 2 according to the first embodiment. FIG. 14 shows a flow of control processing at the time of capturing an image in the image capturing apparatus 1. FIG. 14 has steps S101 to S106. The same processing is repeated each time an image is captured. The steps will be described below in order.

(S101) The control unit 101 sets an angle of view and the like in the setting unit 14 based on a user operation. The imaging control unit 11 grasps the angle of view and the like based on the setting information of the setting unit 14.

(S102) Based on a user operation, the imaging control unit 11 controls the imaging unit 103 to capture an image, obtain the image data 23, and store it in the storage unit 102. The imaging control unit 11 describes the view angle or the view angle related information in the metadata 22 of the image file 24 or the management information 21.

(S103) Branching is performed according to the first method for associating at the time of image capturing or the second method for associating at the time of reproduction display. In the case of the first method, the process proceeds to S104, and in the case of the second method, the process proceeds to S106. That is, since the image pickup apparatus 1 of the first embodiment adopts the first method, the process proceeds to S104, and when the image pickup apparatus 1 of the other embodiment adopts the second method, the process proceeds to S106.

(S104) The associating unit 15 performs a predetermined associating process. Thereby, a plurality of images including the first image and the second image are associated with each other.

(S105) The associating unit 15 creates association information as a result of the association process and describes it in the metadata 22 of the image file 24 or the management information 21. The imaging control unit 11 stores the image file 24 in the storage unit 102 and records it in the recording medium 107 of the recording unit 106.

(S106) Since the imaging control unit 11 does not perform association at this point, it creates the metadata 22 and the management information 21 that do not include association information. The control unit 101 stores the image file 24 in the storage unit 102 and records it in the recording medium 107 of the recording unit 106.

[Processing flow (2)]
FIG. 15 shows a flow of control processing at the time of reproducing and displaying an image in the image pickup apparatus 1 or the display apparatus 2. The control of FIG. 15 corresponds to the control of changing the display mode as shown in FIG. FIG. 15 has steps S201 to S210. The steps will be described below in order. Hereinafter, the case where the imaging device 1 is the main body will be described, but the same applies when the display device 2 is the main body.

(S201) Based on a user operation, the control unit 101 causes the display control unit 12 to display the first screen in the first display mode on the display screen 401 of the display unit 41 of the display device 104 as shown in FIG. The display control unit 12 receives selection of an image in the first screen and the like according to a user input operation.

(S202) The display control unit 12 branches depending on whether the image selected on the first screen in the first display mode is the wide-angle image that is the first image or the narrow-angle image that is the second image. If it is the first image, the process proceeds to S203, and if it is the second image, the process proceeds to S209.

(S203) The display control unit 12 branches depending on whether it is the first method or the second method. In the case of the second method, the process proceeds to S204, and in the case of the first method, the process proceeds to S206. That is, in the case of the image pickup apparatus 1 according to the first embodiment, the procedure is the first method, and thus the process proceeds to step S206.

(S204) The display control unit 12 uses the associating unit 15 to perform a process of associating a plurality of images immediately before the reproduction display.

(S205) As a result of S204, the display control unit 12 reads the image data 23 of the second image associated with the first image.

(S206) In the case of the first method, the association processing has already been performed. The display control unit 12 confirms the association by referring to the association information in the metadata 22 of the image file 24 of the first image or the management information 21. The display control unit 12 reads the image data 23 of the second image associated with the first image.

(S207) The display control unit 12 displays the image data 23 obtained as a result of S205 or S206 on the display screen in the second display mode. That is, the display control unit 12 displays the graphic of the second image on the first image in a superposed and associated manner as in the second screen of FIG. 11. The display control unit 12 accepts selection of a figure of the second image in the second screen and the like in response to a user input operation. The user taps the mark when he or she wants to browse the second image, and taps the area of the first image when he wants to return to the first screen.

(S208) The display control unit 12 branches depending on whether the image selected on the second screen in the second display mode is a mark representing the second image. If it is the mark representing the second image, the process proceeds to S209. If it is another operation, for example, the operation of the area of the first image, it corresponds to the returning operation, and in that case, the processing returns to S201.

(S209) The display control unit 12 uses the image data 23 of the second image corresponding to the mark selected in S208 to display the third image in the third display mode on the display screen as shown in the third screen of FIG. 2 Display the image in detail. The display control unit 12 receives a returning operation and the like within the third screen in response to a user input operation. For example, when the user wants to return to the second screen, the user taps the area of the second image.

(S210) The display control unit 12 returns to S207 as indicated by A1 when there is an input operation corresponding to the return operation on the third screen in the third display mode, and when there is no input operation, the display control unit 12 returns to the third screen in S209. Continue displaying in display mode. In addition, you may make it return to a 1st screen of a 1st display mode from a 3rd display mode according to a predetermined input operation.

[Association processing (1)]
The associating unit 15 performs a process of associating a plurality of images. The association process includes a first process and a second process. The first process is a process of selecting an image that is a candidate for association based on a selection condition of a predetermined selection method. As the predetermined selection method, there are the following first selection method, second selection method, and third selection method. In the first embodiment, the first method is used. In the modification, the second selection method or the third selection method is used. It should be noted that which of the selection methods is adopted may be fixed in terms of mounting, or may be variably selected by user setting.

[First selection method-shooting location]
FIG. 16 is an explanatory diagram showing the first selection method. The first selection method uses a condition regarding a shooting location as a selection condition. In the first selection method, images whose shooting locations are determined to be substantially the same are selected as candidates. In the first selection method, the position information {latitude, longitude, altitude} by the position detection unit 113 is used to determine the shooting location. When associating, the associating unit 15 refers to the position information {latitude, longitude} described in the shooting location item of the metadata 22 at the time of shooting, for example. {Latitude, longitude} is represented by {LA, LO}.

As illustrated in FIG. 16, the associating unit 15 sets the position 1601 representing the shooting location of the first image in the space, for example, the XY plane, as the center point with the wide-angle image that is the first image as a reference. The position 1601 is represented by {LA1, LO1}. The position range 1600 corresponds to the threshold value of the condition, and indicates a circular region centered on the position 1601 and having a predetermined radial distance. The radial distance defining the position range 1600 may be a fixed set value in design or a set value that can be changed by user setting.

The associating unit 15 compares the position 1601 of the first image with the position of each narrow-angle image that is the second image. In this example, the position 1602 {LA2, LO2} of the first narrow-angle image, the position 1603 {LA3, LO3} of the second narrow-angle image, and the position 1604 {LA4, LO4 of the third narrow-angle image are arranged around the wide-angle image. } Is shown. The associating unit 15 searches for an image included in the position range 1600 with respect to the position 1601 of the first image. In this example, the position range 1600 includes the position 1602 of the first narrow-angle image and the position 1603 of the second narrow-angle image, and does not include the position 1604 of the third narrow-angle image.

The associating unit 15 determines that the second image, the position of which is included in the position range 1600, is an image captured at the same shooting location as the first image, and selects it as a candidate for association. An example of the same shooting location is a case where the user shoots in each direction of 360° while circling the circumference along a handrail on a certain observation deck.

[Second selection method-shooting date and time]
FIG. 17 is an explanatory diagram showing the second selection method. The second selection method uses a condition regarding the shooting date and time as the selection condition. In the second selection method, images whose shooting dates and times are judged to be roughly the same are selected as candidates. When associating, the associating unit 15 refers to, for example, the shooting date/time (year/month/day/hour/minute/second) described in the shooting date/time item of the metadata 22.

The associating unit 15 takes a time point t1 indicating the shooting date and time of the first image with the wide-angle image that is the first image as a reference in the time series of the shooting date and time. In this example, the time point t1 is the date and time DT3. The date/time range 1700 corresponds to the threshold value of the condition, and indicates a range based on a predetermined time including the time point t1 as the center point and the time points before and after the time point t1. The time defining the date and time range 1700 may be a fixed design setting value or a setting value that can be changed by user setting. The date and time range 1700 can be set to, for example, 1 day (24 hours), 1 hour, 5 minutes, or the like.

The associating unit 15 searches for the second image included in the predetermined date/time range 1700 with respect to the time point t1 of the shooting date/time of the first image. The associating unit 15 compares the shooting date and time of each narrow-angle image, which is the second image, with the time t1 of the shooting date and time of the first image. In this example, there is a case where there is a time point t2 of the shooting date and time of the first narrow-angle image, a time point t3 of the second narrow-angle image, and a time point t4 of the third narrow-angle image near the time point t1. In this example, the time point t2 of the first narrow-angle image and the time point t3 of the second narrow-angle image are included in the date/time range 1700, and the time point t4 of the third narrow-angle image is not included. The control unit 101 determines that the second image whose time point is included in the date and time range 1700 is an image captured at the same shooting date and time as the first image, and selects it as a candidate for association.

[Third selection method-Setting character string]
FIG. 18 is an explanatory diagram showing the third selection method. The third selection method uses, as a selection condition, a condition regarding a set character string that can be arbitrarily set by the user. In the third selection method, images whose set character strings are determined to be substantially the same are selected as candidates. When associating, the associating unit 15 refers to the character string set as the setting character string of the association management information after imaging, for example.

FIG. 18 shows an example of a list display of information of a plurality of images on the first screen in the first display mode. In this example, for each image, a file name, shooting date, tag, type, etc. are displayed as information. The “tag” item corresponds to the setting character string. The user can arbitrarily set a character string in the "tag" item by editing work. In this example, the user sets a character string representing a shooting location in the “tag” item. For example, the images G1, G2, and G3 are images taken in the same park, and the character string “A park” is set. Images G4 and G5 are images photographed at the same observatory, and the character string "B observatory" is set. The "type" item displays "wide angle" or "narrow angle" as the type of angle of view of the image.

The associating unit 15 refers to the first image selected by the user, for example, the image G1 as a reference, and refers to the set character string of the “tag” item of the first image. The associating unit 15 refers to and compares the set character strings of the first image with the set character strings of the other narrow-angle images, which are the other second images, for example, the set character strings including the same character string. Find the image you have. For example, the set character strings of the images G1, G2, and G3 are the same. The associating unit 15 determines that these images are images captured at the same shooting location, and selects them as candidates for association.

As a modified example, the control unit 101 may automatically set a character string representing a shooting location at the time of capturing an image, etc. without depending on a user operation. For example, the control unit 101 may determine the character string representing the shooting location based on the position information {latitude, longitude}. The control unit 101 may set a character string in the “file name” item or the like of the image. The control unit 101 may automatically give a file name including the set character string, for example, according to a predetermined rule. An example of the rule of the file name is [shooting date/time]+[set character string (shooting location)]+[individual sequence number].

The above selection methods can also be applied in combination. For example, the condition of the shooting location of the first selection method and the condition of the shooting date and time of the second selection method may be combined. As the condition, it is possible to specify, for example, "up to a year ago at the same shooting location".

[Association process (2)]
The second process in the associating process of the associating unit 15 is as follows. In the second process, with respect to the image selected as the candidate in the first process, the association determination is performed using the information of the shooting direction and the angle of view, and the association is performed according to the determination result. The associating unit 15 refers to the metadata 22 of each image file 24 or the information on the shooting direction and the angle of view in the management information 21. As a condition, the associating unit 15 sets the shooting direction {azimuth θ and elevation φ of the candidate image within the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av} of the wide-angle image that is the first reference image. } Is included. The associating unit 15 determines an image that satisfies this condition as an image to be associated.

In the example of FIG. 7 described above, the azimuth θ2=45° of the shooting direction 63b of the first narrow-angle image 6b is included within the range (−50° to +70°) of the horizontal angle of view Ah1 of the wide-angle image 6a. Therefore, the azimuth angle θ3=−20° in the shooting direction 63c of the second narrow-angle image 6c is included. Although not shown, the vertical angle of view Av1 of the wide-angle image 6a (−40° to +40°) includes the elevation angle φ2=0° of the first narrow-angle image 6b in the shooting direction 63b. The elevation angle φ3=−10° in the imaging direction 63c of the 2 narrow-angle image 6c is included. Therefore, the first narrow-angle image 6b and the second narrow-angle image 6c are associated with the wide-angle image 6a.

When associating the first image and the second image, the associating unit 15 may describe the association management information in the management information 21 as described above, or describe similar information in the metadata 22 of each image. May be.

[Video]
In the above, the case of a still image has been mainly described, but basically the same can be applied to a case of a moving image. When shooting a moving image, the imaging control unit 11 obtains image data 23 of the moving image by shooting while changing the electronic zoom magnification, the angle of view, and the like in consecutive shooting modes. The imaging control unit 11 describes, for the image file 24 of the moving image, information such as the electronic zoom magnification and the angle of view according to the time point in the metadata 22 or the management information 21. Further, at the time of reproducing and displaying the moving image, the display control unit 12 displays a mark or the like on the second screen in the second display mode, for example, in association with the position of the image frame at the start time of the moving image or the position of the image frame in the pause state. May be displayed.

[Effects, etc.]
As described above, according to the imaging device 1 and the display device 2 of the first embodiment, it is possible to easily understand the relationship between a plurality of images including a wide-angle image and a narrow-angle image, and to realize better usability. On the display screen, the related narrow-angle images are automatically displayed on top of the wide-angle image, so that the user can easily recognize the plurality of related images and intuitively grasp them, which is preferable. Can be viewed at. The user can recognize the narrow-angle image existing within the angle of view of the wide-angle image, and can quickly browse by referring to the associated narrow-angle image. The user can suitably browse a large number of images while switching the display mode.

The following are examples of modifications of the first embodiment. As a modified example, the relationship between the angles of view may be determined relative to the type of image. That is, the image pickup apparatus 1 or the display apparatus 2 refers to the angle of view in a plurality of images, sets the image having the maximum angle of view as the first image that is the wide-angle image, and determines an image other than the first image. Is a second image that is a narrow-angle image.

[Modification (1)-Second display mode]
FIG. 19 shows an example of the second screen in the second display mode in the image pickup apparatus 1 of the modified example of the first embodiment. FIG. 19 corresponds to the specific example of FIG. 7. In the display screen 401 of FIG. 19, the graphic representing the second image is not a mark but a transparent frame. For example, the transparent frame 9b represents the first narrow-angle image 6b, and the transparent frame 9c represents the second narrow-angle image 6c. In the transparent frame, the four sides of the rectangle are displayed with lines and colors, the inside of which is a transparent region, and the portion of the first image in the background is displayed as it is. The size of the area of the transparent frame on the display is determined in particular in correspondence with the size of the angle of view of the second image. This makes it easier to understand the relationship between the narrow-angle images in the wide-angle image.

In the example of FIG. 19, the wide-angle image 6a is initially displayed on the display screen 401 at the same reference position as in FIG. It shows a state in which the view angle range has been moved and changed. In FIG. 19, a point corresponding to (+15°, +10°) in (azimuth angle, elevation angle) as the viewpoint position (Xa, Ya) of the wide-angle image 6a is arranged at the center point p1 of the display screen. .. On the display screen 401, a 90° range of the 120° range of the horizontal angle of view of the wide-angle image 6a and a 60° range of the 80° range of the elevation angle are enlarged and displayed. The position Xa is 15° in azimuth and the position Ya is 10° in elevation. In the display screen 401, the right end position Xar corresponds to 60°, the left end position Xal corresponds to −30°, the upper end position Yat corresponds to 40°, and the lower end position Yab corresponds to −20°.

The position coordinate (Xb, Yb) of the center point p2 of the transparent frame 9b corresponds to (+45°, 0°) in (azimuth angle, elevation angle). The horizontal width of the transparent frame 9b is a size corresponding to the horizontal angle of view Ah2=21°, is in the range of 45°±10.5° in azimuth, the left end position Xbl is +34.5°, and the right end position Xbr. Is +55.5°. The vertical width of the transparent frame 9b is a size corresponding to the vertical angle of view Av2=14°, and is in the range of 0°±7° in elevation angle, the upper end position Ybt is +7°, and the lower end position Ybb is −7°. Is.

The position and size of the transparent frame 9c are similarly determined. The position coordinates (Xc, Yc) of the center point p3 of the transparent frame 9c correspond to (-20°, -10°). The horizontal width of the transparent frame 9c has a size corresponding to the horizontal angle of view Ah3=21°, is in the range of −20°±10.5° in azimuth, and the position Xcl at the left end is −30.5°, and the position Xcl at the right end is The position Xcr is −9.5°. The vertical width of the transparent frame 9c is a size corresponding to the vertical angle of view Av3=14°, and is in the range of −10°±7° in elevation angle, the upper end position Yct is −3°, and the lower end position Ycb is −. It is 17°.

The display position {Xbr, Xbl, Ybt, Ybb} of the transmission frame 9b and the display position {Xcr, Xcl, Yct, Ycb} of the transmission frame 9c are obtained by the following equations when wide-angle distortion and distortion are ignored. The coefficient Kw and the coefficient Kh are used.

Xbr=(W/2)-Kw(tan50°-tan55.5°) (8)
Xbl=(W/2)-Kw (tan50°-tan34.5°) (9)
Kw=W/(tan 50°+tan 40°) (10)
Ybt=(H/2)-Kh(tan40°-tan7°) (11)
Ybb=(H/2)-Kh(tan 40°-tan(-7°)) (12)
Kh=H/(tan 40°+tan 20°) (13)
Xcr=(W/2)-Kw(tan50°-tan(-9.5°)) (14)
Xcl=(W/2)-Kw(tan50°-tan(-30.5°)) (15)
Yct=(H/2)-Kh(tan40°-tan(-3°)) (16)
Ycb=(H/2)-Kh(tan40°-tan(-17°)) (17)
Further, in the example of FIG. 19, an example regarding the rotation angle of the image is shown by the transparent frame 9e representing the narrow-angle image at the point p5. This narrow-angle image has an angle α with respect to the x direction.

The display control unit 12 determines the size of the transparent frame using the information of the image capturing direction and the angle of view of the image of the image data 23. The display control unit 12 determines the position and size of the transparent frame so as to correspond to the changed display state when scrolling or enlarging/reducing corresponding to the viewpoint change in the display of the wide-angle image.

In this modified example, the display of the transparent frame has an advantage that the range and positional relationship of the narrow-angle image can be easily understood. As another modification, the thumbnails may be displayed in the same size instead of the transparent frame.

[Modification (2)-Second display mode]
FIG. 20 shows an example of a display screen in the image pickup apparatus 1 of the modified example of the first embodiment. On the upper side of FIG. 20, an example of the first screen in the first display mode is shown. On the first screen, thumbnails of wide-angle images and thumbnails of narrow-angle images are displayed vertically and horizontally side by side in a list. At that time, the size of the thumbnail on the display is determined to be the vertical and horizontal sizes according to the size of the angle of view of the image. The wide-angle image thumbnail 2001 of this example is a thumbnail of a panoramic image having a large horizontal angle of view of 180° or more. A narrow-angle image thumbnail 2002 shows a case where the horizontal angle of view is less than 180°. Note that the number of images when arranged in parallel is variable in one direction, such as two, three, or four. The user can distinguish between wide angle and narrow angle based on the size of the thumbnail.

The lower side of FIG. 20 shows an example of the second screen in the second display mode. The display screen 401 has a main area 1001 and areas other than the main area 1001, such as a right side area 1002 and a lower side area 1003 in this example. In the main area 1001, the marks and the like of the second image are superimposed and displayed on the first image in the second display mode. When the entire first image cannot be displayed in the area 1001, the range of partial angle of view is displayed first with the shooting direction as the center point. In this example, a range corresponding to a part of the horizontal angle of view of the wide-angle image selected on the first screen is displayed in the area 1001. The user can change the display state of the first image in the area 1001 by scrolling, enlarging/reducing, or the like according to the viewpoint change or the like.

As the figure of the second image, different figures may be displayed depending on the type of the image, the shooting date and time, and the like. In this example, circular, rhombic, and star-shaped marks are shown as figures. As the type of image, for example, a type according to the difference in the angle of view may be set based on the user setting or the like. Further, information may be added to the graphic and displayed. In this example, the mark is provided with an identification number. In addition, as the information, an attribute value such as a shooting date and time based on the metadata 22 or the management information 21 may be displayed on the figure, or only the attribute value may be displayed.

In the area 1002 on the right side, information of all the plurality of second images associated with the first image, such as marks and attribute values, is displayed in parallel. In the area 1001, some marks of the second image are displayed and other marks of the second image are not displayed according to the display state. The area 1002 also displays information about the second image that is not displayed in the area 1001. When the mark of the second image is selected in the area 1001 or when the mark of the second image is selected in the area 1002, the transition to the third display mode may be performed. In the lower right area, a thumbnail of the second image selected last is displayed as a preview.

A display mode, information of the first image, operation buttons, setting information, and the like are displayed in the area 1003 on the lower side. For example, information indicating that the current display mode is the second display mode and attribute values such as the shooting date and time of the first image is displayed. Further, a button for switching between displaying and hiding the mark in the second display mode is provided. The mark is displayed in the “ON” state, and the mark is hidden in the “OFF” state, that is, only the first image is displayed. In addition, a return button or the like for the operation of returning from the second display mode to the first display mode is provided.

In addition, in the second screen of the second display mode, when a plurality of associated second images are densely present in the same position and in the same area on the first image, the individual marks are displayed. Is not displayed in an overlapping manner, but one representative mark is displayed. In this example, the representative mark is a star. When the user selects and operates the representative mark, information on the plurality of second images associated with the position of the representative mark is displayed. As a method of displaying the information of the plurality of second images at that time, for example, a pop-up area 1005 is displayed on the first image, and the information of the plurality of second images is displayed in the pop-up area 1005. When the user selects a desired second image from the pop-up area 1005, a transition is made to the third display mode in which the selected second image is displayed in detail.

As another method, information on a plurality of second images associated with the representative mark may be displayed in the area 1002 on the right side. As another method, a plurality of associated second images may be sequentially displayed, that is, a slide show, in a predetermined area in accordance with the selection of the representative mark. For example, when there are a plurality of images taken at the same location and in the same shooting direction and at different shooting dates and times, these images can be associated and continuously reproduced and displayed.

In addition, as another method, an area in which a plurality of second images are densely displayed may be enlarged and displayed in accordance with a selection operation of the representative mark so that the marks of the plurality of second images can be distinguished.

[Modification (3)-External device cooperation]
The imaging device 1 and the display device 2 may store the image file 24 in an external device, or may read the image file 24 from an external device and reproduce and display it. The imaging device 1 and the display device 2 may perform the association process for the image file 24 to be transmitted to the external device, and may transmit the image file 24 including the association information to the external device. Further, the imaging device 1 and the display device 2 may perform the association process for the image file 24 received from the external device, and may display and display the image file 24 in the associated state based on the association information. That is, as an association method other than the above-mentioned first method and second method, an association method is performed before the image file 24 is transmitted to an external device, or an association method is performed after the image file 24 is received from the external device. May be adopted. When transmitting a plurality of image files 24 to an external device, the imaging device 1 and the display device 2 may describe the angle-of-view information and the association information as the metadata 22 for each image file 24, or the plurality of image files 24 may be described. You may provide and describe in a management file different from the image file 24.

For example, in the system of FIG. 1, the imaging device 1 performs an association process on a plurality of captured images before transmitting them to the server 3 to create an image file 24 including an angle of view and association information, and the communication network 4 To the server 3 for storage in the image data DB. The imaging device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. At that time, it is possible to collectively acquire a plurality of image files 24 by specifying, for example, a shooting date and time and a shooting location as search conditions. The imaging device 1 or the display device 2 reproduces and displays the image data 23 of the acquired image files 24 in an associated state based on the reference of the association information.

In addition, for example, the imaging device 1 transmits the image file 24 that includes the angle of view and does not include the association information to the server 3 and saves the image data DB in the image data DB without performing the association process for the plurality of captured images. The imaging device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. The imaging device 1 or the display device 2 performs the association process on the acquired image files 24, and reproduces and displays the images in the associated state based on the angle of view and the association information.

Further, the server 3 may perform the association process for the plurality of image data 23 in the image data DB. The server 3 performs the association process and saves it as an image file 24 including association information. The imaging device 1 or the display device 2 reproduces and displays the plurality of image files 24 acquired from the server 3 based on the reference of the association information.

In addition, for example, the imaging device 1 performs an association process on a plurality of captured images and transmits an image file 24 including an angle of view and association information to the display device 2. The display device 2 acquires the image file 24 and reproduces and displays the image data 23 of the acquired image file 24 in an associated state based on the reference of the association information.

Further, for example, the imaging device 1 transmits the image file 24 including the angle of view and not including the association information to the display device 2 without associating the plurality of captured images with each other. The display device 2 acquires the image file 24, performs an association process on the image data 23 of the acquired image file 24, and reproduces and displays the image data 23 in an associated state based on the association information.

Further, the imaging device 1 and the display device 2 may read the respective image files 24 from a plurality of devices of the same user and integrate and manage them. For example, the first image capturing apparatus 1A and the second image capturing apparatus 1B in FIG. 1 each transmit the image file 24 of the captured image to the first display apparatus 2A without making an association. The first display device 2A performs the associating process on the image files 24 and reproduces and displays the images in the associated state.

According to this modification, the association display function can be realized in a system in which a plurality of devices cooperate, and the user can realize association and storage of a large number of images.

[Modification (4)-Editing function]
As the imaging device 1 and the display device 2 of the modified example, an editing function that allows a user to determine association of a plurality of images and manually set and edit the images is provided. The user can add, delete, or change the associated image on the display screen by using the editing function, and can change the display position of each image.

FIG. 21 shows an example of the second screen in the second display mode in the imaging device 1 and the display device 2 of the modified example. The following shows the case of the imaging device 1 having the editing function, but the same applies to the case of the display device 2 having the editing function. Based on a user setting or an input operation, it is possible to switch between an “ON” state in which editing is valid and an invalid “OFF” state. When the imaging device 1 is in the “ON” state, on the second screen in the second display mode, the association of the second image with the first image, the display position of the figure of the second image, and the like are based on the user input operation. Set to variable. The control unit 101 stores and manages the display position of the graphic of the second image as one of the metadata 22 and the management information 21. During reproduction display, the control unit 101 refers to the information and determines the display position of the graphic.

In this example, a case where a transparent frame is displayed as the figure of the second image is shown. First, the imaging device 1 automatically determines and displays the display position of the second image, as in the first embodiment. For example, a transparent frame 9b1 representing the first narrow-angle image 6b is displayed. The user touches the transparent frame 9b1 of the second image, for example, to move it in a desired direction and position. The control unit 101 updates the display state so as to move the display position of the graphic according to the user operation. In this example, the state after the movement of the transparent frame 9b1 is shown as the transparent frame 9b2. The center point of the first narrow-angle image 6b is the point p21 before the movement to the point p22 after the movement. The control unit 101 updates the management value so that the position after the movement of the figure of the second image is stored as a new display position.

Further, the control unit 101 may display the information for confirming the change of the display position after the movement of the graphic of the second image to confirm with the user whether to change and save. The control unit 101 displays, for example, a pop-up area near the position after the movement, and displays "the display position has been moved. Save/not save". The user confirms this and presses, for example, a "save" button. As a result, the control unit 101 saves the display position after the movement.

Further, as an editing function, it has a function of adding a second image to be associated and a function of canceling the association of the second image. In the case of addition, the user touches a desired position on the display screen, for example, the point p23, and specifies addition. For example, the user can select the second image to be added by using the add button in the pop-up area. The control unit 101 displays the figure relating to the selected second image at that position, and stores the display position of the second image in the management value.

In the case of cancellation, the user selects the graphic of the second image displayed on the display screen and specifies cancellation. For example, the user touches the transparent frame 9c of the second narrow-angle image 6c and presses the release button in the pop-up area. The control unit 101 cancels the association of the second image for which cancellation is designated, erases the display, and updates the management value.

According to this modification, the editing function allows the user to freely edit the associated display of the plurality of images in the second display mode so that the user can easily see and understand the image.

(Embodiment 2)
The imaging device and the display device according to the second embodiment of the present invention will be described with reference to FIGS. Hereinafter, parts of the configuration of the second embodiment different from the configuration of the first embodiment will be described. The image pickup apparatus 1 according to the second embodiment has a function of picking up a spherical image as a wide-angle image and a function of displaying the wide-angle image as a first image in association with each other.

[Spherical image]
In this specification, as shown in FIG. 7, a virtual spherical surface 600 that is set with the positions of the imaging device and the user who is the photographer as the center point is referred to as a celestial sphere. In this specification, the omnidirectional image is an image having an angle of view of at least one of a horizontal angle of view and a vertical angle of view of 360°.

FIG. 22 shows the omnidirectional image in the second embodiment with a horizontal angle of view. The spherical surface 600 corresponding to the celestial sphere has a first wide-angle image 6A, a second wide-angle image 6B, and a narrow-angle image 6C. The first wide-angle image 6A is a wide-angle image having a shooting range 2201 with a horizontal angle of view of 180° when viewed from the front camera side of the image pickup apparatus 1 in the front shooting direction (for example, northward). The second wide-angle image 6B is a wide-angle image having a shooting range 2202 of 180° in the horizontal field angle when viewed from the rear camera side in the shooting direction behind (for example, facing south). The narrow-angle image 6C is a narrow-angle image having a shooting range 2203 with a horizontal view angle Ah in the desired shooting direction (for example, northward) from the front camera side. By combining the first wide-angle image 6A and the second wide-angle image 6B, a omnidirectional image that is a wide-angle image having a shooting range of 360° at the horizontal angle of view is configured. Although not shown, the same applies to the vertical angle of view Av, for example, the first wide-angle image 6A has a shooting range of 180° in the vertical angle of view.

[Imaging device]
FIG. 23 shows a configuration of a part related to the image capturing unit 103 of the image capturing apparatus 1. The image pickup apparatus 1 has a first image pickup section 103A, a second image pickup section 103B, and a third image pickup section 103C as the image pickup section 103, which are connected to the interface circuit 131.

The first image pickup unit 103A is an image pickup unit corresponding to the front camera, and includes a drive unit 31a and a lens unit 32a. The lens unit 32a includes a first lens as a super-wide-angle lens capable of capturing the first wide-angle image 6A in a shooting range (that is, a hemisphere) of 180° at the front horizontal and vertical angles of view. The second imaging unit 103B is an imaging unit corresponding to the rear camera, and includes a drive unit 31b and a lens unit 32b. The lens unit 32b includes a second lens as an ultra wide-angle lens capable of capturing the second wide-angle image 6B in a shooting range (that is, a hemisphere) of 180° in the horizontal and vertical angles of view on the back surface.

The third image pickup unit 103C has the same configuration as the image pickup unit 103 of the first embodiment, and includes a drive unit 31c and a lens unit 32c. The lens unit 32c includes a third lens capable of capturing the narrow-angle image 6C within the set angle range of less than 180° in the front horizontal direction and the vertical direction. The angle of view when the image is captured by the image capturing unit 103C can be set based on a user operation within a predetermined angle of view range of less than 180°. For example, assume that a narrow-angle image 6C having a horizontal angle of view Ah of 50° is captured.

The control unit 101 includes a spherical imaging control unit 11A and an imaging control unit 11B. The omnidirectional imaging control unit 11A drives and controls the first imaging unit 103A and the second imaging unit 103B to capture a omnidirectional image. The omnidirectional imaging control unit 11A controls the first wide-angle image 6A captured by the first imaging unit 103A and the second wide-angle image 6B captured by the second imaging unit 103B to be captured at the same timing. The omnidirectional image capturing control unit 11A inputs the first wide-angle image 6A and the second wide-angle image 6B through the signal processing unit 132 or the like and synthesizes them to obtain an omnidirectional image having a 360° angle-of-view range. Image.

It should be noted that the configuration of the image capturing unit 103, the control unit 101, and the like including the optical system capable of capturing the omnidirectional image in this way can be realized using a known technique.

The spherical image capturing control unit 11A stores the image data 23 of the captured spherical image in the storage unit 102 as the image file 24 together with the metadata 22 including the information such as the angle of view, as in the first embodiment. To do. The imaging control unit 11B stores the image data 23 of the captured narrow-angle image in the storage unit 102 as the image file 24 together with the metadata 22 including information such as the angle of view, as in the first embodiment.

In the image capturing mode of the image capturing apparatus 1, it is possible to capture an image using any of the image capturing units under the control of the control unit 101. Under the control of the control unit 101, the image capturing unit 103A, the image capturing unit 103B, and the image capturing unit 103C can simultaneously capture images at one time. Further, under the control of the control unit 101, the image capturing units 103A, 103B, and 103C can also capture images at individual timings. A celestial sphere image can be captured by simultaneously capturing and synthesizing the image capturing units 103A and 103B. In addition, a hemispherical image of the first wide-angle image 6A or the second wide-angle image 6B can be captured by capturing using one of the image capturing unit 103A and the image capturing unit 103B. Further, it is also possible to simultaneously shoot one of the image capturing unit 103A or the image capturing unit 103B and the image capturing unit 103C. In that case, the one wide-angle image and the narrow-angle image 6C have the same shooting date and time, and can be automatically associated. The control unit 101 may add information such as the type representing the omnidirectional image, the hemisphere image, or the narrow-angle image 6C to the metadata 22 or the management information 21 of the captured image. The control unit 101 may add association information to the metadata 22 or the like for a plurality of images that are simultaneously captured.

[Imaging device-appearance]
FIG. 24 shows an appearance of a mounting example of the image pickup apparatus 1. In this example, a case where the imaging device 1 is a smartphone is shown. The left side of FIG. 24 shows the front camera side of the housing 2400 of the image pickup apparatus 1, and the right side shows the rear camera side. On the front camera side, the first lens 2401 is provided at a predetermined position of the housing 2400, and the third lens 2403 is provided at a predetermined position. On the rear camera side, a second lens 2402 is provided at a predetermined position on the opposite side corresponding to the position of the first lens 2401 on the front camera side. Further, on the rear camera side, a display screen 2404 of the display device 104, operation buttons 2405, and the like are provided.

[Display screen-second display mode]
FIG. 25 shows an example of the second screen in the second display mode in the second embodiment. The first screen in the first display mode is the same as in the first embodiment, and it is possible to display a list of information on spherical images and narrow-angle images. When the spherical image is selected and operated on the first screen in the first display mode, the display control unit 12 displays the second screen in the second display mode as shown in FIG. 25.

The display control unit 12 displays the omnidirectional image as indicated by the circular area 2500 in the second screen. The display control unit 12 first displays the area 2500 in a display state in which the shooting direction (azimuth angle and elevation angle, for example, northward in FIG. 22) of the spherical image is located at the center point p0. With respect to the area 2500, operations such as scrolling and enlargement/reduction can be performed so as to change the display state in response to a viewpoint change in accordance with a user's touch input operation or the like. For example, when the user swipes left in the area 2500, the display state in the area 2500 is updated so that the viewpoint in the spherical image moves rightward, that is, in the positive direction at the horizontal angle of view. To be done. The control unit 101 performs image processing to change the display state at that time. Image processing for changing the display state of the spherical image is a known technique. Further, in particular, as the position of the viewpoint in the initial display state of the omnidirectional image, the elevation angle may be 0°, that is, the horizontal direction, or 90°, that is, the zenith direction.

When there is a narrow-angle image associated with the omnidirectional image, the control unit 101 displays the narrow-angle image on the omnidirectional image 2500 in a predetermined shape or in the same manner as in the first embodiment. Display as a reduced image. In this example, a case of displaying with a transparent frame is shown. The case where three narrow-angle images are associated on the area 2500 of the spherical image is shown. The point p11 has a transparent frame 2501 representing the first narrow-angle image, the point p12 has a transparent frame 2502 representing the second narrow-angle image, and the point p13 represents the third narrow-angle image. It has a transparent frame 2503. With the change of the display state in the area 2500 of the spherical image, the position, shape, size, etc. of the figure of the narrow-angle image are also changed. The area of the narrow-angle image corresponding to the transparent frame is arranged in the spherical surface of the celestial sphere, and therefore has a shape distorted in the circumferential direction, that is, an arc portion. In this example, the transparent frame is simply displayed as a rectangle.

Further, similar to the first embodiment, based on a predetermined operation on the second screen in the second display mode, ON/OFF switching of the display of the graphic of the narrow-angle image, an operation of returning to the first display mode, and the like are performed. It is possible. When the graphic representing the narrow-angle image is selected and operated on the second screen, the control unit 101 displays the narrow-angle image in detail in the third display mode as in the first embodiment.

[Effects, etc.]
According to the second embodiment, when a omnidirectional image is displayed as a wide-angle image, narrow-angle images having a predetermined relationship such as the same shooting location are displayed in association with each other, so that the user grasps the relationship between a plurality of images. It is easy to do and you can browse appropriately.

The following is possible as a modification of the second embodiment. The second imaging unit 103B on the rear camera side may be omitted, and the first imaging unit 103A may be used to capture a hemispherical image having a 180° field angle range in front.

On the second screen in the second display mode, the area 2500 of the spherical image may be converted into a rectangular area instead of a circular shape and displayed. In the rectangular area, an image corresponding to a partial view angle range of the omnidirectional image is displayed, and the display state is changed based on the user operation.

The image capturing apparatus 1 may capture a spherical image in its own device, capture a narrow-angle image captured by an external device, and perform association display. Conversely, the imaging device 1 may capture a narrow-angle image in its own device, capture the spherical image captured by an external device, and perform association display. For example, the second imaging device 1B of FIG. 1 captures a celestial sphere image and acquires the narrow-angle image captured by the first imaging device 1A or the narrow-angle image registered in the server 3 in its own device. Then, the association display may be performed.

An image having an angle of view of 360° has an omnidirectional angle of view regardless of the shooting direction of the camera. Therefore, it is not always necessary to change the display direction in the display screen according to the shooting direction. Therefore, when reproducing and displaying a moving image of spherical images in the second display mode or the third display mode, the following two display methods may be selected. First, similar to the reproduction and display of the narrow-angle image, the image that changes according to the shooting direction at the time of shooting is directly displayed and displayed. Alternatively, secondly, the direction of the image is fixed regardless of the shooting direction by rotating the shooting direction by an angle corresponding to, for example, an azimuth angle based on the shooting direction information recorded as metadata at the time of shooting the moving image. , Play and display in that fixed direction.

In addition, regardless of whether it is a moving image or a still image, when displaying an image, the direction of the imaging device 1 (similar to the display device 2) is detected using the shooting direction measurement unit 111 and the position detection unit 113, and the detected direction is detected. May be displayed so as to match the shooting direction of the image.

(Embodiment 3)
The imaging device and the display device according to the third embodiment of the present invention will be described with reference to FIG. Hereinafter, parts of the configuration of the third embodiment different from the configuration of the first embodiment will be described. The image pickup apparatus 1 according to the third embodiment is provided with a unique shooting mode for associating a plurality of images.

[Shooting mode]
The image pickup apparatus 1 according to the third embodiment has, as shooting modes, a first shooting mode which is a normal shooting mode, and a second shooting mode for making an association. The imaging device 1 switches the shooting mode based on a user input operation or a user setting. The user performs a predetermined operation to switch between the first shooting mode and the second shooting mode at an arbitrary timing through the operation input unit 105 and the like. The predetermined operation may be, for example, pressing a mode switching button.

FIG. 26 illustrates switching of shooting modes on the time axis in the image pickup apparatus 1 according to the third embodiment. The image pickup apparatus 1 is initially in the state of the first photographing mode, and does not perform the associating process with respect to the individual images photographed in the first photographing mode. For example, the image captured at the time point t11 is not associated, and the association information is not described in the metadata 22 or the like.

The user performs an operation of switching from the first shooting mode to the second shooting mode at a desired timing, for example, time T2. In the state of the second image capturing mode, the image capturing apparatus 1 automatically performs a process of associating each image captured during the period. For example, images are captured at time points t1 to t4 in the second shooting mode. The control unit 101 causes the associating unit 15 to perform the associating process on the images, and describes the association information in the metadata 22 or the management information 21. At that time, the associating unit 15 associates the plurality of images as a group. In this example, the images at time points t1 to t4 are associated as a first group. The associating unit 15 describes the group identification information as one of the metadata 22 and the management information 21. The group identification information may be, for example, a character string that uses the start date and time and the end date and time of the period of the second shooting mode, or a character string that uses the number of occurrences of the second shooting mode. In addition, a character string that combines the group identification information with the shooting date and time or the sequence number of each individual image may be added to the file names and metadata 22 of those images.

The user performs an operation of switching from the second shooting mode to the first shooting mode at a desired timing, for example, time T4. The image capturing apparatus 1 does not associate each image captured in the first capturing mode, for example, the image captured at time t12. The control unit 101 may execute the association process when returning to the first shooting mode.

Further, the associating unit 15 determines the types of the wide-angle image and the narrow-angle image regarding the plurality of images captured within the period of the second shooting mode. The associating unit 15 does not associate when only one of the wide-angle image and the narrow-angle image exists. When both the wide-angle image and the narrow-angle image exist, the associating unit 15 associates the second image, which is the narrow-angle image, with the first image, which is the wide-angle image.

At the time of reproduction display, the display control unit 12 confirms the association based on the reference of the group identification information or the like in the image file 24 or the management information 21, and performs the association display.

[Effects, etc.]
According to the third embodiment, it is possible for the user to easily switch the shooting modes and perform the association at their own discretion. The third embodiment corresponds to a mode in which the determination regarding the shooting date and time of the above-described second selection method is automated by using a specific shooting mode.

The following is possible as a modification of the third embodiment. The imaging device 1 automatically switches the shooting mode according to the determination. The imaging device 1 determines the type of the angle of view of the image, for example, the distinction between the wide angle and the narrow angle, at the time of capturing the image. Then, the imaging device 1 automatically switches from the first shooting mode to the second shooting mode when the captured image is a wide-angle image. The imaging device 1 determines the narrow-angle image captured in the second shooting mode, and associates the narrow-angle image with the first wide-angle image. The imaging device 1 returns from the second shooting mode to the first shooting mode after a predetermined time based on the setting has elapsed since the second shooting mode was entered.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above embodiments and can be variously modified without departing from the scope of the invention. Some or all of the functions and the like of the present invention may be realized by hardware such as an integrated circuit, or may be realized by software program processing.

DESCRIPTION OF SYMBOLS 1... Imaging device, 2... Display device, 3... Server, 4... Communication network, 11... Imaging control part, 12... Display control part, 13... Touch detection part, 14... Setting part, 15... Association part, 21... Management Information, 22... Metadata, 23... Image data, 24... Image file, 101... Control unit, 102... Storage unit, 103... Imaging unit, 104... Display device, 105... Operation input unit, 106... Recording unit, 107... Recording medium, 108... Communication interface section, 111... Imaging direction measuring section, 112... Internal clock, 113... Position detecting section, 131... Interface circuit section, 132... Signal processing section, 133... Memory, 901... First image, 902 , 903... Graphic, 904... Second image.

Claims (17)

An imaging unit that captures a still image or a moving image to obtain image data,
A display unit for reproducing and displaying the image on a display screen based on the image data,
A storage unit for storing the image data,
A control unit,
Equipped with
The control unit stores the image data and the first information obtained by the image pickup by the image pickup unit in the storage unit,
The first information is seen containing at least one of azimuth and elevation as the photographing direction of the image, at least one of the angle or the angle of the computable is angle-related information of the image, and
The control unit is
When capturing a second image, the second image capturing direction of the second image, first the first angle case is included within the scope of the shooting direction of the first image stored in said storage unit A first method of associating the second image with the first image,
When the second image capturing direction of the second image is included within the range of the first angle of view of the first image in the first image capturing direction, when the first image is reproduced and displayed, A second method for associating the first image with the second image;
Equipped with
The first method and the second method can be selected by the user,
The display unit displays the first image selected based on the operation of the user in the display screen, and displays the second image associated with the first image on the first image, The second image or the second information indicating the second image is displayed in a superimposed state,
Imaging device. The imaging device according to claim 1,
With the storage unit that stores the image data and the first information,
The angle-of-view related information includes an electronic zoom magnification, a focal length, and the size or model of the image sensor,
When the first information includes the view angle related information, the view angle is calculated from the view angle related information, and the view angle is described in the first information.
Imaging device. The imaging device according to claim 1,
The second information includes a reduced image of the second image, a predetermined figure, a transparent frame, or an attribute value of the second image,
Imaging device. The imaging device according to claim 1,
The display size of the second information is a size corresponding to the angle of view of the second image,
Imaging device. The imaging device according to claim 1,
When there are a plurality of associated second images in the area above the first image, a graphic representing a representative is displayed as the second information,
In response to a selection operation of a graphic representing the representative, information regarding the plurality of associated second images is displayed to enable selection of the second image.
Imaging device. The imaging device according to claim 1,
When the image is captured, the first information is created and stored as metadata or management information of the image data,
After the image capturing, the association is performed, and when the second image is associated with the first image, association information between the first image and the second image is created to generate the metadata or the management information. Described in
At the time of the reproduction display, the association information is confirmed and displayed in the superimposed state,
Imaging device. The imaging device according to claim 1,
When the image is captured, the first information is created and stored as metadata or management information of the image data,
When performing the reproduction display, the association is performed, and when the first image is associated with the second image, the images are displayed in the superimposed state.
Imaging device. The imaging device according to claim 1,
Switching between the first shooting mode and the second shooting mode,
For the image captured in the first shooting mode, the association is not performed,
For the plurality of images captured in the second shooting mode, the association is performed as a group,
Imaging device. The imaging device according to claim 1,
At the time of image capturing, position information indicating a shooting location is created and stored as metadata or management information of the image data,
At the time of the association, when the position information of the second image is included in a predetermined position range with respect to the position information of the first image, it is selected as an image of the association candidate.
Imaging device. The imaging device according to claim 1,
At the time of imaging, information including a shooting date and time is created and stored as metadata or management information of the image data,
At the time of the association, if the shooting date and time of the second image is included within a predetermined time range with respect to the shooting date and time of the first image, the image is selected as an image of the association candidate.
Imaging device. The imaging device according to claim 1,
At the time of the image pickup or the playback display, a character string set based on the operation of the user is created and stored as metadata or management information of the image data,
At the time of the association, when the character string of the second image includes the same character string as the character string of the first image, it is selected as an image of the association candidate,
Imaging device. The imaging device according to claim 1,
The imaging unit is capable of capturing a wide-angle image having an angle of view of 180° or more and 360° or less,
The wide-angle image is used as the first image to perform the association,
Imaging device. The imaging device according to claim 1,
During the reproduction display, based on the operation of the user, on the display screen,
As a first screen of the first display mode, a list of information about the plurality of images is displayed,
As a second screen of the second display mode, the second image or the second information associated with the first image is superposed on the first image selected in the first screen. Display,
As the third screen of the third display mode, the second image selected in the second screen or the second image corresponding to the second information is displayed in detail.
Imaging device. The imaging device according to claim 1,
The display position of the second image or the second information on the display screen during the reproduction display is stored as one of the items of metadata or management information of the image data,
At the time of the reproduction display, based on the operation of the user on the display screen,
Moving the second image or the second information on the first image, and storing the display position after the movement in the item,
Adding an association of the second image to a selected position on the first image,
Disassociating the second image at the selected position on the first image,
Imaging device. An imaging display system in which an imaging device and a display device are connected,
The imaging device is
It is equipped with an imaging unit that captures image data by capturing still or moving image.
Acquiring first information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view,
Within the first angle of the first image capturing direction of the first image, if it contains a second imaging direction of the second image, performs association of the second image to the first image,
The association is
When the second image capturing direction of the second image is included in the range of the first angle of view of the first image capturing direction of the first image when capturing the second image, A first method of associating the second image with one image;
When the second image capturing direction of the second image is included within the range of the first angle of view of the first image in the first image capturing direction, when the first image is reproduced and displayed, The user can select a second method of associating the second image with the first image,
The imaging device transmits an image file including the image data of the first image and the second image, which are associated with each other, to the display device,
The display device,
An input unit for inputting and holding the image file including the image data captured by the image capturing device;
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
And displays the first image selected based on the operation of the user on the display screen, the second image associated with the first image, on the first image, the second image or The second information indicating the second image is displayed in a superimposed state,
Imaging display system. An imaging display system in which an imaging device and a display device are connected,
The imaging device is
It is equipped with an imaging unit that captures image data by capturing still or moving image.
First information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view, and the image data, Sending an image file containing the to the display device,
The display device,
An input unit that inputs and holds the image file captured by the image capturing device, the image file including the first information and the image data,
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
Within the first angle of the first image capturing direction of the first image, if it contains a second imaging direction of the second image, performs association of the second image to the first image,
The association is
When the second image capturing direction of the second image is included in the range of the first angle of view of the first image capturing direction of the first image when capturing the second image, A first method of associating the second image with one image;
When the second image capturing direction of the second image is included within the range of the first angle of view of the first image in the first image capturing direction, when the first image is reproduced and displayed, The user can select a second method of associating the second image with the first image,
The display device displays the first image selected based on an operation of the user in the display screen, and displays the second image associated with the first image on the first image, The second image or the second information indicating the second image is displayed in a superimposed state,
Imaging display system. An imaging display system in which an imaging device, a server device, and a display device are connected,
The imaging device is
It is equipped with an imaging unit that captures image data by capturing still or moving image.
First information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view, and the image data, Sending an image file containing the to the server device,
The server device is
The image file received from the imaging device is stored in a database ,
When the second image capturing direction of the second image is included within the range of the first angle of view of the first image capturing direction of the first image of the plurality of images captured by the image capturing device, the first image performs association of the second image, the
The association is
When the second image capturing direction of the second image is included in the range of the first angle of view of the first image capturing direction of the first image when capturing the second image, A first method of associating the second image with one image;
When the second image capturing direction of the second image is included within the range of the first angle of view of the first image in the first image capturing direction, when the first image is reproduced and displayed, The user can select a second method of associating the second image with the first image,
The server device stores, in the database , the image file including the image data of the first image and the second image that are associated with each other,
The display device,
An input unit that acquires and holds the image file from the server device,
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
And displays the first image selected based on the operation of the user on the display screen, the second image associated with the first image, on the first image, the second image or The second information indicating the second image is displayed in a superimposed state,
Imaging display system.

Images