JP2023142979A - Electronic device, control method for electronic device, program, and storage medium - Google Patents

Abstract

To provide a technique that can suitably make a notification on white-out of an image captured using a plurality of optical systems.SOLUTION: An electronic device has acquisition means, detection means, and control means. The acquisition means is for acquiring an image in which a first image area captured through a first optical system and a second image area captured through a second optical system are aligned side by side. The detection means is for detecting a white-out region in each of the first image area and the second image area. The control means is for controlling to make a predetermined notification when a difference between a white-out region in the first image area and a white-out region in the second image area is equal to or greater than a threshold value.SELECTED DRAWING: Figure 12

Description

The present invention relates to an electronic device, a method of controlling the electronic device, a program, and a storage medium.

A digital camera having two optical systems is known. If the two optical systems are arranged to capture images in the same direction, it is possible to obtain two images with parallax using the two optical systems, and stereoscopic viewing is possible from the two images obtained. You can create beautiful images. If the two optical systems each have a fisheye lens and are arranged to capture images in opposite directions, it is possible to obtain a 360-degree image (global image) from the two images obtained using the two optical systems. can be created.

Furthermore, when blown-out highlights occur in a captured image, the area (over-exposed areas) is highlighted (Patent Document 1).

Japanese Patent Application Publication No. 2005-217898

However, in Patent Document 1, it was only possible to notify a blown-out highlight area of an image captured using one optical system.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique that can suitably provide notification regarding blown-out highlights in images captured using a plurality of optical systems.

The electronic device of the present invention acquires an image in which a first image area imaged through a first optical system and a second image area imaged through a second optical system are lined up. means, a detection means for detecting a blown-out highlight area from each of the first image area and the second image area, an area of the blown-out highlight area in the first image area, and an area of the blown-out highlight area in the first image area; The present invention is characterized by comprising a control means for performing control so as to issue a predetermined notification when the difference between the area of the blown-out highlight area and the area of the blown-out highlight area is larger than a threshold value.

According to the present invention, it is possible to suitably notify regarding blown-out highlights in images captured using a plurality of optical systems.

FIG. 1 is a schematic diagram showing the overall configuration of the system. It is an external view of a camera. FIG. 2 is a block diagram showing the configuration of a camera. FIG. 2 is a schematic diagram showing the configuration of a lens unit. FIG. 2 is a block diagram showing the configuration of a PC. FIG. 2 is a schematic diagram showing the structure of an image file. FIG. 3 is a schematic diagram showing lens information and camera information. It is a flowchart showing the operation of the camera. It is a flowchart showing the operation of the PC. It is a flowchart showing the operation of the PC. It is a schematic diagram of left and right swapping. FIG. 3 is a schematic diagram showing an effective area of an image. It is a schematic diagram of a display screen. FIG. 3 is a schematic diagram showing a clustered brightness saturation map. This is a table showing the center coordinates and areas of overexposed areas. It is a schematic diagram of equirectangular cylinder transformation. It is a flowchart showing the operation of the camera. It is a flowchart showing the operation of the camera.

Embodiments according to the present invention will be described in detail below with reference to the drawings.

<Embodiment 1>
In the first embodiment, an example will be described in which a warning display regarding overexposure in an image captured by a digital camera is displayed on a display device of a personal computer connected to the digital camera.

FIG. 1 is a schematic diagram showing an example of the overall configuration of a system according to the first embodiment. The system according to the first embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached to (connected to) the camera 100. Details of the lens unit 300 will be described later, but by attaching the lens unit 300, the camera 100 can capture one image (still image or video) including two image areas having a predetermined parallax. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100. FIG. 1 shows a configuration in which a camera 100 and a PC 500 are connected to each other so as to be able to communicate with each other by wireless or wire.

2(A) and 2(B) are external views showing an example of the external appearance of the camera 100. 2(A) is a perspective view of the camera 100 seen from the front side, and FIG. 2(B) is a perspective view of the camera 100 seen from the back side.

The camera 100 has a shutter button 101, a power switch 102, a mode changeover switch 103, a main electronic dial 104, a sub electronic dial 105, a movie button 106, and a viewfinder outside display section 107 on the top surface. The shutter button 101 is an operation member for issuing a shooting preparation instruction or a shooting instruction. The power switch 102 is an operation member that turns the power of the camera 100 on and off. The mode changeover switch 103 is an operation member for switching between various modes. The main electronic dial 104 is a rotary operating member for changing set values such as shutter speed and aperture. The sub-electronic dial 105 is a rotary operation member for moving a selection frame (cursor), advancing images, and the like. The video button 106 is an operation member for instructing to start or stop video shooting (recording). The outside viewfinder display section 107 displays various setting values such as shutter speed and aperture.

The camera 100 has a display section 108, a touch panel 109, a direction key 110, a SET button 111, an AE lock button 112, an enlarge button 113, a playback button 114, a menu button 115, an eyepiece section 116, an eyepiece detection section 118, and a touch bar on the back. It has 119. The display unit 108 displays images and various information. The touch panel 109 is an operation member that detects a touch operation on the display surface (touch operation surface) of the display unit 108. The direction key 110 is an operation unit composed of keys (four direction keys) that can be pressed in the up, down, left, and right directions. Processing can be performed depending on the position where the direction key 110 is pressed. The SET button 111 is an operation member that is pressed mainly when determining a selection item. The AE lock button 112 is an operation member that is pressed to fix the exposure state in the shooting standby state. The enlargement button 113 is an operation member for switching the enlargement mode on and off in live view display (LV display) in shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. Further, the enlarge button 113 is used to enlarge the reproduced image or increase the enlargement ratio in the reproduction mode. The playback button 114 is an operation member for switching between shooting mode and playback mode. By pressing the playback button 114 in the shooting mode, the mode shifts to the playback mode, and the latest image among the images recorded on the recording medium 227, which will be described later, can be displayed on the display unit 108.

The menu button 115 is an operation member that is pressed to display a menu screen on the display unit 108 on which various settings can be made. The user can intuitively perform various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. The eyepiece section 116 is a part that the eyepiece 117 looks into and looks into. The user can visually recognize an image displayed on an EVF 217 (Electronic View Finder), which will be described later, inside the camera 100 through the eyepiece unit 116. The eyepiece detecting section 118 is a sensor that detects whether or not the user's eye is close to the eyepiece section 116 (eyepiece finder 117).

The touch bar 119 is a line-shaped touch operation member (line touch sensor) that can receive touch operations. The touch bar 119 can be touch-operated with the thumb of the right hand when the grip section 120 is gripped with the right hand (the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. (possible) position. That is, the touch bar 119 can be operated in a state (photographing posture) in which the user looks into the eyepiece finder 117, looks into the eyepiece 116, and is ready to press the shutter button 101 at any time. The touch bar 119 can be used to perform a tap operation on the touch bar 119 (an operation where you touch and then release the touch without moving the touch position within a predetermined period), a slide operation left or right (an operation where you touch and then move the touch position), etc. Acceptable. The touch bar 119 is an operation member different from the touch panel 109 and does not have a display function. The touch bar 119 functions, for example, as a multi-function bar (M-Fn bar) to which various functions can be assigned.

The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like. The grip part 120 is a holding part formed in a shape that is easy to grip with the right hand when the user holds the camera 100. When the camera 100 is held by gripping the grip part 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated with the index finger of the right hand. Further, in a similar state, the sub-electronic dial 105 and the touch bar 119 are arranged at positions that can be operated with the thumb of the right hand. The thumb rest portion 121 (thumb standby position) is a grip portion provided on the back side of the camera 100 at a location where the right thumb can be easily placed while gripping the grip portion 120 without operating any operating member. The thumb rest portion 121 is made of a rubber member or the like to increase the holding force (grip feeling). The terminal cover 122 protects a connector such as a connection cable that connects the camera 100 to an external device (external device). The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227, which will be described later. The communication terminal 124 is a terminal for communicating with a lens unit (such as a lens unit 200 or a lens unit 300 described later) that is detachable from the camera 100.

FIG. 3 is a block diagram showing an example of the configuration of the camera 100. Note that the same components as in FIGS. 2(A) and 2(B) are given the same reference numerals as in FIGS. 2(A) and 2(B), and descriptions of the components will be omitted as appropriate. In FIG. 3, a lens unit 200 is attached to the camera 100.

First, the lens unit 200 will be explained. The lens unit 200 is a type of interchangeable lens unit that can be attached to and detached from the camera 100. The lens unit 200 is a single lens unit, and is an example of a normal lens unit. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

The aperture 201 is configured to have an adjustable opening diameter. Lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from the system control unit 50, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and focuses by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100 side.

Next, the camera 100 will be explained. The camera 100 includes a shutter 210, an imaging section 211, an A/D converter 212, a memory control section 213, an image processing section 214, a memory 215, a D/A converter 216, an EVF 217, a display section 108, and a system control section 50. .

The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an imaging device (image sensor) configured with a CCD, a CMOS device, or the like that converts an optical image into an electrical signal. The imaging unit 211 may include an imaging plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A/D converter 212 converts the analog signal output from the imaging section 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing processing such as reduction, color conversion processing, etc.) on the data from the A/D converter 212 or the data from the memory control unit 213. Further, the image processing unit 214 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. Through this processing, TTL (through-the-lens) type AF processing, AE (automatic exposure) processing, EF (flash pre-emission) processing, etc. are performed. Furthermore, the image processing unit 214 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs TTL-based AWB (auto white balance) processing based on the obtained calculation result.

Image data from the A/D converter 212 is written into the memory 215 via the image processing section 214 and the memory control section 213. Alternatively, the image data from the A/D converter 212 is written to the memory 215 via the memory control unit 213 without passing through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A/D converter 212, and image data to be displayed on the display unit 108 and EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images, a predetermined period of moving images, and audio. Furthermore, the memory 215 also serves as an image display memory (video memory).

The D/A converter 216 converts the image data for display stored in the memory 215 into an analog signal and supplies it to the display unit 108 and the EVF 217. Therefore, the image data for display written in the memory 215 is displayed on the display unit 108 and the EVF 217 via the D/A converter 216. The display unit 108 and the EVF 217 perform display according to the analog signal from the D/A converter 216. The display unit 108 and the EVF 217 are, for example, displays such as an LCD or an organic EL display. The digital signal that has been A/D converted by the A/D converter 212 and stored in the memory 215 is converted into an analog signal by the D/A converter 216, and is sequentially transferred to the display unit 108 or EVF 217 for display. View display is performed.

The system control unit 50 is a control unit including at least one processor and/or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the camera 100 as a whole. The system control unit 50 executes programs recorded in the nonvolatile memory 219 to implement each process in the flowchart described below. The system control unit 50 also performs display control by controlling the memory 215, the D/A converter 216, the display unit 108, the EVF 217, and the like.

The camera 100 also includes a system memory 218, a nonvolatile memory 219, a system timer 220, a communication section 221, an attitude detection section 222, and an eyepiece detection section 118.

For example, a RAM is used as the system memory 218. In the system memory 218, constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are expanded. The nonvolatile memory 219 is electrically erasable/recordable memory, and for example, an EEPROM is used as the nonvolatile memory 219. The nonvolatile memory 219 records constants, programs, etc. for the operation of the system control unit 50. The program here is a program for executing a flowchart described later. The system timer 220 is a clock unit that measures the time used for various controls and the time of a built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from an external device connected via a wireless or wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) and the Internet. The communication unit 221 can also communicate with external devices using Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images captured by the imaging unit 211 (including live images) and images recorded on the recording medium 227, and can receive images and other various information from external devices. The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 222, it is determined whether the image taken by the imaging unit 211 is an image taken with the camera 100 held horizontally or an image taken with the camera 100 held vertically. Identifiable. The system control unit 50 adds orientation information according to the orientation detected by the orientation detection unit 222 to the image file of the image photographed by the imaging unit 211, or rotates the image according to the detected orientation. Is possible. For example, an acceleration sensor, a gyro sensor, or the like can be used for the posture detection section 222. It is also possible to detect movements of the camera 100 (panning, tilting, lifting, whether it is stationary, etc.) using the posture detection unit 222.

The eyepiece detection section 118 can detect the approach of some object to the eyepiece section 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used as the eye proximity detection section 118. When an object approaches, infrared light emitted from the light projecting section of the eye proximity detection section 118 is reflected by the object and is received by the light receiving section of the infrared proximity sensor. The distance from the eyepiece 116 to the object can be determined based on the amount of infrared rays received. In this way, the eyepiece detection section 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece section 116. The eye-approach detection unit 118 is an eye-approach detection sensor that detects the approach (approach) and separation (departure) of the eye (object) from the eye-apparatus 116 . When an object is detected that approaches the eyepiece unit 116 within a predetermined distance from the non-eye-closed state (non-approaching state), it is detected that the object is close to the eyepiece 116 . On the other hand, when the object whose approach was detected moves away from the eye-close state (approach state) by a predetermined distance or more, it is detected that the eye has left the eye. The threshold value for detecting close eye contact and the threshold value for detecting eye separation may be different by providing hysteresis, for example. Furthermore, after detecting eye contact, the eye remains in close contact until eye separation is detected. After eye separation is detected, it is assumed that the eye is not in close contact until eye contact is detected. The system control unit 50 switches the display unit 108 and the EVF 217 between display (display state) and non-display (non-display state) according to the state detected by the eye proximity detection unit 118 . Specifically, when the camera is in at least a shooting standby state and the display destination switching setting is automatic switching, the display destination is set to the display section 108 and the display is turned on while the eye is not close to the eye, and the EVF 217 is not displayed. Further, while the eye is close to the eye, the EVF 217 is set as the display destination and the display is turned on, and the display section 108 is not displayed. Note that the eye proximity detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used as long as the eye proximity detection unit 118 can detect a state that can be considered as eye proximity.

The camera 100 also includes an outside viewfinder display section 107, an outside viewfinder display drive circuit 223, a power supply control section 224, a power supply section 225, a recording medium I/F 226, an operation section 228, and the like.

The outside viewfinder display section 107 is driven by the outside viewfinder display drive circuit 223 and displays various setting values of the camera 100 such as shutter speed and aperture. The power supply control unit 224 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching the block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. Further, the power supply control section 224 controls the DC-DC converter based on the detection result and the instruction from the system control section 50, and supplies the necessary voltage to each section including the recording medium 227 for a necessary period. The power supply section 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The recording medium I/F 226 is an interface with a recording medium 227 such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording photographed images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be removably attached to the camera 100 or may be built into the camera 100.

The operation unit 228 is an input unit that accepts operations from the user (user operations), and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode changeover switch 103, the touch panel 109, and other operation units 229. Other operation units 229 include the main electronic dial 104, sub electronic dial 105, movie button 106, direction key 110, SET button 111, AE lock button 112, enlarge button 113, playback button 114, menu button 115, touch bar 119, etc. is included.

The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (instruction for photographing preparation) during the operation of the shutter button 101, and outputs the first shutter switch signal SW1. The system control unit 50 starts photographing preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed, so-called full press (photography instruction), and outputs the second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of photographing processes from reading out signals from the imaging unit 211 to generating an image file containing the photographed image and writing it to the recording medium 227. do.

The mode changeover switch 103 switches the operation mode of the system control unit 50 to any one of a still image shooting mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image shooting mode include an automatic shooting mode, an automatic scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). There are also various scene modes, custom modes, etc. that provide shooting settings for each shooting scene. The user can directly switch to any of the above-mentioned shooting modes using the mode changeover switch 103. Alternatively, the user can once switch to the shooting mode list screen using the mode changeover switch 103, and then selectively switch to any one of the plurality of displayed modes using the operation unit 228. Similarly, the video shooting mode may include a plurality of modes.

The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured integrally. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that the light transmittance does not interfere with the display on the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI as if the user could directly operate the screen displayed on the display unit 108 is created. (graphical user interface). The touch panel 109 can use any one of various methods, such as a resistive film method, a capacitive method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. Depending on the method, there are methods that detect a touch when there is contact with the touch panel 109, and methods that detect a touch when a finger or pen approaches the touch panel 109. It's okay.

The system control unit 50 can detect the following operations or states on the touch panel 109.
- A finger or a pen that has not touched the touch panel 109 newly touches the touch panel 109, that is, the start of a touch (hereinafter referred to as Touch-Down).
- A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).
- Moving a finger or pen while touching the touch panel 109 (hereinafter referred to as a touch-move).
- The finger or pen that was touching the touch panel 109 is removed from the touch panel 109 (released), that is, the touch ends (hereinafter referred to as Touch-Up).
- A state in which nothing is touched on the touch panel 109 (hereinafter referred to as Touch-Off).

When a touchdown is detected, a touch-on is also detected at the same time. After touchdown, touch-ons typically continue to be detected unless a touch-up is detected. When a touch move is detected, a touch-on is also detected at the same time. Even if a touch-on is detected, if the touch position does not move, a touch move will not be detected. After all touching fingers and pens are detected to have touched up, touch-off occurs.

These operations/states and the coordinates of the position touched by a finger or pen on the touch panel 109 are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch moves, the direction of movement of a finger or pen on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on changes in position coordinates. If it is detected that a touch move has been made over a predetermined distance, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 109 as if flicking. If a touch move over a predetermined distance and at a predetermined speed or higher is detected, and a touch-up is detected as it is, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Further, a touch operation in which a plurality of points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer to each other is called a pinch-in, and a touch operation in which the touch positions are moved away from each other is called a pinch-out. Pinch out and pinch in are collectively called pinch operation (or simply pinch).

FIG. 4 is a schematic diagram showing an example of the configuration of the lens unit 300. FIG. 4 shows a state in which the lens unit 300 is attached to the camera 100. Note that, in the camera 100 shown in FIG. 4, the same components as those explained in FIG. 3 are given the same reference numerals as in FIG. 3, and explanations of the components will be omitted as appropriate.

The lens unit 300 is a type of interchangeable lens unit that can be attached to and detached from the camera 100. The lens unit 300 is a two-lens lens unit that can capture a right image and a left image with parallax. The lens unit 300 has two optical systems, and each of the two optical systems can image a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 has a field of view (angle of view) of 180 degrees in the horizontal direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the vertical direction (vertical angle, elevation angle, pitch angle). It is possible to capture images of a subject of up to In other words, each of the two optical systems can image the range of the front hemisphere.

The lens unit 300 includes a right eye optical system 301R having a plurality of lenses and a reflecting mirror, a left eye optical system 301L having a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right eye optical system 301R has a lens 302R placed on the subject side, and the left eye optical system 301L has a lens 302L placed on the subject side. Lens 302R and lens 302L face the same direction, and their optical axes are substantially parallel.

The lens unit 300 is a two-eye lens unit (VR180 lens unit) for obtaining a VR180 image, which is one of the formats of VR (Virtual Reality) images that allow two-eye stereoscopic vision. The lens unit 300 has a fisheye lens capable of capturing an approximately 180 degree range in each of the right eye optical system 301R and the left eye optical system 301L. Note that the range that can be captured by the lenses of the right-eye optical system 301R and the left-eye optical system 301L may be about 160 degrees, which is narrower than the 180-degree range. The lens unit 300 transmits the right image formed via the right eye optical system 301R and the left image formed via the left eye optical system 301L to one or two cameras of the camera to which the lens unit 300 is attached. An image can be formed on an image sensor. In the camera 100, the right image and the left image are formed on one image sensor (imaging sensor), and the right image area (the area of the right image) and the left image area (the area of the left image) are arranged side by side. One image (binary image) is generated.

The lens unit 300 is attached to the camera 100 via the lens mount section 304 and the camera mount section 305 of the camera 100. By doing this, the system control section 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300. .

In FIG. 4, a right image formed via the right eye optical system 301R and a left image formed via the left eye optical system 301L are imaged side by side on the imaging unit 211 of the camera 100. That is, the right eye optical system 301R and the left eye optical system 301L form two optical images (subject images) on two areas of one image sensor (image sensor), respectively. The imaging unit 211 converts the formed optical image (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, one image including two image areas with parallax can be obtained from two locations (optical systems): the right eye optical system 301R and the left eye optical system 301L. can. By dividing the acquired image into an image for the left eye and an image for the right eye and displaying it in VR, the user can view a three-dimensional VR image in a range of approximately 180 degrees. In other words, the user can view the image of the VR 180 in three dimensions.

Here, the VR image is an image that can be displayed in VR, which will be described later. VR images include omnidirectional images (omnidirectional images) taken with omnidirectional cameras (omnidirectional cameras) and panoramic images with a video range (effective video range) that is wider than the display range that can be displayed at once on the display unit. etc. are included. Further, VR images are not limited to still images, but also include moving images and live images (images acquired from a camera in almost real time). A VR image has a maximum visual range (effective video range) of 360 degrees in the horizontal direction and 360 degrees in the vertical direction. In addition, VR images have a wider angle of view than can be captured with a normal camera, or a display range that can be displayed at once on the display, even if it is less than 360 degrees in the horizontal direction and 360 degrees in the vertical direction. Images with a wider video range are also included. The image captured by the camera 100 using the lens unit 300 described above is a type of VR image. A VR image can be displayed in VR by setting the display mode of a display device (a display device capable of displaying a VR image) to “VR view”, for example. By displaying a partial range of a VR image with a 360-degree viewing angle, and by changing the orientation of the display device in the left-right direction (horizontal rotation direction), the displayed range can be moved and the user can move the displayed range in the left-right direction. You can watch images from all directions without any seams.

VR display (VR view) is a display method (display mode) in which the display range can be changed, in which an image of a VR image is displayed in a viewing range according to the attitude of the display device. VR display includes "single-eye VR display (single-eye VR view)" in which a single image is displayed by mapping a VR image onto a virtual sphere (distortion correction). In addition, for VR display, "two-eye VR display (two-eye VR view)" is used, in which a VR image for the left eye and a VR image for the right eye are mapped onto a virtual sphere and displayed side by side in the left and right areas. ” is there. By performing "two-eye VR display" using a left-eye VR image and a right-eye VR image that have parallax with each other, it is possible to view these VR images stereoscopically. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image is displayed in a viewing range that corresponds to the orientation of the user's face. For example, at a certain point in a VR image, an image with a field of view centered at 0 degrees horizontally (a specific direction, e.g. north) and 90 degrees vertically (90 degrees from the zenith, i.e. horizontal) is displayed. Suppose there is. If you reverse the orientation of the display device from this state (for example, change the display surface from south to north), the same VR image will be 180 degrees in the left and right direction (opposite direction, for example, south), and in the top and bottom directions. The display range is changed to an image with a viewing range centered at 90 degrees. That is, when the user turns his/her face from north to south (that is, turns backward) while wearing the HMD, the image displayed on the HMD is also changed from the north image to the south image. Note that the VR image captured using the lens unit 300 is an image captured in a range of approximately 180 degrees in front (180° image), and there is no video in a range of approximately 180 degrees in the rear. When such an image is displayed in VR and the attitude of the display device is changed to the side where no image exists, a blank area is displayed.

By displaying the VR image in VR in this way, the user can visually feel as if he or she is inside the VR image (in the VR space) (a sense of immersion). Note that the method of displaying the VR image is not limited to the method of changing the attitude of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel, direction buttons, or the like. In addition, during VR display (in the display mode "VR View"), in addition to changing the display range due to changes in posture, the display range can also be changed in response to touch moves on the touch panel, drag operations with a mouse, pressing direction buttons, etc. May be changed. Note that the smartphone attached to VR goggles (head mount adapter) is a type of HMD.

FIG. 5 is a block diagram showing an example of the configuration of the PC 500. The control unit 501 is, for example, a central processing unit (CPU), and controls the entire PC 500. Read Only Memory (ROM) 502 non-temporarily stores programs and parameters. Random Access Memory (RAM) 503 temporarily stores programs and data supplied from external devices. The recording medium 504 is a hard disk or flash memory fixedly installed in the PC 500, or an optical disk, a magnetic card, an optical card, an IC card, a memory card, etc. that is removable from the PC 500. A file of an image taken by the camera 100 is read from the recording medium 504. The operation unit 505 receives user operations on the PC 500. The operation member used by the user when performing an operation may be a button or a touch panel provided on the PC 500, or may be a keyboard, a mouse, etc. that is detachable from the PC 500. The display unit 506 displays data held by the PC 500, data supplied from the outside, and the like. The display unit 506 may be a part of the PC 500 or may be a display device separate from the PC 500. The communication unit 507 communicates with external devices such as the camera 100. System bus 508 communicatively connects the components of PC 500.

Here, the characteristics of an image captured by wearing the lens unit 300 (two-lens lens unit) will be explained. In the case of the lens unit 200 (normal single-lens lens unit), an image that is vertically and horizontally inverted (an image rotated by 180 degrees) is formed on the imaging unit 211 with respect to the actual view. Therefore, the entire formed image is rotated by 180 degrees to obtain (capture) an image that matches the actual appearance. On the other hand, in the case of the lens unit 300 (two-lens lens unit), the right image and the left image are respectively rotated by 180 degrees with respect to the actual appearance and formed on the imaging unit 211. Although the arrangement of the right image and the left image is not particularly limited, in the first embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211. Therefore, as in the case of the lens unit 200 (normal single-lens lens unit), when the entire formed image (image including the right image area and left image area) is rotated 180 degrees, the right image area and the left image area It is possible to match each to the actual appearance. However, the positions of the right image area and the left image area are swapped. In other words, the left-right positional relationship is reversed, and an image is captured in which the right image area is placed on the left side and the left image area is placed on the right side. Embodiment 1 enables display in which the positions of the right image area and the left image area correspond to two optical systems (the right eye optical system 301R and the left eye optical system 301L).

FIG. 6 is a schematic diagram showing an example of the structure of an image file taken with a twin lens unit. The image file in FIG. 6 includes a header 601, a camera information section 602, a lens information section 603, an other information section 604, and an image data section 605. Information such as the type of image taken is recorded in the header 601. In the camera information section 602, information about the camera used for photographing is recorded as metadata. For example, shooting information such as the shutter speed and aperture at the time of shooting, information on the posture of the camera at the time of shooting, etc. are recorded. In the lens information section 603, information about the twin lens unit used for photographing is recorded as metadata. For example, design values and individual values of the binocular lens unit are recorded. Other information is recorded in the other information section 604 as metadata. For example, in the case of a moving image, information that changes from frame to frame is recorded. In the case of a RAW image, data necessary for development is recorded. Image data is recorded in the image data section 605. In the case of a moving image, not only image data but also audio data is recorded. Although an example has been shown in which camera information, twin lens unit information, and other information are recorded in the same image file, they may be recorded in separate files associated with the image file.

FIG. 7(A) is a schematic diagram showing an example of lens information acquired from a binocular lens unit. Lens information is
1. Lens design value 2. Lens individual value 3. Lens flag 4. Lens focal length 5. Including lens temperature, etc.

The lens design value is a design value for correcting aberrations. In the manufacturing process of the binocular lens unit, errors such as lens eccentricity and tilt occur in each of the two optical systems (left eye optical system 301L and right eye optical system 301R). If left and right swapping, equirectangular conversion, etc., which will be described later, are performed without considering the error, the quality of the binocular VR display will deteriorate and good stereoscopic viewing will become difficult. The lens individual value is a measurement result of an error detected during the manufacturing process of the binocular lens unit. Details of the lens design values and lens individual values will be described later using FIG. 7(B).

The lens flag is a flag indicating that it is a binocular lens unit, and can be used to determine whether the binocular lens unit is used. The lens focal length is the distance from the "principal point" which is the center of the lens to the image sensor (image forming position). The lens focal length may or may not be a common parameter for the two optical systems (left eye optical system 301L and right eye optical system 301R) of the binocular lens unit. In order to perform high-quality twin-lens VR display by performing left-right swapping, equirectangular conversion, etc. with high precision, detailed (high-precision) lens focal lengths are required. The lens temperature is the temperature of the twin lens unit, and is used to understand the environmental temperature at the time of shooting.

FIG. 7(B) is a schematic diagram showing details of lens design values and lens individual values. In the first embodiment, the lens design value and the lens individual value are used for left-right swapping and equirectangular cylinder conversion.

The lens design value is
1. Image circle position 2. Image circle diameter 3. Angle of view 4. Includes distortion correction coefficients, etc.

The image circle position is the optical axis center coordinate of the optical system in the captured image (the image including the right image area and the left image area), and is the center coordinate of the optical axis of the optical system in the captured image (the image including the right image area and the left image area). Prepared for each of the academic 301R). That is, the image circle position is the center coordinate of the image circle (circular fisheye image) formed on the image sensor, and is prepared for each of the right image and the left image. The origin of the coordinates is, for example, the center of the image sensor (the center of the captured image). The image circle position includes horizontal coordinates and vertical coordinates. Note that various information regarding the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position (such as the center or the upper left corner) of the captured image to the center of the optical axis can be used.

The image circle diameter is the diameter of an image circle (circumferential fisheye image) formed on the image sensor. The angle of view is the angle of view of an image circle (circular fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the designed image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which no distortion correction coefficient is set, the distortion correction coefficient may be calculated by interpolation using a plurality of distortion correction coefficients. A polynomial that approximates the relationship between the image height and the distortion correction coefficient may be set. The image circle diameter, angle of view, and distortion correction coefficient are
The parameters may or may not be common to the two optical systems (the left eye optical system 301L and the right eye optical system 301R).

When displaying the circumferential fisheye image, the PC 500 may display a magic window on the circumferential fisheye image. The magic window is a display item that indicates the area that is (first) cut out for VR display. For example, a magic window is displayed based on image circle position, image circle diameter, and viewing angle. By doing this, the display quality of the magic window can be improved. In order to display the magic window appropriately, the PC 500 edits and uses the image circle position, image circle diameter, and angle of view as appropriate. For example, the PC 500 multiplies the image circle position and image circle diameter by a coefficient.

The lens individual value is
5. Image circle misalignment 6. Optical axis tilt 7. Includes image magnification deviation, etc. This information is prepared by measuring each of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the binocular lens.

The image circle positional deviation is a deviation of the center coordinates of the image circle (circular fisheye image) formed on the image sensor from the designed value. For example, the image circle positional deviation includes a horizontal deviation and a vertical deviation. With the design value coordinates (two-dimensional coordinates including horizontal and vertical coordinates) as the origin, horizontal coordinates indicate horizontal deviation, and vertical coordinates indicate vertical deviation. . The optical axis tilt is the deviation of the direction of the optical axis on the subject side from a designed value. For example, the optical axis tilt includes a horizontal shift and a vertical shift. The deviation in each direction is expressed in degrees. The image magnification shift is a shift in the size of the image circle (circumferential fisheye image) formed on the image sensor from the designed value. This deviation is expressed, for example, as a ratio to a design value.

Note that the information included in the lens information is not limited to the information described above. For example, the lens information may be the boundary position of the right image area and the left image area in the photographed image (the position of the edge of the circumferential fisheye image; the amount of shift 1005, 1006, 1009, 1010 in FIG. 10(A), etc.). (positions shown). The lens information may include midpoint coordinates between the right image area and the left image area in the captured image. In many cases, the midpoint coordinates coincide with the center coordinates of the captured image. The lens information may include information indicating the area of the magic window (for example, the coordinates of the upper left corner of the magic window, the width of the magic window, and the height of the magic window). The lens information may include correction data (for example, correction values obtained by calibrating the binocular lens unit) for increasing the accuracy of left-right swapping, equirectangular conversion, and the like.

FIG. 7(C) is a schematic diagram showing an example of camera information generated within the camera. For example, camera information is used to perform high-quality VR display. Camera information is
1. Camera recording area information 2. In-camera accelerometer information 3. Contains right exposure compensation information, etc.

Camera recording area information is information on an effective image area. The effective image area that can be displayed differs depending on the camera sensor and recording mode. The PC 500 uses camera recording area information for more accurate display. The in-camera accelerometer information is attitude information obtained using an in-camera acceleration sensor (level), and represents the attitude of the camera in the roll direction, pitch direction, etc. The PC 500 uses in-camera accelerometer information to understand the attitude of the camera at the time of shooting. The PC 500 performs electronic image stabilization, horizontal correction (zenith correction that brings the vertical direction of the display closer to the vertical direction of real space), etc. based on the grasped posture. The right exposure correction information is an exposure setting value that brings the exposure of the right image area closer to the exposure of the left image area. The PC 500 uses the right exposure correction information in order to perform a two-lens VR display that does not feel strange (feel less strange).

FIG. 8 is a flowchart showing an example of the operation of the camera 100. This operation is realized by the system control unit 50 loading the program recorded in the nonvolatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 is activated, the operation shown in FIG. 8 starts. The operations shown in FIG. 8 are operations for a function (PC live view) that displays a live view image captured by a camera on the display section of the PC. The operation in FIG. 8 is executed when the camera 100 is in a shooting standby state. If a recording start instruction is input from the PC 500 during PC live view operation, still image shooting or video shooting is executed. At this time, PC live view may continue.

In step S801, the system control unit 50 determines whether the camera 100 is compatible with a twin lens unit (for example, the lens unit 300). For example, the system control unit 50 determines whether the firmware version of the system control unit 50 is compatible with a binocular lens unit. If it is determined that it is compatible with a binocular lens unit, the process advances to step S802; otherwise, the process advances to step S811. In Embodiment 1, in the case of a two-lens lens unit, information on the two-lens lens unit (lens information; information regarding two optical systems of the two-lens lens unit) is used for post-processing, unlike in the case of a normal single-lens unit. need to be obtained and recorded. Therefore, the process of step S801 is required.

In step S802, the system control unit 50 determines whether a twin lens unit is attached to the camera 100. If it is determined that the binocular lens unit is attached, the process advances to step S803; otherwise, the process advances to step S811. Note that even when the binocular lens unit is attached from a state in which no binocular lens unit is attached, the process proceeds to step S803. When the two-lens unit is removed from the state in which the two-lens unit is attached and the one-lens unit is attached, the process advances to step S811.

In step S803, the system control unit 50 acquires the design value of the binocular lens unit that is attached (connected). The design value is a design parameter, and is used for left-right swapping and equirectangular cylinder conversion, which will be described later. For example, the image circle position, image circle diameter, angle of view, and distortion correction coefficient shown in FIG. 7(B) are obtained.

In step S804, the system control unit 50 acquires the individual value of the attached (connected) binocular lens unit. The individual value is a parameter specific to the lens unit, and is, for example, an error during manufacturing. For example, the image circle position shift, optical axis tilt, and image magnification shift shown in FIG. 7(B) are acquired. By using individual values, image processing can be performed with higher precision than when using only design values.

In step S805, the camera 100 is connected to the PC 500, and the system control unit 50 detects that the camera 100 is connected to the PC 500. In step S806, the system control unit 50 receives a PC live view start request from the PC 500. In step S807, the system control unit 50 receives a live view image request from the PC 500. The live view image request includes information (resolution information) specifying the resolution of the live view image, as will be described later. The system control unit 50 transmits the live view image of the specified resolution to the PC 5.
The process of step S809 is executed to transmit the data to 00.

In step S808, the system control unit 50 converts the information acquired in steps S803 and S804 (lens information of the binocular lens unit) to match the coordinate system of the live view image to be transmitted. Since the captured image (image recorded in the image file) and the live view image have different resolutions, the information acquired in steps S803 and S804 cannot be used as is for image processing of the live view image. . Therefore, in the first embodiment, lens information is converted into information that matches the coordinate system of the live view image.

In step S809, the system control unit 50 transmits the lens information converted in step S808 and the live view image to the PC 500. The system control unit 50 converts the resolution of the live view image based on the resolution information acquired in step S807, and transmits it to the PC 500. Note that in the first embodiment, the system control unit 50 of the camera 100 converts the lens information, but the control unit 501 of the PC 500 may also convert the lens information. At that time, the lens information before conversion and the parameters necessary for converting the lens information are transmitted to the PC 500.

In step S810, the system control unit 50 determines whether to end the PC live view. For example, when the connection between the camera 100 and the PC 500 is canceled or when the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined that the PC live view is to be ended. If it is determined that the PC live view should be ended, the operation in FIG. 8 is ended; otherwise, the process advances to step S807.

If a single lens unit is attached to the camera 100, the process of step S811 is performed. In step S811, the system control unit 50 transmits the live view image captured by the single lens unit to the PC 500. The process in step S811 is similar to the conventional process of transmitting a live view image captured by a single-lens lens unit to an external device, so a detailed explanation will be omitted. When the process of step S811 is completed, the operation of FIG. 8 is ended.

FIG. 9A is a flowchart illustrating an example of the operation of the PC 500. This operation is realized by the control unit 501 loading a program (application program) recorded in the ROM 502 onto the RAM 503 and executing it. For example, when the user instructs the PC 500 to launch a specific application, the operation shown in FIG. 9A starts. The operation in FIG. 9A is an operation for a function (PC live view) of displaying a live view image captured by a camera on the display section of the PC.

In step S901, a camera (for example, camera 100) is connected to the PC 500, and the control unit 501 detects that the camera is connected to the PC 500.

In step S902, the control unit 501 determines whether the camera connected in step S901 is compatible with a twin lens unit (for example, the lens unit 300). For example, the control unit 501 acquires model information of the connected camera from the camera, and determines whether the camera is compatible with a twin-lens unit based on the obtained model information. If it is determined that the camera is compatible with a twin lens unit, the process advances to step S903; otherwise, the process advances to step S920. A camera compatible with a twin lens unit is, for example, a camera to which a twin lens unit can be attached.

In step S903, the control unit 501 determines whether the firmware of the camera connected in step S901 is compatible with a twin lens unit. For example, the control unit 501 acquires information about the firmware version of the connected camera from the connected camera, and determines, based on the acquired information, that the firmware version of the connected camera is compatible with the twin lens unit. Determine whether it exists or not. If it is determined that it is compatible with a binocular lens unit, the process advances to step S904; otherwise, the process advances to step S920.

Even if a camera that is compatible with a twin-lens unit is connected to the PC500, the connected camera may not be compatible with the twin-lens unit due to reasons such as the firmware version of the connected camera being outdated. . Therefore, the process of step S903 is required. Furthermore, various cameras can be connected to the PC 500, and a camera that is not compatible with a twin lens unit may be connected regardless of the firmware version. Therefore, the process in step S902 is required before the process in step S903.

In step S904, the control unit 501 determines whether a twin lens unit is attached to the camera connected in step S901. If it is determined that the binocular lens unit is attached, the process advances to step S905; otherwise, the process advances to step S920.

In step S905, the control unit 501 transmits a PC live view start request to the camera connected in step S901.

In step S906, the control unit 501 determines whether to perform circumferential fisheye display. In the circumferential fisheye display, areas of the circumferential fisheye image are displayed as the right image area and the left image area, respectively. If it is determined that circumferential fisheye display is to be performed, the process advances to step S907; otherwise (if equirectangular cylinder display is to be performed), the process advances to step S913. In the circumferential fisheye display, areas of the circumferential fisheye image are displayed as the right image area and the left image area, respectively. In the equirectangular display, areas of the equirectangular image are displayed as the right image area and the left image area, respectively. In step S906, for example, it is determined whether a circular fisheye display is to be performed depending on whether the radio buttons 1205 in FIGS. 12(A) to 12(D) are in a selected state or in a non-selected state. In FIGS. 12(A) and 12(C), the radio button 1205 is in a selected state, and in FIGS. 12(B) and 12(D), the radio button 1205 is in a non-selected state. If the radio button 1205 is in the selected state, it is determined that circumferential fisheye display is to be performed, and the process advances to step S907. If the radio button 1205 is in a non-selected state, the process advances to step S913. Details of FIGS. 12(A) to 12(D) will be described later.

In step S907, the control unit 501 transmits a live view image request to the camera connected in step S901. In the first embodiment, it is assumed that the live view image request in step S907 is a request for a normal resolution live view image. It is assumed that the normal resolution is 4K resolution in the first embodiment.

In step S908, the control unit 501 receives, from the camera connected in step S901, a live view image captured by the camera and lens information of the twin lens unit attached to the camera. The resolution of the live view image received in step S908 is the normal resolution. The lens information received in step S908 is information converted to match the received live view image (for example, the lens information converted in step S808 of FIG. 8). Note that the live view image and lens information received in step S908 are the live view image and lens information transmitted from the camera in step S809.

In step S909, the control unit 501 determines whether to perform left-right swapping (exchanging the right image area and the left image area). If it is determined that left and right swapping is to be performed, the process advances to step S910; otherwise, the process advances to step S912. In step S909, it is determined whether or not to perform left-right swapping, for example, based on whether the check box 1207 in FIGS. 12(A) and 12(C) is checked. If the check box 1207 is checked, it is determined that left and right swapping is to be performed, and the process advances to step S910. If the check box 1207 is not checked, the process advances to step S912.

In step S910, the control unit 501 exchanges the positions of the right image area and the left image area in the live view image obtained in step S908 based on the lens information obtained in step S908, and generates a processed live view image ( swap left and right). The control unit 501 determines the right image area and the left image in the live view image based on the center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R) included in the lens information received together with the live view image. Swap the positions of the regions and generate a processed live view image. For example, the control unit 501 specifies the right image area based on the center coordinates of the right image area, and specifies the left image area based on the center coordinates of the left image area. Then, the control unit 501 swaps the positions of the two identified areas. In the first embodiment, the right image area and the left image area are arranged side by side in the photographed image, and the horizontal positional relationship between the right image area and the left image area is reversed by the left and right swapping. In order to specify the right image area and the left image area with higher precision, the respective diameters (diameters or radii) of the right image area and the left image area may be acquired from the information of the binocular lens unit.

Note that the method for left and right swapping is not limited to the above method. For example, the deviation amounts 1005, 1006, 1009, and 1010 in FIG. 10(A) are acquired from the information on the binocular lens unit. Then, when exchanging the positions of the right image area 1003 and the left image area 1007, the right image area and the left image area may be arranged so that the obtained amount of deviation is maintained, and the remaining area may be filled with black or the like. . The amount of deviation 1005 is the distance from the left edge of the captured image to the left edge of the right image area 1003, and the amount of deviation 1006 is the distance from the center of the captured image to the right edge of the right image area 1003. When swapping left and right, the amount of shift 1005 is the distance from the left edge of the captured image to the left edge of the left image area, and the amount of shift 1006 is the distance from the center of the captured image to the right edge of the left image area. Make it. Similarly, the amount of shift 1009 is the distance from the right edge of the captured image to the right edge of left image area 1007, and the amount of shift 1010 is the distance from the center of the captured image to the left edge of left image area 1007. When swapping left and right, the shift amount 1009 is the distance from the right edge of the captured image to the right edge of the right image area, and the shift amount 1010 is the distance from the center of the captured image to the left edge of the right image area. Make it.

In step S911, the control unit 501 displays the live view image after the left-right swapping on the display unit 506. The live view image after the left and right swapping is the processed live view image generated in step S910.

In step S912, the control unit 501 displays the live view image acquired in step S908 on the display unit 506.

In step S921, the control unit 501 performs blown-out highlights display processing to distinguishably display blown-out highlights in each of the right image area and the left image area. Details of the blown-out highlight display process will be described later using FIG. 9B.

As described above, when displaying an equirectangular cylinder, the process advances from step S906 to step S913. In step S913, the control unit 501 transmits a live view image request to the camera connected in step S901. In the first embodiment, it is assumed that the live view image request in step S913 is a request for a low resolution (lower resolution than normal resolution) live view image. When performing equirectangular display, equirectangular conversion (conversion from a circumferential fisheye image to an equirectangular image) is required. , and the delay due to equirectangular transformation increases. In the first embodiment, a low-resolution live view image is requested in order to speed up equirectangular conversion (reduce the time required for equirectangular conversion). Note that as long as the delay due to equirectangular conversion is within an allowable range, a live view image of normal resolution may be requested even when equirectangular display is performed.

In step S914, the control unit 501 receives, from the camera connected in step S901, a live view image captured by the camera and lens information of the twin lens unit attached to the camera. The resolution of the live view image received in step S914 is low resolution. The lens information received in step S914 is information converted to match the received live view image (for example, the lens information converted in step S808 in FIG. 8). Note that the live view image and lens information received in step S908 are the live view image and lens information transmitted from the camera in step S809.

In step S915, the control unit 501 determines whether to perform left-right swapping. If it is determined that left and right swapping is to be performed, the process advances to step S916; otherwise, the process advances to step S918. In step S915, it is determined whether or not to perform left-right swapping, for example, based on whether the checkbox 1207 in FIGS. 12(B) and 12(D) is checked. If the check box 1207 is checked, it is determined that left and right swapping is to be performed, and the process advances to step S916. If the check box 1207 is not checked, the process advances to step S918.

In step S916, the control unit 501 swaps the positions of the right image area and the left image area in the live view image acquired in step S914 based on the lens information acquired in step S914, and also switches the positions of the right image area and the left image area in the live view image acquired in step S914. Convert to the domain of an equirectangular image. That is, the control unit 501 performs left-right swapping and equirectangular cylindrical transformation to generate a processed live view image. Equirectangular conversion is a conversion process that treats a circumferential fisheye image as a sphere and converts it so that latitude lines (horizontal lines) and meridian lines (vertical lines) intersect at right angles, similar to the equirectangular projection of maps. . By equirectangular transformation, a circular circumferential fisheye image is converted into a rectangular equirectangular image. The control unit 501 adjusts the lens design value acquired in step S914 based on the lens individual value acquired in step S914. For example, the image circle position (the center coordinates of the right image area and the left image area in the photographed image) is adjusted based on the image circle position shift in FIG. 7(B). When the lens individual value is a difference from the lens design value, the lens individual value is added to the lens design value. If the lens individual value is the same absolute value as the lens design value, the lens design value is replaced with the lens individual value. The control unit 501 generates a map for equirectangular cylinder conversion based on the adjusted center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R). The map indicates to which position in the image before transformation each pixel corresponds after transformation. A map for equirectangular conversion is generated so that not only a circumferential fisheye image can be converted into an equirectangular image, but also the positions of the right image area and the left image area can be corrected. By using the adjusted center coordinates, more accurate equirectangular cylinder transformation becomes possible. In this way, the control unit 501 generates a map based on the lens design value corrected using the lens individual value included in the lens information received together with the live view image, and based on the map, performs correction including left and right swapping. Conversion to cyclocylindrical representation may also be performed. The left and right swapping may be performed as part of the equirectangular conversion, or may be performed as separate processes.

In step S917, the control unit 501 displays, on the display unit 506, the live view image that has been left and right swapped and subjected to equirectangular conversion. The live view image that has undergone left and right swapping and equirectangular conversion is the processed live view image generated in step S916.

In step S918, the control unit 501 corrects each of the right image area and the left image area from the area of the circumferential fisheye image without exchanging the positions of the right image area and the left image area in the live view image acquired in step S914. Convert to a rectangular cylinder image area. That is, the control unit 501 generates a processed live view image by performing equirectangular cylindrical transformation without performing left-right swapping.

In step S919, the control unit 501 displays the live view image after equirectangular conversion on the display unit 506. The live view image after equirectangular conversion is the processed live view image generated in step S918.

If the camera 100 is not compatible with a twin lens unit, or if the camera 100 is equipped with a single lens unit, the process of step S920 is performed. In step S920, the control unit 501 displays the live view image captured by the single lens unit on the display unit 506. The process in step S920 is similar to the conventional process in which a PC or the like displays a live view image captured by a single-lens lens unit, so a detailed explanation will be omitted. When the process of step S920 is finished, the process of FIG. 9A is finished.

In each of step S910, step S916, and step S918, the control unit 501 performs image processing on the live view image obtained from the connected camera. In step S922 after step S910, step S916, and step S918, the control unit 501 determines whether to end the PC live view. If the PC live view is to be continued, the process returns to step S906 before steps S910, S916, and S918. Therefore, in the operation of FIG. 9A, there is a possibility that the image processing in step S910, step S916, or step S918 will be repeatedly executed. If it is determined that the PC live view should be ended, the process in FIG. 9A is ended.

Therefore, in order to speed up the image processing, the control unit 501 may record information related to the executed image processing in the RAM 503, and use the information in the next image processing. For example, the control unit 501 records the correspondence relationship (image processing map) between pixels before image processing and pixels after image processing. The image processing map can continue to be used as long as there is no change in the resolution of the live view image or lens information. When the control unit 501 executes the image processing in step S910, step S916, or step S918, it records an image processing map of the image processing. Then, when executing the same image processing again, the control unit 501 executes the image processing using the recorded image processing map. By doing so, image processing can be speeded up.

10(A) and 10(B) are schematic diagrams of left and right swapping. FIG. 10(A) shows conventional left-right swapping that does not use information on the binocular lens units. FIG. 10(B) shows left-right swapping in the first embodiment using information on the binocular lens units.

As shown in FIGS. 10(A) and 10(B), in the image 1001 before left-right swapping, the right image area 1003, which is the area of the circumferential fisheye image, is located on the left side, and the right image area 1003, which is the area of the circumferential fisheye image, is located on the left side. A left image area 1007, which is an area, is arranged on the right side.

In FIG. 10A, the image 1001 is divided into a left half image area and a right half image area at the center coordinates 1002 of the image 1001, and the left half image area and the right half image area are exchanged. In other words, the left half image area is moved to the right side of the right half image area. Image 1011 is an image after such left and right swapping.

In FIG. 10(A), the amount of deviation 1006 is smaller than the amount of deviation 1005. That is, in the image 1001, the right image area 1003 is closer to the center of the image 1001 than the center of the left half of the image 1001. Similarly, the amount of deviation 1010 is smaller than the amount of deviation 1009. In other words, in image 1001, left image area 1007 extends from the center of the right half of image 1001 to image 1001.
is near the center of Therefore, in the image 1011, the center coordinates 1013 of the left image area 1007 in the left-right direction deviate from the center coordinates 1004 by a distance 1014, and the center coordinates 1016 of the right image area 1003 in the left-right direction deviate by a distance 1017 from the center coordinates 1008.

In the first embodiment, by using lens information, in the image 1037 (FIG. 10B) after left-right swapping, the center of the left image area in the left-right direction is made to match the center coordinates 1004, and the center of the right image area in the left-right direction is The center can be made to coincide with center coordinates 1008.

FIG. 9B is a flowchart illustrating an example of blown-out highlight display processing performed by the control unit 501 in step S921 of FIG. 9A. In the blown-out highlight display process of FIG. 9B, the blown-out highlights are displayed in a distinguishable manner in each of the right image area and the left image area, and the area of the blown-out highlights in the right image area and the area of the blown-out highlights in the left image area are A predetermined notification (warning) is issued when the difference (absolute value) is larger than the threshold value. In the first embodiment, an example will be described in which a predetermined notification is given by displaying on the display unit 506, but the predetermined notification may be given by a method different from the display, such as by outputting audio or lighting a lamp.

In step S923, the control unit 501 determines the right image area (effective area) and left image area (effective area) in the live view image obtained from the connected camera, based on the lens information obtained from the connected camera. do. For example, an area centered on the coordinates based on the optical axis center coordinates (image circle position) of the left eye optical system 301L and having a diameter based on the image circle diameter of the left eye optical system 301L is determined as the left image area. Similarly, an area centered on the coordinates based on the optical axis center coordinates (image circle position) of the right eye optical system 301R and having a diameter based on the image circle diameter of the right eye optical system 301R is determined as the right image area. Similar to step S916, adjustment of the image circle position, etc. may be performed.

Depending on the combination of the lens unit and the shooting settings, there may be an area in each of the right image area and the left image area that is unnecessary for stereoscopic viewing based on them (area that cannot be used for stereoscopic viewing). For example, there may be a shooting range area that exists in the right image area but not the left image area, or a shooting range area that exists in the left image area but not in the right image area. In such a case, in step S923, an area obtained by removing an area unnecessary for stereoscopic viewing from the left image area may be determined as the left image area (effective area). Similarly, an area obtained by removing an area unnecessary for stereoscopic viewing from the right image area may be determined as the right image area (effective area).

When capturing a UHD image using a twin-lens lens unit with a DCI angle of view, which is a type of full-frame compatible lens unit, an image 1101 with both left and right ends missing is captured (recorded) as shown in FIG. Ru. Even when capturing an APS-C or Super35 image using a full-frame compatible lens unit, the image is captured (recorded) with both left and right edges missing. Then, the image 1101 becomes an image 1137 by transposing the left and right sides. Regions 1109 and 1110 in image 1137 are photographed but cannot be used for stereoscopic viewing on an HMD or the like, and therefore are excluded from the effective region. When areas 1109 and 1110 are excluded, the left image area (effective area) in image 1137 becomes area 1107, and the right image area (effective area) in image 1137 becomes area 1103. Note that the regions 1109 and 1110 may or may not be left. For example, the areas 1109 and 1110 may be converted into black areas at a predetermined timing such as when the left and right sides are swapped. The area 1111 including the area 1109 and the area 1112 including the area 1110 may be removed from the image 1037 so that the image size after the left-right swap matches the image size before the left-right swap.

For example, the control unit 501 determines whether both left and right ends of the live view image obtained from the connected camera are missing, based on the lens design value (angle of view of the lens unit) and shooting settings obtained from the connected camera. It is possible to determine areas that are unnecessary for stereoscopic viewing.

In step S924, the control unit 501 controls the brightness value indicating whether each pixel is a pixel with a saturated brightness value (saturated pixel; blown-out highlight pixel) for each of the right image area and left image area determined in step S923. Create a saturation map.

In step S925, the control unit 501 uses the brightness saturation map created in step S924 to determine whether a blown-out highlight pixel exists in at least one of the right image area and the left image area. If it is determined that a blown-out highlight pixel exists, the process advances to step S926; otherwise, the process in FIG. 9B is ended and the process advances to step S922.

In step S926, the control unit 501 uses the brightness saturation map created in step S924 to apply a zebra pattern or the like to overexposed pixels in the right image area and left image area of the live view image displayed on the display unit 506. A predetermined graphic is displayed in a superimposed manner.

In step S927, the control unit 501 uses the brightness saturation map created in step S924 to perform clustering of the blown-out highlights in the right image area and the left image area so that a blown-out area consisting of blown-out highlights is set. conduct.

Through the processing in steps S924 to S927, blown-out highlights are detected from each of the right image area and the left image area determined in step S923, and the detected blown-out highlights are displayed so that they can be identified.

In step S928, the control unit 501 calculates the center coordinates and area (for example, number of pixels) of the overexposed area set in step S927.

In step S929, the control unit 501 associates the blown-out highlights area in the right image area with the blown-out highlights area in the left image area. Since the right image area and the left image area are substantially the same image area, it is considered that the relative position of the blown-out highlight area with respect to the right image area and the relative position of the blown-out highlight area with respect to the left image area are approximately the same. Therefore, from all the blown-out areas set in step S927, two blown-out areas where the relative position of the blown-out area with respect to the right image area and the relative position of the blown-out highlight area with respect to the left image area are approximately the same are searched, The two blown-out highlights areas are associated with each other. If the distance between the center coordinates of two blown-out highlights is equal to or greater than the threshold th_d, the two blown-out highlights are not associated with each other. The threshold value th_d may be determined based on the area of the blown-out highlight area. For example, the threshold value th_d may be determined by considering the ratio of the area of the blown-out area to the area of the effective area. Specifically, the threshold value th_d is increased as the ratio (area of overexposed area/area of effective area) increases. Here, the total area of the blown-out highlights in the left image area is the area S_WL, the total area of the blown-out highlights in the right image area is the area S_WR, the area of the left image area is the area S_L, and the area of the right image area is the area S_WR. Assume that the area is area S_R. The threshold th_d may be determined by considering the ratio of the area S_WL or the area S_WR and the area S_L or the area S_R. The threshold value th_d may be determined by considering the ratio of the area S_WL+S_WR and the area S_L+S_R.

A specific example will be described using FIGS. 13, 14(A), and 14(B). FIG. 13 is a schematic diagram showing an example of a clustered brightness saturation map. The brightness saturation map in FIG. 13 shows overexposed areas. 14(A) and 14(B) are tables showing the center coordinates and area of the overexposed area shown in the brightness saturation map of FIG. 13. FIG. 14(A) is a table showing an example of the center coordinates and area of the overexposed area in the left image area. The center coordinates shown in FIG. 14(A) are coordinates with the upper left corner of the left image area as the origin. be. FIG. 14(B) is a table showing an example of the center coordinates and area of a blown-out highlight area in the right image area. The center coordinates shown in FIG. 14(B) are coordinates with the upper left corner of the right image area as the origin. be.

The left image area 1301A includes blown-out highlights areas 1302, 1303A, and 1304A, and the right image area 1301B includes blown-out highlights areas 1303B and 1304B. The control unit 501 repeats the process of searching for two highlight areas in which the difference between the center coordinates of the highlight area in the left image area and the center coordinates of the highlight area in the right image area is the smallest. First, the control unit 501 selects the blown-out highlight area 1303A and the blown-out highlight area 1303B. It is assumed that the difference (distance) between the center coordinates of the blown-out highlight area 1303A and the center coordinates of the blown-out highlight area 1303B is less than the threshold value th_d. Therefore, the control unit 501 associates the blown-out highlight area 1303A with the blown-out highlight area 1303B. Next, the control unit 501 performs a search excluding the blown-out highlight area 1303A and the blown-out highlight area 1303B, and selects the blown-out highlight area 1304A and the blown-out highlight area 1304B. It is assumed that the difference (distance) between the center coordinates of the blown-out highlight area 1304A and the center coordinates of the blown-out highlight area 1304B is greater than or equal to the threshold value th_d. Therefore, the control unit 501 does not associate the blown-out highlight area 1304A with the blown-out highlight area 1304B. The remaining overexposed area 1302 is also not associated with other areas.

In step S930, the control unit 501 calculates the difference (absolute value) between the areas of the two blown-out areas (over-exposed highlights pair) associated in step S929, and determines whether the difference is greater than or equal to the threshold th_a. do. The threshold value th_a may be determined based on the area of the blown-out highlight area. For example, the threshold value th_a may be determined by considering the ratio between the area of the blown-out area and the area of the effective area. Specifically, the threshold value th_a is increased as the ratio (area of blown-out area/area of effective area) increases. Here, the total area of the blown-out highlights in the left image area is the area S_WL, the total area of the blown-out highlights in the right image area is the area S_WR, the area of the left image area is the area S_L, and the area of the right image area is the area S_WR. Assume that the area is area S_R. The threshold th_a may be determined by considering the ratio of the area S_WL or the area S_WR and the area S_L or the area S_R. The threshold value th_a may be determined by considering the ratio of the area S_WL+S_WR and the area S_L+S_R. If it is determined that the difference in area between the two blown-out areas is greater than or equal to the threshold th_a, the process advances to step S931; otherwise, the process advances to step S932.

In step S931, the control unit 501 sets the whiteout pair as a warning target.

In step S932, the control unit 501 determines whether or not the determination in step S930 has been performed for all whiteout pairs. If it is determined that the determination in step S930 has been performed on all the whiteout pairs, the process advances to step S933, and if not, the process advances to step S930.

In step S933, the control unit 501 sets all overexposure areas that were not associated in step S929 as warning targets. Therefore, even if a blown-out highlight area is detected from only one of the right image area and the left image area, a warning (predetermined notification) will be issued.

In step S934, the control unit 501 determines whether a blown-out highlight area to be warned is set. If it is determined that a warning target overexposure area has been set, the process advances to step S935; otherwise, the process in FIG. 9B is ended and the process advances to step S922.

In step S935, the control unit 501 highlights the overexposed area to be warned. For example, a frame surrounding the blown-out highlight area to be warned is displayed.

In step S936, the control unit 501 displays on the display unit 506 a warning message (text) urging the user to change the shooting conditions to eliminate the difference between the overexposure in the left image area and the overexposure in the right image area. .

12(A) to 12(D) are schematic diagrams showing an example of a display (PC live view display) on an application screen displayed by the control unit 501 on the display unit 506. Screen 1200 is an application screen (remote live view screen). Screen 1200 includes a live view display area 1201, a guide display area 1202, a guide display area 1203, an operation area 1204, and an end button 1208.

Live view display area 1201 is an area where live view images are displayed. The live view display area 1201 consists of a left display area 1201A and a right display area 1201B. The guide display area 1202 contains a character string indicating which of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the binocular lens unit the image displayed in the left display area 1201A is. This is the area to display. The guide display area 1203 contains a character string indicating which of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the binocular lens unit the image displayed in the right display area 1201B is. This is the area to display. The operation area 1204 is an area for accepting operations related to PC live view, and radio buttons 1205 and 1206 and a check box 1207 are displayed in the operation area 1204. Radio button 1205 is a radio button selected when performing circumferential fisheye display, and radio button 1206 is a radio button selected when performing equirectangular cylinder display. When radio button 1205 is in a selected state, radio button 1206 is in a non-selected state, and when radio button 1205 is in a non-selected state, radio button 1206 is in a selected state. Checkbox 1207 is a checkbox that is checked when performing left-right swapping. When check box 1207 is operated, the positions of the right image area (right eye image area; right eye image area) and the left image area (left eye image area; left eye image area) in the live view image are swapped, and the guide The character strings displayed in display areas 1202 and 1203 are also replaced. The end button 1208 is a button for ending PC live view.

The graphic 1209A is a graphic (such as a zebra pattern) superimposed on the overexposed area of the image displayed in the display area 1201A. A graphic 1209B is a graphic (such as a zebra pattern) indicating a blown-out highlight area in the image displayed in the display area 1201B. In the first embodiment, when the difference between the area of the graphic 1209A and the area of the graphic 1209B (difference in overexposure) is equal to or greater than a threshold value, a frame 1210A surrounding the graphic 1209A and a frame 1210B surrounding the graphic 1209B are displayed. By doing so, the user can easily know that there is a difference between the blown-out highlights of the image displayed in the display area 1201A and the blown-out highlights of the image displayed in the display area 1201B. If the difference between the area of the graphic 1209A and the area of the graphic 1209B (difference in overexposure) is greater than or equal to the threshold, a warning message 1211 is further displayed prompting the user to change the shooting conditions to eliminate the difference. . In this way, the user can easily learn how to eliminate the difference in overexposure.

Note that the pattern, color, and other aspects of the graphics 1209A and 1209B (overexposure areas) differ depending on whether the difference between the area of the graphic 1209A and the area of the graphic 1209B (difference in overexposure) is equal to or greater than the threshold value. may be different. If the difference between the area of the graphic 1209A and the area of the graphic 1209B (difference in overexposure) is equal to or greater than a threshold value, the aspect of the graphics 1209A and 1209B may be made more emphasized than in other cases.

In FIG. 12(a), a radio button 1205 for performing circumferential fisheye display is selected. The check box 1207 for performing left and right swapping is not checked. Therefore, the live view image acquired from the camera is displayed as is in the live view display area 1201. Specifically, a right-eye video, which is a circular fisheye image, is displayed in the left display area 1201A, and a left-eye video, which is a circular fisheye image, is displayed in the right display area 1201B.

In FIG. 12(b), a radio button 1206 for displaying an equirectangular cylinder is selected. The check box 1207 for performing left and right swapping is not checked. Therefore, the right-eye image area and left-eye image area (both circumferential fisheye image areas) in the live view image acquired from the camera are converted into equirectangular image areas (left and right swapping is not performed). . Then, the live view image after equirectangular conversion is displayed in the live view display area 1201. Specifically, the left-hand display area 1201A displays a right-eye image that is an equirectangular image, and the right-hand display area 1201B displays a left-eye image that is an equirectangular image.

In FIG. 12C, a radio button 1205 for performing circumferential fisheye display is selected, and a check box 1207 for performing left-right swapping is checked. Therefore, the positions of the right-eye image area and the left-eye image area in the live view image acquired from the camera are swapped. Then, the live view image after the left and right swapping is displayed in the live view display area 1201. Specifically, a left-eye video, which is a circular fisheye image, is displayed in the left display area 1201A, and a right-eye video, which is a circular fisheye image, is displayed in the right display area 1201B.

In FIG. 12(d), a radio button 1206 for displaying an equirectangular cylinder is selected, and a check box 1207 for swapping left and right is checked. Therefore, the positions of the right-eye image area and the left-eye image area in the live view image acquired from the camera are swapped, and the right-eye image area and left-eye image area (both areas of the circular fisheye image) are equidistant. Converted to a cylindrical image. Then, the live view image after performing left-right swapping and equirectangular conversion is displayed in the live view display area 1201. Specifically, a left-eye image that is an equirectangular image is displayed in the left display area 1201A, and a right-eye image that is an equirectangular image is displayed in the right display area 1201B.

FIG. 15 is a schematic diagram showing equirectangular cylinder transformation according to the first embodiment. As shown in FIG. 15, in an image 1501 before equirectangular conversion, a right image area 1502 which is a circumferential fisheye image area is located on the left side, and a left image area which is a circumferential fisheye image area. 1505 is placed on the right side. Image 1508 is an image after equirectangular conversion, and includes image regions 1509 and 1510 that are equirectangular image regions. In the first embodiment, a map of equirectangular cylindrical transformation is generated so that the correspondences shown by arrows 1511 and 1512 are established. In the map of the first embodiment, each pixel in the image area 1509 (region of equirectangular image) placed on the left corresponds to each pixel in the left image area 1505 (region of circumferential fisheye image) placed on the right. can be mapped to. Each pixel in the image area 1510 (region of the equirectangular image) placed on the right side is associated with a position in the right image area 1502 (region of the circumferential fisheye image) placed on the left side. By using such a map, the left image area 1505 (circumferential fisheye image area) placed on the right side is converted into the image area 1509 (equirectangular image area) placed on the left side. That is, the left image area, which is an equirectangular image area, is set as the left image area 1509. Then, the right image area 1502 (circular fisheye image area) placed on the left side is converted into an image area 1510 (equirectangular image area) placed on the right side. That is, the right image area, which is an equirectangular image area, is set as the right image area 1510. In this way, not only the area of the circumferential fisheye image is converted into the area of the equirectangular image, but also the positions of the right and left image areas are swapped, and the positional relationship between the right and left image areas is changed into two. It can be adjusted to the positional relationship of the optical systems (right eye optical system 301R and left eye optical system 301L).

According to the first embodiment, when photographing using a twin-lens unit, it becomes easier to check the difference in overexposure between the left and right image areas on a display device of a PC. This allows the user to easily know that there is unintended overexposure and what countermeasures are required for appropriate photography.

<Embodiment 2>
In the first embodiment, an example has been described in which a warning display regarding overexposure during shooting using a twin lens unit is displayed on a display device of a PC connected to a camera. In the second embodiment, an example will be described in which a warning is displayed on a display device of a camera.

FIG. 16A is a flowchart illustrating an example of the operation of the camera 100 in the shooting mode (shooting mode processing). This operation is realized by the system control unit 50 loading the program recorded in the nonvolatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 is activated in the shooting mode or the mode of the camera 100 is switched to the shooting mode, the operation shown in FIG. 16A starts.

In step S1601, the system control unit 50 determines whether the camera 100 is compatible with a twin lens unit (for example, the lens unit 300). If it is determined that it is compatible with a binocular lens unit, the process advances to step S1602; otherwise, the process advances to step S1615. The process in step S1601 is the same as the process in step S801 in FIG.

In step S1602, the system control unit 50 determines whether a twin lens unit is attached to the camera 100. If it is determined that the binocular lens unit is attached, the process advances to step S1603; otherwise, the process advances to step S1615. The process in step S1602 is the same as the process in step S802 in FIG.

In step S1603, the system control unit 50 acquires the design value of the binocular lens unit that is attached (connected). In step S1604, the system control unit 50 acquires the individual value of the binocular lens unit that is attached (connected) to the binocular lens unit. The processing in steps S1603 and S1604 is the same as the processing in steps S803 and S804 in FIG.

In step S1605, the system control unit 50 acquires an image from the imaging unit 211.

In step S1606, the system control unit 50 displays the image acquired in step S1605 on the EVF 217 or the display unit 108 (live view display processing). In the second embodiment, the live view display process includes a blown-out highlight display process in which blown-out highlights are displayed in a distinguishable manner in each of the right image area and the left image area. Details of the live view display process including the blown-out highlight display process will be described later using FIG. 16B.

In step S1607, the system control unit 50 determines whether the user of the camera 100 has given a recording start instruction. If it is determined that a recording start instruction has been given, the process advances to step S1608; otherwise, the process advances to step S1614. Live view display on the EVF 217 or the display unit 108 continues until a recording start instruction or an end instruction in step S1614, which will be described later, is issued.

The recording start instruction is a full press of the shutter button 101, or the like. The recording start instruction may be an instruction to execute still image shooting, or may be an instruction to start video shooting. The captured image may be a JPEG still image, an MP4 moving image, a RAW still image, a RAW moving image, or the like. During video shooting, the recorded video may be displayed on the EVF 217 or the display unit 108 like a live view display.

In step S1608, the system control unit 50 acquires an image from the imaging unit 211.

In step S1609, the system control unit 50 acquires camera information. The camera information includes shooting information such as the shutter speed and aperture at the time of shooting, and posture information detected by the posture detection unit 222 at the time of shooting. When photographing a RAW image, data (parameters) necessary for development are also acquired.

In step S1610, the system control unit 50 stores the image data (image data) acquired in step S1608 in the recording medium 227 (storage medium) in a file format.

In step S1611, the system control unit 50 stores the information acquired in step S1609 (information on camera 100) in the image file stored in step S1610. As a result, the camera information obtained in step S1609 is added as metadata to the image data obtained in step S1608 in the image file.

In step S1612, the system control unit 50 stores the information acquired in steps S1603 and S1604 (information on the binocular lens unit) in the image file stored in step S1610. As a result, the information on the binocular lens unit obtained in steps S1603 and S1604 is added as metadata to the image data obtained in step S1608 in the image file.

In step S1613, the system control unit 50 determines whether the user of the camera 100 has given an instruction to end recording. If it is determined that a recording end instruction has been given, the process advances to step S1614; otherwise, the process advances to step S1608. By repeating steps S1608 to S1613, frames of a moving image can be recorded one after another in a moving image file, or still images can be continuously shot (continuous shooting).

In the case of still image shooting, the instruction to end recording is, for example, releasing the shutter button 101 fully. If the shutter button 101 is pressed fully and released within a predetermined time, one still image is taken, and if the shutter button 101 is pressed fully for a predetermined time or longer, continuous still image shooting is performed. be exposed. When photographing one still image, one full press of the shutter button 101 may serve as both a recording start instruction and a recording end instruction. In the case of video shooting, the instruction to end recording is a full press of the shutter button 101, or the like. For example, when the shutter button 101 is pressed fully, video shooting starts, and when the shutter button 101 is pressed fully again, the video shooting ends.

In step S1614, the system control unit 50 determines whether a termination instruction has been given by the user of the camera 100. If it is determined that a termination instruction has been given, the operation in FIG. 16A is terminated; otherwise, the process advances to step S1605. The termination instruction may be an instruction to turn off the power of the camera 100 or an instruction to switch the mode of the camera 100 from the shooting mode to another mode. In other words, the termination instruction is a press of the power switch 102, a press of the mode changeover switch 103, or the like.

If a single lens unit is attached to the camera 100, the process of step S1615 is performed. In step S1615, the system control unit 50 performs imaging (photography) using a single lens unit (single lens imaging process). The operation of the single-lens imaging process is similar to the conventional imaging (photographing) operation in a camera equipped with a single-lens unit, so a detailed explanation will be omitted. In the second embodiment, when recording an image file of an image captured by the single lens unit in the recording medium 227, the system control unit 50 stores information about the single lens unit (design values, individual values, etc.) as the single lens unit. It will be acquired from the eye lens unit and stored in an image file.

FIG. 16B is a flowchart illustrating an example of live view display processing performed by the system control unit 50 in step S1606 of FIG. 16A. As described above, the live view display process in step S1606 includes the blown-out highlight display process in which the blown-out highlights are displayed in a distinguishable manner in each of the right image area and the left image area.

In step S1621, the system control unit 50 displays the image acquired from the imaging unit 211 in step S1605 of FIG. 16A on the EVF 217 or the display unit 108 as a live view image.

In step S1622, the system control unit 50 determines the right image area (effective area) and left image area (effective area) in the live view image based on the information on the binocular lens unit acquired in steps S1603 and S1604 of FIG. 16A. to decide. The process in step S1622 is similar to the process in step S923 in FIG. 9B.

In step S1623, the system control unit 50 indicates whether each pixel in the right image area and the left image area determined in step S1622 is a pixel with a saturated luminance value (saturated pixel; blown-out highlight pixel). Create a brightness saturation map. The process in step S1623 is similar to the process in step S924 in FIG. 9B.

In step S1624, the system control unit 50 uses the brightness saturation map to determine whether a blown-out highlight pixel exists in at least one of the right image area and the left image area. If it is determined that a blown-out highlight pixel exists, the process advances to step S1625; otherwise, the process in FIG. 16B is ended and the process advances to step S1607. The process in step S1624 is similar to the process in step S925 in FIG. 9B.

In step S1625, the system control unit 50 uses the brightness saturation map created in step S1623 to superimpose and display a predetermined graphic such as a zebra pattern on the overexposed pixels in the right image area and left image area of the live view image. do. The process in step S1625 is similar to the process in step S926 in FIG. 9B.

In step S1626, the system control unit 50 uses the brightness saturation map created in step S1623 to perform clustering of blown-out highlights in the right image area and the left image area so that blown-out areas made up of blown-out highlights are set. I do. The process in step S1626 is similar to the process in step S927 in FIG. 9B.

In step S1627, the system control unit 50 calculates the center coordinates and area (for example, number of pixels) of the overexposed area set in step S1626. The process in step S1627 is similar to the process in step S928 in FIG. 9B.

In step S1628, the system control unit 50 associates the blown-out highlights area in the right image area with the blown-out highlights area in the left image area. The process in step S1628 is similar to the process in step S929 in FIG. 9B.

In step S1629, the system control unit 50 calculates the difference (absolute value) between the areas of the two blown-out areas (over-exposed highlights pair) associated in step S1628, and determines whether the difference is greater than or equal to the threshold th_a. judge. If it is determined that the difference in area between the two blown-out areas is greater than or equal to the threshold th_a, the process advances to step S1630; otherwise, the process advances to step S1631. The process in step S1629 is similar to the process in step S930 in FIG. 9B.

In step S1630, the system control unit 50 sets the whiteout pair as a warning target.
The process in step S1630 is similar to the process in step S931 in FIG. 9B.

In step S1631, the system control unit 50 determines whether or not the determination in step S1629 has been performed for all whiteout pairs. If it is determined that the determination in step S1629 has been performed on all blown-out highlights pairs, the process advances to step S1632; otherwise, the process advances to step S1629. The process in step S1631 is similar to the process in step S932 in FIG. 9B.

In step S1632, the system control unit 50 sets all overexposure areas that were not associated in step S1628 as warning targets. The process in step S1632 is similar to the process in step S933 in FIG. 9B.

In step S1633, the system control unit 50 determines whether a blown-out highlight area to be warned is set. If it is determined that a warning target overexposure area has been set, the process advances to step S1634; otherwise, the process in FIG. 9B is ended and the process advances to step S1607. The process in step S1633 is similar to the process in step S934 in FIG. 9B.

In step S1634, the system control unit 50 highlights the overexposed area to be warned. The process in step S1634 is similar to the process in step S935 in FIG. 9B.

In step S1635, the system control unit 50 sends a warning message (text) to the EVF 217 or display unit 108 urging the user to change the shooting conditions to eliminate the difference between the overexposure in the left image area and the overexposure in the right image area. to be displayed. The process in step S1635 is similar to the process in step S936 in FIG. 9B.

According to the second embodiment, when shooting using a twin lens unit, it becomes easier to check the difference in overexposure between the left and right image areas on the display device of the camera. This allows the user to easily know that there is unintended overexposure and what countermeasures are required for appropriate photography.

Note that the various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to It may also be possible to perform overall control. Similarly, the various controls described above as being performed by the control unit 501 may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing, allowing the device to It may also be possible to perform overall control.

Further, although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and the present invention includes various forms without departing from the gist of the present invention. Furthermore, each of the embodiments described above is merely one embodiment of the present invention, and it is also possible to combine the embodiments as appropriate. For example, although it has been explained that one image is obtained in which two image areas with parallax are arranged side by side, the number of image areas, that is, the number of optical systems, may be greater than two, or multiple image areas may be used. The arrangement is also not particularly limited.

Further, the present invention is not limited to cameras and PCs, but can be applied to any electronic device that can handle images having a plurality of image areas each corresponding to a plurality of optical systems. For example, the present invention is applicable to PDAs, mobile phone terminals, portable image viewers, printer devices, digital photo frames, music players, game consoles, electronic book readers, and the like. Furthermore, the present invention is applicable to video players, display devices (including projection devices), tablet terminals, smartphones, AI speakers, home appliances, in-vehicle devices, and the like. The present invention is also applicable to multi-lens smartphones that have a plurality of different types of optical systems, such as a standard lens, a wide-angle lens, and a zoom lens. Even in such a case, if the focal lengths (zoom magnifications) of the two optical systems used are matched (common) and photographed, a stereoscopic image can be obtained.

Furthermore, the present invention is applicable not only to the imaging device itself but also to a control device that communicates with the imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices that remotely control the imaging device include devices such as smartphones, tablet PCs, and desktop PCs. The imaging device can be controlled remotely by notifying the imaging device of commands to perform various operations and settings from the control device, based on operations performed on the control device and processing performed on the control device. be. Furthermore, a live view image taken by the imaging device may be received via wired or wireless communication and displayed on the control device side.

<Other embodiments>
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Digital camera (camera) 50: System control unit 500: Personal computer (PC) 501: Control unit

Claims (18)

acquisition means for acquiring an image in which a first image area imaged through the first optical system and a second image area imaged through the second optical system are lined up;
detection means for detecting a blown-out highlight area from each of the first image area and the second image area;
A control unit configured to perform control to perform a predetermined notification when a difference between an area of a blown-out highlight area in the first image area and an area of an blown-out highlight area in the second image area is larger than a threshold value. An electronic device characterized by having: The electronic device according to claim 1, wherein the second optical system has a predetermined parallax with respect to the first optical system. The control means selects two blown-out highlights areas in which the relative position of the blown-out highlights area with respect to the first image area and the relative position of the blown-out highlights area with respect to the second image area are approximately the same, and 3. The electronic device according to claim 1, wherein control is performed so that the predetermined notification is performed when a difference in area of blown-out areas is larger than the threshold value. The electronic device according to any one of claims 1 to 3, wherein the control means determines the threshold value based on the area of the blown-out highlight area. 2. The control means controls the image to be displayed, and controls the blown-out area detected by the detection means to be displayed in a discernible manner in the displayed image. The electronic device according to any one of items 4 to 4. The control means includes:
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is smaller than the threshold value, the detecting means detects a problem in the displayed image. controlling the detected overexposed area to be identifiably displayed in a first manner;
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is larger than the threshold value, the detecting means detects the blown-out highlight area in the displayed image. 6. The electronic device according to claim 5, wherein the detected overexposed area is controlled to be identifiably displayed in a second mode that is more emphasized than the first mode. The control means includes:
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is smaller than the threshold value, the detecting means detects a problem in the displayed image. Controlling the detected overexposure area to be identifiably displayed in the first pattern,
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is larger than the threshold value, the detecting means detects the blown-out highlight area in the displayed image. 7. The electronic device according to claim 6, wherein the electronic device is controlled so that the detected overexposed area is displayed in a second pattern so as to be distinguishable. The control means includes:
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is smaller than the threshold value, the detecting means detects a problem in the displayed image. controlling the detected overexposed area to be identifiably displayed in a first color;
If the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is larger than the threshold value, the detecting means detects the blown-out highlight area in the displayed image. 8. The electronic device according to claim 6, wherein the electronic device is controlled so that the detected overexposed area is identifiably displayed in a second color. The control means controls to display the image, and the difference between the area of the blown-out highlight area in the first image area and the area of the blown-out highlight area in the second image area is larger than the threshold value. The electronic device according to any one of claims 1 to 8, wherein the electronic device is controlled to display a frame surrounding the blown-out highlight area detected by the detection means in the displayed image. . The control means is configured to generate a text prompting a change in shooting conditions when a difference between an area of a blown-out highlight area in the first image area and an area of an blown-out highlight area in the second image area is larger than the threshold value. 10. The electronic device according to claim 1, wherein the electronic device is controlled to display. 2. The control means is further characterized in that the control means performs control to perform the predetermined notification even when a blown-out area is detected from only one of the first image area and the second image area. The electronic device according to any one of items 1 to 10. further comprising determining means for determining the first image area and the second image area,
The acquisition means acquires the image and information regarding the first optical system and the second optical system,
The electronic device according to claim 1, wherein the determining means determines the first image area and the second image area based on the information. The information includes information regarding the optical axis center of the first optical system in the image and information regarding the optical axis center of the second optical system in the image,
The determining means determines the first image area based on the optical axis center of the first optical system in the image, and determines the second image area based on the optical axis center of the second optical system in the image. 13. The electronic device according to claim 12, wherein the electronic device determines an image area of . The first optical system and the second optical system are each a fisheye lens,
The electronic device according to any one of claims 1 to 13, wherein the first image area and the second image area are each an area of a circumferential fisheye image. The electronic device according to claim 1, wherein the first image area and the second image area are each an equirectangular image area. acquiring an image in which a first image area imaged through the first optical system and a second image area imaged through the second optical system are lined up;
detecting a blown-out highlight area from each of the first image area and the second image area;
a step of controlling to perform a predetermined notification when a difference between an area of a blown-out highlight area in the first image area and an area of an blown-out highlight area in the second image area is larger than a threshold value; A method for controlling an electronic device, characterized by: A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 15. A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 15.

Images