JP2022184712A - Information processing apparatus, imaging apparatus, control method, program and storage medium - Google Patents

Abstract

To provide a technique capable of displaying two images captured with a camera attached with a lens unit including two optical systems such that positions of the two images correspond to the two optical systems.SOLUTION: An information processing apparatus includes: obtaining means for obtaining one image including a first image region corresponding to a first optical image input via a first optical system and a second image region corresponding to a second optical image input via a second optical system, the second optical system having predetermined parallax with respect to the first optical system, and correction information on the first optical system and the second optical system; correction means for executing correction processing to correct the position of a pixel included in the first image region and a pixel included in the second image region in the image, based on the correction information; and generation means for executing processing to convert the corrected first and the second image regions to generate a processed image.SELECTED DRAWING: Figure 8

Description

The present invention relates to an information processing device, an imaging device, a control method, a program, and a storage medium.

A technique is known in which two images with parallax are captured by two cameras and the captured two images are displayed in a stereoscopically visible manner. Japanese Unexamined Patent Application Publication No. 2002-200001 discloses a camera that is equipped with a lens unit having two optical systems and that can capture two images with parallax at once.

JP 2013-141052 A

However, if an image captured by a lens unit having two optical systems (one image including two images with parallax) is displayed like a conventional image, the positional relationship between the two optical systems and one The positional relationship of two images in one image is reversed.

An object of the present invention is to provide a technique that enables display such that the positions of two images captured by mounting a lens unit having two optical systems correspond to those of the two optical systems.

An information processing apparatus of the present invention includes a first image area corresponding to a first optical image input via a first optical system, and a second image area having a predetermined parallax with respect to the first optical system. Acquisition for acquiring one image including a second image area corresponding to a second optical image input via an optical system, and correction information regarding the first optical system and the second optical system means for executing correction processing for correcting positions of pixels included in the first image region and pixels included in the second image region in the image based on the correction information; and generating means for generating a processed image by executing a process of transforming the first image area and the second image area.

According to the present invention, it is possible to display two images captured by attaching a lens unit having two optical systems so that the positions of the two images correspond to those of the two optical systems.

1 is a schematic diagram showing the overall configuration of a system; FIG. 1 is an external view of a camera; FIG. 3 is a block diagram showing the configuration of a camera; FIG. It is a schematic diagram which shows the structure of a lens unit. It is a block diagram which shows the structure of PC. 4 is a flow chart showing the operation of the camera; It is a flow chart which shows operation of PC. It is a schematic diagram of left-right interchange. 4 is a schematic diagram showing the structure of an image file; FIG. 4 is a schematic diagram showing lens information and camera information; FIG. 4 is a schematic diagram of a display screen; FIG. FIG. 4 is a schematic diagram of an equirectangular transformation; 4 is a flow chart showing the operation of the camera; It is a flow chart which shows operation of PC. 4 is a schematic diagram of a display screen; FIG. It is a figure which shows the difference of a lens on either side. FIG. 10 is a diagram showing deviation caused by lens error; FIG. 4 is a diagram showing correction of image position on spatial coordinates;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings.

<First embodiment>
A first embodiment of the present invention will be described. FIGS. 1A and 1B are schematic diagrams showing an example of the overall configuration of the system according to this embodiment. The system according to this embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500 . A lens unit 300 is attached (connected) to the camera 100 . Although the details of the lens unit 300 will be described later, by attaching the lens unit 300, the camera 100 can simultaneously capture two images (still image or moving image) having a predetermined parallax. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100 . FIG. 1A shows a configuration in which a camera 100 and a PC 500 are communicably connected to each other wirelessly or by wire. FIG. 1B shows a configuration for inputting an image or the like taken by the camera 100 to the PC 500 via an external storage device on a file basis. The external storage device may or may not be connected to both camera 100 and PC 500 . For example, an external storage device may be connected to the camera 100, and files of images captured by the camera 100 may be stored in the external storage device. After that, the external storage device may be detached from the camera 100 and connected to the PC 500 so that the PC 500 can retrieve the files stored in the external storage device.

2A and 2B are external views showing an example of the external appearance of the camera 100. FIG. 2A is a perspective view of the camera 100 viewed from the front side, and FIG. 2B is a perspective view of the camera 100 viewed from the rear side.

The camera 100 has a shutter button 101, a power switch 102, a mode changeover switch 103, a main electronic dial 104, a sub electronic dial 105, a moving image button 106, and a viewfinder display section 107 on the top surface. A shutter button 101 is an operation member for issuing a photographing preparation instruction or a photographing instruction. A power switch 102 is an operation member for switching the power of the camera 100 between on and off. A mode switch 103 is an operation member for switching between various modes. A main electronic dial 104 is a rotary operation member for changing set values such as shutter speed and aperture. The sub-electronic dial 105 is a rotary operation member for moving a selection frame (cursor) and for image feed. A moving image button 106 is an operation member for instructing start and stop of moving image shooting (recording). A viewfinder display unit 107 displays various setting values such as shutter speed and aperture.

The camera 100 has a display unit 108, a touch panel 109, direction keys 110, a SET button 111, an AE lock button 112, an enlargement button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar on the back. 119. The display unit 108 displays images and various information. The touch panel 109 is an operation member that detects a touch operation on the display surface (touch operation surface) of the display unit 108 . The directional key 110 is an operation unit composed of keys (four-directional keys) that can be pressed up, down, left, and right. Processing can be performed according to the pressed position of the direction key 110 . A SET button 111 is an operation member that is mainly pressed when deciding a selection item. The AE lock button 112 is an operation member that is pressed when fixing the exposure state in the shooting standby state. The enlargement button 113 is an operation member for switching between on and off of the enlargement mode in the live view display (LV display) of the shooting mode. When the enlargement mode is on, operating the main electronic dial 104 enlarges or reduces the live view image (LV image). Also, the enlargement button 113 is used when enlarging a reproduced image or increasing the enlargement ratio in the reproduction mode. A playback button 114 is an operation member for switching between a shooting mode and a playback mode. Pressing the playback button 114 in the shooting mode shifts to the playback mode, and the latest image among images recorded in the recording medium 227 (to be described later) can be displayed on the display unit 108 .

A menu button 115 is an operation member that is pressed to display a menu screen on which various settings can be made on the display unit 108 . The user can intuitively perform various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. FIG. The eyepiece part 116 is a part for looking into an eyepiece finder (looking-in type finder) 117 with an eyepiece. The user can view an image displayed on an EVF 217 (Electronic View Finder) inside the camera 100 through the eyepiece 116 . The eyepiece detection unit 118 is a sensor that detects whether or not the user is eyeing the eyepiece unit 116 (eyepiece viewfinder 117).

The touch bar 119 is a linear touch operation member (line touch sensor) capable of receiving touch operations. The touch bar 119 can be touch-operated with the thumb of the right hand when the grip portion 120 is gripped with the right hand so that the shutter button 101 can be pressed with the index finger of the right hand (with the little finger, ring finger, and middle finger of the right hand). possible). That is, the touch bar 119 can be operated in a state (photographing posture) in which the user looks into the eyepiece finder 117, looks into the eyepiece unit 116, and is ready to press the shutter button 101 at any time. The touch bar 119 can perform a tap operation (an operation of touching and releasing the touch bar 119 without moving the touch position within a predetermined period), a left/right slide operation (an operation of moving the touch position while touching after touching), and the like. Acceptable. The touch bar 119 is an operation member different from the touch panel 109 and does not have a display function. The touch bar 119 functions, for example, as a multi-function bar (M-Fn bar) to which various functions can be assigned.

The camera 100 also has a grip portion 120, a thumb rest portion 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like. The grip portion 120 is a holding portion formed in a shape that allows the user to easily hold the camera 100 with his or her right hand. The shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated with the index finger of the right hand while holding the camera 100 by gripping the grip portion 120 with the little finger, the ring finger and the middle finger of the right hand. Also, in a similar state, the sub-electronic dial 105 and the touch bar 119 are arranged at positions that can be operated with the thumb of the right hand. The thumb rest portion 121 (thumb standby position) is a grip portion provided on the rear side of the camera 100 where the thumb of the right hand holding the grip portion 120 can be easily placed without operating any operation member. The thumb rest portion 121 is made of a rubber member or the like for enhancing holding power (grip feeling). The terminal cover 122 protects a connector such as a connection cable that connects the camera 100 to an external device (external device). The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227, which will be described later. The communication terminal 124 is a terminal for communicating with a detachable lens unit (a lens unit 200, a lens unit 300, etc., which will be described later) with respect to the camera 100. FIG.

FIG. 3 is a block diagram showing an example of the configuration of the camera 100. As shown in FIG. The same components as in FIG. 2 are denoted by the same reference numerals as in FIG. 2, and the description of the components will be omitted as appropriate. In FIG. 3, the lens unit 200 is attached to the camera 100 .

First, the lens unit 200 will be explained. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100 . The lens unit 200 is a monocular lens and an example of a normal lens. The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

The diaphragm 201 is configured such that the aperture diameter can be adjusted. The lens 202 is composed of a plurality of lenses. A diaphragm drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the diaphragm 201 . An AF driving circuit 204 drives the lens 202 to focus. A lens system control circuit 205 controls an aperture drive circuit 203, an AF drive circuit 204, and the like based on instructions from a system control unit 50, which will be described later. A lens system control circuit 205 controls an aperture 201 through an aperture drive circuit 203 and focuses by changing the position of a lens 202 through an AF drive circuit 204 . Lens system control circuit 205 can communicate with camera 100 . Specifically, communication is performed via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100 . A communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100 side.

Next, the camera 100 will be explained. The camera 100 has a shutter 210, an imaging unit 211, an A/D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D/A converter 216, an EVF 217, a display unit 108, and a system control unit 50. .

The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging section 211 based on the instruction from the system control section 50 . The imaging unit 211 is an imaging device (image sensor) configured by a CCD, a CMOS device, or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50 . The A/D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A/D converter 212 or the data from the memory control unit 213 . Further, the image processing unit 214 performs predetermined arithmetic processing using captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic result. Through this processing, TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (flash pre-emission) processing, and the like are performed. Further, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs TTL AWB (auto white balance) processing based on the obtained arithmetic results.

Image data from the A/D converter 212 is written into the memory 215 via the image processing section 214 and the memory control section 213 . Alternatively, the image data from the A/D converter 212 is written to the memory 215 via the memory control section 213 without passing through the image processing section 214 . A memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A/D converter 212 and image data to be displayed on the display unit 108 and the EVF 217 . The memory 215 has a storage capacity sufficient to store a predetermined number of still images, moving images and audio for a predetermined time. The memory 215 also serves as an image display memory (video memory).

The D/A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies the analog signal to the display unit 108 and the EVF 217 . Therefore, the image data for display written in the memory 215 is displayed on the display unit 108 or the EVF 217 via the D/A converter 216 . The display unit 108 and the EVF 217 perform display according to analog signals from the D/A converter 216 . The display unit 108 and the EVF 217 are, for example, displays such as LCD and organic EL. The digital signal A/D converted by the A/D converter 212 and accumulated in the memory 215 is converted into an analog signal by the D/A converter 216, and transferred to the display unit 108 or the EVF 217 for display. A view is displayed.

System controller 50 is a controller comprising at least one processor and/or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. A system control unit 50 controls the entire camera 100 . The system control unit 50 executes the programs recorded in the non-volatile memory 219 to realize each process of the flowcharts described later. The system control unit 50 also performs display control by controlling the memory 215, the D/A converter 216, the display unit 108, the EVF 217, and the like.

The camera 100 also has a system memory 218 , a nonvolatile memory 219 , a system timer 220 , a communication section 221 , an orientation detection section 222 and an eye proximity detection section 118 .

A RAM, for example, is used as the system memory 218 . In the system memory 218, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are expanded. The nonvolatile memory 219 is an electrically erasable/recordable memory, and an EEPROM, for example, is used as the nonvolatile memory 219 . Constants for operation of the system control unit 50, programs, and the like are recorded in the nonvolatile memory 219. FIG. The program here is a program for executing a flowchart to be described later. The system timer 220 is a timer that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from an external device connected wirelessly or by a wired cable. The communication unit 221 can be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with an external device using Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded in the recording medium 227, and can receive images and other various information from external devices. The orientation detection unit 222 detects the orientation of the camera 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 222, whether the image captured by the imaging unit 211 is an image captured with the camera 100 held horizontally or an image captured with the camera 100 held vertically is determined. It is identifiable. The system control unit 50 adds orientation information according to the orientation detected by the orientation detection unit 222 to the image file of the image captured by the imaging unit 211, and rotates the image according to the detected orientation. It is possible. For example, an acceleration sensor, a gyro sensor, or the like can be used for the posture detection unit 222 . The posture detection unit 222 can also be used to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary, etc.).

The eyepiece detection unit 118 can detect the approach of any object to the eyepiece unit 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used for the eye proximity detection unit 118 . When an object approaches, the infrared rays projected from the light projecting unit of the eyepiece detection unit 118 are reflected by the object and received by the light receiving unit of the infrared proximity sensor. The distance from the eyepiece 116 to the object can be determined by the amount of received infrared rays. In this manner, the eye proximity detection unit 118 performs eye proximity detection for detecting the close distance of an object to the eyepiece unit 116 . The eye contact detection unit 118 is an eye contact detection sensor that detects approach (eye contact) and separation (eye separation) of the eye (object) from the eyepiece unit 116 . When an object approaching within a predetermined distance from the eyepiece unit 116 is detected from the non-eyepiece state (non-approach state), it is detected that the eyepiece has been brought into the eyepiece state. On the other hand, when the object whose approach has been detected moves away from the eye contact state (approach state) by a predetermined distance or more, it is detected that the eye has been taken off. The threshold for detecting eye proximity and the threshold for detecting eye detachment may be different, for example, by providing hysteresis. Also, it is assumed that after the eye contact is detected, the eye contact state is maintained until the eye detachment is detected. It is assumed that the eyes are in the non-eye contact state after the eye separation is detected until the eye contact is detected. The system control unit 50 switches display (display state)/non-display (non-display state) of the display unit 108 and the EVF 217 according to the state detected by the eye proximity detection unit 118 . Specifically, at least in the shooting standby state and when the switching setting of the display destination is automatic switching, the display destination is set to the display unit 108 and the display is turned on while the EVF 217 is not displayed. In addition, while the eye is being focused, the display destination is the EVF 217, the display is turned on, and the display unit 108 is turned off. Note that the eye contact detection unit 118 is not limited to the infrared proximity sensor, and other sensors may be used as the eye contact detection unit 118 as long as they can detect a state that can be regarded as eye contact.

The camera 100 also includes an out-of-finder display unit 107, an out-of-finder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium I/F 226, an operation unit 228, and the like.

The viewfinder display unit 107 is driven by the viewfinder display drive circuit 223 and displays various setting values of the camera 100 such as shutter speed and aperture. The power control unit 224 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching blocks to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining amount of the battery. Also, the power supply control unit 224 controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies necessary voltage to each unit including the recording medium 227 for a necessary period. The power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, an AC adapter, or the like. A recording medium I/F 226 is an interface with a recording medium 227 such as a memory card or hard disk. A recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be detachable from the camera 100 or may be built in the camera 100 .

The operation unit 228 is an input unit that receives operations (user operations) from the user, and is used to input various instructions to the system control unit 50 . The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, another operation unit 229, and the like. The other operation unit 229 includes the main electronic dial 104, the sub electronic dial 105, the movie button 106, the direction keys 110, the SET button 111, the AE lock button 112, the zoom button 113, the playback button 114, the menu button 115, the touch bar 119, and the like. is included.

The shutter button 101 has a first shutter switch 230 and a second shutter switch 231 . The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (imaging preparation instruction), and outputs a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed, that is, when the shutter button 101 is fully pressed (imaging instruction), and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of photographing processes from reading signals from the imaging unit 211 to generating an image file including the photographed image and writing it to the recording medium 227. do.

A mode switch 103 switches the operation mode of the system control unit 50 to one of a still image shooting mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes, custom modes, and the like, which are shooting settings for each shooting scene. The user can directly switch to one of the above-described shooting modes by using the mode switch 103 . Alternatively, the user can use the operation unit 228 to selectively switch to any one of the plurality of displayed modes after switching once to the shooting mode list screen using the mode switching switch 103 . Similarly, the movie shooting mode may also include multiple modes.

The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured integrally. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that the light transmittance does not interfere with the display of the display unit 108 . By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI is provided as if the user can directly operate the screen displayed on the display unit 108. (graphical user interface) can be configured. The touch panel 109 can use any one of various methods such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. Depending on the method, there is a method of detecting that there is a touch when there is a contact with the touch panel 109, and a method of detecting that there is a touch when a finger or a pen approaches the touch panel 109. Either method can be used. may

The system control unit 50 can detect the following operations or states on the touch panel 109 .
The touch panel 109 is newly touched by a finger or pen that has not touched the touch panel 109, that is, the start of touch (hereinafter referred to as Touch-Down).
A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).
- The touch panel 109 is moved while being touched by a finger or pen (hereinafter referred to as touch-move).
- The finger or pen touching the touch panel 109 is released from the touch panel 109, ie, the end of the touch (hereinafter referred to as Touch-Up).
A state in which nothing is touched on the touch panel 109 (hereinafter referred to as Touch-Off).

When touchdown is detected, touchon is also detected at the same time. After touchdown, touchon continues to be detected unless touchup is detected. Touch-on is detected at the same time when touch-move is detected. Even if touch-on is detected, touch-move is not detected if the touch position does not move. After it is detected that all the fingers and pens that have touched have touched up, the touch is turned off.

These operations/states and the coordinates of the position where the finger or pen is touching on the touch panel 109 are notified to the system control unit 50 through the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. As for the touch move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component/horizontal component on the touch panel 109 based on the change in the position coordinates. When it is detected that the touch-move has been performed by a predetermined distance or more, it is determined that the slide operation has been performed. An operation of touching the touch panel 109 with a finger and quickly moving it by a certain distance and then releasing it is called a flick. A flick is, in other words, an operation of quickly tracing the touch panel 109 as if flicking it with a finger. It is detected that a touch-move has been performed for a predetermined distance or more at a predetermined speed or more, and if a touch-up is detected as it is, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Further, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called pinch-in, and a touch operation in which the touch positions are moved away from each other is called pinch-out. Pinch-out and pinch-in are collectively called pinch operation (or simply pinch).

FIG. 4 is a schematic diagram showing an example of the configuration of the lens unit 300. As shown in FIG. 4 shows a state in which the lens unit 300 is attached to the camera 100. FIG. In the camera 100 shown in FIG. 4, the same constituent elements as those explained in FIG. 3 are denoted by the same reference numerals as in FIG. 3, and the explanation of those constituent elements is omitted as appropriate.

The lens unit 300 is a type of interchangeable lens that can be attached to and detached from the camera 100 . The lens unit 300 is a twin lens capable of imaging a right image and a left image with parallax. In this embodiment, the lens unit 300 has two optical systems, and each of the two optical systems can image a wide viewing angle range of approximately 180 degrees. Specifically, in each of the two optical systems of the lens unit 300, the horizontal direction (horizontal angle, azimuth angle, yaw angle) of 180 degrees and the vertical direction (vertical angle, elevation/depression angle, pitch angle) of 180 degrees. minutes). That is, each of the two optical systems can image the range of the anterior hemisphere.

The lens unit 300 has a right-eye optical system 301R having a plurality of lenses and reflecting mirrors, a left-eye optical system 301L having a plurality of lenses and reflecting mirrors, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R has a lens 302R arranged on the subject side, and the left-eye optical system 301L has a lens 302L arranged on the subject side. The lenses 302R and 302L face the same direction, and their optical axes are substantially parallel.

The lens unit 300 is a twin lens (VR180 lens) for obtaining a VR180 image, which is one of VR (Virtual Reality) image formats capable of binocular stereoscopic viewing. In this embodiment, the lens unit 300 has a fisheye lens capable of capturing a range of approximately 180 degrees in each of the right-eye optical system 301R and the left-eye optical system 301L. The range that can be captured by the lenses of the right-eye optical system 301R and the left-eye optical system 301L may be about 160 degrees, which is narrower than the range of 180 degrees. The lens unit 300 combines a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L. It can be imaged onto one or two imagers of a mounted camera.

The lens unit 300 is attached to the camera 100 via the lens mount section 304 and the camera mount section 305 of the camera 100 . By doing so, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300. .

In this embodiment, a right image formed via the right-eye optical system 301R and a left image formed via the left-eye optical system 301L are simultaneously (as a set) formed on the imaging unit 211 of the camera 100. be done. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one imaging device. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, two images with parallax can be simultaneously acquired (as a set) from two locations (optical systems) of the right-eye optical system 301R and the left-eye optical system 301L. . By dividing the acquired image into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image within a range of approximately 180 degrees. That is, the user can stereoscopically view the image of the VR 180 .

Here, the VR image is an image that can be displayed in VR, which will be described later. VR images include an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera), or a panoramic image with a wider image range (effective image range) than can be displayed on the display unit at once. etc. are included. Moreover, VR images are not limited to still images, and include moving images and live images (images obtained from a camera almost in real time). A VR image has a maximum video range (effective video range) of 360 degrees in the horizontal direction and 360 degrees in the vertical direction. In addition, even if the VR image is less than 360 degrees in the horizontal direction and less than 360 degrees in the vertical direction, the angle of view that is wider than the angle of view that can be captured by a normal camera, or the display range that can be displayed at once on the display unit Images with wider image coverage are also included. An image captured by the camera 100 using the lens unit 300 described above is a type of VR image. A VR image can be displayed in VR by, for example, setting the display mode of a display device (a display device capable of displaying a VR image) to “VR view”. By displaying a VR image having an angle of view of 360 degrees in VR and changing the posture of the display device in the left-right direction (horizontal rotation direction), the user can enjoy seamless omnidirectional video in the left-right direction. can.

VR display (VR view) is a display method (display mode) that allows the display range to be changed by displaying an image within a viewing range corresponding to the posture of the display device. VR display includes “single-lens VR display (single-lens VR view)” in which one image is displayed by performing deformation (distortion correction) by mapping a VR image onto a virtual sphere. In addition, for VR display, there is a "twin-eye VR display (twin-eye VR view)" in which a VR image for the left eye and a VR image for the right eye are transformed by mapping onto a virtual sphere and displayed side by side in the left and right regions. There is. By performing “twin-eye VR display” using a VR image for the left eye and a VR image for the right eye that have parallax with each other, it is possible to stereoscopically view these VR images. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image within a visual field corresponding to the orientation of the user's face is displayed. For example, of the VR image, at a certain point in time, the image of the viewing range centered at 0 degrees in the horizontal direction (a specific direction, such as north) and 90 degrees in the vertical direction (90 degrees from the zenith, that is, horizontal) is displayed. Suppose there is When the posture of the display device is reversed from this state (for example, the display surface is changed from facing south to facing north), the same VR image can be displayed by 180 degrees in the horizontal direction (opposite direction, for example, south), and by 180 degrees in the vertical direction. The display range is changed to the image of the visual field range centered at 90 degrees. That is, when the user wears the HMD and turns his face from north to south (that is, turns his back), the image displayed on the HMD is changed from the north image to the south image. Note that the VR image captured using the lens unit 300 of the present embodiment is an image (180° image) captured in a range of approximately 180 degrees forward, and there is no video in a range of approximately 180 degrees backward. When such an image is displayed in VR and the posture of the display device is changed to the side where no image exists, a blank area is displayed.

By VR-displaying a VR image in this way, the user can visually feel as if he/she is in the VR image (inside the VR space) (feeling of immersion). Note that the display method of the VR image is not limited to the method of changing the posture of the display device. For example, the display range may be moved (scrolled) according to a user operation via a touch panel, direction buttons, or the like. In addition, when displaying VR (when the display mode is "VR view"), in addition to changing the display range due to changes in posture, the display range can be changed according to touch moves on the touch panel, drag operations with a mouse, etc., pressing direction buttons, etc. You can change it. A smartphone attached to VR goggles (head mount adapter) is a kind of HMD.

FIG. 5 is a block diagram showing an example of the configuration of the PC 500. As shown in FIG. A control unit 501 is, for example, a Central Processing Unit (CPU), and controls the entire PC 500 . A Read Only Memory (ROM) 502 non-temporarily stores programs and parameters. A random access memory (RAM) 503 temporarily stores programs and data supplied from an external device or the like. The recording medium 504 is a hard disk or flash memory fixedly installed in the PC 500, or an optical disc, magnetic card, optical card, IC card, memory card, or the like that is removable from the PC 500. FIG. A file of an image captured by the camera 100 is read from the recording medium 504 . An operation unit 505 receives user operations on the PC 500 . The operation member used when the user performs an operation may be a button or a touch panel provided on the PC 500, or may be a keyboard, mouse, or the like that is detachable from the PC 500. FIG. A display unit 506 displays data held by the PC 500, data supplied from the outside, and the like. The display unit 506 may be a part of the PC 500 or may be a display device separate from the PC 500 . A communication unit 507 communicates with an external device such as the camera 100 . A system bus 508 communicatively connects the components of PC 500 .

Here, the characteristics of an image captured with the lens unit 300 (twin lens) attached will be described. In the case of the lens unit 200 (ordinary single-lens lens), an image that is inverted vertically and horizontally with respect to the actual appearance (an image that is rotated 180 degrees) is formed on the imaging unit 211 . Therefore, the entire formed image is rotated by 180 degrees to acquire (capture) an image that matches the actual appearance. On the other hand, in the case of the lens unit 300 (dual lens), the right image and the left image are formed on the imaging unit 211 after being rotated 180 degrees with respect to the actual appearance. The arrangement of the right image and the left image is not particularly limited, but in this embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211 . Then, as in the case of the lens unit 200 (ordinary single-lens lens), when the entire formed image (image including the image area of the right image and the image area of the left image) is rotated 180 degrees, the right image and Although each left image can be matched to the actual appearance, the positions of the right and left images are exchanged. In other words, the left and right positional relationship is reversed, and an image is captured in which the right image is arranged on the left side and the left image is arranged on the right side. In this embodiment, it is possible to display the positions of the right image and the left image in correspondence with two optical systems (the right eye optical system 301R and the left eye optical system 301L).

FIG. 6 is a flow chart showing an example of the operation of the camera 100 in shooting mode (shooting mode processing). This operation is realized by the system control unit 50 loading the program recorded in the nonvolatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 is activated in shooting mode or the mode of the camera 100 is switched to the shooting mode, the operation of FIG. 6 starts.

In step S601, the system control unit 50 determines whether the camera 100 is compatible with a twin lens (for example, the lens unit 300). For example, the system control unit 50 determines whether or not the version of the firmware of the system control unit 50 is compatible with the binocular lens. If it is determined that the binocular lens is supported, the process proceeds to step S602; otherwise, the process proceeds to step S615. In the present embodiment, in the case of a twin-lens lens, information on the twin-lens lens (lens information; information on two optical systems of the twin-lens lens) is acquired for post-processing, unlike in the case of a normal single-lens lens. Must be recorded. Therefore, the process of step S601 is required.

In step S602, the system control unit 50 determines whether the camera 100 is attached with a twin lens. If it is determined that the binocular lens is attached, the process proceeds to step S603; otherwise, the process proceeds to step S615. It should be noted that even when the twin lens is attached from the state in which the twin lens is not attached, the process proceeds to step S603. When the twin lens is removed from the state in which the twin lens is attached and the single lens is attached, the process proceeds to step S615.

In step S603, the system control unit 50 acquires the design value of the attached (connected) twin lens. The design value is a design parameter, and is used for left-right switching and equirectangular conversion, which will be described later. For example, the image circle position, image circle diameter, angle of view, and distortion correction coefficient shown in FIG. 10(b) are acquired. The image of the subject captured by the lens appears circular on the surface of the image sensor. The image circle is this circular image of the subject (image circle).

In step S604, the system control unit 50 acquires the individual value of the attached (connected) binocular lens. The individual value is a parameter specific to the lens unit, such as manufacturing error. For example, the image circle misalignment, optical axis tilt, and image magnification misalignment shown in FIG. 10(b) are acquired. By using individual values, image processing can be performed with higher accuracy than when only design values are used.

In step S<b>605 , the system control unit 50 acquires an image from the imaging unit 211 .

In step S606, the system control unit 50 displays the image acquired in step S605 on the EVF 217 or the display unit 108 (live view display).

In step S607, the system control unit 50 determines whether or not the user of the camera 100 has issued a recording start instruction. If it is determined that the recording start instruction has been issued, the process proceeds to step S608; otherwise, the process proceeds to step S614. Live view display on the EVF 217 or the display unit 108 continues until a recording start instruction or an end instruction in step S614, which will be described later, is given.

The recording start instruction is, for example, a full press of the shutter button 101 . The recording start instruction may be an instruction to execute still image shooting or an instruction to start moving image shooting. The captured image may be any of JPEG still images, MP4 moving images, RAW still images, RAW moving images, and the like. During movie shooting, the recorded movie may be displayed on the EVF 217 or the display unit 108 like a live view display.

In step S<b>608 , the system control unit 50 acquires an image from the imaging unit 211 .

In step S609, the system control unit 50 acquires shooting information such as shutter speed and aperture at the time of shooting, posture information detected by the posture detection unit 222 at the time of shooting, and the like. When capturing a RAW image, data (parameters) necessary for development are also acquired.

In step S610, the system control unit 50 stores the image data (image data) acquired in step S608 in a file format in the recording medium 227 (storage medium).

In step S611, the system control unit 50 stores the information (information on the camera 100) acquired in step S609 in the image file stored in step S610. As a result, the information acquired in step S609 is added as metadata to the image data acquired in step S608 in the image file.

In step S612, the system control unit 50 stores the information (information on the binocular lens) acquired in steps S603 and S604 in the image file stored in step S610. As a result, the information acquired in steps S603 and S604 is added as metadata to the image data acquired in step S608 in the image file.

In step S613, the system control unit 50 determines whether or not the user of the camera 100 has issued a recording end instruction. If it is determined that the recording end instruction has been issued, the process proceeds to step S614; otherwise, the process proceeds to step S608. By repeating steps S608 to S613, it is possible to record moving image frames one after another in a moving image file, or perform continuous shooting (continuous shooting) of still images.

In the case of still image shooting, the recording end instruction is, for example, releasing the shutter button 101 from full depression. When the shutter button 101 is fully pressed and released within a predetermined time, one still image is captured. will be When shooting one still image, one full press of the shutter button 101 may serve as both a recording start instruction and a recording end instruction. In the case of moving image shooting, the recording end instruction is, for example, a full press of the shutter button 101 . For example, moving image shooting starts when the shutter button 101 is fully pressed, and moving image shooting ends when the shutter button 101 is fully pressed again.

In step S614, the system control unit 50 determines whether or not the user of the camera 100 has issued an end instruction. If it is determined that the end instruction has been issued, the operation of FIG. 6 is terminated, otherwise the process proceeds to step S605. The end instruction is an instruction to turn off the power of the camera 100, an instruction to switch the mode of the camera 100 from the photographing mode to another mode, or the like. In other words, the end instruction is the pressing of the power switch 102, the pressing of the mode changeover switch 103, or the like.

If the camera 100 is equipped with a monocular lens, the process of step S615 is performed. In step S615, the system control unit 50 performs imaging (shooting) using the single-lens lens (single-lens imaging processing). The operation of the single-lens imaging process is the same as the conventional imaging (photographing) operation in a camera equipped with a single-lens lens, and detailed description thereof will be omitted. In this embodiment, when recording an image file of an image captured by the single-lens lens in the recording medium 227, the system control unit 50 stores information (design value, individual value, etc.) of the attached single-lens lens. Assume that the image is obtained from the single-lens lens and stored in an image file.

FIG. 7 is a flowchart showing an example of the operation (display control) of the PC 500 displaying an image based on the image file. This operation is realized by the control unit 501 expanding a program (application program) recorded in the ROM 502 into the RAM 503 and executing the program. For example, when the user of the PC 500 operates the operation unit 505 (operation member) to select an image file captured by a camera from files stored in the recording medium 504, the operations in FIG. 7 start. At this time, the recording medium 227 removed from the camera 100 or the like may be used as the recording medium 504 . In this embodiment, the right image and the left image are circular fisheye images (equidistant projection images). The operation of correcting the positions of the right image and the left image as they are in the circular fisheye image using the information of the two-lens lens as the correction information will be described below.

In step S<b>701 , the control unit 501 reads the image file selected by the user of the PC 500 from the recording medium 504 . Here, not only captured image data but also headers and metadata added to the image data are read. For example, the image file shown in FIG. 9 is read.

In step S702, based on the image file read in step S701, the control unit 501 determines whether or not the image file is a file of an image shot with a twin lens (using a twin lens). If it is determined that the file is an image shot with a twin-lens lens, the process advances to step S703; otherwise, the process advances to step S719. In step S702, for example, based on whether or not the image file contains information on the twin lens (design value, individual value, etc.), it is determined whether or not the file is an image shot with the twin lens. . Determine whether the file is an image shot with a twin lens based on whether the image file contains a flag indicating that the twin lens has been used, not detailed information about the twin lens. You may

In step S703, the control unit 501 acquires (extracts) the information (design value and individual value) of the binocular lens from the image file read in step S701.

In step S704, the control unit 501 acquires the center coordinates of each of the right and left images in the captured image from the design values acquired in step S703. The center coordinates of each of the right and left images correspond to the respective optical axis center coordinates of the two optical systems of the twin lens (the left eye optical system 301L and the right eye optical system 301R). For example, central coordinates 804 and 808 shown in FIGS. 8(a) and 8(b) are acquired. Center coordinates 804 are the center coordinates of the right image in the horizontal direction, and center coordinates 808 are the center coordinates of the left image in the horizontal direction.

In step S705, the control unit 501 acquires the captured image from the image file read in step S701. If the obtained image is displayed as it is, an image in which the positions of the right image and the left image are exchanged is displayed.

In step S706, the control unit 501 determines whether or not to perform circular fisheye display. If it is determined that the circular fisheye display is to be performed, the process proceeds to step S707; otherwise (if the equirectangular display is to be performed), the process proceeds to step S710. In step S706, for example, it is determined whether or not the circular fisheye display is to be performed depending on whether the radio button 1102 in FIGS. 11A and 11B is selected or not selected. The radio button 1102 is in the selected state in FIG. 11(a), and the radio button 1102 is in the non-selected state in FIG. 11(b). If the radio button 1102 is selected, it is determined that circular fisheye display is to be performed, and the process advances to step S707. If radio button 1102 is not selected, the process proceeds to step S710. By default (for example, before displaying the screens shown in FIGS. 11A and 11B), it is determined that the circular fisheye display is to be performed, and the process proceeds to step S707 so that the circular fisheye display is started. . The operation after proceeding to step S710 will be described later.

In step S707, the control unit 501 determines the positions of the right and left images in the captured image based on the center coordinates (centers of the optical axes of the left-eye optical system 301L and the right-eye optical system 301R, respectively) acquired in step S704. are exchanged to generate a processed image (left-right exchange). For example, the control unit 501 identifies the area of the right image in the captured image based on the central coordinates of the right image, and identifies the area of the left image in the captured image based on the central coordinates of the left image. . Then, the control unit 501 replaces the positions of the two identified regions. In this embodiment, the right image and the left image are arranged side by side in the photographed image, and the left-right positional relationship between the right image and the left image is reversed by left-right switching. In order to specify the areas of the right image and the left image with higher accuracy, the respective diameters (diameters or radii) of the right image and the left image may be obtained from the information of the twin lens.

In addition, the method of left-right exchange is not restricted to the said method. For example, the deviation amounts 805, 806, 809, and 810 in FIG. 8A are acquired from the information of the twin lens, and the acquired deviation amounts are maintained when the positions of the right image and the left image are exchanged. A right image and a left image may be arranged, and the remaining area may be filled with black or the like. A shift amount 805 is the distance from the left edge of the captured image to the left edge of the right image, and a shift amount 806 is the distance from the center of the captured image to the right edge of the right image. When switching left and right, the shift amount 805 is the distance from the left edge of the captured image to the left edge of the left image, and the shift amount 806 is the distance from the center of the captured image to the right edge of the left image. . Similarly, the displacement 809 is the distance from the right edge of the captured image to the right edge of the left image, and the displacement 810 is the distance from the center of the captured image to the left edge of the left image. When switching left and right, the shift amount 809 is the distance from the right edge of the captured image to the right edge of the right image, and the shift amount 810 is the distance from the center of the captured image to the left edge of the right image. .

In step S<b>708 , the control unit 501 displays the processed image generated in step S<b>707 on the display unit 506 . For example, the processed image 1101 in FIG. 11A is displayed on the display unit 506. FIG.

In step S709, the control unit 501 determines whether or not the user of the PC 500 has issued an end instruction. If it is determined that the end instruction has been issued, the operation of FIG. 7 is terminated, otherwise the process proceeds to step S706. By doing so, the display (display of the captured image) on the display unit 506 can be switched between a plurality of displays such as circular fisheye display and equirectangular display. The end instruction is issued using the operation unit 505 (operation member). For example, the end instruction is pressing the end button 1106 in FIGS. 11A and 11B.

When an image file shot with a single-lens lens is read in step S701, processing in step S719 is performed. In step S719, the control unit 501 displays on the display unit 506 an image based on the image file shot with the single-lens lens. Since the processing in step S719 is the same as the conventional processing for displaying an image shot with a single-lens lens, detailed description thereof will be omitted.

8(a) and 8(b) are schematic diagrams of left and right switching. FIG. 8(a) shows a conventional left-right switching that does not use the information of the binocular lens. FIG. 8(b) shows left-right switching according to the present embodiment, using the information of the twin lens as the information for correction.

As shown in FIGS. 8A and 8B, in an image 801 before left/right switching, a right image 803, which is a circular fisheye image, is arranged on the left side, and a left image 807, which is a circular fisheye image, is arranged on the left side. is placed on the right.

In FIG. 8A, the image 801 is divided into a left half image and a right half image at the center coordinates 802 of the image 801, and the left half image and the right half image are exchanged. In other words, move the left half image to the right of the right half image. An image 811 is an image after such left-right switching.

In FIG. 8A, the amount of deviation 806 is smaller than the amount of deviation 805 . That is, in the image 801 , the right image 803 is closer to the center of the image 801 than the center of the left half of the image 801 . Similarly, deviation 810 is smaller than deviation 809 . That is, in the image 801 , the left image 807 is closer to the center of the image 801 than the center of the right half of the image 801 . Therefore, in the image 811, the center coordinates 813 of the left image 807 in the left-right direction are shifted from the center coordinates 804 by a distance 814, and the center coordinates 816 of the right image 803 in the left-right direction are shifted from the center coordinates 808 by a distance 817.

In the present embodiment, by using the lens information, in the image 837 (FIG. 8B) after left-right switching, the center coordinates of the left image in the left-right direction are matched with the center coordinates 804, and the center of the right image in the left-right direction is matched. The coordinates can be matched to center coordinates 808 .

FIG. 9 is a schematic diagram showing an example of the structure of an image file shot with a twin lens. The image file in FIG. 9 has a header 901 , camera information section 902 , lens information section 903 , other information section 904 and image data section 905 . A header 901 records information such as the type of captured image. In the camera information section 902, information about the camera used for shooting is recorded as metadata. For example, shooting information such as shutter speed and aperture at the time of shooting, information on the posture of the camera at the time of shooting, etc. are recorded. In the lens information section 903, information on the binocular lens used for shooting is recorded as metadata. For example, the design values and individual values of the binocular lens are recorded. Other information is recorded in the other information section 904 as metadata. For example, in the case of moving images, information that changes for each frame is recorded. In the case of a RAW image, data required for development are recorded. Image data is recorded in the image data portion 905 . In the case of moving images, not only image data but also audio data are recorded. Although an example of recording camera information, twin lens information, and other information in the same image file has been shown, they may be recorded in separate files associated with the image file.

FIG. 10A is a schematic diagram showing an example of lens information acquired from a twin lens. lens information
1. Lens design value 2. Lens individual value 3 . lens flag 4 . 4. lens focal length; Including lens temperature, etc.

A lens design value is a design value for performing aberration correction. In the process of manufacturing a twin-lens lens, errors such as lens eccentricity and tilt occur in each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R). If left-right switching, equirectangular conversion, or the like is performed without considering the error, the quality of the binocular VR display deteriorates, and good stereoscopic viewing becomes difficult. The lens individual value is the measurement result of the error detected in the manufacturing process of the binocular lens. Details of the lens design value and the individual lens value will be described later with reference to FIG.

Fig. 16 shows the reason why the quality of binocular VR display deteriorates and good stereoscopic vision becomes difficult when left-right switching or equirectangular conversion is performed without considering the individual value (manufacturing error) of the lens. Description will be made with reference to (a) and 16(b). FIG. 16(a) is a diagram showing an ideal optical system. In FIG. 16(a), the optical axes of the left lens and the right lens are parallel and face the same direction. Therefore, the closer the subject is to the image sensor, the greater the parallax, and good stereoscopic vision can be achieved. FIG. 16B is a diagram showing an example of an optical system with errors from design values. In FIG. 16(b), the optical axes of the left and right lenses are not parallel, but face different directions. Therefore, parallax occurs at infinity, and good stereoscopic vision cannot be realized. In addition to this, due to the individual values of the left and right lenses, parallax may occur between the upper and lower sides, or the image magnification may differ, which may make it impossible to realize good stereoscopic vision. Therefore, in this embodiment, by correcting the photographed image using the error of the left and right lenses actually used for photographing, the image can be approximated to the image photographed with the ideal optical system. To achieve good stereoscopic vision.

The lens flag is a flag indicating that it is a twin lens, and can be used to determine whether or not the twin lens is used. The lens focal length is the distance from the "principal point", which is the center of the lens, to the imaging device (imaging position). The lens focal length may or may not be a parameter common to the two optical systems of the twin lens (left eye optical system 301L and right eye optical system 301R). Detailed (high-precision) lens focal lengths are required in order to perform high-quality binocular VR display by performing left-right switching, equirectangular conversion, and the like with high precision. The lens temperature is the temperature of the binocular lens, and is used for grasping the environmental temperature and the like at the time of photographing.

FIG. 10B is a schematic diagram showing details of lens design values and individual lens values. In the present embodiment, the lens design value and the lens individual value are used as correction information when left-right switching or equirectangular conversion is performed.

The lens design value is
1. Image circle position 2 . Image circle diameter 3 . angle of view 4. Includes distortion correction coefficients, etc.

The image circle position is the optical axis center coordinates of the optical system in the captured image, and is prepared for each of the two optical systems of the twin lens (the left eye optical system 301L and the right eye optical system 301R). That is, the image circle position is the center coordinates of the image circle (circumferential fisheye image) formed on the imaging device, and is prepared for each of the right and left images. The origin of the coordinates is, for example, the center of the imaging element (the center of the captured image). The image circle position includes horizontal and vertical coordinates. Various information regarding the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position (center, upper left corner, etc.) in the image to the center of the optical axis can be used.

The image circle diameter is the diameter of the image circle (circumferential fisheye image) formed on the imaging device. The angle of view is the angle of view of an image circle (circumferential fisheye image) formed on the imaging device. The distortion correction coefficient is the ratio of the designed image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for an image height for which no distortion correction coefficient is set, the distortion correction coefficient may be calculated by interpolation using a plurality of distortion correction coefficients. A polynomial that approximates the relationship between the image height and the distortion correction coefficient may be set. The image circle diameter, angle of view, and distortion correction coefficient may or may not be common parameters for the two optical systems of the twin lens (left eye optical system 301L and right eye optical system 301R). good.

When displaying the circular fisheye image, the PC 500 may display a magic window on the circular fisheye image. A magic window is a display item that indicates an area that is cut out (first) for VR display. For example, the magic window is displayed based on image circle position, image circle diameter, and angle of view. By doing so, the display quality of the magic window can be improved. In order to properly display the magic window, the PC 500 appropriately edits and uses the image circle position, image circle diameter, and angle of view. For example, the PC 500 multiplies the image circle position and image circle diameter by coefficients.

The lens individual value is
5. Image circle position shift 6 . 7. Optical axis tilt; Including image magnification deviation. These pieces of information are prepared by measuring each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R) of the twin lens.

The image circle positional deviation is the deviation of the center coordinates of the image circle (circumferential fisheye image) formed on the imaging element from the designed value. For example, image circle misalignment includes horizontal misalignment and vertical misalignment. With the coordinate of the design value (two-dimensional coordinate including horizontal and vertical coordinates) as the origin, the horizontal coordinate indicates the horizontal shift, and the vertical coordinate indicates the vertical shift. . FIG. 17A shows an example of image circle positional deviation. A region 1701 indicates the right or left half region of the sensor size screen (imaging surface). Image circle 1702 is the actual right or left lens image circle. An image circle 1703 is an image circle of an ideal optical system (design values). A deviation occurs between the position of the actual image circle 1702 and the ideal image circle 1703 .

The optical axis tilt is the deviation of the direction of the optical axis on the object side from the design value. For example, the optical axis tilt includes horizontal shift and vertical shift. The deviation in each direction is indicated in degrees. FIG. 17B shows an example of horizontal deviation due to optical tilt. A line 1712 indicates the orientation (tilt) of the horizontal optical axis of the actual right or left lens. A line 1713 indicates the orientation of the horizontal optical axis of the ideal optical system (design value). There is a deviation between the actual horizontal optical axis orientation 1712 and the ideal horizontal optical axis orientation 1713 . FIG. 17(c) shows an example of vertical deviation due to optical tilt. Line 1722 indicates the orientation of the horizontal optical axis of the actual right or left lens. A line 1723 indicates the orientation of the horizontal optical axis of the ideal optical system (design value). There is a deviation between the actual horizontal optical axis orientation 1722 and the ideal horizontal optical axis orientation 1723 .

The image magnification deviation is the deviation of the size of the image circle (circumferential fisheye image) formed on the imaging element from the design value. This deviation is indicated, for example, as a ratio to the design value. FIG. 17(d) shows an example of image magnification deviation. An image circle 1732 is an image circle formed on the image sensor through the actual right or left lens. An image circle 1733 is an image circle formed on the imaging device via an ideal optical system (designed values). There is a deviation between the actual image circle 1732 and the image circle 1733 of the ideal optical system.

Information included in the lens information is not limited to the information described above. For example, the lens information may include the respective boundary positions of the right image and the left image in the photographed image (the position of the edge of the circular fisheye image; the positions indicated by the shift amounts 805, 806, 809, 810, etc.). good. The lens information may include midpoint coordinates between the right and left images in the captured image. In most cases, the midpoint coordinates match the center coordinates of the captured image. The lens information may include information indicating the area of the magic window (eg, the coordinates of the upper left corner of the magic window, the width of the magic window, and the height of the magic window). The lens information may include a correction value (for example, a correction value obtained by calibration of the binocular lens) for increasing the accuracy of left-right switching, equirectangular conversion, and the like.

FIG. 10C is a schematic diagram showing an example of camera information generated within the camera. For example, camera information is used to perform high-definition VR display. camera information is
1. Camera recording area information 2 . In-camera accelerometer information 3 . Includes right exposure compensation information and so on.

The camera recording area information is information on the effective image area. The effective image area that can be displayed differs depending on the camera sensor and recording mode. PC 500 uses the camera recording area information for more accurate display. The in-camera accelerometer information is posture information obtained using an acceleration sensor (level gauge) in the camera, and represents the posture of the camera in the roll direction, the pitch direction, and the like. The PC 500 uses the in-camera accelerometer information to grasp the orientation of the camera at the time of shooting. The PC 500 performs electronic image stabilization and horizontal correction (zenith correction that brings the vertical direction of the display closer to the vertical direction of the real space) based on the grasped orientation. The right exposure correction information is an exposure setting value that brings the exposure of the right image closer to the exposure of the left image. The PC 500 uses the right exposure correction information in order to perform binocular VR display without discomfort (reduced discomfort).

FIG. 11A is a schematic diagram showing an example of the display (display in step S708) after switching the left and right of the application screen displayed on the display unit 506 by the control unit 501. FIG. Screen 1100 is an application screen. A screen 1100 in FIG. 11A includes a processed image 1101, radio buttons 1102 and 1103, a check box 1104, and a save button 1105.
, including an exit button 1106 . A processed image 1101 is the image after the left and right interchange, that is, the processed image generated in step S707. A radio button 1102 is a radio button selected when performing circular fisheye display, and a radio button 1103 is a radio button selected when performing equirectangular display. When the radio button 1102 is in the selected state, the radio button 1103 is in the non-selected state, and when the radio button 1102 is in the non-selected state, the radio button 1103 is in the selected state. A check box 1104 and a save button 1105 are display items used when performing equirectangular display. When the radio button 1102 is selected, the check box 1104 and save button 1105 are grayed out and disabled. Uses of the check box 1104 and save button 1105 will be described later. An end button 1106 is a button for ending the application on the screen 1100 .

Next, the case of performing equirectangular display will be described. If it is determined in step S706 in FIG. 7 that the circular fisheye display is not selected and the equirectangular display is selected, the process proceeds to step S710 to display the image in the equirectangular display. For example, when the radio button 1103 is selected among the radio buttons 1102 and 1103 shown in FIGS. 11A and 11B, the circular fisheye display is not selected and the equirectangular display is selected. Suppose that it is determined that

In step S710, the control unit 501 of the PC 500 determines whether or not to perform adjustment (individual value adjustment) using the individual value of the binocular lens. If it is determined that individual value adjustment is to be performed, the process proceeds to step S716; otherwise, the process proceeds to step S711. In step S710, for example, it is determined whether individual value adjustment is to be performed based on whether the check box 1104 in FIG. 11B is checked. If the check box 1104 is checked, it is determined that individual value adjustment is to be performed, and the process proceeds to step S716. If the check box 1104 is not checked, the process proceeds to step S711. The operation after proceeding to step S716 will be described later.

In step S711, the control unit 501 generates a map for equirectangular transformation based on the center coordinates (optical axis centers of the left-eye optical system 301L and right-eye optical system 301R) acquired in step S704. Equirectangular transformation is a transformation process in which the latitude (horizontal line) and longitude (vertical line) intersect at right angles by assuming the circular fisheye image as a sphere, like the equirectangular projection of a map. . A circular circular fisheye image is transformed into a rectangular equirectangular image by the equirectangular transformation. The map indicates where each pixel after transformation corresponds to in the image before transformation. In this embodiment, a map for equirectangular transformation is generated so that not only can a circular fisheye image be transformed into an equirectangular image, but also the positions of the right and left images can be corrected. In step S711, the areas of the right image and the left image in the captured image are identified by the same method as in step S707. A map is then generated based on the two identified regions.

In step S712, the control unit 501 performs equirectangular transformation using the map generated in step S711 to generate a processed image. Note that the right-left permutation is a part of the equirectangular transformation, but the left-right permutation may be performed separately from the equirectangular transformation.

In step S713, the control unit 501 displays on the display unit 506 the processed image generated in step S712 or step S718 described later in another embodiment. For example, the processed image 1111 in FIG. 11B is displayed on the display unit 506. FIG.

In step S714, the control unit 501 determines whether to save the processed image (image after equirectangular transformation) displayed in step S713. If it is determined that the processed image is to be saved, the process proceeds to step S715; otherwise, the process proceeds to step S709. In step S714, for example, based on whether the save button 1105 in FIG.
Determine whether to save the processed image. If the save button 1105 has been pressed, it is determined that the processed image is to be saved, and the process advances to step S715. If the save button 1105 has not been pressed, the process advances to step S709.

In step S<b>715 , the control unit 501 saves the image file of the processed image (image after equirectangular transformation) displayed in step S<b>713 in the recording medium 504 .

FIG. 12 is a schematic diagram showing the equirectangular transformation of this embodiment. As shown in FIG. 12, in an image 1201 before equirectangular transformation, a right image 1202 that is a circular fisheye image is arranged on the left side, and a left image 1205 that is a circular fisheye image is arranged on the right side. ing. An image 1208 is an image after equirectangular transformation and includes equirectangular images 1209 and 1210 . In this embodiment, a map of equirectangular transformation is generated so that the correspondences shown by arrows 1211 and 1212 are made. In the map of this embodiment, each pixel of the equirectangular image 1209 arranged on the left is associated with each position in the left image 1205 arranged on the right, and the equirectangular image 1210 arranged on the right is associated with each pixel. Each pixel is associated with a position in the right image 1202 located on the left. Using such a map, the right-aligned left image 1205 is transformed into a left-aligned equirectangular image 1209 and the left-aligned right image 1202 is transformed into a right-aligned equirectangular image 1209 . Transformed into a cylindrical image 1210 . In other words, not only is the circular fisheye image converted into an equirectangular image, but also the positions of the right and left images are exchanged. Thereby, the positional relationship between the right image and the left image can be matched with the positional relationship between the two optical systems (the right-eye optical system 301R and the left-eye optical system 301L).

FIG. 11B is a schematic diagram showing an example of display (display in step S713) after the equirectangular conversion on the application screen displayed on the display unit 506 by the control unit 501. FIG. In FIG. 11A, a processed image 1101, which is an image after left/right switching, is displayed on the screen 1100 (application screen). In FIG. 11B, a processed image 1111 (processed image generated in step S712 or step S718), which is an image after equirectangular transformation, is displayed on the screen 1100. FIG. Also, in FIG. 11B, the check box 1104 and save button 1105 are not grayed out, and the check box 1104 and save button 1105 are made operable. A check box 1104 is checked when individual value adjustment is performed. A save button 1105 is pressed to save the image file of the processed image 1111 . Since the check box 1104 is not checked, the processed image generated in step S712 is displayed as the processed image 1111. FIG.

Next, an example of performing individual value adjustment when performing equirectangular display will be described. That is, the operation from step S716 in FIG. 7 will be described. Although detailed description is omitted, the individual value of the two-lens lens may also be used when performing circular fisheye display.

In step S716, the control unit 501 adjusts the design value acquired in step S703 based on the individual value acquired in step S703. For example, the image circle position (center coordinates of each of the right and left images in the photographed image) is adjusted based on the image circle position shift in FIG. 10(b). If the individual value is the difference from the design value, the individual value is added to the design value. If the individual value is the same absolute value as the design value, the design value is replaced with the individual value.

In step S717, the control unit 501 generates a map for equirectangular transformation based on the adjusted center coordinates in step S716 (optical axis centers of the left-eye optical system 301L and the right-eye optical system 301R). . FIG. 18 shows an example of adjusting the image position of the spatial coordinates for each pixel based on the center coordinates after adjustment (correction processing for correcting the position of the pixel). For the left image, the image position is adjusted pixel by pixel so that the spatial coordinates are adjusted from image position 1801 to image position 1802 . For the right image, the image position is adjusted pixel by pixel such that the spatial coordinates are adjusted from image position 1803 to image position 1804 . The map generation method is the same as in step S711. By using the center coordinates after adjustment, more accurate equirectangular transformation becomes possible.

In step S718, the control unit 501 performs equirectangular transformation using the map generated in step S717 to generate a processed image. The method of generating the processed image is the same as in step S712. The operations from step S713 are as described above. As described above, by correcting the design value of the lens information based on the individual value of the twin-lens lens, it is possible to more accurately convert the circular fisheye image into the equirectangular image.

According to the information processing apparatus of the present embodiment, when displaying an image based on an image file captured by a twin-lens lens, the centers of the left and right images obtained through the optical systems of the twin-lens lens are corrected. After that, the left and right images can be rearranged appropriately. As a result, it is possible to display a binocular image that does not give a sense of discomfort.

<Second embodiment>
A second embodiment of the present invention will be described. In this embodiment, an example will be described in which the camera 100 and the PC 500 are communicably connected to each other, the live view image captured by the camera 100 is transmitted to the PC 500, and the PC 500 displays the live view image on the display unit 506. .

FIG. 13 is a flow chart showing an example of the operation of the camera 100. FIG. This operation is realized by the system control unit 50 loading the program recorded in the nonvolatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 is activated, the operations of FIG. 13 start. The operation of FIG. 13 is for a function (PC live view) of displaying a live view image captured by the camera on the display unit of the PC. The operation of FIG. 13 is performed when the camera 100 is in a shooting standby state. When a recording start instruction is input from the PC 500 during PC live view operation, still image shooting or moving image shooting is performed. At this time, the PC live view may continue. The control of photographing using the twin lens executes the processing from step S608 to step S613 in FIG. 6, and the description thereof is omitted.

In step S1301, the system control unit 50 determines whether the camera 100 is compatible with a twin lens (for example, the lens unit 300). If it is determined that the binocular lens is supported, the process advances to step S1302; otherwise, the process advances to step S1311. The processing of step S1301 is the same as the processing of step S601 in FIG.

In step S1302, the system control unit 50 determines whether the camera 100 is attached with a twin lens. If it is determined that the double lens is attached, the process advances to step S1303; otherwise, the process advances to step S1311. The processing of step S1302 is the same as the processing of step S602 in FIG.

In step S1303, the system control unit 50 acquires the design value of the attached (connected) twin lens. In step S1304, the system control unit 50 acquires the individual value of the attached (connected) binocular lens. The processing of steps S1303 and S1304 is the same as the processing of steps S603 and S604.

In step S<b>1305 , the camera 100 is connected to the PC 500 and the system control unit 50 detects that the camera 100 is connected to the PC 500 . In step S1306,
The system control unit 50 receives a PC live view start request from the PC 500 . In step S<b>1307 , the system control unit 50 receives a live view image request from the PC 500 . The live-view image request includes information (resolution information) designating the resolution of the live-view image, as will be described later. The system control unit 50 executes the process of step S1309 so as to transmit the live view image with the specified resolution to the PC500.

In step S1308, the system control unit 50 converts the information (lens information of the twin lens) acquired in steps S1303 and S1304 so as to match the coordinate system of the live view image to be transmitted. Since the captured image (image recorded in the image file) and the live view image have different resolutions, the information obtained in steps S1303 and S1304 cannot be used as is for image processing of the live view image. . Therefore, in this embodiment, the lens information is converted into information that matches the coordinate system of the live view image.

In step S<b>1309 , the system control unit 50 transmits the lens information converted in step S<b>1308 and the live view image to the PC 500 . The system control unit 50 converts the resolution of the live view image based on the resolution information acquired in step S1307, and transmits it to the PC500. Although the system control unit 50 of the camera 100 converts the lens information in this embodiment, the control unit 501 of the PC 500 may convert the lens information. At that time, the lens information before conversion and the parameters necessary for converting the lens information are transmitted to the PC 500 .

In step S1310, the system control unit 50 determines whether or not to end the PC live view. For example, when the connection between the camera 100 and the PC 500 is released, or when the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined to end the PC live view. If it is determined that the PC live view should be ended, the operation in FIG. 13 is ended; otherwise, the process advances to step S1307.

If the camera 100 is equipped with a monocular lens, the process of step S1311 is performed. In step S<b>1311 , the system control unit 50 transmits to the PC 500 the live view image captured by the single-lens lens. The processing in step S1311 is the same as conventional processing for transmitting a live view image captured by a single-lens lens to an external device, so detailed description thereof will be omitted. In this embodiment, when transmitting a live view image captured by a single-lens lens to the PC 500, the system control unit 50 transmits information (design values, individual values, etc.) of the attached single-lens lens to the single lens. It is assumed that neither acquisition from the lens nor transmission to the PC 500 is performed.

FIG. 14 is a flow chart showing an example of the operation of the PC 500. As shown in FIG. This operation is realized by the control unit 501 expanding a program (application program) recorded in the ROM 502 into the RAM 503 and executing the program. For example, when the user instructs the PC 500 to start a specific application, the operation of FIG. 14 starts. The operation in FIG. 14 is for a function (PC live view) of displaying a live view image captured by a camera on the display unit of the PC.

In step S<b>1401 , a camera (for example, camera 100 ) is connected to PC 500 , and control unit 501 detects that the camera is connected to PC 500 .

In step S1402, the control unit 501 determines whether or not the camera connected in step S1401 is compatible with a double-lens lens (for example, the lens unit 300). For example, the control unit 501 acquires model information of the camera from the connected camera, and determines whether or not the camera is compatible with a twin-lens lens based on the acquired model information. If the camera is determined to be compatible with a twin-lens lens, the process proceeds to step S1403; otherwise, step S1403.
Proceed to 1421. A camera compatible with a twin-lens lens is, for example, a camera to which a twin-lens lens can be attached.

In step S1403, the control unit 501 determines whether or not the firmware of the camera connected in step S1401 is compatible with the twin lens. For example, the control unit 501 acquires firmware version information of the camera from the connected camera, and based on the acquired information, the firmware version of the connected camera is a version compatible with the twin lens. Determine whether or not If it is determined that the binocular lens can be used, the process advances to step S1404; otherwise, the process advances to step S1421.

Even if a camera compatible with the dual lens is connected to the PC 500, the connected camera may not be compatible with the dual lens because the firmware version of the connected camera is old. Therefore, the process of step S1403 is required. Also, various cameras can be connected to the PC 500, and a camera that is not compatible with a twin lens may be connected regardless of the version of the firmware. Therefore, the process of step S1402 is required before the process of step S1403.

In step S1404, the control unit 501 determines whether or not the camera connected in step S1401 is equipped with a twin lens. If it is determined that two lenses are attached, the process advances to step S1405; otherwise, the process advances to step S1421.

In step S1405, the control unit 501 transmits a PC live view start request to the camera connected in step S1401.

In step S1406, the control unit 501 determines whether or not to perform circular fisheye display. If it is determined that circular fisheye display is to be performed, the process advances to step S1407; if not (if equirectangular display is to be performed), the process advances to step S1414. In step S1406, for example, it is determined whether or not the circular fisheye display is to be performed depending on whether the radio buttons 1505 in FIGS. 15A to 15D are selected or not selected. 15A and 15C, the radio button 1505 is selected, and in FIGS. 15B and 15D, the radio button 1505 is not selected. If the radio button 1505 is selected, it is determined that circular fisheye display is to be performed, and the process advances to step S1407. If the radio button 1505 is not selected, the process advances to step S1414.

In step S1407, the control unit 501 transmits a live view image request to the camera connected in step S1401. In this embodiment, it is assumed that the live-view image request in step S1407 is a request for a normal-resolution live-view image. Assume that the normal resolution is, for example, 4K resolution.

In step S1408, the control unit 501 receives, from the camera connected in step S1401, the live view image captured by the camera and the lens information of the twin lens attached to the camera. The resolution of the live view image received in step S1408 is normal resolution. The lens information received in step S1408 is information converted to match the received live view image (for example, the lens information converted in step S1308 of FIG. 13).

In step S1409, the control unit 501 determines whether or not left-right switching is to be performed. If it is determined that left/right switching is to be performed, the process proceeds to step S1410; otherwise, the process proceeds to step S1412. In step S1409, for example, based on whether or not the check box 1507 in FIGS. If the check box 1507 is checked, it is determined that left/right switching is to be performed, and the process advances to step S1410. If the check box 1507 is not checked, the process advances to step S1412.

In step S1410, based on the lens information acquired in step S1408, the control unit 501 replaces the positions of the right and left images in the live view image acquired in step S1408 to generate a processed live view image (horizontal interchange). ). The method for left/right switching is the same as in step S707 in FIG. Based on the center coordinates (centers of the optical axes of the left-eye optical system 301L and right-eye optical system 301R) included in the lens information received together with the live-view image, the control unit 501 shifts the right and left images in the live-view image. The positions are exchanged to generate a processed live view image.

In step S<b>1411 , the control unit 501 displays the processed live view image generated in step S<b>1410 on the display unit 506 .

In step S<b>1412 , the control unit 501 displays the live view image acquired in step S<b>1408 on the display unit 506 .

In step S1413, the control unit 501 determines whether or not to end the PC live view. For example, when the connection between the camera 100 and the PC 500 is released, or when the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined to end the PC live view. An instruction to end the PC live view is, for example, pressing the end button 1508 in FIGS. If it is determined to end the PC live view, the operation in FIG. 14 is ended; otherwise, the process advances to step S1406.

As described above, when performing equirectangular display, the process proceeds from step S1406 to step S1414. In step S1414, the control unit 501 transmits a live view image request to the camera connected in step S1401. In this embodiment, it is assumed that the live-view image request in step S1414 is a request for a low-resolution (lower than normal resolution) live-view image. When performing equirectangular display, equirectangular conversion (conversion from a circular fisheye image to an equirectangular image) is required. , and the delay due to the equirectangular transformation increases. In this embodiment, a low-resolution live view image is requested in order to speed up the equirectangular transformation (reduce the time required for the equirectangular transformation). If the delay due to the equirectangular conversion is within the allowable range, the normal resolution live view image may be requested even when the equirectangular display is performed.

In step S1415, the control unit 501 receives, from the camera connected in step S1401, the live view image captured by the camera and the lens information of the twin lens attached to the camera. The resolution of the live view image received in step S1415 is low resolution. The lens information received in step S1415 is information converted to match the received live view image (for example, the lens information converted in step S1308 of FIG. 13).

In step S1416, the control unit 501 determines whether or not left-right switching is to be performed. If it is determined that left/right switching is to be performed, the process proceeds to step S1417; otherwise, the process proceeds to step S1419. In step S1416, for example, based on whether or not the check box 1507 in FIGS. If the check box 1507 is checked, it is determined that left/right switching is to be performed, and the process advances to step S1417. If check box 1507 is not checked, the process advances to step S1419.

In step S1417, based on the lens information acquired in step S1415, the control unit 501 switches the positions of the right image and the left image in the live view image acquired in step S1415, and converts the right image and the left image into equirectangular cylinders. Convert to image. That is, the control unit 501 performs left-right switching and equirectangular transformation to generate a processed live view image. It is assumed that the conversion to the equirectangular display including left-right switching is the same as in step S712. The conversion to the equirectangular display including left-right switching may be the same as steps S716 to S718. That is, the control unit 501 generates a map based on the lens design values corrected using the individual values included in the lens information received together with the live view image, and based on the map, the equirectangular display including left-right switching is performed. You can convert to As in the first embodiment, left-right switching may be performed as part of the equirectangular transformation, or may be performed as separate processes.

In step S<b>1418 , the control unit 501 displays the processed live view image generated in step S<b>1417 on the display unit 506 .

In step S1419, the control unit 501 converts the right and left images into equirectangular images without interchanging the positions of the right and left images in the live view image acquired in step S1415. In other words, the control unit 501 performs equirectangular transformation without left-right switching to generate a processed live view image.

In step S<b>1420 , the control unit 501 displays the processed live view image generated in step S<b>1419 on the display unit 506 .

If the camera 100 is not compatible with a two-lens lens, or if the camera 100 is equipped with a single-lens lens, the process of step S1421 is performed. In step S1421, the control unit 501 displays on the display unit 506 the live view image captured by the single-lens lens. The processing in step S1421 is the same as conventional processing for displaying a live view image captured by a single-lens lens on a PC or the like, so detailed description thereof will be omitted.

In each of steps S1410, S1417, and S1419, the control unit 501 performs image processing on the live view image obtained from the connected camera. In step S1413 after steps S1410, S1417, and S1419, the control unit 501 determines whether or not to end the PC live view. If the PC live view is to be continued, the process returns to step S1406 prior to steps S1410, S1417, and S1419. Therefore, in the operation of FIG. 14, there is a possibility that the image processing of any one of steps S1410, S1417, and S1419 will be repeatedly executed.

Therefore, in order to speed up the image processing, the control unit 501 may record information related to the executed image processing in the RAM 503 and use it in subsequent image processing. For example, the control unit 501 records the correspondence (image processing map) between pixels before image processing and pixels after image processing. The image processing map can continue to be used if there is no change in the resolution of the live view image or lens information. The control unit 501 records an image processing map of the image processing when executing any one of the image processing in steps S1410, S1417, and S1419. Then, when executing the same image processing again, the control unit 501 executes the image processing using the recorded image processing map. By doing so, image processing can be speeded up.

15A to 15D are schematic diagrams showing an example of display (PC live view display) on the application screen displayed on the display unit 506 by the control unit 501. FIG. A screen 1500 is an application screen (remote live view screen). Screen 1500 includes live view display area 1501 , guide display area 1502 , guide display area 1503 , operation area 1504 and end button 1508 .

A live view display area 1501 is an area for displaying a live view image. The live view display area 1501 consists of a left display area 1501A and a right display area 1501B. A guide display area 1502 displays a character string indicating which image of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the twin lens is the image displayed in the left display area 1501A. This is the area where A guide display area 1503 displays a character string indicating which image of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the twin lens is the image displayed in the right display area 1501B. This is the area where An operation area 1504 is an area for receiving operations related to PC live view, and radio buttons 1505 and 1506 and a check box 1507 are displayed in the operation area 1504 . A radio button 1505 is a radio button that is selected when performing circular fisheye display, and a radio button 1506 is a radio button that is selected when performing equirectangular display. When the radio button 1505 is in the selected state, the radio button 1506 is in the non-selected state, and when the radio button 1505 is in the non-selected state, the radio button 1506 is in the selected state. A check box 1507 is a check box that is checked when left-right switching is performed. When the check box 1507 is operated, the positions of the right image (right eye image) and left image (left eye image) in the live view image are exchanged, and the character strings displayed in the guide display areas 1502 and 1503 are also exchanged. An end button 1508 is a button for ending the PC live view.

In FIG. 15A, a radio button 1505 for circular fisheye display is selected. The check box 1507 for switching left and right is not checked. Therefore, the live view image obtained from the camera is displayed in the live view display area 1501 as it is. Specifically, the left display area 1501A displays a right-eye image that is a circular fisheye image, and the right display area 1501B displays a left-eye image that is a circular fisheye image.

In FIG. 15B, a radio button 1506 for equirectangular display is selected. The check box 1507 for switching left and right is not checked. Therefore, each of the right-eye video and the left-eye video (both of which are circular fisheye images) in the live view image acquired from the camera is converted into an equirectangular image (left-right switching is not performed). Then, the live view image after equirectangular transformation is displayed in the live view display area 1501 . Specifically, the right-eye image, which is an equirectangular image, is displayed in the left display area 1501A, and the left-eye image, which is an equirectangular image, is displayed in the right display area 1501B.

In FIG. 15C, a radio button 1505 for circular fisheye display is selected, and a check box 1507 for left/right switching is checked. Therefore, the positions of the right-eye video and the left-eye video are switched in the live view image acquired from the camera. Then, the live-view image after left-right switching is displayed in the live-view display area 1501 . Specifically, the left-eye image, which is a circular fish-eye image, is displayed in the left display area 1501A, and the right-eye image, which is a circular fish-eye image, is displayed in the right display area 1501B.

In FIG. 15(d), a radio button 1506 for equirectangular display is selected, and a check box 1507 for left/right switching is checked. Therefore, the positions of the right-eye image and the left-eye image in the live view image acquired from the camera are exchanged, and the right-eye image and the left-eye image (both are circular fisheye images) are converted into equirectangular images. . Then, the live view image after left-right switching and equirectangular conversion is performed is displayed in the live view display area 1501 . Specifically, the left eye image, which is an equirectangular image, is displayed in the left display area 1501A, and the right eye image, which is an equirectangular image, is displayed in the right display area 1501B.

Note that the various controls described above that are performed by the system control unit 50 may be performed by one piece of hardware, or a plurality of pieces of hardware (for example, a plurality of processors or circuits) may share the processing, so that the apparatus Overall control may be performed. Similarly, the various controls described above that are performed by the control unit 501 may be performed by one piece of hardware. Overall control may be performed.

Moreover, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention can be applied to the present invention. included. Furthermore, each embodiment described above merely shows one embodiment of the present invention, and it is also possible to combine each embodiment as appropriate.

Moreover, the present invention is not limited to cameras and PCs, and can be applied to any electronic device that can handle two images with parallax. For example, the present invention can be applied to PDAs, mobile phone terminals, portable image viewers, printers, digital photo frames, music players, game machines, electronic book readers, and the like. In addition, the present invention can be applied to video players, display devices (including projection devices), tablet terminals, smart phones, AI speakers, home appliances, vehicle-mounted devices, and the like.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: Digital camera (camera) 50: System control unit 500: Personal computer (PC) 501: Control unit

Claims (20)

A first image area corresponding to a first optical image input through a first optical system and a second optical system having a predetermined parallax with respect to the first optical system Acquisition means for acquiring one image including a second image area corresponding to a second optical image, and correction information regarding the first optical system and the second optical system;
correction means for executing correction processing for correcting positions of pixels included in the first image region and pixels included in the second image region in the image based on the correction information;
and generating means for generating a processed image by executing a process of transforming the corrected first image area and the corrected second image area. The correction information includes information about the optical axis center of the first optical system in the image and information about the optical axis center of the second optical system in the image,
exchange means for executing exchange processing for exchanging the positions of the first image area and the second image area in the image based on the optical axis centers of the first optical system and the second optical system in the image; 2. The information processing apparatus according to claim 1, further comprising: The replacing means specifies the first image area in the image based on the optical axis center of the first optical system in the image, and specifies the optical axis center of the second optical system in the image, 3. The information processing apparatus according to claim 2, wherein the second image area in the image is specified, and positions of the two specified areas are exchanged. The image is an image in which the first image area and the second image area are arranged side by side,
4. The information processing apparatus according to claim 2, wherein said switching means reverses a horizontal positional relationship between said first image area and said second image area in said image. The first optical system and the second optical system are each a fisheye lens,
5. The information processing apparatus according to claim 1, wherein the first image area and the second image area are areas of a circular fisheye image. 6. The information according to claim 5, wherein said generating means converts each of said corrected first image area and said second image area from a circular fisheye image area to an equirectangular image area. processing equipment. The correction information regarding the first optical system and the second optical system includes design parameters of a lens unit including the first optical system and the second optical system, and a lens unit-specific parameters and
7. The information processing apparatus according to any one of claims 1 to 6, wherein said correction means executes said correction processing based on said design parameter and said inherent parameter. 8. The acquiring means acquires an image file in which the correction information relating to the first optical system and the second optical system is added as metadata to the data of the image. The information processing device according to any one of . The image is a live view image output from an imaging device connectable to a lens unit including the first optical system and the second optical system,
9. The apparatus according to any one of claims 1 to 8, wherein the correction information regarding the first optical system and the second optical system is lens information acquired by the imaging device from the lens unit. The information processing device described. The image is an image captured by an imaging device connectable to a lens unit including the first optical system and the second optical system,
The correction information relating to the first optical system and the second optical system is lens information acquired by the imaging device from the lens unit and added as metadata to the image data. The information processing apparatus according to any one of claims 1 to 8. connecting means connectable to a lens unit including a first optical system and a second optical system having a predetermined parallax with respect to the first optical system;
acquisition means for acquiring information about the first optical system and the second optical system from the lens unit when the lens unit is connected to the connection means;
A first image area corresponding to the first optical image input via the first optical system and a second image area corresponding to the second optical image input via the second optical system and an output means for outputting the information obtained from the lens unit. 12. The imaging apparatus according to claim 11, wherein said output means outputs said image and said information to an external device. 12. The imaging apparatus according to claim 11, wherein said output means stores an image file in which said information is added to said image data as metadata in a storage medium. 14. The information according to any one of claims 11 to 13, wherein the information includes information about the optical axis center of the first optical system in the image and information about the optical axis center of the second optical system in the image. 1. The imaging device according to claim 1. A first image area corresponding to a first optical image input through a first optical system and a second optical system having a predetermined parallax with respect to the first optical system acquiring one image including a second image area corresponding to a second optical image, and correction information regarding the first optical system and the second optical system;
executing a correction process for correcting positions of pixels included in the first image region and pixels included in the second image region in the image based on the correction information;
and executing a process of transforming the corrected first image area and the corrected second image area to generate a processed image. A control method for an imaging device connectable to a lens unit including a first optical system and a second optical system having a predetermined parallax with respect to the first optical system, comprising:
obtaining information about the first optical system and the second optical system from the lens unit when the lens unit is connected to the imaging device;
A first image area corresponding to the first optical image input via the first optical system and a second image area corresponding to the second optical image input via the second optical system and outputting the information obtained from the lens unit. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 10. A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 11 to 14. A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 10. A computer-readable storage medium storing a program for causing a computer to function as each means of the imaging apparatus according to any one of claims 11 to 14.

Images