JP2022191143A - Image processing device and image processing method - Google Patents

Abstract

To provide an image processing device that, even when an imaging apparatus fails to recognize a binocular lens unit, can recognize the type of an image recorded by the imaging apparatus to appropriately process the image.SOLUTION: An image processing device determines whether an image represented by image data includes two circular areas. When determining that the image includes two circular areas, the image processing device executes predetermined processing on the image data.SELECTED DRAWING: Figure 8

Description

The present invention relates to an image processing device and an image processing method.

It is known to display parallax image pairs on a head mounted display (HMD) or the like to provide stereoscopic viewing. Further, Patent Literature 1 describes an image pickup apparatus capable of photographing a pair of parallax images by using a lens having one lens mount and two imaging optical systems (hereinafter referred to as a twin lens). .

JP 2013-141052 A

When a pair of images are formed on one imaging element using a twin lens having two imaging optical systems (a right-eye optical system and a left-eye optical system) with optical axes separated in the horizontal direction, the imaging optical system and the positional relationship of the image is switched between left and right. That is, a captured image is obtained in which the image formed by the right-eye optical system (right image) is positioned on the left side and the image formed by the left-eye optical system (left image) is positioned on the right side.

In order to correctly use the right image and the left image, it is necessary to correctly grasp the relationship between the right image and the left image in the captured image. For this purpose, the imaging apparatus must be able to recognize that a twin-lens unit that forms an image in which the positional relationship between the right image and the left image is reversed is attached.

For example, the imaging device has a twin-lens lens in which the attached lens unit forms an image in which the positional relationship between the right image and the left image is reversed, based on information about the lens unit (lens information) acquired through communication with the lens unit. You may be able to recognize that it is a unit. However, it is not always possible to acquire lens information from all lens units attachable to the imaging apparatus. In addition, there is a possibility that the lens information does not include information that can determine whether or not the mounted lens unit is a twin lens unit that forms an image in which the positional relationship between the right image and the left image is reversed.

The present invention has been made in view of such problems of the prior art. One object of the present invention is to provide an image processing apparatus capable of recognizing the type of image recorded by the imaging device and appropriately processing the image even when the imaging device cannot recognize the twin-lens unit. and to provide an image processing method.

The above objects are provided by obtaining means for obtaining image data, determining means for determining whether an image represented by the image data includes two circular areas, and determining whether the image includes two circular areas by the determining means. and a processing means for executing a predetermined process on the image when it is determined that the image is in the correct state.

It is possible to provide an image processing apparatus and an image processing method capable of recognizing the type of an image recorded by the imaging apparatus and appropriately processing the image even when the imaging apparatus cannot recognize the twin-lens unit. can.

Schematic diagram showing a configuration example of an image processing system according to an embodiment 1 is a perspective view showing an appearance example of a camera according to an embodiment; FIG. 1 is a block diagram showing an example of the functional configuration of a camera according to an embodiment; FIG. Cross-sectional view showing a configuration example of a twin lens unit Schematic diagram regarding the positional relationship of images formed by the single-lens unit and the twin-lens unit FIG. 2 is a block diagram showing an example functional configuration of a PC according to the embodiment; Flowchart relating to the operation of the camera in the embodiment Flowchart relating to PC operation in the embodiment Schematic diagram showing a configuration example of an image processing system according to another embodiment Flowchart of camera operation in another embodiment Flowchart for PC operation in another embodiment Schematic diagram regarding the operation of S703 in FIG.

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. In addition, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In the following embodiments, the case where the present invention is implemented by a personal computer will be described. However, the present invention can be implemented with any electronic device capable of image processing. Such electronic devices include digital cameras, tablet computers, media players, PDAs, mobile phones, smart phones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices.

● (first embodiment)
FIG. 1 shows image processing having a personal computer (PC) 102 as an example of an image processing apparatus according to an embodiment of the present invention, and a digital camera 100 (hereinafter referred to as camera 100) that generates image data to be processed by the PC 500. 1 is a schematic diagram of a system; FIG.

The camera 100 can use the twin-lens unit 300 to acquire data of a captured image (hereinafter referred to as a VR image) with a field of view of 180 degrees for stereoscopy and 180 degrees for stereo. In this embodiment, the camera 100 is assumed to be of an interchangeable lens type, but may not be of an interchangeable lens type. In this embodiment, the twin-lens unit 300 accommodates two imaging optical systems that form a circular fisheye image with a field of view of 180 degrees. A circular fisheye image with parallax can be used to reproduce a virtual reality (VR) image with a stereoscopic effect. The details of the configuration of the twin lens unit 300 will be described later.

The PC 500 is a general personal computer and functions as an image processing apparatus of this embodiment by executing application software. The PC 500 can directly acquire the captured image data acquired by the camera 100 executing the imaging process from the camera 100 through the wired or wireless communication 150 (FIG. 1(a)). Alternatively, the PC 500 may acquire captured image data by reading the captured image data recorded by the camera 100 in a removable medium 160 such as a semiconductor memory card (FIG. 1(b)).

Next, the camera 100 will be explained. FIG. 2 is a perspective view showing an example of the appearance of the camera 100. As shown in FIG. 2A is a perspective view of the camera 100 viewed obliquely from above the front, and FIG. 2B is a perspective view of the camera 100 viewed obliquely from the upper rear.

The camera 100 has a shutter button 101, a power switch 102, a mode changeover switch 103, a main electronic dial 104, a sub electronic dial 105, a moving image button 106, and a viewfinder display section 107 on the top surface. A shutter button 101 is an operation unit for making preparations for photographing or instructing photographing. A power switch 102 is an operation unit that switches the power of the camera 100 between on and off. A mode switch 103 is an operation unit for switching between various modes. A main electronic dial 104 is a rotary operation unit for changing set values such as shutter speed and aperture. The sub-electronic dial 105 is a rotary operation unit for moving a selection frame (cursor), switching images, and the like. A moving image button 106 is an operation unit for instructing start and stop of moving image shooting (recording). A viewfinder display unit 107 displays various setting values such as shutter speed and aperture.

The camera 100 has a display unit 108, a touch panel 109, direction keys 110, a SET button 111, an AE lock button 112, an enlargement button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar on the back. 119. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects a touch operation on the display surface (touch operation surface) of the display unit 108 .

The directional key 110 is an operation unit composed of keys (four-directional keys) that can be pressed up, down, left, and right. An operation can be performed according to the pressed position of the direction key 110 . A SET button 111 is an operation unit that is mainly pressed when deciding a selection item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in the shooting standby state. The enlargement button 113 is an operation unit for switching between on and off of the enlargement mode in the live view display (LV display) of the shooting mode. When the enlargement mode is on, operating the main electronic dial 104 enlarges or reduces the live view image (LV image). Also, the enlargement button 113 is used when enlarging a reproduced image or increasing the enlargement ratio in the reproduction mode.

A playback button 114 is an operation unit for switching between a shooting mode and a playback mode. Pressing the playback button 114 in the shooting mode shifts to the playback mode, and the latest image among the images recorded in the recording medium 227 (to be described later) can be displayed on the display unit 108 . A menu button 115 is an operation unit that is pressed to display a menu screen on which various settings can be made on the display unit 108 . The user can perform various settings of the camera 100 by operating the menu screen displayed on the display unit 108 using the direction key 110 and the SET button 111 . Instead of using the buttons, or in combination with the buttons, the touch panel 109 may be used to enable operation of the menu screen.

An eyepiece 116 is a window for looking through an eyepiece finder (looking-in type finder) 117 . The user can view an image displayed on an internal EVF (Electronic View Finder) 217 , which will be described later, through the eyepiece 116 . The eyepiece detection unit 118 is a sensor that detects whether or not an object is approaching the eyepiece unit 116 .

The touch bar 119 is a linear touch operation unit (line touch sensor) capable of receiving touch operations. The touch bar 119 can be touch-operated with the thumb of the right hand when the grip portion 120 is gripped with the right hand so that the shutter button 101 can be pressed with the index finger of the right hand (with the little finger, ring finger, and middle finger of the right hand). possible). In other words, the touch bar 119 can be operated while looking through the eyepiece finder 117 through the eyepiece unit 116 and ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept a tap operation (touch and release without moving the touch position within a predetermined period), a left/right slide operation (movement of the touch position while being touched), and the like. . A touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 of this embodiment functions as a multi-function bar (M-Fn bar).

The camera 100 also has a grip portion 120, a thumb rest portion 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like. The grip portion 120 is a holding portion formed in a shape that allows the user to easily hold the camera 100 with his or her right hand. The shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated with the index finger of the right hand while holding the camera 100 by gripping the grip portion 120 with the little finger, the ring finger and the middle finger of the right hand. Also, in a similar state, the sub-electronic dial 105 and the touch bar 119 are arranged at positions that can be operated with the thumb of the right hand.

The thumb rest portion 121 (thumb standby position) is a grip portion provided on the rear side of the camera 100 where the thumb of the right hand holding the grip portion 120 can be easily placed without operating any operation portion. The thumb rest portion 121 is made of a rubber member or the like for enhancing holding power (grip feeling). The terminal cover 122 protects a connector such as a connection cable that connects the camera 100 to an external device. The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227, which will be described later. The communication terminal 124 is a terminal for communicating with a lens unit 200 to which the camera 100 is detachable, which will be described later.

<Internal Configuration of Camera 100>
FIG. 3 is a block diagram showing an example of the internal configuration (functional configuration) of a camera system in which the interchangeable lens unit 200 is attached to the camera 100. As shown in FIG. In FIG. 3, the same reference numerals as in FIG. 2 are assigned to the configurations shown in FIG. The description of the configuration already described with reference to FIG. 2 will be omitted as appropriate.

First, the lens unit 200 will be explained.
The lens unit 200 is an example of an interchangeable lens detachable from the camera 100 . The lens unit 200 is a general single eye lens (a lens with one optical axis). The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

The diaphragm 201 is configured such that the aperture diameter can be adjusted. The lens 202 is composed of a plurality of lenses. A diaphragm drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the diaphragm 201 . The AF drive circuit 204 drives the focus lens included in the lens 202 and adjusts the distance at which the lens unit 200 focuses.

The lens system control circuit 205 has, for example, a CPU, a ROM, and a RAM, and controls the operation of each part of the lens unit 200 by loading a program stored in the ROM into the RAM and executing the program by the CPU. The lens unit 200 and the camera 100 are electrically connected through communication terminals 206 and 124, and the lens system control circuit 205 and the system control section 50 of the camera 100 can communicate with each other. A lens system control circuit 205 controls an aperture drive circuit 203, an AF drive circuit 204, and the like based on instructions from the system control unit 50. FIG.

Next, the camera 100 will be explained.
The camera 100 has a shutter 210, an imaging unit 211, an A/D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D/A converter 216, an EVF 217, a display unit 108, and a system control unit 50. .

The shutter 210 is a focal plane shutter that operates based on instructions from the system control unit 50 and controls the exposure time of the imaging unit 211 . The imaging unit 211 is an imaging device (image sensor) configured by a CCD, a CMOS device, or the like that converts an optical image into an electrical signal. In the present embodiment, the imaging unit 211 is an imaging device that supports autofocus (imaging surface phase difference AF) of the imaging surface phase difference detection method. Specifically, the imaging unit 211 can output a focus detection signal pair for realizing phase difference detection autofocus.

The A/D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal (image data). The image processing unit 214 performs predetermined processing (pixel interpolation, resizing processing such as reduction, color conversion processing, etc.) on data input through the A/D converter 212 or the memory control unit 213 . The image processing unit 214 also performs predetermined arithmetic processing using the photographed image data, and calculates evaluation values and the like used for AF and AE. The system control unit 50 performs exposure control and focus detection control based on the obtained calculation result. The image processing unit 214 also calculates the defocus amount based on the pair of focus detection signals obtained from the imaging unit 211 as one of the evaluation values. Further, the image processing unit 214 performs predetermined arithmetic processing using the photographed image data, and performs AWB (Auto White Balance) processing on the image data based on the obtained arithmetic result.

Image data from the A/D converter 212 is written into the memory 215 via the image processing section 214 and the memory control section 213 . Alternatively, the image data from the A/D converter 212 is written to the memory 215 via the memory control section 213 without passing through the image processing section 214 . The memory 215 stores image data output by the A/D converter 212, image data generated by the image processing unit 214, and the like. The image data generated by the image processing unit 214 includes display image data to be displayed on the display unit 108 and the EVF 217 and recording image data to be recorded on the recording medium 227 . The memory 215 has a storage capacity sufficient to store a predetermined number of still image data, and a predetermined time of moving image data and audio data. A partial area of the memory 215 is used as a video memory for the display unit 108 .

A D/A converter 216 converts the image data stored in the memory 215 into an analog signal suitable for display on the display unit 108 or the EVF 217 . Therefore, the display image data written in the memory 215 is displayed on the display unit 108 or the EVF 217 via the D/A converter 216 . The display unit 108 and the EVF 217 perform display according to analog signals from the D/A converter 216 . The display unit 108 and the EVF 217 are, for example, displays such as LCD and organic EL.

While the imaging unit 211 is shooting a moving image, the image data stored in the memory 215 through the A/D converter 212 is converted into an analog signal by the D/A converter 216, and sequentially transferred to the display unit 108 or the EVF 217 for display. do. Accordingly, live view display can be performed on the display unit 108 and the EVF 217 .

The system controller 50 is a controller comprising at least one processor (CPU) and/or at least one circuit. That is, the system control unit 50 may be a processor (CPU), a circuit, or a combination of a processor and a circuit. For example, if the system control unit 50 has a processor (CPU), the system control unit 50 controls the entire camera 100 by loading a program stored in the nonvolatile memory 219 into the system memory 218 and executing it by the processor. The system control unit 50 also performs display control by controlling the memory 215, the D/A converter 216, the display unit 108, the EVF 217, and the like.

The camera 100 also has a system memory 218 , a nonvolatile memory 219 , a system timer 220 , a communication section 221 , an orientation detection section 222 and an eye proximity detection section 118 .
A RAM, for example, is used for the system memory 218 . In the system memory 218, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are expanded.
The nonvolatile memory 219 may be, for example, an electrically erasable/recordable EEPROM. Constants for operation of the system control unit 50, programs, and the like are recorded in the nonvolatile memory 219. FIG.

The system timer 220 is a timer that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives image signals and audio signals to and from an external device connected wirelessly or by a wired cable. The communication unit 221 can communicate with an external device conforming to a wireless LAN (Local Area Network) and a device on the Internet. Also, the communication unit 221 can communicate with an external device conforming to Bluetooth (registered trademark). The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices.

The orientation detection unit 222 outputs a signal representing the orientation of the camera 100 with respect to the direction of gravity. Based on the signal output by the attitude detection unit 222, it is determined whether the image captured by the imaging unit 211 is an image captured with the camera 100 held horizontally or an image captured with the camera 100 held vertically. It is possible. The system control unit 50 can add direction information according to the signal output from the posture detection unit 222 to the image file of the image captured by the imaging unit 211, rotate the image, and record the image. . The orientation detection unit 222 can use, for example, an acceleration sensor, a gyro sensor, or the like. The system control unit 50 can also detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary, etc.) based on the output signal of the posture detection unit 222 .

The eyepiece detection unit 118 can detect the approach of any object to the eyepiece unit 116 of the eyepiece finder 117 incorporating the EVF 217 . For example, an infrared proximity sensor can be used as the eye proximity detection unit 118 . When an object approaches, the infrared rays projected from the light projecting unit of the eyepiece detection unit 118 are reflected by the object and received by the light receiving unit of the infrared proximity sensor. The presence or absence of an object close to the eyepiece 116 can be determined based on the amount of received infrared rays.

The system control unit 50 switches between display (display state) and non-display (non-display state) of the display unit 108 and the EVF 217 according to the presence or absence of a nearby object detected by the eye proximity detection unit 118 . Specifically, at least when the shooting standby state is set and the switching setting of the display destination is automatic switching, if an approaching object is not detected, the display of the display unit 108 is turned on, and the display of the EVF 217 is turned off. do. Also, if an approaching object is detected, the display of the EVF 217 is turned on, and the display of the display unit 108 is turned off. Note that the eye proximity sensor 118 is not limited to the infrared proximity sensor, and other sensors may be used as long as they can detect a state that can be regarded as eye proximity.

The camera 100 also has a viewfinder display unit 107, a viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium I/F 226, an operation unit 228, a video signal output I/F 240, and the like.

The viewfinder display unit 107 displays various setting values of the camera 100 such as shutter speed and aperture via the viewfinder display drive circuit 223 . The power control unit 224 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching blocks to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining amount of the battery. Also, the power supply control unit 224 controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies necessary voltage to each unit including the recording medium 227 for a necessary period. The power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, an AC adapter, or the like. A recording medium I/F 226 is an interface with a recording medium 227 such as a memory card or hard disk.

A recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be detachable or may be built-in. A video signal output I/F 240 is an interface for outputting an image signal from the camera 100 to an external device. The video signal output I/F 240 has one or more interfaces conforming to the standard. Although the standard is not particularly limited, it may be an interface conforming to the HDMI (registered trademark) standard, for example. The camera 100 outputs, for example, video data being captured to an external device (video signal receiving device 241) connected to the video signal output I/F 240. FIG. A PC 500 in FIG. 1A corresponds to the video signal receiving device 241 .

The operation unit 228 is an input unit that receives operations (user operations) from the user, and is used to input various instructions to the system control unit 50 . The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, other operation members 229, and the like. Other operation members 229 include the main electronic dial 104, the sub electronic dial 105, the movie button 106, the direction keys 110, the SET button 111, the AE lock button 112, the zoom button 113, the playback button 114, the menu button 115, the touch bar 119, and the like. is included.

The shutter button 101 has a first shutter switch 230 and a second shutter switch 231 . The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway, that is, during the operation of the shutter button 101, and generates a first shutter switch signal SW1. The system control unit 50 interprets the first shutter switch signal SW1 as a photographing preparation instruction, and starts photographing preparation processing. The shooting preparation processing includes AF processing, AE processing, AWB processing, flash pre-emission processing, and the like.

The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed, that is, when the shutter button 101 is fully pressed, and generates a second shutter switch signal SW2. The system control unit 50 interprets the second shutter switch signal SW2 as a still image shooting instruction, and starts a still image shooting operation based on the exposure conditions determined by the AE processing. Then, each unit is controlled so as to execute a series of photographing processes from signal reading from the image pickup unit 211 to generation of an image file including still image data obtained by photographing and writing to the recording medium 227 .

A mode switch 103 switches the operation mode of the system control unit 50 to one of a still image shooting mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes, custom modes, and the like, which are shooting settings for each shooting scene. The user can directly switch to one of the above-described shooting modes by using the mode switch 103 . Alternatively, the user can use the operation unit 228 to selectively switch to any one of the plurality of displayed modes after switching once to the shooting mode list screen using the mode switching switch 103 . Similarly, the movie shooting mode may also include multiple modes.

The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured integrally. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 . By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI is provided as if the user can directly operate the screen displayed on the display unit 108. can be configured. GUI is an abbreviation for Graphical User Interface. The touch panel 109 can use any one of various methods such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. Depending on the method, there is a method of detecting that there is a touch when there is a contact with the touch panel 109, and a method of detecting that there is a touch when a finger or a pen approaches the touch panel 109. Either method can be used. may

The system control unit 50 can detect the following operations or states on the touch panel 109 .
The touch panel 109 is newly touched by a finger or pen that has not touched the touch panel 109, that is, the start of touch (hereinafter referred to as Touch-Down).
A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).
- The touch panel 109 is moved while being touched by a finger or pen (hereinafter referred to as touch-move).
- The finger or pen touching the touch panel 109 is released from the touch panel 109, ie, the end of the touch (hereinafter referred to as Touch-Up).
A state in which nothing is touched on the touch panel 109 (hereinafter referred to as Touch-Off).

When touchdown is detected, touchon is also detected at the same time. After touchdown, touchon continues to be detected unless touchup is detected. Touch-on is detected at the same time when touch-move is detected. Even if touch-on is detected, touch-move is not detected if the touch position does not move. After it is detected that all the fingers and pens that have touched have touched up, the touch is turned off.

The system control unit 50 is notified of these operations/states and the coordinates of the position where the finger or pen touches the touch panel 109 . Based on the notified information, system control unit 50 determines what kind of operation (touch operation) has been performed on touch panel 109 . As for the touch move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component/horizontal component on the touch panel 109 based on the change in the position coordinates. When it is detected that the touch-move has been performed by a predetermined distance or more, it is determined that the slide operation has been performed. An operation of touching the touch panel 109 with a finger and quickly moving it by a certain distance and then releasing it is called a flick. A flick is, in other words, an operation of quickly tracing the touch panel 109 as if flicking it with a finger. It is detected that a touch-move has been performed for a predetermined distance or more at a predetermined speed or more, and if a touch-up is detected as it is, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Further, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called pinch-in, and a touch operation in which the touch positions are moved away from each other is called pinch-out. Pinch-out and pinch-in are collectively called pinch operation (or simply pinch).

<Structure of multi-lens unit>
FIG. 4 is a schematic diagram showing a configuration example of a twin-lens unit 300 as an example of a multi-lens unit. The twin lens unit 300 is a VR180 lens unit that forms a VR image conforming to the VR180 standard. In this specification, the term "multi-lens" refers to a lens unit having a plurality of imaging optical systems in one lens mount (or lens barrel) and having a plurality of optical axes. FIG. 4 shows a state in which the twin lens unit 300 is attached to the camera 100. As shown in FIG. 4 shows only part of the configuration of the camera 100 shown in FIG.

The twin lens unit 300 is a type of interchangeable lens that can be attached to and detached from the camera 100 . Twin lens unit 300 has two imaging optical systems 301L and 301R in one lens barrel, and therefore has two optical axes.

Here, it is assumed that the two imaging optical systems 301L and 301R are arranged such that when the twin lens unit 300 is attached to the camera 100, the two optical axes are aligned on a horizontal straight line. The two imaging optical systems 301L and 301R have a viewing angle of approximately 180 degrees and can photograph the range of the front hemisphere. Specifically, the two imaging optical systems 301L and 301R each capture a field of view of 180 degrees in the horizontal direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the vertical direction (vertical angle, elevation angle, pitch angle). can. The two imaging optical systems 301L and 301R form a pair of parallax images having left and right parallaxes on the imaging surface of the imaging unit 211 . In the following description, the imaging optical system 301L will be referred to as the left-eye optical system 301L, and the imaging optical system 301R will be referred to as the right-eye optical system 301R.

The right-eye optical system 301R and the left-eye optical system 301L each have a plurality of lenses, reflecting mirrors, and the like. The multiple lenses include at least a focus lens for adjusting the focal length. The twin lens unit 300 also has a lens system control circuit 303 . The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the object side face the same direction, and the respective optical axes are substantially parallel.

Although not shown in FIG. 4, the twin lens unit 300 has the same configuration as the AF driving circuit 204. FIG. In this case, the AF driving circuit that drives the focus lenses of the right-eye optical system 301R and the left-eye optical system 301L in conjunction with each other, and the focus lens of at least one of the right-eye optical system 301R and the left-eye optical system 301L are independently driven. It can have one or more with an AF drive circuit that The focus lens is driven by the lens system control circuit 303 under the control of the system control unit 50 .

In addition, the twin lens unit 300 further has an encoder that detects the rotation amount and rotation direction of the focus ring provided on the lens barrel. The lens system control circuit 303 provides a so-called by-wire manual focus function by controlling the AF drive circuit according to the focus lens operation detected by the encoder. In this case, the twin lens unit 300 may have a switch with which the user can switch the focus lens driven by the focus ring operation.

The twin-lens unit 300 is a VR180 lens for capturing an image in the VR180 format, which is a VR image format that enables twin-lens stereoscopic vision, with the camera 100 . The VR180 lens has a fisheye lens in which the right-eye optical system 301R and the left-eye optical system 301L each have a viewing angle of approximately 180 degrees. The right-eye optical system 301R and the left-eye optical system 301L may acquire an image capable of binocular VR display as VR180, and the viewing angle may be about 160 degrees. The lens for VR180 can form a right image (first image) by the right eye optical system 301R and a left image (second image) by the left eye optical system 301L on the same imaging plane. Here, it is assumed that the imaging unit 211 of the camera 100 has one imaging element, and the twin-lens unit 300 forms right and left images on the imaging surface of one imaging element. However, even if the camera 100 has two image pickup elements arranged in parallel and the twin-lens unit 300 forms a right image on the image pickup surface of one image pickup element and a left image on the image pickup surface of the other image pickup element. good.

The twin-lens unit 300 includes a focus ring for adjusting the focus of the right-eye optical system 301R and a focus ring for adjusting the focus of the left-eye optical system 301L. Alternatively, a focus ring for simultaneously adjusting the focus of the right-eye optical system 301R and the left-eye optical system 301L and a focus ring for adjusting the focus of one of the right-eye optical system 301R and the left-eye optical system 301L are provided. By operating these focus rings, the user can manually adjust the focal distances of the right-eye optical system 301R and the left-eye optical system 301L. These focus rings may be provided individually, or may be realized by switching the function of one focus ring in the case of a by-wire system.

The twin-lens unit 300 is attached to the camera 100 via a mount section, like the (single-lens) lens unit 200 . The mount section is composed of a lens mount section 304 and a camera mount section 305 . When the twin lens unit 300 is attached to the camera 100, the communication terminal 124 of the camera 100 and the communication terminal 306 of the twin lens unit 300 are electrically connected. As a result, the system control section 50 of the camera 100 and the lens system control circuit 303 of the twin lens unit 300 can communicate with each other. However, in the present embodiment, the camera 100 detects from the twin-lens unit 300 that the twin-lens unit 300 is a VR180 lens and that the positional relationship between the right and left circular fisheye images is reversed. Assume that it is not possible to obtain information that can be determined. The information that can determine that the twin-lens unit 300 is a VR180 lens may be, for example, the circular fisheye center coordinates, the circular fisheye radius, and the like.

In this embodiment, the right image and the left image are formed on the imaging surface of the imaging unit 211 while being separated from each other in the horizontal direction. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one imaging device. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. By mounting the twin-lens unit 300 in this manner, a pair of parallax images (right image and left image) formed by the right-eye optical system 301R and the left-eye optical system 301L can be acquired in one shot. . In addition, the acquired right image and left image are VR-displayed as an image for the right eye and an image for the left eye, so that the user can observe a stereoscopic VR image in a range of approximately 180 degrees, a so-called VR180 image. be able to.

FIG. 5 is a schematic diagram of a subject image formed in the imaging unit 211 by the (single-lens) lens unit 200 and the twin-lens unit 300 . The left side shows the (single-lens) lens unit 200, and the right side shows the twin-lens unit 300. FIG. Here, it is assumed that a scene in which a subject (person's face) 401 exists in the center is photographed. Also, for the sake of convenience, the difference in the angle of view of the imaging optical system of each lens unit is ignored.

The subject image formed by the (single-lens) lens unit 200 on the imaging unit 211 is an inverted normal image of the subject 401 (vertically reversed normal image). Therefore, when the captured image 402a is rotated 180 degrees around the center of the image, an image 403a in which the photographing scene and the subject 401 are correctly positioned vertically and horizontally can be obtained.

On the other hand, in the case of the twin-lens unit 300, the left-eye optical system 301L and the right-eye optical system 301R form an inverted normal image of the subject 401 on the imaging unit 211 to obtain a captured image 402b. An image 403b is obtained by rotating the captured image 402b by 180 degrees around the image center. A format for recording a circular fisheye image formed by the left-eye optical system 301L and the right-eye optical system 301R like the image 403b is called a mesh format, and a format in which two images are arranged side by side is called a side-by-side format. There is also In the image 403b, the top, bottom, left, and right of the subject 401 are the same as the shooting scene, but the subject image (right image) formed by the right-eye optical system 301R is on the left, and the subject image (left image) formed by the left-eye optical system 301L is on the left. It exists on the right side, and the positional relationship between the right and left images is reversed. Therefore, if the subject image on the right side of the image 403b is used as the right image and the subject image on the left side as the left image, correct display cannot be performed.

If the camera 100 can recognize that the captured image includes the right image and the left image and that the positional relationship between the right image and the left image is reversed, the information about the right image and the left image is recorded in association with the data of the captured image. Then, the right image and the left image can be used correctly from the captured image. However, if the captured image includes a right image and a left image and the camera 100 cannot recognize that the positional relationship between the right image and the left image is reversed, the information about the right image and the left image is stored in the captured image data. cannot be associated and recorded. Even in such a case, the present embodiment makes it possible to correctly use the right image and the left image from the captured image. Details will be described later.

Here, the VR image is an image that can be displayed in VR, which will be described later. The VR image can be an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera) or a panoramic image with an image range (effective image range) that is wider than the display range that can be displayed on the display unit at once. etc. are included. Also, the VR image may be either a still image or a moving image. The moving images may be prerecorded moving images or live images (images captured from a camera in near real time).

A VR image has an image range (effective image range) of 360 degrees in the horizontal direction and 360 degrees in the vertical direction at maximum. In addition, even if the VR image is less than 360 degrees in the horizontal direction and less than 360 degrees in the vertical direction, the angle of view that is wider than the angle of view that can be taken with a normal camera, or the display range that can be displayed at once on the display unit Images with a wider image range are also included. An image captured by the camera 100 using the twin-lens unit 300 described above is a type of VR image. A VR image can be displayed in VR by, for example, setting the display mode of a display device (a display device capable of displaying a VR image) to “VR view”. A VR image having an angle of view of 360 degrees is VR-displayed, and the user changes the posture of the display device in the left-right direction (horizontal rotation direction), thereby viewing an omnidirectional image seamless in the left-right direction. can.

Here, the VR display (VR view) is a display mode for displaying an image of a predetermined range of the field of view according to the orientation of the display device, out of the field of view captured in the VR image. VR display includes “single-lens VR display (single-lens VR view)” in which one image is displayed by performing deformation (distortion-corrected deformation) that maps a VR image onto a virtual sphere. In addition, for VR display, there is a "twin-eye VR display (twin-eye VR view)" in which a VR image for the left eye and a VR image for the right eye are transformed by mapping onto a virtual sphere and displayed side by side in the left and right regions. There is.

A stereoscopic view is possible by performing “dual-eye VR display” using a VR image for the left eye and a VR image for the right eye that have parallax with each other. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image within a visual field corresponding to the orientation of the user's face is displayed. For example, at a certain point in the VR image, display an image of a viewing range centered at 0 degrees in the horizontal direction (a specific direction, such as north) and 90 degrees in the vertical direction (90 degrees from the zenith, that is, horizontal). Suppose there is When the posture of the display device is reversed from this state (for example, the display surface is changed from facing south to facing north), the same VR image can be displayed by 180 degrees in the horizontal direction (opposite direction, for example, south), and by 180 degrees in the vertical direction. The display range is changed to an image with a visual field range centered at 90 degrees. That is, when the user wears the HMD and turns his face from north to south (that is, turns his back), the image displayed on the HMD is also changed from the north image to the south image.

Note that the VR image captured using the twin lens unit 300 of the present embodiment is a VR180 format image captured in a range of approximately 180 degrees forward, and does not include an image within a range of approximately 180 degrees backward. When such a VR180 format image is displayed in VR and the posture of the display device is changed to the side where no image exists, for example, a blank area is displayed.

By displaying a VR image in VR in this way, the user visually feels as if he or she is in the VR image (inside the VR space). Note that the display method of the VR image is not limited to the method of changing the posture of the display device. For example, the display range may be moved (scrolled) according to a user operation via a touch panel, direction buttons, or the like. In addition, when displaying VR (when the display mode is "VR view"), in addition to changing the display range due to changes in posture, the display range can be changed according to touch moves on the touch panel, drag operations with a mouse, etc., pressing direction buttons, etc. You can change it. A configuration in which a display device such as a smartphone is attached to VR goggles (head mount adapter) is a kind of HMD.

Next, the PC 500 will be explained. FIG. 6 is a block diagram showing a functional configuration example of the PC 500. As shown in FIG. The PC 500 may be an electronic device commercially available as a personal computer.

The control unit 501 is a processor capable of executing programs, such as a CPU (Central Processing Unit). A ROM (Read Only Memory) 502 is, for example, an electrically rewritable nonvolatile memory. The ROM 502 stores programs executed by the control unit 501, various setting values, and parameters. A RAM (Random Access Memory) 503 is used to load programs executed by the control unit 501 and temporarily store various data. A portion of the RAM 503 may be used as a buffer memory or video memory.

The external storage device 504 is a large-scale storage device built into the PC 500 . The external storage device 504 is typically a hard disk drive (HDD) or solid state drive (SSD). The external storage device 504 may include a storage device using removable media, such as a memory card reader, in addition to an HDD or SSD. An external storage device 504 stores an OS (Operating System), application programs, user data, and the like. A portion of the external storage device 504 may be used as a memory swap area. An image data file obtained by receiving from the camera 100 or reading from a memory card is stored in the HDD or SSD of the external storage device 504 .

An operation unit 505 is a general term for input devices that can be operated by a user, such as a keyboard, a mouse, and a touch panel. The display unit 506 is, for example, a liquid crystal display (LCD) and may have a touch panel. A display unit 506 is used by the OS and applications running on the PC 500 to display various information and data.

The communication unit 507 has, for example, a communication circuit for wireless communication with an external device. The communication unit 507 conforms to one or more wireless communication standards. Examples of wireless communication standards include Bluetooth (registered trademark) and wireless LAN (IEEE 802.11x), but are not limited to these.

The external I/F 508 has, for example, a communication circuit for wired communication with an external device. The external I/F 508 conforms to one or more wired communication standards. Typical wired communication standards include, but are not limited to, USB (Universal Serial Bus), HDMI, Thunderbolt (registered trademark), Ethernet (registered trademark), and the like.
A system bus 509 connects the blocks described above so as to be able to communicate with each other.

Next, the photographing operation of camera 100 will be described using the flowchart of FIG. The photographing operation is realized by the system control unit 50 executing a program to perform necessary control. The shooting operation shown in FIG. 7 is started when the user turns on the power of the camera 100 by operating the power switch 102, for example. The system control unit 50 puts the camera 100 into a shooting standby state after executing the activation process associated with power-on. In the shooting standby state, the camera 100 continues to shoot moving images and displays live view on the EVF 217 or the display unit 108 .

In S<b>601 , the system control unit 50 reads an image signal corresponding to one frame of a moving image from the imaging unit 211 and outputs it to the A/D converter 212 .
In S602, the system control unit 50 causes the image processing unit 214 to generate image data for live view display. The image processing unit 214 stores the image data for live view display in the video memory area of the memory 215 . The D/A converter 216 D/A converts the image data stored in the video memory area of the memory 215 and displays it on at least one of the EVF 217 and the display unit 108 . Also, the image processing unit 214 generates an evaluation value based on the image data and outputs it to the system control unit 50 . The system control unit 50 can determine exposure conditions based on the evaluation value and adjust the focal length of the lens unit.

In S603, the system control unit 50 determines whether or not the user has issued a recording start instruction. For example, when the second shutter switch signal SW2 is detected, the system control unit 50 can determine that the recording start instruction has been issued. The system control unit 50 executes S604 if it is determined that the recording start instruction has been issued, and executes again from S601 for live view display of the next frame if it is not determined that the recording start instruction has been issued. In this manner, in the shooting standby state, the system control unit 50 continues to perform the live view display operation until it is determined that the recording start instruction has been issued. Note that the recording start instruction may be an instruction to start recording a moving image by operating the moving image button 106 . Further, when an instruction other than the recording start instruction is detected, the system control unit 50 executes an operation according to the instruction, but the details thereof will be omitted.

In S<b>604 , the system control unit 50 reads the image signal from the imaging unit 211 and outputs it to the A/D converter 212 . The system control unit 50 reads, for example, an image signal with a resolution higher than that during live view display. The system control unit 50 causes the image processing unit 214 to generate image data for recording. The image processing unit 214 stores image data for recording in the memory 215 .

In S605, the system control unit 50 acquires information to be recorded as metadata of the captured image data. The system control unit 50 can acquire, for example, shooting conditions (information on the settings and states of the camera 100 and the lens unit at the time of shooting), parameters used for development processing, and the like. These are examples, and other information may be obtained. The shooting conditions may include, for example, shutter speed, aperture value, exposure conditions such as ISO sensitivity, frame rate, resolution, data compression format, color space, gamma value, attitude of the camera 100, flash lighting, and the like. In the case of moving image data, the metadata also includes information indicating the frame start position in the moving image file. Also, if lens information can be obtained from the lens unit, the lens information can also be included in the metadata. Note that information not exemplified here may be recorded as metadata.

In S<b>606 , the system control unit 50 records the data file storing the image data for recording generated by the image processing unit 214 in S<b>604 in the recording medium 227 , for example.
In S607, the system control unit 50 records the information acquired in S605 as metadata of the image data in association with the data file recorded in S606. In this embodiment, metadata is recorded in the data file recorded in S606.

In S608, the system control unit 50 determines whether or not the user has issued a recording end instruction. For example, if the second shutter switch signal SW2 is not detected, the system control unit 50 can determine that the recording end instruction has been given. In the case of moving image shooting, the system control unit 50 can determine that a recording end instruction has been issued, for example, when detecting an operation of the moving image button 106 . The system control unit 50 ends the shooting operation when it is determined that the recording end instruction has been issued. Note that the system control unit 50 may execute S601 to return to the shooting standby state. If it is determined that the recording end instruction has not been issued, the system control unit 50 returns to S604 to shoot the next frame of the moving image or the still image.

When the two-lens unit 300 is attached to the camera 100, two circular fisheye images are arranged on the left and right (mesh, side-by-side format), and the left and right positional relationships of the circular fisheye images are reversed. An image (still image or video) is recorded. The data format of still images and moving images is not particularly limited, and may be a RAW format or a format after development processing (JPEG format, MPEG format, etc.). Note that the captured image may be transmitted to the PC 500 through the video signal output I/F 240 instead of being recorded on the recording medium 227 or in addition to being recorded on the recording medium 227 . PC 500 stores the image data file received through communication unit 507 or external I/F 508 in external storage device 504 .

Next, the image data display operation of the PC 500 will be described with reference to the flowchart of FIG. The following operations are realized in PC 500 by control unit 501 executing an image processing application stored in external storage device 504 .

Here, the control unit 501 detects that the user has selected an image data file to be processed by operating the GUI presented on the display unit 506 by the image processing application through the operation unit 505, and instructed execution of processing. Action shall be initiated accordingly. It is assumed that the image data file to be processed is selected from image data files stored in the external storage device 504 (HDD, SSD, or memory card), for example.

In S701, the control unit 501 acquires metadata regarding image data in the selected image data file.

In S<b>702 , the control unit 501 uses the metadata acquired in S<b>701 as necessary, and reads image data to be processed from the image data file into the RAM 503 .

In S703, the control unit 501 (image processing means) determines the type of the image data read in S702. Specifically, the control unit 501 determines whether the image data is image data including two circular fisheye images. This determination corresponds to determination of whether or not the image data is VR image data in mesh format. In the case of an image including a circular fisheye image, the peripheral area other than the circular fisheye image is black pixels. Therefore, the control unit 501 generates binarized image data by, for example, applying a threshold close to the luminance value of a black pixel to the luminance value of each pixel of the image data. Then, if it is determined that the image represented by the binarized image data includes two white circular regions, the control unit 501 (determining means) sets the type of the image data read in S702 to two circular regions. It is determined that the image data includes a circular fisheye image.

Whether or not the white area is circular can be determined by any known method. For example, if a plurality of arbitrary straight lines that intersect two points on the outer circumference of the white area are drawn, and perpendicular lines passing through the midpoints of the straight lines intersect at one point, it can be determined that the area is circular. However, considering that an error actually occurs, it may be determined that the area is a circular area if the intersection points of a plurality of perpendicular lines exist within a certain distance.

Also, when two circular regions are detected, the control unit 501 may further determine whether the sizes of the circular regions are the same and whether the distance between the circular regions is less than a threshold. Then, if at least one of the two circular regions having the same size and the distance between the circular regions being less than a threshold is satisfied, the control unit 501 determines that the image data is the two circular fisheye images. It is determined that the VR image data includes

That the two circular regions have the same size and that the distance between the circular regions is less than a threshold are conditions that increase the possibility that the image data is VR image data containing two circular fisheye images. is. The size of a circular area may be, for example, the number of pixels in the area, or the maximum length of a straight line connecting two points on the perimeter of the area. If the difference in the number of pixels or the maximum length is less than a threshold, it can be determined that the sizes of the two circular regions are the same. Also, the distance between the circular regions may be the shortest distance between the perimeters of the two circular regions.

Another method may be used to determine that the image data is VR image data containing two circular fisheye images. For example, the control unit 501 divides the image data so as to divide the image into a left area and a right area. Then, the control unit 501 detects a partial area formed by pixels equal to or larger than the threshold for each area. The control unit 501 determines whether or not the partial area is a circular area by the method described above. If a circular area is detected in each of the right area and the left area, the control unit 501 can determine that the image data is VR image data including two circular fisheye images. Also in this case, the image data contains two circular fisheye images when at least one of the two circular regions having the same size and the distance between the circular regions being less than a threshold is satisfied. It may be determined that the data is VR image data.

The control unit 501 executes S704 if it is determined that the image data to be processed is data of an image containing two circular fisheye images, and otherwise executes S707.

Note that when the size of the effective pixel area of the imaging unit 211 is small, as shown in FIG. A partially missing image 1220 can be recorded. Even if the end portion of the circular fisheye image is missing, the VR image can be displayed and reproduced by using the central portion. However, the viewing angle of the VR image is narrower than when the circular fisheye image is not lacking.

When the white region 1221 reaches the edge of the image 1220, the portion of the periphery of the white region 1221 that does not form the edge 1222 of the image 1220 is used, and the white region 1221 is part of the circular region by the method described above. It can be determined whether there is In other words, if a plurality of arbitrary straight lines that intersect two points included in the portion that does not form the edge 1222 of the image 1220 are drawn from the periphery of the white region 1221, and the perpendiculars that pass through the midpoints of the straight lines intersect at one point, can determine that the white area 1221 is part of the circular area. If it is determined that the white area 1221 is part of a circular area, then it can be determined that the image 1220 contains part of a circular fisheye image.

For example, if it is not determined in S703 that the image includes two circular fisheye images, the control unit 501 further determines whether or not the image includes two partially missing circular fisheye images. can be done. Then, when it is determined that the image includes two partially-missing circular fisheye images, the control unit 501 regards the image as including two circular fisheye images, and determines that the image includes two circular fisheye images. may be performed. In this case, S703 can be regarded as a determination as to whether or not the image data includes at least two partial circular fisheye images, and S704 to S706 can be regarded as processing for at least partial circular fisheye images. can.

In S704, the control unit 501 determines the center coordinates and radii of the two circular fisheye images included in the image. Since the perpendicular bisector of the chord of a circle passes through the center of the circle, draw two arbitrary straight lines that intersect the two points on the circumference of the white area determined to be circular, and find the perpendicular bisector of each straight line. can be used as the center coordinates of the white area.

A specific example will be described with reference to FIG. For example, for each of the two circular regions 1201 detected from the data of the image 1200 in S703, the pixel value is checked horizontally at an arbitrary vertical position (Y coordinate), and the X coordinate X1 of the pixel that changed from black to white, Obtain the X coordinate X2 of the pixel converted from white to black. Also, the pixel value is examined in the vertical direction at an arbitrary horizontal position (X coordinate), and the Y coordinate Y1 of the pixel changed from black to white and the Y coordinate Y2 of the pixel changed from white to black are obtained. At this time, the center coordinates 1202 (X0, Y0) of the circular area 1201 have an X coordinate (X0) of (X1+X2)/2 and a Y coordinate (Y0) of (Y1+Y2)/2. Further, if the pixel value is checked in the horizontal direction from the center coordinates and the X coordinate of the pixel that has changed from white to black is X3, then the radius of the circle can be obtained by X3-((X1+X2)/2). The central coordinates are calculated, for example, as image coordinates with one point in the image (for example, the center of the image) as the origin. It should be noted that the central coordinates and the radius of a circular area 1221 with a part missing as shown in FIG. can ask.

The method described here is only an example, and the center coordinates and radius of the circular fisheye image may be calculated by other image processing. In consideration of errors, the final center coordinates may be the average coordinates of the center coordinates obtained from a plurality of different combinations of two straight lines for one circular area. Similarly, the radius may also be obtained by averaging the results calculated in a plurality of directions from the center coordinates.

In S705, the control unit 501 (generating means) extracts images of two circular regions corresponding to the right and left images from the image data using the center coordinates and radii of the two circular regions obtained in S704. The control unit 501 may extract a surrounding black area so that the circular area is not missing. Then, the control unit 501 generates image data in which the positions of the right image and the left image are interchanged. Note that the control unit 501 arranges the images of the extracted circular areas so that the center coordinates of the left and right circular areas obtained in S704 are maintained even after the arrangement is changed. For example, a circular area (right image) extracted from the left side of the image is arranged so that its central coordinates match the central coordinates calculated for the circular area (left image) on the right side of the image. The same is true for the circular area extracted from the left side of the image.

Alternatively, if the right-eye coordinates and left-eye coordinates of the image after the arrangement is exchanged are known, the arrangement position of the right image is determined based on the right-eye coordinates, and the arrangement position of the left image is determined based on the left-eye coordinates. You may

In S706, the control unit 501 displays the VR180 image data generated in S705 in which the arrangement of the circular fisheye image is replaced on the display unit 506, or displays the VR goggles or the like connected through the communication unit 507 or the external I/F 508. output to a 3D display device. Note that when displaying VR180 image data, the control unit 501 may apply equirectangular transformation to the circular fisheye image as necessary to convert from the mesh format to the equirectangular format.

In S707, the control unit 501 displays the image data acquired in S702 on the display unit 506 as it is.

Although FIG. 8 shows that the display operation ends when S706 and S707 are finished, the process may return to S701 and wait for an instruction to process another image data. Also, when the image data to be processed is VR180 moving image data, the processing of S704 to S706 is repeatedly performed for each frame. As for the moving image data, the processing results of S703 and S704 for the first frame can be used for the second and subsequent frames, so S704 can be omitted for the second and subsequent frames.

It should be noted that the user may specify whether or not to exchange the positions of the right image and the left image at an arbitrary timing of S704 to S706. If it is specified not to change the arrangement, the control unit 501 displays the image data acquired in S702 as it is. Note that if it is determined in S703 that the image does not include two circular fisheye images, it may be determined whether or not the image includes two partially missing circular fisheye images. Then, if it is determined that the image includes two partially-missing circular fisheye images, the processing from S704 onward may be applied to the partially-missing circular fisheye images. .

Even if the metadata does not include information about whether the image data to be processed is VR180 image data, the metadata may include the model name of the lens unit as general shooting information. In this case, in step S701, the model name of the lens unit is read, and the model name list of the VR180 lens registered in advance is referenced to determine the type of image data. It is possible to determine whether or not In this case, the determination in S703 can be performed without image processing. The control unit 501 may execute S704 for image data captured using the VR180 lens, and S707 for image data captured using another lens.

In addition, the control unit 501 controls at least one of image data in which the positions of the right and left images are interchanged, and image data in which the positions of the right and left images are interchanged and equirectangular transformation is applied to the right and left images. may be stored in the external storage device 504 as another image file. These image data can be handled as they are by general VR180 compatible devices such as VR goggles.

When these image data are saved separately, the metadata recorded in the image data file is that the image data is VR180 image data, that the positions of the right and left images have been switched, and that the equirectangular conversion has been completed. Information indicating one or more things can be recorded. This simplifies processing in a VR system that uses image data.

In this embodiment, it is assumed that the camera 100 cannot acquire information from the twin lens unit 300 indicating that the twin lens unit 300 is a VR180 lens. When the camera 100 can determine that the twin-lens unit 300 is a VR180 lens based on information that can be acquired from the twin-lens unit 300, information indicating VR180 image data is recorded in the image data file as metadata. be able to. As a result, the PC 500 can perform the determination of S703 without image processing. The control unit 501 may execute S704 for image data captured using the VR180 lens, and S707 for image data captured using another lens.

Furthermore, if the center coordinates and radius of the circular fisheye images (right and left images) formed by the twin-lens unit 300 can be obtained from the twin-lens unit 300, this information is also recorded in the image data file as metadata. can do. The control unit 501 can skip the processing of S703 and S704 when the metadata acquired in S701 includes information on the center coordinates and radius of the circular fisheye images (right and left images). Therefore, the control unit 501 checks at a timing before executing S703 whether or not the metadata acquired in S701 includes information on the center coordinates and radius of the circular fisheye images (right and left images). You can judge. Then, the control unit 501 executes S703 when it is determined that information about the center coordinates and radius of the circular fisheye images (right image and left image) is not included, and determines that the information is included. If so, S705 is executed.

As described above, according to the present embodiment, when image data is determined to be image data including two circular fisheye images, the type of image data is determined to be VR180 image data. I made it Therefore, even if it is not possible to determine whether the image data is VR180 image data from the metadata of the image data, it is possible to correctly recognize the image type and appropriately process the VR180 image data. Therefore, even if it is not possible to recognize that the lens unit attached to the camera is a VR180 lens, the type of image data recorded by the camera can be correctly recognized and handled appropriately.

● (Second embodiment)
Next, a second embodiment of the invention will be described. The second embodiment relates to an embodiment in which an image signal read out from an imaging device is transmitted in real time as a video signal from a camera 100 equipped with a twin-lens unit 300 to a PC 500 . In the following description, the same reference numerals as in the first embodiment are used for the configurations common to the first embodiment, and the description is omitted.

FIG. 9 is a schematic diagram of an image processing system according to the second embodiment. The image processing system of this embodiment has a configuration in which a camera 100 and a PC 500 are connected by a cable 190, and an image signal read from an imaging unit 211 (image sensor) of the camera 100 is transmitted to the PC 500 in real time. The cable 190 connects the video signal output I/F 240 of the camera 100 and the external I/F 508 of the PC 500, for example. Here, as an example, it is assumed that the image signal is transmitted from the camera 100 to the PC 500 by a transmission method conforming to the HDMI standard. Therefore, the video signal output I/F 240 and the external I/F 508 are interfaces complying with the HDMI standard, and the cable 190 is an HDMI cable. However, video signals may be transmitted according to other standards such as DVI (Digital Visual Interface), SDI, USB Video Class, and the like.

The PC 500 can display the video signal received via the external I/F 508 on the display unit 506 in real time. Also, the PC 500 can convert the received video signal into a streamable format, transmit it to the HMD 191 connected to the PC 500 , or distribute it to a plurality of users 193 via the cloud service 192 .

Next, the video output operation of the camera 100 according to this embodiment will be described using the flowchart of FIG. In FIG. 10, the same reference numerals as in FIG. 7 are given to the steps of performing the same processing as the photographing processing described in the first embodiment, and the description thereof will be omitted. The video output operation is realized by the system control unit 50 executing a program to perform necessary control. The shooting operation shown in FIG. 10 is started when the user turns on the power of the camera 100 by operating the power switch 102, for example. The system control unit 50 puts the camera 100 into a shooting standby state after executing the activation process associated with power-on.

In the shooting standby state, the camera 100 continues to shoot moving images and displays live view on the EVF 217 or the display unit 108 (S601-S602).
In S903, the system control unit 50 determines whether or not the cable 190 is connected to the video signal output I/F 240 based on the voltage of the signal line. The system control unit 50 executes S604 when it is determined that the cable 190 is connected, and executes S601 when it is not determined.

In S604, the system control unit 50 reads the image signal for one frame and causes the image processing unit 214 to generate image data for display. Also, in S605, the system control unit 50 acquires information to be transmitted to the PC 500 as metadata. Also in this embodiment, the system control unit 50 cannot acquire information from the twin lens unit 300 that can determine that the twin lens unit 300 is a VR180 lens.

In S906, in parallel with the operation for live view display in the camera 100, the system control unit 50 stores image data and metadata in a predetermined format in an HDMI signal output buffer prepared as a partial area of the memory 215, for example. Store.

In S<b>907 , the system control unit 50 outputs the image data and metadata stored in the buffer as HDMI signals through the video signal output I/F 240 . Description of details such as encoding processing for converting data into an HDMI signal (TMDS sequence) is omitted.

In S908, the system control unit 50 determines whether or not the cable 190 has been removed from the video signal output I/F 240 based on the voltage of the signal line. If it is determined that the cable 190 has been pulled out, the video output operation is terminated. If it is not determined that the cable 190 has been pulled out, the system control unit 50 repeats the processing from S604. As a result, live view images are continuously displayed on the display unit 506 of the PC 500 .

In this embodiment, the camera 100 is equipped with a twin lens unit 300 that is a VR180 lens. Therefore, the PC 500 receives a video signal of a VR180 image including two circular fisheye images in which the positions of the right image and the left image are reversed (corresponding to the captured image 402b in FIG. 5). output. The signal format of the image data output to the PC 500 may be RGB format, YCbCr format, or RAW format. In the case of the RAW format, the RAW data is stored in the buffer instead of the display image data in S906. Note that the display image data generated by the image processing unit 214 has a resolution that matches the resolution of the display device of the camera 100 . Image data to be transmitted to the PC 500 may have a resolution that takes into consideration the display resolution of the PC 500 .

Next, the video display operation of the PC 500 will be explained using the flowchart of FIG. The following operations are realized in PC 500 by control unit 501 executing an image processing application stored in external storage device 504 . In FIG. 11, steps for performing processing similar to the image data display operation described in the first embodiment are denoted by the same reference numerals as in FIG. 8, and description thereof is omitted. Here, it is assumed that the external I/F 508 and the video signal output I/F 240 of the camera 100 are connected by a cable 190 so that the PC 500 can receive video signals from the camera 100 .

In S<b>1001 , the control unit 501 determines whether or not a video signal has been received through the external I/F 508 . The control unit 501 executes S1002 when it is determined that the video signal has been received, and executes S1001 when it is not determined.

In S1002, the control unit 501 determines whether the reception of the video signal through the external I/F 508 has stopped. The control unit 501 ends the video display operation when it is determined that the reception of the video signal has stopped, and executes S1003 when it is not determined.

In S1003, the control unit 501 decodes the received video signal to acquire metadata.
In S1004, the control unit 501 decodes the received video signal to obtain image data.
Thereafter, the control unit 501 executes the processes after S703 described in the first embodiment.

It should be noted that the user may specify whether or not to exchange the positions of the right image and the left image at an arbitrary timing of S704 to S706. If it is specified not to change the arrangement, the control unit 501 displays the image data acquired in S1004 as it is.

Further, in S706, equirectangular transformation may be applied to the right and left images of the image data in which the positions of the right and left images are interchanged to convert from the mesh format to the equirectangular format before displaying. . The user may be allowed to specify whether to display the mesh format or the equirectangular format.

Even if the metadata does not include information about whether the image data to be processed is VR180 image data, the metadata may include the model name of the lens unit as general shooting information. In this case, in step S1003, the model name of the lens unit is read, and the model name list of the VR180 lens registered in advance is referenced to determine the type of image data to be processed (whether it is image data captured using the VR180 lens). or not) is possible. In this case, the determination in S703 can be performed without image processing. The control unit 501 may execute S704 for image data captured using the VR180 lens, and S707 for image data captured using another lens.

In addition, since the video signal received by the PC 500 in this embodiment corresponds to moving image data, the processing results of S703 and S704 for the first frame can be used for the second and subsequent frames. Therefore, S704 can be omitted for the second and subsequent frames.

In addition, applying equirectangular transformation to the right and left images of the image data in which the positions of the right and left images are exchanged, converting the image data converted from the mesh format to the equirectangular format into a format that can be streamed, It may be supplied externally. For example, the control unit 501 can transmit image data in a streamable format to an HMD connected to the PC 500 or to a distribution server via the communication unit 507 or the external I/F 508 . As a result, the live view image of the camera 100 can be supplied to an external device or distributed to a remote user.

Also, the video signal received by the PC 500 may be directly converted into a streamable format, and the processing corresponding to S703 to S706 may be performed by an external device (distribution server, HMD, etc.).

In this embodiment, it is assumed that the camera 100 cannot acquire information from the twin lens unit 300 indicating that the twin lens unit 300 is a VR180 lens. However, if the camera 100 can determine that the twin-lens unit 300 is a VR180 lens based on information that can be acquired from the twin-lens unit 300, information indicating VR180 image data is included in the video signal as metadata. be able to. As a result, the PC 500 can perform the determination of S703 without image processing. The control unit 501 may execute S704 for image data captured using the VR180 lens, and S707 for image data captured using another lens.

Furthermore, if the center coordinates and radius of the circular fisheye images (right and left images) formed by the twin-lens unit 300 can be obtained from the twin-lens unit 300, this information is also recorded in the image data file as metadata. can do. The control unit 501 can skip the processing of S703 and S704 when the metadata acquired in S1003 includes information on the center coordinates and radius of the circular fisheye images (right and left images). Therefore, at a timing before executing S703, the control unit 501 determines whether or not the metadata acquired in S1003 includes information about the center coordinates and radius of the circular fisheye images (right and left images). You can judge. Then, the control unit 501 executes S703 when it is determined that the information about the center coordinates and the radius of the circular fisheye images (right and left images) is not included, and determines that they are included. If so, S705 is executed.

As described above, according to the present embodiment, when it is determined that the image data transmitted by the video signal transmitted in real time from the camera is image data including two circular areas, the image data is determined to be VR180 image data. Therefore, even if it is not possible to determine whether the image data is VR180 image data from the metadata of the image data, it is possible to appropriately process the VR180 image data. Therefore, even if it is not possible to recognize that the lens unit attached to the camera is a VR180 lens, it is possible to correctly recognize the type of image received from the camera and handle it appropriately.

(Other embodiments)
The operations described using the flowchart of FIG. 8 in the first embodiment need not be executed by the PC 500 and may be executed by the camera 100 . That is, the camera 100 may be the image processing apparatus according to the first embodiment.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The present invention is not limited to the content of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Camera, 102... PC, 501... Control part, 502... ROM, 503... RAM, 504... External storage device, 505... Operation part, 506... Display part, 507... Communication part, 508... External I/F, 509 …system bus

Claims (16)

acquisition means for acquiring image data;
determining means for determining whether an image represented by the image data includes two circular regions;
a processing means for performing a predetermined process on the image when the determination means determines that the image includes two circular regions;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said determination means detects a circular area in each of a right area and a left area of said image. If at least one of a difference in size between two circular regions detected from the image being less than a threshold and a distance between the two circular regions being less than a threshold is satisfied, 3. The image processing apparatus according to claim 1, wherein it is determined that the image includes the two circular areas. 4. The image according to any one of claims 1 to 3, wherein the predetermined process is a process of generating image data in which positions of circular regions included in the image represented by the image data are exchanged. processing equipment. 5. The image processing apparatus according to claim 4, wherein the switching of the positions of the circular regions is a left-to-right switching of the two circular regions. 6. The image processing apparatus according to claim 4, wherein the processing means stores the data of the generated image in association with metadata indicating that it is of a predetermined type. 7. The image processing apparatus according to any one of claims 4 to 6, wherein the processing means further generates an image by applying equirectangular transformation to the image in which the positions of the circular regions are interchanged. 8. The image processing apparatus according to claim 1, wherein said acquisition means acquires said image data from an imaging device in real time. the acquisition means further acquires metadata of the image data;
9. The image processing apparatus according to any one of claims 1 to 8, wherein when said image data is of a predetermined type based on said metadata, said determination means does not perform determination. 10. The image processing apparatus according to any one of claims 1 to 9, wherein said processing means executes said predetermined processing based on an instruction from a user. 11. The image processing apparatus according to claim 1, wherein the circular area included in the image is a circular fisheye image. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus is an imaging apparatus that generates the image data by imaging processing. 13. The image processing apparatus according to claim 12, wherein the image processing apparatus is an interchangeable lens imaging apparatus. 14. The determining means determines that the image includes two circular areas when it is determined that two at least partial circular areas are included. 1. The image processing apparatus according to claim 1. An image processing method executed by an image processing device,
obtaining image data;
determining whether the image represented by the image data includes two circular regions;
if the image is determined to contain two circular regions, performing a predetermined process on the image;
An image processing method characterized by comprising: A program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 14.

Images