JP2020016883A - Image capture device - Google Patents

Abstract

To provide an image capture device capable of capturing a wide-range image which does not include a specific subject such as a photographer yet does not give a feeling that something is wrong.SOLUTION: An image capture device according to the present invention includes: photographing means for acquiring multiple images by photographing multiple times, where acquisition of a first image is controlled such that a specific subject is included in a visual field range other than a first visual field range in the first image, and acquisition of a second image including the other visual field range is controlled such that the specific subject is not included in the other visual field range; and creation means for creating a composite image that has a 360° visual field range around a prescribed axis by compositing the first visual field range in the first image and a second visual field range, which includes the other visual field range and is different than the first visual field range, in the second image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging device.

  2. Description of the Related Art In recent years, imaging devices capable of capturing images in a wider range than a human viewing angle, such as omnidirectional images and omnidirectional images, have become widespread. 2. Description of the Related Art An act of publicizing an image (for example, a spherical image) shot by such an imaging device to an unspecified number of people through a social networking service on the Internet is widely performed.

  When publishing an image to an unspecified number of people, it is desired to take privacy measures as necessary so that information that can identify an individual is not included in the image. However, since the omnidirectional imaging device (imaging device for capturing an omnidirectional image) has almost no blind spot, the photographer is reflected in the captured omnidirectional image. Even if the image is not a spherical image, a similar problem occurs if the image is a wide-area image (image of a wide range).

  Patent Literature 1 discloses performing concealment processing (blur and fill) on a photographer area. Patent Document 2 discloses that an image of a support (including a photographer) supporting an imaging device is determined to be in a predetermined range, and if the image is in a predetermined range, the range is determined by a predetermined texture. Filling is disclosed.

JP 2013-198062 A JP-A-2006-103806

  However, when the techniques described in Patent Literatures 1 and 2 are used, it is possible to make it impossible to determine who the photographer is, but it is possible to use a blurred area or a predetermined color or texture. An area or the like remains in the image.

  SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus capable of capturing a wide-range image in which a specific subject such as a photographer is not reflected and has no uncomfortable feeling.

A first aspect of the present invention provides:
A means for acquiring a plurality of images by photographing a plurality of times, wherein when acquiring a first image, a specific subject is included in a field of view other than the first field of view in the first image. Control to acquire the first image so as to obtain the second image including the other visual field range, so that the specific subject is not included in the other visual field range. Photographing means for performing control for acquiring the second image;
Performing a process of combining the first visual field range in the first image with a second visual field range in the second image that includes the other visual field range and is different from the first visual field range. Generating means for generating a composite image having a 360 ° field of view around a predetermined axis,
An imaging apparatus characterized by having:

A second aspect of the present invention provides:
A step of acquiring a plurality of images by photographing a plurality of times, wherein when acquiring a first image, a specific subject is included in a field of view other than the first field of view in the first image. Control to acquire the first image so as to obtain the second image including the other visual field range, so that the specific subject is not included in the other visual field range. Performing control for acquiring the second image;
Performing a process of combining the first visual field range in the first image and a second visual field range in the second image that includes the other visual field range and is different from the first visual field range. Generating a composite image having a 360 ° field of view around a predetermined axis,
And a control method of the imaging device.

  A third aspect of the present invention is a program for causing a computer to function as each unit of the above-described imaging device. A fourth aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as each unit of the above-described imaging apparatus.

  ADVANTAGE OF THE INVENTION According to this invention, a specific subject like a photographer is not image | photographed, and a wide range image without a sense of incongruity can be image | photographed.

1A and 1B are an external view and a block diagram of a digital camera. 6 is a flowchart of a photographer removal mode 1. 9 is a flowchart of a photographer removal mode 2. FIG. 3 is a diagram illustrating a moving direction of the digital camera. It is a figure showing a situation (method) of photography.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. FIG. 1A is a front perspective view (external view) of the digital camera 100 (imaging device). FIG. 1B is a rear perspective view (external view) of the digital camera 100. The digital camera 100 is a camera (omnidirectional camera; omnidirectional camera) for capturing an omnidirectional image (omnidirectional image).

  The barrier 102a is a protection window for a front camera unit that has a shooting range in front of the digital camera 100. The front camera unit is, for example, a wide-angle camera unit that captures a wide range of 180 degrees or more on the front side of the digital camera 100 up, down, left, and right. The barrier 102b is a protection window for a rear camera unit that covers the back of the digital camera 100 as a shooting range. The rear camera unit is, for example, a wide-angle camera unit that captures a wide range of 180 degrees or more on the rear side of the digital camera 100 in all directions.

  The display unit 28 displays images and various information. The shutter button 61 is an operation unit (operation member) for giving a shooting instruction. The mode switch 60 is an operation unit for switching between various modes. The connection I / F 25 is a connector for connecting a connection cable to the digital camera 100, and an external device such as a smartphone, a personal computer, and a television device is connected to the digital camera 100 using the connection cable. The operation unit 70 includes various switches, buttons, dials, and touch sensors that receive various operations from the user. The power switch 72 is a push button for turning on / off the power. The user can operate the digital camera 100 using the external device wirelessly connected as a remote controller.

The light-emitting units (light-emitting members) 21a and 21b are light-emitting diodes (LEDs) and the like, and notify the user of various states of the digital camera 100 by a light-emitting pattern and a light-emitting color. The light emitting unit 21a is provided on the front surface of the digital camera 100, and the light emitting unit 21b is provided on the back surface of the digital camera 100. The fixing unit 40 is, for example, a tripod screw hole, and is used to fix and install the digital camera 100 with a fixing device such as a tripod.

  FIG. 1C is a block diagram illustrating a configuration example of the digital camera 100.

  The barrier 102a covers the imaging system (the imaging lens 103a, the shutter 101a, the imaging unit 22a, and the like) of the front camera unit, thereby preventing the imaging system from being stained or damaged. The photographing lens 103a is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101a is a shutter having a diaphragm function for adjusting the amount of subject light incident on the imaging unit 22a. The imaging unit 22a is an imaging device configured by a CCD, a CMOS device, or the like that converts an optical image into an electric signal. The A / D converter 23a converts an analog signal output from the imaging unit 22a into a digital signal. Note that, without providing the barrier 102a, the outer surface of the photographing lens 103a may be exposed, and the photographing lens 103a may prevent other imaging systems (the shutter 101a and the imaging unit 22a) from being stained or damaged.

  The barrier 102b covers the imaging system (the imaging lens 103b, the shutter 101b, the imaging unit 22b, and the like) of the rear camera unit, thereby preventing the imaging system from being stained or damaged. The photographing lens 103b is a lens group including a zoom lens and a focus lens, and is a wide-angle lens. The shutter 101b is a shutter having a diaphragm function for adjusting the amount of incident light of the subject on the imaging unit 22b. The imaging unit 22b is an imaging device configured to convert an optical image into an electric signal, such as a CCD or a CMOS device. The A / D converter 23b converts an analog signal output from the imaging unit 22b into a digital signal. Note that, without providing the barrier 102b, the outer surface of the imaging lens 103b may be exposed, and the imaging lens 103b may be used to prevent other imaging systems (the shutter 101b and the imaging unit 22b) from being stained or damaged.

  A VR (Virtual Reality) image is captured by the imaging unit 22a and the imaging unit 22b. The VR image is an image that can be displayed in VR. The VR image includes an omnidirectional image (omnidirectional image) captured by an omnidirectional camera (omnidirectional camera) and a panoramic image having a wider image area (effective image area) than a display area that can be displayed on the display unit at one time. And so on. The VR image includes not only a still image but also a moving image or a live view image (an image obtained almost in real time from a camera). The VR image has a maximum image range (effective image range) of 360 degrees in the vertical direction (vertical angle, angle from the zenith, elevation angle, depression angle, altitude angle) and 360 degrees in the horizontal direction (horizontal angle, azimuth angle). Have.

  Further, even if the VR image is less than 360 degrees vertically and less than 360 degrees left and right, it can be displayed on the display unit at a wide angle of view (field of view) wider than the angle of view that can be photographed by a normal camera. An image having an image range (effective image range) wider than the display range is also included. For example, an image captured by a celestial sphere camera capable of capturing an object in a field of view (equivalent to an angle of view) of 360 degrees in the horizontal direction (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the Zenith is This is a kind of VR image. For example, an image photographed by a camera capable of photographing a subject in a horizontal direction (horizontal angle, azimuth angle) of 180 degrees and a vertical angle of 180 degrees centering on the horizontal direction (field angle) is a VR image. It is a kind. That is, an image having an image range of a visual field of 160 degrees (± 80 degrees) or more in each of the vertical direction and the horizontal direction, and having an image range wider than a range that can be visually recognized at a time by a human being is a VR image. Is a kind of

When this VR image is displayed in VR (display mode: “VR view”), the posture of the display device (display device displaying the VR image) is changed in the left-right rotation direction, so that the VR image is displayed in the left-right direction (horizontal rotation direction). Can watch seamless omnidirectional video. In the vertical direction (vertical rotation direction), a seamless omnidirectional video can be viewed within a range of ± 105 degrees when viewed from directly above (zenith), but an image exceeding 105 degrees from directly above cannot be viewed. The blank area does not exist. The VR image can also be said to be “an image whose video range is at least a part of the virtual space (VR space)”.

  The VR display (VR view) is a display method (display mode in which a display range can be changed by displaying an image of a partial view range (display range; display area) of a VR image according to the orientation of the display device. ). When viewing by mounting a head-mounted display (HMD) as a display device, an image in a visual field range according to the direction of the user's face is displayed. For example, in the VR image, at a certain point in time, an image having a viewing angle (angle of view) centered at 0 degree in the horizontal direction (a specific direction, for example, north) and 90 degrees in the vertical direction (90 degrees from the zenith, that is, horizontal). Is displayed. From this state, when the orientation of the display device is reversed (for example, when the display surface is changed from south to north), the same VR image is shifted 180 degrees in the left and right direction (reverse orientation, for example, south), and up and down. The display range is changed to an image having a viewing angle centered at 90 degrees (horizontal) in the direction. If the user is watching the HMD, if the user turns his face from north to south (that is, if he turns his back), the image displayed on the HMD will change from the north image to the south image. is there. With such VR display, it is possible to provide the user with a sense of being visually as if they were at the place in the VR image (in the VR space). A smartphone mounted on VR goggles (head-mounted adapter) can be said to be a type of HMD.

  The display method of the VR image is not limited to the above. The display range may be moved (scrolled) according to a user operation on the touch panel, the direction button, or the like, instead of the posture change. At the time of VR display (in the VR view mode), a process of changing the display range according to a change in posture and a process of changing the display range according to a touch move operation on a touch panel or a drag operation on an operation member such as a mouse. Both may be performed.

  The image processing unit 24 performs predetermined processing (resize processing such as pixel interpolation and reduction, color conversion processing, and the like) on data from the A / D converter 23 a and the A / D converter 23 b or data from the memory control unit 15. )I do. Further, the image processing unit 24 performs a predetermined calculation process using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation results obtained by the image processing unit 24. Thus, TTL (through the lens) type AF (autofocus) processing, AE (automatic exposure) processing, EF (flash pre-emission) processing, and the like are performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained arithmetic result. Further, the image processing unit 24 performs basic image processing on two images (fisheye images) obtained by the A / D converter 23a and the A / D converter 23b, and performs basic image processing. A single VR image is generated by performing a spliced image process of combining the two images. The image processing unit 24 performs image cutout processing, enlargement processing, distortion correction, and the like for VR display of a VR image during VR display or playback in live view, and performs processing in a predetermined storage area in the memory 32. Perform rendering to draw the result.

  In the spliced image processing, the image processing unit 24 calculates a shift amount between the reference image and the comparison image for each area by pattern matching using one of the two images as the reference image and the other as the comparison image, and Based on the shift amount, a connection position connecting the two images is detected. Then, the image processing unit 24 corrects the distortion of each image by geometrical transformation in consideration of the detected connection position and the lens characteristics of each optical system. As a result, each image is converted into a spherical image. Then, the image processing unit 24 generates one celestial sphere image (VR image) by combining (blend) the two celestial sphere images. The generated celestial sphere image is an image using, for example, the equirectangular projection, and the position of each pixel of the celestial sphere image can be associated with the coordinates of the surface of the sphere (VR space).

  Output data from the A / D converters 23a and 23b is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15. The memory 32 stores image data obtained by the imaging units 22a and 22b and converted into digital data by the A / D converters 23a and 23b, and image data to be output from the connection I / F 25 to an external display device. . The memory 32 has a sufficient storage capacity for storing a predetermined number of still images, moving images and audio for a predetermined time.

  The memory 32 also serves as an image display memory (video memory). The image display data stored in the memory 32 can be output from the connection I / F 25 to an external display device. By sequentially transferring and displaying VR images, which are VR images captured by the imaging units 22a and 22b and generated by the image processing unit 24 and stored in the memory 32, to an external display device, an electronic viewfinder is provided. Function can be realized, and live view display (LV display) can be realized. Hereinafter, an image displayed in the live view display is referred to as a live view image (LV image). Similarly, live view display (remote LV display) can also be realized by sequentially transferring and displaying VR images stored in the memory 32 to an external device (such as a smartphone) wirelessly connected via the communication unit 54.

  The nonvolatile memory 56 is a memory as a recording medium that can be electrically erased and recorded, and is, for example, an EEPROM. The non-volatile memory 56 stores constants, programs, and the like for the operation of the system control unit 50. The program referred to here is a computer program for executing each flowchart of each embodiment described later.

  The system control unit 50 is a control unit having at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 realizes each process of each embodiment described later by executing a program recorded in the above-described nonvolatile memory 56. The system memory 52 is, for example, a RAM, and the system control unit 50 develops, in the system memory 52, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like. The system control unit 50 also performs display control by controlling the memory 32, the image processing unit 24, the memory control unit 15, and the like. The system timer 53 is a time measuring unit that measures the time used for various controls and the time of a built-in clock.

  The mode changeover switch 60, the shutter button 61, the operation unit 70, and the power switch 72 are used to input various operation instructions to the system control unit 50.

  The mode changeover switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image shooting mode, a playback mode, a communication connection mode, a photographer removal mode 1, a photographer removal mode 2, and the like. The modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes, custom modes, and the like, which are shooting settings for each shooting scene. The mode switch 60 allows the user to directly switch to any of these modes. Alternatively, after temporarily switching to the photographing mode list screen with the mode switch 60, the mode may be selectively switched to any one of the plurality of modes displayed on the display unit 28 using another operation member. Similarly, the moving image shooting mode may include a plurality of modes.

The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on by the so-called half-press (instruction for shooting preparation) during the operation of the shutter button 61, and generates a first shutter switch signal SW1. The system control unit 50 starts a shooting preparation operation such as an AF (auto focus) process, an AE (auto exposure) process, an AWB (auto white balance) process, and an EF (flash pre-flash) process in response to the first shutter switch signal SW1. I do. The second shutter switch 64 is turned on when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (photographing instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts an operation of a series of photographing processes from reading signals from the imaging units 22a and 22b to writing image data to the recording medium 200 by the second shutter switch signal SW2.

  The shutter button 61 is not limited to an operation member capable of performing two-stage operation of full-press and half-press, but may be an operation member capable of performing only one-stage press. In that case, the photographing preparation operation and the photographing process are continuously performed by pressing down one step. This is the same operation as when the shutter button that can be half-pressed and fully pressed is fully pressed (when the first shutter switch signal SW1 and the second shutter switch signal SW2 are generated almost simultaneously).

  The operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons and options displayed on the display unit 28, and functions as various function buttons. The function buttons include, for example, an end button, a return button, an image forward button, a jump button, a refinement button, an attribute change button, an INFO button, and the like. For example, when a menu button is pressed, a menu screen on which various settings can be made is displayed on the display unit 28. The user can intuitively perform various settings by operating the operation unit 70 while viewing the menu screen displayed on the display unit 28.

  The power switch 72 is a push button for turning on / off the power. The power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching a block to be energized, and the like, and detects whether or not a battery is mounted, the type of the battery, and the remaining battery level. Further, the power control unit 80 controls the DC-DC converter based on the detection result and the instruction of the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period. The power supply unit 30 includes a primary battery such as an alkaline battery and a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, and an AC adapter.

  The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording a captured image, and includes a semiconductor memory, an optical disk, a magnetic disk, and the like. The recording medium 200 may be a replacement recording medium that is detachable from the digital camera 100 or a recording medium built in the digital camera 100.

  The communication unit 54 transmits and receives a video signal, an audio signal, and the like to and from an external device connected by a wireless or wired cable. The communication unit 54 can be connected to a wireless LAN (Local Area Network) or the Internet, and can also communicate with an external device (such as a server) on the network via the network. The communication unit 54 can also communicate with an external device using Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging units 22a and 22b and images recorded on the recording medium 200, and can receive images and other various information from an external device. .

The posture detection unit 55 detects the posture of the digital camera 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, the image photographed by the imaging units 22a and 22b is an image photographed with the digital camera 100 held horizontally or an image photographed with the digital camera 100 held vertically. It is possible to determine whether there is. In addition, it is possible to determine whether the image captured by the imaging units 22a and 22b is an image captured by tilting the digital camera 100 in a rotation direction such as a yaw direction, a pitch direction, and a roll direction. It can be determined. The system control unit 50 adds orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the VR image captured by the imaging units 22a and 22b, and rotates the image (images to correct the tilt). Or the like). As the posture detecting unit 55, a combination of one or more of a plurality of sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, and an altitude sensor can be used. Using the acceleration sensor, gyro sensor, azimuth sensor, and the like included in the posture detection unit 55, the movement of the digital camera 100 (pan, tilt, lift, whether or not the digital camera 100 is stationary, etc.) can be detected.

  The microphone 20 collects sound around the digital camera 100 recorded as sound of a VR image (VR moving image) that is a moving image. The connection I / F 25 is a connection plug to which an HDMI (registered trademark) cable, a USB cable, and the like are connected in order to perform transmission and reception of video by connecting to an external device.

[First Embodiment]
A first embodiment of the present invention will be described. FIG. 2 is a flowchart according to the first embodiment, and is a flowchart relating to a omnidirectional image photographing process performed by the digital camera 100. This processing is realized by expanding the program recorded in the nonvolatile memory 56 into the system memory 52 and executing the program by the system control unit 50. When the power of the digital camera 100 is turned on and the photographer removal mode 1 is selected by operating the mode changeover switch 60, the processing in FIG. 2 is started.

  In step S201, the system control unit 50 determines whether or not a shooting instruction (full press of the shutter button 61) has been issued. If it is determined that a shooting instruction has been issued, the process proceeds to step S207; otherwise, the process proceeds to step S202. In step S202, system control unit 50 determines whether or not an operation for changing the setting (setting change operation) has been performed. For example, it is determined whether an operation for changing the AF method, the AE method, the white balance method, or the like has been performed. If it is determined that the setting change operation has been performed, the process proceeds to step S203; otherwise, the process proceeds to step S204. In step S203, the system control unit 50 changes the setting by reflecting the setting corresponding to the setting change operation in the system memory 52 (setting change processing). In step S204, system control unit 50 determines whether or not an operation for changing the operation mode (mode change operation; operation of mode switch 60) has been performed. If it is determined that the mode change operation has been performed, the process proceeds to step S205; otherwise, the process proceeds to step S206. In step S205, the system control unit 50 changes the operation mode from the photographer removal mode 1 to another operation mode corresponding to the mode change operation. Then, the processing in FIG. 2 ends. In step S206, system control unit 50 determines whether an end operation such as pressing power switch 72 has been performed. If it is determined that the end operation has been performed, the processing in FIG. 2 ends, and if not, the flow returns to step S201.

  In the present embodiment, two omnidirectional images are obtained by two shootings, and are combined. For example, in each shooting, a spherical image of a specific subject such as a photographer is obtained. The number of times of shooting, the number of shot images (images obtained by shooting), and the like are not particularly limited. More than two shots may be performed, and more than two shot images may be obtained. Further, the captured image may be a wide-range image (a wide-range image; a VR image) other than the spherical image. For example, the captured image may have a field of view of 360 ° around a predetermined axis and have a field of view of less than 360 ° around another axis. The predetermined axis is, for example, a yaw axis (vertical axis).

  If it is determined in step S201 that a shooting instruction has been issued, in step S207, the system control unit 50 shoots the first omnidirectional image and stores (records) the omnidirectional image in the memory 32. ).

  In step S208, the system control unit 50 detects the posture of the digital camera 100 when the first omnidirectional image was captured using the posture detection unit 55, and outputs information on the posture (posture information) to the system. The data is recorded in the memory 52.

  In step S209, the system control unit 50 performs control to notify a plurality of times (two times) of assisting shooting (notification control). Specifically, the system control unit 50 causes the light emitting unit 21a to emit light in one of a plurality of predetermined light emitting patterns.

In the present embodiment, the system control unit 50 determines any of the following states 1 to 3 as the current state.
State 1: The photographer has not moved from the standing position where the first omnidirectional image has been photographed to the opposite side across the digital camera 100. State 2: The posture of the digital camera 100 is not suitable for synthesis. State 3: The photographer moves from the standing position where the first omnidirectional image was taken to the opposite side with the digital camera 100 interposed therebetween, and the posture of the digital camera 100 becomes a posture suitable for synthesis. State

  Then, in the case of the state 1, the system control unit 50 causes the light emitting unit 21a to emit light in the first light emitting pattern. Light emission in the first light emission pattern is, for example, lighting in red. In the case of the state 2, the system control unit 50 causes the light emitting unit 21a to emit light in the second light emitting pattern. Light emission in the second light emission pattern is, for example, blinking in red. In state 3, the system control unit 50 causes the light emitting unit 21a to emit light in the third light emitting pattern. Light emission in the third light emission pattern is, for example, lighting in green. The light emission pattern is not particularly limited as long as states 1 to 3 can be distinguished.

  Thus, the user is notified of the posture of the digital camera 100 so that photographing can be appropriately performed a plurality of times (posture guide).

  States 1 to 3 will be described with reference to FIGS. For example, the photographer moves from position P1 to position P2 after photographing the first omnidirectional image at position P1 in FIG. 5A. At this time, the photographer turns the digital camera 100 only in the yaw direction shown in FIG. 4, and rotates the state (posture) of the digital camera 100 by about 180 degrees in the yaw direction with respect to the state at the time of the first photographing. State. In this state, the photographer at the position P2 captures a second omnidirectional image.

  In the first omnidirectional image, the photographer is not shown in the region (viewing range) 501, but the photographer is shown in the region (viewing range) 502 other than the view range. In the second omnidirectional image, the photographer is not shown in the area 502, but the photographer is shown in the area 501. The system control unit 50 performs at least a process of synthesizing the region 501 of the first spherical image and the region 502 of the second spherical image to obtain a spherical image in which the photographer does not appear. Is generated as a composite image. In this manner, in the present embodiment, it is possible to obtain (capture) a celestial sphere image in which a specific subject such as a photographer is not reflected and which does not cause a sense of discomfort. Note that the composite image may be a wide-range image (a wide-range image; a VR image) other than the spherical image. For example, the composite image may have a 360 ° field of view around a given axis (such as the yaw axis) and less than 360 ° around another axis. In the present embodiment, each of the regions 501 and 502 is a predetermined region. Note that the regions 501 and 502 may be appropriately changed by the digital camera 100.

Here, between the first shooting and the second shooting, a digital camera in a direction other than the yaw direction, that is, a predetermined amount or more in the x direction, y direction, z direction, roll direction, pitch direction, etc. shown in FIG. If 100 moves, it is not possible to properly combine the two spherical images. If the rotation in the yaw direction is small, it is not possible to capture a spherical image in which the photographer is not shown in the area 502. If the digital camera 100 is rotated about 180 degrees in the yaw direction, it is possible to capture a spherical image in which the photographer in the area 502 is not captured.

Therefore, the above-mentioned states 1 to 3 can be said to be the following states 1 'to 3'.
State 1 ': a state in which the digital camera 100 has not moved in a direction other than the yaw direction by a predetermined amount or more, but the rotation angle in the yaw direction is not about 180 degrees. State 2': a state in which the digital camera 100 has a predetermined amount in a direction other than the yaw direction. State 3 ′: state in which the digital camera 100 has not moved a predetermined amount or more in a direction other than the yaw direction, and the rotation angle in the yaw direction is about 180 degrees

  By changing the notification (light emission pattern of the light emitting unit 21a) between the states 1 ′ to 3 ′, the posture of the digital camera 100 is preferably adjusted so that the photographer does not appear in the area 502 after the first shooting. It can notify whether or not it has been changed.

  The state in step S209 (immediately after the first photographing) is state 1 (state 1 '). Therefore, in step S209, the system control unit 50 performs a process of turning on the light emitting unit 21a in red as a process of the posture guide.

  In step S210, system control unit 50 determines which of the current states is states 1 to 3 (states 1 'to 3') (state determination; posture determination). Specifically, the system control unit 50 detects (acquires) the posture information of the digital camera 100 using the posture detection unit 55. Then, based on the current posture information and the posture information recorded in step S208 (posture information at the time of the first photographing), the system control unit 50 changes the current state to states 1 to 3 (states 1 'to 3). ').

  In step S211, the system control unit 50 determines whether the current posture of the digital camera 100 is a posture that can be combined based on the result of the determination in step S210. The determination as to whether or not the posture can be combined is a determination as to whether or not the posture is suitable for combination, and is a determination as to whether or not the posture has not moved by a predetermined amount or more in a direction other than the yaw direction. If it is determined that the postures can be combined, the process proceeds to step S212; otherwise, the process proceeds to step S213.

  In step S212, system control unit 50 determines whether or not the rotation angle of digital camera 100 in the yaw direction is appropriate based on the determination result in step S210. In the present embodiment, it is determined whether or not the rotation angle from the first shooting is about 180 degrees. If it is determined that the rotation angle is appropriate, the process proceeds to step S220; otherwise, the process proceeds to step S214.

  In step S213, the posture of the digital camera 100 is not a posture that can be combined. That is, the current state is state 2 (state 2 '). Therefore, in step S213, the system control unit 50 performs a process of blinking the light emitting unit 21a in red as the process of the posture guide. Then, the process proceeds to step S215.

  In step S214, the posture of the digital camera 100 is a posture that can be combined, but the rotation angle in the yaw direction is not about 180 degrees. That is, the current state is state 1 (state 1 '). Therefore, in step S214, the system control unit 50 performs a process of turning on the light emitting unit 21a in red as a process of the posture guide. Then, the process proceeds to step S215.

  In step S215, system control unit 50 determines whether or not predetermined time T1 has elapsed since the first shooting, that is, whether or not time-out has occurred. The predetermined time T1 is not particularly limited, but is set to 10 seconds in the present embodiment. When it is determined that the predetermined time T1 has elapsed, that is, when it is determined that the timeout has occurred, the process proceeds to step S216, otherwise, the process proceeds to step S218.

  In step S216, system control unit 50 notifies the user that the imaging has failed. In the present embodiment, the user is notified that the photographing has failed by causing the light emitting unit 21a to emit light so that the light emitting color of the light emitting unit 21a alternately switches between green and red. In addition, the method of realizing various notifications is not particularly limited. For example, the display of the display unit 28 may notify the user that the imaging has failed. In step S217, system control unit 50 initializes the shooting state (cancellation of shooting a plurality of times). Specifically, the system control unit 50 clears (deletes) the first omnidirectional image stored in the memory 32. Then, the process returns to step S201.

  In step S218, system control unit 50 determines whether or not a shooting instruction (full press of shutter button 61) has been issued. If it is determined that a shooting instruction has been issued, the process proceeds to step S219; otherwise, the process returns to step S210. In step S219, system control unit 50 notifies the user that shooting cannot be performed. In the present embodiment, the user is notified that photographing cannot be performed by blinking the light emitting unit 21a in red at a cycle earlier than the blink cycle used in the posture guide. Then, the process returns to step S210.

  In step S220, the posture of the digital camera 100 is a posture that can be combined, and the rotation angle in the yaw direction is about 180 degrees. That is, the current state is state 3 (state 3 '). Therefore, in step S220, the system control unit 50 performs a process of turning on the light emitting unit 21a in green as a process of the posture guide.

  In step S221, system control unit 50 determines whether or not a shooting instruction (full press of shutter button 61) has been issued. If it is determined that a shooting instruction has been issued, the process proceeds to step S223; otherwise, the process proceeds to step S222. In step S222, system control unit 50 determines whether or not predetermined time T2 has elapsed since state 3 (state 3 ') was realized. The predetermined time T2 is not particularly limited, but is 3 seconds in the present embodiment. If it is determined that the predetermined time T2 has elapsed, the process proceeds to step S223; otherwise, the process returns to step S210. That is, even if the photographing instruction is not issued, the process of step S223 (the second photographing) is automatically performed when the predetermined time T2 elapses after the state 3 (state 3 ') is realized.

  In step S223, system control unit 50 captures a second omnidirectional image. In step S224, system control unit 50 determines whether or not the first omnidirectional image and the second omnidirectional image stored in memory 32 can be combined. If it is determined that the images can be combined, the process proceeds to step S225; otherwise, the process proceeds to step S227. In the present embodiment, the image processing unit 24 uses one of the two omnidirectional images as a reference image and the other as a comparison image to calculate the amount of deviation between the reference image and the comparison image for each area by pattern matching processing. Then, the image processing unit 24 detects a connection position connecting the area of the reference image and the area of the comparison image based on the shift amount for each area. After that, the image processing unit 24 connects the area of the reference image and the area of the comparison image based on the detected connection position, and generates an omnidirectional image in which the photographer does not appear as a composite image. In step S223, for example, when the connection position cannot be detected, it is determined that the combination is not possible.

  In step S225, the system control unit 50 combines the first omnidirectional image and the second omnidirectional image by using the image processing unit 24, and A sphere image is generated as a composite image. In step S226, the system control unit 50 saves (records) the composite image (omnidirectional image) generated in step S225 in the recording medium 200. Then, the process returns to step S201.

  In step S227, system control unit 50 notifies the user that synthesis cannot be performed. In the present embodiment, the notification that the composition cannot be performed is the same as the notification that the image capturing cannot be performed (step S219; red blinking in an early cycle). Note that the notification that composition cannot be performed may be different from the notification that imaging cannot be performed. In step S228, the system control unit 50 clears (deletes) the second spherical image. Then, the process returns to step S210.

  As described above, in the present embodiment, the omnidirectional image is generated by synthesizing the images obtained by a plurality of shootings. At the time of shooting, the state of the digital camera is set such that, among the images obtained by each shooting, the shooting can be performed such that the photographer is not included in an area used to generate a spherical image. Determine whether it is in the state. By notifying the photographer of the appropriate state, the photographer can easily perform photographing for generating a spherical image that does not include the photographer.

[Second embodiment]
A second embodiment of the present invention will be described. Hereinafter, processing different from the first embodiment will be described in detail, and description of the same processing as in the first embodiment will be appropriately omitted.

  FIG. 5B illustrates an example of a shooting state according to the second embodiment. In the example of FIG. 5B, the photographer moves from position P1 to position P3 after photographing the first omnidirectional image at position P1. At this time, the photographer turns the digital camera 100 only in the yaw direction shown in FIG. In this state, the photographer at the position P3 captures the second spherical image.

  In the first omnidirectional image, the photographer is not shown in the region (viewing range) 503, but the photographer is shown in the region (viewing range) 504. In the second omnidirectional image, the photographer is not shown in the area 504, but the photographer is shown in the area 503. The system control unit 50 performs at least a process of combining the area 503 of the first omnidirectional image and the area 504 of the second omnidirectional image to obtain a omnidirectional image in which the photographer is not captured. Is generated as a composite image.

  In the first embodiment, the rotation angle of the digital camera 100 in the yaw direction (the amount of movement of the photographer in the yaw direction) is set to about 180 degrees (FIG. 5A). However, even if the rotation angle is smaller than 180 degrees, it is possible to prevent the photographer from appearing in the region where the photographer appears in the first omnidirectional image. Therefore, in FIG. 5B, the digital camera 100 is rotated in the yaw direction at a rotation angle smaller than 180 degrees (about 60 degrees), and the photographer moves accordingly. Note that the rotation angle may or may not be predetermined. For example, the system control unit 50 may detect an area in which the photographer is captured in the first omnidirectional image and determine a necessary rotation angle based on the detection result.

  Although the imaging unit used is not particularly limited, in the first embodiment, a final spherical image is generated by combining two images captured by the imaging unit 22b. On the other hand, in the present embodiment, the image obtained by the imaging unit 22a in the first shooting, the image obtained by the imaging unit 22b in the first shooting, and the image obtained by the imaging unit 22a in the second shooting The images are combined to generate a final spherical image.

[Third Embodiment]
A third embodiment of the present invention will be described. FIG. 3 is a flowchart according to the third embodiment, and is a flowchart relating to a omnidirectional image photographing process performed by the digital camera 100. This processing is realized by expanding the program recorded in the nonvolatile memory 56 into the system memory 52 and executing the program by the system control unit 50. When the power of the digital camera 100 is turned on and the photographer removal mode 2 is selected by operating the mode switch 60, the processing in FIG. 3 is started. In the present embodiment, the digital camera 100 is operated using a smartphone wirelessly connected via the communication unit 54 as a remote controller of the digital camera 100. Note that the digital camera 100 may be directly operated.

  In step S301, the system control unit 50 causes the light emitting unit 21a (front LED) to emit light in a predetermined light emitting pattern 1, and causes the light emitting unit 21b (back LED) to emit light in a predetermined light emitting pattern 2 (different from the light emitting pattern 1). Flash. Although the light emission patterns 1 and 2 are not particularly limited, in the present embodiment, the light emission pattern 1 is turned on in green and the light emission pattern 2 is turned on and off in green.

  In step S302, the system control unit 50 controls the communication unit 54 to display the shooting guide on the display surface of the smartphone used as the remote controller. The shooting guide is a guide (notification) that assists shooting, and is a screen on which a message such as “Please give a shooting instruction at a position where the same pattern of light can be seen” is described. When the digital camera 100 is directly operated, a shooting guide is displayed on the display unit 28.

  In step S303, system control unit 50 controls communication unit 54 to display the installation guide on the display surface of the smartphone. The installation guide is a guide (notification) that prompts the user to fix the digital camera 100 to a tripod, and is a screen on which a message such as “Please install the digital camera on a tripod” is described. When the digital camera 100 is directly operated, an installation guide is displayed on the display unit 28.

  In step S304, system control unit 50 uses communication unit 54 to determine whether a shooting instruction (shooting operation) has been issued to the smartphone. When the digital camera 100 is directly operated, the determination in step S304 is to determine whether or not the shutter button 61 has been fully pressed. If it is determined that a shooting instruction has been issued, the process proceeds to step S310; otherwise, the process proceeds to step S305. In step S305, system control unit 50 determines whether a setting change operation has been performed. The setting change operation may be performed on the smartphone or the digital camera 100. If it is determined that the setting change operation has been performed, the process proceeds to step S306; otherwise, the process proceeds to step S307. In step S306, system control unit 50 changes the setting by reflecting the setting according to the setting change operation in system memory 52 (setting change processing). In step S307, system control unit 50 determines whether or not a mode change operation has been performed. The mode change operation may be performed on the smartphone or the digital camera 100. If it is determined that the mode change operation has been performed, the process proceeds to step S308; otherwise, the process proceeds to step S309. In step S308, system control unit 50 changes the operation mode from photographer removal mode 2 to another operation mode according to the mode change operation. Then, the processing in FIG. 3 ends. In step S309, system control unit 50 determines whether an end operation such as pressing power switch 72 has been performed. If it is determined that the end operation has been performed, the process of FIG. 3 ends, and if not, the process returns to step S304.

  If it is determined in step S304 that a shooting instruction has been issued, in step S310, the system control unit 50 shoots the first omnidirectional image and stores (records) the omnidirectional image in the memory 32. ).

  In step S311, the system control unit 50 switches the light emitting pattern of the light emitting unit 21a and the light emitting pattern of the light emitting unit 21b. That is, the system control unit 50 switches the light emitting pattern of the light emitting unit 21a from the light emitting pattern 1 to the light emitting pattern 2, and switches the light emitting pattern of the light emitting unit 21b from the light emitting pattern 2 to the light emitting pattern 1. By exchanging the light emission patterns, the photographer moves to the rear side of the digital camera 100 based on the photographing guide “Please give a photographing instruction at a position where the same pattern of light can be seen”.

  In step S312, system control unit 50 determines whether or not a shooting instruction has been issued. If it is determined that a shooting instruction has been issued, the process proceeds to step S313; otherwise, the process proceeds to step S319.

  In step S319, system control unit 50 determines whether or not a predetermined time T1 has elapsed since the first shooting, that is, whether or not a timeout has occurred. In the present embodiment, the predetermined time T1 is set to 10 seconds. If it is determined that the predetermined time T1 has elapsed, that is, if it has been timed out, the process proceeds to step S320; otherwise, the process returns to step S312.

  In step S320, system control unit 50 notifies the user that the imaging has failed. For example, by causing the light emitting units 21a and 21b to emit light such that the light emission colors of the light emitting units 21a and 21b are alternately switched to green and red, the user is notified that the photographing has failed. Alternatively, by displaying a predetermined screen on the display surface of the smartphone, the user is notified that the shooting has failed. By displaying a predetermined screen on the display unit 28, the user may be notified that the imaging has failed. In step S321, the system control unit 50 initializes a shooting state (cancellation of shooting a plurality of times). Specifically, the system control unit 50 clears (deletes) the first omnidirectional image stored in the memory 32. Then, the process returns to step S301.

  If it is determined in step S312 that a shooting instruction has been issued, in step S313, the system control unit 50 shoots a second omnidirectional image. In step S314, the system control unit 50 determines whether or not the first omnidirectional image and the second omnidirectional image stored in the memory 32 can be combined. If it is determined that the images can be combined, the process proceeds to step S315; otherwise, the process proceeds to step S317.

  In step S315, the system control unit 50 uses the image processing unit 24 to combine the first omnidirectional image and the second omnidirectional image, so that the photographer does not see the entire omnidirectional image. A sphere image is generated as a composite image. Specifically, the first visual field range in the first omnidirectional image and the second visual field range in the second omnidirectional image are combined. Here, for example, the first field of view is the field of view 501 in FIG. 5A, and the second field of view is the field of view 502 in FIG. 5B. In step S316, the system control unit 50 saves (records) the composite image (omnidirectional image) generated in step S315 in the recording medium 200. Then, the process returns to step S301.

  In step S317, system control unit 50 notifies the user that synthesis cannot be performed. For example, by blinking each of the light emitting units 21a and 21b in red, the user is notified that synthesis cannot be performed. Alternatively, by displaying a predetermined screen on the display surface of the smartphone, the user is notified that synthesis cannot be performed. By displaying a predetermined screen on the display unit 28, the user may be notified that synthesis cannot be performed. In step S318, the system control unit 50 clears (deletes) the second omnidirectional image. Then, the process returns to step S312.

As described above, according to each embodiment, it is possible to capture a wide-range image in which a specific subject such as a photographer is not reflected and has no uncomfortable feeling.

  The above-described various controls described as being performed by the system control unit 50 may be performed by a single piece of hardware, or may be performed by a plurality of pieces of hardware (for example, a plurality of processors or circuits) sharing the processing. Control of the entire apparatus may be performed.

  Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the gist of the present invention are also included in the present invention. included. Furthermore, each of the above-described embodiments is merely an embodiment of the present invention, and the embodiments can be appropriately combined.

  Further, in the above-described embodiment, the case where the present invention is applied to a digital camera has been described as an example. However, the present invention is not limited to this example, and is applicable to any electronic device capable of capturing a VR image. It is possible. Of course, it is also possible to apply to a smartphone instead of a digital camera.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

100: digital camera 22a, 22b: imaging unit 24: image processing unit 50: system control unit

Claims (13)

A means for acquiring a plurality of images by photographing a plurality of times, wherein when acquiring a first image, a specific subject is included in a field of view other than the first field of view in the first image. Control to acquire the first image so as to obtain the second image including the other visual field range, so that the specific subject is not included in the other visual field range. Photographing means for performing control for acquiring the second image;
Performing a process of combining the first visual field range in the first image with a second visual field range in the second image that includes the other visual field range and is different from the first visual field range. Generating means for generating a composite image having a 360 ° field of view around a predetermined axis,
An imaging device comprising: Assisting shooting for acquiring the first image so that the other field of view includes the specific subject, and setting the second field of view so as not to include the specific subject in the other field of view. The imaging apparatus according to claim 1, further comprising: a notification control unit configured to perform a notification for assisting photographing for acquiring an image. The second image is an image acquired next to the first image,
The notification control unit may be configured to perform the imaging after the imaging for acquiring the first image so that the imaging is performed without including the specific subject in the second visual field range in the second image. The imaging apparatus according to claim 2, wherein control is performed so as to notify whether or not the posture of the camera has been changed. The second image is an image acquired next to the first image,
The specific subject is a photographer,
The notification control unit is configured to control the photographer such that the photographer is photographed without being included in the second visual field range in the second image after the photographing for acquiring the first image. The imaging apparatus according to claim 2, wherein control is performed such that notification for prompting movement is performed. A plurality of light emitting units respectively corresponding to the plurality of shootings,
The imaging apparatus according to claim 2, wherein the notification control unit controls the corresponding light emitting unit to emit light in a predetermined light emitting pattern in each of the plurality of shootings. The imaging apparatus according to claim 4, wherein the notification control unit performs control so as to give a notification that prompts a user to fix the imaging apparatus to a tripod before the plurality of shootings. The imaging device according to any one of claims 1 to 6, wherein each of the first field of view and the second field of view is a predetermined field of view. A control unit configured to control so as to cancel the plurality of shootings when a predetermined time has elapsed from the first shooting of the plurality of shootings before the plurality of shootings is completed. The imaging device according to claim 1, wherein: The imaging apparatus according to claim 1, wherein each of the plurality of images is an image of the specific subject. The imaging device according to claim 1, wherein each of the plurality of images is an image having a 360 ° field of view around the predetermined axis. A step of acquiring a plurality of images by photographing a plurality of times, wherein when acquiring a first image, a specific subject is included in a field of view other than the first field of view in the first image. Control to acquire the first image so as to obtain the second image including the other visual field range, so that the specific subject is not included in the other visual field range. Performing control for acquiring the second image;
Performing a process of combining the first visual field range in the first image and a second visual field range in the second image that includes the other visual field range and is different from the first visual field range. Generating a composite image having a 360 ° field of view around a predetermined axis,
A method for controlling an imaging device, comprising:   A program for causing a computer to function as each unit of the imaging device according to claim 1.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the imaging apparatus according to claim 1.

Images