JP2017055268A - Imaging device, control method and program for imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the image quality of a wide-angle composite image, by determining an appropriate material image according to the projection method used in wide-angle image synthesis.SOLUTION: An imaging device 100 for generating a wide-angle composite image by using a material image includes imaging means 22, detection means 80 for detecting at least any one of the position and angle of the imaging device 100, determination means for determining whether or not an image captured by the imaging means 22 is used as the material image, based on at least any one of the position and angle of the imaging device 100 detected by the detection means 80 when imaging by the imaging means 22 and a set projection method, and generation means for converting the image, determined as the material image by the determination means, by a conversion method corresponding to the projection method, and generating a wide-angle composite image.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging apparatus, an imaging apparatus control method, and a program.

In recent years, solid-state imaging devices such as CCD imaging devices and CMOS imaging devices are often used as image capturing means for television cameras, digital cameras, and the like. For the purpose of expanding the dynamic range and noise of this solid-state image sensor, reducing camera shake during hand-held shooting, and widening the angle of view, etc., multiple images are shot in the same scene, and these images are combined by some calculation. A technique has been proposed.
For example, Patent Document 1 proposes a method of generating a wide-angle image by continuously shooting while moving a shooting range of an imaging apparatus, and aligning and synthesizing a plurality of obtained images. Hereinafter, such a method is called wide-angle image composition, an image used for wide-angle image composition is called a material image, and the synthesized image is called a wide-angle composition image. In Patent Document 1, a wide-angle image to be combined is divided into regions, and a material image suitable for combining is stored in a memory for each region.

  In wide-angle image composition, various projection methods are used. For example, there are a method of projecting onto a plane and a method of projecting onto a cylindrical surface. The appropriate material image differs depending on the projection method used. In the following, the method of projecting onto a plane is referred to as plane projection, the method of projecting onto a cylindrical surface is referred to as cylindrical projection, and the geometric deformation projected onto the cylindrical surface is referred to as cylindrical transformation.

Here, planar projection and cylindrical projection will be described with reference to FIG.
First, planar projection will be described with reference to FIG. Planar projection is a projection method in which a captured image 701 is projected from a projection center 702 onto a projection plane 700 that is a reference plane. In FIG. 12A, there are five captured images 701, and a plane including the image 701 at the center is a projection plane 700.
When plane projection is used for wide-angle image composition, first, the material image is converted into an image projected on the projection plane 700. Then, the converted material images are combined to generate a wide-angle combined image. For the conversion of the material image, projective conversion which is a conversion method corresponding to planar projection is used. Therefore, a material image is desirable because an image taken while being translated with respect to the projection plane 700 has a small amount of geometric deformation.

Next, cylindrical projection will be described with reference to FIG. Cylindrical projection is a projection method in which a captured image 711 is projected from a projection center 712 onto a projection plane 710 that is a reference cylindrical plane. FIG. 12C is a perspective view of one of the images 711 in FIG. 12B and a part of the projection plane 710.
When using cylindrical projection for wide-angle image composition, first, the material image is converted into an image projected on the projection plane 710. For the conversion of the material image, cylindrical conversion which is a conversion method corresponding to cylindrical projection is used. Then, the converted material images are combined to generate a wide-angle combined image. In the case of cylindrical projection, it is widely known that the material image is desirably an image 711 that is taken with a point called parallax point (NPP) having no parallax as the center of rotation.

JP 2011-205395 A

  However, Patent Document 1 does not consider that the appropriate material image differs depending on the projection method used in the wide-angle image composition, and the image quality of the wide-angle composite image may not be sufficient.

  The present invention has been made in view of such a situation, and an object of the present invention is to determine an appropriate material image according to a projection method used in wide-angle image composition to improve the image quality of the wide-angle composite image.

  The imaging device of the present invention is an imaging device that generates a wide-angle composite image using a material image, and includes an imaging unit, a detection unit that detects at least one of a position and an angle of the imaging device, Based on at least one of the position and angle of the imaging device detected by the detection unit during imaging by the imaging unit and a set projection method, the image captured by the imaging unit is used as a material image. Determining means for determining whether to generate a wide-angle composite image by converting the image determined to be a material image by the determining means by a conversion method corresponding to the projection method and combining the converted images And means.

  According to the present invention, whether or not a captured image is used as a material image based on at least one of the position and angle of the imaging device detected during imaging by the imaging unit and the set projection method. To decide. Therefore, an appropriate material image can be determined according to the projection method used in the wide-angle image composition, and the image quality of the wide-angle composite image can be improved.

It is an external view which shows the back external appearance of the digital camera in 1st Embodiment. It is a block diagram which shows the structure of the digital camera in 1st Embodiment. It is a conceptual diagram explaining how to represent the position and angle of the digital camera in the first embodiment. It is a conceptual diagram for demonstrating the wide angle image composition mode in 1st Embodiment, (a) is a conceptual diagram which shows the relationship between the direction to which a user moves a digital camera, and the angle of view of a raw material image, (b ) Is a conceptual diagram showing the relationship between the angle of view of the wide-angle composite image and the angle of view of the material image. It is a flowchart which shows the process in the wide angle image composition mode of the digital camera in 1st Embodiment. It is a figure which shows a screen when setting the division number of the imaging area in 1st Embodiment, (a) is a figure which shows a division number setting screen, (b) is a figure which shows setting screens, such as a reference | standard screen. is there. It is a figure which shows the screen which sets the projection method in 1st Embodiment, Comprising: (a) is a figure which shows a screen when plane projection is set, (b) is a screen when cylindrical projection is set. FIG. It is a figure which shows the guide display in 1st Embodiment, Comprising: (a) is a figure which shows guide display when plane projection is set, (b) shows guide display when cylindrical projection is set. FIG. It is a figure which shows the material image and wide-angle synthetic | combination image in 1st Embodiment, Comprising: (a) is a figure which shows a 1st material image, (b) is a figure which shows a 2nd material image, (c) is a figure. It is a figure which shows a 3rd material image, (d) is a figure which shows a wide angle synthetic | combination image. It is a conceptual diagram explaining NPP in 2nd Embodiment. It is a figure which shows the material image and wide-angle synthetic image in 2nd Embodiment, Comprising: (a) is a figure which shows a 1st material image, (b) is a figure which shows a 2nd material image, (c) is a figure. It is a figure which shows a 3rd material image, (d) is a figure which shows the 1st material image converted by cylindrical projection, (e) is a figure which shows the 2nd material image converted by cylindrical projection, (F) is a figure which shows the 3rd material image converted by cylindrical projection, (g) is a figure which shows a wide angle synthetic | combination image. It is a conceptual diagram explaining a projection method, (a) is a conceptual diagram explaining planar projection, (b) is a conceptual diagram explaining cylindrical projection, (c) is a cylindrical projection about one image. FIG.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
(First embodiment)
A digital camera 100 that is an example of an imaging apparatus will be described. The digital camera 100 uses a first image continuously shot as a reference image for alignment, and performs wide-angle image synthesis that has a wide angle in the horizontal direction using planar projection as a projection method. Details of the wide-angle image composition will be described later. Note that description of processing that is generally performed, such as aperture correction that corrects the influence of a pixel having a finite opening, is omitted.

First, the external appearance of the digital camera 100 will be described with reference to FIG. FIG. 1 is an external view showing the rear external appearance of the digital camera 100.
The display unit 28 displays images and various information. As the display unit 28, for example, a liquid crystal display is used.
The connector 112 is a component for connecting the connection cable 111 connected to an external device to the digital camera 100.
The operation unit 70 receives various operations from the user, and includes a shutter button 61, a mode switch 60, a power switch 72, and a controller wheel 73. Moreover, the operation part 70 may be provided with operation members, such as various switches other than these, a button, and a touch panel.

The shutter button 61 is an operation member for issuing a shooting instruction.
The mode switch 60 is an operation member for switching various modes.
The power switch 72 is an operation member for switching power on and power off.
The controller wheel 73 is an operation member that can be rotated.
The recording medium 200 is a storage unit that stores image data and the like. For example, a memory card or a hard disk is used as the recording medium 200.
The recording medium slot 201 is a slot for storing the recording medium 200, and a lid 202 for closing the recording medium slot 201 is attached. The recording medium 200 stored in the recording medium slot 201 can communicate with the digital camera 100.

Next, the configuration of the digital camera 100 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the digital camera 100.
The photographing lens 103 is a lens group including a zoom lens and a focus lens.
The shutter 101 is a shutter provided with a diaphragm means.
The imaging unit 22 is an imaging device configured by a CCD, a CMOS device, or the like that converts an optical image into an electrical signal.
The A / D converter 23 converts an analog signal into a digital signal. The A / D converter 23 is used to convert an analog signal output from the imaging unit 22 into a digital signal. Output data from the A / D converter 23 is directly written into the memory 32 via the image processing unit 24 and the memory control unit 15 or via the memory control unit 15.
The barrier 102 covers the imaging system including the imaging lens 103 to prevent the imaging system including the imaging lens 103, the shutter 101, and the imaging unit 22 from being soiled or damaged.

  The image processing unit 24 performs resizing processing such as predetermined pixel interpolation and reduction and color conversion processing on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. As a result, TTL (through-the-lens) AF (autofocus) processing and AE (automatic exposure) processing are performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained arithmetic result. Further, the image processing unit 24 performs geometric transformation such as projective transformation and cylindrical transformation on the image data. Further, the image processing unit 24 performs alignment of the image data, and performs wide-angle image synthesis that generates a wide-angle composite image by combining the aligned images. Details of the wide-angle image composition in this embodiment will be described later.

The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23 and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time. The memory 32 also serves as an image display memory (video memory).
The memory control unit 15 controls data writing to the memory 32 and data reading from the memory 32.
The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28. Thus, the display image data written in the memory 32 is displayed on the display unit 28 via the D / A converter 13.
The display unit 28 performs display according to the analog signal from the D / A converter 13 on a display such as an LCD. In addition, the display unit 28 can function as an electronic viewfinder, and a through image can be displayed on the display unit 28. When the display unit 28 functions as an electronic viewfinder, the digital signal once A / D-converted by the A / D converter 23 and stored in the memory 32 is converted into an analog signal by the D / A converter 13. Transfer sequentially.

The nonvolatile memory 56 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants, programs, and the like for operating the system control unit 50. The program here is a program for executing various flowcharts described later in the present embodiment.
The system control unit 50 controls the entire digital camera 100. By executing the program recorded in the non-volatile memory 56 described above, each process of the present embodiment to be described later is realized. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 13, the display unit 28, and the like.
The system memory 52 is a RAM, in which constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are expanded.
The system timer 53 is a time measuring unit that measures the time used for various controls and the time of a built-in clock.

  The operation unit 70 such as the mode switch 60, the first shutter switch 62, and the second shutter switch 63 is an operation unit for inputting various operation instructions to the system control unit 50. The mode switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene discrimination mode, a manual mode, various scene modes for shooting settings for each shooting scene, a program AE mode, a wide-angle image composition mode, a custom mode, and the like. The mode changeover switch 60 can directly switch to one of these modes included in the still image shooting mode. Alternatively, after switching to the still image shooting mode once with the mode switch 60, the mode may be switched to any one of these modes included in the still image shooting mode using another operation member. Similarly, the moving image shooting mode may include a plurality of modes.

The first shutter switch 62 is turned on when the shutter button 61 provided in the digital camera 100 is being operated, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the image processing unit 24 starts operations such as AF processing, AE processing, and AWB processing.
The second shutter switch 63 is turned on when the operation of the shutter button 61 is completed, so-called full depression (shooting instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processing operations from reading a signal from the imaging unit 22 to writing image data on the recording medium 200.

Each operation member of the operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 28, and functions as various function buttons.
Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button. For example, when a menu button is pressed, various setting menu screens are displayed on the display unit 28. The user can make various settings intuitively using the menu screen displayed on the display unit 28, and the four-way buttons and the SET button.

  The controller wheel 73 is a rotatable operation member included in the operation unit 70, and is used when a selection item is instructed together with a direction button. When the controller wheel 73 is rotated, an electrical pulse signal is generated according to the operation amount, and the system control unit 50 controls each unit of the digital camera 100 based on the pulse signal. From this pulse signal, it is possible to determine the angle at which the controller wheel 73 is rotated, how many rotations, and the like. The controller wheel 73 may be any operation member that can detect a rotation operation. For example, it may be a dial operation member that generates a pulse signal by rotating the controller wheel 73 itself according to the rotation operation of the user. Further, an operation member made of a touch sensor may detect the rotation operation of the user's finger on the controller wheel 73 without rotating the controller wheel 73 itself (so-called touch wheel).

The power control unit 31 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. Further, the power supply control unit 31 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period.
The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.
The recording medium I / F 18 is an interface with the recording medium 200 such as a memory card or a hard disk.
The recording medium 200 is a recording medium such as a memory card for recording a captured image, and includes a semiconductor memory, a magnetic disk, or the like.
The detection unit 80 includes a gyro sensor, an acceleration sensor, and the like. The system control unit 50 calculates the position and angle of the digital camera 100 from the detection result of the detection unit 80. For example, when the detection result of the detection unit 80 is angular velocity or acceleration, the system control unit 50 calculates the position and angle of the digital camera 100 by integrating each.

Next, the position and angle of the digital camera 100 will be described with reference to FIG. FIG. 3 is a conceptual diagram illustrating the coordinate system of the position and angle of the digital camera 100. As shown in FIG. 3, the position and angle of the digital camera 100 are represented using an orthogonal coordinate system including an X axis, a Y axis, and a Z axis.
The position of the digital camera 100 is represented as (x, y, z) using the coordinate values of the X axis, the Y axis, and the Z axis. Here, the position of the reference digital camera 100 is (0, 0, 0). Further, the angle of the digital camera 100 is expressed as (α, β, γ) using angles rotated about the X-axis direction axis, the Y-axis direction axis, and the Z-axis direction axis as rotation axes. Here, it is assumed that the angle of the reference digital camera 100 is (0 °, 0 °, 0 °).
The reference position and angle of the digital camera 100 are, for example, the position and angle of the digital camera 100 when the digital camera 100 first shoots. The system control unit 50 sets the detection result of the detection unit 80 synchronized with the first shooting as a reference value, that is, an angle (0 °, 0 °, 0 °) and a position (0, 0, 0). After calculating the reference value, the system control unit 50 calculates the position and angle of the digital camera 100 as a relative value from the reference value.

Next, an overview of the wide-angle image composition mode of the digital camera 100 will be described with reference to FIG. FIG. 4 is a conceptual diagram for explaining the wide-angle image composition mode, and FIG. 4A is a conceptual diagram showing the relationship between the direction in which the user moves the digital camera and the angle of view G1 of the material image. 4B is a conceptual diagram showing the relationship between the angle of view G2 of the wide-angle composite image and the angle of view G1 of the material image.
An angle of view G1 shown in FIGS. 4A and 4B is an angle of view of a material image used in the wide-angle image composition mode. An angle of view G2 illustrated in FIG. 4B is an angle of view of a wide-angle synthesized image synthesized in the wide-angle image synthesis mode. In the present embodiment, a case will be described in which synthesis is performed using planar projection as a projection method.

  As shown in FIG. 4A, in the present embodiment, the first image captured by the digital camera 100 is used as a reference image for alignment, and the user continuously captures images while moving the imaging range of the digital camera 100 in parallel in the horizontal direction. To do. In FIG. 4B, the vertical angle of view of the angle of view G2 is narrower than the vertical angle of view of the angle of view G1, but this is because a vertical shift during continuous shooting is assumed. is there. Since the user moves the digital camera 100 in the horizontal direction, the shooting range shifts in the vertical direction, and there is a range in which the subject data does not exist. The angle of view G2.

Next, processing in the wide-angle image composition mode of the digital camera 100 will be described with reference to FIG. FIG. 5 is a flowchart showing processing in the wide-angle image composition mode of the digital camera 100. The digital camera 100 executes the process of FIG. 5 when the mode changeover switch 60 selects the wide-angle image composition mode.
In step S100, the system control unit 50 determines whether or not the setting button of the operation unit 70 has been pressed. When it is determined that the setting button has been pressed, the system control unit 50 proceeds to step S101. When it is determined that the setting button has not been pressed, the system control unit 50 proceeds to step S102.

In step S <b> 101, the system control unit 50 changes the setting of the projection method of the wide-angle composite image, the angle of view of the wide-angle composite image, the number of shooting areas, the shooting order of the material images, the reference image, etc. Then, the process proceeds to step S102. Further, the system control unit 50 may set a region to be a wide-angle composite image in accordance with a user designation. Note that depending on the setting of the projection method, the angle of view of the wide-angle composite image, etc., there is a case where it is not necessary to change the setting in step S101.
In the present embodiment, it is assumed that the user sets planar projection as a projection method and sets an image captured at the beginning of continuous shooting as a reference image. In addition, it is assumed that the number of shooting areas is not set, and the range of continuous shooting by the user moving the digital camera 100 in parallel is set as the angle of view of the wide-angle composite image.
When the processing in the wide-angle image composition mode is performed for the first time after the digital camera 100 is activated, when the processing proceeds to step S102 without pressing the setting button, a predefined initial setting value is used.

  When setting the number of sections, the system control unit 50 displays a section number setting screen 300 shown in FIG. The division number setting screen 300 displays a plurality of numerical values as the number of divisions. When the user selects any numerical value, the system control unit 50 sets the selected numerical value as the number of divisions. Further, the system control unit 50 displays a reference screen setting screen 310 shown in FIG. 6B on the display unit 28. The reference screen setting screen 310 displays a plurality of combinations of the reference image and the shooting order according to the setting of the angle of view of the wide-angle composite image and the number of shooting areas. When the user selects any combination, the system control unit 50 sets the reference image and the shooting order.

  In step S102, the system control unit 50 displays the explanation screen 400 shown in FIG. The explanation screen 400 is a screen that displays a projection method and an explanation of how to move the digital camera 100 when taking a picture. In the present embodiment, as described above, since the planar projection is set, the explanation screen 400 displays that the projection method is the planar projection. Further, when the projection method is planar projection, an image taken by moving the digital camera 100 in parallel is desirable as a material image. Therefore, the explanation screen 400 displays that the user needs to move the digital camera 100 in parallel during shooting.

In step S103, the system control unit 50 determines whether or not the first shutter switch 62 has been pressed. When it is determined that the first shutter switch 62 has been pressed, the system control unit 50 proceeds to step S104, and when it is determined that the first shutter switch 62 has not been pressed, the process returns to step S100.
In step S104, the system control unit 50 controls the image processing unit 24 to perform AE processing and AF processing.
In step S105, the system control unit 50 determines shooting conditions such as shooting sensitivity based on the AE processing result in step S103.
In step S106, the system control unit 50 determines whether or not the second shutter switch 63 has been pressed. When it is determined that the second shutter switch 63 has been pressed, the system control unit 50 proceeds to step S108, and when it is determined that the second shutter switch 63 has not been pressed, the process proceeds to step S107.
In step S107, the system control unit 50 determines whether or not the first shutter switch 62 remains pressed (whether or not the so-called half-press is kept). When it is determined that the first shutter switch 62 has been released, the system control unit 50 returns the process to step S100, and when it is determined that the first shutter switch 62 remains pressed, the process returns to step S106.

In step S108, the system control unit 50 captures an image with the image capturing conditions determined in step S105. The image photographed in step S108 is used as a reference image for alignment. The system control unit 50 stores the image captured in step S108 in the memory 32 as a material image.
In step S109, the system control unit 50 detects the position and angle of the digital camera 100 from the result detected by the detection unit 80 in synchronization with the timing when the second shutter switch 63 is pressed. The system control unit 50 sets the position and angle of the digital camera 100 as the reference position and angle of the digital camera 100.
In step S <b> 110, the system control unit 50 sets a target position and target angle, which are the target position and angle of the digital camera 100, a weighting factor, and a threshold according to the projection method and focal length of the wide-angle composite image. To do. The target position is expressed as (X _tar , Y _tar , Z _tar ), which are the coordinate values of the X axis, the Y axis, and the Z axis, respectively. The target angles are expressed as (α _tar , β _tar , γ _tar ), and are angles rotated about the X-axis direction axis, the Y-axis direction axis, and the Z-axis direction axis, respectively. The weighting coefficient is composed of six numerical values, and each numerical value is represented as (ω _X , ω _Y , ω _Z , ω _α , ω _β , ω _γ ). The target position and the target angle may be calculated by calculation, or values stored in a lookup table stored in advance in the nonvolatile memory 56 may be used. As the weighting coefficient and threshold value, values stored in a lookup table stored in advance in the nonvolatile memory 56 are used.
In the present embodiment, since the projection method is planar projection, an image obtained by moving the digital camera 100 in parallel is desirable as a material image. Therefore, the target position and the target angle are set so as to have values when the digital camera 100 is paralleled from the reference position and angle. For example, when the target position and target angle are determined so that the digital camera 100 is parallel to the X-axis direction shown in FIG. 3, the target position and target angle are Y _tar = 0, Z _tar = 0, α _tar = 0, β _tar = 0 and _γtar = 0 are set and used in an evaluation function described later.

  In step S111, the system control unit 50 displays guide information 500 on the display unit 28 as shown in FIG. 8A in order to prompt the user to move the shooting range of the digital camera 100. The guide information 500 serves as a guide for the position and angle of the digital camera 100 so that an image taken in step S112 described below is used as a material image. The system control unit 50 calculates the direction in which the digital camera 100 is moved from the difference between the target position and the position of the digital camera 100 and the difference between the target angle and the angle of the digital camera 100. The guide information 500 is a figure such as an arrow representing the calculation result. In the example shown in FIG. 8A, guide information 500 that prompts the digital camera 100 to translate in the direction of the arrow is displayed. The process in step S111 is an example of a process performed by a display unit. The guide information 500 may be a display other than an arrow, for example, a graphic or a sentence other than an arrow, or may be a voice guidance.

In step S112, the system control unit 50 captures an image with the image capturing conditions determined in step S105. At this time, imaging by the imaging unit 22 is performed.
In step S113, the system control unit 50 detects the position and angle of the digital camera 100 during imaging by the imaging unit 22 from the result detected by the detection unit 80 in synchronization with the imaging timing in step S112. The system control unit 50 sets the detected position of the digital camera 100 as (X _cor , Y _cor , Z _cor ) and the angles as (α _cor , β _cor , γ _cor ).
In step S114, the system control unit 50 calculates an evaluation value VAL for determining whether or not the image captured in step S112 is suitable as a material image, using the evaluation function shown in the following expression 1.
VAL = ω _X (X _tar −X _cor ) + ω _Y (Y _tar −Y _cor ) + ω _Z (Z _tar −Z _cor ) + ω _α (α _tar −α _cor ) + ω _β (β _tar −β _cor ) + ω _γ ( γ _tar −γ _cor ) (Formula 1)

In step S115, the system control unit 50 determines whether the image captured in step S112 is suitable as a material image by comparing the evaluation value VAL with a threshold value. This threshold value is the value set in step S110.
If the evaluation value VAL is greater than or equal to the threshold value, the system control unit 50 determines that the image captured in step S112 is not suitable as a material image, and advances the process to step S117. If it is determined that the evaluation value VAL is smaller than the threshold value, the system control unit 50 determines that the image captured in step S112 is suitable as a material image, and determines to handle the image captured in step S112 as a material image. Then, the process proceeds to step S116. Note that the process of step S115 is an example of a process performed by a determination unit.

In step S116, the system control unit 50 stores the image captured in step S112 in the memory 32 as a material image. Note that the system control unit 50 may store the position and angle information of the digital camera 100 detected in step S113 in the memory 32 in association with the material image.
In step S <b> 117, the system control unit 50 determines whether shooting is continued using the position and angle information of the digital camera 100. For example, the system control unit 50 compares the position and angle information of the digital camera 100 at the previous shooting (one frame before) with the position and angle information of the digital camera 100 at the current shooting. Then, when the system control unit 50 determines that the digital camera 100 has moved in a direction significantly different from the direction displayed in the guide information, the system control unit 50 determines that shooting is not continued. When it is determined that shooting is continued, the system control unit 50 returns the process to step S110, and when it is determined that shooting is not continued, the process proceeds to step S118. Note that the processing in step S117 is an example of processing by the control means.

In step S <b> 118, the system control unit 50 determines whether two or more material images are stored in the memory 32. The system control unit 50 proceeds to step S120 when two or more material images are stored in the memory 32, and proceeds to step S119 when two or more material images are not stored in the memory 32. Proceed.
In step S119, the system control unit 50 notifies the composition failure. The notification of the composition failure is performed, for example, by displaying on the display unit 28 that the wide-angle image composition has failed. Then, the system control unit 50 returns the process to step 100.

In step S120, the system control unit 50 controls the image processing unit 24 to convert (geometrically deform) the material image stored in the memory 32 by projective transformation. The projective transformation of the material image is performed so that the plane including the reference image becomes the projection plane. Then, the system control unit 50 controls the image processing unit 24 to perform the alignment of the converted material image using the reference image as a reference for alignment. The alignment is performed by a method of dividing the image into small blocks of an arbitrary size, calculating corresponding points at which the luminance SAD (Sum of Absolute Difference) is minimum for each small block, and calculating a motion vector. .
In step S121, the system control unit 50 controls the image processing unit 24 to synthesize the converted material images aligned in step S120 and generate a wide-angle synthesized image. Note that the processing in steps S120 and S121 is an example of processing by the generation unit.
In step S122, the system control unit 50 records the wide-angle composite image generated in step S121 on the recording medium 200, and ends the wide-angle image composite mode.

  Next, a generation example of a wide-angle composite image will be shown with reference to FIG. The images shown in FIGS. 9A, 9B, and 9C are a first material image 600, a second material image 601, and a third material image 602, respectively. Then, by combining the wide angle images using the first to third material images 600 to 602, a wide angle combined image 603 shown in FIG. 9D is obtained.

As described above, in the present embodiment, the projection method is planar projection, and an image determined to be suitable as a material image by the process of step S115 in FIG. 5 is handled as a material image of a wide-angle composite image. Therefore, in wide-angle image synthesis that performs projective transformation corresponding to planar projection, an image suitable as a wide-angle synthesized image can be used as a material image, and the image quality of the wide-angle synthesized image can be improved.
Further, as shown in FIG. 8A, guide information 500 is displayed on the display unit 28. Therefore, by moving the digital camera 100 according to the guide information 500, the user can easily shoot an appropriate material image as a wide-angle composite image.

(Second Embodiment)
In the above-described embodiment, the example in which the wide-angle image synthesis with a wide angle in the horizontal direction is performed using planar projection as the projection method has been described. In the present embodiment, an example will be described in which a first captured image is used as a reference image for alignment, and wide-angle image composition is performed with a wide angle in the horizontal direction using cylindrical projection as a projection method. Note that the description of the same points as in the above embodiment is omitted or simplified.
The position and angle of the digital camera 100 are represented using an orthogonal coordinate system including an X axis, a Y axis, and a Z axis, as in the case of the first embodiment shown in FIG. As shown in FIG. 10, the origins of the X axis, the Y axis, and the Z axis are NPPs.

In the present embodiment, it is assumed that in step S101 in FIG. 5, the user sets cylindrical projection as the projection method and sets the image captured at the beginning of continuous shooting as the reference image. This setting is called a cylindrical projection setting.
When the cylindrical projection setting is set, the system control unit 50 displays the explanation screen 401 shown in FIG. 7B on the display unit 28 in step S102 of FIG. An explanation screen 401 is a screen that displays a projection method and an explanation of how to move the digital camera 100 when photographing. In the present embodiment, as described above, since cylindrical projection is set, the explanation screen 401 displays that the projection method is cylindrical projection. Further, when the projection method is cylindrical projection, an image obtained by rotating the digital camera 100 around the NPP is desirable as a material image. In view of this, the explanation screen 401 displays that the user needs to rotate the digital camera 100 about the Y axis passing through the NPP as the rotation axis.

When the cylindrical projection setting is set, an image obtained by rotating the digital camera 100 around the NPP is desirable as a material image. Therefore, in the present embodiment, in step S110 in FIG. 5, the target position and the target angle are determined so that the digital camera 100 is rotated about the NPP and the Y axis as the rotation axis. For example, the target position and target angle are set as X _tar = 0, Y _tar = 0, Z _tar = 0, α _tar = 0, and γ _tar = 0, and are used in the evaluation function (Equation 1) in step S114.

When the cylindrical projection setting is set, in step S111, the system control unit 50 displays the display unit as shown in FIG. 8B in order to prompt the user to move the shooting range of the digital camera 100. 28, guide information 501 is displayed. In the example shown in FIG. 8B, guide information 501 that prompts the digital camera 100 to rotate about the Y axis as a rotation axis is displayed.
When the cylindrical projection setting is set, in step S120, the system control unit 50 controls the image processing unit 24 to convert (geometrically deform) the material image stored in the memory 32 by cylindrical conversion. Then, the system control unit 50 controls the image processing unit 24 to align the converted material image.

  Next, a generation example of a wide-angle composite image will be shown with reference to FIG. Images shown in FIGS. 11A, 11B, and 11C are a first material image 610, a second material image 611, and a third material image 612, respectively. By converting the first material image 610 into a cylinder, a first converted material image 613 shown in FIG. 11D is generated. Similarly, by converting the second material image 611 and the third material image 612 into cylinders, the second conversion material image 614 shown in FIG. 11 (e) and the third conversion material shown in FIG. 11 (f), respectively. An image 615 is generated. Then, by combining the first to third conversion material images 613 to 615 after positioning, a wide-angle composite image 616 shown in FIG. 11G is obtained.

  In the present embodiment, an image obtained by rotating the digital camera 100 around the NPP and having the Y axis as a rotation axis is used as a material image. However, a predetermined axis other than the Y axis, for example, the X axis may be used as the rotation axis.

As described above, in this embodiment, the projection method is cylindrical projection, and an image determined to be suitable as a material image by the process of step S115 in FIG. 5 is treated as a material image of a wide-angle composite image. Therefore, in wide-angle image synthesis that performs cylindrical transformation corresponding to cylindrical projection, an appropriate image as a wide-angle synthesized image can be used as a material image, and the image quality of the wide-angle synthesized image can be improved.
Further, as shown in FIG. 8B, guide information 501 is displayed on the display unit 28. Therefore, the user can easily shoot an appropriate material image as a wide-angle composite image by moving the digital camera 100 according to the guide information 501.

(Other embodiments)
In the above embodiment, as shown in steps S115 and S116 of FIG. 5, the system control unit 50 indicates a process of leaving all images in the memory 32 as material images when the evaluation value VAL is smaller than the threshold value. However, the wide-angle composite image to be combined may be divided into regions, and an image optimal for combining may be stored in the memory 32 for each region.
For example, the system control unit 50 divides the wide-angle composite image into a predetermined number of regions (first region to Nth region). In step S110, the system control unit 50 uses the target position, the target angle, the weighting factor, and the threshold value as a material image used for the nth region of the wide-angle composite image based on the projection method used in the wide-angle composite image. Set to a value that allows you to obtain the correct image.

In the above embodiment, the material image is stored in the memory 32. However, the system control unit 50 may record the material image on the recording medium 200 and delete the material image from the recording medium 200 after generating the wide-angle image composition.
In the above-described embodiment, processing related to planar projection and cylindrical projection has been described. However, projection may be performed by other projection methods such as spherical projection which is a projection method for projecting onto a spherical surface. When the projection method is a spherical projection, for example, an image captured while rotating the digital camera 100 around the NPP and the NPP as a center is used as a material image.
In the above embodiment, the detection unit 80 detects the position and angle of the digital camera 100. However, the detection unit 80 may detect only one of the position and the angle of the digital camera 100. For example, when planar projection is used as the projection method, the detection unit 80 calculates the evaluation value VAL by setting ω _α , ω _β and ω _γ to 0 in Equation 1 so as to detect only the position. When cylindrical projection is used as the projection method, the detection unit 80 calculates the evaluation value VAL by setting ω _X , ω _Y, and ω _Z to 0 in Equation 1 so as to detect only the angle.

The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
Although the present invention has been described together with the embodiments, the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention is interpreted in a limited manner by these. It must not be. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  100 digital camera, 28 display unit, 50 system control unit

Claims (10)

An imaging device that generates a wide-angle composite image using a material image,
Imaging means;
Detecting means for detecting at least one of a position and an angle of the imaging device;
Based on at least one of the position and angle of the imaging device detected by the detection unit during imaging by the imaging unit and a set projection method, an image captured by the imaging unit is defined as a material image. A determination means for determining whether or not to do;
Generating means for converting the image determined as the material image by the determining means by a conversion method corresponding to the projection method, and generating a wide-angle composite image by combining the converted images. Imaging device.   When the projection method is planar projection, the determination unit determines whether the position and angle of the imaging device detected by the detection unit during imaging by the imaging unit is a position and angle translated from a reference position and angle. 2. The imaging apparatus according to claim 1, wherein whether or not to use the image captured by the imaging unit as a material image is determined based on whether or not the image is captured.   When the projection method is cylindrical projection, the determination unit is configured such that the position and angle of the imaging device detected by the detection unit during imaging by the imaging unit is a predetermined axis from the reference position and angle as a rotation axis. 3. The image pickup apparatus according to claim 1, wherein whether or not to use the image picked up by the image pickup means as a material image is determined based on whether or not the position and angle are rotated.   When the projection method is spherical projection, the determination unit rotates the position and angle of the imaging device detected by the detection unit during imaging by the imaging unit around a predetermined point from a reference position and angle. 4. The imaging apparatus according to claim 1, wherein it is determined whether or not an image captured by the imaging unit is used as a material image based on whether or not the position and the angle are determined. .   The imaging apparatus according to claim 1, further comprising a control unit that controls the imaging unit so as to repeatedly perform imaging.   The imaging device according to claim 1, wherein the determining unit determines the material image for each region into which the wide-angle composite image is divided.   The imaging apparatus according to claim 1, wherein the determination unit determines whether or not the captured image is used as a material image every time the imaging unit captures an image.   The imaging apparatus according to claim 1, further comprising display means for displaying a guide for changing at least one of a position and an angle of the imaging apparatus. A method of controlling an imaging apparatus that generates a wide-angle composite image using a material image,
Imaging step;
A detection step of detecting at least one of a position and an angle of the imaging device;
Based on at least one of the position and angle of the imaging device detected in the detection step during imaging in the imaging step, and a set projection method, the image captured in the imaging step is defined as a material image. A decision step for deciding whether or not to do;
A generation step of converting the image determined as the material image in the determination step by a conversion method corresponding to the projection method, and generating a wide-angle composite image by combining the converted images. Control method of imaging apparatus. A program for controlling an imaging device that generates a wide-angle composite image using a material image,
Imaging step;
A detection step of detecting at least one of a position and an angle of the imaging device;
Based on at least one of the position and angle of the imaging device detected in the detection step during imaging in the imaging step, and a set projection method, the image captured in the imaging step is defined as a material image. A decision step for deciding whether or not to do;
Converting the image determined as the material image in the determination step by a conversion method corresponding to the projection method, and generating a wide-angle composite image by combining the converted images; program.

Images