JP6642661B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging device.

  2. Description of the Related Art Conventionally, a panoramic image is generated by capturing a plurality of images while panning a digital camera or the like and sequentially connecting the images. For example, Patent Literature 1 discloses a technique of performing panorama synthesis processing using a frame image of a moving image.

JP 2000-244814 A

  However, according to the related art, if the movement amount due to panning is too large to join the images, the image must be taken again from the beginning.

  An object of the present embodiment is to provide an imaging device capable of performing suitable imaging.

One mode of an imaging device illustrating the present invention is an imaging unit that captures an image of a subject that has passed through an imaging optical system to generate image data, and an imaging unit that captures an image while changing an imaging direction of the imaging optical system. both an image processing unit for synthesis a plurality of images based on the plurality of image data generated, the image based on the image data generated by the imaging of the imaging unit, the direction indicating the direction to change the imaging direction of the imaging optical system A display unit that displays the image and the combined image combined by the image processing unit; and, when the imaging direction is changed to a direction different from the direction indicated by the directional image, an image generated by imaging by the imaging unit on the display unit. A control unit for displaying an image different from the image based on the data, the directional image, and the composite image is provided .

  According to the present embodiment, it is possible to provide an imaging device capable of performing suitable imaging.

1 is a block diagram illustrating a configuration of a digital camera according to an embodiment. 4 is a flowchart illustrating an image processing operation of the digital camera according to the embodiment. Diagram showing an example of display on a monitor in panoramic mode (part 1) Diagram showing an example of display on the monitor in the panorama mode (part 2) Figure showing another example of monitor display in panoramic mode FIG. 10 is a diagram showing another example of display on the monitor in the panorama mode.

  FIG. 1 is a block diagram showing a configuration of a digital camera according to one embodiment of the present invention.

  The digital camera according to the present embodiment includes an imaging optical system 11, an imaging element 12, an AFE 13, a CPU 14, a memory 15, a monitor 16, a recording interface (I / F) 17, an operating member 19, a release button 20, a touch panel 21, and a bus. Is done. The CPU 14, the memory 15, the monitor 16, and the recording I / F 17 are connected so as to be able to transmit information via a bus. The operation member 19, the release button 20, and the touch panel 21 are connected to the CPU 14, respectively. It is assumed that the digital camera according to the present embodiment has a panorama mode for generating one panorama image from a plurality of images obtained by dividing and capturing an object scene, as one function of a shooting mode.

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, FIG. 1 illustrates the imaging optical system 11 as a single lens.

  The imaging element 12 is a device that captures an image of an object field formed by a light beam that has passed through the imaging optical system 11. The output of the image sensor 12 is input to the AFE 13. Note that the imaging device 12 of the present embodiment may be a solid-state imaging device of a progressive scanning type (CCD or the like) or a solid-state imaging device of an XY addressing system (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. Red (R), green (G), and blue (B) color filters are arranged in each light receiving element of the imaging element 12 according to a known Bayer arrangement. Therefore, each light receiving element of the image sensor 12 outputs an image signal corresponding to each color by color separation by a color filter. Thereby, the imaging element 12 can acquire a color image.

  Here, in the shooting mode of the digital camera, the image sensor 12 captures a recorded image (main image) in response to a full-press operation of the release button 20. Further, in the photographing mode, the image sensor 12 captures a low-resolution image (through image) for confirming the composition at a predetermined time interval even during the photographing standby. The data of the through image is output from the image sensor 12 by thinning-out reading. The data of the through-the-lens image is used for image display on the monitor 16 and various arithmetic processes by the CPU 14. When the shooting mode is the moving image mode or the panoramic mode, the imaging element 12 captures a moving image as a main image at a predetermined frame rate (30 fps, 60 fps, or the like).

  The AFE 13 is an analog front-end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, adjustment of the gain of the image signal, and A / D conversion of the image signal. The output of the AFE 13 is connected to the CPU 14. Note that, in the present embodiment, an imaging unit is configured by the imaging element 12 and the AFE 13.

  The CPU 14 is a processor that totally controls the operation of the digital camera. By executing the control program, the CPU 14 performs known autofocus (AF) control, automatic exposure (AE) calculation, automatic white balance calculation, and the like based on the through image data based on the phase difference detection method and the contrast detection method. Do. The CPU 14 also performs a setting screen, display processing of a captured main image and a through image, and the like. Further, the CPU 14 operates as an image processing unit 22 that performs image processing on the digital image signal output from the AFE 13 by executing the image processing program stored in the memory 15.

  The image processing unit 22 performs various image processing (for example, color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, etc.) on the digital image signal output from the AFE 13. Further, as described later, when the shooting mode is set to the panorama mode, the image processing unit 22 according to the present embodiment outputs a frame (a target image) to be captured as a moving image based on a known panorama synthesis process. A panoramic image (synthesized image) is generated by connecting the images in sequence and performing panoramic synthesis.

  The memory 15 stores image data captured by a digital camera, a control program executed by the CPU 14, an image processing program, and the like. As the memory 15, a nonvolatile semiconductor memory or the like can be used.

  The monitor 16 is a monitor such as a liquid crystal monitor, and displays various setting screens and images according to instructions from the CPU 14.

  A connector for connecting the storage medium 18 is formed in the recording I / F 17. Then, the recording I / F 17 writes / reads data to / from the storage medium 18 connected to the connector. The storage medium 18 includes a hard disk, a memory card having a built-in semiconductor memory, and the like. FIG. 1 shows a memory card as an example of the storage medium 18.

  The operation member 19 includes, for example, a command dial, a cross-shaped cursor key, an enter button, and the like. The operation member 19 receives various inputs of the digital camera from the user. For example, the operation member 19 is used for a switching operation of an operation mode of the digital camera, an input operation on a setting screen, and the like.

  The release button 20 receives from the user an instruction input for starting an AF operation before imaging by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

  The touch panel 21 detects the position of the stylus (or fingertip or the like) that has touched the panel surface, and outputs the detected position information to the CPU 14 to receive an instruction input from the user. The configuration of the touch panel 21 may be any of a capacitance type and a pressure-sensitive type. The touch panel 21 of the present embodiment is configured by a transparent panel having the same shape as the monitor 16, and is stacked and disposed on the entire surface of the monitor 16. The user inputs an instruction to the CPU 14 via the touch panel 21 while visually recognizing the button of the GUI format displayed on the monitor 16.

  Next, the operation of image processing by the digital camera of the present embodiment will be described with reference to the flowchart of FIG. In the following, it is assumed that the user has previously set the shooting mode to the panorama mode via the operation member 19.

  When the power button of the operation member 19 is pressed by the user, the CPU 14 turns on the power to the digital camera and initializes each unit. The CPU 14 causes the image sensor 12 to start capturing a through image when in the imaging standby mode in the image capturing mode. The CPU 14 causes the image sensor 12 to start capturing a moving image of the object scene in the panorama mode in response to an imaging instruction of a full-press operation of the release button 20 by the user. At the same time, the user starts a panning operation of the digital camera. In the present embodiment, the user is described as panning the digital camera in the horizontal direction. However, the same applies to the case in the vertical direction, and the description is omitted.

  Further, when starting capturing a moving image in the panoramic mode, the CPU 14 switches the display on the monitor 16 from the display shown in FIG. 3A to the display shown in FIG. 3B. In other words, as shown in FIG. 3B, the CPU 14 reduces the frame of the captured moving image to the display area 40 arranged on the right side of the display range 30 of the monitor 16 and displays the frame in real time instead of the through image. On the other hand, the CPU 14 displays a part of a panorama image sequentially generated by a panorama synthesis process described later in a display area 50 arranged on the left side of the display range 30 in real time. However, as shown in FIG. 3B, at the start of the moving image capturing, there is no panorama image subjected to the panorama synthesis processing, so that nothing is displayed in the display area 50.

  Step S101: The CPU 14 reads the first frame output from the image sensor 12 via the AEF 13, and sets the first frame as a reference image for panorama synthesis processing. Then, the CPU 14 reads the second frame and sets it as a target image to be joined to the composite image.

  Step S102: The image processing unit 22 extracts image feature amounts such as edge amounts and color information of the target image and the composite image by known image processing. The image processing unit 22 can detect the movement amount and the pan direction of the target image with respect to the composite image by the pan operation of the digital camera by the user by performing a correlation process between the target image and the composite image based on the extracted image feature amount. Is determined. The image processing unit 22 proceeds to step S103 (YES side) when the movement amount and the pan direction are detected, and proceeds to step S106 (NO side) when the movement amount and the pan direction are not detected.

  Step S103: The image processing unit 22 performs a panorama synthesis process of connecting an image area newly captured by the target image to a synthesized image based on the movement amount and the pan direction detected in step S102.

  Step S104: The image processing unit 22 displays the obtained pan direction on the monitor 16 with an arrow 60, and simultaneously slides the composite image panorama-combined in step S103 in the display area 50 in the direction opposite to the pan direction. Display in real time (FIG. 4a). Here, FIG. 4A shows, as an example, the display state of the monitor 16 immediately after the start of the panorama composition processing.

  In the case where the target image is an image that is vertically shifted or obliquely captured with respect to the composite image, the CPU 14 displays the composite image obtained by panoramic composite of the target image, It is preferable that the region is displayed after being trimmed. Alternatively, the CPU 14 may reduce and display the composite image so as to display the protruding image area.

  Step S105: The CPU 14 determines whether or not the imaging end instruction of the full-press operation of the release button 20 by the user is received again. When receiving the imaging end instruction, the CPU 14 ends the moving image imaging in the panorama mode. The image processing unit 22 generates a composite image as a panoramic image file, and records the panoramic image file in the memory 15 or the storage medium 18. The CPU 14 ends a series of processing. In the present embodiment, it is preferable that the CPU 14 generate the panoramic image file by trimming the vertical size of the composite image with the vertical width of the image of the first frame, for example. When the user pans the digital camera in the vertical direction, the CPU 14 preferably trims the horizontal size of the composite image using the horizontal width of the image of the first frame, for example.

  On the other hand, if the CPU 14 has not received the imaging end instruction, the process proceeds to step S101 (NO side), and reads the next captured frame as the target image. The CPU 14 performs the processing of steps S101 to S104 on the target image.

  Step S106: The image processing unit 22 determines the amount of movement of the target image with respect to the synthesized image, such as a case where the panning speed of the digital camera by the user is too fast and the amount of movement is too large, or the posture of the digital camera changes too much. If the pan and pan directions cannot be detected, the detection failure is notified to the user. That is, the image processing unit 22 displays a warning display as shown in FIG. For example, the image processing unit 22 displays the panning direction of the digital camera to the user with an arrow 70 instead of the arrow 60 so that the user can capture a target image whose moving amount can be detected and is combined with the composite image. And instruct. The image processing unit 22 may display a warning such as a symbol 71 or a sentence 72 on the monitor 16 together with the arrow 70. Alternatively, as another example of the warning display, the image processing unit 22 may indicate a region where the target images are to be joined in the composite image as a shaded region 73.

  Note that the CPU 14 may display the above warning message to the user even when the user stops the digital camera for a certain period of time without panning, or pans the digital camera in a direction different from the arrow 60, for example. preferable.

  Step S107: The image processing unit 22 temporarily stores the image feature amount of the composite image extracted in step S102 in, for example, an internal memory (not shown) of the CPU 14. The stored image feature amount of the composite image is the same as that of the target image or 70 to 80% of the image area from the end of the composite image to which the target image is to be joined. The quantity alone may be sufficient. As a result, the memory capacity can be reduced, and the circuit size can be reduced.

  Step S108: The image processing unit 22 causes the user to pan in the direction of the arrow 70, for example, based on the warning display while referring to the frame in the display area 40, and performs the next frame (for example, (Third frame) is read as a new target image.

  Step S109: The image processing unit 22 extracts the image feature amount of the target image read in step S108. The image processing unit 22 can detect the movement amount and the pan direction of the target image for the synthesis processing based on the extracted image feature amount of the target image and the image feature amount of the composite image stored in the internal memory (not shown). Is determined. When the movement amount and the pan direction have been detected, the image processing unit 22 proceeds to step S110 (YES side). On the other hand, when the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S108 (NO side). The image processing unit 22 performs the processing of steps S108 to S109 until the movement amount and the pan direction of the target image for the synthesis processing are detected.

  Step S110: The image processing unit 22 cancels the warning display shown in FIG. 4B and switches to the display shown in FIG. 4C. The image processing unit 22 proceeds to step S103, and restarts the panorama synthesizing process of connecting the target image newly read in step S108 to the synthesized image.

As described above, in the present embodiment, even if the digital camera is greatly panned with respect to the synthesized image, the panorama synthesis is performed by displaying the moving image and the synthesized image obtained by the panorama synthesis in real time. It is possible to easily determine where the panorama synthesis has failed, and easily return the imaging direction of the digital camera to the point where the panorama synthesis has failed. Thus, a panoramic image can be generated with high accuracy without performing the imaging in the panorama mode again from the beginning.
<< Supplementary information of the embodiment >>
(1) In the above embodiment, the example in which the processing of the image processing unit 22 is realized by software in the CPU 14 has been described. However, the processing may be realized by hardware using an ASIC.

  (2) In the above embodiment, the image processing unit 22 detects the moving amount and the panning direction of the target image based on the image feature amounts of the entire image area of the target image and the composite image. Not limited. For example, a margin portion having a predetermined width is set around the target image and the composite image, and the image processing unit 22 performs the target image combining process based on the image feature amount in the image area other than the margin portion. And the pan direction may be detected.

  Further, the digital camera includes an acceleration sensor, an electronic gyro, and the like, and the image processing unit 22 synthesizes based on the movement amount and the pan direction detected by the image processing unit 22 and the movement amount and the pan direction detected by the acceleration sensor and the like. The target image may be joined to the image. This enables more accurate panoramic synthesis.

  (3) In the above embodiment, when the user starts panning of the digital camera, the CPU 14 displays the moving image frame in the right display area 40 in the display range 30 of the monitor 16 in real time, and the left display area Although the synthesized image subjected to the panorama synthesis processing is displayed as a slide at 50, the present invention is not limited to this. For example, as shown in FIG. 5A, the CPU 14 displays a frame of a moving image in real time in the display area 40 ′ above the center of the monitor 16 and displays the synthesized image in the lower display area 50 ′ as the progress of the panorama synthesis processing. It may be displayed in real time in response. FIG. 5B shows an example of a warning display when the detection of the movement amount and the pan direction of the target image with respect to the composite image has failed.

  Prior to the start of imaging in the panorama mode, the CPU 14 receives a designation of an imaging range for panoramic imaging such as 180 degrees or 360 degrees from the user via the operation member 19 or the like, and displays a display area 50 ′ corresponding to the imaging range. May be set. This allows the user to easily recognize to what extent imaging has been completed. Further, when the imaging is completed up to the set imaging range, the CPU 14 may automatically end the imaging in the panorama mode even if the release button 20 is not fully pressed again.

  Further, while the panorama synthesizing process is being performed smoothly, the CPU 14 displays the full-screen of the frame of the moving image in the display range 30 shown in FIG. 3A, and detects the moving amount and the pan direction of the target image for the synthesizing process. If it fails, the display may be switched to the warning display shown in FIG. 4B or FIG. 5B.

  FIG. 6 shows an example of the display on the monitor 16 when the user pans the digital camera in the vertical direction to image a tower, a high-rise building, or the like.

  (4) In the above embodiment, in step S105, the CPU 14 terminates the imaging in the panorama mode when the user presses the release button 20 again and again, but the present invention is not limited to this. At any time during imaging in the panoramic mode, the user can perform a full-press operation of the release button 20, and it is preferable that the CPU 14 can receive an interruption to end the imaging.

  (5) In the above-described embodiment, panorama synthesis is performed by capturing a moving image. However, the present invention is not limited to this, and can be applied to continuous shooting or panorama synthesis by capturing still images one by one. .

  The features and advantages of the embodiments will be apparent from the above detailed description. It is intended that the claims extend to the features and advantages of the embodiments described above without departing from the spirit and scope of the claims. In addition, any person having ordinary knowledge in the technical field should be able to easily conceive any improvements and changes, and there is no intention to limit the scope of the inventive embodiment to the above. And any suitable modifications and equivalents included within the scope disclosed in the above.

11 imaging optical system, 12 imaging element, 13 AFE, 14 CPU, 15 memory, 16 monitor, 17 recording I / F, 18 storage medium, 19 operating member, 20 release button, 21 ... Touch panel, 22 ... Image processing unit

Claims (9)

An imaging unit that captures an image of a subject that has passed through the imaging optical system and generates image data ;
An image processing unit a plurality of images to synthesis based on a plurality of image data generated is captured by the imaging unit while changing the imaging direction of the imaging optical system,
Both an image based on the image data generated by the imaging of the imaging unit, a display unit for displaying the direction image indicating the direction to change the imaging direction of the imaging optical system, and a synthetic image combined by the image processing unit When,
When the imaging direction is changed to a direction different from the direction indicated by the directional image, the display unit, an image based on image data generated by imaging of the imaging unit, the directional image, the composite image, A control unit for displaying a different image,
An imaging device comprising:   An imaging unit that captures an image of a subject that has passed through the imaging optical system and generates image data;
  An image processing unit that synthesizes a plurality of images based on a plurality of image data generated by being imaged by the imaging unit while changing an imaging direction of the imaging optical system;
  A display unit that displays an image based on image data generated by the imaging unit, a direction image indicating a direction in which an imaging direction of the imaging optical system is changed, and a synthesized image synthesized by the image processing unit. When,
  Control for displaying, on the display unit, an image based on image data generated by imaging by the imaging unit, the direction image, and the composite image, based on a change speed for changing the imaging direction. Department and
  An imaging device comprising: The imaging device according to claim 1, wherein
Before SL control section, when unable to synthesize an image by the image processing unit by the imaging direction of the changing speed of the imaging optical system, an image based on the image data on the display unit, and the direction image, the composite image When, the that imaging device to display different images. The imaging device according to any one of claims 1 to 3,
Wherein the display unit, an imaging device for displaying Ri警 tell by the symbol or sentence. The imaging device according to any one of claims 1 to 4,
Wherein the display unit, an imaging device for displaying the progress of the composition processing of the interleaf Naruga image.   The imaging device according to any one of claims 1 to 5,
  An imaging device configured to reduce and display the composite image;   The imaging device according to any one of claims 1 to 5,
  The imaging device, wherein the display unit displays an image based on the image data and an image obtained by reducing the synthesized image.   The imaging device according to any one of claims 1 to 7,
  The imaging device, wherein the display unit simultaneously displays an image based on image data generated by the imaging unit, the directional image, and the composite image.   The imaging device according to any one of claims 1 to 7,
  The imaging device, wherein the display unit displays an image based on image data generated by the imaging unit, the directional image, and the composite image on the same screen.

Images