JP2010263270A - Image pickup device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image pickup device for supporting layout setting when changing an angle of view. <P>SOLUTION: When operation for reducing photography angle of view (zooming operation to increase zoom magnification) is performed during photography of a target moving image, a wide angle image 210 is acquired by performing still image photography before decreasing a photography angle of view, and then the photography angle of view is actually decreased. When a zoom lens for decreasing the photography angle of view is being moved or has been moved, a small angle image 220 at a relatively narrow photography angle of view is obtained. The image pickup device uses block matching or the like to detect a position of a subject on the narrow angle image 220 on the wide angle image 210 and place a contracted image of the narrow angle image 220 onto the wide angle image 210 according to the detection result, so that a display image 230 to be displayed on a display picture is generated. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital camera.

  In an imaging apparatus provided with a zoom function, it is possible to shoot a subject by enlarging or reducing the subject. In the optical zoom, the zoom lens in the optical system is driven in conjunction with the operation of the zoom switch, and the shooting angle of view changes by driving the zoom lens. The user sets the desired angle of view of the shooting angle of view while checking a monitor that displays the current shot image that reflects the current angle of view of the shooting.

  However, due to the time lag between the zoom operation and the lens driving, it is necessary to repeat the enlargement and reduction zoom operations before and after the desired angle of view, and it may be difficult to adjust to the desired angle of view. In addition, when shooting at a high zoom magnification, the image to be shot and the video displayed on the monitor are a small part of the entire shooting scene. It may be difficult to keep supplementing. This can also be understood from the fact that, even with the same amount of camera shake, when the zoom magnification is 30 times, the resulting image blur is 30 times compared to when the zoom magnification is 1 time. Further, if the subject of interest is lost during the zoom operation, it is difficult to place the subject of interest within the photographing range again, especially at a high magnification.

  The following technique is disclosed in Patent Document 1 below. Using image size reduction processing and image cutout processing, still image display data and moving image display data are generated from image data for one image, and a moving image based on moving image display data is added to a still image based on still image display data. The image on which the image is arranged is displayed on the monitor (or the reverse image is displayed on the monitor).

JP 2007-104352 A

  As described above, the conventional imaging apparatus only displays the current captured image that reflects the current shooting angle of view. Depending on the display content, the user performs camera work including zoom operation and pan operation, and performs composition settings such as placing the subject of interest at the center of the shooting range, but the actual composition is desired particularly at a high zoom magnification. It may be difficult to set the composition. In addition, the technique of patent document 1 is not a technique which solves such a subject.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus that supports composition setting related to a change in angle of view.

  A first imaging device according to the present invention includes an imaging unit that captures a still image or a moving image, a display unit, a first image having a first angle of view obtained by imaging, and a time different from the first image. A display control unit that causes the display unit to simultaneously display a second image having a second angle of view obtained by the shooting of the target moving image formed including the second image, The first and second angles of view are different from each other.

  As a result, the captured images before and after the angle of view change can be confirmed on the display unit at the same time, and the actual composition is supported close to the desired composition.

  Specifically, for example, the display control unit changes the display size of the first image and the display size of the second image on the display screen of the display unit, and displays the second image on the first image. The arranged display image is displayed on the display unit during shooting of the target moving image.

  More specifically, for example, the imaging apparatus further includes a position detection unit that detects a position of the subject on the second image on the first image, and the display control unit detects the position detection unit. Based on the result, an arrangement position of the second image on the first image is determined.

  Or, specifically, for example, the imaging apparatus pays attention to each of a plurality of different partial images included in the entire first image and detects a feature size of each partial image. The display control unit determines an arrangement position of the second image on the first image based on a detection result of the feature amount detection unit.

  More specifically, for example, the first image is taken before the second image, the first angle of view is wider than the second angle of view, and the imaging device When receiving an operation for reducing the first angle of view from the first angle of view toward the second angle of view, the first image is photographed prior to the reduction of the photographing angle of view, and then the photographing angle of view is actually set to the first angle of view. The second image is photographed while decreasing from the angle of view to the second angle of view.

  A second imaging device according to the present invention includes an imaging unit that captures a still image or a moving image, a display unit, and a shooting angle of view from the first angle of view to the first angle of view during shooting of the target moving image. When receiving an operation to reduce the angle of view to a narrow second angle of view, the first image having the first angle of view is taken by the imaging unit prior to the reduction of the angle of view, and then the angle of view of the image is actually taken. And a position control for detecting the position of the subject on the second image captured at the second angle of view on the first image. And a display control unit that superimposes and displays an index representing a shooting range of the second image on the first image after the second image is shot during shooting of the target moving image. It is characterized by having.

  As a result, the photographing range before and after the change in the angle of view can be confirmed on the display unit at the same time, and it is supported to bring the actual composition closer to the desired composition.

  In addition, for example, the first or second imaging device may further include a recording control unit that records the first image as the still image and the target moving image on the recording medium in association with each other. For example, the display control unit includes a reproduction control unit that causes the display unit to reproduce a moving image or a still image recorded on the recording medium, and the reproduction control unit performs a predetermined operation for designating the first image. When the image is received, it is preferable that the display unit reproduces a moving image within a predetermined period with reference to the shooting time of the first image in the entire shooting period of the target moving image.

  Characteristic scenes are often photographed before and after the time at which the angle of view changes. Therefore, by making it possible to reproduce a moving image within the predetermined period as described above, the user can find a desired moving image in a short time from a plurality of moving images that can be recorded on the recording medium. It becomes possible.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the imaging device which supports the composition setting which concerns on a view angle change.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part provided in the imaging device of FIG. It is a figure which shows XY coordinate surface as a two-dimensional coordinate surface, and a two-dimensional image. It is a schematic external view of the display part provided in the imaging device of FIG. It is a partial block diagram of the imaging device of FIG. It is a figure showing the relationship between a normal display period and a special display period. 5 is an operation flowchart of the imaging apparatus according to the first embodiment of the present invention, particularly focusing on a display operation during shooting of a target moving image. It is a figure which shows the wide-angle image, narrow-angle image, and display image which concern on 1st Example of this invention. FIG. 6 is a diagram illustrating a manner in which a narrow-angle image for alignment is generated by reducing the image size of a narrow-angle image according to the first embodiment of the present invention. It is a figure which shows the position on the wide-angle image which concerns on 1st Example of this invention and the center point of the superimposing narrow-angle image and the superimposing narrow-angle image should be arrange | positioned. FIG. 6 is a diagram illustrating a manner in which an entire image area such as a display image is classified into two image areas according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating a state in which a display image generation unit is included in the display control unit of FIG. 5 according to the second embodiment of the present invention. It is a figure which shows the image for a display which concerns on 4th Example of this invention. It is a figure which shows a mode that all the image areas, such as a display image, are classified into two image area | regions concerning 4th Example of this invention. It is a figure which shows a mode that the image for a display is produced | generated from a wide angle image and a narrow angle image concerning 5th Example of this invention. It is a figure which shows a mode that all the image areas, such as a display image, are classified into two image area | regions concerning 5th Example of this invention. It is an operation | movement flowchart of the imaging device which concerns on 6th Example of this invention paying special attention to the display operation during imaging | photography of an object moving image. It is a figure which shows the image for a display which concerns on 6th Example of this invention. FIG. 10 is a diagram illustrating a structure of an image file generated in the external memory of FIG. 1 according to an eighth embodiment of the present invention. It is a figure which shows a mode that several thumbnails about several moving images are displayed concerning 8th Example of this invention. It is a figure which shows a mode that the thumbnail of the related still image linked | related with the object moving image is displayed concerning 8th Example of this invention.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to eighth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described. In the present specification, for the sake of simplification of description, names corresponding to the reference numerals may be omitted or abbreviated by referring to the reference numerals. For example, the alignment narrow-angle image 220a (see FIG. 9) is sometimes simply referred to as the image 220a, but both indicate the same image.

[Description of basic configuration]
FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging device 1 is a digital video camera, and can capture a moving image and a still image, and also can simultaneously capture a still image during moving image capturing. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. The display unit 27 and / or the speaker 28 may be provided in an external device (not shown) of the imaging device 1.

  The imaging unit 11 captures a subject using an imaging element. FIG. 2 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging device (solid-state imaging device) 33 including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the optical system 35 and the diaphragm 32. And a driver 34 for drive control. The optical system 35 is formed of a plurality of lenses including a zoom lens 30 for adjusting the angle of view of the imaging unit 11 and a focus lens 31 for focusing. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. Based on a control signal from the CPU 23, the positions of the zoom lens 30 and the focus lens 31 and the opening of the diaphragm 32 in the optical system 35 are controlled.

  The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions. Each light receiving pixel of the image sensor 33 photoelectrically converts an optical image of an object incident through an optical system and a diaphragm, and outputs an electric signal obtained by the photoelectric conversion to an AFE 12 (Analog Front End).

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The amplification degree of signal amplification in the AFE 12 is controlled by a CPU (Central Processing Unit) 23. The video signal processing unit 13 performs necessary image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and records a video signal and an audio signal compressed by the compression processing unit 16.

  The decompression processing unit 19 decompresses the compressed video signal and audio signal read from the external memory 18. The video signal expanded by the expansion processing unit 19 or the video signal from the video signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the audio signal that has been expanded by the expansion processing unit 19 is sent to the speaker 28 via the audio output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, an operation key 26c, and the like. Accept the operation. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The operation mode of the imaging apparatus 1 includes a shooting mode in which an image (still image or moving image) can be shot and recorded, and an image (still image or moving image) recorded in the external memory 18 is reproduced and displayed on the display unit 27. Playback mode to be included. Transition between the modes is performed according to the operation on the operation key 26c.

  In the shooting mode, a subject is periodically shot at a predetermined frame period, and shot images of the subject are sequentially acquired. A digital video signal representing an image is also called image data. Image data for a certain pixel may be referred to as a pixel signal. The pixel signal includes, for example, a luminance signal and a color difference signal. One image is represented by image data for one frame period. One image represented by image data for one frame period is also called a frame image. The size of a certain image or the size of an image area is called an image size. The image size of the target image or target image area can be expressed by the number of pixels forming the target image or the number of pixels belonging to the target image area.

  In the present specification, image data of a certain image may be simply referred to as an image. In addition, since the compression and expansion of image data is not related to the essence of the present invention, the existence of compression and expansion of image data is ignored in the following description. Therefore, for example, recording compressed image data for a certain image is simply expressed as recording image data or recording an image.

  FIG. 3 shows an XY coordinate plane as a two-dimensional coordinate plane on which an arbitrary two-dimensional image is to be arranged. In FIG. 3, a rectangular frame denoted by reference numeral 200 represents an outer shape frame of a two-dimensional image. The XY coordinate plane has an X axis extending in the horizontal direction of the two-dimensional image 200 and a Y axis extending in the vertical direction of the two-dimensional image 200 as coordinate axes. All images described herein are two-dimensional images unless otherwise specified. The position of a certain point of interest on the XY coordinate plane and the two-dimensional image 200 is represented by (x, y). x represents the X-axis coordinate value of the target point, and also represents the horizontal position of the target point on the XY coordinate plane and the two-dimensional image 200. y represents the Y-axis coordinate value of the target point, and also represents the vertical position of the target point on the XY coordinate plane and the two-dimensional image 200. In the XY coordinate plane and the two-dimensional image 200, as the value of x that is the X-axis coordinate value of the attention point increases, the position of the attention point moves to the right side, and the value of y that is the Y-axis coordinate value of the attention point is changed. It is assumed that the position of the attention point moves upward as it increases.

  FIG. 4 shows a schematic external view of the display unit 27. The display unit 27 includes a display screen 27 a such as a liquid crystal display and a display unit housing 27 b that is a part of the housing of the imaging device 1. Frame images periodically obtained at a predetermined frame period are updated and displayed on the display screen 27a. That is, a collection of frame images obtained sequentially is displayed as a moving image on the display screen 27a. The user can check the shooting range of the imaging unit 11 (in other words, the field of view of the shooting of the imaging unit 11) by viewing the display content.

  FIG. 5 shows a block diagram of a portion particularly related to the characteristic function of the imaging apparatus 1. Each part shown in FIG. 5 is inherent in the imaging apparatus 1. In FIG. 5, the imaging unit 11, the display unit 27, and the external memory 18 are the same as those in FIG. The imaging control unit 41 is realized by the CPU 23, for example. The display control unit 42 is realized by the display processing unit 20 or by the display processing unit 20 and the CPU 23, for example. The recording control unit 43 is realized by, for example, the CPU 23 or a memory driver (not shown) provided in front of the CPU 23 and the external memory 18. The image processing unit 44 is realized, for example, by the video signal processing unit 13 or by the video signal processing unit 13 and the CPU 23.

  The imaging control unit 41 controls the imaging operation of the imaging unit 11, and in particular controls the optical zoom magnification by driving the zoom lens 30. As the optical zoom magnification is increased from 1 ×, which is the minimum magnification, the angle of view (hereinafter simply referred to as “shooting angle of view”) of the imaging unit 11 is reduced and the imaging range (in other words, the field of view) of the imaging unit 11 is also reduced. On the other hand, the size of each subject on the frame image increases. Conversely, as the optical zoom magnification is decreased from a certain magnification to 1 ×, the shooting angle of view increases and the shooting range (in other words, the field of view) of the imaging unit 11 increases. The subject size decreases. By changing the position of the zoom lens 30 within the movable range of the zoom lens 30, the optical zoom magnification changes from a predetermined minimum magnification to a predetermined maximum magnification.

  The user can instruct the operation unit 26 to change the optical zoom magnification by performing a predetermined zoom operation. The zoom operation is divided into an enlargement zoom operation for increasing the optical zoom magnification and a reduction zoom operation for reducing the optical zoom magnification. The imaging control unit 41 controls the optical zoom magnification according to the contents of the zoom operation.

  The display control unit 42 controls the display content of the display unit 27. An image displayed on the display screen 27a of the display unit 27 is particularly called a display image. The recording control unit 43 performs recording control of the external memory 18. The image processing unit 44 performs various image processing using the image data of the frame image. Details of the functions of the display control unit 42, the recording control unit 43, and the image processing unit 44 will be described later.

  Below, the 1st-8th Example is described as an Example related to the characteristic function which the imaging device 1 has. If no contradiction arises, the matters described in one embodiment among the first to eighth embodiments can be applied to other embodiments, and two or more embodiments can be combined. .

<< First Example >>
A first embodiment will be described. First, the outline of the operation according to the first embodiment will be described with reference to FIG. A moving image to which a characteristic operation according to the present invention is applied is particularly called a target moving image. The target moving image is, for example, a moving image to be photographed according to an operation on the recording button 26a. The shooting period of the target moving image is roughly divided into a normal display period and a special display period.

  In the normal display period, each frame image itself forming the target moving image functions as a display image, and the latest frame images obtained sequentially are updated and displayed using the entire display area of the display screen 27a. Such a display is also called a normal display.

  In a normal display period, an image obtained by performing predetermined image processing (for example, edge enhancement processing or noise removal processing) on the frame image, or an arbitrary index (for example, remaining recording capacity of the external memory 18 or driving of the imaging device 1) is applied to the frame image. An image in which an index indicating the remaining battery level) is superimposed, or an image in which the image processing and the index are superimposed on a frame image may be displayed as a display image. This is also true for the display image during the special display period. However, in the following description, for the sake of simplification of description, the existence of such image processing and indices that are not highly relevant to the characteristics of the present invention is ignored.

  The time immediately after the start of capturing the target moving image belongs to the normal display period. When an enlargement zoom operation is performed after the start of capturing the target moving image, the capturing control unit 41 causes the image capturing unit 11 to capture a still image before increasing the optical zoom magnification by moving the zoom lens 30. The still image taken here is particularly called a wide-angle image. The wide-angle image is a single frame image that forms the target moving image. Alternatively, the image obtained by performing predetermined image processing on the frame image may be a wide-angle image.

  The photographing control unit 41 actually increases the optical zoom magnification by moving the zoom lens 30 after the photographing of the wide-angle image is completed, and stops the zoom lens 30 at a position where a desired optical zoom magnification is obtained. A special display period is provided after the wide-angle image is obtained. Although the end of the special display period will be described later, when the special display period ends, the normal display period starts again.

  With reference to FIG. 7, a display operation during shooting of the target moving image will be described. FIG. 7 is a flowchart of the operation of the imaging apparatus 1 with particular attention paid to the display operation. When the start of capturing the target moving image is instructed, the imaging unit 11 sequentially captures the frame images forming the target moving image at a predetermined frame period from the start of capturing until the end of capturing of the target moving image is instructed. Is done. The recording control unit 43 can record each frame image forming the target moving image in the external memory 18.

  After the start of capturing the target moving image, in step S11, it is confirmed whether or not the zoom operation has been performed by the CPU 23. If there is no zoom operation, the confirmation operation in step S11 is repeatedly executed, while the zoom operation is performed. Shifts to step S12 to confirm whether the zoom operation is an enlargement zoom operation. When the zoom operation performed on the operation unit 26 is an enlargement zoom operation, the process proceeds from step S12 to step S13, and the process of steps S13 to S17 is sequentially performed before reaching step S18. When the zoom operation performed on the operation unit 26 is a reduction zoom operation, the process proceeds from step S12 to step S20, and the process of steps S20 and S21 is sequentially performed before the process proceeds to step S18.

  When the zoom operation performed on the operation unit 26 is a reduction zoom operation, the shift from the normal display period to the special display period is not performed, and in step S20, the movement of the zoom lens 30 according to the reduction zoom operation is immediately started. In step S21, the latest frame image is normally displayed. That is, the frame image after the optical zoom magnification is reduced according to the reduction zoom operation is normally displayed. Thereafter, the process proceeds to step S18.

  When the zoom operation performed on the operation unit 26 is an enlargement zoom operation, a still image is captured as a wide-angle image in step S13 before the zoom lens 30 is moved in accordance with the enlargement zoom operation, and the subsequent steps. In S14, the movement of the zoom lens 30 according to the enlargement zoom operation is started. The movement of the zoom lens 30 continues until the optical zoom magnification according to the enlargement zoom operation is obtained. FIG. 8A shows a wide-angle image 210 as an example of the wide-angle image obtained in step S13. The wide-angle image 210 is an image obtained by shooting so that three persons standing in front of the forest fall within the shooting range. The significance of the broken line frame 211 will be described later. The wide-angle image data obtained in step S13 is stored in the internal memory 17 for the required time.

  In step S15 subsequent to step S14, the latest frame image and the wide-angle image obtained in step S13 are aligned. The latest frame image here is a frame image in the middle of movement of the zoom lens 30 for increasing the optical zoom magnification or a frame image obtained after the movement is completed. The angle of view at is narrower than that of wide-angle images. Therefore, the latest frame image to be aligned is particularly called a narrow-angle image. FIG. 8B shows a narrow-angle image 220 as an example of the narrow-angle image that is the object of alignment in step S15. The narrow-angle image 220 is an image taken after increasing the optical zoom magnification and adjusting the shooting range so that the face of the person located at the center among the three people is projected larger. This is an image obtained by enlarging and photographing the subject in the broken line frame 211 shown in (a). The contents of alignment will be described later. The alignment is performed by the image processing unit 44 in FIG.

  In step S16, a display image is generated by arranging a narrow-angle image on the wide-angle image based on the alignment result in step S15. At this time, the image size of the narrow angle image is reduced, and the narrow angle image after the image size reduction is arranged on the wide angle image. FIG. 8C shows a display image 230 generated based on the wide-angle image 210 and the narrow-angle image 220 shown in FIGS. 8A and 8B. The display image 230 is generated by disposing a reduced image of the narrow-angle image 220 (a superimposing narrow-angle image 220b described later; see FIG. 10) on the wide-angle image 210 so that the wide-angle image 210 is the background. . On the display image 230, a boundary frame 231 representing the outer shape of a reduced image of the narrow-angle image 220 (a superimposing narrow-angle image 220b described later; see FIG. 10) is also shown. The process of generating the display image from the wide-angle image and the narrow-angle image is performed by the display control unit 42 in FIG. However, the processing may be performed by the image processing unit 44.

  The display image generated in step S16 is displayed on the display screen 27a. A period during which the display image based on the wide-angle image and the narrow-angle image is displayed corresponds to the special display period. In step S17 following step S16, the display control unit 42 determines whether or not to end the special display period. If it is determined that the special display period should be ended, the process proceeds from step S17 to step S18. If not, the process returns to step S15 and the processes of steps S15 and S16 are repeatedly executed. Simply, for example, the time point when the wide-angle image is captured, the time point when the zoom lens 30 starts moving according to the enlargement zoom operation, or the time point when the movement of the zoom lens 30 according to the enlargement zoom operation ends is the start time of the special display period. The special display period can be ended when a certain time has elapsed from the start of the special display period. In the process of repeatedly executing steps S15 and S16, the image on the display screen 27a where the wide-angle image is displayed is a still image, but the image on the portion where the superimposing narrow-angle image is displayed. Is a moving image.

  In step S18 which moves from step S17 or step S21, it is confirmed whether or not the end of shooting of the target moving image has been instructed. When the end of shooting of the target moving image is instructed, the shooting of the target moving image is ended. However, when the end of shooting of the target moving image is not instructed, the process returns to step S11 and the processes of the above steps are repeatedly executed. The

The significance of the alignment in step S15, a method for determining the arrangement position of the narrow-angle image on the wide-angle image using the alignment, and a method for generating the display image according to the determination will be described. For clarity of explanation, attention is focused on the wide-angle image 210 and the narrow-angle image 220 shown in FIGS. 8A and 8B as the wide-angle image and the narrow-angle image. Further, it is assumed that the optical zoom magnification at the time of photographing the wide-angle image 210 is Z _W and the optical zoom magnification at the time of photographing the narrow-angle image 220 is Z _N. Here, 1 ≦ Z _W <Z _N. When the optical zoom magnification is k times, the size of the subject of interest on the frame image is k times in each of the horizontal and vertical directions (therefore, the size of the subject of interest on the frame image is k ² times as an area ratio). Become).

There are various methods for alignment, and a method using block matching will be described. The image sizes of the wide-angle image 210 and the narrow-angle image 220 obtained by shooting (that is, the number of pixels in the horizontal and vertical directions) are the same. It shrinks to (Z _W / Z _N ) in each of the vertical directions. The narrow-angle image after the image size reduction is referred to as an alignment narrow-angle image, and as shown in FIG. 9, the alignment narrow-angle image for the narrow-angle image 220 is referred to by reference numeral 220a. The process of generating the alignment narrow-angle image from the narrow-angle image is realized by using a known resolution conversion (for example, resampling or thinning process using interpolation). For example, if Z _W = 1 and Z _N = 2, the number of pixels of the narrow-angle image 220 is reduced to ½ in each of the horizontal and vertical directions using a thinning process, resulting in an area or number of pixels. The image size of the alignment narrow-angle image 220 a is ¼ of the image size of the narrow-angle image 220.

Next, a search frame having the same image size as the image size of the image 220a is set on the wide-angle image 210, and the similarity between the image in the search frame and the image 220a is evaluated. The similarity between two images to be compared can be evaluated using SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) between pixel signals of the two images. In general, a luminance signal is used as a pixel signal when calculating SSD or SAD. The evaluation of the similarity is sequentially executed while shifting the center position in the search frame by one pixel in the horizontal or vertical direction. Then, the center position (x _A , y _A ) in the search frame where the maximum similarity is obtained is obtained.

When tracking processing for tracking the position and size of the subject of interest is performed in the target moving image, the center position (x _A , y _A ) is obtained using the result of the tracking processing. Also good. However, the subject of interest needs to be included in both the wide-angle image and the narrow-angle image. The position (x _A , y _A ) can be obtained based on the center positions of the subject of interest on the wide-angle image 210 and the narrow-angle image 220 obtained by the tracking process and the optical zoom magnifications Z _W and Z _N. . The tracking process can be performed by, for example, the image processing unit 44.

The position (x _A , y _A ) refers to the position where the image data of the subject located at the center of the narrow-angle image 220 and the center of the image 220 a is present on the wide-angle image 210. If the subject at the center position of the narrow-angle image 220 or the image 220a is called a center subject, the position of the center subject on the narrow-angle image 220 or the image 220a on the wide-angle image 210 is (x _A , y _A ). . The arrangement position of the narrow angle image on the wide angle image is determined by the position (x _A , y _A ). The alignment can be said to be a process for deriving the center position (x _A , y _A ).

  On the other hand, the display control unit 42 generates the superimposing narrow-angle image 220b by reducing the image size of the narrow-angle image 220 (see FIG. 10). The process of generating the superimposing narrow-angle image from the narrow-angle image is also realized by using known resolution conversion (for example, resampling or thinning processing using interpolation). In FIG. 10, the point 221 shown on the superimposing narrow-angle image 220 b is arranged at the center position of the superimposing narrow-angle image 220 b that matches the point to be arranged at the center position of the narrow-angle image 220. Is a point.

The display control unit 42 arranges the superimposing narrow-angle image 220b on the wide-angle image 210 so that the point 221 is arranged at the position (x _A , y _A ) on the wide-angle image 210, thereby displaying the display image 230 ( (See FIG. 8C).

More specifically, for example, as shown in FIG. 11, the entire image area of the wide-angle image or the display image is centered on the position (x _A , y _A ) and the superimposing narrow-angle image (in this example, the image 220b). Are classified into a first image area having the same image size and a second image area that is the other image area. In FIG. 11, the area filled with diagonal lines corresponds to the first image area, and the area filled with dots corresponds to the second image area. Then, an image obtained by combining the image in the second image region of the wide-angle image 210 and the superimposing narrow-angle image 220 b is generated as the display image 230. In this case, the image in the first image region of the display image 230 is the superimposing narrow-angle image 220b itself, but the image in the first image region of the wide-angle image 210 and the superimposing narrow-angle image 220b are blended ( An image in the first image area of the display image 230 may be generated by performing alpha blending.

  As described above, when the display image is generated, the image size of the narrow-angle image is reduced. Therefore, the display size of the wide-angle image on the display screen 27a is different from the display size of the narrow-angle image, and the wide-angle image is displayed on the display screen 27a. The outer frame of the narrow-angle image (strictly speaking, the narrow-angle image for superimposition) fits in the outer frame.

  The image size of the superimposing narrow-angle image 220b can be determined by any one of a plurality of image size determination methods described below.

  In the first image size determination method, the image size of the superimposing narrow-angle image 220b is set to a predetermined constant size. In this case, the reduction ratio when generating the superimposing narrow-angle image 220b from the narrow-angle image 220 is always constant.

In the second image size determination method, the image size of the superimposing narrow-angle image 220b is set to the optical zoom magnification Z _W when the wide-angle image 210 is photographed, the optical zoom magnification Z _N when the narrow-angle image 220 is photographed, or these It is determined based on the ratio (Z _W / Z _N ). Data for determining the image size of the overlapping narrow-angle image 220b from Z _W , Z _N , or (Z _W / Z _N ) can be determined in advance.

  In the third image size determination method, the image size of the superimposing narrow-angle image 220b is determined based on the size of the specific subject detected in the specific subject detection process on the image. For example, the image size of the overlapping narrow-angle image 220b is determined so that the size of the specific subject on the display image is equal to or larger than a predetermined reference size.

  The specific subject detection process can be executed by the image processing unit 44 of FIG. In the specific subject detection process, the size of a specific type of subject on the input image is detected based on the image data of the input image. As the input image, the narrow-angle image 220 or the superimposing narrow-angle image 220b is used. An image in the first image area of the wide-angle image 210 can also be used as an input image.

  The specific type of subject is typically a person or a person's face, for example. When the specific type of subject is a person's face, the specific subject detection process is a so-called face detection process. That is, when the specific type of subject is a person's face, the face of the person appearing on the input image is detected based on the input image, and a face area that is an image area representing the face is extracted from the input image. The image size of the superimposing narrow-angle image 220b can be determined so that the image size of the extracted face area is equal to or larger than a predetermined reference size on the display image 230.

  Of course, the specific type of subject may be other than a person and a person's face (for example, a car or the like). When tracking processing for tracking the position and size of a specific type of subject in the target moving image is performed, the image size of the superimposing narrow-angle image 220b is determined using the tracking processing result. good.

  As described in the description of the prior art, when shooting at a relatively high optical zoom magnification, the shot image and the video displayed on the display unit are a small part of the entire shooting scene. In some cases, it may be difficult to keep the subject of interest within the shooting range while performing the zoom operation. Further, if the subject of interest is lost during the zoom operation, it is difficult to place the subject of interest within the photographing range again, especially at a high magnification. In view of these circumstances, in the first embodiment, a wide-angle image obtained prior to the increase of the optical zoom magnification is used as a background, and a narrow-angle image (narrow for superimposition) obtained after the optical zoom magnification is increased on the wide-angle image. Corner image). As a result, the user can easily recognize which part of the entire shooting scene is currently being shot, supports composition setting by camera work, and the subject of interest during and after the zoom operation. It is supported that we continue to grasp steadily.

  In the description of the flowchart of FIG. 7, it is described that the latest frame image is simply displayed when the zoom operation performed on the imaging apparatus 1 is the reduction zoom operation (see step S21). When the display image immediately before the display image is the display image 230 of FIG. 8C, a display image based on the wide-angle image 210 and the latest frame image may be displayed. Specifically, for example, the latest frame image is regarded as a narrow-angle image, and a display image is generated by arranging a reduced narrow-angle image obtained by reducing the image size of the narrow-angle image on the wide-angle image 210. May be. As a method for determining the arrangement position of the reduced narrow-angle image based on the latest frame image on the wide-angle image 210, it is possible to apply that of the above-described superimposing narrow-angle image. The display size of the reduced narrow-angle image based on the latest frame image may be the same as that of the display image 230 described above, but the display size may be increased as the optical zoom magnification decreases.

<< Second Example >>
A second embodiment will be described. The second embodiment corresponds to a modification of the first embodiment, and the description of the first embodiment applies to the second embodiment as long as there is no contradiction regarding matters not specifically described in the second embodiment.

  For clarity of explanation, the wide-angle image itself obtained by photographing with the imaging unit 11 is particularly referred to as an original wide-angle image, and the narrow-angle image itself obtained by photographing with the imaging unit 11 is particularly referred to as an original narrow-angle image. A wide-angle image 210 in FIG. 8A is an original wide-angle image, and a narrow-angle image 220 in FIG. 8B is an original narrow-angle image. An image obtained by photographing by the imaging unit 11 is a color image, and image data of the image includes luminance information and color information. As shown in FIG. 12, the display image generation unit 45 included in the display control unit 42 generates a display image from the original wide-angle image and the original narrow-angle image in the special display period.

  In the second embodiment, first to fifth display controls related to a method for generating a display image and a method for determining a transition timing from a special display period to a normal display period will be described. Any of the first to fifth display controls can be implemented in combination with the operation described in the first embodiment. Hereinafter, the first to fifth display controls will be described individually. However, as long as there is no contradiction, two or more display controls of the first to fifth display controls can be combined and executed. In the description of each display control, it is assumed that the original wide-angle image and the original narrow-angle image are the wide-angle image 210 and the narrow-angle image 220, respectively, for the sake of concrete description.

[First display control]
The first display control will be described. The first display control is the same display control as that described in the first embodiment. That is, a superimposing narrow-angle image 220b is generated by reducing the image size of the original narrow-angle image 220 (see FIG. 10), and the superimposing narrow-angle image 220b is arranged on the original wide-angle image 210 for display. An image 230 is generated (see FIG. 8C). Therefore, the image in the second image region of the display image in the first display control is the image itself in the second image region of the original wide-angle image 210 (see FIG. 11), and unlike other display controls described later, A process for reducing the luminance of the background portion of the image is not performed. The background portion of the display image refers to an image portion in the second image area of the display image.

  Then, the special display period ends when a predetermined holding time elapses from the start point of the special display period, and thereafter, the normal display of the latest frame image (that is, the latest original narrow-angle image) is performed. Like that.

[Second display control]
The second display control will be described. In the second display control, a luminance reduction process for reducing the luminance of the original wide-angle image 210, a gray-scale process for converting the original wide-angle image 210 into a gray scale, or a low-low process for reducing the contrast of the original wide-angle image 210. A processed wide-angle image is generated from the original wide-angle image 210 by contrast processing. Then, the display-use image is generated by arranging the superimposing narrow-angle image 220b on the processed wide-angle image. The method of arranging the image 220b on the processed wide-angle image is the same as the method of arranging the image 220b on the original wide-angle image 210.

  As the luminance reduction process, the gray scale process, and the low contrast process, it is possible to use any process that realizes the functions that should be performed. For example, in the luminance reduction process, the luminance of each pixel of the original wide-angle image 210 is reduced by a predetermined amount, or the original wide-angle image 210 so that the average luminance of the original wide-angle image 210 is lower than the average luminance of the superimposing narrow-angle image 220b. The luminance value of each pixel of the image 210 is multiplied by a constant value less than 1. The luminance value of a certain pixel is a value indicating the brightness of the pixel, and the brightness of the pixel increases as the value increases. Further, for example, in the gray scale processing, the original wide angle image 210 is converted into a gray scale image having only luminance information, and the gray scale image is generated as a processed wide angle image. Further, for example, in the low contrast processing, the original wide-angle image 210 is filtered by using a filter that smoothes the image to reduce the contrast of the original wide-angle image 210, and the result of the filtering is generated as a processed wide-angle image. .

  Note that the brightness reduction process, the gray scale process, or the low contrast process may be performed only on the image in the second image area of the original wide-angle image 210. In addition, the processed wide-angle image may be generated by performing two or more processes among the luminance reduction process, the gray scale process, and the low-contrast process on the original wide-angle image 210.

  Also in the second display control, as in the first display control, the special display period can be ended when a predetermined holding time elapses from the start time of the special display period. After the end of the special display period, the latest frame image (that is, the latest original narrow-angle image) is normally displayed.

  During the special display period, the image of the background portion of the display image can be kept constant, but the image can be changed as time passes.

  For example, when the luminance reduction process or the low contrast process is used, the degree of reduction in luminance or contrast when generating a processed wide-angle image from the original wide-angle image 210 is increased as time elapses from the start point of the special display period. You may do it. As a result, the background portion of the display image becomes darker or lower in contrast with the passage of time.

  Further, for example, when the gray scale processing is used, instead of generating a gray scale image from the original wide angle image 210 at once, the saturation and hue of the original wide angle image 210 as time passes from the start of the special display period. Is gradually reduced, and finally the gray scale image is generated as a processed wide-angle image. In the process of reducing the saturation and hue of the original wide-angle image 210, an image for display may be generated using the image obtained by reducing the saturation and hue of the original wide-angle image 210 as a processed wide-angle image. .

  When the display image 230 (see FIG. 8C) is displayed, the photographer is paying attention to the subject in the superimposing narrow-angle image 220b. Therefore, it is not preferable that the background image is too conspicuous. Considering this, the second display control provides a display that makes it easy to see the subject of interest. In addition, during the special display period, there may be a deviation between the background image of the display image and the actual image, and the deviation tends to increase with time. In consideration of this, processing such as gradually darkening the background portion as time passes may be performed as described above.

[Third display control]
The third display control will be described. In the third display control, the display size of the superimposing narrow-angle image is gradually increased as time elapses during the special display period.

  That is, for example, the degree of reduction of the image size when generating the narrow-angle image for superimposition by reducing the image size of the original narrow-angle image is gradually increased as the elapsed time from the start point in the special display period increases. Decrease. If the degree of reduction decreases, the image size of the superimposing narrow angle image (in other words, the display size of the superimposing narrow angle image) increases. Then, the special display period ends when a predetermined holding time elapses from the start point of the special display period, and thereafter, the normal display of the latest frame image (that is, the latest original narrow-angle image) is performed. Like that.

  As described in the second control method, a deviation may occur between the background image of the display image and the actual image during the special display period, and the deviation tends to increase with time. is there. It is not preferable to display an image with a large shift for a long time using a large display area. The third display control corresponds to such a situation.

[Fourth display control]
The fourth display control will be described. Although it has been described that the alignment between the wide-angle image and the narrow-angle image is performed in step S15 in FIG. 7, the reliability of the alignment is used in the fourth display control.

The alignment reliability represents the certainty of the performed alignment. In the case of performing alignment using block matching as described in the first embodiment, for example, the alignment reliability can be obtained from the similarity calculated at the time of executing block matching. More specifically, for example, the smaller the size of the SSD or SAD corresponding to the position (x _A , y _A ), the higher the alignment reliability required. The reliability of alignment can be obtained so that the reliability of alignment can be reduced.

  In addition, performing alignment between the wide-angle image and the narrow-angle image is substantially equivalent to obtaining a motion vector representing a positional shift between the centers of the wide-angle image and the narrow-angle image, and the motion vector is obtained. The alignment can be performed. In view of this, the reliability of the motion vector representing the positional deviation between the wide-angle image and the narrow-angle image is obtained using a known method (for example, the method described in Japanese Patent Application Laid-Open No. 2008-60927), and The reliability of the motion vector may be used as the alignment reliability. In addition, in the case of performing alignment using the above-described tracking process, the reliability of the tracking process becomes the reliability of alignment.

  The following display control can be performed according to the above-described alignment reliability.

  For example, the luminance or contrast of the background portion of the display image during the special display period can be changed according to the alignment reliability. That is, for example, the luminance reduction process or the low contrast process described above may be executed such that the lower the alignment reliability, the lower the luminance or contrast of the background portion of the display image during the special display period. Good. If the alignment reliability is relatively low, there is a high possibility that the display image based on the alignment result is not the desired display image. The narrow-angle image to be superimposed should be conspicuous.

  Further, for example, the image size of the superimposing narrow angle image (in other words, the display size of the superimposing narrow angle image) may be changed in accordance with the reliability of alignment. That is, for example, the image size of the narrow-angle image for superimposition may be increased as the alignment reliability is lower. As a result, the lower the alignment reliability, the larger the area occupied by the superimposing narrow-angle image in the display image, and the display image to be displayed approaches the original narrow-angle image (the display to be displayed). Image is close to that in normal display). As described above, when the alignment reliability is relatively low, there is a high possibility that the display image based on the alignment result is not a desirable display image, so that the display image to be displayed is brought close to that in the normal display. .

  Further, for example, the color of the border frame 231 of the superimposing narrow-angle image may be changed in accordance with the alignment reliability in order to notify the photographer of the high or low alignment reliability (FIG. 8 (c). )reference).

  Further, for example, the length of the special display period (that is, the length of the holding time) may be changed according to the reliability of alignment. That is, for example, the length of the special display period may be shortened as the alignment reliability is lower. When the alignment reliability is relatively low, there is a high possibility that the display image based on the alignment result is not a desirable display image. Therefore, the length of the special display period is shortened and the normal display is quickly restored.

  Further, for example, when the alignment reliability is lower than a predetermined lower limit reliability, the length of the special display period may be set to zero. That is, when the alignment reliability is lower than the predetermined lower limit reliability, the latest images that are sequentially obtained without performing display of the display image based on the wide-angle image and the narrow-angle image even if an enlargement zoom operation is performed. These frame images may be normally displayed.

[Fifth display control]
The fifth display control will be described. When the optical zoom magnification is relatively high, image blur caused by camera shake acting on the imaging device 1 is relatively large. When the blurring of the image increases, the reliability of alignment tends to decrease.

  Considering this, when the optical zoom magnification specified by the user's enlargement zoom operation is larger than the predetermined upper limit magnification, the length of the special display period is set to zero as described in the fourth display control. You may make it do. In that case, the user may be warned that image blur due to camera shake becomes considerably large. In that case, the actual enlargement of the zoom magnification may be limited to the above upper limit magnification regardless of the contents of the user's enlargement zoom operation.

<< Third Example >>
A third embodiment will be described. In the first and second embodiments, the position (x _A , y _A ) is determined based on the wide-angle image 210 and the narrow-angle image 220. It is also possible to determine the position (x _A , y _A ) using the figure.

The motion sensor is, for example, an angular velocity sensor that detects the angular velocity of the housing of the imaging device 1 or an acceleration sensor that detects the acceleration of the housing of the imaging device 1, and is also used as a sensor for detecting so-called camera shake. The Based on the detection result of the motion sensor during a certain attention period and the state of change in the optical zoom magnification during the attention period, what trajectory the point image of an arbitrary stationary point in the real space is on the image sensor 33 during the attention period. You can derive what to draw. A period between the shooting time of the wide-angle image 210 and the shooting time of the narrow-angle image 220 (more specifically, for example, between the intermediate time of the exposure period of the wide-angle image 210 and the intermediate time of the exposure period of the narrow-angle image 220) ) Can be determined from the derivation result of the trajectory, and the position (x _A , y _A ) can be determined.

<< 4th Example >>
A fourth embodiment will be described. The fourth embodiment corresponds to a modification of the first embodiment, and the description of the first embodiment applies to the fourth embodiment as long as there is no contradiction regarding matters not specifically described in the fourth embodiment.

  In the first embodiment, the arrangement position of the narrow-angle image for superimposition on the display image is determined from the alignment result between the wide-angle image and the narrow-angle image. In the fourth embodiment, the wide-angle image and the narrow-angle image are determined. The arrangement position is determined by searching a partial image area with few features from the entire image area of the wide-angle image without depending on the alignment result between the two. This determination can be performed by the display control unit 42 or the image processing unit 44 in FIG.

  A specific method for determining the arrangement position will be described. The image processing unit 44 in FIG. 5 sets an evaluation block having the same image size as the superimposing narrow-angle image 220b on the wide-angle image 210, and an image in an evaluation block that is a partial image of the wide-angle image 210 (hereinafter, evaluation) A feature evaluation value representing the size of the feature of the evaluation image is derived.

  For example, by performing spatial filtering using an edge extraction filter on each pixel of the evaluation block, the gradient of the pixel signal at the position of each pixel of the evaluation block is obtained, and the sum of the obtained gradient magnitudes is used as the feature evaluation value To derive. The gradient of the pixel signal is, for example, the gradient of the luminance signal in the horizontal and vertical directions of the evaluation block, and this can also be referred to as edge strength in the horizontal and vertical directions. Alternatively, for example, the standard deviation in the luminance value of each pixel in the evaluation block may be derived as the feature evaluation value.

  The feature evaluation value obtained in this way represents the size of the feature included in the evaluation image, and the more the feature size of the evaluation image increases as the evaluation image includes more edges, the more the feature evaluation value corresponds. The feature evaluation value increases, and conversely, the smaller the number of edges included in the evaluation image and the smaller the feature size of the evaluation image, the smaller the corresponding feature evaluation value.

The image processing unit 44 derives a feature evaluation value for each movement while moving the center position of the evaluation block one pixel at a time in the horizontal or vertical direction on the wide-angle image 210. Thus, m feature evaluation values for m evaluation images having different center positions are obtained (m is an integer of 2 or more). Assume that m feature evaluation values are composed of first to m-th feature evaluation values, and among m evaluation images, an evaluation image corresponding to an i-th feature evaluation value is referred to as an i-th evaluation image ( i is an integer of 1 to m. The center position of the i-th evaluation image on the wide-angle image 210 is represented by (x _i , y _i ).

The display control unit 42 sets the center position of the evaluation image corresponding to a relatively small feature evaluation value among the first to m-th feature evaluation values as the position (x _B , y _B ). The display control unit 42 arranges the point 221 arranged at the center position of the superimposing narrow angle image 220b (see FIG. 10) at the position (x _B , y _B ) on the wide angle image 210 and the display image. The display image is generated by disposing the superimposing narrow-angle image 220b on the wide-angle image 210.

  FIG. 13 shows a display image 300 according to the fourth embodiment, which is generated based on the wide-angle image 210 and the narrow-angle image 220 shown in FIG. The display image 300 is generated by placing the superimposing narrow-angle image 220b on the wide-angle image 210 so that the wide-angle image 210 is the background. On the display image 300, a boundary frame 301 representing the outer shape of the superimposing narrow-angle image 220b is also shown.

More specifically, for example, as shown in FIG. 14, the entire image area of the wide-angle image or the display image is centered on the position (x _B , y _B ) and the superposed narrow-angle image (in this example, the image 220b). Are classified into a third image region having the same image size and a fourth image region which is the other image region. In FIG. 14, the area filled with diagonal lines corresponds to the third image area, and the area filled with dots corresponds to the fourth image area. Then, an image obtained by combining the image in the fourth image region of the wide-angle image 210 and the superimposing narrow-angle image 220 b is generated as the display image 300. In this case, the image in the third image area of the display image 300 is the superimposing narrow-angle image 220b itself, but the image in the third image area of the wide-angle image 210 and the superimposing narrow-angle image 220b are blended ( An image in the third image region of the display image 300 may be generated by performing alpha blending.

A method for setting the position (x _B , y _B ) will be described. Simply for example, of the feature evaluation value of the first to m, identifying the minimum feature evaluation value, the position of the center position of the evaluation image corresponding to the characteristic evaluation value of the minimum (x _B, y _B) to Can be set. That is, if the minimum feature evaluation value is the third feature evaluation value, the center position (x ₃ , y ₃ ) of the third evaluation image may be set to the position (x _B , y _B ). it can.

Based on the setting method of the position (x _B , y _B ), the result of the face detection process may be further considered. For example, let us consider a case where the smallest feature evaluation value among the first to mth feature evaluation values is the third feature evaluation value, and the second smallest feature evaluation value is the fourth feature evaluation value. Then, it is assumed from the result of face detection processing on the wide-angle image 210 that the third evaluation image includes a face area while the fourth evaluation image does not include a face area. In such a case, priority is given to including the face of the person in the display image, and the center position (x ₄ , y ₄ ) of the fourth evaluation image is set to the position (x _B , y _B ). Also good.

  According to the fourth embodiment, the same effect as that achieved in the first embodiment can be obtained.

<< 5th Example >>
A fifth embodiment will be described. In the fourth embodiment, the narrow-angle image is reduced and displayed in a portion having a small feature of the wide-angle image, but in the fifth embodiment, the reverse display process is performed.

  Assuming that the wide-angle image and the narrow-angle image are the wide-angle image 210 and the narrow-angle image 220 shown in FIGS. 8A and 8B and FIG. 15, during the special display period according to the fifth embodiment. A method for generating the display image will be described. In this case, immediately after the narrow-angle image 220 is obtained, a display image 310 as shown in FIG. 15 is generated and displayed based on the wide-angle image 210 and the narrow-angle image 220. The display image 310 is generated by disposing the superimposing wide-angle image 210 b obtained by reducing the image size of the wide-angle image 210 on the narrow-angle image 220.

  In the first embodiment, the method for determining the image size of the superimposing narrow-angle image 220b has been described. However, this method can be applied to the method for determining the image size of the superimposing wide-angle image 210b. When considering this application, the superimposing narrow-angle image 220b in the description of the method for determining the image size of the superimposing narrow-angle image 220b may be read as the superimposing wide-angle image 210b.

  The arrangement position of the superimposing wide-angle image 210b on the narrow-angle image 220 and the display image 310 can be determined by a method similar to the method described in the fourth embodiment. That is, an evaluation block having the same image size as that of the wide-angle image 210b for superimposition is set on the narrow-angle image 220, and the feature size of the evaluation image is based on the image in the evaluation block that is a partial image of the narrow-angle image 220. A feature evaluation value representing the height is derived. The evaluation image described in the fifth example indicates an image in the evaluation block set in the narrow-angle image 220. The method for deriving the feature evaluation value is the same as that described in the fourth embodiment.

The image processing unit 44 derives a feature evaluation value for each movement while moving the center position of the evaluation block one pixel at a time in the horizontal or vertical direction on the narrow-angle image 220, and the obtained first to m-th images are obtained. Among the feature evaluation values, the center position of the evaluation image corresponding to a relatively small feature evaluation value is set to a position (x _C , y _C ) (m is an integer of 2 or more). The display control unit 42 arranges the narrow-angle image 220 so that the point arranged at the center position of the superimposing wide-angle image 210b is arranged at the position (x _C , y _C ) on the narrow-angle image 220 and the display image 310. The display image 310 is generated by arranging the superimposing wide-angle image 210b on the top. The display image 310 is generated by disposing the superimposing wide-angle image 210b on the narrow-angle image 220 so that the narrow-angle image 220 becomes the background. On the display image 310, a boundary frame 311 representing the outer shape of the superimposing wide-angle image 210b is also shown.

More specifically, for example, as shown in FIG. 16, the entire image area of the narrow-angle image or the display image is centered on the position (x _C , y _C ) and the wide-angle image for superimposition (in this example, the image 210b). Are classified into a fifth image region having the same image size and a sixth image region which is the other image region. In FIG. 16, the area filled with diagonal lines corresponds to the fifth image area, and the area filled with dots corresponds to the sixth image area. Then, an image synthesized by combining the image in the sixth image area of the narrow-angle image 220 and the wide-angle image for superimposing 210 b is generated as the display image 310. In this case, the image in the fifth image area of the display image 310 is the superposition wide-angle image 210b itself, but the image in the fifth image area of the narrow-angle image 220 and the superposition wide-angle image 210b are blended (alpha An image in the fifth image area of the display image 310 may be generated by blending.

The method for setting the position (x _C , y _C ) is the same as the method for setting the position (x _B , y _B ) described in the fourth embodiment. That is, for example, among the first to m-th feature evaluation values, the minimum feature evaluation value is specified, and the center position of the evaluation image corresponding to the minimum feature evaluation value is determined as the position (x _C , y _C ) Can be set.

Based on this position (x _C , y _C ) setting method, the result of face detection processing may be taken into consideration. For example, let us consider a case where the smallest feature evaluation value among the first to mth feature evaluation values is the third feature evaluation value, and the second smallest feature evaluation value is the fourth feature evaluation value. Then, it is assumed from the result of the face detection process on the narrow-angle image 220 that it is found that the third evaluation image includes a face area, but the fourth evaluation image does not include a face area. In such a case, priority is given to including the face of the person in the display image, and the center position (x ₄ , y ₄ ) of the fourth evaluation image corresponding to the fourth feature evaluation value is set to the position (x _C , y _C ) may be set.

  According to the fifth embodiment, since a wide-angle image captured in the past and a narrow-angle image representing the current subject of interest can be simultaneously confirmed on the display screen 27a, composition setting by camera work is supported. The same effect as that obtained in the embodiment can be obtained.

<< Sixth Example >>
A sixth embodiment will be described. In the first embodiment, during the special display period, the display image is generated by arranging the superimposing narrow-angle image on the wide-angle image. However, in the sixth embodiment, the narrow-angle image is displayed on the wide-angle image. A display image is generated by superimposing an index representing the imaging range. This will be described more specifically.

  FIG. 17 is a flowchart of the operation of the imaging apparatus 1 according to the sixth embodiment, focusing particularly on the display operation during shooting of the target moving image. The flowchart in FIG. 17 is obtained by replacing step S16 in the flowchart in FIG. 7 with step S30. In other points, both flowcharts are the same.

  Accordingly, if there is a reduction zoom operation after the start of capturing the target moving image, the process proceeds to step S20 via steps S11 and S12 to execute the processes of steps S20 and S21, and then the process of step S18 is executed. On the other hand, if there is an enlargement zoom operation, the process proceeds to step S13 via steps S11 and S12, and the processes of steps S13, S14, S15, S30, and S17 are executed, and then the process of step S18 is performed. Executed. Since the contents of the processes in steps S11 to S15, S17, S18, S20, and S21 are the same as those described in the first embodiment, a duplicate description is omitted.

The position (x _A , y _A ) where the image data of the subject located at the center of the narrow-angle image 220 exists on the wide-angle image 210 is determined by the alignment in step S15. As described in the first embodiment, if the subject at the center position of the narrow-angle image 220 is called the center subject, the position of the center subject on the narrow-angle image 220 on the wide-angle image 210 is (x _A , y _A ).

In step S30, the display control unit 42, the position on the wide-angle image 210 (x _A, y _A) a rectangular frame around the same as the image size for positioning the narrow-angle image 220a (see FIG. 9) A rectangular frame having an image size is set on the wide-angle image 210. This rectangular frame represents the outer shape of the shooting range of the narrow-angle image 220, and the rectangular frame is referred to as a shooting range frame of the narrow-angle image. The shooting range frame of the narrow-angle image 220 is the same as the boundary frame 231 in FIG.

  FIG. 18 shows a display image 330 generated in step S30 based on the wide-angle image 210 and the narrow-angle image 220. The display image 330 is obtained by superimposing the shooting range frame 331 of the narrow-angle image 220 on the wide-angle image 210.

  The display image 330 generated in step S30 is displayed on the display screen 27a. A period during which the display image based on the wide-angle image and the narrow-angle image is displayed corresponds to the special display period. After the display image 330 is generated and displayed, the process proceeds to step S17, and if it is determined that the special display period should be terminated, the process proceeds to step S18. Otherwise, the process returns to step S15 and steps S15 and S30 are performed. This process is repeatedly executed. In the process of repeatedly executing the processes in steps S15 and S30, the displayed image is a still image, but the shooting range frame of the narrow-angle image can move on the display screen 27a.

  According to the sixth embodiment, it is possible to check the current shooting range while checking the state of the entire shooting scene, and it is possible to obtain the same effect as the effect brought about by the first embodiment. That is, the user can easily recognize which part of the entire shooting scene is currently being shot, supports composition setting by camera work, and selects a subject of interest during and after the zoom operation. It is supported to continue to capture it reliably.

<< Seventh Embodiment >>
A seventh embodiment will be described.

A first special display mode for generating and displaying a display image 230 (see FIG. 8C) from the wide-angle image 210 and the narrow-angle image 220 as described in the first or second embodiment;
A second special display mode for generating and displaying a display image 300 (see FIG. 13) from the wide-angle image 210 and the narrow-angle image 220 as described in the fourth embodiment;
A third special display mode for generating and displaying a display image 310 (see FIG. 15) from the wide-angle image 210 and the narrow-angle image 220 as described in the fifth embodiment;
A fourth special display mode for generating and displaying a display image 330 (see FIG. 18) from the wide-angle image 210 and the narrow-angle image 220 as described in the sixth embodiment;
Regardless of the presence or absence of the zoom operation and the content, the normal display mode for simply displaying the latest frame images obtained sequentially,
The imaging device 1 is formed so that the imaging device 1 can realize display in any two or more display modes among a plurality of display modes including the actual display in any display mode. It is better that the user can select these.

  Further, as described above, in the first or fourth special display mode, alignment is required for generating the display image. However, when the alignment reliability is low, the first or fourth special display mode is used. The display image may be different from the originally intended image. Considering this, when the imaging apparatus 1 is formed so that display in the first or fourth special display mode can be realized, when the reliability of alignment is lower than a predetermined reference reliability, the first Alternatively, the display control unit 42 automatically switches the display mode so that the display is performed in a display mode different from the fourth special display mode (that is, the normal display mode, or the second or third special display mode). May be performed.

  In addition, as described above, in the second special display mode, the narrow-angle image is reduced and displayed on the portion with less features on the wide-angle image, and in the third special display mode, the wide-angle image is reduced on the portion with less features on the narrow-angle image. If there is no portion with few features, the display in the second or third special display mode may not be useful. Considering this, when the imaging apparatus 1 is formed so that display in the first or fourth special display mode can be realized, the minimum value among the m feature evaluation values calculated is a predetermined reference. If the value is larger than the value, it is determined that there are no features having a small feature, and the display mode is different from the second or third special display mode (that is, the normal display mode or the first or fourth special display mode). The display control unit 42 may automatically switch the display mode so that the display is performed.

<< Eighth Example >>
An eighth embodiment will be described. The wide-angle image as a still image taken before the movement of the zoom lens 30 described in the first to sixth embodiments is referred to as a related still image in the eighth embodiment. In addition, the target moving image in the eighth embodiment refers to a moving image in which one or a plurality of related still images are captured during capturing of the moving image. However, in the eighth embodiment, such a target moving image may be simply referred to as a moving image.

  At the time of shooting the target moving image described in the first to sixth embodiments, the recording control unit 43 in FIG. 5 records the image data of the target moving image in the external memory 18, while the image data of the related still image. Can also be recorded in the external memory 18. The recording of the image data of the related still image can be executed, for example, at the stage of the process in step S13 in FIG. However, at this time, the image data of the related still image is recorded in the external memory 18 after being associated with the image data of the target moving image.

  Specifically, for example, the recording control unit 43 creates an image file including a main body area and a header area as shown in FIG. 19 in the external memory 18, and the image data of the target moving image is stored in the main body area of the image file. On the other hand, additional data including image data of a related still image is stored in the header area of the image file. However, the image data of the related still image stored in the additional data is the image data of the thumbnail of the related still image. The thumbnail of the related still image is an image obtained by reducing the image size of the related still image. Since the main body area and the header area of one image file are recording areas associated with each other, the target moving image and the related still image are associated with each other by this storage. Further, the additional data includes related time data indicating the shooting time of the related still image, and thumbnail image data of the target moving image.

  The display control unit 42 in FIG. 5 includes a reproduction control unit that reproduces and displays still images or moving images recorded in the external memory 18 on the display unit 27 (or functions as a reproduction control unit) in the reproduction mode. Now, assume that a plurality of moving images including the target moving image are recorded in the external memory 18.

  In the playback mode, the display control unit 42 performs thumbnail playback for simultaneously displaying a plurality of thumbnails for a plurality of still images or moving images on the display screen 27a, and displays one designated still image or moving image on the display screen 27a. It is possible to selectively execute normal playback and display using the whole. The user can perform thumbnail reproduction or normal reproduction by performing a predetermined operation on the operation unit 26.

  FIG. 20 shows a display image 400 displayed on the display screen 27a during thumbnail reproduction. The display image 400 is divided into four image areas, and one thumbnail is drawn in each image area obtained by this division. As shown in FIG. 20, it is assumed that the display image 400 includes thumbnails 401 to 404. A thumbnail for a still image and a thumbnail for a moving image may be mixed in one display image, or such a mixture may not occur. However, all thumbnails 401 to 404 are now moving images. Assume a thumbnail about the image. Furthermore, it is assumed that the moving image corresponding to the thumbnail 401 is a target moving image in which two related still images are associated.

  When the first designation operation for designating any thumbnail displayed on the display image 400 is performed by the user, the display control unit 42 displays the moving image corresponding to the thumbnail designated by the first designation operation as the moving image. Normal playback from the beginning of the video recording period. The first designation operation is performed on the operation unit 26, for example (the same applies to the second designation operation described later).

The user can also perform a second designation operation different from the first designation operation. For example, when the user designates the thumbnail 401 by the second designation operation while the display image 400 is displayed, the display control unit 42 displays the display image 410 as shown in FIG. 21 on the display screen 27a. indicate. Two thumbnails 411 and 412 are drawn on the display image 410. The thumbnails 411 and 412 are thumbnails of related still images associated with the moving image corresponding to the thumbnail 401, and they are extracted from the header area of the image file for the moving image corresponding to the thumbnail 401. Hereinafter, the moving image corresponding to the thumbnail 401 is referred to by reference numeral MV ₄₀₁ and the image file storing the moving image MV ₄₀₁ is referred to by reference numeral FL ₄₀₁ .

The display image displayed when the second designation operation is performed is divided into four image areas, and one thumbnail can be drawn in each image area obtained by this division. Since the number of related still images of MV ₄₀₁ is two, only thumbnails of the two related still images are drawn on the display image 410.

  In a state where the display image 410 is displayed, the user can perform a digest reproduction operation on the imaging apparatus 1. The digest reproduction operation is performed by designating the thumbnail 411 or 412 on the display image 410 using the operation unit 26 or the like.

When the thumbnail 411 is designated by the digest playback operation, the playback display unit 42 determines a first digest playback period that is a part of the entire shooting period of the moving image MV ₄₀₁ based on the related time data in the image file FL ₄₀₁ . The moving image MV ₄₀₁ is reproduced on the display screen 27a (for example, normal reproduction) only during the first digest reproduction period. That is, out of all the frame images forming the moving image MV ₄₀₁ , only the frame image belonging to the first digest reproduction period is extracted, and the moving image composed of the extracted frame images is reproduced on the display screen 27a. Similarly, when the thumbnail 412 is designated by the digest playback operation, the playback display unit 42 performs the second digest playback that is a part of the entire shooting period of the moving image MV ₄₀₁ based on the related time data in the image file FL ₄₀₁ . A period is determined, and the moving image MV ₄₀₁ is reproduced on the display screen 27a only during the second digest reproduction period (for example, normal reproduction).

First digest reproduction period is a period relative to photographing time t ₄₁₁ of the related still image corresponding to the thumbnail 411, eg, T from T _A seconds before time as the commencement and time t ₄₁₁ than the time t _{411 This} is the period that ends when _B seconds have passed. The second digest reproduction period is a period relative to photographing time t ₄₁₂ of the related still image corresponding to the thumbnail 412, eg, T from T _A seconds before time as the commencement and time t ₄₁₂ than the time t _{412 This} is the period that ends when _B seconds have passed. T _A and T _B can be a predetermined positive number (for example, about 5).

  In many cases, a large number of moving images are recorded in the external memory 18, and it may be difficult to find a desired moving image from among the large number of moving images. On the other hand, since a photographer usually performs a zoom operation while paying attention to a specific subject, characteristic scenes are often photographed before and after the time when the zoom operation is performed. Considering this, reproduction by the digest reproduction operation as described above is made possible. Since only a characteristic scene in the moving image is extracted and reproduced by the digest reproduction operation, the user finds a desired moving image in a short time from among a large number of moving images recorded in the external memory 18. It becomes possible.

When thumbnails for a plurality of moving images are displayed, the number of related still images associated with each moving image may be displayed on or near the corresponding thumbnail. For example, when the display image 400 of FIG. 20 is displayed, “2” that is the number of related still images associated with the moving image MV ₄₀₁ is displayed at the end of the thumbnail 401 or next to the thumbnail 401. Anyway. The same applies to the thumbnails 402 to 404.

  In addition, when thumbnails for a plurality of moving images are displayed, the color of the thumbnail frame of the moving image to which one or more related still images are associated, and the color of the thumbnail frame of the moving image that is not May be different. That is, when the display image 400 including the thumbnails 401 to 404 is displayed, a frame surrounding the outer periphery of each thumbnail is displayed. For example, a moving image corresponding to the thumbnails 401 and 403 is associated with a related still image. , While the related still images are not associated with the moving images corresponding to the thumbnails 402 and 404, the color of the frame surrounding the outer periphery of the thumbnails 401 and 403 in the display image 400 and the thumbnails 402 and 404 You may make it differ. Thereby, the user can confirm the presence or absence of a related still image easily. It is also possible to notify the user of the presence or absence of the related still image by a method other than changing the color of the frame.

  The display formats of the display images 400 and 410 in FIGS. 20 and 21 are similar. For this reason, when the display image 400 or 410 is displayed, the user can determine whether the display image is an image displaying thumbnails of a plurality of moving images, or can display thumbnails of related still images. It may be difficult to determine whether the image is displayed. Taking this into consideration, the display frames 400 and 410 have different thumbnail frame colors, or different display areas other than the thumbnail display portion so that the former and the latter can be easily distinguished. You may make it let it.

  In the above description, the thumbnail image data of the related still image is stored in the header area of the image file of the moving image. However, the image data of the thumbnail of the related still image (or the image data of the related still image itself) You may make it store in the image file for still images different from the image file of a moving image. The image file for the still image is created in the external memory 18 at the time of executing the process of step S13 in FIG. 7 or FIG. 17, for example, and the recording control unit 43 is associated with the main area of the image file for the still image. Image thumbnail image data (or related still image image data) is stored. On the other hand, link information indicating the still image file (for example, the file number of the still image file) is written in the header area of the image file of the target moving image to which the related still image should be associated. Thereby, desired association recording is realized.

  In the above description, the maximum number of thumbnails displayed simultaneously is four, but the maximum number may be other than four.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 and 2 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The imaging apparatus 1 in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

[Note 2]
For example, it can be considered as follows. The image processing unit 44 in FIG. 5 can function as a position detection unit, or can function as a feature amount detection unit. The position detection unit can detect the position of the subject on the narrow angle image on the wide angle image. The feature amount detection unit can set the evaluation block in the wide-angle image or the narrow-angle image, extract the evaluation image, and detect the feature size of the evaluation image.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 18 External memory 27 Display part 30 Zoom lens 31 Focus lens 33 Imaging element 41 Shooting control part 42 Display control part 43 Recording control part 44 Image processing part

Claims (6)

An imaging unit that captures still images or moving images;
A display unit;
A first image having a first angle of view obtained by photographing and a second image having a second angle of view obtained by photographing at a time different from the first image are formed including the second image. A display control unit for simultaneously displaying on the display unit during shooting of the target moving image
The image pickup apparatus, wherein the first and second angles of view are different from each other. The display control unit changes a display size of the first image and a display size of the second image on a display screen of the display unit, and displays a display image in which the second image is arranged on the first image. The image pickup apparatus according to claim 1, wherein the image pickup device is displayed on the display unit during shooting of the target moving image. A position detector for detecting a position of the subject on the second image on the first image;
The imaging apparatus according to claim 2, wherein the display control unit determines an arrangement position of the second image on the first image based on a detection result of the position detection unit. The first image is taken before the second image, and the first angle of view is wider than the second angle of view;
The imaging device captures the first image prior to the decrease in the shooting angle of view when receiving an operation for decreasing the shooting angle of view from the first angle of view toward the second angle of view, 4. The imaging apparatus according to claim 1, wherein the second image is actually captured by reducing the shooting angle of view from the first angle of view to the second angle of view. 5. An imaging unit that captures still images or moving images;
A display unit;
When receiving an operation to reduce the shooting angle of view from the first angle of view to the second angle of view narrower than the first angle of view during shooting of the target moving image, the first angle of view is set prior to the reduction of the shooting angle of view. An imaging control unit that causes the imaging unit to capture a first image having an angle of view and then actually reduces the imaging angle of view from the first angle of view to the second angle of view;
A position detection unit for detecting a position of the subject on the second image captured at the second angle of view on the first image;
A display control unit configured to superimpose and display an index representing a shooting range of the second image on the first image after the second image is shot during shooting of the target moving image. An imaging apparatus characterized by the above. A recording control unit that records the first image as the still image and the target moving image on a recording medium in association with each other;
The display control unit includes a reproduction control unit that causes the display unit to reproduce a moving image or a still image recorded on the recording medium,
When the reproduction control unit receives a predetermined operation for designating the first image, the reproduction control unit selects a moving image within a predetermined period with reference to the shooting time of the first image within the entire shooting period of the target moving image. The image pickup apparatus according to claim 1, wherein the image display apparatus reproduces the image on the display unit.

Images