JP5471425B2 - Imaging apparatus, imaging control program, and imaging method - Google Patents

Description

The present invention relates to an imaging apparatus , an imaging control program, and an imaging method .

  Conventionally, for example, in Patent Document 1, the similarity between a region of interest (template image) that appears in the image of the previous frame and each region in the tracking area in the image of the subsequent frame is calculated in frames that move forward and backward in time. In addition, a technique has been proposed for detecting a region (a region of interest in a captured image of a later frame) where the similarity is equal to or greater than a predetermined reference. In this way, the technique described in Patent Document 1 tracks the attention area. Note that the attention area is an area that is first detected as an area that satisfies a predetermined condition, and it is considered that a main subject (a subject that a user is interested in the image, such as a human face) appears in the image. It is an area that is For this reason, the image of the main subject (main subject image) is tracked on the image by tracking the attention area.

JP 2009-17271 A

  However, in the technique described in Patent Document 1, when the main subject image in the attention area is out of the imaging range (on the captured image), the main subject image image may not be tracked by tracking the attention area. It was.

The present invention has been made in view of the above-described problems, and provides an imaging apparatus , an imaging control program, and an imaging method that accurately track a region of interest and track a main subject image. Objective.

In order to solve the above problems, an imaging apparatus according to the first aspect of the present invention provides:
An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
Image data acquisition means for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking unit that detects a region of interest that satisfies a predetermined condition on the moving image captured by the image capturing unit based on the still image data sequentially acquired by the image data acquisition unit, and tracks the detected region of interest. When,
When the region of interest tracked by the tracking unit moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled, and the direction in which the imaging unit captures the image of the attention Imaging direction control means for changing according to the movement or position of the area;
Equipped with a,
The imaging direction control means is a range of a still image captured by the imaging unit in the first direction after the region of interest has changed from the first direction to the second direction after the imaging unit has changed the imaging direction from the first direction to the second direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
It shall be the features a.

In order to solve the above problem, an imaging control program according to the second aspect of the present invention provides:
An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
Control the computer,
Image data acquisition means for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking unit that detects a region of interest that satisfies a predetermined condition on the moving image captured by the image capturing unit based on the still image data sequentially acquired by the image data acquisition unit, and tracks the detected region of interest. When,
When the region of interest tracked by the tracking unit moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled, and the direction in which the imaging unit captures the image of the attention An imaging direction control means for changing according to the position of the area,
To function as,
The imaging direction control means is a range of a still image captured by the imaging unit in the first direction after the region of interest has changed from the first direction to the second direction after the imaging unit has changed the imaging direction from the first direction to the second direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
It shall be the features a.
In order to solve the above problem, an imaging method according to a third aspect of the present invention includes:
An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
An imaging method using an imaging device comprising:
An image data acquisition step for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking step of detecting a region of interest that satisfies a predetermined condition on the moving image captured by the imaging unit based on the still image data sequentially acquired by the image data acquisition step, and tracking the detected region of interest. When,
When the region of interest tracked by the tracking step moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled so that the direction in which the imaging unit captures the image of the attention An imaging direction control step that changes according to the position of the region;
Including
In the imaging direction control step, after changing the direction in which the imaging unit captures an image from the first direction to the second direction, the region of interest is a range of a still image captured by the imaging unit in the first direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
It is characterized by that.

According to the imaging apparatus , the imaging control program, and the imaging method according to the present invention, it is unlikely that the attention area can be tracked.

It is a figure for demonstrating the structure of the imaging device which concerns on one Embodiment of this invention. It is a figure for demonstrating the hardware constitutions of the imaging device which concerns on one Embodiment of this invention. It is a flowchart for demonstrating the imaging control process which the imaging device which concerns on one Embodiment of this invention performs. FIG. 6 is a diagram illustrating a display screen displayed on the display unit of the imaging apparatus according to the embodiment of the present invention and illustrating a relationship between a region of interest and an imaging direction. It is a figure for demonstrating the relationship between an attention area and a captured image. FIG. 6 is a diagram illustrating a display screen displayed on the display unit of the imaging apparatus according to the embodiment of the present invention and illustrating a relationship between a region of interest and an imaging direction. It is an explanatory view showing a display screen displayed on a display part of an imaging device concerning modification 1 of one embodiment of the present invention, and explaining relation between an attention field and an imaging direction. It is an explanatory view showing a display screen displayed on a display part of an imaging device concerning modification 2 of one embodiment of the present invention, and explaining relation between an attention field and an imaging direction. It is a figure for demonstrating the relationship between the attention area | region and captured image about the modification 3 of one Embodiment of this invention. It is an explanatory view showing a display screen displayed on a display part of an imaging device concerning modification 3 of one embodiment of the present invention, and explaining relation between a field of attention and an imaging direction. FIG. 10 is another explanatory diagram illustrating a display screen displayed on a display unit of an imaging apparatus according to Modification Example 3 of the embodiment of the present invention, and illustrating a relationship between a region of interest and an imaging direction.

  Embodiments according to the present invention will be described with reference to the drawings. In addition, this invention is not limited by the following embodiment and drawing. It goes without saying that changes (including deletion of components) can be added to the following embodiments and drawings. Further, in the following description, in order to facilitate understanding of the present invention, description of known unimportant technical matters is appropriately omitted.

  First, the configuration of the imaging apparatus 100 according to the present embodiment will be described with reference to FIGS. 1 and 2. The imaging device 100 according to the present embodiment is a digital camera.

  As illustrated in FIG. 1, the imaging apparatus 100 includes a control unit 110, an I / F unit 120, an imaging unit 130, an operation unit 140, a display unit 150, a signal processing unit 160, and a control mechanism 170. Prepare.

  Under the control of the control unit 110, the imaging unit 130 captures an image (captured image) such as a person and a background in the imaging direction, generates an imaging signal, and supplies the generated imaging signal to the signal processing unit 160. .

  The signal processing unit 160 generates image original data representing a captured image based on the imaging signal supplied from the imaging unit 130 under the control of the control unit 110, and supplies the generated image original data to the control unit 110. .

  The control mechanism 170 mechanically controls the imaging unit 130 under the control of the control unit 110 to change the direction in which the imaging unit 130 captures an image in another direction.

  The control unit 110 controls the entire imaging apparatus 100 (each component such as the imaging unit 130). The control unit 110 includes an image data acquisition unit 110a, a tracking unit 110b, and an imaging direction control unit 110c. The image data acquisition unit 110a, the tracking unit 110b, and the imaging direction control unit 110c operate as follows to perform imaging control processing.

  The image data acquisition unit 110 a sequentially generates still image data based on the image original data supplied from the signal processing unit 160. The image data acquisition unit 110a sequentially generates still image data at a predetermined frame rate by controlling the imaging unit 130 and the signal processing unit 160, for example. The frame rate is expressed by the number of still images acquired per second (the number of frames). The still image data is data representing a captured image captured by the imaging unit 130. In this embodiment, in this way, the image data acquisition unit 110a sequentially acquires still image data representing still images (captured images) captured by the imaging unit 130 (image data acquisition processing). A moving image is constituted by a plurality of continuous still images represented by a plurality of still image data.

  First, the tracking unit 110b searches for a region of interest in a still image represented by the still image data based on, for example, still image data acquired first after the image data income unit 110a starts the imaging control process (search). To do). The attention area is an area in which the main subject is shown, and is an area that satisfies a predetermined condition on the image. The attention area is, for example, an area where a face appears as a main subject (subject to be tracked). For example, the tracking unit 110b uses a template image including a predetermined face (main subject) image prepared in advance to search for an area similar to the template image by a template matching or the like. Such a region is a region of interest, and if there is this region, the tracking unit 110b detects this region as a region of interest. A known technique (so-called face detection technique or the like) can be used for such attention area search and detection. Note that the tracking unit 110b detects one region of interest. When there are a plurality of candidates for the attention area, for example, one of the candidates determined at random is set as the attention area. When the tracking area 110b cannot detect the attention area using one still image data, the tracking area 110b detects the attention area using the still image data acquired next by the image data acquisition section 110a.

  When the tracking unit 110b detects the attention area, the tracking area 110b uses the series of still image data continuously acquired by the image data acquisition unit 110a, and tracks the attention area on the moving image represented by the series of still image data. . For this tracking, a known technique (for example, the technique of Patent Document 1) for automatic tracking such as so-called autofocus can be used. The tracking unit 110b, for example, includes a region of interest (template image) in a still image of a previous frame and a still image of a subsequent frame of still image data (frames) that are temporally continuous acquired by the image data acquisition unit 110a. The degree of similarity with each area in the tracking area is calculated, and an area where the degree of similarity is equal to or greater than a predetermined reference and is maximized is detected. This area is an attention area in a still image of a later frame. By repeating this process, the region of interest is tracked, and the subject image 157 (main subject image) in the region of interest is tracked. The similarity is calculated by template matching using, for example, SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or the like. The tracking area is an area within a predetermined range (preset) from the position of the attention area in the still image of the previous frame.

  Here, as described above, the tracking unit 110b detects a region of interest that satisfies a predetermined condition on the moving image captured by the imaging unit 130 based on the still image data sequentially acquired by the still image acquisition unit 110a. Then, the detected attention area (the subject image 157 in the attention area) is tracked (tracking process).

  The imaging direction control unit 110c monitors the position, movement, and the like of the attention area (subject image 157 in the attention area) tracked by the tracking section 110b. The imaging direction control unit 110c determines whether the attention area tracked by the tracking unit 110b has moved to the outside on the image based on the detection result, and determines that the control mechanism 170 has been moved to the outside on the image. The direction in which the imaging unit 130 captures an image is controlled to be changed. Details of these processes (imaging direction control processes) will be described later.

  The I / F unit 120 is an interface (I / F) with the outside. The I / F unit 120 is used when the control unit 110 reads / writes data from / to the storage medium 50. For example, the I / F unit 120 is used when the control unit 110 records still image data (particularly, still image data generated when a shutter key or the like is pressed) or the like in the storage medium 50.

  The operation unit 140 receives an operation input from the user under the control of the control unit 110, and supplies an operation signal corresponding to the received operation input to the control unit 110. The control unit 110 performs processing according to the supplied operation signal.

  The display unit 150 displays a menu screen, a setting screen, a live view image, and the like necessary for operating the imaging apparatus 100 under the control of the control unit 110.

  The storage medium 50 is an external storage device of the imaging device 100 that is connected to the imaging device 100. The storage medium 50 is constituted by, for example, a memory card equipped with a flash memory. The control unit 110 records still image data and the like on the storage medium 50 via the I / F unit 120.

  Next, a hardware configuration of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG.

  The imaging apparatus 100 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 114, a TG (Timing Generator) 115, a driver 116, a media controller 121, and an imaging. An element 131, an imaging lens group 132, an imaging drive unit 133, an operation key group 141, a display 151, an AFE (Analog Front End) 161, a camera shake correction mechanism 171, and a bus 199 are provided. The CPU 111 and the like are appropriately connected to the bus 199.

  The imaging unit 130 includes an imaging element 131, an imaging lens group 132, and an imaging drive unit 133 (see FIG. 2).

  The imaging lens group 132 includes a lens group including a focus lens and a zoom lens.

  The plurality of lenses constituting the imaging lens group 132 are each driven by a lens driving unit included in the imaging driving unit 133. The lens driving unit includes a motor that moves the imaging lens group 132 in the optical axis direction and a motor driver that drives the motor. The lens driving unit drives the motor by a motor driver under the control of the CPU 111 to move the imaging lens group 132 in the optical axis direction.

  Further, the imaging drive unit 133 includes an aperture / shutter unit. The diaphragm / shutter unit includes a diaphragm / shutter and a drive circuit. The drive circuit operates the aperture / shutter under the control of the CPU 111. This aperture / shutter functions as an aperture and shutter. The diaphragm is a mechanism that controls the amount of light incident through the imaging lens group 132, and the shutter is a mechanism that controls the time during which light is applied to the imaging element 131, that is, the time during which the light receiving element receives light. I mean.

  The image sensor 131 is configured by, for example, various CCD (Charge Coupled Device) image sensors having a Bayer color filter. The image sensor 131 may be another image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  In accordance with the drive signal supplied from the driver 116, the image sensor 131 captures an image of light incident through the imaging lens group 132, performs photoelectric conversion for each pixel, and outputs a signal based on the charge for each pixel generated by the photoelectric conversion. It supplies to AFE161 as an imaging signal. In this way, the image sensor 131 is controlled by the CPU 111 via the driver 116 and supplies the image signal to the AFE 161.

  The signal processing unit 160 in FIG. 1 includes an AFE 161. The AFE 161 performs a predetermined process (hereinafter referred to as process A) on the imaging signal supplied from the imaging element 131, and generates and outputs digital data (original image data) representing the captured image captured by the imaging element 131. . The process A is a process including an OB (Optical Black) clamp process, a CDS (Correlated Double Sampling) process, an AGC (Automatic Gain Control) process, an A / D (Analog / digital) process, and the like. The AFE 161 records (supplies) the generated original image data in the RAM 114.

  Here, the original image data is image data for one frame expressed by a pixel value or the like in a still image. The original image data is, for example, Bayer data.

  The control mechanism 170 in FIG. 1 includes a camera shake correction mechanism 171. For example, the camera shake correction mechanism 171 mechanically moves at least the image sensor 131 under the control of the CPU 111. Accordingly, the direction in which the image sensor 131 captures an image changes to another direction, and camera shake when the user uses the image capturing apparatus 100 is corrected. The camera shake correction mechanism 171 is, for example, an image sensor shift system (a system in which camera shake is sensed by a vibration gyro sensor and the image sensor 131 is moved in accordance with camera shake) or an image unit swing system (a camera shake is detected by a vibration gyro sensor). This is a mechanism for moving an imaging unit including the element 131 and the imaging lens group 132 in accordance with camera shake. A known camera shake correction mechanism 171 can be used. Further, the camera shake correction mechanism 171 moves at least the image sensor 131 and changes the imaging direction of the image sensor 131 to another direction under the control of the CPU 111, particularly for the purpose other than the purpose of camera shake correction in the imaging control process. It is configured to be able to In the present embodiment, the camera shake correction mechanism 171 will be described as an image sensor shift type mechanism.

  The control unit 110 (image data acquisition unit 110a, tracking unit 110b, and imaging direction control unit 110c) in FIG. 1 includes a CPU 111, a ROM 112, a RAM 114, a TG 115, and a driver 116.

  The CPU 111 controls each component of the imaging device 100. In addition, the CPU 111 uses various data recorded in the ROM 112 in accordance with the imaging control program 113 recorded in the ROM 112, and also includes a driver 116, a TG 115, a camera shake correction mechanism 171, an imaging drive unit 133, and the like. By controlling this, processing performed by the control unit 110 (the image data acquisition unit 110a, the tracking unit 110b, and the imaging direction control unit 110c), such as imaging control processing, is executed. The imaging control program 113, data, and the like recorded in the ROM 112 may be read once into the RAM 114 and used.

  At least a part of the functions of the CPU 111 may be realized by various dedicated circuits. That is, at least a part of the control unit 110 may be configured by various dedicated circuits. Examples of various dedicated circuits include a DSP (Digital Signal Processor), an encoder, and a decoder.

  The ROM 112 is configured by an appropriate semiconductor storage device. The ROM 112 stores various programs such as the imaging control program 113, data used by the CPU 111, and the like.

  The RAM 114 is configured by an appropriate semiconductor memory device. The RAM 114 functions as a working memory of the CPU 111 and temporarily stores data used by the CPU 111 (template image data), data generated by the CPU 111 (still image data, etc.), data supplied to the CPU 111 (original image data), and the like. Remember.

  Note that the imaging control program 113 may cause the CPU 111 to perform imaging control processing in cooperation with an OS (Operation System). In this case, the OS is also recorded in the ROM 112. The imaging control program 113 is recorded on a portable storage medium (for example, a CD-ROM (Compact Disk Read Only Memory) or a DVD-ROM (Digital Versatile Disk Read Only Memory)), and is supplied to the imaging apparatus 100. Also good. Further, the imaging control program 113 may be supplied to the imaging device 100 via a network.

  The ROM 112, RAM 114, or portable storage medium in which the imaging control program 113 is recorded is a computer-readable program product (a computer-readable storage medium that stores the program). The imaging control program 113 may operate a computer in which at least a part of functions is realized by a dedicated circuit. In other words, the computer only needs to perform an imaging control process as a whole, and the imaging control program 113 only needs to operate such a computer.

  The CPU 111 performs various processes (hereinafter referred to as process B) on the original image data supplied from the AFE 161, and converts the original image data into still image data in the YUV (luminance / color difference) format. The process B includes a process for improving the image quality, such as contour enhancement, auto white balance, and auto iris. As described above, the CPU 111 acquires still image data by generating still image data based on the original image data supplied from the AFE 161.

  When the control unit 110 includes a DSP, the DSP receives original image data from the AFE 161, and converts the received original image data into still image data (YUV format) by the same process as the process B described above. Thereby, the control unit 110 acquires still image data.

  The CPU 111 appropriately converts the still image data generated above into display data and supplies it to the display 151. In addition, the CPU 111 appropriately generates display data representing a menu screen, a setting screen, and the like (the data of these template images is recorded in the ROM 112) and supplies the display data to the display 151. The display data is, for example, RGB (Red-Green-Blue) data.

  The CPU 111 appropriately compresses still image data recorded in the RAM 114, and generates compressed compressed image data (this is also an example of still image data). When the control unit 110 includes an encoder, the encoder may perform compression. The CPU 111 appropriately records the compressed image data on the storage medium 50 via the media controller 121. The compressed image data is, for example, still image data in JPEG (Joint Photographic Experts Group) format.

  The TG 115 is controlled by the CPU 111, generates a pulse signal for operating the driver 116, and supplies the pulse signal to the driver 116. That is, the interval (shutter speed) at which the image pickup signal for one frame (that is, one image) is supplied from the image pickup device 131 depends on the pulse signal supplied from the TG 115 to the driver 116 or the like. The TG 115 also supplies a similar pulse signal to the AFE 161. The TG 115 synchronizes operation timings of the AFE 161 and the driver 116 by supplying a pulse signal. In this way, the AFE 161 and the driver 116 are controlled by the CPU 111 via the TG 115.

  The driver 116 is a drive circuit that supplies a drive signal to the image sensor 131 in accordance with a pulse signal from the TG 115. The image sensor 131 is controlled by the CPU 111 via the TG 115 and the driver 116.

  The CPU 111 sequentially acquires still image data at a predetermined frame rate (image data acquisition processing) by controlling the driver 116, the TG 115, and the like, and uses the still image data acquired sequentially to control the camera shake correction mechanism 171 and the like. To perform imaging control processing (tracking processing, imaging control processing). Note that the data format of the still image data used at this time may be appropriately changed by the CPU 111. That is, the still image data only needs to represent a predetermined still image. When using still image data in the imaging control process, the control unit 110 can convert the still image data into an appropriate data format (for example, RGB format) and use it.

  The I / F unit 120 in FIG. 1 includes a media controller 121. Under the control of the CPU 111, the media controller 121 receives data (still image data, difference data, etc.) from the CPU 111, and stores (records) the received data in the storage medium 50. To do. Thus, still image data is recorded in the storage unit.

  The operation unit 140 in FIG. 1 includes an operation key group 141. The operation key group 141 includes a plurality of operation keys such as a shutter key, a mode switching key, a cross key, a zoom key, a menu key, a set key, and a power key that can be operated in two steps by half-pressing and full-pressing. The operation key group 141 further includes a signal output unit (configured by a substrate, a contact, etc.) that outputs an operation signal corresponding to the operation of each key. The operation key group 141 is controlled by the CPU 111 and supplies an operation signal corresponding to the operation of each key to the CPU 111.

  The display unit 150 in FIG. The display 151 includes a display panel and a display controller. Display data is supplied to the display controller under the control of the CPU 111. The display controller converts the supplied display data into an analog signal under the control of the CPU 111, and displays an image on the display panel based on the converted analog signal. That is, the display 151 displays an image (a menu screen, a setting screen, a live view image, or the like) represented by the display data based on the supplied display data under the control of the CPU 111.

  Next, a detailed example of the imaging control process performed by the imaging apparatus 100 will be described with reference to FIGS. This process is started when the operation unit 140 of the imaging apparatus 100 is operated and the imaging mode is selected. Further, this processing is appropriately terminated when the imaging mode is terminated (for example, instructed by an operation of the operation unit 140).

  The image data acquisition unit 110a starts displaying a live view image (step S101). The image data acquisition unit 110a sequentially acquires still image data by the image data acquisition process described above, and generates display data based on the acquired still image data each time still image data is acquired. The image is supplied to the display unit 150 and the still image represented by the still image data is displayed on the display unit 150. Thereby, the display of the live view image (moving image) is started. Thereafter, the image data acquisition unit 110a sequentially acquires still image data at a predetermined frame rate, and sequentially displays the still images on the display unit 150, whereby the live view image is displayed on the display unit 150.

  When the image data acquisition unit 110a starts displaying the live view image, the tracking unit 110b searches the attention area based on the still image data first acquired by the image data acquisition unit 110a (step S102). The tracking unit 110b determines whether the attention area has been detected (step S103). If the attention area cannot be detected (step S103; NO), the processing of step S102 is performed again. In this way, the tracking unit 110b searches the attention area for each frame of the still image after the start of the display of the live view image. When the tracking unit 110b detects the attention area (step S103; YES), the tracking section 110b starts tracking the attention area (step S104). The processing in steps S102 to S104 is the tracking processing described above. The tracking unit 110b performs the process of step S104 for each frame of the still image.

  When the tracking unit 110b detects the attention area, the image data acquisition unit 110a is an image in which a frame (attention frame) indicating the attention area detected by the tracking section 110b is superimposed on a still image (hereinafter referred to as a tracking image). Is generated using still image data and attention frame data (assumed to be recorded in the ROM 112), and this display data is supplied to the display unit 150, whereby a tracking image is generated. Displayed on the display unit 150. The image data acquisition unit 110a performs this process (superimposition process) for each frame of the still image. Through the superimposition process, a moving image (superimposed moving image) in which the frame of interest is superimposed on the live view image is displayed on the display unit 150. Note that this superimposed moving image reflects the superimposing process, and when the superimposed moving image is displayed on the display unit 150, the tracking process of the attention image performed by the tracking unit 110b is expressed on the display unit 150. become. Further, the still image captured by the imaging unit 130 varies depending on the imaging direction of the imaging unit 130. For this reason, the live view image (superimposed moving image) displayed on the display unit 150 also changes as the imaging direction changes. That is, the imaging direction control process performed by the imaging direction control unit 110 c is also expressed on the display unit 150. Hereinafter, the tracking process and the imaging direction control process will be described with reference to the superimposed image displayed on the display unit 150 (see FIG. 4).

  For example, as illustrated in FIG. 4A, the display unit 150 displays a superimposed moving image (a tracking image when viewed instantaneously) in which the attention frame 156 surrounding the attention region 155 is displayed on the live view image. The subject image 157 (main subject image) in the attention area 155 here is an image of a human face. When the subject image 157 moves, the attention area 155 moves along with this movement (see the dotted arrow), and the attention frame 156 also moves (see FIG. 4B). Note that the attention frame 156 indicated by a two-dot chain line in FIG. 4B is the attention frame 156 at a time point several frames before (a time point in FIG. 4A). In this way, the tracking unit 110b tracks the attention area 155.

  When the tracking unit 110b starts tracking the attention area 155 (step S104), the imaging direction control section 110c determines whether the attention area 155 has moved outside the still image (step S105). Since the tracking unit 110b detects and tracks the attention area 155, the position of the attention area 155 is grasped. Accordingly, the imaging direction control section 110c can grasp the position of the attention area 155 (the grasped position is predetermined). And the movement of the attention area 155 can be grasped. Note that the imaging direction control unit 110c can also specify movement information such as a motion vector for the attention area 155 based on a plurality of still image data acquired by the image data acquisition unit 110a. Further, the imaging direction control unit 110c performs the process of step S105 that is repeatedly performed for each frame from the still image in which the tracking unit 110b detects the attention area 155.

  An example of the concept of the discrimination process in step S105 is shown below. For example, the imaging direction control unit 110c performs a determination process based on the following concept. For example, as shown in FIG. 5, on the image P, xy coordinates (x and y define the position for each pixel) with the lower left as the origin are defined. The upper right coordinate is determined by the number of pixels. Here, the upper right coordinate is (x1, y1). The position of the attention area 155 can also be defined by the coordinates of the xy coordinates. Here, the coordinates of the vertices of the attention area 155 are (xa, ya), (xb, yb), (xc, yc), and (xd, yd), respectively. In FIG. 5, two of the four vertices of the attention area 155 are located on the right side of the image. That is, xb = x1 and xd = x1.

  The imaging direction control unit 110c determines whether any of the four vertices of the attention area 155 is an edge on the image (upper side, lower side, right side, left side on the still image) of the still image immediately before the still image to be determined this time. ), The motion vector of the attention area 155 is specified using the current still image data and the still image data of the previous frame. At this time, when the direction indicated by the motion vector is toward the outside of the edge where the vertex of the attention area 155 overlaps, the imaging direction control unit 110c determines that the attention area 155 has moved to the outside on the still image ( Step S105; YES), Step S106 is performed. For example, when the attention area 155 is located at a position as shown in FIG. 5 with respect to the previous frame, when the motion vector indicates the right side toward the image (see the dotted arrow in FIG. 5), the attention area 155 is stationary. It is determined that the image has moved to the outside on the image.

  When the attention area 155 is not in the above state, the imaging direction control unit 110c determines that the attention area 155 has not moved outward on the still image (step S105; NO), and the process of step S107 I do.

  In the process of step S106, the imaging direction control unit 110c controls the control mechanism 170 in accordance with the direction toward the attention area 155, and changes (moves) the imaging direction of the imaging unit 130 by moving the imaging element 131. Here, it is assumed that the moving amount of the image sensor 131 is set in advance. That is, the amount of change in the imaging direction that is changed at a time is set in advance.

  The imaging direction control unit 110c moves the imaging element 131 to change the imaging direction so that the attention area 155 moves in a direction opposite to the direction (direction indicated by the motion vector) toward the attention area 155 on the image. . For example, as described above, when the direction of the attention area 155 is the right side in the state illustrated in FIG. 5, the imaging direction control unit 110 c causes the imaging element 131 to be a subject (a subject in the real space of the subject image 157. Hereinafter, this subject is appropriately moved so as to face the right side toward the subject 157), and the imaging direction is changed so that the imaging direction faces the right side toward the subject 157. As a result, the attention area 155 moves to the left on the image P.

  Note that since the camera shake correction mechanism 171 mechanically moves the image sensor 131, the movable range of the image sensor 131 is limited. In step S106, when the imaging direction control unit 110c cannot move the imaging element 131 (imaging direction) any more in the direction in which the imaging element 131 (imaging direction) is desired to be moved, the process of step S107 is skipped.

  As shown in FIG. 4C, when the attention area 155 (motion vector) is directed to the outer right side (see the dotted arrow), the imaging direction control unit 110c determines that the attention area 155 is on the left side (the direction in which the attention area is located). The imaging direction is changed so as to move in the opposite direction. As a result, as shown in FIG. 4D, the imaging direction is changed so that the subject image 157 moves to the left side on the image (see the dotted arrow), and the still image captured correspondingly is also displayed. As it changes. In FIG. 4D, the boundary L indicates the right side of the first still image. Further, the imaging direction control unit 110c always recognizes where the first still image area (initial image area) before changing the imaging direction was, even after the imaging direction was changed. The imaging direction (or imaging range) has moved to the right as a whole toward the subject 157. By such processing, the subject image 157 in the attention area 155 is not deviated from the imaging range, and the attention area 155 is positioned in the image, and the tracking unit 110b can continue to track the attention area 155 ( (Refer FIG.4 (d)). In this embodiment, this makes it possible to track the subject image 157 by tracking the attention area 155 with high accuracy.

  As shown in FIG. 4E, the tracking unit 110b continues to track the subject image 157 in the attention area 155. For example, the face that is the subject image 157 in the attention area 155 changes from the side face to the front face. Even so, the tracking unit 110b can continue to track the subject image 157 of the attention area 155. Further, as shown in FIG. 4F, the tracking unit 110b continues to track the attention area 155. For example, even if the subject image 157 in the attention area 155 is inverted and changes from a front face to a side face. The tracking unit 110b can continue to track the subject image 157 of the attention area 155. Since the attention area 155 is used as a template image when searching for the attention area 155 in the still image of the next frame, the subject image 157 in the attention area 155 is gradually changed when the image in the attention area 155 changes gradually. Can be tracked sequentially.

  Conventionally, when the subject image 157 in the attention area 155 deviates from the image, there is a case where the attention area 155 with the subject image 157 cannot be detected because the imaging direction is not moved. Conventionally, in this case, the detection area (tracking area) of the attention area 155 may be set near the attention area 155 immediately before the subject image 157 deviates outside the image so that the attention area 155 can be easily detected again. there were. However, when the subject image 157 deviates from the imaging range and enters the imaging range again (the state from FIG. 4C to FIG. 4F), the orientation of the subject image 157 is usually reversed. For example, the profile of the face becomes the profile of the opposite side. In such a case, the subject image 157 deviating from the image and the subject image 157 entering the image are no longer similar to each other. Therefore, it is difficult to re-detect the attention area 155 where the subject image 157 exists, and the subject image 157 is continuously tracked. It was difficult. In the present embodiment, since the imaging range is changed, the subject image 157 in the attention image 155 is gradually changed, so that the attention image 155 can be continuously tracked and the subject image 157 can be continuously tracked. it can.

  The imaging direction control unit 110c determines whether the attention area 155 has moved (positioned) in the initial image area in the process of step S107. Since the imaging direction control unit 110c always grasps the initial image area as described above, such a determination is possible. The imaging direction control unit 110c performs this repeated processing for each frame of a still image.

  When it is determined that the attention area 155 is not located in the initial image area (step S107; NO), the imaging direction control unit 110c performs the process of step S105 again. The imaging direction control unit 110c determines NO in this determination process when the imaging direction is the initial imaging direction. When it is determined that the attention area 155 is located in the initial image area (step S107; YES), the imaging direction control unit 110c moves the imaging element 131 so that the current imaging direction becomes the initial imaging direction, and the imaging direction Is changed (step S108), and then the process of step S105 is performed again.

  As shown in FIGS. 4F to 4G, when the attention area 155 moves into the initial image area, the imaging element 131 is moved, the imaging direction is changed (step S108), and the imaging direction is restored (FIG. 4). (Return to the imaging direction in (a) etc.).

  The imaging direction control unit 110c performs the imaging direction control process by performing the processes of steps S105 to S108, for example.

  Here, another specific example of the imaging direction control process performed by the imaging direction control unit 110c will be described with reference to FIG. For example, when the attention area 155 moves to the right side toward the image (FIG. 6A) and further moves to the right side, the imaging direction control unit 110c performs imaging so as to shift the imaging range toward the subject 157 to the right side. The direction is changed (FIG. 6B). Thereafter, when the attention area 155 moves downward toward the image (FIG. 6C) and further moves downward, the imaging direction control unit 110c shifts the imaging range downward toward the subject 157. In this manner, the imaging direction is changed (FIG. 6D). The boundary H indicates the lower side of the initial image area. A boundary L indicates the right side of the initial image area.

  As described above, the imaging direction control unit 110c controls the control mechanism 170 to capture an image when the attention area 155 tracked by the tracking unit 110b moves on the moving image to the outside of the image. The direction is changed according to the movement of the attention area.

  Thereby, in this embodiment, since the imaging range is changed, the attention area is tracked with high accuracy to track the main subject image.

  As described above, the tracking unit 110b detects a region similar to the region of interest in the older still image of two still images adjacent in time series from the newer still image. The attention area is tracked by repeating the detected area as a new attention area. Thereby, in this embodiment, it is possible to track even if the subject image in the attention area gradually changes.

  In addition, since the camera shake correction mechanism 171 that mechanically controls the imaging unit 130 to correct camera shake is employed, in the present embodiment, the conventional camera shake correction mechanism 171 is used to perform the imaging control process. I can do it.

  As described above, the imaging direction control unit 110c changes the direction in which the imaging unit 130 captures an image from the first direction to the second direction, and then the region of interest is displayed in the first direction. When moving within the range of the still image imaged in (initial image area), the control mechanism 170 is controlled to return the imaging direction of the imaging unit 130 to the first direction. Thereby, in this embodiment, the imaging range can be quickly returned to the range that the user originally wants to capture.

(Modification 1)
Note that the above-described process of returning the direction in which the imaging unit 130 captures an image to the first direction may be performed by another method. As shown in FIG. 7, when the region of interest 155 moves from the upper right to the left on the image after the imaging direction is changed so that the imaging range becomes the upper right toward the subject (FIG. 7A and the dotted arrow) First, the imaging direction control unit 110c returns the imaging range to the left side (FIG. 7B). When the attention area 155 subsequently moves from the top to the bottom on the image (see FIG. 7C and the dotted arrow), the imaging direction control unit 110c returns the imaging range to the original imaging direction ( FIG. 7 (d)).

  When moving the imaging direction, the imaging direction control unit 110c stores and holds the moving direction (for example, the right side) of the imaging range on the image, and the attention area 155 moves to the opposite side (the left side). ), The imaging direction control unit 110c moves the imaging element 131 so as to cancel the moving direction (right side) of the imaging direction, and changes the imaging direction to this direction (horizontal). Undo with respect to (direction). In this way, the imaging direction control unit 110c changes the imaging direction of the imaging unit 130 from the first direction to the second direction, and then captures the region of interest in the first direction. When moving within the range of the still image (within the initial image area), the control mechanism 170 may be controlled to return the direction in which the imaging unit 130 captures an image to the first direction. Thus, the imaging range can be changed finely according to the position of the attention area 155, and the imaging range can be returned to the range that the user originally wants to capture (the first imaging range).

(Modification 2)
In step S106, the amount of movement (the degree of change in the imaging direction (change angle)) by which the imaging direction control unit 110c moves the imaging element 131 is fixed, but the imaging is performed with the degree of change according to the moving speed of the region of interest. The direction in which the unit 130 captures an image may be changed. In this case, for example, the imaging direction control unit 110c uses the magnitude of the motion vector (particularly the magnitude of the motion vector in the direction toward the outside of the image) as a speed, and changes the imaging direction with a degree of change corresponding to the speed. Let Further, the degree of change may be determined by a speed that can be expressed by a difference in the position of the attention area of a still image between predetermined frames.

  The degree of change may be calculated based on the speed using a predetermined relational expression, or by referring to a table that is recorded in advance in a storage unit such as the ROM 112 and shows the relationship between the speed and the moving amount. It may be derived. Further, for example, as illustrated in FIG. 8A, the imaging direction control unit 110c causes the movement amount on the image to be W1 when the speed of the attention area 155 is slow and the speed is less than the threshold. The imaging direction may be changed depending on the degree of change (see FIG. 8B). In addition, for example, as illustrated in FIG. 8C, the imaging direction control unit 110c causes the movement amount on the image to be W1 when the speed of the attention area 155 is fast and the speed is less than the threshold. The imaging direction is changed according to the degree of change. In FIG. 8, what is drawn with a dotted line is a plurality of pre-frame attention area 155, subject image 157, and the like. FIG. 8 is drawn on the assumption that the attention area 155 and the like move at a constant speed.

  According to this modification, the degree of change is changed according to the speed of the region of interest, so that it is possible to prevent the imaging direction from being changed too much, and to prevent the user from changing the imaging range that the user originally wants to capture, It is less likely to miss the image.

(Modification 3)
The imaging direction control unit 110c controls the control mechanism 170 when the attention area is located at the end of the moving image on the moving image, and changes the direction in which the imaging section 130 captures the image according to the position of the attention area. You may let them.

  In this case, the imaging direction control unit 110c detects whether or not the attention area is at the end on the image in step S105. Here, the end is an area on the image that is within a predetermined range from the edge of the image (four sides of the image) to the inside of the image. Further, the attention area being at the end means that at least a part of the attention area is located at the end.

  For example, as shown in FIG. 9, as in FIG. 5, xy coordinates (x and y define the position of each pixel) with the lower left as the origin are defined on the image A. The coordinates of the four corners of the attention area 155 are (xa, ya), (xb, yb), (xc, yc), and (xd, yd), respectively. The end M is an area outside the central area 158 having (xe, ye), (xf, yf), (xg, yg), and (xh, yh) as vertices.

  For example, in step S105 of the flowchart illustrated in FIG. 3, the imaging direction control unit 110c determines that the attention area 155 is located at the end M when any of the four vertices of the attention area 155 is located within the end M ( Step S105; YES), Step S106 is performed. When none of the four vertices of the attention area 155 is located within the end M, the imaging direction control unit 110c determines that the attention area 155 is not at the end on the still image (step S105; NO).

  In the process of step S106, the imaging direction control unit 110c controls the control mechanism 170 according to the position of the attention area 155, and changes (moves) the imaging direction of the imaging unit 130 by moving the imaging element 131.

  The end portion M is divided into a first region M1 to an eighth region M8 that are located in the upper, lower, left, and right directions and in an oblique direction therebetween with the central region 158 as the center. The imaging direction control unit 110c moves the imaging element 131 depending on which of the first area M1 to the eighth area M8 the vertex of the attention area 155 located in the end M is located ( The direction in which the imaging direction is changed is specified, and the image sensor 131 is moved in the specified direction.

  Normally, at least two of the vertices of the attention area 155 are located in the end M, but one of the first area M1, the third area M3, and the sixth area is located in an oblique direction. When located in M6 and the eighth area M8, the vertex of the attention area 155 is assumed to be located in these areas. Accordingly, priority is given to detection in an oblique direction, and the imaging direction can be appropriately changed.

  The imaging direction control unit 110c has a direction (direction on the image centered on the center of the image) of any one of the first area M1 to the eighth area M8 where the vertex of the attention area 155 is located, and on the image. The imaging element 131 is moved to change the imaging direction so that the region of interest 155 moves in the opposite direction. That is, the imaging range moves in the same direction as the direction of the attention area 155 on the image and the direction toward the subject. For example, when the vertex of the attention area 155 is located in the fifth area M5, the imaging direction control unit 110c moves the imaging element 131 so as to face the right side toward the subject 157, and the imaging direction is the subject 157. The imaging direction is changed so that it faces toward the right side toward. For example, when the vertex of the attention area 155 is located in the eighth area M8, the imaging direction control unit 110c moves the imaging element 131 to face the upper right side, and the imaging direction faces the upper right side. Thus, the imaging direction is changed. The concept of the relationship between the imaging direction to be changed and the position of the attention area is basically the same as described above.

  As shown in FIG. 10A, when the attention area 155 is located at the end portion (here, the fifth area M5 on the right side on the image) M, the imaging direction control unit 110c indicates that the attention area 155 is on the left side ( The imaging direction is changed so as to move in the direction opposite to the direction in which the attention area 155 is located. As a result, as shown in FIG. 10B, the imaging direction changes so that the subject image 157 moves to the left side, and the imaging range that is imaged corresponding to this changes as a whole.

  By such control, the same effect as described above can be obtained in this modification. Further, since the imaging direction is changed before the attention area 155 goes outside the image, the subject image 157 can be accurately tracked even if the subject image 157 within the attention area 155 moves suddenly. .

  In the above description, when the attention area 155 is located at the end M, the imaging direction control unit 110c changes the imaging direction. However, the imaging direction may be changed depending on the movement direction of the attention area 155. That is, the imaging direction control unit 110c is a direction in which the attention area 155 is located at the end M on the moving image, and the movement direction of the attention area 155 is directed to the side on the image closest to the position of the attention area 155. In some cases, the control mechanism 170 may be controlled to change the direction in which the imaging unit 130 captures an image according to the moving direction of the attention area 155.

  For example, in step S106, the imaging direction control unit 110c determines that the attention area 155 is positioned at the end M based on the movement direction of the attention area 155 based on the motion vector or the movement direction of the attention area 155 for several frames. The movement direction of the attention area 155 before the detection is detected, and when the movement direction satisfies the above condition, the imaging direction is changed (refer to the above for how to change the imaging direction). In step S106, the imaging direction control unit 110c does not change the imaging direction when the moving direction does not satisfy the above condition.

  As shown in FIG. 11A, when the attention area 155 is located at an end (here, the fifth area M5 on the right side on the image) M, and the attention area 155 is in the right direction (the attention area 155). When moving to the closest side (the direction toward the right side), the imaging direction control unit 110c performs imaging so that the attention area 155 moves to the left (the direction opposite to the direction in which the attention area 155 is located). Change direction. As a result, as shown in FIG. 10B, the imaging direction changes so that the subject image 157 moves to the left side, and the imaging range that is imaged corresponding to this changes as a whole. As shown in FIG. 11C, even if the attention area 155 is located at the end M, the attention area 155 is in the lower direction (the direction toward the side that is not the closest side (right side) at the position of the attention area 155). When moving to, the imaging direction control unit 110c does not change the imaging direction.

  According to such processing, the imaging direction may not be changed depending on the moving direction of the attention area 155, and it is possible to suppress the user from changing the imaging range that the user originally wants to capture.

DESCRIPTION OF SYMBOLS 50 ... Storage medium, 100 ... Imaging device, 110 ... Control part, 110a ... Image data acquisition part, 110b ... Tracking part, 110c ... Imaging direction control part, 111 ... CPU, 112 ... ROM, 113 ... imaging control program, 114 ... RAM, 115 ... TG, 116 ... driver, 120 ... I / F unit, 121 ... media controller, DESCRIPTION OF SYMBOLS 130 ... Imaging part, 131 ... Imaging element, 132 ... Imaging lens group, 133 ... Imaging drive part, 140 ... Operation part, 141 ... Operation key group, 150 ... Display 151, display, 155 ... attention area, 156 ... attention frame, 157 ... subject image, 160 ... signal processing section, 161 ... AFE, 170 ... control mechanism, 171 ... Camera shake compensation Mechanism, A · · · image, H · · · boundary, L · · · boundary, M · · · end

Claims (6)

An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
Image data acquisition means for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking unit that detects a region of interest that satisfies a predetermined condition on the moving image captured by the image capturing unit based on the still image data sequentially acquired by the image data acquisition unit, and tracks the detected region of interest. When,
When the region of interest tracked by the tracking unit moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled, and the direction in which the imaging unit captures the image of the attention Imaging direction control means for changing according to the movement or position of the area;
Equipped with a,
The imaging direction control means is a range of a still image captured by the imaging unit in the first direction after the region of interest has changed from the first direction to the second direction after the imaging unit has changed the imaging direction from the first direction to the second direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
An imaging apparatus characterized by that. The control mechanism is a camera shake correction mechanism that corrects camera shake by moving only the imaging unit.
The imaging apparatus according to claim 1. The imaging direction control means changes the direction in which the imaging unit captures images with a degree of change corresponding to the moving speed of the region of interest.
The imaging apparatus according to claim 1 or 2 , wherein The imaging direction control means is a direction in which the region of interest is located at the end on the moving image, and the moving direction of the region of interest is closest to the side on the image at the position of the region of interest. In some cases, the control mechanism is controlled, and the imaging direction of the imaging unit is changed according to the moving direction of the region of interest.
The imaging apparatus according to any one of claims 1 to 3, characterized in that. An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
Control the computer,
Image data acquisition means for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking unit that detects a region of interest that satisfies a predetermined condition on the moving image captured by the image capturing unit based on the still image data sequentially acquired by the image data acquisition unit, and tracks the detected region of interest. When,
When the region of interest tracked by the tracking unit moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled, and the direction in which the imaging unit captures the image of the attention An imaging direction control means for changing according to the position of the area,
To function as,
The imaging direction control means is a range of a still image captured by the imaging unit in the first direction after the region of interest has changed from the first direction to the second direction after the imaging unit has changed the imaging direction from the first direction to the second direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
An imaging control program characterized by the above. An imaging unit that captures a moving image by continuously capturing a plurality of still images;
A control mechanism for controlling a direction in which the imaging unit captures an image;
An imaging method using an imaging device comprising:
An image data acquisition step for sequentially acquiring still image data representing the still image captured by the imaging unit;
A tracking step of detecting a region of interest that satisfies a predetermined condition on the moving image captured by the imaging unit based on the still image data sequentially acquired by the image data acquisition step, and tracking the detected region of interest. When,
When the region of interest tracked by the tracking step moves on the moving image to the outside of the image or is positioned at the end of the image, the control mechanism is controlled so that the direction in which the imaging unit captures the image of the attention An imaging direction control step that changes according to the position of the region;
Including
In the imaging direction control step, after changing the direction in which the imaging unit captures an image from the first direction to the second direction, the region of interest is a range of a still image captured by the imaging unit in the first direction. When moving in, the control mechanism is controlled, and the imaging direction of the imaging unit is returned to the first direction.
An imaging method characterized by the above.

Images