JP6443486B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques relating to subject extraction have been considered. For example, in the invention of Patent Document 1, an imaging device that automatically adjusts an automatic focus on a subject even when the subject moves in the camera direction by deforming an AF distance measurement frame according to the amount of change in the subject shape. It is disclosed.

JP 2009-069748 A

  Since the imaging device disclosed in Patent Document 1 deforms the AF distance measurement frame in accordance with the amount of change in the subject shape, a favorable effect can be obtained when the subject moves in the camera direction. However, an actual subject may move in various ways, and the purpose of a user who performs shooting also varies, and focus adjustment may not always be performed on a subject desired for the user.

  The present invention has been made in view of the above points, and an object of the present invention is to continuously extract a main subject region easily and accurately.

An image processing apparatus according to the present invention includes: an acquisition unit that sequentially acquires a plurality of pieces of image data generated continuously in time; a feature amount of an image indicated by the image data; and a subject based on the feature amount An area detection unit that detects an area, and the area detection unit by comparing the subject areas detected in at least two images among images indicated by the plurality of image data acquired by the acquisition unit A determination unit that determines whether or not the region detected by the step is stable; and when the determination unit determines that the region is stable, the subject region that is the target of the determination is set as a main subject region comprising a setting unit that, the, the determination unit includes a detection result of the recording unit the area detecting unit, the determination unit determination result recorded in the recording unit, the less the On the other hand, on the basis, to determine whether any of the subject region is determined in the past, based on the determination result, changing the criteria for determining that the stable.

  In addition, a display unit that displays the main subject region set by the setting unit so as to be visible may be further provided so that the positional relationship with the image can be understood.

  Further, a recording unit that records information on the main subject region set by the setting unit may be further provided.

  The determination unit may determine that the region is stable when the same region of the subject is detected continuously or intermittently in a predetermined number of images by the region detection unit.

  The region detection unit detects the subject region by performing mask extraction based on the feature amount, and the determination unit detects the extraction condition of the mask extraction and the position of the mask extracted by the mask extraction. The subject areas may be compared based on at least one of a change and a change in the size of the mask.

  In addition, a history recording unit that records a detection result by the region detection unit and a determination result by the determination unit is further provided, and the determination unit performs comparison of the subject region based on at least a change in the position of the mask. In addition, the subject areas may be compared by comparing the amount of change in the mask position change with a threshold value based on at least one of the detection result and the determination result recorded in the history recording unit. .

  In addition, a history recording unit that records a detection result by the region detection unit and a determination result by the determination unit is provided, and the determination unit compares the subject regions based on at least a change in the size of the mask. In this case, the subject region is compared by comparing the amount of change in the size of the mask with a threshold value based on at least one of the detection result and the determination result recorded in the history recording unit. Also good.

  In addition, a history recording unit that records a detection result by the region detection unit and a determination result by the determination unit is further provided, and the determination unit includes the detection result and the determination result recorded in the history recording unit. Based on at least one, it is determined whether or not any subject area has been detected in the past, and if any subject area has been detected in the past, an arbitrary number less than the predetermined number If the same subject area is detected continuously or intermittently in the image by the area detection unit, it may be determined that the area is stable.

  The image forming apparatus further includes a second determination unit that determines whether or not the detection by the region detection unit is valid, and the determination unit compares the subject region in at least two images of the plurality of images. Thus, it may be determined whether or not the region is stable.

  Further, when a plurality of the subject regions are detected by the region detection unit, the determination unit stops determining whether the region is stable, and the setting unit sets the main subject region. May be canceled.

  The acquisition unit may sequentially acquire a plurality of image data generated intermittently in time instead of the plurality of images generated continuously in time as the plurality of images. .

  An imaging apparatus according to the present invention is an imaging apparatus that includes an imaging unit that captures an image by an optical system and generates image data, and the above-described image processing apparatus, and includes a sensor that detects the movement of the imaging apparatus. A motion detection unit; the acquisition unit acquires an image generated by the imaging unit; and the second determination unit is effective in detecting by the region detection unit based on a detection result of the motion detection unit. It is determined whether or not.

  Another imaging device of the present invention is an imaging device including an imaging unit that captures an image by an optical system to generate image data, and the above-described image processing device, and the plurality of images acquired by the acquisition unit The image processing apparatus further includes a motion detection unit that detects a motion of the imaging device based on at least two images, and the acquisition unit acquires an image generated by the imaging unit, and the second determination unit Determines whether the detection by the region detection unit is effective based on the detection result by the motion detection unit.

  Note that a control unit that controls at least one of the region detection unit, the determination unit, the setting unit, and the motion detection unit based on a determination result by the second determination unit may be further provided. .

  In addition, a control unit may be provided that stops detection of the subject area by the area detection unit based on a determination result by the second determination unit.

  In addition, when the determination unit detects again the subject region detected in the past by the region detection unit, and can estimate that the imaging device has made a predetermined movement based on the detection result by the motion detection unit, It may be determined that the region is stable.

  In addition, if the determination unit detects again the subject region detected in the past by the region detection unit and can estimate that the imaging apparatus has made a predetermined movement based on the detection result by the motion detection unit, It may be determined whether the region is stable by loosening the conditions than in other cases.

  Further, when the determination unit detects the subject region in the region detection unit, and can estimate that the imaging apparatus has continuously moved at a constant speed for a certain time or more based on a detection result by the motion detection unit. May determine that the region is stable.

  The setting unit can switch between a first mode in which a main subject region is set based on a determination result by the determination unit and a second mode in which a main subject region is set based on focus adjustment information. If the main subject area cannot be set in the first mode, the mode may be switched to the second mode.

  Another imaging device of the present invention includes an imaging unit that captures an image by an optical system to generate image data, and an acquisition unit that sequentially acquires a plurality of temporally generated image data from the imaging unit. Calculating a feature amount in the image indicated by the image data and detecting a subject region based on the feature amount; and among the images indicated by the plurality of image data acquired by the acquisition unit, A determination unit that determines whether the region detected by the region detection unit is stable by comparing the subject regions detected in at least two images, and the region is stabilized by the determination unit. A setting unit that sets the subject area that is the subject of the determination as a main subject area.

  The image processing program of the present invention calculates, in the computer, an acquisition procedure for sequentially acquiring a plurality of pieces of image data generated sequentially in time, and a feature amount of an image indicated by the image data. An area detection procedure for detecting a subject area on the basis of the area detected by comparing the subject areas detected in at least two images among images indicated by the plurality of image data acquired in the acquisition procedure; A determination procedure for determining whether or not the area detected by the detection procedure is stable, and if it is determined that the area is stable in the determination procedure, the subject area that is the object of the determination is a main subject area The setting procedure to be set as is executed.

  According to the present invention, the main subject region can be continuously extracted easily and accurately.

2 is a block diagram illustrating configurations of a lens barrel 10, an imaging device 20, and a storage medium 40. FIG. It is a flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is a figure explaining determination of whether the mask extracted by the mask extraction process is stable. It is another flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is another flowchart (continuation) which shows operation | movement of CPU26 at the time of automatic detection mode execution.

<First Embodiment>
Hereinafter, a first embodiment will be described in detail with reference to the drawings.

  In the first embodiment, an apparatus including a lens barrel 10, an imaging device 20, and a recording medium 40 as illustrated in FIG. 1 will be described as an example.

  The imaging device 20 captures an optical image incident from the lens barrel 10. The obtained image is stored in the storage medium 40 as a still image or a moving image.

  The lens barrel 10 includes a focus adjustment lens (hereinafter referred to as an “AF (Auto Focus) lens”) 11, a lens driving unit 12, an AF encoder 13, and a lens barrel control unit 14. The lens barrel 10 may be detachably connected to the imaging device 20 or may be integrated with the imaging device 20.

  The imaging device 20 includes an imaging unit 21, an image processing device 22, a display unit 23, a buffer memory unit 24, a storage unit 25, a CPU 26, an operation unit 27, and a communication unit 28. The imaging unit 21 includes an imaging element 29 and an A / D (Analog / Digital) conversion unit 30. The imaging unit 21 is controlled by the CPU 26 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  In the lens barrel 10, the AF lens 11 is driven by the lens driving unit 12 and guides an optical image to the light receiving surface (photoelectric conversion surface) of the imaging element 29 of the imaging device 20. The AF encoder 13 detects the movement of the AF lens 11 and outputs a signal corresponding to the movement amount of the AF lens 11 to the lens barrel control unit 14. Here, the signal corresponding to the movement amount of the AF lens 11 may be, for example, a sine wave signal whose phase changes according to the movement amount of the AF lens 11.

  The lens barrel control unit 14 controls the lens driving unit 12 in accordance with a drive control signal input from the CPU 26 of the imaging device 20. Here, the drive control signal is a control signal for driving the AF lens 11 in the optical axis direction. The lens barrel control unit 14 changes, for example, the number of steps of the pulse voltage output to the lens driving unit 12 according to the drive control signal. Further, the lens barrel control unit 14 outputs the position (focus position) of the AF lens 11 in the lens barrel 10 to the CPU 26 of the imaging device 20 based on a signal corresponding to the movement amount of the AF lens 11. Here, the lens barrel control unit 14 integrates, for example, signals according to the movement amount of the AF lens 11 according to the movement direction of the AF lens 11, thereby moving the AF lens 11 in the lens barrel 10 ( Position) may be calculated. The lens driving unit 12 drives the AF lens 11 according to the control of the lens barrel control unit 14 and moves the AF lens 11 in the optical axis direction within the lens barrel 10.

  In the imaging device 20, the imaging element 29 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 10 (optical system) into an electric signal, and supplies the electric signal to the A / D conversion unit 30. Output. The imaging element 29 is configured by a photoelectric conversion element such as a CMOS (Complementary Metal Oxide Semiconductor), for example. Further, the image sensor 29 may convert an optical image into an electric signal for a partial region of the photoelectric conversion surface (image cutout). Further, the image sensor 29 outputs an image obtained when a photographing instruction from the user is received via the operation unit 27 to the storage medium 40 via the A / D conversion unit 30 and the communication unit 28. On the other hand, the image pickup device 29 converts the continuously obtained image into a through image and converts it into a buffer memory unit 24 and the display unit 23 in the state before accepting a shooting instruction from the user via the operation unit 27. The data is output via the unit 30.

  The A / D conversion unit 30 digitizes the electrical signal converted by the image sensor 29 and outputs an image, which is a digital signal, to the buffer memory unit 24 and the like.

  The image processing device 22 performs image processing on the image temporarily stored in the buffer memory unit 24 based on the image processing conditions stored in the storage unit 25. The image after image processing is stored in the storage medium 40 via the communication unit 28. Further, the image processing device 22 performs mask extraction processing on the image temporarily stored in the buffer memory unit 24 (details will be described later). The extracted mask information is output to the CPU 26 and stored in the storage unit 25, the storage medium 40, and the like.

  The display unit 23 is, for example, a liquid crystal display, and displays an image generated by the imaging unit 21, an operation screen, and the like. The buffer memory unit 24 temporarily stores the image generated by the imaging unit 21. The storage unit 25 stores imaging conditions, determination conditions referred to by the CPU 26 in various determinations, and the like.

  The CPU 26 appropriately acquires necessary information from the image processing unit 22, the storage unit 25, and the like, and comprehensively controls each unit in the imaging device 20 based on the acquired information. Control by the CPU 26 includes focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, flash emission amount change setting, subject tracking setting, and various shooting mode settings. , Various image processing settings, various display settings, brightness optimization settings linked to zoom magnification, and the like. In addition, the CPU 26 monitors the operation state of the operation unit 27 and outputs image data to the display unit 23.

  The operation unit 27 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys. The operation unit 27 receives a user operation input when operated by the user, and outputs a signal corresponding to the operation input to the CPU 26. Output to.

  The communication unit 28 is connected to a removable storage medium 40 such as a card memory, and writes, reads, or deletes information (image data, area information, etc.) to the storage medium 40.

  The storage medium 40 is a storage unit that is detachably connected to the imaging device 20 and stores information (image data, area information, and the like). Note that the storage medium 40 may be integrated with the imaging device 20.

  The imaging device 20 includes an automatic detection mode for automatically detecting the main subject area in addition to the normal mode for detecting the main subject area based on the focus adjustment information at the time of shooting. In the automatic detection mode, a main subject area is automatically and continuously detected based on a through image for composition confirmation, etc., and information on the detected main subject area is displayed on the display unit 23, and the buffer memory unit 24 and storage This mode is stored in the unit 25 or the like. This automatic detection mode may be set by a user operation via the operation unit 27, or may be automatically set by the CPU.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S <b> 101, the CPU 26 controls the imaging unit 21 to start acquiring a through image. The acquired image information of the through image is temporarily stored in the buffer memory unit 24. This through image is continuously generated at a predetermined time interval. The acquisition of the through image by the CPU 26 is sequentially performed sequentially in time.

  In step S102, the CPU 26 controls the image processing device 22 to perform normal image processing. Normal image processing includes white balance adjustment, interpolation processing, color tone correction processing, gradation conversion processing, and the like. Since the specific method of each process is the same as that of a well-known technique, description is abbreviate | omitted. The image processing device 22 acquires the image data of the target image from the buffer memory unit 24, performs image processing, and then outputs the image data to the buffer memory unit 24 again.

  In step S103, the CPU 26 controls the image processing device 22 to perform mask extraction processing. The mask extraction process is a technique for calculating a feature amount in an image and detecting a subject area based on the feature amount. For example, the mask extraction is performed by obtaining an evaluation value from the feature amount in the image and obtaining a continuous region having the same evaluation value. Any method of mask extraction may be used. In addition, since a specific method for extracting a mask is the same as that of a known technique, description thereof is omitted.

  In step S104, the CPU 26 determines whether or not the subject area has been extracted. If the CPU 26 determines that the subject area has been extracted, the CPU 26 proceeds to step S105. On the other hand, if it is determined that the subject area cannot be extracted, the CPU 26 returns to step S102 and performs the processing from step S102 onward for the image of the next frame.

  Whether or not the subject area has been extracted means that the mask extraction has succeeded by the mask extraction processing in step S103. For example, if the feature amount cannot be calculated or the evaluation value is too low in the mask extraction process in step S103, the mask extraction fails and the subject area cannot be extracted. If the subject area cannot be extracted, the CPU 26 repeats the normal image processing in step S102 and the mask extraction process in step S103 until the subject area can be extracted without performing the subsequent processing.

  In step S105, the CPU 26 records the mask information in the buffer memory unit 24, the storage unit 25, and the like. The mask information includes information such as the extraction conditions (color classification, mask classification, etc.) of the mask extraction process in step S103, the position of the mask, and the size and shape of the mask.

  In step S106, the CPU 26 determines whether or not the same mask has been extracted continuously n times (n frames) or more. When the CPU 26 determines that the same mask has been extracted n times or more consecutively as shown in FIG. 3, the CPU 26 proceeds to step S108 described later. On the other hand, when determining that the same mask has not been extracted n times or more in succession, the CPU 26 proceeds to step S108.

  The case where the same mask is extracted is a case where masks of substantially the same color and size are extracted at substantially the same position in a plurality of frames. For this determination, the mask information described in step S105 may be used. In any case, when the amount of change between frames is within a specified range, it can be determined that they are the same. In this way, by comparing the extracted masks, the subject regions that are candidates for the main subject region can be compared between frames. Based on the mask information and the like described in step S105, the tendency of mask extraction performed in the past (extracted mask type, appearance frequency, etc.) is obtained, and the amount of change described above is within the specified range according to this tendency. The threshold for determining whether or not can be changed as appropriate. For example, the threshold value may be changed to a larger value for a mask extracted at a high frequency in the past.

  This determination is for determining whether or not the mask extracted by the mask extraction process in step S103 is stable. If the mask extracted by the mask extraction process in step S103 is stable, it can be determined that the main subject region is correctly detected. Note that n described above is a predetermined threshold (for example, n = 7), and is appropriately determined based on the frame rate at the time of image capturing by the image capturing unit 21, the zoom magnification in the lens barrel 10, the user operation via the operation unit 27, and the like. It may be changeable. Further, the tendency of mask extraction performed in the past (the type of extracted mask, the appearance frequency, etc.) is obtained based on the mask information described in step S105, and the value of n described above is appropriately set according to this tendency. It may be changeable. For example, for a mask extracted at a high frequency in the past, the value of n may be changed to a smaller value.

  Instead of determining whether the same mask has been extracted n times (n frames) or more, it is determined whether the same mask has been extracted N / M times (N / M frames). Also good. For example, when N = 3 and M = 5 and the same mask is extracted 3 times (3 frames) out of 5 times (5 frames), the mask extracted by the mask extraction process in step S103 is stable. You may determine that you are doing. N and M may be appropriately changed in the same manner as the value of n described above.

  Further, the contents of the determination made in step S106 may be recorded in addition to the mask information described in step S105, and used in subsequent processing relating to frames.

  In step S107, the CPU 26 determines whether or not the same mask has been extracted continuously until the previous time. For this determination, the mask information described in step S105 may be used. If the CPU 26 determines that the same mask has been extracted continuously until the previous time, the process proceeds to step S108. On the other hand, when determining that the same mask has not been extracted n times or more in succession, the CPU 26 proceeds to step S111 described later.

  If the same mask is extracted continuously until the previous time, there is a possibility that temporary unextraction has occurred only this time. The unextraction described above occurs due to a temporary change in the subject, a change in the light source, an error in extraction processing, and the like. Therefore, in such a case, if the same mask is extracted in the next frame without invalidating the previous accumulation due to temporary unextraction, the stable state up to the previous time is maintained. Then, the process proceeds to step S108 described later. It should be noted that when such a temporary unextraction occurs, the configuration may return to step S102. Further, after returning to step S102, the value of n in step S106 may be changed to a smaller value. For example, although it is normally determined that it is stable after 7 times or more, an example is considered in which it is determined that it is stable after 3 times or more after a temporary undetection occurs.

  In step S108, the CPU 26 sets a main subject area based on the extracted mask. The main subject area is obtained based on the mask extracted in step S103 in the image of the frame determined in step S106 or step S107.

  In step S109, the CPU 26 records information on the main subject area in the buffer memory unit 24, the storage unit 25, and the like. The information on the main subject area includes mask information related to the mask used, information on the position of the main subject area, and the size and shape of the main subject area.

  In step S <b> 110, the CPU 26 displays information on the main subject area on the display unit 23. The CPU 26 controls the display unit 26 to display the main subject area so as to be visible by superimposing it on the through image acquired in step S101. For example, the CPU 26 displays a frame or the like in the main subject area. The frame may be in accordance with the shape of the main subject area, or may be a predetermined shape such as a rectangle or a circle. Also, the line thickness, shading, color, etc. may be anything. In any case, by performing such display, the user can easily grasp what the main subject area is automatically detected by the CPU 26 is.

  In step S111, the CPU 26 determines whether or not the setting has failed continuously m times (m frames) or more. If the CPU 26 has not been able to perform the main subject area setting process described in step S108 continuously for at least m times, the CPU 26 determines that the process has failed continuously for at least m times and proceeds to step S112. On the other hand, when determining that it has not failed continuously for m times or more, the CPU 26 proceeds to step S113 described later. Note that m described above is a predetermined threshold (for example, m = 30), and may be appropriately changed based on a user operation via the operation unit 27.

  In step S112, the CPU 26 determines that detection in the automatic detection mode is difficult, changes to the normal mode in which the main subject region is detected based on the focus adjustment information, and ends the series of processes. In addition, when changing from automatic detection mode, you may alert | report to a user using the display part 23 grade | etc.,.

  In step S113, the CPU 26 performs normal 3A processing in preparation for a case where a shooting instruction is issued in a state where the main subject area setting processing is not performed. The 3A process is a focus adjustment (AF) setting process, an exposure adjustment (AE) setting process, and a white balance adjustment process (AWB), and is performed in the same manner as a known technique.

  If an arbitrary subject area has been set as the main subject area by the processing of step S108, the CPU 26 cancels the setting and ends the display of information on the main subject area on the display unit 23. Proceed to S114.

  In step S114, the CPU 26 determines whether or not a shooting instruction has been issued. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S115. On the other hand, if it is determined that the shooting instruction is not performed, the CPU 26 returns to step S102 and performs the processing after step S102 on the image of the next frame. The shooting instruction is given by a user operation via the shutter button of the operation unit 27. This user operation may be a so-called half shutter or all shutter.

  In step S115, the CPU 26 controls each unit to perform shooting. At this time, the CPU 26 performs shooting based on information on the main subject area set in step S108. When the main subject area is set in step S108, the CPU 26 sets focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, and white balance adjustment based on the set main subject area information. In addition to performing 3A processing (AWB), conditions for various image processing in the image processing apparatus 22 are determined. When normal 3A processing is performed in step S113, focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, and white balance adjustment processing (AWB) are performed based on the result of the 3A processing. 3A processing is performed, and various image processing conditions in the image processing device 22 are determined.

  In step S116, the CPU 26 records the image generated by the imaging on the storage medium 40 via the communication unit 28, and ends the series of processes.

Through the series of processes described above, the CPU 26 can continuously detect the transition state of the main subject. Transition states include the following.
Detection: A certain subject is set as a main subject region by performing mask extraction stably over a specified frame (step S106 → step S108, etc.).
Continuation: The set main subject area is stably mask extracted even in subsequent frames (step S108, etc.).
Cancel: The set main subject area disappears in the subsequent frame, and the setting is canceled (step S113, etc.).
Update: The set main subject area disappears in the subsequent frame, and a new subject is set as the main subject area (step S113 → (omitted) → step S106 → step S108, etc.).
Restoration: The set main subject area once disappears in the subsequent frame, appears again, and is set as the main subject area (step S107 → step S108, etc.).

  As described above, according to the first embodiment, a plurality of pieces of image data generated successively in time are sequentially acquired, the feature amount of the image indicated by the image data is calculated, and based on the feature amount. Detect the subject area. Then, by comparing subject areas detected in at least two images of the plurality of images, it is determined whether the detected area is stable. The set subject area is set as the main subject area.

  Usually, the most common case of shooting is a case of shooting a stationary subject. In such a case, there is little movement of the imaging apparatus itself, and there is almost no moving object in the screen to be imaged. Therefore, with the above-described configuration, the main subject area can be continuously extracted easily and accurately. In particular, unlike a conventional so-called tracking technique, a user's designation operation is not required, and even when a new subject appears, it is possible to respond flexibly. Further, unlike a so-called face recognition technique in the past, even a subject other than a face can be set as a main subject region.

  Further, according to the first embodiment, the set main subject area is displayed so as to be visible so that the positional relationship with the image can be understood. Therefore, the user can easily grasp what the main subject area is automatically detected by the imaging apparatus.

  Further, according to the first embodiment, information regarding the set main subject area is recorded. Therefore, information on the main subject area can be used preferentially at the time of shooting or image processing after shooting.

Second Embodiment
Hereinafter, the second embodiment will be described in detail with reference to the drawings. The second embodiment is a modification of the first embodiment. Therefore, only the parts different from the first embodiment will be described, and the description of the same parts as the first embodiment will be omitted.

  In the second embodiment, a description will be given by taking as an example an apparatus including the lens barrel 10, the imaging device 20, and the recording medium 40 shown in FIG. 1 of the first embodiment. However, in the second embodiment, the imaging device 20 further includes a motion sensor (not shown) that detects the motion of the imaging device 20 itself. This motion sensor is constituted by an acceleration sensor or the like, and the detection result is output to the CPU 26.

In the first embodiment, as described above, the processing has been described on the premise of the case of photographing a stationary subject. In the second embodiment, processing that can handle various cases shown in 1) to 5) will be described below. First, each case will be described, and then a series of processes for dealing with each case will be described.
1) Shooting of a stationary subject The movement of the imaging apparatus itself is small, and there is almost no moving object in the screen of the shooting target. In such a case, since the composition is almost determined, the same processing as in the first embodiment is performed.
2) Shooting a moving subject (moving vehicle, animal, etc.) The amount of movement of the imaging device itself is relatively below a certain level, and there is a moving object in the screen to be imaged. In such a case, the composition is roughly determined, and there is a high possibility that the moving subject is being photographed in a favorite state and a favorite position. Therefore, even if the position of the extracted mask changes to some extent, the process of setting as the main subject region is performed.
3) Shooting from a moving vehicle during composition determination The amount of movement of the image pickup apparatus itself is a certain level and relatively large, and the moving speed is slower than a general framing speed. In such a case, the subject to be photographed is determined, and there is a high possibility that the subject is being confirmed, the composition is being adjusted, and the photo opportunity is on standby. For this reason, while continuing extraction, the process waits for setting of the main subject area, and performs processing for setting the main subject area when it is stabilized to some extent.
4) While searching for a subject The amount of movement of the imaging apparatus itself is a certain level and relatively large, and the moving speed is slightly faster than 3) described above. In such a case, there is a high possibility that the subject is being searched. Therefore, a process of waiting for setting of the main subject area is performed while the extraction is continued.
5) Panning shooting during tracking of the subject The imaging device itself is moving at a certain moving speed and is following the subject near the center of the screen to be shot. In such a case, there is a high possibility that the subject is being tracked or a panning shot is being taken. For this reason, while continuing extraction, the process waits for setting of the main subject area, and performs processing for setting the main subject area when it is stabilized to some extent. However, in the case of panning shots, the probability that the extracted mask is the main subject area is extremely high, so if the extracted mask once disappears and reappears (subject restoration), the condition for determining that it is stable May be loosened.

  In any case, basically, it is determined whether or not the mask extraction itself is effective, and only the subject area based on the mask extracted by the mask extraction determined to be effective is targeted by the mask extraction process. It is determined whether or not the extracted mask is stable.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowcharts of FIGS. 4 and 5.

  In step S <b> 201, the CPU 26 controls the imaging unit 21 and starts acquiring a through image, similarly to step S <b> 101 of the first embodiment.

  In step S202, the CPU 26 controls the image processing device 22 to perform normal image processing, similarly to step S102 of the first embodiment.

  In step S <b> 203, the CPU 26 detects the amount of motion of the imaging device 20 based on at least two images acquired successively. The amount of movement is detected in the same manner as in the known technique. For example, the amount of motion can be detected by image analysis, frame difference, matching error, motion vector, and the like. The amount of motion detected here may be used as appropriate for a mask extraction process in step S207 described later, a determination process in steps S210 to S213 described later, and each process in steps S218 to S220. For example, the process may be performed by adding the amount of motion to the mask extraction process in step S207. When the amount of motion is equal to or greater than the predetermined upper limit amount, the amount of motion of the imaging device 20 is too large and is not suitable for automatic detection. Therefore, the process returns to step S202 without performing the subsequent processing. In this case, the normal image processing in step S202 and the detection of the motion amount in step S203 are repeated until the motion amount becomes a motion amount suitable for automatic detection.

  In step S204, the CPU 26 determines whether or not the amount of motion detected in step S203 is equal to or greater than a predetermined threshold value Ta. If the CPU 26 determines that the amount of motion detected in step S203 is greater than or equal to the predetermined threshold Ta, the CPU 26 proceeds to step S205. On the other hand, when determining that the amount of motion detected in step S203 is less than the predetermined threshold Ta, the CPU 26 proceeds to step S206 described later.

  This determination is for determining whether or not the imaging apparatus 20 itself can be regarded as stationary. If the imaging device 20 itself can be considered to be stationary, the CPU 26 proceeds to step S206 described later in order to perform mask extraction. On the other hand, if the imaging apparatus 20 itself cannot be regarded as stationary, the CPU 26 proceeds to step S205 to obtain more detailed information. Note that Ta described above is a predetermined threshold determined empirically, and may be changed as appropriate based on a user operation via the operation unit 27 or the like.

  In step S205, the CPU 26 determines whether or not the output value of a motion sensor (not shown) is equal to or greater than a predetermined threshold value Tb. If CPU26 determines with the output value of a motion sensor being more than predetermined threshold value Tb, it will return to step S202. On the other hand, when determining that the output value of the motion sensor is less than the predetermined threshold value Tb, the CPU 26 proceeds to step S206.

  This determination is for determining how fast the imaging apparatus 20 itself is moving. When the amount of movement of the imaging device 20 itself is too large, there is a high possibility that the imaging device 20 is moving at a high speed or that rapid zooming is being performed. Thus, if it is not suitable for automatic detection, the CPU 26 returns to step S202 without performing the subsequent processing. On the other hand, when the amount of motion of the imaging device 20 itself is small to some extent, the CPU 26 proceeds to step S206 in order to perform mask extraction. Note that Tb described above is a predetermined threshold that is determined empirically, and may be appropriately changed based on a user operation via the operation unit 27.

  In step S206, the CPU 26 records the motion information in the buffer memory unit 24, the storage unit 25, and the like. The motion information includes information such as the amount of motion detected in step S203 and the output value of the motion sensor (not shown) described in step S205.

  In step S207, the CPU 26 controls the image processing device 22 to perform mask extraction processing, as in step S103 of the first embodiment. At this time, the CPU 26 may appropriately use the motion information recorded in step S206.

  In step S208, the CPU 26 determines whether or not the subject area has been extracted, as in step S104 of the first embodiment. If the CPU 26 determines that the subject area has been extracted, the CPU 26 proceeds to step S209. On the other hand, if it is determined that the subject area cannot be extracted, the CPU 26 returns to step S202, and performs the processing after step S202 on the image of the next frame.

  In step S209, the CPU 26 records the mask information in the buffer memory unit 24, the storage unit 25, and the like, similarly to step S105 of the first embodiment.

  In step S210, the CPU 26 determines whether or not the imaging device 20 is moving at a low speed. If the CPU 26 determines that the imaging device 20 is moving at a low speed, the CPU 26 proceeds to step S211. On the other hand, when determining that the imaging device 20 is not moving at a low speed, the CPU 26 proceeds to step S212 described later. This determination is performed by comparing the motion information recorded in step S206 with a predetermined threshold value.

  In step S211, the CPU 26 determines whether or not the tracking mask has reappeared. If the CPU 26 determines that the tracking mask has reappeared, the CPU 26 proceeds to step S218 described later. On the other hand, when determining that the tracking mask has not reappeared, the CPU 26 proceeds to step S213 described later. The tracking mask is a mask having a certain appearance frequency among masks extracted in the past by mask extraction and set as the main subject region. This determination can be made based on the mask information described in step S209. When the tracking mask reappears, the main subject area that has been set once disappears in the subsequent frame and reappears, or the imaging device 20 has moved continuously for a certain time at a certain speed. Includes cases that can be guessed. In such a case, the CPU 26 proceeds to step S218 to set the main subject area without determining whether or not the mask extracted by the mask extraction process in step S207 is stable. However, the main subject area may be set and the main subject area information only recorded, and the main subject information may not be displayed. This is because the subject area may change one after another and the display may be difficult to see.

  When the tracking mask reappears, instead of setting the main subject area (without determining whether or not the mask extracted by the mask extraction process in step S207 is stable), the process proceeds to step S207. The determination may be performed by loosening the conditions for determination regarding whether or not the mask extracted by the mask extraction process is stable. Further, when the tracking mask reappears, the extraction may be performed again by relaxing the extraction condition of the mask extraction process in step S207.

  In step S212, the CPU 26 determines whether or not the same mask has been extracted p times (p frames) or more in a manner substantially similar to step S106 of the first embodiment. If the CPU 26 determines that the same mask has been extracted continuously p times or more, the CPU 26 proceeds to step S218 described later. On the other hand, when determining that the same mask has not been extracted p times or more in succession, the CPU 26 proceeds to step S215 to be described later.

  Note that p described above is a predetermined threshold value, and may be changed as appropriate based on a frame rate at the time of image capturing by the image capturing unit 21, a zoom magnification in the lens barrel 10, a user operation via the operation unit 27, and the like. Furthermore, the tendency of mask extraction performed in the past (the type of extracted mask, the appearance frequency, etc.) is obtained based on the mask information described in step S209, and the value of p described above is appropriately set according to this tendency. It may be changeable. For example, for a mask extracted at a high frequency in the past, the value of p may be changed to a smaller value.

  In step S213, the CPU 26 determines whether or not the same mask has been extracted q times (q frames) or more in a manner substantially similar to step S106 of the first embodiment. If the CPU 26 determines that the same mask has been extracted continuously q times or more, the process proceeds to step S214. On the other hand, if it is determined that the same mask has not been extracted q times or more, the CPU 26 proceeds to step S215 to be described later.

  In this determination, the extraction condition of the mask extraction process in step S207 is relaxed and extraction is performed again, and it is determined whether or not the same mask has been extracted q times or more. In addition, q mentioned above is a predetermined threshold value, and is a value smaller than the threshold value p mentioned above. As a result, the case where it is determined in step S213 that the same mask has been extracted q times or more in succession is a case where the target mask corresponds to the tracking mask described in step S211. Proceed to step S214 for setting.

  In step S214, the CPU 26 records the mask information in the buffer memory unit 24, the storage unit 25, and the like. The mask information in step S214 is information indicating that the target mask is the tracking mask described in step S211.

  In step S215, the CPU 26 determines whether or not the setting has failed continuously for m times (m frames) or more in substantially the same manner as in step S111 of the first embodiment. If the CPU 26 has not been able to perform the main subject area setting process described in step S218 continuously for m times or more, the CPU 26 determines that the process has failed continuously for m times or more and proceeds to step S216. On the other hand, when determining that it has not failed continuously for m times or more, the CPU 26 proceeds to step S217 described later.

  In step S216, as in step S112 of the first embodiment, the CPU 26 determines that detection in the automatic detection mode is difficult, and changes to the normal mode in which the main subject region is detected based on the focus adjustment information. Terminate the process.

  In step S217, as in step S113 of the first embodiment, the CPU 26 performs a normal 3A process in preparation for a case where a shooting instruction is issued in a state where the main subject area setting process is not performed.

  In step S218, the CPU 26 sets a main subject area based on the extracted mask, as in step S108 of the first embodiment.

  In step S219, the CPU 26 records information on the main subject area in the buffer memory unit 24, the storage unit 25, and the like, as in step S109 of the first embodiment.

  In step S220, the CPU 26 displays information on the main subject area on the display unit 23 as in step S110 of the first embodiment.

  In step S221, the CPU 26 determines whether or not a shooting instruction has been issued, in substantially the same manner as in step S114 of the first embodiment. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S222. On the other hand, if it is determined that no shooting instruction is given, the CPU 26 returns to step S202, and performs the processing from step S202 onward for the image of the next frame.

  In step S222, the CPU 26 performs shooting by controlling each unit as in step S115 of the first embodiment.

  In step S223, as in step S116 of the first embodiment, the CPU 26 records an image generated by imaging on the storage medium 40 via the communication unit 28, and ends the series of processes.

  In the flowcharts of FIG. 4 and FIG. 5, an example is shown in which the motion amount obtained based on the image in step S204 is compared with the threshold value Ta, and the output value of the motion sensor is compared with the threshold value Tb in step S205. Any one of them may be performed. That is, without providing a motion sensor, the amount of motion obtained based on the image in step S204 may be compared with the threshold value Ta to determine subsequent processing, or without obtaining the amount of motion based on the image, In step S205, the output value of the motion sensor and the threshold value Tb may be compared to determine subsequent processing.

  Further, the cases 1) to 5) described above may be determined based on image analysis, scene analysis, shooting mode type, and the like, and a predetermined process may be performed for each case. In this case, it is preferable to determine an appropriate threshold value and determination condition for each case.

  As described above, according to the second embodiment, whether or not the detection of the subject area is valid is determined, and the detection is performed by comparing the subject areas detected by the detection determined to be valid. It is determined whether or not the subject area is stable. In general, until the composition and the angle of view are determined to some extent, the detection of the subject area and the extraction of the main subject area are likely to be wasteful and often not effective. With the above-described configuration, in addition to the effects of the first embodiment, it is possible to cope with the problem that unnecessary subject detection and main subject region extraction are performed. In particular, the display and recording associated with the setting of the main subject area may be disadvantageous for the user if performed in the dark. With the above-described configuration, display and recording associated with the setting of the main subject area can be performed only when an effect can be expected.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In each of the embodiments described above, when a plurality of subject areas are extracted by mask extraction, the following processing (a) or (b) is performed.

When a plurality of subject areas are extracted by mask extraction, the following processes can be considered.
(A) The determination of whether or not the mask extracted in the mask extraction process is stable is stopped.

Only the mask extraction is continuously performed, and the determination of whether the mask is stable and the setting process of the main subject area are stopped. This process is the same whether a series of processes is started or a plurality of subject areas are extracted during the process.
(B) The subsequent processing is performed in accordance with preset conditions.

  For example, each of the subsequent processes may be performed only on the subject area with the highest evaluation value, or each of the subsequent processes may be performed on all of the plurality of subject areas. The subsequent processes may be performed for only a part of the subject area. When only some of the subject areas are targeted, the display unit 23 or the like may be used to cause the user to select the subject area, and the evaluation value, position, and appearance frequency may be selected. For example, a configuration may be adopted in which a part of the subject area is automatically selected.

  When a plurality of subject areas are set as the main subject areas, it is preferable to devise display contents when displaying information on the main subject areas on the display unit 23 (step S110 in FIG. 2, step S220 in FIG. 5). . For example, it is preferable to make the distinction possible by changing the color of the display frame, changing the thickness of the line, changing the density of the line, changing the type of line (solid line / dotted line), and the like.

  In addition, when performing the following processes for only a part of the subject areas of the plurality of subject areas, perform a part of the processes for the subject areas that are not targeted, and only record information. Also good. When a plurality of subject areas are extracted, an error display may be performed using the display unit 23 or the like.

  In each of the above-described embodiments, an example in which an automatic detection mode for automatically detecting a main subject area and a normal mode for detecting a main subject area based on focus adjustment information has been described. An S1 activation mode (details will be described later) may be provided, or an S1 activation mode may be provided in addition to the normal mode and the automatic detection mode.

  Unlike the automatic detection mode in each embodiment described above, the S1 activation mode is different from the automatic detection mode in each of the above-described embodiments. The main subject region setting and the main subject region information are set based on a user operation via the operation unit 27 (for example, a half shutter of a shutter button). This mode is for displaying and recording. This S1 activation mode may be used in place of the normal mode described in the above-described embodiments. For example, when the detection in the automatic detection mode is difficult, the S1 activation mode may be executed.

  Moreover, in S1 starting mode, it is good also as a structure which loosens various conditions rather than the automatic detection mode in each embodiment mentioned above. That is, the extraction conditions for mask extraction may be relaxed (easy to extract), or the determination conditions for determining whether the mask extracted by the mask extraction process is stable may be relaxed. Conversely, in the automatic detection mode in each embodiment described above, various conditions may be stricter than in the S1 activation mode. This is to make the accuracy as high as possible instead of performing automatic detection.

  In each of the above embodiments, the automatic detection mode interruption processing and resumption processing may be appropriately executed. For example, as described above, after changing from the automatic detection mode to the normal mode or the S1 activation mode, the automatic detection mode may be resumed according to the amount of motion of the imaging device 20 or the output of the motion sensor. Further, for example, even when detection in the automatic detection mode is difficult, the processing may be continued without changing to the normal mode or the S1 activation mode.

  In each of the above-described embodiments, an example in which normal 3A processing is performed in preparation for a case where a shooting instruction is performed in a state where the main subject region setting processing is not performed has been described. It is not limited to examples. For example, when a predetermined user operation including a photographing instruction is performed in a state where the main subject area setting process is not performed, the main information can be simply simplified using the mask information (step S105 in FIG. 2, step S209 in FIG. 4). A configuration may be adopted in which a subject area is set.

  In each of the above embodiments, an example in which a series of processing is performed based on a through image for composition confirmation has been described, but the present invention is not limited to this example. For example, the present invention can be similarly applied to a case where a live view image for composition confirmation generated in a single-lens reflex camera or the like is targeted. Further, the present invention can be similarly applied to a case where a moving image recorded on the recording medium 40 or the like is targeted.

  In each of the above embodiments, an example in which a series of processing is performed for all frames has been described. However, the present invention is not limited to this example. For example, a plurality of images generated intermittently in time may be targeted. Specifically, a plurality of images subjected to frame thinning as appropriate may be targeted. In this case, a series of processing in each of the above embodiments may be performed for a plurality of thinned images, and display may be performed for all images. By performing such processing, the processing load can be reduced.

  Further, the image processing described in each of the above-described embodiments may be realized by software by a “computer system” including a computer and an image processing program. In this case, the computer system may be configured to execute part or all of the processing of the flowcharts described in the embodiments. For example, part or all of the processing from step S101 to step S113 in FIG. 2 may be executed by a computer. Also, part or all of the processing from step S201 in FIG. 4 to step S220 in FIG. 5 may be executed by a computer. By adopting such a configuration, it is possible to perform the same processing as in the above-described embodiments.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a www system is used. The computer-readable recording medium is a writable nonvolatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, and a storage such as a hard disk built in the computer system. Refers to the device.

  Further, the computer-readable recording medium is a volatile memory (for example, DRAM (Dynamic Random Access) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory)) that holds a program for a certain period of time is also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 20 ... Imaging device, 21 ... Imaging part, 22 ... Image processing apparatus, 23 ... Display part, 26 ... CPU

Claims (1)

An acquisition unit that sequentially acquires a plurality of pieces of image data generated continuously in time;
A region detecting unit that calculates a feature amount of an image indicated by the image data and detects a subject region based on the feature amount;
By comparing the subject regions detected in at least two images among the images indicated by the plurality of image data acquired by the acquisition unit, the region detected by the region detection unit is stabilized. A determination unit for determining whether or not,
When the area by the determination unit is determined to be stable, and a setting unit for setting the object region as a target of the determination as the main subject area,
The determination unit determines whether any subject area has been recorded in the past based on at least one of the detection result of the area detection unit recorded in the recording unit and the determination result of the determination unit recorded in the recording unit. An image processing apparatus that determines whether or not a determination is made, and changes a criterion for determining that the determination is stable based on a determination result .

Images