JP2011160044A - Imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically replicate imaging control in accordance with the preference of a user. <P>SOLUTION: By using an object detection part for detecting each object on a target image by being sorted into one of a plurality of categories (persons, dogs, mountains and the like), the category, the size and the position of a focused object are learned by being related to a combination (for instance, a person and a mountain) of the detection categories of the plurality objects on the target image every time the target image is imaged. When a halfway pressing operation of a shutter button is performed after the completion of a certain amount of learning, object sorting for an input image at the time is performed, a learning content corresponding to a combination of detection categories for the input image is read from a learning memory, and automatic field angle adjustment and automatic focus adjustment in accordance with the read content are performed. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital camera.

  In recent years, digital cameras are often provided with a function of automatically selecting a shooting mode and shooting a target image (see, for example, Patent Document 1). This function automatically recognizes the subject and shooting scene on the camera side, selects the optimum shooting mode from a plurality of shooting modes, and sets the shooting conditions (focus, ISO sensitivity, signal processing) specified in the selected shooting mode. The target image is taken under the above conditions. However, the shooting conditions defined in the selected shooting mode as the optimum shooting mode are preset by the camera manufacturer. For this reason, some users need to perform a manual operation so that a truly desired shooting condition is set after camera control in the selected shooting mode is performed.

  As a method for dealing with such a problem, a method for allowing a user to select an optimum process in a question format has been proposed (for example, see Patent Document 2). However, this method places a burden on the response operation on the user.

  There has also been proposed a method for finely classifying subjects and controlling shooting conditions according to the classification result (see, for example, Patent Document 3). However, with this method, only the shooting mode is individually set for a specific subject, and control that takes into account the subject with which the subject of interest is captured and in what kind of shooting scene cannot be performed. . That is, for example, the optimal shooting conditions to be applied when a person is photographed with a dog may differ from the optimal shooting conditions to be applied when a person is photographed with a mountain. Although the specific contents vary depending on the user, the above method cannot cope with individual user preferences.

JP 2003-344891 A JP 2007-110619 A JP 2007-74141 A

  It is important to perform shooting control according to the user's preference (preference) without imposing a special burden on the user.

  Similarly, when acquiring an image accompanied by an acoustic signal, it would be beneficial if voice control according to the user's preference (preference) could be achieved without imposing a special burden on the user.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus that can perform imaging control or audio control in accordance with user preferences without imposing a special burden on the user.

  An image pickup apparatus according to the present invention includes an image pickup device that outputs a signal obtained by photoelectrically converting an optical image of a subject, and generates a target image from an output signal of the image pickup device obtained when a predetermined operation is performed. In the imaging device, a plurality of target images including the first and second target images are generated by repeating the predetermined operation, the second target image is generated after the first target image, and the imaging device A subject detection unit that detects and classifies each subject present on an image based on an output signal of the image sensor into any of a plurality of categories, and detection of the subject detection unit for a plurality of subjects on the first target image A memory unit that stores a combination of categories as a specific combination and stores learning information according to the characteristics of the first target image or the generation conditions of the first target image in association with the specific combination; An image based on an output signal of the imaging element after the generation of the first target image and before the generation of the second target image is used as an evaluation image, and the detection category of the subject detection unit for a plurality of subjects on the evaluation image And a shooting control unit that generates the second target image using the learning information when the combination matches the specific combination.

  This makes it possible to generate a target image using learning information that reflects user preferences. That is, it is possible to reproduce shooting control according to the user's preference from the learning information without imposing a special burden on the user.

  Specifically, for example, the learning information is information according to the characteristics of the first target image, and which category of subjects among the plurality of subjects on the first target image is in focus. Contains focus information that represents.

  As a result, the user's preference regarding the focus can be reproduced from the learning information.

  For example, the learning information further includes size information indicating the size of the subject in focus on the first target image.

  For example, the learning information further includes position information indicating the position of the focused subject on the first target image.

  Thus, the user's preference regarding the size and composition of the subject can be reproduced from the learning information.

  In addition, for example, the imaging apparatus further includes an operation unit that receives specification of a generation condition for the first target image, and the first target image is set according to the generation condition for the first target image specified via the operation unit. The learning information is information according to the generation condition of the first target image.

  As a result, the user-specified content via the operation unit can be stored in the learning information, and thereafter, the shooting control according to the user's preference can be reproduced from the learning information.

  Specifically, for example, when the combination of detection categories of the subject detection unit for a plurality of subjects on the evaluation image matches the specific combination, the shooting control unit responds to the characteristics of the first target image. Based on the learning information, focus control and zoom control are performed on the second target image so that the second target image has characteristics according to the characteristics of the first target image, or the first target image Based on the learning information according to the generation condition, the second target image is generated under a generation condition according to the generation condition of the first target image.

  Another imaging apparatus according to the present invention includes an imaging element that outputs a signal obtained by photoelectrically converting an optical image of a subject, and a target image is obtained from an output signal of the imaging element that is obtained when a predetermined operation is performed. The plurality of target images including the first and second target images are generated by repeating the predetermined operation, and the second target image is generated after the first target image, and the imaging device Includes a subject detection unit that detects a subject existing on an image based on an output signal of the image sensor by classifying it into any of a plurality of categories, and a plurality of registration scenes of an image based on the output signal of the image sensor A scene determination unit for determining by selecting from the scenes, a detection category of the subject detection unit for a subject on the first target image, and a determination scenario of the scene determination unit for the first target image. A memory unit that stores learning information according to characteristics of the first target image or generation conditions of the first target image in association with the specific combination, and generation of the first target image An image based on the output signal of the image sensor after and before the generation of the second target image is used as an evaluation image, and the detection category of the subject detection unit for the subject on the evaluation image and the scene determination for the evaluation image And a shooting control unit that generates the second target image using the learning information when the combination with the determination scene matches the specific combination.

  This makes it possible to generate a target image using learning information that reflects user preferences. That is, it is possible to reproduce shooting control according to the user's preference from the learning information without imposing a special burden on the user.

  Still another imaging device according to the present invention includes an imaging device that outputs a signal obtained by photoelectrically converting an optical image of a subject and a microphone unit including a plurality of microphones. When a predetermined operation is performed, the imaging device In the imaging device that generates a target image from the output signals of the plurality of microphones and generates a target sound signal based on the output sound signals of the plurality of microphones and associates the target sound signal with the target image, the first operation is repeated. A plurality of target images including a first target image and a second target image are generated, the second target image is generated after the first target image, and the imaging device is present on an image based on an output signal of the imaging element A combination of a subject detection unit that detects each subject to be classified into any of a plurality of categories and a detection category of the subject detection unit for a plurality of subjects on the first target image A memory unit that stores learning information corresponding to the characteristics of the target acoustic signal associated with the first target image in association with the specific combination, and after the generation of the first target image and the first The image based on the output signal of the image sensor before the generation of the two target images or the second target image is used as the evaluation image, and the combination of detection categories of the subject detection unit for the plurality of subjects on the evaluation image is the specified And a target acoustic signal generation unit configured to generate a target acoustic signal to be associated with the second target image using the learning information when matching the combination.

  Thereby, it becomes possible to generate the target acoustic signal using the learning information reflecting the user's preference. That is, it is possible to reproduce the voice control according to the user's preference from the learning information without imposing a special burden on the user.

  ADVANTAGE OF THE INVENTION According to this invention, it is possible to provide the imaging device which can perform imaging | photography control or audio | voice control according to a user's liking without imposing a special burden on a user.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is the figure which showed a mode that operation | movement of the special imaging | photography mode which concerns on embodiment of this invention is divided roughly into learning step operation | movement and control step operation | movement. FIG. 6 is a block diagram of a part related particularly to the operation of the special imaging mode according to the first embodiment of the present invention. It is a flowchart of the learning stage operation | movement which concerns on 1st Example of this invention. FIG. 4A is a diagram illustrating an example of a target input image captured during a learning stage operation, and FIG. 5B is a diagram illustrating a subject area on the target input image. It is a figure which shows the structure of the characteristic information which concerns on 1st Example of this invention. It is a figure which shows a mode that nine blocks are set in the whole image area | region of an input image. It is a figure which shows the recorded content of the learning memory at the time of execution of control step operation | movement concerning 1st Example of this invention. 3 is a flowchart of a control stage operation according to the first embodiment of the present invention. It is a figure which concerns on 1st Example of this invention and shows the image for evaluation acquired at the time of execution of control step operation | movement. It is a figure which shows the example of the input image which concerns on 1st Example of this invention and is obtained after an angle of view automatic adjustment. It is a figure which shows the input image assumed in 4th Example of this invention. It is a figure which shows the structure of the characteristic information which concerns on 4th Example of this invention. It is a block diagram of the site | part which concerns on 8th Example of this invention and is especially concerned in operation | movement of special imaging | photography mode. It is a flowchart of the learning stage operation | movement which concerns on 8th Example of this invention. It is a figure which shows the structure of the production | generation condition information based on 8th Example of this invention. It is a figure which shows the recorded content of the learning memory at the time of execution of control step operation | movement concerning 8th Example of this invention. It is a flowchart of the control step operation | movement which concerns on 8th Example of this invention. It is a figure which shows the scene determination part which concerns on 9th Example of this invention. It is an internal block diagram of the microphone part of FIG. It is an external appearance perspective view of the imaging device of FIG. It is a figure which concerns on 10th Example of this invention and shows the target input image acquired in learning step operation | movement. It is a figure which shows the structure of the sound control information based on 10th Example of this invention. It is a figure concerning the 10th example of the present invention and shows the contents of a learning memory recorded at the time of execution of a control phase operation. It is a flowchart of the control step operation | movement which concerns on 10th Example of this invention. It is a figure concerning the 10th example of the present invention and is a figure showing an image for evaluation acquired at the time of execution of a control stage operation. It is a figure which shows the mode of the display screen which concerns on 11th Example of this invention. It is a figure which shows the mode of the display screen which concerns on 12th Example of this invention. It is a figure which concerns on 13th Example of this invention and shows the characteristic information based on actual imaging | photography. It is a figure which concerns on 13th Example of this invention and shows the characteristic information produced | generated artificially.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging device 1 is a digital video camera, and can capture a moving image and a still image, and also can simultaneously capture a still image during moving image capturing. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. It should be noted that the display unit 27 and / or the speaker 28 can be interpreted as being provided in an external device (not shown) of the imaging device 1.

  The imaging unit 11 captures a subject using an imaging element. FIG. 2 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging device (solid-state imaging device) 33 including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the optical system 35 and the diaphragm 32. And a driver 34 for drive control. The optical system 35 is formed of a plurality of lenses including a zoom lens 30 for adjusting the angle of view of the imaging unit 11 and a focus lens 31 for focusing. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. Based on the photographing control signal from the CPU 23, the positions of the zoom lens 30 and the focus lens 31 and the opening of the diaphragm 32 in the optical system 35 are controlled.

  The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions. Each light receiving pixel of the image sensor 33 photoelectrically converts an optical image of a subject incident through the optical system 35 and the diaphragm 32, and outputs an electric signal obtained by the photoelectric conversion to an AFE 12 (Analog Front End).

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the image signal processing unit 13. The amplification degree of signal amplification in the AFE 12 is controlled by a CPU (Central Processing Unit) 23. The image signal processing unit 13 performs necessary image processing on the image represented by the output signal of the AFE 12, and generates an image signal representing the image after image processing. The image signal includes, for example, a luminance signal and a color difference signal. The microphone unit 14 converts the ambient sound of the imaging device 1 into an analog acoustic signal, and the acoustic signal processing unit 15 converts the analog acoustic signal into a digital acoustic signal.

  The compression processing unit 16 compresses the image signal from the image signal processing unit 13 and the acoustic signal from the acoustic signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and can record various signals such as an image signal and an acoustic signal after being compressed by the compression processing unit 16.

  The expansion processing unit 19 expands the compressed image signal and sound signal read from the external memory 18. The image signal after being decompressed by the decompression processing unit 19 or the image signal from the image signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the acoustic signal after being decompressed by the decompression processing unit 19 is sent to the speaker 28 via the acoustic signal output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, and a zoom button for designating a zoom magnification. 26c and the like, and accepts various operations by the user. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The operation mode of the imaging apparatus 1 includes a shooting mode in which an image (still image or moving image) can be shot and recorded, and an image (still image or moving image) recorded in the external memory 18 is reproduced and displayed on the display unit 27. Playback mode to be included. In accordance with an operation on the operation unit 26, transition between the modes is performed. In the shooting mode, the imaging apparatus 1 can sequentially acquire captured images of the subject by periodically shooting the subject at a predetermined frame period.

  In the present specification, an image signal of a certain image may be simply referred to as an image. In addition, since the compression and expansion of the image signal and the sound signal are not related to the essence of the present invention, the presence of the compression and expansion of the image signal and the sound signal is ignored in the following description unless otherwise required. Thus, for example, recording a compressed image signal for an image may be simply expressed as recording an image signal or recording an image (the same applies to an acoustic signal). Further, the size of a certain image or the size of an image area is also referred to as an image size. The image size of the target image or target image area can be expressed by the number of pixels forming the target image or the number of pixels belonging to the target image area. In this specification, recording of a signal or information in a memory may be expressed as storage. Moreover, in this specification, the name of the term corresponding to the symbol or code | symbol may be abbreviated by showing a symbol or code | symbol. Therefore, for example, the external memory 18 and the memory 18 indicate the same thing.

  The imaging device 1 extracts information related to the user's preference as a photographer from past shooting results, manual operations, etc., and based on that information, the shooting results that match the preference at the present time are reproduced. A special function is provided to control shooting. The operation mode of the image pickup apparatus 1 that is a kind of shooting mode and in which the special function operates is referred to as a special shooting mode.

  As shown in FIG. 3, the operation in the special shooting mode is roughly divided into a learning stage operation for learning the user's preference and a control stage operation for performing shooting control using the learning result. The control phase operation is executed through a learning phase operation. Even after the learning stage operation is once completed and the control stage operation can be executed, the user's preference learning can be continued and the learning result can be updated.

  Hereinafter, operations and configuration examples of the imaging apparatus 1 related to the special imaging mode will be described as first to thirteenth embodiments. As long as there is no contradiction, the matters described for one embodiment can be applied to other embodiments, and a plurality of embodiments of the first to thirteenth embodiments can be combined. is there. The following description is an explanation of the operation of the imaging apparatus 1 in the special imaging mode unless otherwise specified.

<< First Example >>
A first embodiment will be described. FIG. 4 is a block diagram of a part particularly related to the operation in the special imaging mode. The subject detection unit 51, the feature information generation unit 52, the memory control unit 53, and the imaging control unit 55 can be realized by the image signal processing unit 13 or a combination of the image signal processing unit 13 and the CPU 23. The learning memory 54 is formed from a nonvolatile memory such as a flash memory provided in the internal memory 17.

  An image itself represented by the output signal of the AFE 12 for one frame or a still image obtained by performing predetermined image processing (such as demosaicing processing or noise reduction processing) on the image is referred to as an input image. Further, an input image obtained according to a predetermined shutter operation is particularly called a target input image. The shutter operation is an operation of pressing the shutter button 26b. However, the shutter operation may be an operation other than the operation of pressing the shutter button 26b (for example, a predetermined touch panel operation).

[Learning stage operation]
FIG. 5 is a flowchart showing the procedure of the learning stage operation according to the first embodiment. In the learning stage operation, each process of steps S11 to S14 is executed. In step S11, the target input image is acquired by the user performing a shutter operation. In step S12, the subject detection unit 51 performs subject detection and subject category classification of the target input image. In step S <b> 13, the feature information generation unit 52 extracts and generates feature information that can be said to be information relating to user preference from the target input image. In step S14, the processing results of steps S12 and S13 are reflected in the recorded contents of the learning memory 54 (simply, for example, the feature information generated in step S13 is stored in the learning memory 54 as it is). Information recording control for the learning memory 54 is performed by the memory control unit 53. Of the respective parts shown in FIG. 4, only the imaging control unit 55 functions beneficially in the control stage operation. Hereinafter, the operation of each part in FIG. 4 will be described in detail with a specific example of the target input image.

  In FIG. 6A, reference numeral 300 represents an example of the target input image acquired in step S11. It is assumed that the target input image 300 is obtained by performing a shutter operation in a state where a person and a mountain are included in the shooting range as subjects. In FIG. 6A, reference numerals 301 and 302 represent subjects of the target input image 300. The subjects 301 and 302 are a person and a mountain, respectively.

  An image signal of each input image including the target input image is input to the subject detection unit 51 and the feature information generation unit 52. The subject detection unit 51 classifies and detects each subject present on the input image in any of a plurality of categories based on the image signal of the input image. In other words, for each subject existing on the input image, it is detected to which category the subject belongs. Categories can be read as types. The plurality of categories are categories registered in advance in the subject detection unit 51 and include persons, dogs, cats, birds, automobiles, mountains, sea, sky, flowers, and the like. Since the process (face detection process or the like) for detecting the category of each subject on the input image based on the image signal of the input image is known, a detailed description is omitted here.

  When the image signal of the target input image 300 is input to the subject detection unit 51, the subject detection unit 51 detects the subjects 301 and 302 from the target input image 300 based on the image signal of the target input image 300, and the subject 301 and An image region where 302 exists is extracted as subject regions 311 and 312, and it is detected to which category the subjects 301 and 302 belong (see FIG. 6B). As described above, since the subjects 301 and 302 are a person and a mountain, respectively, the subject detection unit 51 detects that the categories of the subjects 301 and 302 are a person and a mountain, respectively.

  Note that the subject region 311 may be an image region in which the entire image of the person as the subject 301 appears, but in this example, an image region including only the face portion of the person is extracted as the subject region 311. (The same applies to other examples described later). In FIG. 6B, each subject area to be extracted is a rectangular area, but each subject area may be other than the rectangular area (the same applies to other embodiments described later).

  The feature information generation unit 52 can generate focus information, in-focus subject size information, and in-focus subject position information for one input image. The focus information represents the category of the focused subject on the input image. The focused subject refers to a subject that is in focus. With respect to a certain input image, a subject positioned within the depth of field of the imaging device 1 when the input image is captured is included in the subject in focus. Focused subject size information (hereinafter sometimes abbreviated as size information) represents the size of the focused subject on the input image. The size of a certain subject refers to, for example, the image size of the subject area of the subject. Focused subject position information (hereinafter sometimes abbreviated as position information) represents the position of the focused subject on the input image. Strictly speaking, the position of a certain subject indicates, for example, the center position of the subject area of the subject.

  The feature information for an input image includes focus information, focused subject size information, and focused subject position information for the input image, and additional information is added to the feature information as necessary (see FIG. 7). Although it is possible to consider that the additional information is also a part of the feature information, in this example, it is considered that the additional information is not a component of the feature information. Arbitrary information can be included in the additional information, such as the time at the time of shooting the input image and weather information. The memory control unit 53 pays attention to a combination of categories of a plurality of subjects detected by the subject detection unit 51 with respect to the target input image, and associates the feature information with respect to the target input image with respect to the target combination. Save in the learning memory 54.

FIG. 7 shows the contents stored in the learning memory 54 for the target input image 300. Now, the subject in focus on the target input image 300 is the subject 301 (the subject 302 is not in focus), and the image size of the subject area 311 on the target input image 300 is SIZE _A1 . there are, and the center position of the subject region 311 on the target input image 300 is assumed to belong to the block BL _5. Then, the focus information for the target input image 300 is a person is a category of the object 301, and the size information and position information for the target input image 300 are respectively SIZE _A1 and BL _5. The fact that the focus information is a person means that the category of the subject in focus indicated by the focus information is a person.

The significance of the block BL ₅ will be described. FIG. 8 shows a focused input image. By dividing the noted input image into three equal parts along the horizontal and vertical directions, the entire image area of the noted input image is divided into nine equal parts, and the nine obtained image areas are defined as blocks BL _{1 to} BL _9. To do. The block BL ₅ described above is one of the blocks BL _{1 to} BL ₉ . It should be noted that 9 as the number of divisions of the input image (that is, the number of the blocks) is, of course, an example, and can be other than 9. However, in the following description, it is assumed to be 9.

  The feature information generation unit 52 generates feature information 305 including focus information, size information, and position information of the target input image 300, and the memory control unit 53 converts the feature information 305 into the subject category on the target input image 300. Are stored in the learning memory 54 in association with these combinations. The combination of categories of the subject on the target input image 300 is a combination of categories of the subjects 301 and 302, that is, a combination of “person” and “mountain”. Hereinafter, a combination of subject categories is also referred to as a category combination. Therefore, the combination of “person” and “mountain” is represented as a category combination “person and mountain”.

  The feature information generation unit 52 compares the AF evaluation value for the subject region 311 and the AF evaluation value for the subject region 312 and determines that the subject 301 is a focused subject if the former is larger than the latter, while the latter is more than the former. If it is larger, the subject 302 is determined as a focused subject, and focus information of the target input image 300 is generated from the determination result. In this example, since the former is larger than the latter, it is determined that the subject 301 is a focused subject. In addition, it is assumed that the subject in focus is included in the plurality of subjects on the target input image 300 detected by the subject detection unit 51 (other target input images and in-focus subjects are included). The same applies to any input image discussed).

  The AF evaluation value is calculated by an AF evaluation unit (not shown) in the image signal processing unit 13. It may be considered that the AF evaluation unit is inherent in the feature information generation unit 52. The AF evaluation unit divides the entire image area of the input image that is the target of AF evaluation value calculation into a plurality of AF evaluation blocks, and for each AF evaluation block, an AF evaluation value corresponding to the contrast amount of the image in the AF evaluation block Is calculated. The imaging apparatus 1 can perform autofocus control by a contrast detection method based on the AF evaluation value. The AF evaluation value of a certain AF evaluation block increases as the contrast of the image in the AF evaluation block (in other words, the edge strength) increases.

  Regarding the target input image 300, the average value of AF evaluation values of the AF evaluation blocks belonging to the subject area 311 is handled as the AF evaluation value of the subject area 311 and the average value of the AF evaluation values of the AF evaluation blocks belonging to the subject area 312 is used. May be handled as the AF evaluation value of the subject area 312 (assuming that a plurality of AF evaluation blocks belong to each subject area). Alternatively, the subject areas 311 and 312 may be handled as the first and second AF evaluation blocks, and the respective AF evaluation values may be calculated. Of a plurality of AF evaluation values calculated for a certain input image, an AF evaluation block corresponding to the maximum AF evaluation value or an AF evaluation block corresponding to an AF evaluation value equal to or greater than a predetermined threshold value, It can also be determined that this is an image area where a focused subject exists.

After detecting that the focused subject of the target input image 300 is the subject 301 as described above, the feature information generation unit 52 determines the size and center of the subject region 311 that is an image region where the image signal of the subject 301 exists. By detecting the position, size information (SIZE _A1 ) and position information (BL ₅ ) for the target input image 300 are generated. It should be noted that detection of the size and center position of the subject region 311 itself may be performed by the subject detection unit 51.

In the learning stage operation, the user's preference is repeatedly learned by repeatedly executing a series of processes consisting of steps S11 to S14. For one category combination, the number of learnings required to shift from the learning stage operation to the control stage operation is represented by _LNUM . L _NUM is set to an integer equal to or greater than 1. For the sake of concrete explanation, consider the case where L _NUM = 3. FIG. 9 shows the recorded contents of the learning memory 54 after the target input image 300 is acquired and the target input images 300a, 300b, 330, 330a, 330b, 331, and 331a (all not shown) are further acquired. Yes. Each category combination of the target input images 300 a and 300 b is “person and mountain”, and each category combination of the target input images 330, 330 a, and 330 b is “dog and sea”, and the target input image 331. And 331a are assumed to be “person and sea”.

The feature information generation unit 52 generates the feature information 305a of the target input image 300a and the feature information 305b of the target input image 300b by a method similar to the method of generating the feature information 305 of the target input image 300. When the feature information 305, 305a, and 305b is generated, the memory control unit 53 stores them in the learning memory 54 while associating them with the category combination “person and mountain”. On the other hand, the feature information generation unit 52 or the memory control unit 53 creates feature information W1 _A that matches or resembles the feature information 305, 305a, and 305b based on statistics, and the feature information W1 _A is also combined with the category combination “person and It is stored in the learning memory 54 in association with “mountain”. The feature information of each target input image is also called element feature information, and the feature information generated based on statistics from the feature information of a plurality of target input images is also called comprehensive feature information. In this example, each of the feature information 305, 305a, and 305b is element feature information, and the feature information W1 _A is comprehensive feature information.

The element feature information 305 in FIG. 9 is the same as that shown in FIG. Furthermore, as shown in FIG. 9, the focus information, size information, and position information in the element feature information 305a are person, SIZE _A2 and BL ₅ , respectively, and the focus information, size information, and position information in the element feature information 305b are , consider respectively, the person, the case is SIZE _A3 and BL _4. In this case, the focus information, size information, and position information in the general feature information W1 _A are a person, SIZE _A, and BL ₅ , respectively. In the additional information accompanying the information W1 _A , the number of learning times for the category combination “person and mountain” is recorded. The information W1 _A is generated based on the learning results for three times, that is, based on the feature information 305, 305a, and 305b. Accordingly, 3 is recorded in the additional information of the information W1 _A as a numerical value indicating the number of learnings.

Of focus information elements characteristic information as the original overall characteristic information W1 _A, the most frequent more focus information may be the focus information comprehensive feature information W1 _A. In the example of FIG. 9, since the focus information of the element feature information 305, 305a, and 305b is all “person”, the focus information of the comprehensive feature information W1 _A is also “person”. Even if only two of the focus information of the element feature information 305, 305a, and 305b are “person”, the focus information of the comprehensive feature information W1 _A is “person”.
Size information SIZE _A in General characteristic information W1 _A is, for example, a mean size information elements characteristic information as the original overall characteristic information W1 _A (i.e., the average of the SIZE _A1 ~SIZE _A3). However, the size information of the element feature information that does not match the general feature information W1 _A in the focus information is not reflected in the size information SIZE _A. Therefore, if the focus information of the element feature information 305, 305a, and 305b is “person”, “person”, and “mountain”, respectively, the average size information of the element feature information 305, 305a (that is, SIZE _A1 and SIZE). The average of _A2 ) is the size information SIZE _A.
Of the positional information of the element characteristic information as the original overall characteristic information W1 _A, the most frequent more location information may be a position information of the overall characteristic information W1 _A. However, the position information of the element characteristic information that does not match the overall characteristic information W1 _A in the focus information shall not reflected in the position information of the overall characteristic information W1 _A. In the example of FIG. 9, since the focus information of the element feature information 305, 305a, and 305b is all “person”, the position information BL ₅ with the highest frequency among the position information of the element feature information 305, 305a, and 305b is comprehensive. is the position information of the feature information W1 _a. If the focus information of the element feature information 305, 305a, and 305b is “person”, “mountain”, and “person”, respectively, the position information BL ₅ of the element feature information 305 or the position information BL _{4 of the} element feature information 305b. Is the position information of the general feature information W1 _A. In this case, if the target input image 300b has been shot after shooting of the target input image 300, giving priority to target input image 300b towards the newly captured, comprehensive feature position information BL ₄ elements characteristic information 305b The position information of the information W1 _A may be used.

In FIG. 9, the feature information group shown in the central portion is a feature information group for the category combination “dog and sea” based on the target input images 330, 330 a, and 330 b (not shown). W1 _B is comprehensive feature information for the category combination “dog and sea” based on the element feature information of the target input images 330, 330a, and 330b. The comprehensive feature information W1 _{B is} also created in the same manner as the comprehensive feature information W1 _A. In FIG. 9, the feature information group shown in the lower part is a feature information group for the category combination “person and sea” based on the target input images 331 and 331 a (not shown). In the state shown in FIG. 9, since there is only two element feature information for the category combination “person and sea”, the category combination “person and sea” cannot shift from the learning stage operation to the control stage operation. For the category combination “people and mountains” and “dogs and the sea”, the number of learning times has reached the required number of learning times L _NUM (= 3), so that the control stage operation can be executed.

After the target input images 300, 300a, and 300b are photographed, if the target input image 300c (not shown) whose category combination is “people and mountains” is photographed, the feature information of the latest target input image 300c is captured. It is preferable to update the comprehensive feature information W1 _A using. For example, the comprehensive feature information W1 _A can be recreated based on the element feature information of the images 300, 300a, 300b, and 300c, or based on the element feature information of the images 300a, 300b, and 300c. Further alternatively, without updating the overall characteristic information W1 _A at the time the target input image 300c is captured, when the category combinations "people and mountain" to become the target input image 300d and 300e are further captured, the target input image The comprehensive feature information W1 _A may be recreated based on the element feature information of 300c, 300d, and 300e.

[Control stage operation]
For the category combination “people and mountains”, the comprehensive feature information W1 _A of FIG. 9 based on the element feature information 305, 305a, and 305b is stored in the learning memory 54, and some comprehensive feature information other than the comprehensive feature information W1 _{A is} stored. For convenience, the state in which (including information W1 _B ) is stored in the learning memory 54 is referred to as a learning state in FIG. In the first embodiment, the control stage operation under the learning state of FIG. 9 will be described below. FIG. 10 is a flowchart showing the procedure of the control stage operation according to the first embodiment. In the control stage operation, each process of steps S21 to S28 is executed.

  First, in step S21, it is confirmed by the CPU 23 in FIG. 1 or the photographing control unit 55 in FIG. 4 whether the shutter button 26b is in a half-pressed state. The process proceeds to step S22, and the processes of steps S22 to S27 are sequentially executed. On the other hand, if it is not half-pressed, the confirmation operation of step S21 is repeatedly executed. The shutter button 26b can be pressed in two stages. When the user lightly presses the shutter button 26b, the shutter button 26b is half pressed, and when the shutter button 26b is further pressed from this state, the shutter button 26b is pressed. The state is fully pressed. The operation of setting the shutter button 26b to the fully depressed state is a shutter operation. The operation of setting the shutter button 26b to the half-pressed state is referred to as a half-press operation.

  Whether or not to move from step S21 to step S22 may be determined on the basis of the stationary state (in other words, the moving state) of the housing of the imaging device 1, not whether or not the half-press operation has been performed. That is, for example, when it is determined in step S21 that the housing of the imaging apparatus 1 is stationary for a certain period of time, the transition from step S21 to step S22 may be executed. When the amount of movement representing the magnitude of the movement of the housing of the imaging device 1 is not more than a predetermined value continuously for a certain time, it can be determined that the housing of the imaging device 1 is stationary for a certain time. . If a motion sensor (not shown) for detecting the movement of the housing of the imaging device 1 is provided in the imaging device 1, the amount of motion can be detected using the detection result of the motion sensor. Alternatively, the amount of motion can be detected based on an optical flow between input images acquired adjacent in time. The motion sensor is, for example, an angular velocity sensor that detects an angular velocity of the housing of the imaging device 1 or an acceleration sensor that detects an acceleration of the housing of the imaging device 1.

  In the shooting modes including the special shooting mode, the subject is periodically shot at a predetermined frame period (for example, 1/60 seconds) to sequentially acquire the input images, and the sequentially acquired input images are displayed one after another. It is updated and displayed on the section 27 (the same is true in the learning stage operation, and the same is true in other embodiments described later). In step S22, the subject detection unit 51 in FIG. 4 handles the latest input image as an evaluation image, and performs the above-described subject detection and subject category classification on the evaluation image. That is, based on the image signal of the evaluation image, each subject present on the evaluation image is classified into any of a plurality of categories and detected. Thereby, the category combination about the image for evaluation is determined by the same method as described above.

In step S23 subsequent to step S22, the imaging control unit 55 in FIG. 4 reads, from the learning memory 54, the comprehensive feature information of the category combination that matches the category combination of the evaluation image. For example, if the combination category of the evaluation image is “person and mountain”, the comprehensive feature information W1 _A is read, and if the combination category of the evaluation image is “dog and sea”, the comprehensive feature information W1. _B is read (see FIG. 9). In the following, the case where the evaluation image is the image 350 in FIG. 11 is considered. On the image 350, there are image signals of a subject 351 that is a person and a subject 352 that is a mountain. As a result, it is assumed that the combination category of the evaluation image 350 is determined to be “person and mountain”. Then, the comprehensive feature information W1 _A is read out in step S23. Hereinafter, the comprehensive feature information read in step S23 is also referred to as read feature information.

In step S24 subsequent to step S23, the imaging control unit 55 sets the subject indicated by the focus information of the readout feature information (the subject in the category indicated by the focus information of the readout feature information) as the main subject. Since the category of the subject indicated in the focus information of the general feature information W1 _A that is the read feature information is “person”, the person is set as the main subject.

An image region 361 in FIG. 11 is a subject region where an image signal of the subject 351 exists. The subject region 361 is detected based on the image signal of the evaluation image 350 by a method similar to the method for detecting the subject region 311 in FIG. It is assumed that the image size of the subject area 361 on the evaluation image 350 is SIZE _A ′. In step S25 following step S24, the imaging control unit 55, a SIZE _A is the size information of the read characteristic information, on the basis of the above SIZE _A ', to perform the angle automatic adjustment should be called a zoom automatic control.

The angle automatic adjustment, the optical zoom magnification required to change the image size indicating the size of the main object SIZE _A 'to SIZE _A was calculated as the target zoom magnification, the target zoom magnification of the actual optical zoom magnification Change towards If the optical zoom magnification is changed so that the actual optical zoom magnification matches the target zoom magnification, the size of the main subject (image size of the subject area of the main subject) on the input image obtained after the change is Ideally SIZE _A. The change of the optical zoom magnification is realized by changing the position of the zoom lens 30 in FIG. 2, and the change of the optical zoom magnification changes the shooting angle of view of the image pickup unit 11 which should be called the angle of view of the input image. Specifically, for example, if SIZE _A 'expressed as the area of the subject area of the main subject is 1/4 of SIZE _A , the optical zoom magnification is doubled by the automatic adjustment of the angle of view (the optical zoom magnification is 2). This is because the area of the subject area of the main subject will be quadrupled if doubled). An image 370 in FIG. 12 represents an input image obtained through the automatic adjustment of the angle of view to increase the optical zoom magnification by 1.5 in comparison with that before the automatic adjustment of the angle of view.

However, the automatic adjustment of the angle of view is performed on the assumption that the position of the main subject does not protrude from the block indicated in the position information of the readout feature information. The automatic adjustment of the angle of view is forcibly terminated immediately before the protrusion occurs. Strictly speaking, the position of the main subject is, for example, the center position of the subject area of the main subject. For example, the position of the main subject (the center position of the subject area 361) on the evaluation image 350 belongs to the block BL ₅ , the optical zoom magnification at the time of shooting the evaluation image 350 is ZF ₁ , and , SIZE _A > SIZE _A ′, a target zoom magnification ZF ₂ larger than ZF ₁ is set, and the optical zoom magnification increases toward the target zoom magnification ZF ₂ . when the position of the main subject on the latest input image is about to protrude from the block BL _5, to fix the optical zoom magnification to cancel an increase in optical zoom magnification immediately before the squeeze out occurs. In this case, the optical zoom magnification after the automatic adjustment of the angle of view in step S25 is larger than ZF ₁ but smaller than ZF ₂ .

  After the automatic adjustment of the angle of view in step S25, in step S26, the imaging control unit 55 performs automatic focus adjustment so that the main subject is in focus. Automatic focus adjustment can be realized by autofocus control based on the contrast detection method using the AF evaluation value described above. That is, the main subject can be focused by adjusting the position of the focus lens 31 in FIG. 2 so that the AF evaluation value of the subject area of the main subject is maximized.

  By continuously performing subject detection for each input image acquired during automatic focus adjustment, the subject area of the main subject for which the AF evaluation value should be maximized is sequentially detected from each of the input images, Automatic focus adjustment may be performed using the result of the sequential detection. Alternatively, the position of the main subject on the current input image is estimated from the optical zoom magnification at the time of shooting the evaluation image and the current optical zoom magnification and the position of the main subject on the evaluation image, and the result of the estimation is obtained. It may be used to perform automatic focus adjustment.

  After the automatic focus adjustment in step S26, in step S27, the process waits for the shutter button 26b to be fully pressed. When it is confirmed that the shutter button 26b is fully pressed, the process proceeds from step S27 to step S28. A new target input image is taken. The target input image obtained in step S28 is recorded in the external memory 18.

  In the above description, it is assumed that the zoom magnification for determining the angle of view of the input image is the optical zoom magnification by the optical zoom. However, the zoom magnification for determining the angle of view of the input image may be an electronic zoom magnification by electronic zoom, and automatic adjustment of the angle of view may be realized by electronic zoom. In the same way, automatic adjustment of the angle of view may be realized by a combination of optical zoom and electronic zoom.

  The feature information recorded in the learning memory 54 represents the feature of the target input image taken in the past. Here, the feature of the target input image includes a first feature that indicates which category of subjects from among a plurality of subjects on the target input image is a focused subject (a focused subject), target input Second and third features representing the size and position of the focused subject on the image are included. The first to third features are indicated by focus information, size information, and position information, respectively.

  By recording the focus information in the learning memory 54, it becomes possible to reproduce the user's preferences regarding the focus, and by recording the size information and the position information in the learning memory 54, the user regarding the size and composition of the focused subject. Can be reproduced. In other words, in the control stage operation, feature information corresponding to the current subject category is read from the learning memory 54, and zoom control (automatic view angle adjustment) and focus control (automatic focus adjustment) are performed. Automatically reproduces the appropriate field angle setting, composition setting, and focus setting. Thereby, the angle of view setting, the composition setting and the focus setting adapted to the user's preference are supported, and the convenience of the user is improved.

  In the following second to seventh embodiments, modifications to the first embodiment that can be applied to the first embodiment are shown. However, as long as there is no contradiction, the techniques shown in the second to seventh embodiments can be applied to other embodiments other than the first embodiment.

<< Second Example >>
A second embodiment will be described. In the first embodiment, it is assumed that the required number of learning times L _NUM is 3. However, the required number of learning times L _NUM may be 1, and when L _NUM = 1, in the learning stage operation, FIG. when the target input image 300 is acquired element characteristic information 305 (see FIGS. 7 and 9) is generated in (a), the element characteristic information 305 is to function as a comprehensive feature information W1 _a. However, in order to appropriately reproduce the user's preference in the control stage operation, it is desirable to set _LNUM to 2 or more.

<< Third Example >>
A third embodiment will be described. When a person is present in the input image, the person may be subjected to personal recognition processing and categorized according to the result of the personal recognition processing. In the personal recognition processing, for example, a face dictionary database corresponding to the image signals of a plurality of registered person's face images is prepared in the subject detection unit 51 in advance, and input is performed based on the face dictionary database and the image signal of the input image. It recognizes which registered person is the person existing on the image. If it is recognized that the person existing on the input image is the i-th registered person, the category of the person may be detected as the i-th registered person (i is an integer).

  When the person on the first input image is the first registered person and the person on the second input image is the second registered person, the category of the person on the first input image and the person on the second input image The categories are determined to be different from each other, and as a result, the category combination is determined to be different between the first and second input images. When i and j are different integers, it is assumed that the i-th registered person and the j-th registered person are different persons.

<< 4th Example >>
A fourth embodiment will be described. In the input image assumed in the first embodiment, there is only one person. However, when there are a plurality of persons, the plurality of persons may be grouped into one category.

  For example, consider a case where an image 400 shown in FIG. 13 is input to the subject detection unit 51 as an input image. The input image 400 includes image signals of subjects 401 and 402 that are two persons and a subject 403 that is a mountain, and the subject areas 411 to 413 of the subjects 401 to 403 are input by the subject detection unit 51. 400. The subject detection unit 51 collectively classifies the subjects 401 and 402 into one category “two people”, and the feature information generation unit 52 may regard the category combination for the input image 400 as “two people and a mountain”. it can.

Feature information 405 in FIG. 14 shows an example of feature information generated for the input image 400. Assume that the focused subjects in the input image 400 are the subjects 401 and 402. Then, the focus information in the element feature information 405 is two persons, and the size information in the element feature information 405 is an average of the sizes of the subjects 401 and 402 on the input image 400 (strictly, for example, an image of the subject region 411). SIZE _D (average of the size and the image size of the subject area 412), and the position information in the element feature information 405 is the positions of the subjects 401 and 402 on the input image 400. Assume that the center positions of the subject areas 411 and 412 on the input image 400 are located in blocks BL ₄ and BL ₅ on the input image 400, respectively. Then, as shown in FIG. 14, the position information in the element feature information 405 is BL ₄ and BL ₅ .

Now, for simplification of description, consider the case where the feature information 405 itself functions as general feature information for the category combination “two people and a mountain”. In this case, when an evaluation image having a category combination of “two people and a mountain” is acquired during the control stage operation, feature information 405 is read from the learning memory 54 as read feature information (FIG. 10). Step S23), two persons are set as main subjects based on the focus information of the feature information 405 (Step S24), and the average size of the two persons on the input image after the automatic adjustment of the angle of view (subjects of the two persons) The angle of view is automatically adjusted so that the average image size of the region is SIZE _D (step S25), and the focus is automatically adjusted so that the two people are in focus (step S26). Thereafter, when the shutter button 26b is fully pressed (Y in step S27), a new target input image is shot and recorded (step S28).

However, in the process of automatic angle of view adjustment, whether or not the condition that one person's position on the input image is located in the block BL ₄ and the other person's position is located in the block BL ₅ is confirmed. If it is determined that the condition does not hold, the change of the optical zoom magnification (or electronic zoom magnification) by the automatic adjustment of the angle of view is ended at that time as described in the first embodiment.

<< 5th Example >>
A fifth embodiment will be described. In the first embodiment (and the other embodiments described above), it is assumed that the number of categories forming the category combination is 2, but the number may be 3 or more. For example, when the number is 3, and a target input image including a person, a dog, and a mountain as subjects is acquired during the learning stage operation, the category combination of the target input image is “person, dog, and mountain”. Thus, the feature information of the target input image is stored in the learning memory 54 in association with the category combination “people, dogs and mountains”.

  After the comprehensive feature information about the category combination “people, dogs and mountains” is generated, when an input image including people, dogs and mountains as subjects is acquired during the control phase operation, the category combination “ The comprehensive feature information about “people, dogs and mountains” is read as read feature information, and the processes after step S24 in FIG. 10 are executed.

<< Sixth Example >>
A sixth embodiment will be described. When the number of categories forming the category combination is 2, during the learning stage operation, the comprehensive feature information 420 of the category combination “people and dogs”, the comprehensive feature information 421 of the category combination “dogs and mountains”, the category combination “ Consider a case where the comprehensive feature information 422 of “people and mountains” is generated and stored in the learning memory 54 (the comprehensive feature information 420 to 422 is not shown).

  In this case, when an evaluation image (not shown) including a person, a dog, and a mountain as subjects is acquired during the control stage operation, any of the comprehensive feature information 420 to 422 is obtained in step S23 of FIG. Read out. At this time, referring to the number of learnings stored in the additional information of the general feature information 420 to 422, the general feature information with the largest number of learnings may be read in step S23.

  When there are a plurality of pieces of the total feature information having the same number of times of learning in the total feature information 420 to 422, the total feature information learned at the latest learning time may be read in step S23. In order to realize this, a learning time may be given to each additional information in the learning memory 54. The learning time of certain feature information represents the shooting time of the target input image that is the source of the feature information. For example, when the number of learning times of the comprehensive feature information 420 to 422 is “3”, “4”, and “4”, respectively, the latest learning among the learning times of the four element feature information that is the basis of the comprehensive feature information 421. While extracting the time as the first learning time, the latest learning time is extracted as the second learning time among the learning times of the four element feature information that is the source of the comprehensive feature information 422. If the first learning time is later than the second learning time, the comprehensive feature information 421 is read in step S23. Conversely, if the second learning time is later than the first learning time, the comprehensive feature information 422 is read in step S23. Read it out.

<< Seventh Embodiment >>
A seventh embodiment will be described. In the first embodiment, by including focus information, size information, and position information in the feature information, automatic adjustment of the angle of view and automatic adjustment of the focus are realized in the control stage operation.

  However, by deleting the size information and the position information from the feature information, execution of the automatic adjustment of the angle of view may be omitted in the control stage operation (see FIG. 10). When the size information and the position information are deleted from the feature information, the angle of view automatic adjustment in step 25 is not executed in the control stage operation, and after the automatic focus adjustment based on the focus information (step S26), the target input image in step S28 Is filmed.

  Alternatively, only the position information may be deleted from the feature information. In the first embodiment, the angle of view is automatically adjusted under the assumption that the position of the main subject does not protrude from the block indicated by the position information of the readout feature information. Although the automatic adjustment of the angle of view is forcibly terminated immediately before such protrusion, such forcible termination does not occur when the position information is deleted from the feature information. That is, when the position information is deleted from the feature information, the size of the main subject on the target input image in step S28 is the size according to the size information of the read feature information regardless of the position of the main subject on the input image. The angle of view is automatically adjusted so that

<< Eighth Example >>
An eighth embodiment will be described. In the eighth embodiment, it is assumed that the generation condition of the target input image generated during the learning stage operation is designated by the user's manual operation. The manual operation is an operation on the operation unit 26 by the user. In the case where the display unit 27 has a touch panel function, a manual operation may be realized by a touch panel operation. In this case, the display unit 27 also functions as a second operation unit that accepts specification of a target input image generation condition.

  The generation conditions for the target input image include any condition that changes the image quality of the target input image (more broadly, changes the image signal of the target input image). Depending on the specified generation conditions, camera shake correction ON / OFF, AFE 12 gain, and image processing (sharpening processing and white balance adjustment processing) performed on the output signal of the AFE 12 in the process of generating the target input image The contents of are defined. More specifically, the target input image generation conditions include a camera shake correction ON / OFF condition that determines whether camera shake correction is performed when the target input image is generated, a sensitivity condition that determines the sensitivity of the target input image, and a target A sharpening condition that defines the sharpness of the input image, a white balance condition that defines the white balance state of the target input image, and the like are included.

When camera shake correction is OFF, if the housing of the imaging device 1 moves during the exposure period performed by the imaging device 33 to obtain an image signal of the target input image, the image on the imaging device 33 is displayed. The blur is mixed in the target input image. When the camera shake correction is ON, a shake that may be mixed into the target input image is removed optically or electronically using a known method.
The sensitivity specified by the sensitivity condition is ISO sensitivity. The ISO sensitivity means sensitivity defined by ISO (International Organization for Standardization), and the brightness (luminance level) of the target input image can be adjusted by adjusting the ISO sensitivity. Actually, the amplification factor of the signal amplification in the AFE 12 is determined according to the ISO sensitivity.
A target input image can be generated by performing a sharpening process on the image represented by the output signal itself of the AFE 12. The sharpening conditions define the contents of the sharpening process, including whether or not to execute the sharpening process itself.
Whether or not to generate a target input image using auto white balance control is defined by the white balance condition.

  In the following description of the eighth embodiment, it is assumed that only the camera shake correction ON / OFF condition and the sensitivity condition are designated by a manual operation for the sake of simplicity.

  FIG. 15 is a block diagram of a portion particularly related to the operation in the special imaging mode according to the eighth embodiment. The generation condition information generation unit 56 generates generation condition information described later. The generation condition information generation unit 56 can be realized by the image signal processing unit 13 or a combination of the image signal processing unit 13 and the CPU 23.

[Learning stage operation]
FIG. 16 is a flowchart showing the procedure of the learning stage operation according to the eighth embodiment. In the learning stage operation, each process of steps S51 to S54 is executed. In step S51, prior to the shutter operation, the imaging apparatus 1 accepts specification of the target input image generation condition by the manual operation. After that, when the shutter operation is performed by the user in step S52, the specified generation condition is satisfied. To generate a target input image. After the generation of the target input image, in step S53, the subject detection unit 51 performs subject detection of the target input image and subject category classification. Finally, in step S54, the generation condition information indicating the generation condition of the target input image is reflected in the recorded content of the learning memory 54 in a state associated with the category combination of the target input image.

  For example, in step S51, it is specified by the user's manual operation that the ISO sensitivity should be set to “ISO 100” and camera shake correction should be turned on, and the target input image 300 in FIG. Consider the generated case. In this case, generation condition information 505 shown in FIG. 17 is generated based on the contents of the manual operation on the target input image 300, and learning is performed in a state where the generation condition information 505 is associated with the combination category “person and mountain” of the target input image 300. It is stored in the memory 54. The combination category detection method is as described in the first embodiment.

  The generation condition information for a certain input image is composed of sensitivity information indicating sensitivity conditions and camera shake correction information indicating camera shake correction ON / OFF conditions. Additional information as described above is added to the generation condition information as necessary. (See FIG. 17). Although it is possible to consider that the additional information is also a part of the generation condition information, in this example, it is considered that the additional information is not a component of the generation condition information. In the generation condition information 505, the sensitivity information is “ISO100”, and the camera shake correction information is “ON”.

In the learning stage operation, the user's preference is repeatedly learned by repeatedly executing a series of processes consisting of steps S51 to S54. L _NUM can be any integer greater than or equal to 1, but now _assume that L _NUM is 3. The significance of _LNUM is as described above. In FIG. 18, after the target input image 300 is acquired, the above-described target input images 300 a, 300 b, 330, 330 a, 330 b, 331, and 331 a (all not shown) that are the generation sources of the information in FIG. The contents recorded later in the learning memory 54 are shown.

The generation condition information generation unit 56 is a method similar to the method for generating the generation condition information 505 of the target input image 300, and the generation condition information of the target input image 300a based on the contents of the manual operation on the target input images 300a and 300b. The generation condition information 505b of 505a and the target input image 300b is generated. When the generation condition information 505, 505a and 505b are generated, the memory control unit 53 stores them in the learning memory 54 while associating them with the category combination “person and mountain”. On the other hand, generation condition information generating unit 56 or the memory control unit 53, a generation condition information W2 _A matching or similar to the generation condition information 505,505a and 505b created based on the statistics, generates condition information W2 _A also Category It is stored in the learning memory 54 in association with the combination “person and mountain”. The generation condition information for each target input image is also called element generation condition information, and the generation condition information generated based on statistics from the generation condition information for a plurality of target input images is also called total generation condition information. . In this example, each of the generated condition information 505,505a and 505b people are elements generating condition information, generates condition information W2 _A is a total generation condition information.

In the element generation condition information 505, 505a, and 505b, the sensitivity information is “ISO100”, “ISO100”, and “ISO200”, respectively, and the camera shake correction information is “ON”, “OFF”, and “ON”, respectively. is there. Sensitivity information of information 505, 505a, and 505b by a method similar to the method of generating the focus information of the comprehensive feature information (the method described in the first embodiment) from the focus information of the plurality of element feature information (see FIG. 9). Sensitivity information of the information W2 _A can be generated from the image information, and camera shake correction information of the information W2 _A can be generated from the camera shake correction information of the information 505, 505a, and 505b. That is, for example, of the sensitivity information of the underlying element generation condition information of the information W2 _A, the most frequent sensitivity information may be the sensitivity information of the information W2 _A (image stabilization information as well). Therefore, the sensitivity information and image stabilization information in the information W2 _A, respectively, the "ISO100" and "ON". In the additional information accompanying the information W2 _A , the number of learning times for the category combination “person and mountain” is recorded. Information W2 _A is based on three times the learning result is generated based on the words information 505,505a and 505b. Therefore, the additional information of the information W2 _A, 3 is recorded as a numerical value representing the number of times of learning.

18, indicated generation condition information group in the central portion is a generation condition information group for the category combinations "dog and sea", W2 _B is a comprehensive generation condition information about the category combinations "dog and sea" . The total generation condition information W2 _{B is} also created in the same manner as the total generation condition information W2 _A. In FIG. 18, the generation condition information group shown in the lower part is a generation condition information group for the category combination “person and sea”. In the state shown in FIG. 18, for the category combination “person and mountain” and “dog and sea”, the number of learning times has reached the required number of learning times L _NUM (= 3), so that the control stage operation can be executed.

As described in the first embodiment, after the target input images 300, 300a, and 300b are captured, the target input image 300c (not shown) whose category combination is “person and mountain” is further captured. May update the comprehensive generation condition information W2 _A using the generation condition information of the latest target input image 300c.

[Control stage operation]
For category combinations "People and Mountain", General generation condition information W2 _A of Figure 18 based on the element generating condition information 505,505a and 505b are stored in the learning memory 54, and several non-comprehensive generation condition information W2 _A a state in which total generation condition information (including information W2 _B) is stored in the learning memory 54, for convenience, referred to as the learning state of FIG. In the eighth embodiment, the control stage operation under the learning state of FIG. 18 will be described below. FIG. 19 is a flowchart showing the procedure of the control stage operation according to the eighth embodiment. In the control stage operation, steps S61 to S65 are executed.

  The processing contents of steps S61 and S62 are the same as those of steps S21 and S22 of FIG. Accordingly, when the shutter button 26b is half-pressed, or when it is determined that the housing of the imaging device 1 is stationary for a certain period of time, the process proceeds from step S61 to step S62. The subject detection unit 51 treats the latest input image as an evaluation image, and classifies and detects each subject existing on the evaluation image based on the image signal of the evaluation image. . Thereby, the category combination about the image for evaluation is determined by the same method as described above.

In step S <b> 63 following step S <b> 62, the imaging control unit 55 in FIG. 4 reads, from the learning memory 54, the comprehensive generation condition information of the category combination that matches the category combination of the evaluation image. For example, if the evaluation image combination category is “person and mountain”, the information W2 _A is read, and if the evaluation image combination category is “dog and sea”, the information W2 _B is read. (See FIG. 18). In the following, the case where the evaluation image is the image 350 in FIG. 11 is considered. Since the combination category of the evaluation image 350 is “person and mountain”, the comprehensive generation condition information W2 _A is read in step S63. Hereinafter, the comprehensive generation condition information read in step S63 is also referred to as read generation condition information.

After the reading process in step S63, in step S64, the process waits for the shutter button 26b to be fully pressed. When it is confirmed that the shutter button 26b is fully pressed, the process proceeds from step S64 to step S65. A new target input image is taken and the image signal of the target input image is recorded in the external memory 18. The generation condition of the target input image acquired in step S65 is in accordance with the read generation condition information. That is, if the read generation condition information is information W2 _A, ISO sensitivity of the target input image acquired in step S65 is the "ISO100", and, subject the input of step S65 in a state where the camera shake correction to ON An image is taken.

  The generation condition information recorded in the learning memory 54 reflects user preference. In the control stage operation, the generation condition information corresponding to the user's preference (ISO sensitivity, etc.) is automatically reproduced by reading out the generation condition information corresponding to the current subject category from the learning memory 54. Thereby, the setting of the generation conditions (ISO sensitivity, etc.) adapted to the user's preference is supported, and the convenience of the user is improved.

<< Ninth Embodiment >>
A ninth embodiment will be described. In each of the embodiments described above, category combinations are formed by associating a plurality of subjects. However, using scene determination, the subject on the input image and the scene determined for the input image are related. A category combination may be formed. This method will be specifically described.

  In the ninth embodiment, a scene determination unit 58 shown in FIG. 20 is used. The scene determination unit 58 can be provided in the image signal processing unit 13 of FIG. The scene determination unit 58 determines the shooting scene of the input image based on the image signal of the input image. This determination can be made for each input image. The determination of the shooting scene of the input image is performed using detection of the subject of the input image, category classification of the subject of the input image, analysis of the hue of the input image, estimation of the light source state of the subject at the time of shooting of the input image, etc. Any known method (for example, the method described in JP-A-2009-71666) can be used for the determination.

  A plurality of registered scenes are set in the scene determination unit 58 in advance. The registered scenes include, for example, a portrait scene that is a shooting scene in which a person is noted, a mountain scene that is a shooting scene in which a mountain is noted, a sea scene that is a shooting scene in which the sea is noted, and daytime shooting. A daytime scene representing a state, a night scene representing a night scene shooting state, and the like may be included. The scene determination unit 58 extracts feature data effective for scene determination from the image signal of the focused input image, thereby selecting the shooting scene of the focused input image from the plurality of registered scenes. The shooting scene of the focused input image is determined. The shooting scene determined by the scene determination unit 58 is called a determination scene.

  A method of applying the scene determination unit 58 to the first or eighth embodiment will be described. Determination scene information representing a determination scene is transmitted to the feature information generation unit 52 in FIG. 4 or the generation condition information generation unit 56 in FIG. The feature information generation unit 52 or the generation condition information generation unit 56 sets a category combination for the input image based on the detection result of the subject detection unit 51 for the arbitrary input image and the determination scene information. For example, regarding the target input image 300 in FIG. 6A, the subject 301 on the target input image 300 is detected by the subject detection unit 51, and the shooting scene of the target input image 300 is determined as a mountain scene by the scene determination unit 58. And the person who is the category of the subject 301 and the mountain scene are related, and the combination of “person” and “mountain scene” is set as the category combination of the target input image 300. A combination of “person” and “mountain scene” is described as a category combination “person and mountain scene”. Similarly, for example, when a subject of a dog category is detected from a certain target input image and it is determined that the determination scene of the target input image is a sea scene, the category combination of the target input image is the category combination “ Dog and sea scene ".

  The learning stage operation and the control stage operation when the scene determination unit 58 is applied to the first or eighth embodiment are different from the first or eighth embodiment except that the category combination setting method is different from the first or eighth embodiment. Similar to that of the example. That is, for example, the words “mountain” and “sea” in the description of the first or eighth embodiment are appropriately replaced with the words “mountain scene” and “sea scene”, respectively, and then in the first or eighth embodiment. The matters described can be applied to the ninth embodiment (the same applies to the second to seventh embodiments).

  When the first embodiment is applied, feature information of each target input image is generated by repeatedly capturing the target input image in the learning stage operation, and each category combination that is a combination of the category of the subject and the determination scene is set. The feature information is associated and stored in the learning memory 54. When the comprehensive feature information is generated in the learning memory 54, the process proceeds to the control stage operation. In the control stage operation (see FIG. 10), the category classification of the subject is performed on the evaluation image and the determination by the scene determination unit 58 is performed, whereby the category combination of the evaluation image (the category of the subject and the evaluation image) The combination of the image with the judgment scene) is determined, and the comprehensive feature information corresponding to the determined category combination is read from the learning memory 54. Then, based on the read comprehensive feature information, the view angle automatic adjustment and the focus automatic adjustment focusing on the main subject are performed according to the method described in the first embodiment, and then a new target input image may be taken.

  When the eighth embodiment is applied, in the learning stage operation, the target input image is repeatedly photographed through manual operation to generate the generation condition information of each target input image, and the combination of the subject category and the determination scene Each generation condition information is associated with the category combination and stored in the learning memory 54. When the comprehensive generation condition information is generated in the learning memory 54, the process proceeds to the control stage operation. In the control stage operation (see FIG. 19), the category classification of the subject is performed on the evaluation image and the determination by the scene determination unit 58 is performed to combine the categories of the evaluation image (the subject category and the evaluation image in the evaluation image). The combination of the image with the determination scene) is determined, and the comprehensive generation condition information corresponding to the determined category combination is read from the learning memory 54. Then, according to the method described in the eighth embodiment, a new target input image may be taken under the generation conditions specified in the read total generation condition information.

<< Tenth Embodiment >>
A tenth embodiment will be described. In the tenth embodiment, characteristic control is performed on the acoustic signal. The microphone unit 14 in FIG. 1 is formed from a plurality of microphones. Now, as shown in FIG. 21, it is assumed that the microphone unit 14 includes two microphones 14L and 14R. Although it is possible to employ directional microphones having directivity as the microphones 14L and 14R, it is assumed that the microphones 14L and 14R are omnidirectional microphones having no directivity.

  FIG. 22 is an external perspective view of the imaging apparatus 1. The microphones 14L and 14R are installed at different positions on the housing of the imaging device 1. As shown in FIG. 22, the microphone 14 </ b> L is installed on the left side on the casing of the imaging apparatus 1, and the microphone 14 </ b> R is installed on the right side on the casing of the imaging apparatus 1. As shown in FIG. 22, the direction in which a subject that can be photographed by the imaging unit 11 is defined as the front, and the opposite direction is defined as the rear. The front and rear are directions along the optical axis of the imaging unit 11. Further, right and left mean right and left when the front side is viewed from the rear side.

  Each of the microphones 14L and 14R converts the sound collected by itself into an analog acoustic signal and outputs it. The acoustic signal processing unit 15 in FIG. 1 converts analog acoustic signals output from the microphones 14L and 14R into digital acoustic signals. The digital acoustic signals obtained by this conversion and based on the output signals of the microphones 14L and 14R are referred to as a left original signal and a right original signal, respectively. The acoustic signal processing unit 15 can generate an acoustic signal having directivity by performing known directivity control on the left original signal and the right original signal.

  Moreover, the imaging device 1 has a function of generating a still image with sound. That is, when a target input image is captured according to the shutter operation, an acoustic signal for a certain period based on the capturing time of the target input image is generated as the target acoustic signal, and the target acoustic signal corresponds to the image signal of the target input image. In addition, it can be recorded together with the image signal in the external memory 18. The generation of the target acoustic signal is performed by a target acoustic signal generation unit (not shown) included in the acoustic signal processing unit 15 of FIG. When the reproduction of the target input image is instructed in the reproduction mode, the target input image is displayed and reproduced on the display unit 27 and the target acoustic signal is reproduced by the speaker 28 as sound.

  Furthermore, when generating the still image with sound, the imaging apparatus 1 can generate an acoustic signal having directivity in the direction of the enhancement target subject on the target input image as the target acoustic signal by directivity control. The user can designate which subject is to be an enhancement subject using the operation unit 26 or the like. Directivity control for generating an acoustic signal having directivity in the direction of the enhancement target subject on the target input image as the target acoustic signal is particularly referred to as specific direction enhancement control. The imaging device 1 learns by regarding the user-specified content as user preference information regarding sound, and can be used for subsequent imaging. Hereinafter, a detailed method of realizing such a method will be described. In the tenth embodiment, each part shown in FIG. 4 or FIG. 15 is used.

[Learning stage operation]
First, the learning stage operation will be described. In the learning stage operation, the target input image is taken according to the shutter operation. Consider a case where a target input image 600 shown in FIG. 23 is obtained. The target input image 600 is an input image that is captured with the first and second registered persons described in the third embodiment included in the subject, and the subjects 601 and 602 are the first images on the target input image 600, respectively. 1 and 2 registered persons. The subject detection unit 51 detects the subjects 601 and 602 from the target input image 600, detects that the subjects 601 and 602 are people, and sets different subject areas 611 and 612. The subject areas 611 and 612 are image areas where the image signals of the subjects 601 and 602 exist, respectively. Further, through the above-described personal recognition process based on the image signal of the target input image 600 (see the third embodiment), the subject detection unit 51 recognizes that the subjects 601 and 602 are the first and second registered persons, respectively.

In the subject detection unit 51, the first and second registered persons are regarded as subjects of different categories. Now, it is assumed that the user uses the operation unit 26 or the like to specify that the first registered person should be the highlight target subject. This designation can be performed before the target input image 600 is captured, or can be performed after the target input image 600 is captured. By this designation, the subject 600 is set as the enhancement target subject. The acoustic signal processing unit 15 sets an extraction period P _{600 based} on the shooting time of the target input image 600, and arrives from the subject 601 from the left original signal and the right original signal in the extraction period P ₆₀₀ by specific direction enhancement control. An acoustic signal in which the sound component is emphasized is generated as the target acoustic signal SD ₆₀₀ . This target acoustic signal SD ₆₀₀ is recorded in the external memory 18 together with the image signal of the target input image 600 in association with the target input image 600.

Strictly speaking, the shooting time of an input image refers to, for example, the start time, intermediate time, or end time of exposure performed by the image sensor 33 in order to acquire an image signal of the input image. The extraction period based on a certain shooting time refers to a period starting from a time that is Δt _A seconds before the shooting time and ending at a time that is Δt _B seconds after the shooting time (Δt _A and Δt _B is a predetermined positive value). If the left original signal and the right original signal for a predetermined time are recorded in the internal memory 17, even if the subject to be emphasized is designated after the target input image 600 is photographed, the recorded signal in the internal memory 17 is recorded. Can generate the target acoustic signal SD ₆₀₀ (the same applies to other target acoustic signals).

The acoustic signal processing unit 15 determines the direction of the subject 601 viewed from the imaging device 1 from the position of the subject region 611 on the target input image 600 and the focal length at the shooting time of the target input image 600 (focal length of the imaging unit 11). And the target acoustic signal SD ₆₀₀ is generated so that the signal component of the sound coming from the estimated direction (that is, the signal component of the sound coming from the subject 601 as a sound source) is emphasized. A signal component of sound arriving from the estimated direction and extracted as required component from the left original signal and the right original signals during the extraction period P _600, the extracted necessary ingredient itself can be the target sound signal SD ₆₀₀ a. Alternatively, while extracting from the left original signal and the right original signals during the extraction period P ₆₀₀ the necessary components, as unwanted components from the left original signal and the right original signals during the extraction period P ₆₀₀ signal components other than the necessary ingredients After the extraction, the target acoustic signal SD ₆₀₀ may be generated by weighted addition of the necessary component and the unnecessary component so that the mixing ratio of the necessary component becomes relatively large. That is, coefficients k _A and k _B satisfying 0 <k _B <k _A are set, and a signal obtained by multiplying the necessary component by the coefficient k _A and a signal obtained by multiplying the unnecessary component by the coefficient k _B are added. signal may be generated as a target sound signal SD ₆₀₀ a.

Sound signal processing section 15, 'instead of, target sound signal SD ₆₀₀ from the left original signal and the right original signals during the extraction period P ₆₀₀ by Sureteo reduction control is a kind of directivity control' target sound signal SD ₆₀₀ produces a You can also The target acoustic signal by the stereo control is a stereo signal composed of an L signal and an R signal generated regardless of the position of the enhancement target subject. The L signal and the R signal are acoustic signals having directivity and having directivity axes in different directions. Signal SD ₆₀₀ 'when is generated, signal signal SD ₆₀₀ in place of SD _600' are recorded in association with the target input image 600 with the image signal of the target input image 600 in the external memory 18.

  The category combination “first and second registered persons” is set as the category combination of the target input image 600 by the subject detection unit 51, and the sound control information regarding the directivity of the target acoustic signal of the target input image 600 is set by the memory control unit 53. It is recorded in the learning memory 54 in a state associated with the category combination “first and second registered persons”. For example, the sound control information 605 shown in FIG. 24 is recorded in the learning memory 54 in a state associated with the category combination “first and second registered persons” of the target input image 600. The sound control information is generated, for example, by a sound control information generation unit (not shown) included in the CPU 23 of FIG.

  The sound control information for a certain input image is composed of control ON / OFF information indicating ON / OFF of specific direction emphasis control and emphasis target information indicating an emphasis target subject, and the sound control information is as described above as necessary. Additional information is added (see FIG. 24). Although it is possible to consider that the additional information is also a part of the sound control information, in this example, it is considered that the additional information is not a component of the sound control information.

Control ON / OFF information is set to ON or OFF. When the control ON / OFF information is ON, it means that the target acoustic signal is generated using the specific direction enhancement control for the corresponding target input image, and the control ON / OFF information is OFF. Means that the target acoustic signal is generated using the stereo control for the corresponding target input image.
The emphasis target information indicates which category of subject on the target input image is the emphasis target subject, and has meaningful data only when the control ON / OFF information is ON.
It is assumed that the target acoustic signal SD ₆₀₀ is generated for the target input image 600. Then, in the sound control information 605, the control ON / OFF information is “ON”, and the emphasis target information is “first registered person”.

In the learning stage operation, the user's preference is repeatedly learned by repeatedly performing capturing of the target input image. L _NUM can be any integer greater than or equal to 1, but now _assume that L _NUM is 3. The significance of _LNUM is as described above. FIG. 25 shows the recorded contents of the learning memory 54 after the target input image 600 is acquired and the target input images 600a and 600b (all not shown) are further acquired. Each of the target input images 600a and 600b includes first and second registered persons as subjects, and each combination category of the target input images 600a and 600b is “first and second registered persons”. To do. Further, it is assumed that the target acoustic signal is generated for each of the target input images 600a and 600b as in the case of the target input image 600.

The CPU 23 generates sound control information 605a of the target input image 600a and sound control information 605b of the target input image 600b by a method similar to the method of generating the sound control information 605 of the target input image 600. When the sound control information 605, 605a, and 605b is generated, the memory control unit 53 stores them in the learning memory 54 while associating them with the category combination “first and second registered persons”. On the other hand, the CPU 23 or the memory control unit 53 creates sound control information W3 _A that matches or is similar to the sound control information 605, 605a, and 605b based on statistics, and the sound control information W3 _A is also classified into the category combination “first and The information is stored in the learning memory 54 in association with the “second registered person”. The sound control information for each target input image is also called elemental sound control information, and the sound control information generated based on statistics from the sound control information for a plurality of target input images is also called comprehensive sound control information. . In this example, a sound control information 605,605a and 605b respectively are elements sound control information, the sound control information W3 _A is a total sound control information.

In the element sound control information 605, 605a, and 605b, the control ON / OFF information is “ON”, “ON”, and “OFF”, respectively, and the emphasis target information in the element sound control information 605 and 605a is “first”. "Registered person". Control ON of information 605, 605a, and 605b is performed in the same manner as the method of generating the focus information of the comprehensive feature information (the method described in the first embodiment) from the focus information of the plurality of element feature information (see FIG. 9). The control ON / OFF information of the information W3 _A can be generated from the / OFF information, and the emphasis target information of the information W3 _A can be generated from the emphasis target information of the information 605, 605a, and 605b. That is, for example, of the emphasis target information elements sound control information as a source of information W3 _A, similar to the most frequent more emphasis target information may be the highlight object information of the information W3 _A (also controls ON / OFF information ). Therefore, the control ON / OFF information and the emphasis target information in the information W3 _A are “ON” and “first registered person”, respectively. In the additional information accompanying the information W3 _A , the number of learning times for the category combination “first and second registered persons” is recorded. The information W3 _A is generated based on the learning results for three times, that is, based on the information 605, 605a, and 605b. Therefore, 3 is recorded in the additional information of the information W3 _A as a numerical value indicating the number of learnings.

As described in the first embodiment, after the target input images 600, 600a, and 600b are captured, the target input image 600c (not shown) whose category combination is “first and second registered person” is the target. When generated together with the sound signal, the comprehensive sound control information W3 _A may be updated using the sound control information of the latest target input image 600c.

[Control stage operation]
For the category combination “first and second registered persons”, the integrated sound control information W3 _{A of} FIG. 25 based on the element sound control information 605, 605a, and 605b is stored in the learning memory 54, and other than the integrated sound control information W3 _A Assuming a state in which some integrated sound control information is stored in the learning memory 54, the control stage operation under this assumed state will be described. FIG. 26 is a flowchart showing the procedure of the control stage operation according to the tenth embodiment. In the control stage operation, steps S81 to S84 are executed.

  When the shutter operation is performed in the control stage operation, in step S81, a new target input image is captured, while the subject detection unit 51 performs detection processing on the evaluation image. The evaluation image here is usually the target input image photographed in step S81. However, the input image obtained during the control stage operation and captured before capturing the target input image in step S81 (for example, the input image captured immediately before capturing the target input image in step S81). May be used as an evaluation image. The subject detection unit 51 detects and classifies each subject existing on the evaluation image into one of a plurality of categories based on the image signal of the evaluation image. Thereby, the category combination about the image for evaluation is determined by the same method as described above.

In step S <b> 82 following step S <b> 81, the acoustic signal processing unit 15 reads out the comprehensive sound control information of the category combination that matches the category combination of the evaluation image from the learning memory 54. For example, if the combination category of the evaluation image is “first and second registered person”, the information W3 _A is read, and the combination category of the evaluation image is “third and fourth registered person”. Then, the integrated sound control information corresponding to the combination category “third and fourth registered persons” is read out. Hereinafter, a case where the evaluation image is the image 650 in FIG. 27 will be considered. Furthermore, it is assumed that the evaluation image 650 is a target input image captured in step S81. On the image 650, there are image signals of the subject 651 as the first registered person and the subject 652 as the second registered person. Then, since the combination category of the evaluation image 650 is “first and second registered persons”, the integrated sound control information W3 _A is read in step S82. Hereinafter, the comprehensive sound control information read in step S82 is also referred to as read sound control information. The subject areas 661 and 662 on the image 650 in FIG. 27 are image areas where the image signals of the subjects 651 and 652 exist, respectively.

After the reading process in step S82, in step S83, the acoustic signal processing unit 15 sets an extraction period _{P650 based} on the shooting time of the target input image shot in step S81, and the target input image in step S81. The target acoustic signal SD ₆₅₀ to be associated with is generated. The method of generating the target acoustic signal SD ₆₅₀ is as described above, but the target acoustic signal SD ₆₅₀ is generated according to the conditions defined in the read sound control information.

Now, since it reads sound control information is information W3 _A is assumed (see Fig. 25), the left original signal during the extraction period P ₆₅₀ on the emphasis target object is considered to be the first registered person and The target acoustic signal SD ₆₅₀ is generated by performing the above-described specific direction emphasis control on the right original signal. When the target input image and the evaluation image 650 are the same, the position of the subject region 661 on the target input image that coincides with the evaluation image 650 and the focal length at the shooting time of the target input image in step S81 (of the imaging unit 11). The direction of the subject 651 viewed from the imaging device 1 is estimated from the focal length), and the signal component of the sound coming from the estimated direction (that is, the signal component of the sound coming from the subject 651 as a sound source) is emphasized. A target acoustic signal SD ₆₅₀ is generated. Although the assumption is different from the above assumption, if the control ON / OFF information in the readout sound control information is “OFF”, the target sound is detected from the left original signal and the right original signal in the extraction period P ₆₅₀ by the threshold control. Signal SD ₆₅₀ is generated.

  Thereafter, in step S84, the image signal of the target input image taken in step S81 and the target acoustic signal generated in step S83 are associated with each other and recorded in the external memory 18.

  The sound control information recorded in the learning memory 54 reflects the user's preference for sound. In the control stage operation, the sound control information corresponding to the current subject category is read from the learning memory 54 to automatically reproduce the sound characteristics suitable for the user's preference. Thereby, the user's convenience is improved.

  In the above-described specific operation example, the use of the personal recognition process is assumed, but the same process can be performed even when the personal recognition process is not used. For example, if a person is designated as an object to be emphasized when the category combination of the target input image obtained during the learning stage operation is “person and dog”, control ON / OFF information of “ON” and “person” are designated. The sound control information including the emphasis target information is created, and the sound control information is stored in the learning memory 54 in a state associated with the category combination “person and dog”. After comprehensive sound control information for the category combination “person and dog” is generated on the learning memory 54 by repeating the shooting of the target input image, the category combination becomes “person and dog” in step S81 of the control stage operation. When the target input image (or the image for evaluation) is photographed, comprehensive sound control information corresponding to the category combination “person and dog” is read as read sound control information. If the control ON / OFF information and the emphasis target information in the read sound control information are “ON” and “person”, respectively, the person on the target input image in step S81 is regarded as the emphasis target subject and specified. What is necessary is just to produce | generate the target acoustic signal of step S83 using direction emphasis control.

  The above-described method can also be applied to moving images. By performing subject detection and category classification on the target moving image based on the image signal of the target moving image, a category combination of the target moving image can be set as in the case of the target input image as a still image. After the comprehensive sound control information for the category combination “person and dog” is generated on the learning memory 54 by repeating the shooting of the target moving image, the target moving image in which the category combination becomes “person and dog” in the control stage operation. Is taken, comprehensive sound control information corresponding to the category combination “person and dog” is read out as read sound control information. If the control ON / OFF information and the emphasis target information in the read sound control information are “ON” and “person”, respectively, the person on the target moving image captured during the control stage operation is set as the emphasis target subject. What is necessary is just to produce | generate a target acoustic signal using specific direction emphasis control after considering. In this case, a target acoustic signal that tracks the sound from the enhancement target subject is generated and associated with the target moving image.

  The method described in the tenth embodiment can be implemented in combination with the methods described in other embodiments.

<< Eleventh embodiment >>
An eleventh embodiment will be described. In the learning phase operation, the remaining number of learnings necessary for shifting from the learning phase operation to the control phase operation may be presented to the user.

For example, if the can transition category combinations to control stage operation from the learning stage operation when photographing _once more m the target input image is a "person and mountains" in the first embodiment, the indicating _once m 1 The index may be displayed on the display unit 27 (m ₁ is a natural number). When L _NUM = 3 and only the feature information 305 and 305a of FIG. 9 is recorded in the learning memory 54, m ₁ = 1.
Similarly, for example, if the target input image whose category combination is “person and mountain” is photographed m _{2 more} times in the eighth embodiment, the learning phase operation can be shifted to the control phase operation, indicating m ₂ times. The second index may be displayed on the display unit 27 (m ₂ is a natural number). When L _NUM = 3 and only the generation condition information 505 and 505a of FIG. 18 is recorded in the learning memory 54, m ₂ = 1.
Similarly, for example, in the tenth embodiment, if the target input image whose category combination is “first and second registered persons” is photographed m _{3 more} times, the learning stage operation can be shifted to the control stage operation. _{A third} index indicating ₃ times may be displayed on the display unit 27 (m ₃ is a natural number). When L _NUM = 3 and only the sound control information 605 and 605a in FIG. 25 is recorded in the learning memory 54, m ₃ = 1.
The values of m _{1 to} m ₃ can be easily determined by referring to the recorded contents of the learning memory 54.

  The first index may be composed of characters as shown in FIG. 28 (a), may be composed of figures as shown in FIG. 28 (b), or may be composed of characters and figures. You may comprise by the combination of these. The same applies to the second and third indices. FIGS. 28A and 28B are diagrams illustrating display screen examples in a state where the first index is displayed. Unless otherwise specified, the display screen refers to the display screen of the display unit 27. Further, when the imaging apparatus 1 is formed so as to be able to realize the operations of the above-described plurality of embodiments (for example, the first, eighth, and tenth embodiments), the first to third indices These two or more indices may be displayed simultaneously.

<< Twelfth embodiment >>
A twelfth embodiment will be described. In the twelfth embodiment, a technique applicable to the eighth embodiment regarding manual operation will be described.

  For convenience of explanation, in the twelfth embodiment, it is assumed that each generation condition information includes sharpening information representing a sharpening condition in addition to sensitivity information and camera shake correction information (the significance of the sharpening condition is , Described in the eighth embodiment).

  The user can change all or a part of the sensitivity condition, the sharpening condition, and the camera shake correction ON / OFF condition of each target input image from the initial conditions by performing a manual operation in the learning stage operation. When those changes are made, the memory control unit 53 in FIG. 15 records what conditions are changed and the number of changes for each condition in the additional information of the learning memory 54. Based on the additional information, the CPU 23 sets a condition that the user changes relatively frequently as the specific condition. Then, after the specific condition is set, a user interface that makes it easy to change the specific condition is realized.

  A specific example will be described. Immediately after the start of the learning stage operation, when an input image with the category combination “person and mountain” is photographed, the input image is simply displayed on the display unit 27 as shown in FIG. In this state, if the user desires to change the sensitivity condition, after performing the first operation for displaying the operation menu that can change the sensitivity condition, the sharpening condition, and the image stabilization ON / OFF condition, the sensitivity is changed. It is necessary to sequentially perform a second operation for selecting a sensitivity condition from among a condition, a sharpening condition, and a camera shake correction ON / OFF condition, and a third operation for designating a specific value of ISO sensitivity. A series of operations including the first to third operations is collectively referred to as a basic operation.

  In the learning stage operation, n target input images having a category combination of “person and mountain” are captured, and the generation conditions of the target input images are changed at the time of capturing the n target input images. It is assumed that a manual operation is performed (n is an integer of 2 or more). However, this change is a change with respect to only the sensitivity condition, and it is assumed that the sharpening condition is constant among n target input images. In this case, the CPU 23 determines that the number of changes to the sensitivity condition (ie, n times) is greater than the number of changes (ie, 0) to the sharpening condition and the camera shake correction ON / OFF condition based on the additional information in the learning memory 54. As a result, the sensitivity condition is set to a specific condition. Then, after the setting, when an input image whose category combination is “person and mountain” is acquired in the control stage operation (or learning stage operation), the imaging apparatus 1 displays as shown in FIG. The operation icon 721 is displayed together with the input image on the unit 27 (for example, the operation icon 721 is superimposed on the input image). FIG. 29B is a display screen in a state where the operation icon 721 is displayed. The user can complete the designation of sensitivity information by performing only the third operation after selecting the operation icon 721 by an operation on the operation unit 26 or a touch panel operation. That is, the first operation by the user is not necessary, and the second operation is simpler than the basic operation. Although different from the above assumption, if the number of changes to the sharpening condition is relatively large, an icon (not shown) for changing the sharpening condition is also displayed alongside the operation icon 721.

  When the operation icon 721 is selected by the user, the CPU 23 extracts the ISO sensitivity value set relatively frequently by the user from each sensitivity information in the learning memory 54, and displays a display using the extraction result. 27. For example, when the ISO sensitivity at the time of shooting the n target input images is ISO 50 or ISO 100, when an operation for selecting the operation icon 721 is received, the display screen of FIG. 29B is displayed as shown in FIG. Change to the display screen. In the display screen of FIG. 29C, an icon 731 for “ISO 50” and an icon 732 for “ISO 100” are shown together with the current input image.

  In the state where the display of FIG. 29C is made, the user can set the current ISO sensitivity to “ISO50” simply by performing an operation of selecting the icon 731 or select the icon 732. The current ISO sensitivity can be set to “ISO100” simply by performing an operation. Since this setting content is immediately reflected in the generation conditions of the input image, the user can easily confirm the reflection result on the display screen. In the third operation in the basic operation, it is necessary to select a desired ISO sensitivity from among three or more (about 10) ISO sensitivities. However, the third operation is performed by displaying as shown in FIG. Is easier than basic operations.

Further, in the case of the display of FIG. 29C, the learning memory is used to arrange the icons 731 and 732 on the display screen so that the ISO sensitivity icons set relatively frequently are arranged above the display screen. It may be determined based on each sensitivity information in 54. For example, in the case where the ISO sensitivity for n ₁ target input images among the n target input images is “ISO 50” and the ISO sensitivity for n ₂ target input images is “ISO 100”. (N ₁ and n ₂ are integers and n ₁ + n ₂ = n), and if n ₁ > n ₂ , the icon 731 is displayed above the icon 732 on the display screen as shown in FIG. On the other hand, if n ₂ > n ₁ , the icon 732 may be displayed above the icon 731 on the display screen. As a result, the ISO sensitivity set more frequently can be selected with a smaller number of operations. Here, it is assumed that the icon arranged at the top on the display screen can be selected with a smaller number of operations than the icon arranged at the bottom.

<< Thirteenth embodiment >>
A thirteenth embodiment will be described. The above-described feature information, generation condition information, and sound control information (see FIGS. 9, 18, and 25) can be collectively referred to as learning information. In the method described above, learning information is generated for each category combination. Therefore, it takes a certain amount of time (corresponding to the number of times of photographing) to generate learning information for all category combinations, and the category that is classified and detected. As the number of types increases, the time also increases.

  In the thirteenth embodiment, a technique for reducing the time (number of times of photographing) necessary for generating learning information will be described. The technique described in the thirteenth embodiment can be applied to any of the embodiments described above. For simplification of description, it is assumed that the number of categories forming the category combination is 2, and a method of applying the technique of this embodiment to the feature information according to the first embodiment will be described.

  Assume that the comprehensive feature information 801 and 802 and the feature information 803 shown in FIGS. 30A to 30C are stored in the learning memory 54.

The comprehensive feature information 801 is comprehensive feature information for the category combination “car and mountain”, and the comprehensive feature information 802 is comprehensive feature information for the category combination “car and person”. In the general feature information 801, the focus information, the size information, and the position information are “car”, “SIZE ₁ ”, and “BL ₅ ”, respectively. In the general feature information 802, the focus information, the size information, and the position information are “Automobile”, “SIZE ₂ ” and “BL ₆ ”, respectively. Also, the comprehensive feature information 801 is created based on N ₁ element feature information for the category combination “car and mountain”, and the comprehensive feature information 802 is N ₂ for the category combination “car and person”. It is assumed that it has been created based on individual element feature information. That is, the learning times of the general feature information 801 and 802 are N ₁ and N ₂ respectively (N ₁ and N ₂ are natural numbers).

In the thirteenth embodiment, feature information for a single category is also generated by the feature information generation unit 52 (see FIG. 4) and stored in the learning memory 54. The learning information 803 in FIG. 30C is feature information about a single category. In the learning stage operation, the subject detection unit 51 and the feature information generation unit 52 detect only a car from a certain target input image, and detect that the focused subject of the target input image is a car. It is detected that the size of the automobile on the target input image (image size of the subject area of the automobile) is SIZE ₃ and that the position of the automobile on the target input image belongs to block BL _5. Shall be. Then, the feature information generation unit 52 generates feature information 803 in which the focus information, the size information, and the position information are “automobile”, “SIZE ₃ ”, and “BL ₅ ”, respectively, and the feature information 803 includes a single category “ It is associated with the “car” and stored in the learning memory 54.

  In a state where only the information 801 to 803 is stored in the learning memory 54, the feature information generation unit 52 artificially generates the feature information 804 for the category combination “person and mountain” based on the information 801 to 803. Can do. FIG. 31 shows the characteristic information 804 generated in a pseudo manner. This generation is performed as follows.

  The feature information generation unit 52 first determines the priority order of subjects to be focused based on the information 801 and 802, and generates focus information of the feature information 804 from the determination result. It is estimated from the information 801 that it is preferable to focus on the car rather than the mountain for the user who realized the generation of the information 801 to 803, and it is possible to focus on the car rather than the person. It is estimated from the information 802 that it is preferable. Therefore, among automobiles, people, and mountains, automobiles have the highest priority. On the other hand, superiority or inferiority between a person and a mountain cannot be estimated only by the information 801 and 802. However, when comparing a person and a subject other than the person, it is usually desirable to focus on the person. Considering this, the feature information generation unit 52 sets the focus information of the feature information 804 to “person”. Although different from the above assumption, if the focus information of the information 801 is “mountain”, the priority is “mountain> automobile> person” by referring also to the information 802. Therefore, the feature information 804 The focus information is “mountain”. Hereinafter, it is assumed that the focus information of the feature information 804 is “person”.

The size information SIZE ₁ in the information 801 can be considered to be the size of the automobile on the basis of the size of the mountain on the image. Similarly, the size information SIZE ₂ in the information 802 can be considered to be the size of the automobile on the basis of the size of the person on the image. Considering this, the size of the person can be estimated based on the size of the mountain from the ratio of SIZE ₁ and SIZE ₂ . At this time, however, the size information SIZE ₃ of the feature information 803 is used as a reference value for the size of the automobile. That is, the size information SIZE ₄ obtained according to “SIZE ₄ = (SIZE ₁ / SIZE ₂ ) × SIZE ₃ ” can be used as the size information of the feature information 804.

From the position information of the information 801 and 802, the preference of the placement position of the focused subject (user preference) can be estimated. If the position information is the same between the information 801 and 802, it is sufficient to include the same position information in the feature information 804. However, if the position information is different between the information 801 and 802, the position information with the larger number of learning times. Is adopted as the position information of the feature information 804. In this example, it is assumed that N ₁ > N ₂ . Then, the position information BL ₅ of the information 801 is substituted into the position information of the feature information 804 (assuming that N ₁ <N ₂ , the position information BL ₆ of the information 802 is substituted into the position information of the feature information 804. ).

  After the generation of the feature information 804, the feature information 804 can be regarded as comprehensive feature information and can be shifted from the learning stage operation to the control stage operation, and an input image whose category combination is “person and mountain” in the control stage operation If acquired as an image (FIG. 10), the feature information 804 can be read as feature information and read from the learning memory 54 to perform the operation described in the first embodiment.

Here, the number of learning times N ₄ of the artificially generated feature information 804 is set so as to satisfy 0 <N ₄ <1 (for example, N ₄ = 0.5) for convenience. By setting in this way, in the control stage operation, when a person, a mountain, and a car are included in the subject at the same time, the feature information 804 is not read as feature information by the method of the sixth embodiment ( In this case, since N ₁ > N ₂ > N ₄ , the feature information 801 is read as read feature information by the method of the sixth embodiment).

After the feature information 804 is generated, if a target input image whose category combination is “person and mountain” is actually captured, feature information based on the target input image is generated. At this time, the feature information 804 can be discarded and the feature information based on the target input image can be stored in the learning memory 54 as the feature information of the category combination “person and mountain”. In this case, the learning frequency N ₄ for the category combination “person and mountain” is changed to 1. Alternatively, feature information on the category combination “person and mountain” may be recreated from the feature information based on the target input image and the feature information 804 (in other words, feature information based on the target input image is used). The feature information 804 may be corrected. In this case, N ₄ = 1 or 1 <N ₄ <2.

  Although the method of generating feature information in a pseudo manner has been described, generation condition information and sound control information (see FIGS. 18 and 25) can also be generated in a similar manner.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values.

  The imaging apparatus 1 in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Image signal processing part 14 Microphone part 15 Acoustic signal processing part 30 Zoom lens 31 Focus lens 32 Aperture 33 Imaging element 51 Subject detection part 52 Feature information generation part 53 Memory control part 54 Learning memory 55 Shooting control part 56 Generation Condition Information Generation Unit 58 Scene Determination Unit

Claims (6)

In an imaging apparatus that has an image sensor that outputs a signal obtained by photoelectrically converting an optical image of a subject and generates a target image from an output signal of the image sensor that is obtained when a predetermined operation is performed.
By repeating the predetermined operation, a plurality of target images including the first and second target images are generated, and the second target image is generated after the first target image,
The imaging device
A subject detection unit that classifies and detects each subject present on the image based on the output signal of the image sensor into any of a plurality of categories;
A combination of detection categories of the subject detection unit for a plurality of subjects on the first target image is set as a specific combination, and learning information according to characteristics of the first target image or generation conditions of the first target image is used as the specific combination. A memory part to be stored in association with
An image based on an output signal of the image sensor after generation of the first target image and before generation of the second target image is used as an evaluation image, and a detection category of the subject detection unit for a plurality of subjects on the evaluation image An imaging apparatus comprising: an imaging control unit configured to generate the second target image using the learning information when the combination matches the specific combination. The learning information is information corresponding to the characteristics of the first target image,
The imaging apparatus according to claim 1, further comprising: focus information indicating which category of subjects in the plurality of subjects on the first target image is in focus. The imaging apparatus further includes an operation unit that receives specification of a generation condition of the first target image, and generates the first target image according to the generation condition of the first target image specified via the operation unit. ,
The imaging apparatus according to claim 1, wherein the learning information is information according to a generation condition of the first target image. When the combination of detection categories of the subject detection unit for the plurality of subjects on the evaluation image matches the specific combination,
Based on the learning information according to the characteristics of the first target image, focus control and zoom control for the second target image are performed so that the second target image has characteristics according to the characteristics of the first target image. Do or
The second target image is generated under a generation condition according to the generation condition of the first target image based on the learning information according to the generation condition of the first target image. Item 4. The imaging device according to any one of Items 3 to 3. In an imaging apparatus that has an image sensor that outputs a signal obtained by photoelectrically converting an optical image of a subject and generates a target image from an output signal of the image sensor that is obtained when a predetermined operation is performed.
By repeating the predetermined operation, a plurality of target images including the first and second target images are generated, and the second target image is generated after the first target image,
The imaging device
A subject detection unit for classifying and detecting a subject existing on an image based on an output signal of the image sensor into any of a plurality of categories;
A scene determination unit for determining by selecting a shooting scene of an image based on an output signal of the image sensor from a plurality of registered scenes;
The combination of the detection category of the subject detection unit for the subject on the first target image and the determination scene of the scene determination unit for the first target image is a specific combination, and the characteristics of the first target image or the first target A memory unit that stores learning information according to image generation conditions in association with the specific combination;
An image based on an output signal of the imaging element after the generation of the first target image and before the generation of the second target image is used as an evaluation image, and the detection category of the subject detection unit for the subject on the evaluation image and the A shooting control unit configured to generate the second target image using the learning information when a combination of the determination scene of the scene determination unit with respect to the evaluation image matches the specific combination; An imaging device. An image sensor that outputs a signal obtained by photoelectrically converting an optical image of a subject and a microphone unit that includes a plurality of microphones, and generates a target image from an output signal of the image sensor when a predetermined operation is performed. In the imaging device that generates a target acoustic signal based on the output acoustic signals of the plurality of microphones and associates the target acoustic signal with the target image,
By repeating the predetermined operation, a plurality of target images including the first and second target images are generated, and the second target image is generated after the first target image,
The imaging device
A subject detection unit that classifies and detects each subject present on the image based on the output signal of the image sensor into any of a plurality of categories;
A combination of detection categories of the subject detection unit for a plurality of subjects on the first target image is set as a specific combination, and learning information corresponding to the characteristics of the target acoustic signal associated with the first target image is set as the specific combination. A memory portion to store in association;
An image based on an output signal of the imaging element after the generation of the first target image and before the generation of the second target image or the second target image is used as an evaluation image, and the plurality of subjects on the evaluation image are selected. A target acoustic signal generation unit configured to generate a target acoustic signal to be associated with the second target image using the learning information when a combination of detection categories of the subject detection unit matches the specific combination. An imaging apparatus characterized by that.

Images