JP2008219450A - Imaging device and control method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To eliminate such a trouble that unnecessary photographing operation is carried out while securing the convenience of automatic photography with a voice as a trigger. <P>SOLUTION: The digital camera 1 is provided with a person extracting means 41, a voice analyzing means 43, and a composition deciding means 44. The person extracting means 41 analyzes image data to extract an image area representing a person. The voice analyzing means 43 analyzes an input voice to detect a predetermined features associated with the voice. The composition deciding means decides whether the composition of the image data are favorable based upon the extraction result of the person extracting means and the detection result of the voice analyzing means. The timing of recording the image data is determined based upon whether the composition is favorable, and a recording means is controlled so that the image data are recorded in the determined timing. Alternately, the determined timing is notified to a user. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus that performs automatic imaging using sound as a trigger, and a method for controlling the imaging apparatus.

Digital cameras are usually equipped with a microphone for recording audio during movie shooting. There is known a digital camera that identifies a specific voice phrase input from the microphone and automatically captures an image when the voice phrase is detected (for example, Patent Document 1).
JP 2006-184589 A

  A digital camera that performs automatic shooting using voice as a trigger is convenient, but may perform unnecessary operations in response to irrelevant voice. For example, in places with a lot of people such as sightseeing spots, the camera may react to the voice of an irrelevant person nearby. In addition, when a group photo is taken, the shooting may be performed because someone has made a voice such as “High, Cheese” even though it is not yet ready for shooting.

  An object of the present invention is to eliminate the inconvenience that an unnecessary shooting operation is performed while ensuring the convenience of automatic shooting using voice as a trigger.

  In order to achieve the above object, the present invention provides two types of imaging devices. Each of these imaging apparatuses includes an imaging unit that captures a scene and generates image data representing the scene, and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium. A person extraction unit, a voice analysis unit, and a composition determination unit described below are provided.

  The person extracting unit extracts an image area representing the person by analyzing the image data generated by the imaging unit. For example, a face included in image data is searched, and information indicating the number of faces detected by the search, the position of each face, and the size of each face is output as an extraction result. In this case, the facial expression detected by the search may be identified, and information indicating the identified facial expression may be further output. Moreover, it is good also as what identifies the gesture of the person contained in image data, and outputs the information which shows the identified gesture as an extraction result.

  The voice analysis means detects a predetermined feature related to the voice by analyzing the input voice. For example, as a voice feature, a predetermined volume change, a predetermined voice phrase, a feature registered in advance as a voice feature of a predetermined person, and the like are detected. The composition determination means determines the quality of the composition of the image data based on the extraction result of the person extraction means and the detection result of the voice analysis means.

  The first imaging device of the present invention determines the timing for recording image data based on the determination result of the composition determination means in addition to the imaging means, recording means, person extraction means and composition determination means, and at the determined timing. A recording control unit is provided for controlling the recording unit so that the image data is recorded. According to the first imaging apparatus of the present invention, even if a sound that can be a trigger for automatic shooting is emitted, automatic shooting is not performed unless the composition satisfies a predetermined condition. There is no worry about unnecessary shooting.

  Further, the second imaging apparatus of the present invention determines the timing for recording image data based on the determination result of the composition determination means instead of the recording control means of the first imaging apparatus, and the arrival of the determined timing is reached. Informing means for informing is provided. The second device does not perform automatic shooting, but when the composition satisfies a predetermined condition and a predetermined sound is produced, it is determined that the time has come for the photographer to press the shutter release button. Therefore, the user can enjoy the same convenience as automatic shooting. In addition, since the photographing operation is not automatically performed, the image capturing apparatus does not perform an unnecessary operation against the user's intention.

  The recording means preferably records the extraction result of the person extraction means and the detection result of the sound analysis means together with the image data on a recording medium. Thereby, when the image data recorded on the recording medium is edited with a personal computer or the like, editing using the extraction result can be performed.

  The first control method of the present invention is a method of operating as the first imaging device by controlling the imaging device according to the following procedure. First, an image region representing a person is extracted by analyzing the image data generated by the imaging means. In parallel with this, the input voice is analyzed to detect a predetermined feature related to the voice. Subsequently, the quality of the composition of the image data is determined based on the person extraction result and the sound detection result. Then, the timing for recording the image data is determined based on the determination result, and the recording means is controlled so that the image data is recorded at the determined timing.

  The second control method of the present invention is a method of operating as the second imaging device by controlling the imaging device according to the following procedure. First, an image region representing a person is extracted by analyzing the image data generated by the imaging means. In parallel with this, the input voice is analyzed to detect a predetermined feature related to the voice. Subsequently, the quality of the composition of the image data is determined based on the person extraction result and the sound detection result. And the arrival of the determined timing is alert | reported to a user by controlling operation | movement of a predetermined | prescribed output means.

  Hereinafter, a digital camera that performs control by selectively using a first control method and a second control method will be disclosed as an embodiment of the method and apparatus of the present invention. This digital camera has four operation modes: a normal shooting mode, an image reproduction mode, an automatic shooting mode, and a shooting assist mode. The digital camera when set to the automatic shooting mode corresponds to the first imaging device of the present invention, and the digital camera when set to the shooting assist mode corresponds to the second imaging device of the present invention. However, the two apparatuses and the control method of the present invention can be implemented independently, and are not limited to the embodiments described below.

  First, the configuration of the digital camera will be described. 1A and 1B are views showing the appearance of the digital camera 1. FIG. 1A shows the front of the digital camera, and FIG. 1B shows the back. As shown in these drawings, the digital camera 1 includes an imaging lens 2, a shutter release button 3, a microphone 4, an operation dial or buttons 5a to 5f, a monitor 6, and an LED lamp 9. Further, at the lower part of the digital camera 1, there is a slot cover (not shown) that can be opened and closed with a speaker 8, and a card slot into which a memory card 7 is loaded is provided in the slot cover.

  FIG. 2 is a diagram illustrating an internal configuration of the digital camera 1. As shown in this figure, the digital camera 1 includes an imaging unit including an imaging lens 2, a lens driving unit 16, a diaphragm 13, a diaphragm driving unit 17, a CCD 14, and a timing generator (TG) 18. The imaging lens 2 includes a plurality of functional lenses such as a focus lens for focusing on a subject and a zoom lens for realizing a zoom function. The lens driving unit 16 is a small motor such as a stepping motor, and adjusts the position of each functional lens so that the distance from the CCD 14 to each functional lens is a suitable distance for the purpose. The diaphragm 13 is composed of a plurality of diaphragm blades. The aperture drive unit 17 is a small motor such as a stepping motor, and adjusts the position of the aperture blades so that the aperture size of the aperture becomes a size suitable for the purpose. The CCD 14 is a CCD having 5 to 12 million pixels with a primary color filter, and discharges accumulated charges in response to an instruction signal from the timing generator 18. The timing generator 18 sends a signal to the CCD 14 so that charges are accumulated in the CCD 14 only for a desired time, thereby adjusting the shutter speed.

  The digital camera 1 also transfers an A / D conversion unit 15 that converts the output signal of the CCD 14 into a digital signal, and image data output from the A / D conversion unit 15 to another processing unit via the system bus 24. An image input control unit 23 and a memory 22 for temporarily storing image data transferred from the image input control unit 23 are provided.

  In addition, the digital camera 1 determines a focus adjustment unit 20 that instructs the lens driving unit 16 to move the lens and performs focusing, determines an aperture value and a shutter speed, and sends an instruction signal to the aperture driving unit 17 and the timing generator 18. An exposure adjusting unit 21 is provided. The digital camera 1 also includes an image processing unit 25 that performs image processing on the image data stored in the memory 22. The image processing unit 25 performs color tone correction and brightness correction to make an image have natural color and brightness, processing for correcting red eyes when the image data includes red eyes, and when the image composition is poor Various finishing processes are performed to improve the appearance of the image, such as a process of correcting the composition. The processed image data processed by the image processing unit 25 is stored in the memory 22 again.

  The digital camera 1 also includes a display control unit 26 that controls output of image data stored in the memory 22 to the monitor 6. The display control unit 26 thins out the number of pixels of the image data stored in the memory 22 so as to have a size suitable for display, and then outputs it to the liquid crystal monitor 9. The display control unit 26 also controls display of setting screens such as operating conditions.

  Further, the digital camera 1 includes a recording / reading control unit 27 that controls writing of image data stored in the memory 22 to the memory card 7 and loading of image data recorded on the memory card 7 to the memory 22. Prepare. The recording / reading control unit 27 records the image data obtained by photographing according to the user setting as it is or compression-encoded, and stores it in the memory card 7 as an Exif (Exchangeable Image File Format) file. Exif is a file format standard defined by the Japan Electronics Industry Promotion Association (JEIDA). When an operation for requesting reproduction of an image file recorded on the memory card 7 is performed, the image data included in the Exif file is loaded into the memory 22. If the image data is compressed, it is decompressed and loaded into the memory 22.

  The digital camera 1 also includes an LED control unit 19 that controls lighting and extinguishing of the LEDs 9. Furthermore, a microphone 4, a speaker 8, an A / D converter 10, a D / A converter 11, and a voice input / output controller 12 that controls voice input / output are provided. Audio data input from the microphone 4 and converted into digital data by the A / D converter 10 is transferred to the memory 22 via the system bus 24 by the audio input / output controller 12 and stored therein. Also, audio data supplied from each processing unit or the overall control unit described later to the audio input / output control unit 12 is converted by the D / A conversion unit 11 and then output to the speaker 8.

  The digital camera 1 also includes a timing detection unit 28 that detects the timing at which an image should be acquired. The timing detection unit 28 analyzes the image data and audio data stored in the memory 22 and outputs a signal indicating that the timing for acquiring an image has arrived when those data satisfy a predetermined condition. .

  In addition, the digital camera 1 includes a CPU (Central Processor Unit) 31, a RAM (Random Access Memory) 32 in which an operation / control program is stored, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) in which various setting values are stored. An overall control unit 30 is provided. The CPU 31 of the overall control unit 30 refers to the setting value stored in the EEPROM 33, and selects and executes a program stored in the RAM 32 based on the setting value. As a result, the overall control unit 30 detects the operation of the shutter release button 3 and the operation dial / buttons 5a to 5f, or receives the processing result of each processing unit, and the LED control unit 19, the focus adjustment unit 20, the exposure adjustment. An instruction signal for instructing a process to be executed is sent to the unit 21, the image input control unit 23, the image processing unit 25, the display control unit 26, the recording / reading control unit 27, the timing detection unit 28, and the voice input / output control unit 12. To do. Thereby, the operation of the digital camera 1 is controlled.

  In the normal shooting mode, the automatic shooting mode, and the shooting assist mode, each processing unit performs focus adjustment, exposure control, flash control, image processing, recording, and the like under the control of the overall control unit 30 to obtain an image. To be acquired. In the playback mode, an image recorded on the memory card 7 is output to the monitor 6 under the control of the overall control unit 30. In the setting mode, a setting screen is displayed on the monitor 6 under the control of the overall control unit 30, and operation inputs from the operation dial / buttons 5a to 5f are accepted. Information selected by the user using the operation dials / buttons 5 a to 5 f on the setting screen or information taken from the memory card 7 is stored in the EEPROM 33.

  Hereinafter, the automatic shooting mode and the shooting assist mode will be further described. FIG. 3 is a flowchart showing an outline of the operation of the digital camera 1 when the automatic shooting mode is set. When the digital camera 1 is set to the automatic shooting mode, the digital camera 1 starts generating image data representing a scene captured on the lens (S101). Then, the composition of the image represented by the generated image data is determined (S102), and if the composition is good, the image is recorded on the memory card 7 regardless of whether the shutter release button 3 is operated (S103). If the composition is bad, a better composition is proposed (S104), and the operation of the imaging unit is controlled or the image processing unit 25 is controlled so that the image data generated in step S101 becomes the composition proposed in step S104. A predetermined process is performed (S105). For example, when the main subject is too small, the imaging unit is caused to perform a zoom operation. When the main subject is biased, the image processing unit 25 is instructed to perform image processing for trimming and moving or enlarging only the region where the subject is shown. Alternatively, if a subject that should be standing vertically appears obliquely, a rotation process is performed so that the subject becomes vertical.

  The image data regenerated (S101) by the imaging unit or the image processing unit 25 is evaluated again in step S102. The above process is repeated until a mode switching operation is detected (S106).

  FIG. 4 is a flowchart showing an outline of the operation of the digital camera 1 when the photographing assist mode is set. When the digital camera 1 is set to the photographing assist mode, the digital camera 1 starts generating image data representing a scene captured on the lens (S201). Then, the composition of the image represented by the generated image data is determined (evaluated) (S202).

  If the composition is good, the arrival of timing is notified (S203). 5 and 6 show notification examples of timing arrival. FIG. 5 shows an example in which the arrival of timing is notified by displaying a mark 34 prompting the user to press the shutter release button on the monitor 6. Instead of the mark, a message such as “It is a photo opportunity” may be displayed. FIG. 6 shows an example in which the arrival of timing is notified by blinking the LED lamp 9. In addition, a method of notifying the arrival of timing by sound output from a speaker is also conceivable.

  If the composition is bad, a better composition is proposed (S204). Then, by displaying the proposed composition on the monitor 6 (hereinafter referred to as assist display), the photographer is prompted to correct the way the camera is held or to operate a predetermined operation button (S205). 7 and 8 show examples of assist display. FIG. 7 shows an example in which a preferred framing method is presented by displaying a framed frame 35 superimposed on image data that has been shot. FIG. 8 shows an example in which a preferred framing image generated by image processing is displayed, and a mark 36 is displayed that suggests how framing can be acquired at the edge of the screen to obtain image data such as the image being displayed. It is. In addition, a preferred framing may be presented by displaying a message such as “Please zoom in” or “Please turn the camera slightly to the left.” In the photographing assist mode, the above processing is repeated until a mode switching operation is detected (S206).

  Next, the process for determining the composition in steps S102 and S202 and the process for proposing the composition in steps S104 and S204 will be described in more detail. Composition determination and composition proposal are performed by the timing detection unit 28 in FIG. FIG. 9 is a diagram illustrating a configuration of the timing detection unit 28. As shown in the figure, the timing detection unit 28 includes a person extraction unit 41, a non-person extraction unit 42, a voice analysis unit 43, a composition determination unit 44, and a composition proposal unit 45. Note that the timing detection unit 28 may be a circuit constituted by an LSI functioning as the means 41 to 45, or a microcomputer incorporating a software program for executing the processing of the means 41 to 45.

  The person extracting means 41 reads the image data stored in the memory 22 and searches for a person area in the image data. In this embodiment, a person is detected by searching for a face. When the face is detected, the person extraction unit 41 adds an identifier such as a serial number, and then calculates the area of the face area, the area of the entire body including the face (hereinafter referred to as the whole body area), and the barycentric coordinates. . The center-of-gravity coordinates are obtained as the center-of-gravity coordinates of the face area when the area of the face exceeds a predetermined value, and the center-of-gravity coordinates of the whole body area when the area is less than the predetermined value. For example, when the face area is relatively large as illustrated in FIGS. 10A and 10B, the barycentric coordinates of each face area are calculated. When the face area is small as shown in FIGS. 10C and 10D, the barycentric coordinates of the whole body area indicated by the cross mark are calculated. When the search of all areas is completed, the person extracting means 41 uses the total number of detected persons, the range and area of the face area of each person, and the whole body as information indicating the number, position, and size of the detected persons. The range and area of the region and the barycentric coordinates are stored in a memory (not shown) provided in the timing detection unit 28.

  As a method for searching for and detecting a face, as introduced in Japanese Patent Application Laid-Open No. 2001-51338, a method for detecting a skin color area as a face, presence of parts having geometric features such as hair, eyes and mouth There are known a method for detecting a face by determining whether or not it is a face, and various other methods. Any known method can be used for the face detection processing executed by the person extracting means 41.

  The person extraction means 41 then identifies the detected facial expression. However, facial expression identification is performed only when the display identification function is set to ON in the detailed settings of the automatic shooting mode. Alternatively, facial expressions may be identified only when the size of the detected face is greater than or equal to a predetermined value. In the present embodiment, the person extraction unit 41 identifies four facial expressions, such as a laughing face, an angry face, a crying face, and a surprised face exemplified in FIGS. 11A, 11B, 11C, and 11D. As is clear from the example in the figure, each of these facial expressions is characterized by the degree of opening of the eyes and mouth and the degree of rise of the eyebrows and corners of the mouth, so that the facial expressions are identified based on the image characteristics of those parts. Can do. Various methods for identifying facial expressions are known, including the method introduced in Japanese Patent Laid-Open No. 2001-51338, and any known method is used for facial expression identification processing performed by the person extracting means 41. can do. The person extraction unit 41 stores the identified facial expression in a memory provided in the timing detection unit 28.

  The person extracting means 41 further identifies a gesture of a person having the detected face. Gesture identification is executed only when the gesture identification function is set to ON in the detailed settings of the automatic shooting mode. Alternatively, facial expressions may be identified when the detected face size is greater than or equal to a predetermined value, and gestures may be identified when the detected face size is less than or equal to a predetermined value.

  In the present embodiment, a well-known gesture is registered in advance in the memory included in the timing detection unit 28 as data representing the geometric feature of the gesture. For example, a gesture (piece) in which only the forefinger and middle finger are exemplarily illustrated in FIG. 12A, a gesture for raising both arms illustrated in FIG. There are registered gestures such as gesturing to raise only the thumb and pointing up (good). The person extracting means 41 collates the geometric features extracted from the periphery of the face of the image data read from the memory 22 with the registered data. When the extracted feature matches the registered gesture feature, the gesture name or a predetermined gesture identifier is stored in a memory included in the timing detection unit 28.

  Various methods are known as a method for identifying a gesture, including the method introduced in Japanese Patent Laid-Open No. 2001-51338. Any known technique can be used for the gesture identification process executed by the person extracting means 41.

  The person extraction means 41 then calculates the total area of the face area. For example, in the example of FIGS. 10A to 10D, the total area of the regions displayed as dotted line frames is calculated. However, the total area of the whole body region may be calculated.

  When the total of the calculated area areas exceeds a predetermined threshold, the person extracting means 41 determines the number of faces stored in the memory included in the timing detection unit 28, the area of the face area, the area of the whole body area, the center of gravity. Information on coordinates, facial expressions, and gestures is supplied only to the composition determination means 44. On the other hand, when the calculated total area is equal to or smaller than a predetermined threshold value, the information stored in the memory is supplied to the composition determination unit 44 and the non-person extraction unit 42.

  The non-person extracting means 42 extracts main subjects other than the person included in the image data. In the present embodiment, the non-person extracting means 42 reads the image data stored in the memory 22 and replaces the pixel value of the area corresponding to the person including the face or the body with 0 in the image data. Erase the person area data from. For example, it is assumed that the read image data is image data including persons 50a and 50b and a subject 51 other than the person as illustrated in FIG. 13A. In addition, it is assumed that information such as the center of gravity is supplied from the person extraction unit 41 for the regions 52a and 52b surrounded by the dotted frame in the figure. If the pixel data in the person areas 52a and 52b are deleted from the image data, image data including only the subject 51 can be obtained as illustrated in FIG. 13B.

  Subsequently, the non-person extracting unit 42 performs a filtering process using a high-pass filter on the image data from which the information of the person areas 52a and 52b has been deleted. Thereby, as illustrated in FIG. 13C, an edge image 53 in which the edge portion of the subject 51 is extracted is obtained. Since the edge image 53 is an image composed of the contour of a subject other than a person included in the image data, by analyzing this image, as shown in FIG. 13D, a rough area where the subject 53 is arranged is illustrated. 54 can be identified. The non-person extracting unit 42 calculates the area and barycentric coordinates of the identified region 54 and supplies the calculated area 54 to the composition determining unit 44.

  As a method of identifying a subject area other than a person, a method of extracting only a specific frequency component (corresponding to an edge) using Fourier transform instead of filtering processing using a high-pass filter is conceivable. Also, a method of extracting a main subject by performing analysis using color information instead of frequency analysis is also conceivable. For example, when the pixel value is a value representing a predetermined color, the value is left as it is, and when the pixel value is a value representing a color other than the predetermined color, the pixel value is replaced with 0 or 1. As a result, the image is divided into two regions, and the region where the subject of the predetermined color or the subject other than the predetermined color is arranged is extracted. In addition, an object that is likely to be photographed with a person (for example, an animal often kept as a pet) can be identified using a discrimination algorithm based on learning, such as the Adaboost algorithm, to generate data indicating a subject area.

  Note that the sharpness of the image may depend on the shutter speed when the image data is acquired, and the color of the image may depend on the photometric value or the aperture. For this reason, when performing image analysis, the subject area may be easily identified by taking into account various adjustment values and setting values when the image data is generated.

  The voice analysis means 43 analyzes the voice input from the microphone 4 and detects the following voice. First, the voice analysis means 43 constantly measures the volume of the voice input from the microphone 4 and compares it with a predetermined threshold value. FIG. 14 shows a graph with the horizontal axis representing time and the vertical axis representing volume. The voice analysis means 43 detects the time point T in the example of FIG. 14, that is, the time point when the volume suddenly changes and exceeds the threshold value Th. When shooting at a sporting event or shooting at a party, the moment of cheering, such as a soccer goal scene or a wedding toast scene, is often a photo opportunity. Therefore, it is possible to detect a photo opportunity by detecting a time point when the sound volume changes suddenly. Alternatively, instead of detecting that the volume has changed, it may only be detected that the volume exceeds the threshold Th. This is because it can always be considered as a photo opportunity while cheers are raised. On the other hand, when it is a photo opportunity when it is quiet, for example, when a baby's sleeping face is desired to be photographed, a situation where the volume falls below or below the threshold may be detected. As a result of analyzing the volume, which time point is detected can be changed by setting.

  Moreover, the voice analysis means 43 identifies the phrase represented by the voice, and collates it with a specific phrase registered in advance. This registration data is stored in a memory provided in the timing detection unit 28. For example, phrases that are highly likely to be issued in synchronization with the timing of pressing the shutter release button, such as “high, cheese” and “campai” are registered. ing. Moreover, in this embodiment, a voice can be registered as one of the registration data, and furthermore, a voice and a phrase can be registered in association with each other. The voice analysis means 43 collates with these registration data, so that (a) when the registered phrase is uttered as voice, (b) when the registrant utters voice, (c) when the registrant registers. It is possible to detect when a phrase is uttered as speech. In principle, which time point (a), (b), or (c) is detected depends on the setting. However, processing different from the setting may be executed according to the registration status of the registration data. For example, even if (c) is set as the time point to be detected, if the voice is not registered, the time point (a) is detected.

  Note that it is dependent on the setting whether both the detection process based on volume and the detection process based on phrase matching are executed, or only one or only one is executed.

  Next, processing of the composition determination unit 44 will be described. As shown in FIG. 9, the composition determination means 44 is supplied with image data read from the memory 22, extraction results by the person extraction means 41 and non-person extraction means 42, and detection results by the sound analysis means 43. The However, when extraction or detection is not performed, a value (for example, 0) indicating that there is no information to be supplied is input.

  FIG. 15 is a flowchart illustrating an example of processing of the composition determination unit 44. For the composition determination means 44, information on the face area range and area of each person, the whole body area range and area, the person's barycentric coordinates, facial expression, and gesture information from the person extraction means 41 is received from the non-person extraction means 42. It is assumed that information on the range, area, and barycentric coordinates of a subject other than a person and information on the detection result of sound are supplied from the sound analysis means 43.

  The composition determination unit 44 first evaluates the balance of the arrangement of the subject including the person (S301). If the person extracting means 41 detects N persons (N is an integer) and the non-person extracting means 42 detects M (M is an integer) subjects, the composition determining means 44 The center-of-gravity coordinates of the entire N + M areas are calculated from the center-of-gravity coordinates of the person area or the subject area. For example, in the case of the images illustrated in FIGS. 13A to 13D, as shown in FIG. 16, the centroid G of the three regions is based on the coordinates of the centroids g1 and g2 of the person regions 52a and 52b and the centroid g3 of the subject region 54. The coordinates of are calculated. If the center of gravity G is within the predetermined range 55 in the center of the image, it is determined that the arrangement balance is good, and if it is outside the predetermined range 55, it is determined that the arrangement balance is bad.

  When calculating the center-of-gravity coordinates of the entire N + M regions, the center-of-gravity coordinates may be obtained after weighting the center of gravity of each region according to the area of the target. If the weighting is increased for an object having a larger area, the center of gravity of the entire region becomes a position closer to the object having a larger area. For example, in the illustration of FIG. 17, the center of gravity obtained by weighting each region equally is the point GA that is out of the range 55, and it is determined that the arrangement balance is poor. However, in the calculation method in which the weighting is increased as the area is larger, the calculated center of gravity is the point GB within the range 55, and it is determined that the arrangement balance is good.

  In addition, the composition determination unit 44 evaluates not only the arrangement balance but also the rotation blur for some subjects. Here, the rotation blur means that the direction and direction of the subject included in the image are different from the direction and direction of the subject in the real world. For example, when a high-rise building that stands vertically is arranged obliquely in the image, it is determined that there is rotational shake. In the form in which the non-person extracting unit 42 performs subject extraction using a discrimination algorithm based on learning, the non-person extracting unit 42 can determine not only the contour of the subject but also the type of the subject. For such a subject, information indicating the type of the subject is supplied from the non-person extracting unit 43 to the composition determining unit 44. The composition determination means 44 calculates the orientation and direction of the extracted subject when the extracted subject is an object that is vertically or horizontally arranged in the real world, such as a high-rise building or a horizontal line. Determine presence or absence.

  The composition determination unit 44 outputs a determination result that the composition is bad (NG) when it is determined in step S302 that the arrangement balance is poor or when there is a rotational shake (S306).

  When the composition determination means 44 determines that the arrangement balance is good and there is no rotation blur, the facial expression of the person is a specific expression suitable for pressing the shutter release button based on the facial expression information supplied from the person extraction means 41. It is determined whether or not (S303). Alternatively, it is compared with the facial expression information supplied immediately before to determine whether or not the facial expression has changed. However, facial expression determination may be performed only when the area of the face region is equal to or greater than a predetermined value. If the expression is specific (or has changed), a determination result that the composition is good (OK) is output (S307).

  If the facial expression is not a specific facial expression (or no change), the composition determination means 44 performs a gesture suitable for the person to press the shutter release button based on the gesture information supplied from the person extraction means 41. It is determined whether or not there is (S304). Alternatively, it is checked against the gesture information supplied immediately before to determine whether there has been a change in the movement of the person. However, the determination of the gesture may be performed only when the area of the person region is equal to or larger than a predetermined value. When there is a specific gesture or movement, a determination result that the composition is good is output (S307).

  If there is no specific gesture or movement, the composition determination unit 44 determines whether a specific voice is detected based on the information supplied from the voice analysis unit 43 (S305). If a specific voice is not detected, a determination result that the composition is bad is output (S306), and if a specific voice is detected, a determination result that the composition is good is output. (S307).

  When the composition determination unit 44 outputs a determination result indicating that the composition is good, the timing detection unit 28 transmits the determination result to the overall control unit 30. The overall control unit 30 that has received this determination result sends the image data stored in the memory 22 to the memory card 7 when the digital camera 1 is set to the automatic shooting mode. To record. If the photographing assist mode is set, the display control unit 26 is instructed to display a mark or a message indicating the arrival of a photo opportunity on the monitor (FIG. 5). Alternatively, the LED controller 19 is instructed to turn on the LED 9 (FIG. 6).

  In the present embodiment, the recording / reading control unit 27 records the information used for composition determination on the memory card 7 as information attached to the image data. Specifically, it is recorded in the tag portion of the Exif file. On the other hand, when the composition determination unit 44 determines that the composition is bad, the composition determination unit 44 supplies information used to determine the composition to the composition proposal unit 45. The composition proposing means 45 executes the processing described below using these pieces of information.

  The composition proposing means 45 analyzes the information supplied from the composition determining means 44 and proposes a more preferable composition for an image determined to have a poor composition. Proposing a composition here means determining an arrangement that satisfies a composition determination criterion for a subject included in an image. The determined composition is output together with information on processing to be executed to acquire an image of the composition. For example, as illustrated in FIG. 18A, when the regions 52a, 52b, and 54 extracted from the captured image are arranged to be shifted to the lower left, the center of gravity of the regions 52a, 52b, and 54 is illustrated as illustrated in FIG. 18B. A composition in which G is arranged at the center of the image is proposed. Alternatively, as illustrated in FIG. 18C, a composition is proposed in which the center of gravity G of the regions 52a, 52b, and 54 is arranged at the center of the image and the subject is shown larger. Then, two types of information are output as information of processing to be executed to acquire an image of the composition.

  The first information output by the composition proposing means 45 is information necessary when converting image data acquired by photographing into image data having a preferable composition by image processing. For example, in the example of FIG. 18B, the trimming range (thick frame in the figure) and the moving direction (movement vector) of the center of gravity G are output. In the example of FIG.

  The second information output by the composition proposing means 45 is information necessary for acquiring image data having a more preferable composition by re-photographing. For example, in the example of FIG. 18B, information indicating an operation for turning the camera to the left is output. Further, in the example of FIG. 18C, an operation for turning the camera to the left and information on the photographing magnification to be set are output.

  In addition, there is rotational blurring as a cause for determining that the composition is bad. In the case of rotational blurring, the first information includes information on the rotation direction and angle when correcting the tilt, and second information. Information indicating that the camera should be tilted to the left or right is output. In addition, if the factor determined to be poor composition is a facial expression, gesture or voice, it cannot be corrected by image processing, so information indicating that the judgment factor is a facial expression, gesture or voice is output. The

  Each information output from the composition proposing means 45 is transmitted to the overall control unit 30. Receiving this determination result, the overall control unit 30 determines whether the digital camera 1 is set to the automatic shooting mode or the shooting assist mode, and performs processing according to each mode.

  When the digital camera 1 is set to the automatic shooting mode, the overall control unit 30 reads image data from the memory 22 into the image processing unit 25 and performs image processing (trimming and movement) necessary for improving the composition. , Scaling, rotation, etc.). Further, the display control unit 26 is instructed to display the image data processed by the image processing unit 25 on the monitor 6. Furthermore, it instructs the recording / reading control unit 27 to record the processed image data by the image processing unit 25 in the memory card 7.

  In the present embodiment, when the display control unit 26 receives the above instruction, as illustrated in FIG. 19, the display control unit 26 records image data (captured image) of the composition as photographed, or an image of the proposed composition. A selection screen for allowing the user to select whether to record data (proposed image) or to record both a captured image and a proposed image is displayed. Then, the recording / reading control unit 27 records the image data selected on this selection screen in the memory card 7. However, such a selection screen may not be displayed, and only the proposed image or both the captured image and the proposed image may be recorded unconditionally.

  In the present embodiment, the recording / reading control unit 27 that has received the above instruction receives information used by the composition determination unit 44 for composition determination, that is, the center-of-gravity coordinates of the entire N + M areas, the orientation of the subject for which rotation blur has been determined, Information such as the facial expression and gesture of the person and the detected voice is recorded in the memory card 7 as the auxiliary information of the image data. Furthermore, the first information output by the composition proposing means 45, that is, information necessary for converting image data acquired by photographing into image data having a preferable composition by image processing is also attached as image data. Record in the memory card 7. Specifically, these pieces of information are recorded in the tag portion of the Exif file.

  The information recorded in the tag portion of the Exif file can be used when editing an image using a personal computer. For example, if there is a photographed image and the first information output by the composition proposing means 45, an image equivalent to the proposed image can be generated on a personal computer. File size can be reduced. In addition, an image having a composition different from the composition proposed by the composition proposing means 45 can be generated by slightly modifying the first information output from the composition proposing means 45.

  On the other hand, when the digital camera 1 is set to the shooting assist mode, the overall control unit 30 reads image data from the memory 22 into the image processing unit 25 and performs image processing (trimming) necessary for improving the composition. , Move, scale, rotate, etc.). Further, the display control unit 26 is instructed to display the image data processed by the image processing unit 25 and the mark or message generated based on the second information output from the composition proposal unit 45. As a result, the assist display described with reference to FIGS. 7 and 8 is performed.

  In the present embodiment, if the digital camera is set to the automatic shooting mode, the composition of the shot image is good and the image data is automatically recorded on the memory card when a predetermined sound is detected. The digital camera does not react if only a predetermined sound is detected, so a shooting operation is performed before the camera is pointed at the subject, or an unnecessary shooting operation is performed in response to the voice of a person who happened to be near Nor. That is, it is possible to eliminate the inconvenience of unnecessary shooting while ensuring the convenience of automatic shooting using voice as a trigger.

  In the photographing assist mode, the photographer is notified of the photo opportunity, but this notification is not performed only when a predetermined sound is detected. Only when the composition of the photographed image is good and a predetermined sound is detected, it is notified that there is a photo opportunity, so that no erroneous notification is made. In the shooting assist mode, an image with a good composition can be easily acquired by simply pressing the shutter release button at the notified timing, and the same convenience as in automatic shooting can be enjoyed.

  In the above embodiment, the non-person extracting unit 42 extracts a target other than a person, but a mode in which the non-person extracting unit 42 is not provided is also conceivable. In this case, the composition determination means determines the composition by evaluating the arrangement of only people. Alternatively, the arrangement may not be evaluated, and the composition may be determined as long as a person is included, or may be determined as a bad composition if the person is not included. Various other determination methods, information used for the determination, and determination criteria for determining the composition are conceivable and are not limited to the above-described embodiment. In the above embodiment, the composition proposal unit 45 proposes a composition, but a configuration in which the composition proposal unit 45 is not provided is also conceivable.

  In the description of the above embodiment, a case where still image shooting is performed is described as an example. However, the present invention is also useful when it is desired to automatically start recording in moving image shooting.

Diagram showing the appearance of a digital camera (front) Figure showing the appearance of the digital camera (back) Diagram showing the internal configuration of the digital camera Flow chart showing the outline of digital camera operation (automatic shooting mode) Flow chart showing the outline of digital camera operation (shooting assist mode) Diagram showing an example of timing notification The figure which shows the other example of information of timing Diagram showing an example of assist display The figure which shows the other example of an assist display Diagram showing the configuration of the timing detector The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating gesture identification processing The figure for demonstrating gesture identification processing The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person Diagram showing an example of speech analysis A flowchart showing an example of composition determination processing The figure for demonstrating composition determination processing The figure for demonstrating composition determination processing Diagram for explaining composition proposal processing Diagram for explaining composition proposal processing Diagram for explaining composition proposal processing The figure which shows an example of a recording image selection screen

Explanation of symbols

1 digital camera, 2 lens, 3 shutter release button, 4 microphone,
5a to 5f Operation dial / button, 6 monitor, 7 memory card,
8 Speaker, 9 LED lamp

Claims (11)

An imaging apparatus comprising: an imaging unit that shoots a scene and generates image data representing the scene; and a recording unit that records image data generated by the imaging unit on a predetermined recording medium,
A person extracting means for extracting an image region representing a person by analyzing the image data generated by the imaging means;
Voice analysis means for detecting a predetermined feature of the voice by analyzing the input voice;
Composition determining means for determining the quality of the composition of the image data based on the extraction result of the person extracting means and the detection result of the voice analyzing means;
An imaging apparatus comprising: a recording control unit that determines a timing for recording image data based on a determination result of the composition determination unit, and controls the recording unit so that the image data is recorded at the determined timing. An imaging apparatus comprising: an imaging unit that shoots a scene and generates image data representing the scene; and a recording unit that records image data generated by the imaging unit on a predetermined recording medium,
A person extracting means for extracting an image region representing a person by analyzing the image data generated by the imaging means;
Voice analysis means for detecting a predetermined feature of the voice by analyzing the input voice;
Composition determining means for determining the quality of the composition of the image data based on the extraction result of the person extracting means and the detection result of the voice analyzing means;
An imaging apparatus comprising: a notification unit that determines a timing for recording image data based on a determination result of the composition determination unit, and notifies the arrival of the determined timing.   3. The imaging apparatus according to claim 1, wherein the recording unit records the extraction result of the person extraction unit and the detection result of the voice analysis unit together with the image data on the recording medium.   The imaging apparatus according to claim 1, wherein the sound analysis unit detects a predetermined volume change as a characteristic of the sound.   The imaging apparatus according to claim 1, wherein the voice analysis unit detects a predetermined voice phrase as a feature of the voice.   The imaging apparatus according to claim 1, wherein the voice analysis unit detects a characteristic registered in advance as a voice characteristic of a predetermined person as the voice characteristic.   The person extracting means searches for a face included in the image data, and outputs, as the extraction result, information indicating the number of faces detected by the search, the position of each face, and the size of each face. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is characterized.   8. The imaging apparatus according to claim 7, wherein the person extracting unit identifies facial expressions detected by the search, and further outputs information indicating the identified facial expressions.   The said person extraction means identifies the gesture of the person contained in the said image data, The information which shows the identified gesture is output as the said extraction result, The any one of Claim 1 to 8 characterized by the above-mentioned. Imaging device. An imaging apparatus control method comprising: an imaging unit that captures a scene and generates image data representing the scene; and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium,
By analyzing the image data generated by the imaging means, an image region representing a person is extracted,
Analyzing the input speech to detect certain features related to the speech,
Based on the extraction result of the person and the detection result of the sound, the quality of the composition of the image data is determined,
Determine the timing for recording image data based on the result of the determination,
An image pickup apparatus control method for controlling the recording unit so that image data is recorded at a determined timing. An imaging apparatus control method comprising: an imaging unit that captures a scene and generates image data representing the scene; and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium,
By analyzing the image data generated by the imaging means, an image region representing a person is extracted,
Analyzing the input speech to detect certain features related to the speech,
Based on the extraction result of the person and the detection result of the sound, the quality of the composition of the image data is determined,
A method for controlling an imaging apparatus, which notifies a user of the arrival of the determined timing by controlling an operation of a predetermined output unit.

Images