JP2008219451A - Imaging device and control method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a digital camera for easily taking a picture with a favorable composition. <P>SOLUTION: The digital camera 1 is provided with a person extracting means 41, a non-person extracting means 42, a composition deciding means 44, and a composition guide means 45. The person extracting means 41 analyzes image data to extract an image area representing a person. The non-person extracting means 42 analyzes an image area other than the image area extracted by the person extracting means 41 to extract a main subject other than the person. The composition deciding means 44 evaluates the arrangement of the person and a main subject other than the person based upon the extraction result to decide whether the composition is favorable. The composition guide means 45 determines an arrangement meeting predetermined conditions of the person and the main subject. The operation of an imaging means is controlled so as to generate image data including the person and the main subject in the determined arrangement. Alternately, the image data are processed so that the person and the main subject are in the determined arrangement. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus that performs imaging control based on the composition of an image being shot, and a control method for the imaging apparatus.

A digital camera normally starts taking an image as soon as it is set to a shooting mode. Then, when the shutter release button is pressed, the image taken at that time is recorded on the memory card. An image shot before the shutter release button is pressed is displayed on the monitor, and is also used for shooting control such as focusing and exposure adjustment. Attempts have also been made to eliminate the operation of the shutter release button by controlling the timing of recording image data using this image. For example, Patent Document 1 discloses a camera that recognizes the orientation of a person's face in person shooting, determines whether or not the person faces a predetermined direction, and controls the recording operation according to the determination result. Has been. Patent Document 2 discloses a camera that analyzes the composition of a photographed scene and sets photographing conditions according to the composition. Further, Patent Document 3 discloses an imaging device that focuses on a predetermined portion of an image being shot and operates the shutter when the portion changes.
Japanese Patent Laid-Open No. 2001-051338 JP 2006-203346 A JP 2000-196934 A

  The AE (Auto Exposure) and AF (Auto Focus) functions of digital cameras have been improving year by year, and recently, even those who are not familiar with camera operations can take clear, bright and well-colored photographs. However, how to hold the camera and when to press the shutter release button still largely depends on the skill of the photographer. For this reason, it is still difficult for beginners to take pictures with good composition. For example, Patent Document 1 shows a control method for shooting one person, but does not show a control method for shooting a plurality of persons or a target other than a person. The camera of Patent Document 2 proposes to perform control by detecting the sky other than the face of a person and further detecting the inclination of the subject image. The correct control method is not shown. The imaging apparatus of Patent Document 3 also performs control only focusing on a predetermined portion designated by the user, and does not perform control considering the composition of the entire image.

  In view of the above circumstances, an object of the present invention is to provide an imaging apparatus that allows even a beginner to easily take a photograph with a good composition.

  In order to achieve the above object, the present invention provides two types of imaging devices. Each of these imaging apparatuses includes an imaging unit that captures a scene and generates image data representing the scene, and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium. A person extracting unit, a non-person extracting unit, a composition determining unit, and a composition proposing unit described below are provided.

  The person extracting unit extracts an image area representing the person by analyzing the image data generated by the imaging unit. For example, a face included in image data is searched, and information indicating the number of faces detected by the search, the position of each face, and the size of each face is output as an extraction result. In this case, the facial expression detected by the search may be identified, and information indicating the identified facial expression may be further output. Moreover, it is good also as what identifies the gesture of the person contained in image data, and outputs the information which shows the identified gesture as an extraction result.

  The non-person extracting unit analyzes an image area other than the image area extracted by the person extracting unit of the image data generated by the imaging unit, and extracts a main subject other than the person. For example, the main subject is extracted by performing a filtering process using a high-pass filter on the image data. Alternatively, by identifying a predetermined target registered in advance from the targets included in the image data, the target may be extracted as the main subject. Furthermore, the main subject may be extracted using both of these methods.

  The composition determination unit evaluates whether the arrangement of the main subject other than the person and the person satisfies a predetermined condition based on the extraction results of the person extraction unit and the non-person extraction unit, and the image based on the evaluation of the arrangement Judge the quality of the data composition. Further, the composition proposing means determines an arrangement satisfying a predetermined condition of the person and the main subject using the extraction results of the person extracting means and the non-person extracting means.

  The first image pickup apparatus of the present invention includes a person and a main subject in an arrangement determined by the composition proposal means in addition to the image pickup means, recording means, person extraction means, non-person extraction means, composition determination means and composition proposal means. Imaging control means for controlling the operation of the imaging means is provided so that the arranged image data is generated. According to the first imaging apparatus of the present invention, when the composition of the captured image is poor, the operation of the subsequent imaging means is changed (for example, the imaging magnification is changed), and the composition is automatically improved.

  The second image pickup apparatus of the present invention performs image processing on image data so that the arrangement of the person and the main subject is determined by the composition proposing means instead of the image pickup control means of the first image pickup apparatus. Image processing means for applying According to the second imaging device of the present invention, when the composition of the captured image is poor, an image with a good composition is automatically generated by image processing, and the composition is improved.

  The first and second apparatuses of the present invention determine recording timing for recording image data based on the determination result of the composition determination unit, and control the recording unit so that the image data is recorded at the determined timing. Means may be provided. In the configuration provided with this recording control means, image data is automatically recorded when an image with a good composition can be obtained, so that an image with a good composition can always be obtained regardless of the skill of the photographer. .

  Alternatively, instead of the recording control unit, a notification unit that determines the timing of recording image data based on the determination result of the composition determination unit and notifies the arrival of the determined timing may be provided. With the configuration provided with the notification means, the photographer can know the timing at which an image with a good composition was captured. Therefore, by pressing the shutter release button at the notified timing, an image with a good composition can be easily acquired.

  The recording means preferably records the extraction results of the person extracting means and the non-person extracting means on a recording medium together with the image data. Thereby, when the image data recorded on the recording medium is edited with a personal computer or the like, editing using the extraction result can be performed.

  In addition to the above means, both the first imaging device and the second imaging device may further include voice analysis means for detecting a predetermined feature related to the voice by analyzing the input voice. For example, as a voice feature, a predetermined volume change, a predetermined voice phrase, a feature registered in advance as a voice feature of a predetermined person, and the like are detected. In this case, the composition determination means determines the quality of the composition of the image data based on the characteristics of the sound detected by the sound analysis means and the above-described evaluation of the arrangement. In this case, it is preferable that the recording unit records the extraction results of the person extraction unit and the non-person extraction unit and the detection result of the sound analysis unit together with the image data on a recording medium. Image data can be recorded at a more appropriate timing by considering the sound when determining the quality of the composition.

  The first control method of the present invention is a method of operating as the first imaging device by controlling the imaging device according to the following procedure. First, an image region representing a person is extracted by analyzing the image data generated by the imaging means. Next, the main subject other than the person is extracted by analyzing the image area other than the image area representing the person in the image data generated by the imaging means. Subsequently, it is evaluated whether or not the arrangement of the extracted person and main subject satisfies a predetermined condition, and the quality of the composition of the image data is determined based on the evaluation of the arrangement. Then, the arrangement of the extracted person and the main subject that satisfies the predetermined condition is determined, and the operation of the imaging unit is controlled so that image data in which the person and the main subject are arranged in the determined arrangement is generated. .

  The second control method of the present invention is a method of operating as the second imaging device by controlling the imaging device according to the following procedure. First, an image region representing a person is extracted by analyzing the image data generated by the imaging means. Next, the main subject other than the person is extracted by analyzing the image area other than the image area representing the person in the image data generated by the imaging means. Subsequently, it is evaluated whether or not the extracted person and the arrangement of main subjects other than the person satisfy a predetermined condition, and the quality of the composition of the image data is determined based on the evaluation of the arrangement. Then, an arrangement that satisfies the predetermined condition for the extracted person and main subject is determined, and image processing is performed on the image data so that the arrangement of the main subject other than the person and the person is determined.

  Hereinafter, a digital camera that performs control using a first control method and a second control method is disclosed as an embodiment of the method and apparatus of the present invention. This digital camera has four operation modes: a normal shooting mode, an image reproduction mode, an automatic shooting mode, and a shooting assist mode. However, the two apparatuses and the control method of the present invention can be implemented independently, and are not limited to the embodiments described below.

  First, the configuration of the digital camera will be described. 1A and 1B are views showing the appearance of the digital camera 1. FIG. 1A shows the front of the digital camera, and FIG. 1B shows the back. As shown in these drawings, the digital camera 1 includes an imaging lens 2, a shutter release button 3, a microphone 4, an operation dial or buttons 5a to 5f, a monitor 6, and an LED lamp 9. Further, at the lower part of the digital camera 1, there is a slot cover (not shown) that can be opened and closed with a speaker 8, and a card slot into which a memory card 7 is loaded is provided in the slot cover.

  FIG. 2 is a diagram illustrating an internal configuration of the digital camera 1. As shown in this figure, the digital camera 1 includes an imaging unit including an imaging lens 2, a lens driving unit 16, a diaphragm 13, a diaphragm driving unit 17, a CCD 14, and a timing generator (TG) 18. The imaging lens 2 includes a plurality of functional lenses such as a focus lens for focusing on a subject and a zoom lens for realizing a zoom function. The lens driving unit 16 is a small motor such as a stepping motor, and adjusts the position of each functional lens so that the distance from the CCD 14 to each functional lens is a suitable distance for the purpose. The diaphragm 13 is composed of a plurality of diaphragm blades. The aperture drive unit 17 is a small motor such as a stepping motor, and adjusts the position of the aperture blades so that the aperture size of the aperture becomes a size suitable for the purpose. The CCD 14 is a CCD having 5 to 12 million pixels with a primary color filter, and discharges accumulated charges in response to an instruction signal from the timing generator 18. The timing generator 18 sends a signal to the CCD 14 so that charges are accumulated in the CCD 14 only for a desired time, thereby adjusting the shutter speed.

  The digital camera 1 also transfers an A / D conversion unit 15 that converts the output signal of the CCD 14 into a digital signal, and image data output from the A / D conversion unit 15 to another processing unit via the system bus 24. An image input control unit 23 and a memory 22 for temporarily storing image data transferred from the image input control unit 23 are provided.

  In addition, the digital camera 1 determines a focus adjustment unit 20 that instructs the lens driving unit 16 to move the lens and performs focusing, determines an aperture value and a shutter speed, and sends an instruction signal to the aperture driving unit 17 and the timing generator 18. An exposure adjusting unit 21 is provided. The digital camera 1 also includes an image processing unit 25 that performs image processing on the image data stored in the memory 22. The image processing unit 25 performs color tone correction and brightness correction to make an image have natural color and brightness, processing for correcting red eyes when the image data includes red eyes, and when the image composition is poor Various finishing processes are performed to improve the appearance of the image, such as a process of correcting the composition. The processed image data processed by the image processing unit 25 is stored in the memory 22 again.

  The digital camera 1 also includes a display control unit 26 that controls output of image data stored in the memory 22 to the monitor 6. The display control unit 26 thins out the number of pixels of the image data stored in the memory 22 so as to have a size suitable for display, and then outputs it to the liquid crystal monitor 9. The display control unit 26 also controls display of setting screens such as operating conditions.

  Further, the digital camera 1 includes a recording / reading control unit 27 that controls writing of image data stored in the memory 22 to the memory card 7 and loading of image data recorded on the memory card 7 to the memory 22. Prepare. The recording / reading control unit 27 records the image data obtained by photographing according to the user setting as it is or compression-encoded, and stores it in the memory card 7 as an Exif (Exchangeable Image File Format) file. Exif is a file format standard defined by the Japan Electronics Industry Promotion Association (JEIDA). When an operation for requesting reproduction of an image file recorded on the memory card 7 is performed, the image data included in the Exif file is loaded into the memory 22. If the image data is compressed, it is decompressed and loaded into the memory 22.

  The digital camera 1 also includes an LED control unit 19 that controls lighting and extinguishing of the LEDs 9. Furthermore, a microphone 4, a speaker 8, an A / D converter 10, a D / A converter 11, and a voice input / output controller 12 that controls voice input / output are provided. Audio data input from the microphone 4 and converted into digital data by the A / D converter 10 is transferred to the memory 22 via the system bus 24 by the audio input / output controller 12 and stored therein. Also, audio data supplied from each processing unit or the overall control unit described later to the audio input / output control unit 12 is converted by the D / A conversion unit 11 and then output to the speaker 8.

  The digital camera 1 also includes a timing detection unit 28 that detects the timing at which an image should be acquired. The timing detection unit 28 analyzes the image data and audio data stored in the memory 22 and outputs a signal indicating that the timing for acquiring an image has arrived when those data satisfy a predetermined condition. .

  In addition, the digital camera 1 includes a CPU (Central Processor Unit) 31, a RAM (Random Access Memory) 32 in which an operation / control program is stored, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) in which various setting values are stored. An overall control unit 30 is provided. The CPU 31 of the overall control unit 30 refers to the setting value stored in the EEPROM 33, and selects and executes a program stored in the RAM 32 based on the setting value. As a result, the overall control unit 30 detects the operation of the shutter release button 3 and the operation dial / buttons 5a to 5f, or receives the processing result of each processing unit, and the LED control unit 19, the focus adjustment unit 20, the exposure adjustment. An instruction signal for instructing a process to be executed is sent to the unit 21, the image input control unit 23, the image processing unit 25, the display control unit 26, the recording / reading control unit 27, the timing detection unit 28, and the voice input / output control unit 12. To do. Thereby, the operation of the digital camera 1 is controlled.

  In the normal shooting mode, the automatic shooting mode, and the shooting assist mode, each processing unit performs focus adjustment, exposure control, flash control, image processing, recording, and the like under the control of the overall control unit 30 to obtain an image. To be acquired. In the playback mode, an image recorded on the memory card 7 is output to the monitor 6 under the control of the overall control unit 30. In the setting mode, a setting screen is displayed on the monitor 6 under the control of the overall control unit 30, and operation inputs from the operation dial / buttons 5a to 5f are accepted. Information selected by the user using the operation dials / buttons 5 a to 5 f on the setting screen or information taken from the memory card 7 is stored in the EEPROM 33.

  Hereinafter, the automatic shooting mode and the shooting assist mode will be further described. FIG. 3 is a flowchart showing an outline of the operation of the digital camera 1 when the automatic shooting mode is set. When the digital camera 1 is set to the automatic shooting mode, the digital camera 1 starts generating image data representing a scene captured on the lens (S101). Then, the composition of the image represented by the generated image data is determined (S102), and if the composition is good, the image is recorded on the memory card 7 regardless of whether the shutter release button 3 is operated (S103). If the composition is bad, a better composition is proposed (S104), and the operation of the imaging unit is controlled or the image processing unit 25 is controlled so that the image data generated in step S101 becomes the composition proposed in step S104. A predetermined process is performed (S105). For example, when the main subject is too small, the imaging unit is caused to perform a zoom operation. When the main subject is biased, the image processing unit 25 is instructed to perform image processing for trimming and moving or enlarging only the region where the subject is shown. Alternatively, if a subject that should be standing vertically appears obliquely, a rotation process is performed so that the subject becomes vertical.

  The image data regenerated (S101) by the imaging unit or the image processing unit 25 is evaluated again in step S102. The above process is repeated until a mode switching operation is detected (S106).

  FIG. 4 is a flowchart showing an outline of the operation of the digital camera 1 when the photographing assist mode is set. When the digital camera 1 is set to the photographing assist mode, the digital camera 1 starts generating image data representing a scene captured on the lens (S201). Then, the composition of the image represented by the generated image data is determined (evaluated) (S202).

  If the composition is good, the arrival of timing is notified (S203). 5 and 6 show notification examples of timing arrival. FIG. 5 shows an example in which the arrival of timing is notified by displaying a mark 34 prompting the user to press the shutter release button on the monitor 6. Instead of the mark, a message such as “It is a photo opportunity” may be displayed. FIG. 6 shows an example in which the arrival of timing is notified by blinking the LED lamp 9. In addition, a method of notifying the arrival of timing by sound output from a speaker is also conceivable.

  If the composition is bad, a better composition is proposed (S204). Then, by displaying the proposed composition on the monitor 6 (hereinafter referred to as assist display), the photographer is prompted to correct the way the camera is held or to operate a predetermined operation button (S205). 7 and 8 show examples of assist display. FIG. 7 shows an example in which a preferred framing method is presented by displaying a framed frame 35 superimposed on image data that has been shot. FIG. 8 shows an example in which a preferred framing image generated by image processing is displayed, and a mark 36 is displayed that suggests how framing can be acquired at the edge of the screen to obtain image data such as the image being displayed. It is. In addition, a preferred framing may be presented by displaying a message such as “Please zoom in” or “Please turn the camera slightly to the left.” In the photographing assist mode, the above processing is repeated until a mode switching operation is detected (S206).

  Next, the process for determining the composition in steps S102 and S202 and the process for proposing the composition in steps S104 and S204 will be described in more detail. Composition determination and composition proposal are performed by the timing detection unit 28 in FIG. FIG. 9 is a diagram illustrating a configuration of the timing detection unit 28. As shown in the figure, the timing detection unit 28 includes a person extraction unit 41, a non-person extraction unit 42, a voice analysis unit 43, a composition determination unit 44, and a composition proposal unit 45. Note that the timing detection unit 28 may be a circuit constituted by an LSI functioning as the means 41 to 45, or a microcomputer incorporating a software program for executing the processing of the means 41 to 45.

  The person extracting means 41 reads the image data stored in the memory 22 and searches for a person area in the image data. In this embodiment, a person is detected by searching for a face. When the face is detected, the person extraction unit 41 adds an identifier such as a serial number, and then calculates the area of the face area, the area of the entire body including the face (hereinafter referred to as the whole body area), and the barycentric coordinates. . The center-of-gravity coordinates are obtained as the center-of-gravity coordinates of the face area when the area of the face exceeds a predetermined value, and the center-of-gravity coordinates of the whole body area when the area is less than the predetermined value. For example, when the face area is relatively large as illustrated in FIGS. 10A and 10B, the barycentric coordinates of each face area are calculated. When the face area is small as shown in FIGS. 10C and 10D, the barycentric coordinates of the whole body area indicated by the cross mark are calculated. When the search of all areas is completed, the person extracting means 41 uses the total number of detected persons, the range and area of the face area of each person, and the whole body as information indicating the number, position, and size of the detected persons. The range and area of the region and the barycentric coordinates are stored in a memory (not shown) provided in the timing detection unit 28.

  As a method for searching for and detecting a face, as introduced in Japanese Patent Application Laid-Open No. 2001-51338, a method for detecting a skin color area as a face, presence of parts having geometric features such as hair, eyes and mouth There are known a method for detecting a face by determining whether or not it is a face, and various other methods. Any known method can be used for the face detection processing executed by the person extracting means 41.

  The person extraction means 41 then identifies the detected facial expression. However, facial expression identification is performed only when the display identification function is set to ON in the detailed settings of the automatic shooting mode. Alternatively, facial expressions may be identified only when the size of the detected face is greater than or equal to a predetermined value. In the present embodiment, the person extraction unit 41 identifies four facial expressions, such as a laughing face, an angry face, a crying face, and a surprised face exemplified in FIGS. 11A, 11B, 11C, and 11D. As is clear from the example in the figure, each of these facial expressions is characterized by the degree of opening of the eyes and mouth and the degree of rise of the eyebrows and corners of the mouth, so that the facial expressions are identified based on the image characteristics of those parts. Can do. Various methods for identifying facial expressions are known, including the method introduced in Japanese Patent Laid-Open No. 2001-51338, and any known method is used for facial expression identification processing performed by the person extracting means 41. can do. The person extraction unit 41 stores the identified facial expression in a memory provided in the timing detection unit 28.

  The person extracting means 41 further identifies a gesture of a person having the detected face. Gesture identification is executed only when the gesture identification function is set to ON in the detailed settings of the automatic shooting mode. Alternatively, facial expressions may be identified when the detected face size is greater than or equal to a predetermined value, and gestures may be identified when the detected face size is less than or equal to a predetermined value.

  In the present embodiment, a well-known gesture is registered in advance in the memory included in the timing detection unit 28 as data representing the geometric feature of the gesture. For example, a gesture (piece) in which only the forefinger and middle finger are exemplarily illustrated in FIG. 12A, a gesture for raising both arms illustrated in FIG. There are registered gestures such as gesturing to raise only the thumb and pointing up (good). The person extracting means 41 collates the geometric features extracted from the periphery of the face of the image data read from the memory 22 with the registered data. When the extracted feature matches the registered gesture feature, the gesture name or a predetermined gesture identifier is stored in a memory included in the timing detection unit 28.

  Various methods are known as a method for identifying a gesture, including the method introduced in Japanese Patent Laid-Open No. 2001-51338. Any known technique can be used for the gesture identification process executed by the person extracting means 41.

  The person extraction means 41 then calculates the total area of the face area. For example, in the example of FIGS. 10A to 10D, the total area of the regions displayed as dotted line frames is calculated. However, the total area of the whole body region may be calculated.

  When the total of the calculated area areas exceeds a predetermined threshold, the person extracting means 41 determines the number of faces stored in the memory included in the timing detection unit 28, the area of the face area, the area of the whole body area, the center of gravity. Information on coordinates, facial expressions, and gestures is supplied only to the composition determination means 44. On the other hand, when the calculated total area is equal to or smaller than a predetermined threshold value, the information stored in the memory is supplied to the composition determination unit 44 and the non-person extraction unit 42.

  The non-person extracting means 42 extracts main subjects other than the person included in the image data. In the present embodiment, the non-person extracting means 42 reads the image data stored in the memory 22 and replaces the pixel value of the area corresponding to the person including the face or the body with 0 in the image data. Erase the person area data from. For example, it is assumed that the read image data is image data including persons 50a and 50b and a subject 51 other than the person as illustrated in FIG. 13A. In addition, it is assumed that information such as the center of gravity is supplied from the person extraction unit 41 for the regions 52a and 52b surrounded by the dotted frame in the figure. If the pixel data in the person areas 52a and 52b are deleted from the image data, image data including only the subject 51 can be obtained as illustrated in FIG. 13B.

  Subsequently, the non-person extracting unit 42 performs a filtering process using a high-pass filter on the image data from which the information of the person areas 52a and 52b has been deleted. Thereby, as illustrated in FIG. 13C, an edge image 53 in which the edge portion of the subject 51 is extracted is obtained. Since the edge image 53 is an image composed of the contour of a subject other than a person included in the image data, by analyzing this image, as shown in FIG. 13D, a rough area where the subject 53 is arranged is illustrated. 54 can be identified. The non-person extracting unit 42 calculates the area and barycentric coordinates of the identified region 54 and supplies the calculated area 54 to the composition determining unit 44.

  As a method of identifying a subject area other than a person, a method of extracting only a specific frequency component (corresponding to an edge) using Fourier transform instead of filtering processing using a high-pass filter is conceivable. Also, a method of extracting a main subject by performing analysis using color information instead of frequency analysis is also conceivable. For example, when the pixel value is a value representing a predetermined color, the value is left as it is, and when the pixel value is a value representing a color other than the predetermined color, the pixel value is replaced with 0 or 1. As a result, the image is divided into two regions, and the region where the subject of the predetermined color or the subject other than the predetermined color is arranged is extracted. In addition, an object that is likely to be photographed with a person (for example, an animal often kept as a pet) can be identified using a discrimination algorithm based on learning, such as the Adaboost algorithm, to generate data indicating a subject area.

  Note that the sharpness of the image may depend on the shutter speed when the image data is acquired, and the color of the image may depend on the photometric value or the aperture. For this reason, when performing image analysis, the subject area may be easily identified by taking into account various adjustment values and setting values when the image data is generated.

  The voice analysis means 43 analyzes the voice input from the microphone 4 and detects the following voice. However, in the present embodiment, the processing described below is executed only when the voice analysis function is set to ON. First, the voice analysis means 43 constantly measures the volume of the voice input from the microphone 4 and compares it with a predetermined threshold value. FIG. 14 shows a graph with the horizontal axis representing time and the vertical axis representing volume. The voice analysis means 43 detects the time point T in the example of FIG. 14, that is, the time point when the volume suddenly changes and exceeds the threshold value Th. When shooting at a sporting event or shooting at a party, the moment of cheering, such as a soccer goal scene or a wedding toast scene, is often a photo opportunity. Therefore, it is possible to detect a photo opportunity by detecting a time point when the sound volume changes suddenly. Alternatively, instead of detecting that the volume has changed, it may only be detected that the volume exceeds the threshold Th. This is because it can always be considered as a photo opportunity while cheers are raised. On the other hand, when it is a photo opportunity when it is quiet, for example, when a baby's sleeping face is desired to be photographed, a situation where the volume falls below or below the threshold may be detected. As a result of analyzing the volume, which time point is detected can be changed by setting.

  Moreover, the voice analysis means 43 identifies the phrase represented by the voice, and collates it with a specific phrase registered in advance. This registration data is stored in a memory provided in the timing detection unit 28. For example, phrases that are highly likely to be issued in synchronization with the timing of pressing the shutter release button, such as “high, cheese” and “campai” are registered. ing. Moreover, in this embodiment, a voice can be registered as one of the registration data, and furthermore, a voice and a phrase can be registered in association with each other. The voice analysis means 43 collates with these registration data, so that (a) when the registered phrase is uttered as voice, (b) when the registrant utters voice, (c) when the registrant registers. It is possible to detect when a phrase is uttered as speech. In principle, which time point (a), (b), or (c) is detected depends on the setting. However, processing different from the setting may be executed according to the registration status of the registration data. For example, even if (c) is set as the time point to be detected, if the voice is not registered, the time point (a) is detected.

  Note that it is dependent on the setting whether both the detection process based on volume and the detection process based on phrase matching are executed, or only one or only one is executed.

  Next, processing of the composition determination unit 44 will be described. As shown in FIG. 9, the composition determination means 44 is supplied with image data read from the memory 22, extraction results by the person extraction means 41 and non-person extraction means 42, and detection results by the sound analysis means 43. The However, when extraction or detection is not performed, a value (for example, 0) indicating that there is no information to be supplied is input.

  FIG. 15 is a flowchart illustrating an example of processing of the composition determination unit 44. For the composition determination means 44, information on the face area range and area of each person, the whole body area range and area, the person's barycentric coordinates, facial expression, and gesture information from the person extraction means 41 is received from the non-person extraction means 42. It is assumed that information on the range, area, and barycentric coordinates of a subject other than a person and information on the detection result of sound are supplied from the sound analysis means 43.

  The composition determination unit 44 first evaluates the balance of the arrangement of the subject including the person (S301). If the person extracting means 41 detects N persons (N is an integer) and the non-person extracting means 42 detects M (M is an integer) subjects, the composition determining means 44 The center-of-gravity coordinates of the entire N + M areas are calculated from the center-of-gravity coordinates of the person area or the subject area. For example, in the case of the images illustrated in FIGS. 13A to 13D, as shown in FIG. 16, the centroid G of the three regions is based on the coordinates of the centroids g1 and g2 of the person regions 52a and 52b and the centroid g3 of the subject region 54. The coordinates of are calculated. If the center of gravity G is within the predetermined range 55 in the center of the image, it is determined that the arrangement balance is good, and if it is outside the predetermined range 55, it is determined that the arrangement balance is bad.

  When calculating the center-of-gravity coordinates of the entire N + M regions, the center-of-gravity coordinates may be obtained after weighting the center of gravity of each region according to the area of the target. If the weighting is increased for an object having a larger area, the center of gravity of the entire region becomes a position closer to the object having a larger area. For example, in the illustration of FIG. 17, the center of gravity obtained by weighting each region equally is the point GA that is out of the range 55, and it is determined that the arrangement balance is poor. However, in the calculation method in which the weighting is increased as the area is larger, the calculated center of gravity is the point GB within the range 55, and it is determined that the arrangement balance is good.

  In addition, the composition determination unit 44 evaluates not only the arrangement balance but also the rotation blur for some subjects. Here, the rotation blur means that the direction and direction of the subject included in the image are different from the direction and direction of the subject in the real world. For example, when a high-rise building that stands vertically is arranged obliquely in the image, it is determined that there is rotational shake. In the form in which the non-person extracting unit 42 performs subject extraction using a discrimination algorithm based on learning, the non-person extracting unit 42 can determine not only the contour of the subject but also the type of the subject. For such a subject, information indicating the type of the subject is supplied from the non-person extracting unit 43 to the composition determining unit 44. The composition determination means 44 calculates the orientation and direction of the extracted subject when the extracted subject is an object that is vertically or horizontally arranged in the real world, such as a high-rise building or a horizontal line. Determine presence or absence.

  The composition determination unit 44 outputs a determination result that the composition is bad (NG) when it is determined in step S302 that the arrangement balance is poor or when there is a rotational shake (S306).

  When the composition determination means 44 determines that the arrangement balance is good and there is no rotation blur, the facial expression of the person is a specific expression suitable for pressing the shutter release button based on the facial expression information supplied from the person extraction means 41. It is determined whether or not (S303). Alternatively, it is compared with the facial expression information supplied immediately before to determine whether or not the facial expression has changed. However, facial expression determination may be performed only when the area of the face region is equal to or greater than a predetermined value. If the expression is specific (or has changed), a determination result that the composition is good (OK) is output (S307).

  If the facial expression is not a specific facial expression (or no change), the composition determination means 44 performs a gesture suitable for the person to press the shutter release button based on the gesture information supplied from the person extraction means 41. It is determined whether or not there is (S304). Alternatively, it is checked against the gesture information supplied immediately before to determine whether there has been a change in the movement of the person. However, the determination of the gesture may be performed only when the area of the person region is equal to or larger than a predetermined value. When there is a specific gesture or movement, a determination result that the composition is good is output (S307).

  If there is no specific gesture or movement, the composition determination unit 44 determines whether a specific voice is detected based on the information supplied from the voice analysis unit 43 (S305). If a specific voice is not detected, a determination result that the composition is bad is output (S306), and if a specific voice is detected, a determination result that the composition is good is output. (S307).

  When the composition determination unit 44 outputs a determination result indicating that the composition is good, the timing detection unit 28 transmits the determination result to the overall control unit 30. The overall control unit 30 that has received this determination result sends the image data stored in the memory 22 to the memory card 7 when the digital camera 1 is set to the automatic shooting mode. To record. If the photographing assist mode is set, the display control unit 26 is instructed to display a mark or a message indicating the arrival of a photo opportunity on the monitor (FIG. 5). Alternatively, the LED controller 19 is instructed to turn on the LED 9 (FIG. 6).

  In the present embodiment, the recording / reading control unit 27 records the information used for composition determination on the memory card 7 as information attached to the image data. Specifically, it is recorded in the tag portion of the Exif file. On the other hand, when the composition determination unit 44 determines that the composition is bad, the composition determination unit 44 supplies information used to determine the composition to the composition proposal unit 45. The composition proposing means 45 executes the processing described below using these pieces of information.

  The composition proposing means 45 analyzes the information supplied from the composition determining means 44 and proposes a more preferable composition for an image determined to have a poor composition. Proposing a composition here means determining an arrangement that satisfies a composition determination criterion for a subject included in an image. The determined composition is output together with information on processing to be executed to acquire an image of the composition. For example, as illustrated in FIG. 18A, when the regions 52a, 52b, and 54 extracted from the captured image are arranged to be shifted to the lower left, the center of gravity of the regions 52a, 52b, and 54 is illustrated as illustrated in FIG. 18B. A composition in which G is arranged at the center of the image is proposed. Alternatively, as illustrated in FIG. 18C, a composition is proposed in which the center of gravity G of the regions 52a, 52b, and 54 is arranged at the center of the image and the subject is shown larger. Then, two types of information are output as information of processing to be executed to acquire an image of the composition.

  The first information output by the composition proposing means 45 is information necessary when converting image data acquired by photographing into image data having a preferable composition by image processing. For example, in the example of FIG. 18B, the trimming range (thick frame in the figure) and the moving direction (movement vector) of the center of gravity G are output. In the example of FIG.

  The second information output by the composition proposing means 45 is information necessary for acquiring image data having a more preferable composition by re-photographing. For example, in the example of FIG. 18B, information indicating an operation for turning the camera to the left is output. Further, in the example of FIG. 18C, an operation for turning the camera to the left and information on the photographing magnification to be set are output.

  In addition, there is rotational blurring as a cause for determining that the composition is bad. In the case of rotational blurring, the first information includes information on the rotation direction and angle when correcting the tilt, and second information. Information indicating that the camera should be tilted to the left or right is output. In addition, if the factor determined to be poor composition is a facial expression, gesture or voice, it cannot be corrected by image processing, so information indicating that the judgment factor is a facial expression, gesture or voice is output. The

  Each information output from the composition proposing means 45 is transmitted to the overall control unit 30. Receiving this determination result, the overall control unit 30 determines whether the digital camera 1 is set to the automatic shooting mode or the shooting assist mode, and performs processing according to each mode.

  When the digital camera 1 is set to the automatic shooting mode, the overall control unit 30 reads image data from the memory 22 into the image processing unit 25 and performs image processing (trimming and movement) necessary for improving the composition. , Scaling, rotation, etc.). Further, the display control unit 26 is instructed to display the image data processed by the image processing unit 25 on the monitor 6. Furthermore, it instructs the recording / reading control unit 27 to record the processed image data by the image processing unit 25 in the memory card 7.

  In the present embodiment, when the display control unit 26 receives the above instruction, as illustrated in FIG. 19, the display control unit 26 records image data (captured image) of the composition as photographed, or an image of the proposed composition. A selection screen for allowing the user to select whether to record data (proposed image) or to record both a captured image and a proposed image is displayed. Then, the recording / reading control unit 27 records the image data selected on this selection screen in the memory card 7. However, such a selection screen may not be displayed, and only the proposed image or both the captured image and the proposed image may be recorded unconditionally.

  In the present embodiment, the recording / reading control unit 27 that has received the above instruction receives information used by the composition determination unit 44 for composition determination, that is, the center-of-gravity coordinates of the entire N + M areas, the orientation of the subject for which rotation blur has been determined, Information such as the facial expression and gesture of the person and the detected voice is recorded in the memory card 7 as the auxiliary information of the image data. Furthermore, the first information output by the composition proposing means 45, that is, information necessary for converting image data acquired by photographing into image data having a preferable composition by image processing is also attached as image data. Record in the memory card 7. Specifically, these pieces of information are recorded in the tag portion of the Exif file.

  The information recorded in the tag portion of the Exif file can be used when editing an image using a personal computer. For example, if there is a photographed image and the first information output by the composition proposing means 45, an image equivalent to the proposed image can be generated on a personal computer. File size can be reduced. In addition, an image having a composition different from the composition proposed by the composition proposing means 45 can be generated by slightly modifying the first information output from the composition proposing means 45.

  On the other hand, when the digital camera 1 is set to the shooting assist mode, the overall control unit 30 reads image data from the memory 22 into the image processing unit 25 and performs image processing (trimming) necessary for improving the composition. , Move, scale, rotate, etc.). Further, the display control unit 26 is instructed to display the image data processed by the image processing unit 25 and the mark or message generated based on the second information output from the composition proposal unit 45. As a result, the assist display described with reference to FIGS. 7 and 8 is performed.

  In this embodiment, even if an image with a good composition cannot be captured, the operation of the imaging unit is controlled or an image process is performed so that an image with a good composition can be obtained. An image with a good composition can be acquired without repositioning the camera or adjusting the zoom magnification or other settings.

  In the above embodiment, the composition determination is performed in consideration of the detected voice. However, a configuration in which the voice analysis unit 43 is not provided and the composition determination is performed based only on information extracted from the image is also conceivable. In the above embodiment, several determination / proposition methods such as a method for determining or proposing the composition by calculating the barycentric coordinates of the region are exemplified. However, the conditions to be satisfied by the composition and the data used for the condition determination are other Various examples are also conceivable, and the present invention is not limited to the above-described embodiments. In the description of the above embodiment, the case where still image shooting is performed is described as an example. However, the present invention is also useful when determining the recording start timing in moving image shooting.

Diagram showing the appearance of a digital camera (front) Figure showing the appearance of the digital camera (back) Diagram showing the internal configuration of the digital camera Flow chart showing the outline of digital camera operation (automatic shooting mode) Flow chart showing the outline of digital camera operation (shooting assist mode) Diagram showing an example of timing notification The figure which shows the other example of information of timing Diagram showing an example of assist display The figure which shows the other example of an assist display Diagram showing the configuration of the timing detector The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating the detection process of a face The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating facial expression identification processing The figure for demonstrating gesture identification processing The figure for demonstrating gesture identification processing The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person The figure for demonstrating the extraction process of subjects other than a person Diagram showing an example of speech analysis A flowchart showing an example of composition determination processing The figure for demonstrating composition determination processing The figure for demonstrating composition determination processing Diagram for explaining composition proposal processing Diagram for explaining composition proposal processing Diagram for explaining composition proposal processing The figure which shows an example of a recording image selection screen

Explanation of symbols

1 digital camera, 2 lens, 3 shutter release button, 4 microphone,
5a to 5f Operation dial / button, 6 monitor, 7 memory card,
8 Speaker, 9 LED lamp

Claims (17)

An imaging apparatus comprising: an imaging unit that shoots a scene and generates image data representing the scene; and a recording unit that records image data generated by the imaging unit on a predetermined recording medium,
A person extracting means for extracting an image region representing a person by analyzing the image data generated by the imaging means;
Non-person extraction means for extracting a main subject other than a person by analyzing an image area other than the image area extracted by the person extraction means of the image data generated by the imaging means;
Based on the extraction results of the person extraction means and the non-person extraction means, it is evaluated whether or not the arrangement of the main subject other than the person and the person satisfies a predetermined condition, and the image data based on the evaluation of the arrangement Composition determining means for determining the quality of the composition of
Using the extraction results of the person extracting means and the non-person extracting means to determine an arrangement that satisfies the predetermined condition of the person and the main subject;
An imaging apparatus comprising: an imaging control unit that controls an operation of the imaging unit so that image data in which the person and the main subject are arranged in an arrangement determined by the composition proposing unit is generated. An imaging apparatus comprising: an imaging unit that shoots a scene and generates image data representing the scene; and a recording unit that records image data generated by the imaging unit on a predetermined recording medium,
A person extracting means for extracting an image region representing a person by analyzing the image data generated by the imaging means;
Non-person extraction means for extracting a main subject other than a person by analyzing an image area other than the image area extracted by the person extraction means of the image data generated by the imaging means;
Based on the extraction results of the person extraction means and the non-person extraction means, it is evaluated whether or not the arrangement of the person and the main subject satisfies a predetermined condition, and the composition of the image data is determined based on the evaluation of the arrangement. Composition determination means for determining pass / fail;
Using the extraction results of the person extracting means and the non-person extracting means to determine an arrangement that satisfies the predetermined condition of the person and the main subject;
An image pickup apparatus comprising: an image processing unit that performs image processing on the image data so that the arrangement of the person and the main subject is an arrangement determined by the composition proposal unit.   A recording control unit configured to determine a timing for recording the image data based on a determination result of the composition determination unit, and to control the recording unit so that the image data is recorded at the determined timing; The imaging apparatus according to claim 1 or 2.   The imaging apparatus according to claim 1, further comprising notification means for determining a timing at which image data is recorded based on a determination result of the composition determination means and notifying arrival of the determined timing.   5. The imaging apparatus according to claim 1, wherein the recording unit records the extraction results of the person extracting unit and the non-person extracting unit together with the image data on the recording medium. 6.   The person extracting means searches for a face included in the image data, and outputs, as the extraction result, information indicating the number of faces detected by the search, the position of each face, and the size of each face. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized in that:   The imaging apparatus according to claim 6, wherein the person extracting unit identifies facial expressions detected by the search and further outputs information indicating the identified facial expressions.   The said person extraction means identifies the gesture of the person contained in the said image data, The information which shows the identified gesture is output as the said extraction result, The any one of Claim 1 to 7 characterized by the above-mentioned. Imaging device.   9. The image pickup apparatus according to claim 1, wherein the non-person extracting unit extracts the main subject by performing a filtering process using a high-pass filter on the image data. 10.   10. The non-person extracting unit extracts the target as the main subject by identifying a predetermined target registered in advance from the targets included in the image data. The imaging apparatus of any one of Claims. Voice analysis means for detecting a predetermined feature of the voice by analyzing the input voice;
11. The composition determination unit according to claim 1, wherein the composition determination unit determines whether the composition of the image data is acceptable based on the characteristics of the sound detected by the sound analysis unit and the evaluation of the arrangement. The imaging apparatus according to 1.   12. The imaging apparatus according to claim 11, wherein the recording unit records the extraction result of the person extraction unit and the non-person extraction unit and the detection result of the voice analysis unit together with the image data on the recording medium. .   The imaging apparatus according to claim 11, wherein the voice analysis unit detects a predetermined volume change as a characteristic of the voice.   The imaging apparatus according to claim 11, wherein the voice analysis unit detects a predetermined voice phrase as a feature of the voice.   The imaging apparatus according to claim 11, wherein the voice analysis unit detects a characteristic registered in advance as a voice characteristic of a predetermined person as the characteristic of the voice. An imaging apparatus control method comprising: an imaging unit that captures a scene and generates image data representing the scene; and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium,
By analyzing the image data generated by the imaging means, an image region representing a person is extracted,
By analyzing an image area other than the image area representing the person in the image data generated by the imaging means, a main subject other than the person is extracted,
Evaluate whether the extracted person and main subject placement meet a given condition,
Determining the quality of the composition of the image data based on the evaluation of the arrangement;
Determining an arrangement of the extracted person and main subject that satisfies the predetermined condition;
An imaging apparatus control method for controlling an operation of the imaging unit so that image data in which the person and the main subject are arranged in a determined arrangement is generated. An imaging apparatus control method comprising: an imaging unit that captures a scene and generates image data representing the scene; and a recording unit that records the image data generated by the imaging unit on a predetermined recording medium,
By analyzing the image data generated by the imaging means, an image region representing a person is extracted,
By analyzing an image area other than the image area representing the person in the image data generated by the imaging means, a main subject other than the person is extracted,
Evaluate whether the extracted person and main subject placement meet a given condition,
Determining the quality of the composition of the image data based on the evaluation of the arrangement;
Determining an arrangement of the extracted person and main subject that satisfies the predetermined condition;
A control method for an imaging apparatus, wherein image processing is performed on the image data so that the arrangement of the person and the main subject is the determined arrangement.

Images