JP2011030089A - Image processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and program for accurately determining a user-desired subject scene from among consecutive images in reflection with the intention of a user who captures images. <P>SOLUTION: The image processing apparatus includes: a program memory 22 storing audio frequency band information indicative of audio frequency bands corresponding to kinds of a plurality of subjects; imaging and recording systems 11-13, 16, 17 for capturing a plurality of image data items with which audio data are attached; an image processing section 13 for recognizing the type of a subject captured in the image from the plurality of image data items captured; a control section 20, a main memory 21 and the program memory 22 in which audio frequency band information corresponding to the subject type recognized in the image processing section 13 is read from the program memory 22, the readout audio frequency band is set and image data are extracted with which audio data are attached for reproducing a volume for which the voice of a sound belonging to the set audio frequency band is determined equal to or higher than a specified volume among sounds reproduced from audio data of the plurality of image data items captured. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program suitable for an imaging apparatus such as a digital camera that captures a subject that shows an unpredictable movement.

2. Description of the Related Art Conventionally, there has been proposed a technique for determining a scene in which a sound volume of a predetermined level or higher is detected for a predetermined time or more as a highlight scene while shooting an image with audio data. (For example, Patent Document 1)
According to the technique of this patent document, for example, a cheer of a spectator or a loud voice of an announcer can be detected in a sports video, and it can be determined that the scene is a highlight scene.

  The above-mentioned patent document also proposes a technique for determining a scene in which a frequency similar to a predetermined audio frequency is detected for a predetermined time or longer in a captured image as a highlight scene. According to such a technique, for example, it is possible to detect a singer's treble or bass in a music video and determine that it is a highlight scene.

JP 2009-027430 A

  In the technique described in the above-mentioned patent document, the highlight scene is determined based on the time when the volume of a predetermined level or higher or the volume of a frequency band similar to a preset audio frequency is equal to or higher than the predetermined level. For this reason, it is also possible to determine that the subject is not a highlight scene and to perform corresponding processing on the sound produced by the subject that the user who is the photographer does not want in the photographed image.

  The present invention has been made in view of the above circumstances, and its purpose is to reflect the intention of the user who is the photographer and accurately determine the scene of the subject desired by the user from among the continuous shot images. An object of the present invention is to provide an image processing apparatus and a program that can be used.

  One aspect of the present invention is a first storage unit that stores audio frequency band information indicating audio frequency bands respectively corresponding to a plurality of types of subjects, and an acquisition unit that acquires a plurality of image data attached with audio data. Recognizing means for recognizing the type of the subject in each of the images represented by the plurality of image data acquired by the acquiring means, and audio frequency band information corresponding to the type of the subject recognized by the recognizing means. Among the voices reproduced from the voice data attached to the plurality of image data acquired by the acquisition means and the setting means for setting the voice frequency band indicated by the read voice frequency band information Image data with sound data that reproduces the sound volume determined to be greater than or equal to the specified sound volume in the sound frequency band set by the means. Characterized by comprising an extraction means for extracting the data.

  According to another aspect of the present invention, an acquisition unit that acquires a plurality of image data attached with audio data and a facial expression of a subject that is reflected in an image that is represented by the plurality of image data acquired by the acquisition unit are detected. Detection means, first extraction means for extracting image data in which the facial expression of the subject detected by the detection means is a preset facial expression, and an image showing the subject extracted by the first extraction means The determination unit determines whether or not the volume of the sound reproduced from the audio data attached to the data is equal to or higher than a specified volume, and the image data including the subject extracted by the first extraction unit. And a second extraction means for extracting image data accompanied by voice data capable of reproducing a sound having a sound volume equal to or higher than a prescribed volume.

  According to the present invention, it is possible to accurately determine a scene of a subject desired by a user from continuous shot images, reflecting the intention of the user who is a photographer.

1 is a block diagram showing a functional configuration of an electronic circuit of a digital camera according to a first embodiment of the present invention. The flowchart which shows the processing content at the time of the one push shot function execution based on the embodiment. 6 is a flowchart showing the processing content of a subroutine for selecting one-shot shot according to the embodiment. The figure which illustrates a part of a series of picked-up images according to the embodiment. The figure which illustrates a part of a series of picked-up images according to the embodiment. The flowchart which shows the processing content at the time of 1 push shot function execution which concerns on the 2nd Embodiment of this invention. 6 is a flowchart showing the processing content of a subroutine for selecting one-shot shot according to the embodiment. The figure which illustrates a part of a series of picked-up images according to the embodiment.

(First embodiment)
A first embodiment when the present invention is applied to a digital camera will be described below with reference to the drawings.

  FIG. 1 shows a circuit configuration of a digital camera 10 according to the present embodiment. In the figure, a light image of a subject is formed on the imaging surface of a CMOS image sensor 12 that is a solid-state imaging device by an optical lens unit 11 disposed on the front surface of the camera casing.

  In a monitor state, also referred to as through image display or live view image display, an image signal obtained by imaging with the CMOS image sensor 12 is sent to the image processing unit 13.

  The image processing unit 13 digitizes the image signal by performing correlation square sampling, automatic gain adjustment, and A / D conversion processing.

  The image processing unit 13 further performs color process processing including pixel interpolation processing and γ correction processing on the digitized signal, and temporarily stores the digitized signal in the image buffer 14 via the system bus SB. Hereinafter, the image signal held in the image buffer 14 is referred to as “image data”.

  The image buffer 14 has a capacity capable of storing image data for a certain time, for example, a total of 300 frames if the frame rate is 30 frames / second for 10 seconds so as to correspond to a circular storage operation described later. It shall be.

The image data held in the image buffer 14 is read out to the image processing unit 13 via the system bus SB, sent again to the display unit 15 via the system bus SB, and displayed as a through image.
The image processing unit 13 detects a face area included in an image represented by image data held in the image buffer 14 by a face recognition technique described later.

  Similarly to the optical lens unit 11, a microphone 16 having an acoustic directivity substantially equal to the shooting angle of view of the optical lens unit 11 is disposed on the front surface of the camera casing. Entered. The microphone 16 converts the input sound into an electrical signal and outputs it to the sound processing unit 17.

  The audio processing unit 17 converts the audio signal input from the microphone 16 into digital data when recording audio. Hereinafter, the digitized audio signal is referred to as “audio data”. The audio processing unit 17 temporarily stores the digital audio data in the audio buffer 18 formed of a ring buffer via the system bus SB.

  In addition, the audio processing unit 17 has an audio filter function corresponding to a plurality of frequency bands. The voice processing unit 17 is, for example, 80 [Hz] to 150 [Hz] corresponding to the voice of an adult male, 200 [Hz] to 350 [Hz] corresponding to the voice of an adult female, and 300 [Hz] corresponding to the voice of a child. ] To 800 [Hz]. The voice processing unit 17 can extract the voice in the frequency band corresponding to the adult male, the adult female, and the child by using the voice filter function for the voice data held in the voice buffer 18.

  Further, the audio processing unit 17 partially cuts out the audio data held in the audio buffer 18 as necessary, and compresses the data in a predetermined data file format, for example, AAC (moving picture experts group-4 Advanced Audio Coding) format. An audio data file is created and sent to a recording medium to be described later.

  In addition, the sound processing unit 17 includes a sound source circuit such as a PCM sound source, uncompresses and converts the sound data file sent during sound reproduction into an analog signal, and a speaker 19 provided on the rear side of the casing of the digital camera 10. To sound a loud sound.

The control unit 20 performs overall control of the above circuit. The control unit 20 is composed of a CPU and is directly connected to the main memory 21 and the program memory 22. The main memory 21 is composed of SDRAM (synchronous DRAM) and functions as a work memory. The program memory 22 is composed of an electrically rewritable non-volatile memory such as a flash memory, for example, and fixedly stores an operation program, data including control in a photographing mode described later.
In addition, the program memory 22 stores in advance audio frequency band information corresponding to a plurality of types of subjects, information indicating the priority order of the types of subjects, and the like.

  The control unit 20 reads out necessary programs, data, and the like from the program memory 22 and executes the control operation of the entire digital camera 10 while temporarily expanding and storing it in the main memory 21 as appropriate.

  Further, the control unit 20 executes various control operations in response to key operation signals directly input from the key input unit 23. In addition to the image processing unit 13, the image buffer 14, the display unit 15, the audio processing unit 17, and the audio buffer 18, the control unit 20 further includes a lens driving unit 24, a flash driving unit 25, and a CMOS drive via the system bus SB. Are also connected to the unit 26, the compression / decompression processing unit 27, the memory card controller 28, and the USB interface (I / F) 29.

  The key input unit 23 includes, for example, a power key, shutter key, zoom key, shooting mode key, playback mode key, menu key, cursor (“↑” “→” “↓” “←”) key, set key, playback key, and the like. Is provided.

  The lens driving unit 24 receives a control signal from the control unit 20 and controls the rotation of the lens stepping motor (M) 30, and a part of a plurality of lens groups constituting the optical lens unit 11, specifically In this case, the positions of the focus lens and the zoom lens are individually moved along the optical axis direction.

  The flash drive unit 25 receives a control signal from the control unit 20 during still image shooting, and drives the flash unit 31 including a plurality of white high-intensity LEDs to be lit in synchronization with the shooting timing.

  The CMOS drive unit 26 scans and drives the CMOS image sensor 12 in accordance with the photographing conditions set at that time.

  The compression / decompression processing unit 27 converts the image data stored in the image buffer 14 into a predetermined data file format, for example, DCT if it is a JPEG (Joint Photographic Experts Group), at the time of shooting an image accompanying the shutter key operation of the key input unit 23. Data compression processing such as (discrete cosine transform) and Huffman coding is performed to create an image data file with a greatly reduced data amount. The created image data file is recorded on the memory card 32 via the system bus SB and the memory card controller 28.

The compression / decompression processing unit 27 receives the image data read from the memory card 32 via the memory card controller 28 in the playback mode via the system bus SB, and decompresses the decompression process in the reverse order of the recording process. Thus, the original size image data is obtained and held in the image buffer 14 via the system bus SB. Then, display for reproduction is executed on the display unit 15 by the image data held in the image buffer 14.
The memory card controller 28 is connected to the memory card 32 via the card connector 33. The memory card 32 is a memory for recording image data and the like that is detachably attached to the digital camera 10 and serves as a recording medium for the digital camera 10, and has a nonvolatile memory that can be electrically rewritten in units of blocks. A flash memory and a driving circuit thereof are provided.
The USB interface 29 controls transmission / reception of data when the digital camera 10 is connected to an external device such as a personal computer via the USB connector 34.

Next, the operation of the above embodiment will be described.
In the following operation, the control unit 20 reads out the operation program and data stored in the program memory 22 when the still image is shot by the “one-shot shot” function in the shooting mode, and the main memory 21. It is executed after being expanded and memorized.

  Here, the “one-shot shot” function is a function for automatically selecting and recording the best scene from a series of continuous shot images on the digital camera 10 side.

  The operation program, fixed data, and the like stored in the program memory 22 are stored in the program memory 22 at the time of shipment from the manufacturing factory of the digital camera 10, for example, when the digital camera 10 is upgraded. Are stored by downloading a new operation program, data, and the like from the outside via the USB connector 34 and the USB interface 29 when the user connects to the personal computer.

  The shutter key that constitutes a part of the key input unit 23 has a two-step operation stroke, and enters a shooting preparation state at the first step operation stroke (hereinafter referred to as “half-press”). Is used to start detection of the face area of the subject and past shooting.

  Further, after the shutter key reaches the second stage operation stroke (hereinafter referred to as “full press”), the continuous shooting is continued until the full press state is released.

Here, the detection of a face area from an image using the face recognition technique will be briefly described.
Various methods are known as a method for detecting a face area included in an image, and any method can be adopted for the digital camera 10 according to the present embodiment.
For example, a face region may be detected by extracting a skin color region from an input image, as in the method described in Japanese Patent Application Laid-Open No. 2000-105819, or Japanese Patent Application Laid-Open No. 2006-21111 or Japanese Patent Application Laid-Open No. 2006-072770. The face area may be detected using the technique described in the Japanese Patent Publication.

Typically, for example, the image of the region of interest set in the image is compared with a predetermined template face image to determine the similarity of both images, and the face is included in the region of interest based on the similarity Whether or not the region of interest is a face region is detected.
The similarity is determined by extracting a feature amount effective for identifying whether the face is a face. The feature amount includes a horizontal edge, a vertical edge, a right diagonal edge, a left diagonal edge, and the like.

The region of interest in the image is shifted in the horizontal direction or the vertical direction by one pixel. Then, the image in the region of interest after the shift is compared with the template face image, the similarity between both images is determined again, and the same detection is performed.
In this way, the region of interest is updated and set while being shifted pixel by pixel from the upper left to the lower right in the image, for example. Further, the image is reduced at a certain rate, and the face area detection process is similarly performed on the reduced image.
By repeating such processing, it is possible to detect a face region of an arbitrary size from the image.
In the digital camera 10 of this embodiment, the image processing unit 13 performs the face area detection process using the image data held in the image buffer 14 under the control of the control unit 20.

  FIG. 2 shows the processing contents of image data handling when the one-press shot function is selected in which continuous shooting is performed including the image before the shutter key operation, and the still image with the highest evaluation is automatically selected and recorded. .

  Initially, the control unit 20 starts a monitoring operation, also referred to as through image display, temporarily stores image data captured by the CMOS image sensor 12 in the image buffer 14, and displays the display unit 15 based on the stored image data. By performing the display at, the content captured by the CMOS image sensor 12 is displayed on the display unit 15 in real time (step S101).

While maintaining this through image display state, the control unit 20 also determines whether or not the shutter key of the key input unit 23 is half-pressed (step S102).
Here, if the shutter key is not half-pressed, the process returns to step S101 to wait for the shutter key to be half-pressed while continuing through image display.

  When the shutter key is half-pressed, the control unit 20 determines that in step S102, and starts past photographing that continuously acquires still images with sound for a predetermined period of time (step S103). .

  In this past shooting, image data for a maximum of 60 frames in total for 2 seconds at a preset frame rate and time, for example, 30 [frames / second], and audio data attached thereto are acquired as needed. By keeping the sound buffer 18 in a circular manner, the latest image with sound for a maximum of 2 seconds is always kept.

At the time of past photographing, detection of a face area is performed on the latest image data from the image expressed by the image data by the face recognition technique as described above.
Depending on the setting in advance, the detection result of the face area may be notified to the user of the digital camera 10 by displaying a frame indicating the face area superimposed on the image displayed on the display unit 15.

  Simultaneously with the execution of past photography with sound, the control unit 20 continuously determines whether or not the shutter key is fully pressed (step S104) and whether or not the shutter key is half-pressed (step S105). to decide.

  By repeatedly executing the processing of steps S103 to S105 in this way, the control unit 20 waits for the shutter key to be fully pressed or the half-pressed operation to be released while continuing the circular recording by past shooting. .

  When the half-press operation of the shutter key is released in the past shooting state, the control unit 20 determines that in step S105, and the image data and audio buffer held in the image buffer 14 in the past shooting so far. After the audio data held in 18 is erased in a batch (step S106), the process returns to step S103.

  If it is determined in step S104 that the shutter key has been fully pressed, the control unit 20 starts acquiring a still image with sound as the main shooting following the past shooting so far (step S107).

  In this actual photographing, the control unit 20 acquires image data and accompanying audio data at a preset frame rate, for example, 30 [frames / second] as needed, and stores them in the image buffer 14 and the audio buffer 18.

  During the execution of the main photographing with sound, the control unit 20 either releases the shutter key full-press operation or whether a certain time, for example, 8 seconds has elapsed since the full-press operation was started. It is determined whether or not the state becomes (step S108).

  If it is determined that neither of these states is present, the process returns to step S107, and the shutter key full-press operation is canceled while the main photographing operation with sound is executed, or for a certain period of time. Wait for it to elapse.

  If it is determined that the shutter key full-press operation is released or a certain time has elapsed, the control unit 20 determines that in step S108, and stops the actual photographing operation at that time (step S109).

  Next, the control unit 20 recognizes the type of the subject from the series of image data held in the image buffer 14 based on the series of audio data held in the audio buffer 18 by past imaging and main imaging. Data of a one-shot shot image in which a high-priority type subject is reflected is selected (step S110).

FIG. 3 shows the processing contents of a subroutine related to selection of a one-shot shot image in consideration of the type of subject.
At the beginning, the control unit 20 causes the image processing unit 13 to execute face recognition processing for the image data corresponding to the first image located at the head of the main photographing held in the image buffer 14, and is expressed by the image data. The number of faces of the subject in the image to be displayed and the type of each face, specifically, adult male, adult female, and child are identified (step S301).

  A technique for identifying the gender and age group of a person from a face area reflected in an image represented by image data is known, including the technique described in Japanese Patent Laid-Open No. 11-175724, for example. .

  For example, in Japanese Patent Application Laid-Open No. 11-175724, a large number of learning face images are grouped into genders and age groups, the attribute learning unit stores the grouped learning face images in association with each group, and for each attribute. A set of grouped learning face image vectors is expanded into a group-specific subspace to generate an eigenvector for each group, and the eigenvector for each group generated by this subspace generator is unknown. A determination storage unit that stores data for performing group determination of the input face image, and determines the similarity between the unknown input face image and each group with reference to the data in the determination storage unit, and determination The attribute of the input face image is identified from the stored group and the content stored in the attribute storage unit.

  Based on the processing result of step S301, the control unit 20 first determines whether or not at least one person's face area has been detected by the image processing unit 13 using the face recognition technique (step S302).

  If no person's face area can be detected, image data corresponding to the type of subject cannot be selected, so the audio data held in the audio buffer 18 is used. Instead, only the series of image data stored in the image buffer 14 and the image processing unit 13 selects the image with the least subject blur in the image (step S303), and the subroutine of FIG. To do.

  Regarding the selection of the image data, the image processing unit 13 calculates an evaluation value for each image, for example, by extracting edge components from the entire image, and the image with the highest evaluation value is the image with the least blurring. As an object, corresponding image data is selected.

  If the control unit 20 determines in step S302 that the image processing unit 13 has detected an image of at least one person's face, there are a plurality of types of subjects that can be detected next. Whether or not (step S304).

  If it is determined that the number of types of the face of the subject detected here is not plural but one, the audio frequency band is read from the program memory 22 and set in accordance with the recognized type (step). S305).

FIG. 4A illustrates an operation screen on the display unit 15 in the case where setting is performed using the sound from the subject as a trigger when the one-press shot function is selected. Here, regarding the setting that triggers the sound from the subject side,
"OFF" (sound is not triggered)
"ON" (sound as a trigger)
"During face recognition only (sound as a trigger)"
The state where “only at the time of face recognition (using voice as a trigger)” is selected from the three options is shown.

  When the setting shown in FIG. 4A is performed, a part of an image displayed on the display unit 15 at the time of actual photographing is illustrated in FIG. 4B. In FIG. 5B, the face recognition process for the subject T1 in the image is executed by the image processing unit 13, and the detected face area FA is displayed in the image so that the face area of the subject T1 is detected. That the user of the digital camera 10 is informed.

  When it is recognized in step S305 that the type of the subject is “child”, for example, the control unit 20 reads out the audio frequency band information from the program memory 22, thereby enabling the audio frequency band of 300 [Hz] to 800 [Hz]. Set.

  In addition, when it is determined that there are a plurality of types of face of the subject that can be detected in the above step S304, the control unit 20 sets various priority orders stored in the program memory 22 for the plurality of types that can be detected next. Based on the highest type, the audio frequency band information is read from the program memory 22 and set (step S306).

  For example, as shown in FIG. 5A, two subjects T2 and T3 are detected from the first image of the main image, and as a result of the face recognition, the subject T2 is of the type “adult male” and the subject T3 is of the type “ Suppose you are an “adult woman”.

In this digital camera 10, the priority order of types is determined by information stored in advance in the program memory 22.
“Children”> “Adult Women”> “Adult Men”
It is assumed that the type “child” is set at the top and the type “adult male” is set at the bottom.

  In the image data showing the subjects T2 and T3 shown in FIG. 5A, the control unit 20 selects the type “adult female” on the subject T3 side as the higher priority. According to the selection result, in step S306, the control unit 20 reads out the audio frequency band information corresponding to the subject type “adult female” from the program memory 22, and sets the audio frequency band of 200 [Hz] to 350 [Hz]. To do.

  After the audio frequency band is set in step S305 or step S306, the audio data subjected to the past shooting and the main shooting stored in the audio buffer 18 according to the set audio frequency band is subjected to the audio filter process, and the processed audio is processed. The sound volume level reproduced by the data is detected (step S308).

  As a result of the detection, audio data having the highest volume level is selected, and image data corresponding to the audio data is selected from the image buffer 14 (step S308).

For the series of image data shown in FIG. 5A as the first image at the time of photographing the main image, the image data as shown in FIG. Data shall be selected.
When the image data is selected, the subroutine shown in FIG. 3 is terminated, and the process returns to the main routine shown in FIG.

In the main routine of FIG. 2, the selected image data is read from the image buffer 14 and compressed by the compression / decompression processing unit 27 in a predetermined data file format to create an image data file, which is then recorded via the memory card controller 28. Recording is performed on the memory card 32 as a medium (step S111).
As described above, the photographing by the one-press shot function based mainly on the audio data by the control unit 20 is completed, and the process returns to the processing from step S101 in preparation for the next photographing.

  As described above, according to the present embodiment, it is possible to reflect the intention of the user who is the photographer, and to accurately determine and record the scene of the subject desired by the user from the continuous shot images based on the displacement of the volume level.

  In particular, according to the present embodiment, one of a plurality of types of subjects is selected, and the volume level is detected based on the audio frequency band corresponding to the selected type of subjects. A more accurate scene can be determined without being confused by the sound associated with the subject.

  In this case, in the present embodiment, since the type of the subject to be processed is automatically selected according to a preset priority order, the highest priority order can be obtained without requiring the user to perform complicated operations. Since the process moves to the process of selecting one image data from a series of image data after selecting a subject of a type that seems to be high, the user's labor can be simplified and photographing can be performed easily.

Although not described in the above embodiment, the user manually selects the subject type by operating the key input unit 23, and the control unit 20 that receives the selection manually stores the selection information in the program memory 22. You may be able to.
In this case, the control unit 20 determines the priority order of the subject type according to the user selection information stored in the main memory 21, so that it is possible to select appropriate image data that reliably reflects the user's intention to shoot. It becomes like this.

  Moreover, in the said embodiment, the audio | voice data with the highest audio | voice volume level among audio | voice data corresponding to image data shall be selected, and the corresponding image data shall be selected. This further simplifies the selection criteria for image data, reduces the burden on the apparatus side, and allows images to be selected quickly.

  Further, although not shown in the above embodiment, regarding the audio data corresponding to the image, the user arbitrarily sets the volume level by operating the key input unit 23 in advance of shooting, and the control unit 20 receives and sets this operation. The recorded volume level is stored in the program memory 22, and if the control unit 20 has reached or exceeded the volume level stored in the program memory 22 during actual shooting, the corresponding image data is recorded in the memory card 32 as a recording medium. It is good also as what to do.

  As described above, by allowing the user to arbitrarily set the reference volume level, it is possible to select one of a series of image data, particularly when there is a free space in the memory card 32 as a recording medium. More image data can be automatically recorded and left depending not only on the number of images but also on the volume level according to the user's setting.

  In the above embodiment, adult males, adult females, and children are recognized as face types by the face recognition technology, and the type is selected according to a preset priority order. With regard to the priority order of these types, the user can arbitrarily set in advance according to the shooting intention and store it in the program memory 22, thereby simplifying the user's trouble of selecting the main subject during shooting, Image data can be recorded by accurately determining the main subject.

(Second Embodiment)
A second embodiment when the present invention is applied to a digital camera will be described below with reference to the drawings.

  Note that the circuit configuration of the digital camera 10 'according to the present embodiment is basically the same as the contents shown in FIG. 1, and the same reference numerals are used for the same parts to illustrate and explain the same. Is omitted.

Next, the operation of the above embodiment will be described.
In the following operation, the control unit 20 reads out the operation program and data stored in the program memory 22 when the still image is shot by the “one-shot shot” function in the shooting mode, and the main memory 21. It is executed after being expanded and memorized.

  Here, the “one-shot shot” function is a function that automatically selects and records the best scene from a series of continuous shot images on the digital camera 10 side.

  The operation program stored in the program memory 22 is stored in the program memory 22 at the time of shipment of the digital camera 10 ′, for example, when the digital camera 10 ′ is upgraded. Also included are those in which new operation programs, data, and the like are downloaded and stored from the outside via the USB connector 34 and the USB interface 29 when the user connects to the personal computer.

  The shutter key that constitutes a part of the key input unit 23 has a two-step operation stroke, and enters a shooting preparation state at the first step operation stroke (hereinafter referred to as “half-press”). Is used to start detection of the face area of the subject and past shooting.

  Further, after the shutter key reaches the second stage operation stroke (hereinafter referred to as “full press”), the continuous shooting is continued until the full press state is released.

  Here, a technique for detecting a face area from an image using the face recognition technique is the same as that in the first embodiment, and a description thereof will be omitted.

  In the digital camera 10 ′ of the present embodiment, the image processing unit 13 executes face area detection processing using image data held in the image buffer 14 under the control of the control unit 20.

  FIG. 6 shows the processing contents of image data handling when the one-shot shot function is selected in which continuous shooting is performed including the image before the shutter key operation, and the still image having the highest evaluation is automatically selected and recorded. .

  Initially, the control unit 20 starts a monitoring operation, also referred to as through image display, temporarily stores image data captured by the CMOS image sensor 12 in the image buffer 14, and displays the display unit based on the stored image data. By performing the display at 15, the content captured by the CMOS image sensor 12 is displayed on the display unit 15 in real time (step S501).

While continuing this through image display state, the control unit 20 also determines whether or not the shutter key of the key input unit 23 has been half-pressed (step S502).
Here, if the shutter key is not half-pressed, the control unit 20 returns to the processing from step S501 and waits for the shutter key to be half-pressed while continuing through image display.

  When the shutter key is half-pressed, the control unit 20 determines that in step S502, and starts past imaging in which a still image with sound is continuously acquired cyclically for a predetermined time (step S503). .

  In this past shooting, image data for a maximum of 60 frames in total for 2 seconds at a preset frame rate and time, for example, 30 [frames / second], and audio data attached thereto are acquired as needed. By keeping the sound buffer 18 in a circular manner, the latest image with sound for a maximum of 2 seconds is always kept.

At the time of past photographing, detection of a face area is performed on the latest image data from the image expressed by the image data by the face recognition technique as described in detail in the first embodiment.
Depending on the setting in advance, the detection result of the face area may be notified to the user of the digital camera 10 by displaying a frame indicating the face area superimposed on the image displayed on the display unit 15.

  Simultaneously with the execution of past photography with sound, the control unit 20 continuously determines whether or not the shutter key is fully pressed (step S504) and whether or not the shutter key is half-pressed (step S505). to decide.

  By repeatedly executing the processing of steps S503 to S505 in this way, the control unit 20 waits for the shutter key to be fully pressed or the half-pressed operation to be released while continuing the circular recording by past shooting. .

  When the half-press operation of the shutter key is released in the past shooting state, the control unit 20 determines that in step S505, and the image data and audio buffer held in the image buffer 14 in the previous bust shooting. After the audio data held in 18 is erased at once (step S506), the process returns to step S503.

  If it is determined in step S504 that the shutter key has been fully pressed, the control unit 20 starts acquiring a still image with sound as the main shooting following the past shooting (step S507).

  In this actual photographing, the control unit 20 acquires image data and accompanying audio data at a preset frame rate, for example, 30 [frames / second] as needed, and stores them in the image buffer 14 and the audio buffer 18.

  During the execution of the main photographing with sound, the control unit 20 either releases the shutter key full-press operation or whether a certain time, for example, 8 seconds has elapsed since the full-press operation was started. It is determined whether or not the state becomes (step S508).

  If it is determined that neither state is found, the process returns to step S507, and the shutter key full-press operation is canceled or a certain period of time is executed while performing the main photographing operation with sound. Wait for it to elapse.

  If it is determined that the shutter key full-press operation is released or a certain time has elapsed, the control unit 20 determines that in step S508 and stops the actual photographing operation at that time (step S509).

  Next, the control unit 20 recognizes the facial expression of the subject from the series of image data held in the image buffer 14 based on the series of audio data held in the audio buffer 18 by past imaging and main imaging. A one-shot shot image is selected (step S510).

FIG. 7 shows the processing contents of a subroutine related to selection of a one-shot shot image taking into account the expression of the subject.
At the beginning, the control unit 20 causes the image processing unit 13 to execute face recognition processing for the image data corresponding to the first image located at the head of the main photographing held in the image buffer 14, and is expressed by the image data. The face of the subject that is reflected in the image to be identified is identified (step S701).

Based on the processing result of step S701, it is first determined whether or not at least one face portion of a person has been detected by the face recognition technique (step S702).
Since the face recognition technique is a well-known technique as described in the first embodiment, the detailed description thereof is omitted.

  Here, when no person's face can be detected, it is not possible to select an image according to the facial expression of the subject. Therefore, without using the audio data held in the audio buffer 18, Only the series of image data stored in the image buffer 14 is selected by the image processing unit 13 with the smallest subject blur in the image (step S703), and the subroutine of FIG.

  Regarding the selection of the image data, the image processing unit 13 calculates an evaluation value for each image, for example, by extracting edge components from the entire image, and the image with the highest evaluation value is the image with the least blurring. As an object, corresponding image data is selected.

  If it is determined in step S702 that the control unit 20 has detected an image of at least one person's face by the image processing unit 13, then whether or not there are a plurality of detected subject faces is determined. Is determined (step S704).

  Here, only when it is determined that there are a plurality of detected faces of the subject, the face having the largest face area is selected from the plurality of faces (step S705).

  Thereafter, with respect to the selected face area, the smile degree is calculated by the image processing unit 13 for each same face area reflected in each image represented by a series of image data, and the image is equal to or higher than a preset smile degree. Data is searched (step S706).

Various methods are known as a method for calculating the smile level, and any method can be adopted for the digital camera according to the present embodiment.
For example, a technique described in Japanese Patent Application Laid-Open No. 2004-046591 can be adopted as one technique for calculating the smile level. When this method is employed, the image processing unit 13 under the control of the control unit 20 refers to each of the eyebrows, pupils, and lips that constitute the face image with reference to a smile evaluation criterion table stored in advance in the program memory 22. The evaluation points of the element shape are calculated, and the results obtained by weighting each evaluation point by a predetermined coefficient are added together to obtain the smile level of the face image shown in the image.

  Next, based on each image data obtained as a result of the search and having a smile level equal to or higher than a preset smile level, a sound volume level reproduced by each corresponding sound data held in the image buffer 14 is detected ( Step S707).

  As a result of the detection, audio data having the highest volume level is selected, and image data corresponding to the audio data is selected from the image buffer 14 (step S708).

  8A to 8D show smiles preset by the image processing unit 13 from the series of image data held in the image buffer 14 under the control of the control unit 20 by the processing in step S706. An example of the result of searching and extracting image data of a degree or more is illustrated.

  As a result of detecting the image data searched in this way with the highest volume level of the audio data, the control unit 20 detects the image data shown in FIG. 8C as a detection result as shown in FIG. 8E. Is obtained.

When the image data is selected, the subroutine shown in FIG. 7 is terminated and the process returns to the main routine shown in FIG.
In the main routine of FIG. 6, the image data selected by the control unit 20 is read from the image buffer 14 and compressed in a predetermined data file format by the compression / decompression processing unit 27 to create an image data file. Through the memory card 32 (step S511).
As described above, photographing by the one-press shot function based mainly on the audio data by the control unit 20 is completed, and the processing returns to the processing from step S501 in preparation for the next photographing.

  As described above, according to the present embodiment, a scene that reflects the intention of the user who is a photographer and is rich in changes in the facial expression of the subject desired by the user is determined more accurately together with the detection of the volume level. Can be recorded.

  In the above-described embodiment, the face recognition technology is used and the smile level of the subject is detected as a preset expression. However, the present invention is not limited to this. For example, the degree of crying face and angry face is quantified. It may be set.

  As described above, when the user selects a facial expression that can be taken by the user in advance and the degree thereof, the control unit 20 accepts this and stores the selected contents in the program memory 22, so that the control unit 20 at the time of actual photographing can be used. The setting information stored in the program memory 22 can be read, the user's intention of photographing can be judged more accurately, and the most suitable image can be selected from a series of image data and recorded.

  The first and second embodiments have been described with reference to a case where the present invention is applied to a digital camera that captures an image. However, the present invention is not limited to this, and a digital video movie that can also capture still images. The present invention can be similarly applied to a camera, a mobile phone terminal with a camera function, and other various imaging devices, or a digital photo data storage device that records a series of image data with audio data without having a shooting function.

  In addition, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention in the implementation stage. Further, the functions executed in the above-described embodiments may be combined as appropriate as possible. The above-described embodiment includes various stages, and various inventions can be extracted by an appropriate combination of a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect is obtained, a configuration from which the constituent requirements are deleted can be extracted as an invention.

  DESCRIPTION OF SYMBOLS 10,10 '... Digital camera, 11 ... Optical lens unit, 12 ... CMOS image sensor, 13 ... Image processing part, 14 ... Image buffer, 15 ... Display part, 16 ... Microphone, 17 ... Sound processing part, 18 ... Sound buffer , 19 ... Speaker, 20 ... Control unit, 21 ... Main memory, 22 ... Program memory, 23 ... Key input unit, 24 ... Lens drive unit, 25 ... Flash drive unit, 26 ... CMOS drive unit, 27 ... Compression / decompression processing unit 28 ... Memory card controller, 29 ... USB interface, 30 ... Lens stepping motor (M), 31 ... Flash unit, 32 ... Memory card, 33 ... Card connector, 34 ... USB connector, SB ... System bus.

Claims (11)

First storage means for storing audio frequency band information indicating audio frequency bands respectively corresponding to a plurality of types of subjects;
Acquisition means for acquiring a plurality of image data accompanied by audio data;
Recognizing means for recognizing the type of subject appearing in each of the images represented by the plurality of image data acquired by the acquiring means;
Setting means for reading audio frequency band information corresponding to the type of subject recognized by the recognition means from the first storage means and setting the audio frequency band indicated by the read audio frequency band information;
Of the sound reproduced from the audio data attached to the plurality of image data acquired by the acquisition means, the volume determined to be equal to or higher than a prescribed volume of the sound belonging to the audio frequency band set by the setting means An image processing apparatus comprising: extraction means for extracting image data accompanied with sound data to be reproduced. A selecting means for selecting one of the recognized object types when the recognizing means recognizes the types of the plurality of objects;
The setting means reads the audio frequency band information corresponding to one type selected by the selection means from the plurality of object types recognized by the recognition means from the first storage means, and reads the read audio frequency band The image processing apparatus according to claim 1, wherein an audio frequency band indicated by the information is set. A second storage means for storing priorities respectively corresponding to the types of the plurality of subjects;
When the recognition unit recognizes a plurality of different subject types, the selection unit corresponds to the highest priority order among the recognized different subject types in the priority order stored in the second storage unit. The image processing apparatus according to claim 2, wherein one type is selected. An instruction means for instructing any one of a plurality of different subject types;
The selecting means, when the recognition means recognizes a plurality of different subject types, selects any one of the recognized different subject types designated by the instruction means. The image processing apparatus according to claim 2.   The image processing apparatus according to claim 1, further comprising a determination unit that arbitrarily determines a prescribed volume that is a determination criterion of the extraction unit.   2. The image processing apparatus according to claim 1, wherein the extraction unit extracts image data accompanied by audio data having a maximum volume of audio belonging to the audio frequency band set by the setting unit.   The image processing apparatus according to claim 1, wherein the plurality of types of subjects include at least one of a male, a female, an adult, and a child. Acquisition means for acquiring a plurality of image data accompanied by audio data;
Detecting means for detecting facial expressions of the subject in the images respectively represented by the plurality of image data acquired by the acquiring means;
First extraction means for extracting image data in which the facial expression of the subject detected by the detection means is a preset facial expression;
Determining means for determining whether or not the volume of the sound reproduced from the audio data attached to the image data in which the subject extracted by the first extracting means is greater than or equal to a predetermined volume;
Second image extracting means for extracting image data accompanied by audio data capable of reproducing sound of a sound volume higher than a specified volume by the determining means from image data showing the subject extracted by the first extracting means; An image processing apparatus comprising:   9. The image processing apparatus according to claim 8, further comprising selection means for arbitrarily selecting a preset facial expression from a plurality of candidates as a reference for extraction by the first extraction means. A program executed by a computer incorporated in an image processing apparatus, the computer being
First storage means for storing audio frequency band information indicating audio frequency bands respectively corresponding to a plurality of types of subjects;
Acquisition means for acquiring a plurality of image data accompanied by audio data;
Recognizing means for recognizing the type of subject in each of the images represented by the plurality of image data acquired by the acquiring means;
The setting means for reading the audio frequency band information corresponding to the type of the subject recognized by the recognition means from the content stored in the first storage means, and setting the audio frequency band indicated by the read audio frequency band information, and the acquisition Of the sound reproduced from the audio data attached to the plurality of image data acquired by the means, the sound volume determined to be equal to or higher than the prescribed sound volume belonging to the sound frequency band set by the setting means is reproduced. A computer-executable program that functions as an extraction unit that extracts image data accompanied by audio data. A program executed by a computer incorporated in an image processing apparatus, the computer being
Acquisition means for acquiring a plurality of image data accompanied by audio data;
Recognizing means for recognizing the facial expression of the subject in the images respectively represented by the plurality of image data acquired by the acquiring means;
First extraction means for extracting image data in which the facial expression of the subject recognized by the recognition means is a preset facial expression;
Determining means for determining whether or not the volume of the sound reproduced from the audio data attached to the image data including the subject extracted by the first extracting means is equal to or higher than a prescribed volume; and the first extraction And functioning as second extraction means for extracting image data attached with sound data capable of reproducing sound of a sound level equal to or higher than a prescribed volume by the determination means from image data showing the subject extracted by the means. A program executable by the computer.

Images