JP2010147587A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To represent a feeling obtained from hearing during photographing in a photographed image. <P>SOLUTION: The imaging device is provided with a microphone for acquiring sound information, a sound analyzing circuit for analyzing a feature amount of a sound signal, and a system control means for controlling an image processing parameter and a photographing parameter on the basis of the feature amount of the sound signal acquired by the microphone, wherein the imaging device can represent a feeling obtained from hearing during photographing and can generate a photographed image having presence. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus and an image pickup method, and more particularly to a technique suitable for use in taking an image reflecting an atmosphere and a sense of reality at the time of shooting.

  Conventionally, an emotion detection sensor for detecting the photographer's body temperature, sweating, grip pressure, etc., and the position / external temperature / external humidity at the time of photographing for the purpose of reflecting the photographer's emotion at the time of photographing in the photographed image. Has an environment detection sensor for detecting Patent Document 1 discloses a technique for changing an image processing parameter of an acquired image in accordance with a detection value of each detection sensor.

  In addition, in order to make the “atmosphere at the time of photographing” feel from a photographed image, the volume and inflection of the inputted voice are analyzed in a camera capable of voice input. Patent Document 2 discloses a technique for converting a character image into an optimal format according to the analysis result and acquiring a photographed image synthesized with a subject image.

Japanese Patent Laid-Open No. 2005-250977 JP 2003-348411 A

  In general, an imaging apparatus converts an optical image formed through an imaging optical system into an electrical signal to acquire image information, and records the information on a recording medium or displays it on a display device. . In other words, it is a device that can record and store the photographer's visual information.

  However, in reality, the photographer stores not only visual information but also information obtained from other sensory organs such as auditory and tactile sensation, and stores and stores the “video” at the time of photographing. For this reason, when the photographed image is looked back after a lapse of time after photographing, it may feel different from the “video” stored in its own memory.

  On the other hand, in the technique disclosed in Patent Document 1, a tactile sensation at the time of shooting, in particular, temperature sensation / humidity can be detected and reflected in an image by an outside air temperature / outside humidity sensor. A certain image can be acquired. However, the auditory information is not touched.

  On the other hand, in the technique disclosed in Patent Document 2, the atmosphere at the time of photographing can be produced by synthesizing auditory information with a photographed image in the form of characters. However, there is a problem that the form of expressing in characters is slightly different from the original shooting purpose of recording a realistic video.

  In view of the above-described problems, an object of the present invention is to make it possible to express a feeling obtained from hearing at the time of shooting in a shot image.

  An imaging apparatus according to the present invention includes an imaging unit that captures a subject and generates still image data or moving image data, and an image processing unit that performs predetermined image processing on the still image data or moving image data generated by the imaging unit. Voice input means for acquiring a voice signal when the imaging means captures the subject, voice analysis processing means for analyzing a feature amount of the voice signal acquired by the voice input means, and the image processing means System control means for controlling an image processing parameter used for performing predetermined image processing on still image data or moving image data based on a feature amount of an audio signal analyzed by the audio analysis processing means. And

  The imaging method of the present invention includes an imaging step of capturing a subject to generate still image data or moving image data, and an image processing step of performing predetermined image processing on the still image data or moving image data generated in the imaging step. A voice input step of acquiring a voice signal when photographing the subject in the imaging step, a voice analysis processing step of analyzing a feature amount of the voice signal acquired in the voice input step, and the image processing step. A system control step of controlling image processing parameters used for performing predetermined image processing on still image data or moving image data based on a feature amount of an audio signal analyzed in the audio analysis processing step. And

  The computer program of the present invention includes an imaging process for capturing a subject to generate still image data or moving image data, and an image processing process for performing predetermined image processing on the still image data or moving image data generated in the imaging process. A voice input step of acquiring a voice signal when photographing the subject in the imaging step, a voice analysis processing step of analyzing a feature amount of the voice signal acquired in the voice input step, and the image processing step. A system control step of controlling an image processing parameter used for performing predetermined image processing on still image data or moving image data based on a feature amount of an audio signal analyzed in the audio analysis processing step; The computer is executed.

  According to the present invention, it is possible to express a feeling obtained from hearing at the time of photographing in a photographed image, and a photographed image with a more realistic feeling can be obtained.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to the first exemplary embodiment of the present invention.
In FIG. 1, reference numeral 100 denotes an imaging device. Reference numeral 121 denotes a lens unit 200 that converts an optical image of a subject (not shown) from an imaging lens 210 through an imaging optical system including an aperture 211, lens mounts 102 and 202, and a shutter 144 into an electrical signal. This is an image sensor to be converted.

  Reference numeral 122 denotes an A / D converter that converts an analog signal output output from the image sensor 121 into a digital signal and generates digital still image data or moving image data. The digital signal A / D converted by the A / D converter 122 is controlled by the memory controller 124 and the system controller 120 and stored in the memory 127.

  An image processing unit 123 performs predetermined pixel interpolation processing and color conversion processing on digital signal data or data read from the memory control unit 124. The image processing unit 123 also includes a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like.

  It is also possible to read an image stored in the memory 127, perform compression processing or decompression processing, and write the processed data to the memory 127. In the moving image shooting mode, a series of moving image data captured in the memory 127 can be compressed by the MPEG method or the like at the time of recording.

  Further, the image processing unit 123 can perform predetermined arithmetic processing using the captured image data and perform image processing such as contrast, brightness, saturation, white balance, and sharpness. It also performs TTL (through the lens) AWB (auto white balance) processing.

  A memory control unit 124 controls transmission / reception of data between the A / D conversion unit 122, the image processing unit 123, the liquid crystal panel display unit 125, the external removable memory unit 131, and the memory 127. Data digitized by the A / D conversion unit 122 is written into the memory 127 via the image processing unit 123 and the memory control unit 124. Alternatively, digital data output from the A / D converter 122 is directly written in the memory 127.

  A liquid crystal display type display device 110 includes a liquid crystal panel display unit 125 and a backlight illumination unit 126. The liquid crystal panel display unit 125 can display a menu screen written in the display data storage area of the memory 127 or an image file stored in the external removable memory unit 131.

  The backlight illumination unit 126 is configured by using an LED, an organic EL, a fluorescent tube, or the like as a light source element that irradiates the back surface of the liquid crystal panel display unit 125, and arbitrarily turns on or off illumination according to an instruction from the system control unit 120. Is possible. Further, the system control unit 120 has a dimming function for adjusting the display luminance of the display device 110 by limiting the energization current of the light source element by either the voltage driving method or the PWM driving method.

  Reference numeral 127 denotes a memory for storing captured still images and moving images, images for reproduction display, and audio files, and has a sufficient storage capacity for storing a predetermined number of still images and moving images. . The memory 127 can also be used as a work area for the system control unit 120.

  Reference numeral 128 denotes an electrically erasable / recordable nonvolatile memory such as a flash memory or an EEPROM. Further, the nonvolatile memory 128 stores a shooting state storage and a program for controlling the imaging apparatus 100.

  Reference numeral 130 denotes an eyepiece detection unit, which is composed of an infrared light emitter and a light receiving circuit. The infrared light is emitted from the infrared light emitter at regular intervals, and the infrared light reflected by the detection object is received by the light receiving circuit. It receives light and detects whether there is an object to be detected at a specified position based on the received light quantity.

  Reference numeral 131 denotes an external detachable memory unit for recording and reading image files on a recording medium such as a compact flash (registered trademark) or an SD card. A power supply unit 138 includes a battery, a battery detection circuit, a DC / DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level. Further, the DC / DC converter is controlled based on the detection result and an instruction from the system control unit 120, and a necessary voltage is supplied to each block unit for a necessary period.

  The system control unit 120 performs power management control for power saving by stopping power supply to unnecessary blocks in accordance with the operation state of the imaging apparatus 100. As an example, in the case of image reproduction display or menu screen display, since the camera control block does not need to be operated, to the camera control unit 140, photometry unit 142, distance measurement unit 143, image sensor 121, lens unit 200, and strobe unit 300. Shut off the power supply.

  Reference numeral 141 denotes a shutter control unit that controls the shutter 144 in conjunction with the lens control unit 203 that controls the aperture 211 based on photometric information from the photometry unit 142. Reference numeral 142 denotes a photometric unit for performing AE (automatic exposure) processing. In this embodiment, the photometric unit 142 is also used to measure the brightness of the peripheral portion of the subject.

  That is, the light beam incident on the photographing lens 210 is incident on the photometry unit 142 via the stop 211, the lens mounts 202 and 102, and a photometric lens (not shown). Thereby, the exposure state of the subject formed as an optical image can be measured. The photometry unit 142 also has an EF (flash dimming) processing function in cooperation with the flash unit 300. The flash unit 300 also has an AF auxiliary light projecting function and a flash light control function.

  Reference numeral 143 denotes a distance measuring unit for performing AF (autofocus) processing. The light beam incident on the photographing lens 210 is measured through the aperture 211, the lens mounts 202 and 102, and a distance measuring mirror (not shown). The light is incident on the portion 143. Thereby, the focus state of the image formed as an optical image can be measured. As described above, since the distance measurement unit 143 and the photometry unit 142 are provided exclusively, the AF (auto focus) process, the AE (automatic exposure) process, and the EF (flash adjustment) are performed using the distance measurement unit 143 and the photometry unit 142. It is configured to perform each processing of (light) processing.

  A camera control unit 140 controls a series of operations as a camera through transmission / reception communication with the shutter control unit 141, the photometry unit 142, and the distance measurement unit 143. The camera control unit 140 can also control the lens unit 200 and the strobe unit 300.

  Reference numerals 132, 133, 134, 135, 136, and 137 constitute an operation unit for inputting various operation instructions of the system control unit 120. It is composed of a single or a combination of switches, dials, touch panels, pointing by line-of-sight detection, voice recognition devices, etc. constituting the operation unit.

Here, a specific description of the operation tool constituting the operation unit will be given.
The reproduction switch 132 is a switch for performing a reproduction display mode operation for displaying predetermined image data on the liquid crystal panel display unit 125. When the image file stored in the external detachable memory unit 131 is reproduced and displayed, the reproduction switch 132 must be operated. If this operation has already been performed in the playback display mode, the playback display mode can be switched to the shooting mode.

  The menu switch 133 is a switch for displaying a list of various items on the liquid crystal panel display unit 125. The display contents include a state setting related to shooting, a recording medium format, a clock setting, a development parameter setting, and a user function setting (custom function setting).

  The mode dial 134 is a switch for switching and setting various function photographing modes. In this embodiment, automatic shooting mode, program shooting mode, shutter speed priority shooting mode, aperture priority shooting mode, manual shooting mode, portrait shooting mode, landscape shooting mode, sports shooting mode, night scene shooting mode, video mode, etc. Can be set.

  The release switch 135 is a switch that is turned on when the release button is half-pressed (SW1) and fully pressed (SW2). In the half-pressed state, the start of operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash dimming) processing is instructed.

  In the fully-pressed state, an analog signal read from the image sensor 121 is converted into digital image data by the A / D converter 122, and imaging processing is performed in which the analog signal is written to the memory 127 according to the control of the memory controller 124. Further, development processing is performed so that the image processing unit 123 and the memory control unit 124 perform calculation so that the image can be viewed. Further, a series of processing operations of reading image data from the memory 127, further compressing the image data by the image processing unit 123, and performing a recording process of writing the image data to a recording medium (not shown) mounted in the external removable memory unit 131. Instruct the start.

  An operation unit 136 including various button switches can perform shooting mode, continuous shooting mode, set, macro, page feed, flash setting, menu movement, white balance selection, shooting image quality selection, exposure correction, date / time setting, and the like. it can. Further, there are a moving image shooting switch for starting and stopping moving image shooting, an up / down / left / right direction switch, a zoom magnification change switch for a reproduced image, and an image display ON / OFF switch of the liquid crystal panel display unit 125. There are also a quick review ON / OFF switch for automatically reproducing captured image data immediately after shooting, and an image erasing switch for deleting a reproduced image.

  In addition, there is a compression mode switch for selecting each compression rate of JPEG and MPEG compression and a CCD RAW mode for digitizing and recording the signal of the image sensor as it is. In addition, there is an AF mode setting switch for setting a one-shot AF mode that keeps the autofocus in-focus state when the release switch is half-pressed and a servo AF mode that keeps the autofocus operation continuously.

  Reference numeral 137 denotes a power switch that can switch and set the power-on and power-off modes of the imaging apparatus 100. In addition, the power-on and power-off settings of various accessory devices such as the lens unit 200, the strobe unit 300, and the recording medium connected to the imaging device 100 can be switched and set together. The timer function 139 includes a clock function, a calendar function, a timer counter function, and an alarm function, and is used for system management such as a transition time to the sleep mode and alarm notification.

  Reference numerals 102 and 202 denote lens mounts, which are interfaces for connecting the imaging apparatus 100 to the lens unit 200. Reference numerals 101 and 201 denote connectors that electrically connect the imaging apparatus 100 to the lens unit 200, and are controlled by the camera control unit 140. Reference numerals 111 and 301 denote accessory shoes, which are interfaces for connecting the imaging apparatus 100 to the strobe unit 300.

  The lens control unit 203 includes a memory that controls the entire lens unit 200 and stores operation constants, variables, programs, and the like. In addition, a non-volatile memory (not shown) that holds identification information such as a number unique to the lens unit 200, management information, function information such as an open aperture value, minimum aperture value, and focal length, and current and past set values. It has. The lens control unit 203 also has a function of controlling the diaphragm 211, controlling the focusing of the photographing lens 210, and controlling zooming of the photographing lens 210.

  The strobe unit 300 has a strobe light emission control unit 302. The strobe light emission control unit 302 is connected to the accessory shoe 111 via the interface 301 and operates with respect to a light emission unit such as a xenon tube (not shown) based on information from the photometry unit 142 and a light emission amount or light emission. Control timing.

  Reference numeral 150 denotes a microphone provided as an audio input unit that acquires an audio signal, which converts the audio into an electric signal and inputs the audio signal to the imaging apparatus 100. Reference numeral 151 denotes an A / D converter that converts an analog signal output from the microphone 150 into a digital signal. The digital signal that has been A / D converted by the A / D converter 151 is encoded by the Audio Codec unit 152. The signal encoded by the audio codec unit 152 is controlled by the memory control unit 124 and the system control unit 120 and stored in the memory 127.

  At the time of audio reproduction, the audio codec 152 decodes the audio file stored in the memory 127. The decoded signal is converted from a digital signal to an analog signal by the D / A converter 153. Reference numeral 154 denotes a speaker for voice output, which outputs an analog voice signal converted by the D / A converter 153 as voice.

  Reference numeral 155 denotes a voice analysis processing unit for acquiring a voice feature amount from a voice signal. The voice analysis processing unit 155 includes a spectrum analyzer circuit, and obtains the volume and frequency distribution of the input voice signal. The feature amount of the input voice signal obtained by the voice analysis processing unit 155 is calculated by the system control unit 120, and the calculation result is reflected in the parameters of the image processing performed by the image processing unit 123.

  Reference numeral 156 denotes a voice recognition mode switch, which extracts the voice feature amount and changes the image processing parameter and the shooting parameter. The implementation mode (voice recognition mode) of the present embodiment is started by turning on the voice recognition mode switch 156. Is done.

FIG. 2 is a flowchart illustrating an example of a processing procedure executed in the imaging apparatus 100 according to the first embodiment.
In the imaging apparatus of FIG. 1, when the voice recognition mode is selected by the voice recognition mode switch 156, the following flow is processed.
In step S101, it is determined whether or not the release switch 135 is half pressed (SW1). If SW1 is ON, the process proceeds to step S102.

In step S102, an audio signal is acquired from the microphone 150.
In step S103, the sound analysis processing unit 155 analyzes the volume level and frequency of the input sound signal.

  In step S104, the system control unit 120 performs predetermined calculation processing from the volume level and frequency distribution of the acquired audio signal, and calculates the average volume level, the volume change rate per unit time, the average frequency, and the unit time within the audio acquisition period. The rate of change of the average frequency per hit is calculated. Then, voice feature values such as the volume, tempo, pitch, and intonation of the input voice are extracted from the calculation result. Details will be described later.

  In step S105, it is determined whether or not the release switch 135 is fully pressed (SW2). As a result of the determination, if the release switch is fully pressed and SW2 is ON, the voice acquisition is stopped and the release operation is started. If the release switch is not fully pressed and SW2 is not ON, the process returns to step S102. That is, the period from the time when the release switch 135 is turned on to the time when SW2 is turned on is the voice acquisition period.

In step S <b> 106, an imaging process is performed in which a signal read from the image sensor 121 is converted into image data and written to the memory 127.
In step S107, image processing parameter values such as contrast, brightness, saturation, and sharpness corresponding to the audio feature value obtained in step S104 are set.

In step S108, in addition to predetermined pixel interpolation processing and color conversion processing, image processing based on the parameters set in step S107 is performed, and image data compression processing is performed.
In step S109, the image data obtained in step S108 is recorded in the external removable memory unit 131. In addition, an image is displayed on the display device 110.

Next, a method for extracting voice feature amounts will be described.
As described above, the voice signal is analyzed by the voice analysis processing unit 155. The voice analysis processing unit 155 includes a spectrum analyzer function, and can obtain the frequency distribution of the acquired voice signal. Thereby, as shown in FIG. 3, the volume level in each frequency range can be acquired.

From the frequency distribution obtained as described above, the average volume A within the voice acquisition time, the volume change rate B per unit time, the average frequency C, and the unit time from the following formula 1, formula 2, formula 3, and formula 4. The change rate D of the average frequency per hit can be obtained. Here, T is the total number of times of voice acquisition sampling, K is the number of frequency samplings, _ft , _k are the _kth frequency of sampling at the tth time of voice acquisition sampling. a _t and _k represent volume levels at frequencies f _t and _k .

  Here, the volume change rate B per unit time corresponds to the tempo of the input audio signal, the average frequency C corresponds to the pitch, and the change rate D of the average frequency per unit time corresponds to the inflection. As described above, the volume, tempo, pitch, and intonation, which are the feature quantities of the input voice, can be acquired. Note that the above-described four formulas may be moving average formulas for simplicity.

  Next, a method for setting image processing parameters will be described. The image processing parameters to be changed are contrast, sharpness, brightness, saturation, and white balance. For the default image processing, the strength of each effect is adjusted according to the strength, volume, tempo, pitch, and inflection, which are the features of the input sound.

  Hereinafter, the setting method will be described by taking contrast as an example. The contrast has an image processing table in which the strength of the effect changes in proportion to the contrast setting value Cr. The greater the contrast setting value Cr, the stronger the contrast, and the smaller the contrast setting value Cr, the weaker the contrast.

The audio feature amount includes volume Vcr, tempo Tcr, pitch Pcr, and inflection Icr as parameters for contrast adjustment, and each value is set according to the strength of the acquired feature amount. Using the parameters, the contrast setting value Cr is obtained from Equation 5.
Cr = Cro × Vcr × Tcr × Pcr × Icr (Formula 5)
Here, Cro is a default setting value of contrast.

  Here, it is assumed that each image processing parameter and the audio feature amount have the relationship shown in FIG. Taking the contrast setting value as an example, it is assumed that the contrast increases as the volume / inflection increases, and the contrast decreases as the pitch / tempo increases. That is, if each voice feature amount is larger than the default in Equation 5, the volume Vcr / inflection Icr is a value greater than 1, and the pitch Pcr / tempo Tcr is a value smaller than 1.

  The setting method has been described above by taking contrast as an example, but the same image processing table is used for sharpness, lightness, saturation, and white balance, and each effect varies depending on the value of the audio feature amount. However, as shown in FIG. 4, the direction of change of each effect on the audio feature amount is different.

  For example, in an environment where the volume and inflection are high, the brightness and saturation of the image are increased to make an impressive image. In addition, by reducing the contrast and sharpness and shifting the white balance to the yellow side, we create a picture that gives a soft and bright impression.

  On the other hand, in an environment where the volume / inflection is small, the sound is high, and the tempo is fast, the brightness / saturation is lowered and the white balance is shifted to the blue side. Then, the effect of contrast and sharpness is enhanced to produce a tight image.

  As described above, according to the configuration of the first embodiment, it is possible to acquire the environmental sound immediately before shooting and reflect it in the image processing. In the first embodiment described above, the period from when the release switch 135 is half-pressed (SW1) to fully pressed (SW2) is the voice acquisition period, but buttons other than the release switch 135 may be used. .

  The voice acquisition period may be any other period. For example, image processing may be performed based on a voice feature amount acquired in advance before shooting. In addition, a voice feature amount may be extracted after shooting, and image processing may be performed on an already shot image.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
In the first embodiment described above, an example in which the present invention is executed during still image shooting has been described. In the second embodiment, an example in which the present invention is applied during moving image shooting will be described. Since the configuration of the imaging apparatus of the present embodiment is the same as that of the first embodiment, drawings for describing the configuration of the imaging apparatus are omitted.

FIG. 5 shows a flowchart for explaining a processing procedure executed in the second embodiment.
When the voice recognition mode is selected by the voice recognition mode switch 156 provided in the imaging apparatus 100 of FIG. 1, the following flow is processed.

First, in step S201, a moving image switch (not shown) of the operation unit 136 is pressed to determine whether moving image shooting is in a start state. If the result of this determination is that moving image shooting has been started, the process proceeds to step S202.
In step S202, an imaging process is performed in which the signal read from the image sensor 121 is converted into image data and written to the memory 127. An audio signal is acquired from the microphone 150. Next, in step S203, it is selected whether to process the image data acquired in step S202 or to process the audio signal, and when processing the audio signal, the process proceeds to step S204. When processing image data, the process proceeds to step S206.

In step S204, the sound analysis processing unit 155 analyzes the volume level and frequency of the input sound signal.
Next, in step S205, the system control unit 120 performs predetermined arithmetic processing from the volume level and frequency distribution of the audio signal acquired in step S204. Thus, the average sound volume level, the sound volume change rate per unit time, the average frequency, and the change rate of the average frequency per unit time are calculated. Then, the voice feature quantity such as the volume, tempo, pitch, and intonation of the input voice is extracted from the calculation result. The details of extracting the audio feature amount are as shown in the first embodiment.

  However, in the case of video shooting, the acquired audio feature value needs to be dynamically reflected in the image processing, so the volume change rate per unit time, average frequency, average per unit time using the moving average formula It is desirable to calculate the rate of change of frequency.

Next, in step S206, image processing parameter values such as contrast, brightness, saturation, and sharpness corresponding to the audio feature amount obtained in step S205 are set.
In step S207, in addition to predetermined pixel interpolation processing and color conversion processing, image processing based on the parameters set in S207 is performed.

  Next, in step S208, it is determined whether the moving image switch of the operation unit 136 is pressed and moving image shooting is in a stopped state. If the result of this determination is that movie shooting has been stopped, the process proceeds to step S209. If the moving image shooting is not stopped, the process returns to step S202, and the above-described operation is executed.

In step S209, the acquired moving image data is compressed by the image processing unit 123 using the MPEG method. The audio data is compressed by the audio codec unit 152.
Next, in step S210, the moving image file and the audio file compressed in step S209 are recorded in the external detachable memory unit 131 as a moving image file (moving image data + audio data).

  As described above, in the second embodiment described above, the audio feature extraction method and the image processing parameter change method are the same as those in the first embodiment described above, and thus the description thereof is omitted. According to the second embodiment, it is possible to apply the present invention that changes the image processing based on the sound feature amount during shooting to a moving image.

(Third embodiment)
In the first embodiment and the second embodiment described above, the example in which the image processing parameter is changed based on the audio feature amount has been described. On the other hand, in the third embodiment, an example in which the shooting parameter is changed based on the audio feature amount will be described. Since the configuration of the imaging apparatus is the same as that of the first embodiment described with reference to FIG. 1, drawings for describing the configuration of the imaging apparatus are omitted.

  FIG. 6 is a flowchart for explaining the processing procedure of the imaging method performed in the third embodiment. This process is executed when the voice recognition mode is selected by the voice recognition mode switch 156 in the imaging apparatus of FIG.

First, in step S301, the eyepiece detection unit 130 determines whether an eyepiece is detected. If it is determined that the eyepiece is in the eyepiece state, the process proceeds to step S302.
In step S302, an audio signal is acquired from the microphone 150. Thereafter, the process proceeds to step S303. In step S303, the sound analysis processing unit 155 analyzes the volume level and frequency of the sound signal input from the microphone 150.

  Next, in step S304, the system control unit 120 performs predetermined calculation processing from the volume level and frequency distribution of the acquired audio signal, and the average volume level within the audio acquisition period, the volume change rate per unit time, the average frequency, Calculate the rate of change of average frequency per unit time. Then, the voice feature quantity such as the volume, tempo, pitch, and intonation of the input voice is extracted from the calculation result. Details are shown in the first embodiment.

Next, in step S305, it is determined whether or not the release switch 135 is half pressed (SW1). If the result of this determination is that SW1 is ON, voice acquisition is stopped and the mode is shifted to the shooting mode. That is, the period from when the eyepiece is detected by the eyepiece detection unit 130 to when SW1 is turned on is the voice acquisition period.
In step S306, automatic exposure correction (AE) is performed based on the photometric value indicating the brightness of the subject and the peripheral portion of the subject obtained from the photometric unit 142, and the aperture and shutter speed are set.

In step S307, the aperture and shutter speed set values obtained in step S306 are changed based on the audio feature values obtained in step S304.
In step S308, it is determined whether or not the release switch 135 is fully pressed (SW2).

  As a result of the determination in step S308, if SW2 is ON, the process proceeds to step S309. In step S309, a shooting operation is performed based on the aperture and shutter speed setting values set in step S307. Note that in the third embodiment, the method for extracting voice feature amounts is the same as that in the first embodiment, and thus detailed description thereof is omitted.

Next, a method for setting shooting parameters will be described. The shooting parameters to be changed are the aperture and shutter speed settings.
With respect to the default aperture and shutter speed obtained in step 306 in FIG. 6, the set values are adjusted in accordance with the volume, tempo, pitch, and inflection strength, which are the feature amounts of the input sound.

  As an audio feature parameter for exposure (Ev), aperture (Av), and shutter speed (Tv), volume V *, tempo T *, pitch P *, inflection I * (* = ev is exposure, * = av Is an aperture, and * = tv is a shutter speed parameter. In this embodiment, each value is set according to the strength of the feature amount acquired in step S304.

  That is, the exposure correction parameter Kev is obtained from the following equation (6). Kev is a parameter that determines how much exposure should be increased. If the default exposure value is Evo, the corrected exposure Ev is a value obtained by increasing Evo by Kev level.

  Similarly, with respect to the aperture Av and the shutter speed Tv, the aperture correction parameter Kav is obtained from the following equation 7, and the shutter speed correction parameter Ktv is obtained from the following equation 8. The corrected aperture Av is a value lowered by a Kav step relative to the default set value Avo, and the corrected shutter speed Tv is a value raised by a Ktv step relative to the default set value Tvo.

  Here, it is assumed that the relationship between each audio feature amount and the shooting parameter is as shown in FIG. For example, in an environment where the volume and the inflection are high, an image having a soft and bright impression can be taken by correcting the aperture to the open side, decreasing the shutter speed, and increasing the exposure. On the other hand, in an environment where the volume / inflection is low, the sound is high, and the tempo is fast, the exposure is reduced, the aperture is narrowed down, and the shutter speed is increased to produce a tight image.

  According to the third embodiment described above, the present invention can be applied to an imaging apparatus having a poor image processing function. Further, a synergistic effect can be expected by combining with the first embodiment.

  In the third embodiment, still image shooting is assumed, but it may be applied during moving image shooting. For example, in the case of a moving image, in addition to exposure and aperture control, the frame rate at the time of shooting may be changed instead of adjusting the shutter speed. For example, if the tempo is fast among the audio feature amounts, the frame rate of the moving image is thinned out, resulting in a fast-forward effect during reproduction, and a sense of realism is transmitted more.

  In the third embodiment, the period from when the eyepiece is detected by the eyepiece detection unit 130 to when the release switch 135 is half-pressed (SW1) is set as the voice acquisition period. Good. The voice acquisition period may be any other period. For example, shooting parameters may be set based on audio feature values acquired in advance before shooting.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described.
In the first to third embodiments described above, the image processing parameters and the shooting parameters are changed with the feature amount of the audio signal as the volume, tempo, pitch, and inflection. On the other hand, in the fourth embodiment, it is determined whether the distribution of the acquired frequency component is within the audible range, and the image processing parameter and the imaging parameter set according to the determination result are changed.

  In general, there is almost no sound exceeding the human audible range (20 kHz) in urban areas. FIG. 8 is a schematic diagram showing the frequency distribution of environmental sounds in an urban city area, but the upper limit of the average frequency is only about 10 kHz.

  On the other hand, FIG. 9 is a schematic diagram of the frequency distribution of the environmental sound in a natural environment such as a rainforest, but is filled with high frequency components that cannot be heard as sound reaching 150 kHz or higher. Therefore, in the imaging apparatus having the configuration shown in the first to third embodiments, the average volume level of frequency components exceeding the audible range (20 kHz) is calculated from the environmental sound acquired at the time of shooting. Then, it is determined whether the shooting environment is a natural environment or an urban environment, and image processing and shooting parameters for the natural environment or the urban environment are set.

FIG. 10 is a flowchart for explaining the processing procedure of the fourth embodiment. Since other configurations are the same as those in the first to third embodiments, a block diagram and a description for describing the configuration of the imaging apparatus are omitted.
First, in step S401, an audio signal is acquired by the microphone 150.
In step S402, the sound analysis processing unit 155 analyzes the volume level and frequency of the sound signal output from the microphone 150.

  Next, in step S403, predetermined calculation processing is performed by the system control unit 120 from the volume level and frequency distribution of the acquired audio signal, and the average volume level of frequency components exceeding the audible range (20 kHz) is calculated.

  Next, in step S404, it is determined whether or not the average volume level of frequency components exceeding the audible range (20 kHz) is equal to or higher than a predetermined threshold value. If it is determined that the threshold value is equal to or greater than the threshold value, the process proceeds to step S406, and if it is less than the threshold value, the process proceeds to step S405.

  In step S405, the shooting environment is determined to be an urban area, and parameters for the urban area are set. For example, the brightness / saturation is lowered and the white balance is shifted to the blue side. Then, the effect of contrast and sharpness is enhanced and the image is tightened to create an atmosphere surrounded by artifacts.

  On the other hand, in step S406, it is determined that the shooting environment is a natural environment, and parameters for the natural environment are set. For example, the brightness / saturation of the image is increased to make the image colorful. Also, by reducing the contrast and sharpness, the image is created to give a soft and bright impression.

  The configuration of the fourth embodiment has been described above. In the fourth embodiment, only the image processing parameters are changed, but the shooting parameters may be changed. Moreover, you may use together other audio | voice feature-values, such as volume, pitch, tempo, and inflection.

  Further, in the above-described embodiment, the environmental sound is classified into two within the audible range and outside the audible range, and the image processing parameter is changed. However, the classification may be made finer. For example, the volume may be determined at 10 kHz, 20 kHz, 50 kHz, and 100 kHz, and the image processing parameter and the shooting parameter may be changed by analogy with the shooting environment.

The four embodiments have been described above.
In the description of the embodiment, the imaging apparatus 100 is configured to assume a digital camera with interchangeable lenses. However, the imaging apparatus 100 may be a lens-integrated compact digital camera, a camera-equipped mobile phone, or a digital video camera. It is good also as a structure.

  Moreover, although four embodiment was demonstrated separately, it is good also as a structure by which each was combined. Further, although the still image shooting and the moving image shooting are separately described, the present invention may be applied to a still image shot during moving image shooting.

  In the above-described embodiment, the setting examples of the image processing parameter and the shooting parameter for the audio feature amount are shown as in FIGS. 4 and 7, but the relationship between the audio feature amount and each parameter is the same as that of the present embodiment. Different settings may be used.

  In the embodiment, the parameters to be changed based on the image feature amount are contrast, brightness, saturation, sharpness, and white balance. However, other image processing parameters may be changed.

  In the above description, only image data that has undergone image processing based on audio feature values is stored in the recording medium. However, CCD RAW data before image processing and image data that has undergone default image processing are simultaneously stored. You may comprise so that it can do.

  Further, although the configuration has been described in which the external detachable memory unit 131 is mounted on the imaging apparatus 100, the recording medium may be configured to be a single or a combination of a plurality. In the above-described embodiment, the camera control unit 140 and the system control unit 120 have independent circuit configurations. However, the system control unit 120 may have a configuration in which the camera control unit 140 is also used.

  In the description of the embodiment, the imaging apparatus 100 is assumed to have a configuration in which a nonvolatile memory includes a program and display data, but may be configured by a hard disk, a DVD-ROM, a CD-ROM, or the like.

(Other embodiments according to the present invention)
Each unit constituting the imaging apparatus according to the above-described embodiment of the present invention can be realized by operating a program stored in a RAM or a ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 2, 5, 6, and 10) that executes each step in the imaging method described above is directly transferred to a system or apparatus. Alternatively, it is supplied remotely. In addition, this includes a case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Various recording media can be used as a recording medium for supplying the program. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer may perform part or all of the actual processing. The functions of the above-described embodiments can be realized.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to a first embodiment of the present invention. 5 is a flowchart illustrating an example of a processing procedure executed in the imaging apparatus according to the first embodiment of this invention. It is a figure which shows the example of the frequency distribution of the audio | voice signal acquired in the 1st Embodiment of this invention-4th Embodiment. It is a conceptual diagram showing the relationship between the audio | voice feature-value and image processing parameter in the 1st Embodiment and 2nd Embodiment of this invention. 10 is a flowchart illustrating a processing procedure executed in the imaging apparatus according to the second embodiment of this invention. 10 is a flowchart illustrating an example of a processing procedure executed in the imaging apparatus according to the third embodiment of this invention. It is a figure which shows the 3rd Embodiment of this invention and represents the relationship between an audio | voice feature-value and imaging | photography parameter. It is a figure which shows the 4th Embodiment of this invention and shows an example of frequency distribution of the environmental sound of an urban area. It is a figure which shows the 4th Embodiment of this invention and shows an example of frequency distribution of the environmental sound in a natural environment. 14 is a flowchart illustrating an example of a processing procedure executed in the imaging apparatus according to the fourth embodiment of this invention.

Explanation of symbols

100 Imaging device 101 Lens connector 102 Lens mount (camera side)
DESCRIPTION OF SYMBOLS 110 Display apparatus 111 Accessory shoe 120 System control part 121 Image pick-up element 122 A / D conversion part 123 Image processing part 124 Memory control part 125 Liquid crystal panel display part 126 Backlight illumination 127 Memory 128 Non-volatile memory 130 Eyepiece detection part 131 External removable memory Section 132 Playback switch 133 Menu switch 134 Mode dial 135 Release switch 136 Operation section 137 Power switch 138 Power section 139 Timer 140 Camera control section 141 Shutter control section 142 Photometry section 143 Distance measurement section 144 Shutter 150 Microphone 151 A / D conversion section 152 Audio Codec 152
153 D / A converter 154 Speaker 155 Voice analysis processor 156 Voice recognition mode switch 200 Lens unit 201 Lens connector 202 Lens mount (lens side)
203 Lens control unit 210 Shooting lens 211 Aperture 300 Strobe unit 301 Interface 302 Strobe light emission control unit

Claims (9)

Imaging means for capturing a subject and generating still image data or moving image data;
Image processing means for performing predetermined image processing on still image data or moving image data generated by the imaging means;
Audio input means for acquiring an audio signal when the imaging means images the subject;
Voice analysis processing means for analyzing a feature amount of the voice signal acquired by the voice input means;
System control means for controlling image processing parameters used by the image processing means for performing predetermined image processing on the still image data or moving image data based on the feature amount of the audio signal analyzed by the audio analysis processing means. An imaging apparatus comprising:   The imaging means includes an imaging element that converts an optical image formed through an imaging optical system including a lens, a diaphragm, and a shutter into an electrical signal, a photometric means that measures the brightness of a peripheral portion of the subject, and the photometric means The image pickup apparatus according to claim 1, further comprising: a lens control unit that controls the diaphragm in conjunction with a photometric value obtained by the step (a), and a shutter control unit that controls a speed of the shutter.   The system control means includes the aperture and the shutter based on a photometric value indicating brightness of a peripheral portion of the subject measured by the photometry means and a feature amount of an audio signal acquired when the subject is photographed. The imaging apparatus according to claim 2, wherein imaging parameters including the speed of the imaging are controlled.   The feature amount of the audio signal includes at least one of volume, tempo, pitch, and inflection, and the audio analysis processing means performs a predetermined calculation based on the volume and frequency distribution of the audio signal, and The imaging device according to claim 1, wherein a feature amount of the audio signal is obtained.   5. The imaging according to claim 1, wherein the image processing parameter controlled by the system control unit includes at least one of contrast, sharpness, brightness, saturation, and white balance. apparatus.   6. The system control unit according to claim 1, wherein when the imaging unit performs moving image shooting, the system control unit controls a frame rate of the moving image based on a feature amount of an audio signal acquired at the time of shooting. The imaging device according to item. The sound analysis processing means determines whether the distribution of frequency components acquired when the subject is photographed is within an audible range,
The system control unit controls an image processing parameter set when the image processing unit performs image processing and a shooting parameter when the imaging unit captures an image according to a determination result of the voice analysis processing unit. The imaging apparatus according to claim 1, wherein An imaging step of shooting a subject to generate still image data or moving image data;
An image processing step of performing predetermined image processing on still image data or moving image data generated in the imaging step;
An audio input step of acquiring an audio signal when shooting the subject in the imaging step;
A voice analysis processing step of analyzing a feature amount of the voice signal acquired in the voice input step;
A system control step for controlling image processing parameters used for performing predetermined image processing on the still image data or moving image data in the image processing step based on a feature amount of the audio signal analyzed in the audio analysis processing step. An imaging method characterized by comprising: An imaging step of shooting a subject to generate still image data or moving image data;
An image processing step of performing predetermined image processing on still image data or moving image data generated in the imaging step;
An audio input step of acquiring an audio signal when shooting the subject in the imaging step;
A voice analysis processing step of analyzing a feature amount of the voice signal acquired in the voice input step;
A system control step for controlling image processing parameters used for performing predetermined image processing on the still image data or moving image data in the image processing step based on a feature amount of the audio signal analyzed in the audio analysis processing step. A computer program that causes a computer to execute an imaging method including:

Images