JP2005039451A - Imaging device, imaging method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize a diversified use environments by effectively using a plurality of camera units. <P>SOLUTION: The imaging device is provided with: two camera units 16 and 19; microphones 18 and 20 provided corresponding to the camera units 16 and 19; and a control unit 35 which drives them at the same time to photograph moving pictures, compares sound signals obtained by the microphones 18 and 20 with each other, composites the images obtained by the camera parts 16 and 19 according to the comparison result and outputs the composite imsge. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus, an imaging method, and a program suitable for a mobile phone with a camera function, for example.
[0002]
[Prior art]
Recently, as mobile phones with camera functions have become widespread, the camera unit's shooting direction can be rotated, two camera units provided in different directions of the housing, camera unit Various models have been commercialized, such as those having an optical zoom mechanism formed in the axial direction of the hinge portion of the folding casing.
[0003]
However, any one provided with two camera units can selectively output only one of them depending on the object to be photographed, and only one image sensor is used. It is also considered that images can be taken in different directions by selectively switching with an optical member such as a half mirror. (For example, see Patent Document 1.)
[0004]
[Patent Document 1]
JP 2001-223924 A
[0005]
[Problems to be solved by the invention]
However, any of the above two camera sections or one having two shooting optical paths in one shooting section can be used to selectively shoot an image in one direction according to the shooting target. Met.
[0006]
The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an imaging apparatus and an imaging method capable of realizing various usage environments by more effectively using a plurality of camera units. And providing a program.
[0007]
[Means for Solving the Problems]
According to the first aspect of the present invention, a plurality of image pickup means, a plurality of sound input means provided corresponding to each of the plurality of image pickup means, and the plurality of image pickup means and the plurality of sound input means are simultaneously driven to generate a moving image. A photographing drive means for photographing, a comparing means for comparing a plurality of audio signals obtained by driving a plurality of sound input means by the photographing driving means, and a plurality of imaging means based on the comparison result obtained by the comparing means An image synthesizing means for synthesizing the obtained image signals and an output means for outputting the image signals obtained by the image synthesizing means are provided.
[0008]
With such a configuration, it is possible to generate and output a combined image in a natural form, for example, by changing the size of the corresponding image according to the sound pressure level of the audio signal at that time and combining the images.
[0009]
According to a second aspect of the present invention, in the first aspect of the present invention, a plurality of image pickup means, a plurality of sound input means provided corresponding to each of the plurality of image pickup means, the plurality of image pickup means, and a plurality of sound An imaging drive means for capturing a moving image by simultaneously driving the input means, a comparison means for comparing a plurality of audio signals obtained by driving a plurality of audio input means by the imaging drive means, and a comparison result obtained by the comparison means The image selecting means for selecting one of the image signals obtained by the plurality of imaging means based on the above and an output means for outputting the image signal obtained by the image selecting means.
[0010]
With such a configuration, for example, by selecting and outputting an image corresponding to the side with the higher sound pressure level of the audio signal at that time, a plurality of persons can be photographed while the plurality of persons are being photographed. It is possible to generate and output a synthetic image having a natural shape, such as switching images according to the time.
[0011]
According to a third aspect of the present invention, a plurality of image pickup means, a plurality of sound input means provided corresponding to each of the plurality of image pickup means, and the plurality of image pickup means and the plurality of sound input means are simultaneously driven to generate a moving image. An imaging drive unit for imaging, an image synthesis unit for synthesizing image signals obtained by the plurality of imaging units reflecting the positional relationship of the plurality of imaging units, and an output unit for outputting the image signals obtained by the image synthesis unit It was characterized by comprising.
[0012]
With such a configuration, for example, when a plurality of persons are photographed, a combined image in which each person is arranged corresponding to the positional relationship is generated and output. Can be output.
[0013]
According to a fourth aspect of the present invention, in the first aspect of the present invention, the synthesizing unit generates an image corresponding to an audio signal having a lower sound pressure level in an image corresponding to an audio signal having a higher sound pressure level. An image signal combined and synthesized is generated.
[0014]
With such a configuration, in addition to the operation of the invention described in claim 1 above, an image corresponding to the side where the sound pressure level of the audio signal is large at that time is large and an image corresponding to the side where the sound pressure level is small By generating and outputting a composite image of picture-in-picture with a reduced size, a composite image of a natural shape so that the image of the speaker in it is larger than the others while shooting multiple people Can be generated and output.
[0015]
According to a fifth aspect of the present invention, in the invention of the third aspect, the output means outputs a plurality of audio signals obtained by the plurality of audio input means together with the image signals to positions of a plurality of corresponding imaging means, respectively. It is characterized by being output in a separated state reflecting the relationship.
[0016]
With such a configuration, in addition to the operation of the invention described in claim 3, the sound corresponding to the composition of the image can also be output, for example, the sound signal can be output in stereo according to the positional relationship of the plurality of image signals. By outputting, it is possible to output more natural video shooting content.
[0017]
The invention according to claim 6 is the invention according to any one of claims 1 to 3, wherein the output means outputs a plurality of sound signals obtained by the plurality of sound input means together with an image signal. To do.
[0018]
With such a configuration, in addition to the operation of the invention according to any one of the first to third aspects, it is possible to generate and output very natural moving image data combined with sound.
[0019]
A seventh aspect of the invention is characterized in that, in the sixth aspect of the invention, the output means outputs the plurality of audio signals in a separated state.
[0020]
With such a configuration, in addition to the operation of the invention described in claim 6 above, since stereo sound corresponding to the image can be output, very natural and high-quality moving image data in which the localization of the sound image is clear is obtained. Can be generated and output.
[0021]
The invention described in claim 8 is characterized in that, in the invention described in claim 1 or 2, the output means outputs an image signal that goes back in time in consideration of a fixed time lag.
[0022]
With such a configuration, in addition to the operation of the invention described in claim 1 or 2, an image that goes back a certain time is output in consideration of a time lag when outputting an image according to comparison of a plurality of audio signals. Therefore, it is possible to more naturally generate and output, for example, an original image of a person who is a subject to be photographed with simple processing.
[0023]
According to a ninth aspect of the present invention, in the invention according to any one of the first to eighth aspects, the output means records at least an image signal on a recording medium.
[0024]
With such a configuration, in addition to the operation of the invention according to any one of claims 1 to 8, in order to record an image obtained by photographing on a recording medium, a later reproduction process using the recording medium, etc. Can be used for
[0025]
According to a tenth aspect of the present invention, in the invention according to any one of the first to eighth aspects, the output means transmits at least an image signal to a communication medium.
[0026]
With such a configuration, in addition to the operation of the invention according to any one of the first to eighth aspects, not only can it be applied to, for example, transmission of an e-mail attached with an image, but also it has the same function. It is possible to apply to communication using a mutual image in real time, such as a video phone, by communicating with the apparatus.
[0027]
According to an eleventh aspect of the present invention, a plurality of imaging units and a plurality of audio input units provided corresponding to each of the plurality of imaging units are simultaneously driven to capture a moving image, and the imaging driving step A comparison step of comparing a plurality of audio signals obtained by driving a plurality of audio input units, an image synthesis step of synthesizing the image signals obtained by the plurality of imaging units based on the comparison results obtained in the comparison step, And an output process for outputting the image signal obtained in the image synthesis process.
[0028]
With such a method, it is possible to generate and output a synthesized image having a natural shape, for example, by changing the size of the corresponding image in accordance with the sound pressure level of the audio signal at that time.
[0029]
The invention according to claim 12 is a program executed by a computer built in an imaging apparatus provided with a plurality of imaging units and a plurality of audio input units provided corresponding to each of the plurality of imaging units, A shooting drive step of simultaneously driving the imaging unit and the plurality of audio input units to shoot a moving image, a comparison step of comparing a plurality of audio signals obtained by driving the plurality of audio input units in the shooting drive step, An image synthesizing step for synthesizing the image signals obtained by the plurality of imaging units based on the comparison result obtained in the comparing step, and an output step for outputting the image signal obtained in the image synthesizing step. Features.
[0030]
With such program content, it is possible to generate and output a synthetic image in a natural form, for example, by changing the size of the corresponding image according to the sound pressure level of the audio signal at that time and synthesizing it. .
[0031]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
A first embodiment in the case where the present invention is applied to a CDMA (Code Division Multiple Access) mobile phone with a camera function will be described below with reference to the drawings.
[0032]
FIGS. 1A and 1B show an external configuration of the mobile phone 10 according to the first embodiment. Two casings 12 and 13 are integrally formed with a hinge 11 interposed therebetween. FIG. 1A shows the inner surface in a fully opened state, and FIG. 1B mainly shows the outer surface of the upper housing 12 in a folded state.
[0033]
As shown in FIG. 1A, a speaker 14 serving as a receiver, a main display unit 15, and a first camera unit 16 are provided on the inner surface of the upper housing 12.
[0034]
On the other hand, various keys 17 including dial keys and a first microphone 18 serving as a transmitter are provided on the inner surface of the lower housing 13.
[0035]
As shown in FIG. 1B, the outer surface of the upper housing 12 is provided with a second camera unit 19, a second microphone 20, a photographing light 21 made of a high-intensity LED, and a sub display unit 22.
[0036]
Further, an antenna 23 extending also into the lower housing 13 is formed so as to protrude from the outer surface side of the hinge portion 11.
[0037]
Although not shown here, an outer surface speaker 24 larger than the speaker 14 is provided on the outer surface side of the lower housing 13 to emit a beep sound, a melody, or the like at the time of incoming call.
[0038]
FIG. 2 shows a circuit configuration of the mobile phone 10. In the figure, the antenna 23 performs CDMA communication with the nearest base station, and an RF unit 31 is connected to the antenna 23.
[0039]
The RF unit 31 separates the signal input from the antenna 23 from the frequency axis by a duplexer at the time of reception, and converts the frequency into an IF signal by mixing with a local oscillation signal having a predetermined frequency output from the PLL synthesizer. Only the reception frequency channel is extracted by the broadband BPF, and the signal level of the desired reception wave is made constant by the AGC amplifier, and then output to the modulation / demodulation unit 32 at the next stage.
[0040]
On the other hand, after transmitting the modulation power of OQPSK (Offset Quadri-Phase Shift Keying) sent from the modulation / demodulation unit 32, the RF unit 31 uses the AGC amplifier to control the transmission power based on the control from the control unit 35 described later. The signal is mixed with a local oscillation signal having a predetermined frequency output from the PLL synthesizer, frequency-converted to an RF band, amplified to a large power by a PA (Power Amplifier), and radiated and transmitted from the antenna 23 via the duplexer.
[0041]
The modulation / demodulation unit 32 separates the IF signal from the RF unit 31 into a baseband I · Q (In-phase / Quadrature-phase) signal by a quadrature detector at the time of reception, digitizes it, and outputs it to the CDMA unit 33.
[0042]
On the other hand, at the time of transmission, the modem unit 32 converts the digital I / Q signal sent from the CDMA unit 33 into an analog signal, performs OQPSK modulation with a quadrature modulator, and sends the result to the RF unit 31.
[0043]
The CDMA unit 33 inputs the digital signal from the modulation / demodulation unit 32 at the time of reception to a PN (Pseudo Noise) timing extraction circuit and a plurality of demodulation circuits that perform despreading and demodulation in accordance with instructions from the timing circuit. Are synchronized with each other by the synthesizer and output to the speech processing unit 34.
[0044]
On the other hand, at the time of transmission, the CDMA unit 33 spreads the output symbol from the voice processing unit 34 and then limits the band with a digital filter to form an I / Q signal, which is sent to the modem unit 32.
[0045]
The voice processing unit 34 deinterleaves the output symbols from the CDMA unit 33 at the time of reception, and then performs error correction processing by a Viterbi demodulator, and then, from a digital signal compressed by a voice processing DSP (Digital Signal Processor), The digital audio signal is expanded into an analog signal, and the speaker 14 or, if necessary, the external speaker 24 is loudly driven.
[0046]
On the other hand, at the time of transmission, the audio processing unit 34 digitizes the analog audio signals input from the first microphone 18 and the second microphone 20 and then compresses the data amount by the audio processing DSP, and then the error is detected by the convolutional encoder. After performing the correction encoding, interleaving is performed, and the output symbol is sent to the CDMA unit 33.
[0047]
Then, a control unit 35 is connected to the RF unit 31, the modem unit 32, the CDMA unit 33, and the audio processing unit 34, and a GPS receiver 36, an image capturing unit 37, a moving image processing unit 38, The main display unit 15, the sub display unit 22, the memory card 39, the vibrator unit 40, and the LED drive unit 41 are connected.
[0048]
Here, the control unit 35 is composed of a CPU and a ROM in which an operation program including a moving image communication operation, which will be described later, is fixedly stored, a RAM used as a work memory, and the like, and controls the overall operation of the mobile phone 10. To do.
[0049]
The GPS receiver 36 calculates the latitude, longitude and altitude of the current position and the accurate current time based on the positioning information from the plurality of GPS satellites received by the GPS antenna 42 and outputs the calculated current time to the control unit 35.
[0050]
Under the control of the control unit 35, the image photographing unit 37 performs the photographing operation with the CCD 44 disposed behind the photographing optical axis of the optical lens system 43 constituting the first camera unit 16, and the second camera unit 19. The imaging operation of the CCD 46 disposed behind the imaging optical axis of the optical lens system 45 constituting the above is controlled, and two image data obtained by these imaging are digitized and output.
[0051]
The moving image processing unit 38 compresses the image data obtained from the image photographing unit 37 and the audio data obtained from the first microphone 18 and the second microphone 20 based on, for example, the MPEG4 (Moving Pictures coding Expert Group 4) system. While generating the moving image data, the received moving image data is decompressed and decompressed to obtain the original bitmap image data and audio data.
[0052]
The memory card 39 is detachably attached to the cellular phone 10 and stores moving image data photographed by itself, moving image data obtained by reception, and the like.
[0053]
The vibrator unit 40 vibrates with a vibration pattern and vibration intensity set in advance when an incoming call is received.
[0054]
The LED driving unit 41 includes a high-intensity white LED that constitutes the photographing light 21 and a driving circuit for the high-intensity white LED. If necessary, the LED driving unit 41 emits auxiliary light toward a subject to be photographed by the second camera unit 19. .
[0055]
Each of the main display unit 15 and the sub display unit 22 is composed of a reflective / transmissive color liquid crystal panel with a backlight, and can be displayed as a transmissive liquid crystal by turning on the backlight. Although the brightness is somewhat reduced, it is also possible to display as a reflective liquid crystal using external light with the backlight turned off.
[0056]
Although not shown, the hinge portion 11 has a mechanism for detecting the expanded state and the folded state of the upper housing 12 and the lower housing 13. In accordance with information from the detection mechanism, the control unit 35 operates the camera key 17a of the various keys 17 to shift to take a picture by using the first camera unit 16 and the second camera unit 19 to make a call. In FIG. 1A, the user of the cellular phone 10 identifies himself / herself with the first camera unit 16 and the second camera unit 19 with the upper housing 12 and the lower housing 13 opened. Therefore, it is determined that the other person is going to be photographed at the same time, the display on the sub display unit 22 is stopped, and the monitor image based on the photographing by the first camera unit 16 and the second camera unit 19 is displayed on the main display unit 15. Let
[0057]
Next, the operation of the above embodiment will be described.
FIG. 3 shows the contents of the call processing in the video phone mode that is basically executed by the control unit 35 based on the operation program fixedly stored in advance, and is received from the mobile phone of the other party having the same function. It is assumed that the reproduction of the moving image data is executed in parallel with this processing, and the description thereof is omitted here, and only the process from the acquisition of the moving image data to the transmission is described.
[0058]
Initially, the first microphone 18 and the second microphone 20 are used to record sound in both directions of the inner surface and the outer surface of the upper housing 12 (referred to as “voice A, voice B” in the figure) (step A01). The system waits for the timing of shooting individual images constituting the moving image (step A02).
[0059]
In this case, for example, assuming that the frame rate of a moving image is 15 [frames / second], the resolution is 160 dots wide × 120 dots long, and the audio sampling frequency is 12 [KHz], the audio signal is sampled at 12 [KHz]. During execution, the shooting timing is determined every 1/15 [seconds], and this is determined in step A02, and images of both the inner surface and the outer surface of the upper housing 12 by the first camera unit 16 and the second camera unit 19 (see FIG. Then, “image A, image B”) is photographed (step A03).
[0060]
At the same time, the sound pressure levels of the two audio data obtained by the recording are compared to determine which is greater (step A04).
[0061]
If the sound pressure level of the sound data obtained by the first microphone 18 is equal to or higher than the sound pressure level of the sound data obtained by the second microphone 20, the first camera unit 16 obtains an image corresponding to the sound. Then, a picture-in-picture composite image in which a part of the image data, for example, the image data obtained by the second camera unit 19 is inserted in the lower right is created (step A05).
[0062]
As this composite image, the image data having the resolution of 160 dots horizontally × 120 dots vertically obtained by the second camera unit 19 as described above is thinned out for each dot both vertically and horizontally, and image data of 80 dots wide × 60 dots vertically is obtained. It can be easily created by replacing it with a portion in the range over the lower right quarter of the image data having the resolution of 160 horizontal dots × 120 vertical dots obtained by the first camera unit 16.
[0063]
If it is determined in step A04 that the sound pressure level of the sound data obtained by the second microphone 20 is higher than the sound pressure level of the sound data obtained by the first microphone 18, the sound is handled. A picture-in-picture composite image in which a part of the image data obtained by the second camera unit 19 as an image to be processed, for example, the image data obtained by the first camera unit 16 is inserted in the lower right is created (step A06). ).
[0064]
In this way, the composite image data obtained in step A05 or A06 is used, and two audio data for 1/15 [seconds] acquired between the previous image shooting and the current image shooting are superimposed to obtain a predetermined value. The data is formatted (step A07), transmitted to the other party of the call (step A08), and the series of processing in units of image data is completed. Then, the processing returns to the processing from step A01. The above process is repeated until the call is finished.
[0065]
FIG. 4 illustrates the concept of audio data corresponding to the image data that has undergone the processing in step A05 and that has been formatted in a predetermined format in step A07. As the image data, a part of the image data obtained by the first camera unit 16, for example, a picture-in-picture composite image in which the image data obtained by the second camera unit 19 is inserted in the lower right is arranged. On the other hand, as the audio data, the audio data obtained by the first microphone 18 and the audio data obtained by the second microphone 20 are superposed.
[0066]
When the moving image data is actually transmitted after being compressed in accordance with, for example, the MPEG4 standard, the moving image processing unit 38 performs processing such as GOP (Group Of Pictures) and motion compensation on the image data of a plurality of frames. Therefore, the concept is different from the data format in which audio data corresponding to the time is added to the image data of one frame as shown in FIG.
[0067]
In this way, for example, by generating a picture-in-picture composite image in which the image corresponding to the side with the higher sound pressure level of the audio signal is enlarged and the image corresponding to the side with the lower sound pressure level is reduced. It is possible to generate a composite image in a natural form, for example, by enlarging an image on the speaking side of a single mobile phone 10 while simultaneously photographing two persons.
[0068]
(Second Embodiment)
A second embodiment when the present invention is applied to a CDMA mobile phone with a camera function will be described below with reference to the drawings.
[0069]
The external configuration is basically the same as that in FIG. 1 and the circuit configuration is the same as in FIG. 2, and therefore, the same reference numerals are used for the same parts, and illustration and description thereof are omitted.
[0070]
Next, the operation of the above embodiment will be described.
FIG. 5 shows the contents of the call processing in the video phone mode that is basically executed by the control unit 35 based on the operation program fixedly stored in advance, and is received from the mobile phone of the other party having the same function. It is assumed that the reproduction of the moving image data is executed in parallel with this processing, and the description thereof is omitted here, and only the process from the acquisition of the moving image data to the transmission is described.
[0071]
Initially, the first microphone 18 and the second microphone 20 are used to record sound in both directions of the inner surface and the outer surface of the upper housing 12 (referred to as “voice A, voice B” in the figure) (step B01). The system waits for the timing of photographing individual images constituting the moving image (step B02).
[0072]
In this case, for example, assuming that the frame rate of a moving image is 15 [frames / second], the resolution is 160 dots wide × 120 dots long, and the audio sampling frequency is 12 [KHz], the audio signal is sampled at 12 [KHz]. During execution, the shooting timing is determined every 1/15 [seconds], and this is determined in step B02, and images of both the inner surface and the outer surface of the upper housing 12 by the first camera unit 16 and the second camera unit 19 (see FIG. Then, “image A, image B”) is photographed (step B03).
[0073]
At the same time, the sound pressure levels of the two audio data obtained by the recording are compared, and it is determined which one is greater (step B04).
[0074]
If the sound pressure level of the sound data obtained by the first microphone 18 is equal to or higher than the sound pressure level of the sound data obtained by the second microphone 20, the first camera unit 16 obtains an image corresponding to the sound. The selected image data is selected (step B05).
[0075]
As this selected image, the image data having the resolution of 160 horizontal pixels × 120 vertical pixels obtained by the first camera unit 16 as described above is used as it is.
[0076]
If it is determined in step B04 that the sound pressure level of the sound data obtained by the second microphone 20 is higher than the sound pressure level of the sound data obtained by the first microphone 18, the sound is handled. The image data obtained by the second camera unit 19 is selected as the image to be performed (step B06).
[0077]
In this way, the image data selected in step B05 or B06 is used, and two audio data for 1/15 [seconds] acquired between the previous image shooting and the current image shooting are superimposed to form a predetermined format. (Step B07), transmitted to the other party (step B08), the series of processing in units of image data is completed, and the process returns to the processing from step B01. The above process is repeated until the end of.
[0078]
FIG. 6 exemplifies the concept of audio data corresponding to the image data that has undergone the processing in step B05 and that has been formatted in a predetermined format in step B07. As the image data, only the image data obtained by the first camera unit 16 is used, while as the audio data, the audio data obtained by the first microphone 18 and the audio data obtained by the second microphone 20 are superimposed. It is arranged in the state.
[0079]
When the moving image data is actually transmitted after being compressed in accordance with, for example, the MPEG4 standard, the moving image processing unit 38 performs processing such as GOP (Group Of Pictures) and motion compensation on the image data of a plurality of frames. Therefore, the concept is different from the data format in which audio data corresponding to the time is added to the image data of one frame as shown in FIG.
[0080]
In this way, for example, by selecting the image side corresponding to the side where the sound pressure level of the audio signal is large at that time, one mobile phone 10 is simultaneously shooting two persons. It is possible to generate a natural image such as switching to the speaking side.
[0081]
(Third embodiment)
A third embodiment when the present invention is applied to a CDMA mobile phone with a camera function will be described below with reference to the drawings.
[0082]
The external configuration is basically the same as that in FIG. 1 and the circuit configuration is the same as in FIG. 2, and therefore, the same reference numerals are used for the same parts, and illustration and description thereof are omitted.
[0083]
Next, the operation of the above embodiment will be described.
FIG. 7 shows the contents of the call processing in the videophone mode that is basically executed by the control unit 35 based on the operation program fixedly stored in advance, and is received from the mobile phone of the other party having the same function. It is assumed that the reproduction of the moving image data is executed in parallel with this processing, and the description thereof is omitted here, and only the process from the acquisition of the moving image data to the transmission is described.
[0084]
Initially, the first microphone 18 and the second microphone 20 are used to record sound in both directions of the inner surface and the outer surface of the upper housing 12 (referred to as “voice A, voice B” in the figure) (step C01). It waits for the timing of shooting of individual images constituting the moving image (step C02).
[0085]
In this case, for example, assuming that the frame rate of a moving image is 15 [frames / second], the resolution is 160 dots wide × 120 dots long, and the audio sampling frequency is 12 [KHz], the audio signal is sampled at 12 [KHz]. During execution, the shooting timing is determined every 1/15 [second], and this is determined in step C02, and images of the inner surface and the outer surface of the upper housing 12 by the first camera unit 16 and the second camera unit 19 (see FIG. Then, “image A, image B”) is photographed (step C03).
[0086]
Then, using the image data obtained by the first camera unit 16 and the image data obtained by the second camera unit 19, a composite image in which these are arranged on the left and right is created (step C04).
[0087]
As the composite image, as described above, the image data having the resolution of horizontal 160 dots × vertical 120 dots obtained by the first camera unit 16 and the second camera unit 19 is thinned out for each dot only in the horizontal direction, and the horizontal 80 dots. X Image data of 120 dots vertically is generated, and the two images are simply combined horizontally so that the image data obtained by the first camera unit 16 is on the left and the image data obtained by the second camera unit 19 is on the right. The image data has a resolution of 160 dots × 120 dots vertically.
[0088]
In this way, the composite image data obtained in step C04 is used, and two audio data for 1/15 [seconds] acquired between the previous image shooting and the current image shooting are used as the left channel (Lch) and The right channel (Rch) is separated into a predetermined format (step C05), transmitted to the other party (step C06), and the series of processing in units of image data is completed. Returning to the process, the above process is repeated until the telephone call in the videophone mode is finished.
[0089]
FIG. 8 illustrates the concept of audio data corresponding to the image data that has undergone the processing in step C04 and that has been formatted in a predetermined format in step C05.
[0090]
As image data, the image data obtained by the first camera unit 16 is arranged on the left side, and the image data obtained by the second camera unit 19 is arranged on the right side and arranged as a single composite image. As the audio data, the audio data obtained by the first microphone 18 is arranged in the state of stereo audio so that the audio data obtained by the second microphone 20 becomes the right channel.
[0091]
Therefore, if the other party of the call is also reproduced using the equivalent mobile phone 10, a composite image showing the two persons is displayed on the main display unit 15 and, for example, mainly the first microphone 18. Corresponding to the positional relationship of the person who obtained the voice data, the voice data obtained in step 2 is emitted from the outer speaker 24 and the voice data obtained in the second microphone 20 is emitted by the first microphone 18. The speaker 14 and the external speaker 24 can be driven separately to emit different sounds.
[0092]
When the moving image data is actually transmitted after being compressed in accordance with, for example, the MPEG4 standard, the moving image processing unit 38 performs processing such as GOP (Group Of Pictures) and motion compensation on the image data of a plurality of frames. Therefore, the concept is different from the data format in which audio data corresponding to the time is added to the image data of one frame as shown in FIG.
[0093]
Thus, for example, by generating a composite image in which each person is arranged corresponding to the positional relationship in a state where a plurality of persons are photographed, a natural-shaped composite image can be generated.
[0094]
In addition, since the audio data obtained by the first microphone 18 and the second microphone 20 is transmitted in the state of being separated as a stereo signal, if the other party is also equipped with two speakers, each audio is transmitted. The sound image can be output separately, and the sound image can be output clearly and the moving image can be output with a very high quality so that the difference in the person who speaks can be inferred from the output position of the sound.
[0095]
In the first and second embodiments, the state of the immediate image is switched depending on the level of the sound pressure level of the two audio signals. However, the switching has been made since the previous switching. Switching may be performed in consideration of time and the difference in sound pressure level between the two audio signals.
[0096]
In that case, immediately after the previous image switching, if there is no difference in sound pressure level exceeding a certain threshold value, the image is not switched, and the contents of the threshold are changed as time elapses from switching. Reduce in steps.
[0097]
By doing so, it can be avoided that the contents of the composite image are switched unnaturally in a short cycle and become difficult to see.
[0098]
Also, when switching the composite image in the first and second embodiments, the image obtained at that time is held for a certain amount of time by the buffer memory so that the user does not feel uncomfortable with the sound at the time of switching. An image that is traced back by a certain time lag, for example, 0.2 [seconds] may be extracted and used for output.
[0099]
This is because, for example, when an image is switched due to a change in the sound state, an image that is traced back in time is used instead of the point in time when the change in the sound state is detected. By outputting an image from the state immediately before the utterance of the person who started the speech, not from the state of the utterance, a visually very natural image can be provided.
[0100]
Further, in each of the first to third embodiments, the case where the signals generated from the two image signals and the two corresponding audio signals are immediately transmitted to the call partner has been described. However, it may be recorded once on a recording medium such as the memory card 39 and subjected to processing such as editing or reproduction, or may be used as an attached file of an e-mail. .
[0101]
Furthermore, the first to third embodiments described all describe the case where the present invention is applied to a mobile phone with a camera function. However, the present invention is not limited to this, and moving image shooting is possible. The present invention can be applied to a digital camera, a PDA with a camera function, a video camera device, a conference recording system, and the like.
[0102]
In addition, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention.
[0103]
Further, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, at least one of the problems described in the column of the problem to be solved by the invention can be solved, and described in the column of the effect of the invention. In a case where at least one of the obtained effects can be obtained, a configuration in which this configuration requirement is deleted can be extracted as an invention.
[0104]
【The invention's effect】
According to the first aspect of the present invention, it is possible to generate and output a synthesized image in a natural form, for example, by changing the size of the corresponding image according to the sound pressure level of the audio signal at that time, and combining the images. it can.
[0105]
According to the second aspect of the present invention, for example, by selecting and outputting an image corresponding to the side with the higher sound pressure level of the audio signal at that time, a remark in the state where a plurality of persons are being photographed It is possible to generate and output a natural composite image, such as switching a person's image according to the situation.
[0106]
According to the third aspect of the present invention, for example, when a plurality of persons are photographed, a combined image in which each person is arranged corresponding to the positional relationship is generated and output. Can be generated and output.
[0107]
According to the invention described in claim 4, in addition to the effect of the invention described in claim 1, the image corresponding to the side where the sound pressure level of the audio signal is large at that time is enlarged and the image corresponding to the side where the sound pressure level is low is supported. By generating and outputting a composite image of a picture-in-picture with a reduced size, it is possible to create a natural image so that the image of the speaker within it is larger than the others while shooting multiple people. A composite image can be generated and output.
[0108]
According to the fifth aspect of the present invention, in addition to the effect of the third aspect of the present invention, in addition to the effects of the third aspect of the invention, the audio signal is also stereoized and output in accordance with the positional relationship of the plurality of image signals. By outputting sound, more natural video shooting content can be output.
[0109]
According to the invention described in claim 6, in addition to the effect of the invention described in any one of claims 1 to 3, it is possible to generate and output very natural moving image data combined with sound.
[0110]
According to the seventh aspect of the invention, in addition to the effect of the sixth aspect of the invention, since stereo sound corresponding to the image can be output, a very natural and high quality moving image with a clear localization of the sound image. Data can be generated and output.
[0111]
According to the invention described in claim 8, in addition to the effect of the invention described in claim 1 or 2, an image that goes back a certain time in consideration of a time lag when outputting an image according to comparison of a plurality of audio signals. Therefore, it is possible to more naturally generate and output, for example, an original image of a person who is a subject to be photographed with simple processing.
[0112]
According to the ninth aspect of the invention, in addition to the effects of the first to eighth aspects of the invention, an image obtained by photographing is recorded on the recording medium. It can be used for processing.
[0113]
According to the invention described in claim 10, in addition to the effect of the invention described in any one of claims 1 to 8, it can be applied not only to, for example, transmission of an e-mail attached with an image, but also to the same function. Communication with other devices can be applied to communication using mutual images in real time, such as a videophone.
[0114]
According to the eleventh aspect of the invention, it is possible to generate and output a synthesized image in a natural form, for example, by changing the size of the corresponding image in accordance with the sound pressure level of the audio signal at that time. it can.
[0115]
According to the twelfth aspect of the invention, it is possible to generate and output a synthetic image having a natural shape, for example, by changing the size of the corresponding image in accordance with the sound pressure level of the audio signal at that time. it can.
[Brief description of the drawings]
FIG. 1 is a diagram showing an external configuration of a mobile phone according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing a functional configuration of an electronic circuit of the mobile phone according to the embodiment;
FIG. 3 is a flowchart showing the processing content when acquiring moving image data according to the embodiment;
FIG. 4 is a diagram illustrating a data configuration generated per unit time according to the embodiment;
FIG. 5 is a flowchart showing the processing contents when moving image data is acquired according to the second embodiment of the present invention.
FIG. 6 is a diagram illustrating a data configuration generated per unit time according to the embodiment;
FIG. 7 is a flowchart showing the processing contents when acquiring moving image data according to the third embodiment of the present invention;
FIG. 8 is a diagram illustrating a data configuration generated per unit time according to the embodiment;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 ... Mobile phone, 11 ... Hinge part, 12 ... Upper housing | casing, 13 ... Lower housing | casing, 14 ... Speaker, 15 ... Main display part, 16 ... 1st camera part, 17 ... Various keys, 18 ... 1st microphone, DESCRIPTION OF SYMBOLS 19 ... 2nd camera part, 20 ... 2nd microphone, 21 ... Shooting light, 22 ... Sub display part, 23 ... Antenna, 24 ... External speaker, 31 ... RF part, 32 ... Modulation / demodulation part, 33 ... CDMA part, 34 ... Audio processing unit, 35 ... control unit, 36 ... GPS receiver, 37 ... image capturing unit, 38 ... moving image processing unit, 39 ... memory card, 40 ... vibrator unit, 41 ... LED drive unit, 42 ... GPS antenna, 43 ... optical Lens system, 44 ... CCD, 45 ... Optical lens system, 46 ... CCD.

Claims (12)

A plurality of imaging means;
A plurality of voice input means provided corresponding to each of the plurality of imaging means;
Shooting driving means for simultaneously driving the plurality of imaging means and the plurality of audio input means to shoot a moving image;
Comparison means for comparing a plurality of sound signals obtained by driving a plurality of sound input means by the photographing driving means,
Image synthesizing means for synthesizing the image signals obtained by the plurality of imaging means based on the comparison result obtained by the comparing means;
An image pickup apparatus comprising: output means for outputting an image signal obtained by the image composition means. A plurality of imaging means;
A plurality of voice input means provided corresponding to each of the plurality of imaging means;
Shooting driving means for simultaneously driving the plurality of imaging means and the plurality of audio input means to shoot a moving image;
Comparison means for comparing a plurality of sound signals obtained by driving a plurality of sound input means by the photographing driving means,
Image selecting means for selecting one of the image signals obtained by the plurality of imaging means based on the comparison result obtained by the comparing means;
An image pickup apparatus comprising: output means for outputting an image signal obtained by the image selection means. A plurality of imaging means;
A plurality of voice input means provided corresponding to each of the plurality of imaging means;
Shooting driving means for simultaneously driving the plurality of imaging means and the plurality of audio input means to shoot a moving image;
Image combining means for combining the image signals obtained by the plurality of imaging means to reflect the positional relationship of the plurality of imaging means;
An image pickup apparatus comprising: output means for outputting an image signal obtained by the image composition means. 2. The synthesizing unit generates an image signal obtained by inserting and synthesizing an image corresponding to an audio signal having a lower sound pressure level into an image corresponding to an audio signal having a higher sound pressure level. The imaging device described. The output means outputs a plurality of audio signals obtained by the plurality of audio input means together with the image signal in a state of being separated by reflecting a positional relationship between a plurality of corresponding imaging means. Item 4. The imaging device according to Item 3. The imaging apparatus according to claim 1, wherein the output unit outputs a plurality of audio signals obtained by the plurality of audio input units together with the image signal. The imaging apparatus according to claim 6, wherein the output unit outputs the plurality of audio signals in a separated state. The image pickup apparatus according to claim 1, wherein the output unit outputs an image signal that goes back in time in consideration of a certain time lag. The image pickup apparatus according to claim 1, wherein the output unit records at least an image signal on a recording medium. The imaging apparatus according to claim 1, wherein the output unit transmits at least an image signal to a communication medium. A shooting driving step of simultaneously driving a plurality of imaging units and a plurality of audio input units provided corresponding to each of the plurality of imaging units to shoot a moving image;
A comparison step of comparing a plurality of audio signals obtained by driving a plurality of audio input units in this photographing driving step;
An image synthesis step of synthesizing the image signals obtained by the plurality of imaging units based on the comparison result obtained in the comparison step;
And an output step of outputting an image signal obtained in the image composition step. A program executed by a computer built in an imaging apparatus provided with a plurality of imaging units and a plurality of audio input units provided corresponding to each of the plurality of imaging units,
A shooting driving step of simultaneously driving the plurality of imaging units and the plurality of audio input units to shoot a moving image;
A comparison step for comparing a plurality of audio signals obtained by driving a plurality of audio input units in this photographing driving step;
An image synthesis step for synthesizing the image signals obtained by the plurality of imaging units based on the comparison result obtained in the comparison step;
A program for causing a computer to execute an output step of outputting an image signal obtained in the image synthesis step.

Images