JP2010093671A - Camera, reproducing device, and reproducing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera capable of picking up a sound for sound effect free of a sense of incongruity when reproducing a synthesized still picture or moving picture on the basis of a plurality of images, and a reproducing device and a reproducing method that performs sound reproduction with sound effect free of a sense of incongruity when reproducing a synthesized still picture or moving picture on the basis of a plurality of images. <P>SOLUTION: The camera includes an imaging unit 2 which continuously photographs a subject, and a right and a left sound pickup unit 7 that can change a sound pickup range of a sound from a subject direction. The imaging unit can generate a synthesized image by synthesizing a plurality of images obtained successively by the imaging unit 2, and sound pickup ranges 33a and 33b are changed by the right and the left sound pickup units 7 on the basis of movements of the images when the plurality of images are successively photographed. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a camera and a playback device, and more particularly, to a camera that can be recorded during shooting, and a playback device and playback method for a shot image shot with this camera.

  In recent years, large screen televisions have become widespread, and captured images are played back and displayed on large screen televisions. Also, since the image quality of televisions has been improved and the power consumption has been reduced, captured images are displayed like posters and the images can be enjoyed as interiors. In addition, digital photo frames for displaying digital images have become widespread. Thus, recently, life has been colored with photographed images.

  When displaying in an interior sensation, images are not intrusive, and there is a need for something that can heal, such as majestic landscapes and beautiful flower-and-birds, and a different shooting method and display method are required. It has become.

For display, a panoramic image display is proposed by combining a plurality of images. For example, Patent Document 1 discloses an image display device that expresses the entire panoramic image by using the entire size of one of the images to be combined and performs powerful display. In addition, a proposal has been made to use a panoramic image, combine a plurality of images, and display the panoramic image as a still image on a large screen having an aspect ratio of 16: 9.
Japanese Patent Laid-Open No. 5-308553

Moreover, when displaying a photographed image as if it were an interior sensation, the sound recorded at the time of photographing can be played back, and it can be immersed in memories of the past. Various proposals have been made to record audio during shooting. For example, Patent Document 2 discloses a sound source stabilization device that converts a monaural sound image into a stereo sound image. That is, this sound source stabilization device analyzes the object in the image, the movement (position) of the object, the operation of the camera, etc. from the divided image using the information of the image knowledge database, and the object is emitted. The sound source that is thought to be present is separated from the sound information, and the separated sound source is rearranged in a sound field space suitable for video.
JP 2000-295700 A

  Thus, by compositing images, it is possible to obtain an image suitable for a large screen, and by reproducing the sound recorded at the time of shooting, it is possible to immerse yourself in healing effects and memories. However, since the individual images to be synthesized are recorded with directivity, if they are synthesized as they are, the position of the sound source moves during the reproduction of the sound, and the reproduction is not suitable for viewing the image.

  In this regard, the image display device disclosed in the above-mentioned Patent Document 1 is silent about the reproduction of sound. The device disclosed in Patent Document 2 converts a monaural sound image into a stereo sound image, requires an image knowledge database, and increases the size. In addition, the conventional cameras are supposed to be used to record the voice of my child at, for example, athletic meet and academy, and match the sound within the field of view. It did not consider the case of compositing images.

  The present invention has been made in view of such circumstances, and a camera that enables audio recording with an acoustic effect without a sense of incongruity when a synthesized still image or video is reproduced based on a plurality of images. It is an object of the present invention to provide a playback apparatus and a playback method for playing back sound with a sound effect that does not cause a sense of incongruity when playing back a composite still image or moving image based on a plurality of images.

  In order to achieve the above object, a camera according to a first aspect of the invention includes an imaging unit that continuously shoots a subject, a sound collection change unit that can change a sound collection range of sound from the subject direction, and the imaging unit. An image synthesis unit that synthesizes the plurality of obtained images and generates a synthesized image, and a control unit that changes the sound collection range of the sound collection change unit when the plurality of images are captured.

  A camera according to a second invention is the camera according to the first invention, wherein the control unit obtains each image for generating the composite image from right to left and from left to right. Accordingly, the sound collection range is changed from left to right or from right to left, respectively.

  A camera according to a third aspect of the invention is obtained continuously by an imaging unit that captures continuous images while moving the camera field of view left and right, an audio acquisition unit that records audio in a plurality of directions at the time of shooting, and the imaging unit. An image synthesizing unit that synthesizes the plurality of images and generates a synthesized image; and a control unit that changes the synthesis of audio in a plurality of directions obtained by the audio acquisition unit when synthesizing the plurality of images. Have.

  In the camera according to a fourth aspect, in the third aspect, the control unit changes the voice synthesis in accordance with a change in position of a predetermined subject in each image for generating the synthesized image.

  According to a fifth aspect of the present invention, there is provided a camera according to a fifth aspect of the present invention, an imaging unit that continuously shoots a subject, a sound collection changing unit that can change a sound collection range from the direction of the subject, and a plurality of images obtained continuously by the imaging unit. And a control unit that changes the sound collection range depending on whether the image is obtained from the right to the left or the left to the right.

  According to a sixth aspect of the present invention, there is provided a camera that continuously captures images of a subject, a sound collection changing unit that can change a sound collection range of sound from the direction of the subject, and a plurality of images obtained continuously by the imaging unit. The sound collection range of the sound collection change unit is changed in the direction of the image obtained by the image pickup unit when the plurality of images are combined with the image composition unit that generates the composite image and the composite image. And a control unit.

  A camera according to a seventh aspect of the invention includes an imaging unit that continuously shoots a subject, a sound collection change unit that can change a sound collection range of the sound from the direction of the subject, a motion determination unit that determines the movement of the camera, A control unit that changes a sound collection range of the sound collection change unit based on a determination result by the motion determination unit when the continuous shooting is performed.

In the camera according to an eighth aspect based on the seventh aspect, the motion determination unit makes a determination based on image data output from the imaging unit.
According to a ninth aspect of the present invention, in the seventh aspect, the camera further includes a face detection unit that determines whether the face of the subject exists based on the image data output from the imaging unit. The control unit controls the sound collection range of the sound collection changing unit based on the face when a face is detected by the face detection unit.

  According to a tenth aspect of the present invention, there is provided a playback device for storing continuously captured image data, a storage unit for storing stereo sound data recorded in stereo during the continuous shooting, and a display unit for reproducing and displaying an image based on the image data. And an audio playback unit that can be played back by changing the left / right balance based on the stereo audio data, a motion determination unit that determines the movement of the camera, and at the time of playback of the image data and the stereo audio data, And a control unit that controls the left-right balance of the stereo audio data based on the movement.

  According to an eleventh aspect of the present invention, in the playback apparatus according to the tenth aspect, in the tenth aspect, the control unit obtains a correction angle from the angular velocity of the camera and the timing of each frame of the continuous shooting, and the control unit determines the correction angle according to the correction angle. Controls the left / right balance of stereo audio data.

  According to a twelfth aspect of the present invention, there is provided a reproduction method for storing image data that has been continuously captured and stereo sound data that has been stereo recorded at the time of continuous capture, determining the movement of the camera, and reproducing the image data and the stereo sound data. Sometimes, the left and right balance of the stereo audio data is controlled based on the movement of the camera.

  A program according to a thirteenth aspect of the present invention stores continuously captured image data and stereo sound data recorded in stereo during the continuous shooting, determines the movement of the camera, and reproduces the image data and the stereo sound data. And causing the computer to control the left-right balance of the stereo audio data based on the movement of the camera.

  ADVANTAGE OF THE INVENTION According to this invention, when reproducing | regenerating a synthetic | combination still image and a moving image based on a some image, the camera which enabled the audio | voice recording which becomes an acoustic effect without a sense of incongruity can be provided. In addition, it is possible to provide a playback device and a playback method for playing back sound with a sound effect that does not cause a sense of incongruity when playing back a composite still image or moving image based on a plurality of images.

  Hereinafter, preferred embodiments using a digital camera to which the present invention is applied will be described with reference to the drawings. The digital camera according to the present embodiment records sound with directivity according to the situation while continuously shooting, so that an acoustic effect without a sense of incongruity during reproduction is obtained.

  FIG. 1 is a block diagram showing the configuration of a camera 10 and an external device 20 according to the first embodiment of the present invention. The camera 10 is a digital camera, and includes a signal processing and control unit 1, an imaging unit 2, a change determination unit 3, a recording unit 4, an operation determination unit 6, a left / right audio recording unit 7, a display unit 8, a clock unit 9, and a communication. Part 12.

  The signal processing and control unit 1 in the camera 10 includes a signal processing LSI dedicated to the camera 10 and controls the entire camera 10 and performs image processing of image data output from the imaging unit 2. The imaging unit 2 includes a photographic lens and an imaging element that converts a subject image formed by the photographic lens into image data.

  The recording unit 4 records the image data output from the imaging unit 2 after being subjected to image processing and compression processing by the signal processing and control unit 1. The change determination unit 3 determines a change in the field of view of the camera using the image data output from the imaging unit 2. That is, when the camera 10 moves from right to left, the image moves from left to right. When the camera 10 moves from left to right, the image moves from right to left, and this movement is detected. Based on the determination result by the change determination unit 3, audio recording is performed in consideration of the movement of the camera 10. Further, the change determination unit 3 detects the presence of a human face using image data, and also determines the position of a face part and the position of a face part such as a mouth when a face part is present.

  The left and right audio recording unit 7 has a stereo microphone 7a, and records left and right audio, respectively. The left and right audio recording unit 7 can process an audio signal from the stereo microphone and change the audio recording range (that is, change directivity in collecting sound). The audio data output from the left and right audio recording unit 7 is signal-processed by the signal processing and control unit 1 and then recorded in the recording unit 4 together with the image data.

  The operation determination unit 6 includes an operation member such as a release button and a switch linked to the operation member. The operation state determined by the operation determination unit 6 is sent to the signal processing and control unit 1, and the signal processing and control unit 1 executes a process according to the operation state. The clock unit 9 has a calendar / time keeping function such as date and time, and outputs shooting date and time information at the time of shooting. The shooting date / time information is recorded in the recording unit 4 together with the image data.

  The display unit 8 performs live view display of the subject image for framing based on the image data output from the imaging unit 2, and reproduces and displays the image data recorded in the recording unit 4. The communication unit 12 performs transmission and reception with an external device 20 such as a television. As communication means, wireless LAN, proximity wireless communication, infrared communication, wired communication using a USB cable or the like can be used, and image data and audio data captured by the camera 10 can be transmitted. In recent years, HDMI and the like tend to be used for transmitting images and sounds to a high-definition display, and the communication unit 12 may include an HDMI terminal and may perform wired communication.

  The external device 20 such as a television or a photo stand includes a signal processing and control unit 21, a communication unit 22, a display / playback unit 23, a display priority unit 24, and a remote control reception unit 25. Similar to the signal processing and control unit 1 of the camera 10, the signal processing and control unit 21 is configured by a signal processing LSI dedicated to the external device 20, etc., and controls the entire external device 20 and received via the communication unit 22. Controls playback and display of image data and audio data.

  The communication unit 22 communicates with the camera 10 and receives image data and audio data from the camera 10. Similar to the communication unit 12 of the camera 10, communication by wireless LAN, proximity wireless communication, infrared communication, USB cable, HDMI cable, or the like is possible. The display priority unit 24 determines the priority of the image. That is, it is determined whether or not the image is a priority image to be displayed first on the display unit 8 built in the camera 10.

  The display / playback unit 23 includes a thin large screen monitor and a speaker, and plays back and displays image data and audio data received from the camera 10. In reproduction, the signal processing and control unit 21 performs reproduction control according to the determination result of whether or not the image is the priority image in the display priority unit 24. In addition, when the external device 20 is a television, a normal television broadcast or the like is also displayed.

  The remote control receiving unit 25 receives an instruction signal from the remote control device by infrared communication. For example, the remote controller can receive, reproduce, or interrupt a designated image or sound from the camera 10.

  Next, a method of using the camera 10 will be described with reference to FIG. As shown in FIG. 2A, the user 15 holds the camera 10 and shoots a subject image through the photographic lens of the imaging unit 2, and can also record audio from the front by the stereo microphone 7a. As shown in FIG. 2B, the image and the sound thus captured are transmitted to the external device 20 via the communication unit 12 of the camera 10 and the communication unit 22 of the external device 20. The external device 20 reproduces and displays the received image and sound on the display / reproduction unit 23.

  Next, shooting and sound recording by the camera 10 in this embodiment will be described with reference to FIGS. 3 and 4. FIG. 3 shows a state where the camera 10 performs shooting and audio recording. The user first starts shooting at the position of the camera 10a, moves the camera 10 toward the position of the camera 10b, and continuously performs shooting during this time. At this time, a still panoramic image can be obtained as shown in FIG. 4A by superimposing and synthesizing similar portions of the obtained images. In this example, three images 51a, 51b, and 51c are combined to generate a still panoramic image.

  FIG. 4B is a still panoramic image obtained by synthesizing three images 51a, 51b, and 51c as in FIG. 4A. When sound in the center of the screen of each image 51a, 51b, 51c is recorded at the time of shooting, the positions of the sound sources 53a, 53b, 53c are moved from left to right during reproduction, resulting in unnatural sound reproduction. .

  Therefore, in the present embodiment, when shooting at the position of the camera 10a, as shown in FIG. 3, within the angle of view 31a, sound is picked up in the sound collection range 33a from the right, and shooting is performed at the position of the camera 10b. In this case, sound is collected in the sound collection range 33b from the left in the angle of view 31b. When the images thus captured are combined with the panorama image and the sound is reproduced, the sound source position 52a when the image 51a is captured and the sound source position 52b when the image 51c is captured are substantially at the center of the panorama image. It becomes. That is, as shown in FIG. 4C, the sound collected from the sound source position 52c at the substantially center of the panoramic image is heard. As described with reference to FIG. 4B, there is almost no movement of the sound source position, and sound reproduction can be performed with a natural feeling.

  As described above, in the present embodiment, a change in sound is suppressed by correction so that an image created by pasting together images taken in time series is not felt. In this embodiment, three images are pasted together. However, it takes time to record a large number of images, select a plurality of images from these images, synthesize the images, and reproduce the sound. But of course it does n’t matter.

  Next, the operation in this embodiment will be described with reference to the flowchart shown in FIG. This flowchart is controlled by the signal processing and control unit 1 of the camera 10.

  If the camera control flow shown in FIG. 5 is entered, it is first determined whether or not the camera is in shooting mode (S101). The camera 10 has a shooting mode and a playback mode. If the result of determination in step S101 is shooting mode, image capture is performed and face detection is performed (S102). In this step, the image data output from the imaging unit 2 for live view display is acquired, and the change determination unit 3 performs face detection using the acquired image data. Subsequently, image display is performed (S103). Here, the subject image is displayed on the display unit 8 based on the image data acquired in step S102. The photographer can perform framing while viewing the subject image.

  Once image display has been performed, it is next determined whether or not a face has been detected (S104). In step S102, face detection is performed. At this time, it is determined in this step whether or not a face portion has been detected from the image. If it is determined that a face has been detected as a result of this determination, the position of the face and the position of a facial part such as a mouth are detected (S105). Here, the detected face position is used for focusing and exposure control.

  If the face position or mouth position is determined, or if it is determined in step S104 that there is no face, it is next determined whether or not to start recording (S106). Here, the operation state of the release button is detected, and it is determined whether to start moving image shooting, panoramic shooting, or the like. If the result of this determination is that recording has not started, processing returns to step S101 and the aforementioned operation is executed.

  If the result of determination in step S106 is that recording has started, shooting and sound recording are performed (S107). In this subroutine, image and sound are continuously recorded and screen motion is detected as needed, and the sound collection range is changed according to the motion detection result. The photographing / sound collecting operation continues until an end determination is made in this subroutine. This shooting / sound recording subroutine will be described later with reference to the flowchart shown in FIG.

  If the result of determination in step S101 is that shooting mode has not been set, it is determined whether or not playback mode has been set (S112). If the result of this determination is that playback mode has not been set, processing returns to step S101. On the other hand, if the result of determination in step S111 is that the playback mode has been set, playback is performed (S112).

  In step S112, the photographed image recorded from the recording unit 4 is read out, and the image is displayed on the display unit 8 in a thumbnail format. When an image is selected by the operation unit, the image is enlarged and displayed. Further, when the audio data is recorded together with the display of the image, it is reproduced. If no speaker is provided in the camera 10, only image playback is performed and audio playback is not performed.

  Once reproduction has been performed, it is next determined whether or not transmission is to be performed (S113). Here, it is determined whether or not an operation member for transmission instruction has been operated in order to transmit an image to the external device 20 such as a television. If the result of this determination is transmission, a display image is transmitted (S114). In this step, the image being displayed in step S112 is transmitted to the external device 20. When a plurality of images are selected, these images may be transmitted together. When the display image is transmitted, if the result of determination in step S113 is not transmission, or when shooting / sound recording is completed in step S107, the camera control flow is terminated. Returning to, the above-described operation is executed.

  Next, the photographing / sound recording subroutine in step S107 will be described with reference to the flowchart shown in FIG.

  If this flow is entered, it is first determined whether or not panoramic shooting is performed (S1). Here, it is determined whether or not the user has set the panoramic shooting mode with the operation member. It may be determined whether or not the shooting is suitable for the panorama after shooting, and the panorama may be created. However, in this embodiment, in order to simplify the flow, the panorama shooting mode is performed by user setting.

  If the result of determination in step S <b> 1 is that panoramic shooting mode has not been set, normal shooting of normal moving images or the like is performed (S <b> 11). Subsequently, it is determined whether or not telephoto shooting is performed (S12). Here, it is determined whether or not the zooming operation is performed on the telephoto side. If the result of this determination is telephoto, centralized sound collection is performed (S13). On the other hand, if the result of determination is not telephoto, sound collection that emphasizes the left and right stereo feeling is performed (S14). ). That is, depending on the angle of view of the photographing lens 2a of the camera 10, the recording is switched between focusing on the center or recording with a wide sound collection range and emphasizing stereo. Note that the change of the sound collection range is performed by the left and right audio recording unit 7.

  If the centralized recording or the stereo emphasized recording is performed, it is next determined whether or not the normal photographing is finished (S15). Here, it is determined whether or not an end operation has been performed with the release button. If the result of this determination is not end, processing returns to step S11 and shooting is continued. On the other hand, if it is finished, the shooting / sound recording subroutine is finished and the flow returns to the original flow.

  If the result of determination in step S1 is panoramic shooting mode, imaging is first performed and an image at the edge of the image is recorded (S2). The edge image is recorded so that it can be used later, for example, in a different memory area or by leaving the feature of the edge image on the tag. When the camera 10 is moved in the direction of the arrow by performing this edge image recording, an image of the edge in the screen (the tree 55 in the example of FIG. 9B) is as shown in FIG. 9A. Then, the panoramic image can be generated so that the end image (tree 55) is centered by sequentially moving through the image 56a → the image 56b → the image 56c. Even when the camera 10 is moved, the recording (recording image direction recording) is performed so that the sound recording directivity is always given in the direction of the image (tree 55) at the end portion, and the unnaturalness at the time of sound reproduction is reduced. is doing.

  Edge image recording is started, and continuous shooting is performed (S3). Subsequently, it is first determined whether or not there is a face of a predetermined size in the center (S4). Here, the change determination unit 3 determines whether or not a face having a predetermined size, for example, a face having a width of 1/5 of the screen width, exists in the approximate center of the screen. This is because, in the case of the scene shown in FIG. 7, when shooting is started from the left side, if the person 57 occupies a size larger than a predetermined size, the person 57 is likely to be the main subject.

  In this case, since the person 57 may speak while looking at the landscape, when it is determined that there is a face at the beginning of the shooting (timing in step S105), the process branches from step S4 to step S21. Direction tracking emphasis recording is performed (S21). In face direction tracking emphasis recording, the sound collection ranges 58 a to 58 c are sequentially changed in accordance with the movement of the camera 10, and sound is collected in the direction of the person 57. It is possible to collect sound suitable for photographing mainly of the person 57.

  If the result of determination in step S4 is that there is no face of a predetermined size at the center, next, screen motion determination is performed (S5). As shown in FIG. 9, it is determined whether the captured image is moving from left to right or from right to left according to a change in the target direction of the camera 10. Subsequently, it is determined whether or not the screen movement is from left to right (S6). If the result of this determination is that the screen motion is from left to right, the audio recording enhancement direction is changed from right enhancement to left enhancement (S8). On the other hand, when the movement of the screen is from right to left, the recording is made from left enhancement to right enhancement (S7).

  The operation from right enhancement to left enhancement recording in step S8 will be described with reference to the flowchart shown in FIG. In this flow, first, the image at the right end of the screen is read (S31). Subsequently, recording in the direction of the recording image is performed (S32). Here, as described in FIG. 4 and FIG. 9, sound is initially collected toward the right side of the screen, but sound is gradually collected toward the left side of the screen as the screen moves. This sound collecting operation is performed by the left and right audio recording unit 7.

  Note that the left-emphasized to right-emphasized recording in step S7 may be performed in the opposite manner to the right-emphasized to left-emphasized recording. In the flow shown in FIG. 10, the sound collection direction is moved according to the movement of the image in the screen, but only the direction of the screen movement is detected, and 120 ° is simply moved in a few seconds according to the direction of movement. The sound collection range may be changed depending on the degree. That is, assuming that the user changes the angle of view at a predetermined speed, it is also possible to switch the audio recording direction without using the image from the imaging unit 2.

  If the emphasized recording in step S7, S8 or S21 is performed, it is next determined whether or not the photographing is finished (S10). In this step, the operation state of the release button is determined. If the result of this determination is that photography has not ended, processing returns to step S3 and photography continues. On the other hand, if the result of the determination is that the processing has been completed, the shooting / sound recording subroutine is terminated and the flow returns to the original flow.

  As described above, the camera 10 according to the present embodiment moves the sound collection range according to the movement of the camera 10 when continuously capturing images such as the panoramic shooting mode. For this reason, when the images are combined and reproduced and displayed, the sound source position of the sound does not move unnaturally, and the sound can be reproduced with an acoustic effect without a sense of incongruity.

  Next, the configuration and operation of the left and right audio recording unit 7 for changing the sound collection range will be described. The left and right audio recording unit 7 includes a stereo microphone 7a, an AD converter 42, and an adder / multiplier 43, as shown in FIG.

  The stereo microphone 7a includes a right microphone 41a and a left microphone 41b, and is disposed on the front side of the camera body 10. The stereo microphone 7a is connected to the AD converter 42, and the audio signal is digitized. That is, the right microphone 41a is connected to the AD converter 42a, and the left microphone 41b is connected to the AD converter 42b to output digital audio data.

  The output terminal of the AD converter 42 is connected to an adder / multiplier 43 to calculate the difference between the left and right sounds. That is, the AD converter 42a that outputs the audio data of the right microphone 41a is connected to the plus side input end of the adder 43a and the minus side input end of the adder 43d. Further, the AD converter 42b that outputs the audio data of the left microphone 41b is connected to the minus side input end of the adder 43a and the plus side input end of the adder 43d.

  The output of the adder 43a is connected to the input terminal of the multiplier 43b, and the output terminal of the adder 43d is connected to the input terminal of the multiplier 43e. Control ends of the multiplier 43b and the multiplier 43e are connected to the signal processing and control unit 1, and input gains of the multipliers 43b and 43e. The input terminal of the adder 43c is connected to the output terminal of the AD converter 42a and the output terminal of the multiplier 43b. The input terminal of the adder 43f is connected to the output terminal of the AD converter 42b and the output terminal of the multiplier 43e.

  The output terminal of the adder / multiplier 43 is an output unit as the left and right audio recording unit 7 and is connected to the recording unit 4. That is, the output terminal of the adder 43c and the output terminal of the adder 43f output right audio data and left audio data, respectively, and each audio data is recorded in the recording unit 4 via these output terminals.

  Thus, the left and right audio recording unit 7 is configured, and can emphasize either left or right of the audio data input in stereo. The audio signals input by the two microphones 41a and 41b of the left and right audio recording unit 7 are converted into digital audio data by the AD converters 42a and 42b, and (right audio data)-(left audio data) by the adder 43a. Is calculated, and (additional left audio data) − (right audio data) is calculated by the adder 43d. That is, the difference between the left and right audio data is calculated by the adders 43a and 43b. Here, the calculated difference is a difference between left and right sounds, and by emphasizing this difference, it is possible to obtain an audio output in which the spread is emphasized to the right or left, and this addition operation is a preprocessing for that purpose. .

  The differences obtained by the adders 43a and 43d are multiplied by multipliers 43b and 43e based on the gain from the signal processing and control unit 1, respectively, and the multiplication results are added to the right audio data by the adders 43c and 43f. And the left audio data are added respectively. Since the outputs of the adders 43a and 43d are positive, they are substantially added. Here, if either one of the gains in the multipliers 43b and 43e is increased, the voice having the higher gain is emphasized, and the voice output having the spread enhanced on the right side or the left side can be obtained. Also, by increasing both gains, it is possible to obtain an audio output that emphasizes the center. The signal processing and control unit 1 can change the sense of spread by controlling the gain for the multipliers 43b and 43e at the timings of steps S7, S8, S13, S14, and S21.

  In this way, the left and right audio recording unit 7 in the present embodiment can change the direction of the sound collection range from right to left and from left to right using a pair of microphones having the same performance. Further, since the screen motion is determined and the emphasis recording in the left and right audio recording unit 7 is controlled, even if the shooting field of view changes, the sound is reproduced from a pseudo sound source position in a pseudo manner. can do.

  As described above, according to the first embodiment of the present invention, since the sound collection range is changed in accordance with the movement of the screen, there is no sense of incongruity when a composite still image is reproduced based on a plurality of images. It is possible to perform sound reproduction with acoustic effects.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment of the present invention, the sound collection range is changed in accordance with the movement of the camera when shooting with the camera 10. In the second embodiment, stereo recording is performed without changing the sound collection range at the time of shooting, and the position of the sound source is changed in accordance with the movement of the camera at the time of reproducing a composite image such as a panoramic image. The configuration in this embodiment is the same as that of the first embodiment shown in FIG.

  The operation of this embodiment will be described with reference to the flowchart of camera control shown in FIG. If the camera control flow is entered, it is first determined whether or not the shooting mode is set (S201). Steps S201 to S203 are the same as the camera control flow in the first embodiment shown in FIG. However, face detection was performed at the time of image capture in step S202, but this is omitted in this embodiment. Of course, face detection may be performed for exposure control and automatic focus adjustment.

  If image display (live view display) is performed in step S203, it is next determined whether or not recording is started as in step S106 (S204). Here, it is determined whether or not the release button is operated. If the result of this determination is that recording has not started, processing returns to step S201 and the above-described operation is executed.

  If the result of determination in step S204 is that recording has started, continuous shooting is started, and stereo recording is started simultaneously (S206). Next, motion determination is performed (S207). In this step, the movement of the camera 10 is determined from the image change based on the image data. If there is a motion as a result of this determination, the feature of the motion is recorded (S208). Here, the feature of the motion is recorded in the recording unit 4 together with the image data.

  If motion feature recording is performed or if the result of determination in step S207 is that there is no motion, it is next determined whether or not to end recording (S209). In this step, the determination is made based on the operation state of the release button. If the result of this determination is that recording has not ended, processing returns to step S206 and continuous shooting is continued. On the other hand, if the result of determination is that recording is complete, this flow is terminated, and the flow returns to step S201 again.

  If the result of determination in step S201 is not shooting mode, it is next determined whether or not playback mode is set (S211). Here, it is determined whether or not an operation member such as a playback button is operated and a playback start instruction is issued from the user. If the result of this determination is that playback mode has not been set, processing returns to step S201. On the other hand, if the result of determination is that the playback mode has been set, it is next determined whether or not panoramic playback is to be performed (S212).

  In step S212, it is determined in step S208 whether or not a motion feature is recorded together with the image data. That is, when motion features are recorded, a panoramic image can be obtained by combining a plurality of frames. Further, when determining whether or not the panorama playback is performed, it may be determined including whether or not the panorama playback mode is set by the user.

  If the result of determination in step S212 is not panorama playback, normal movie playback is performed (S213), and stereo playback is performed (S214). Here, the moving image recorded in step S206 and the recorded sound are reproduced.

  If the result of determination in step S212 is panorama playback, image connection playback is performed (S221). That is, in this step, a panoramic image is generated as a still image by pasting the common parts of the continuous images taken in step S206. Subsequently, the sound is reproduced with the direction sound enhancement of the panorama central image (S222). In this step, the center image of the completed panoramic image is determined, the sound obtained at each position is corrected from the image position and the motion feature data recorded in step S208, and the panoramic image is displayed in the center of the panoramic image. Play sound as if the sound source is located.

  The audio reproduction in the audio enhancement in step S222 is controlled by the left and right audio recording unit 7 shown in FIG. In stereo recording in step S206, the left and right audio recording unit 7 does not perform audio enhancement, records the left and right stereo sounds as they are in the recording unit 4, and performs reproduction with audio enhancement in step S222. The audio reproduction in the left and right audio recording unit 7 will be described later with reference to FIG.

  Further, in the audio enhancement reproduction in step S222, in the first embodiment, as shown in FIG. 9, the audio is targeted at the image of the end in the screen (the tree 55 in the example of FIG. 9B). I was trying to balance the direction. For this reason, the end image needs to be included in the center of the panoramic image. In the present embodiment, even when there is no target end image at the center of the panoramic image, audio enhancement playback is performed using a sensor (for example, a gyroscope) that detects an angular velocity during handling of the camera. ing. This voice enhancement reproduction will be described later with reference to FIGS.

  If reproduction by voice emphasis in step S222 or stereo reproduction in step S214 is performed, it is next determined whether or not reproduction has ended (S215). Here, the determination is made based on whether or not a reproduction end instruction is given by the user, such as an operation member such as a reproduction button being operated again. If the result of this determination is that playback has not ended, processing returns to step S212 and playback operation continues.

  If the result of determination in step S215 is that reproduction has ended, it is next determined whether or not to transmit (S226). Here, it is determined whether or not an operation member such as a transmission button is operated to transmit the displayed reproduction image to the external device 20. If the result of this determination is transmission, the display image is transmitted to the external device 20 (S277). If the display image is transmitted or if the result of determination in step S226 is not transmission, the camera control flow is terminated, and the process is executed again from step S201.

  Next, the configuration of the left and right audio recording unit 7 that performs audio enhancement processing in the direction of the panoramic central image in step S222 will be described with reference to FIG.

  FIG. 12 is a block diagram showing the configuration of the left and right audio recording unit 7. The left and right audio recording unit 7 adjusts the balance of left and right audio reproduction and performs audio enhancement processing during reproduction. Compared with the configuration in the first embodiment shown in FIG. 8, the point that the recording unit 4 is connected between the AD converter 42 and the adder / multiplier 43 is different, but the configuration in each circuit is as follows. This is the same as the left and right audio recording unit 7 in the first embodiment.

  That is, the output end of the AD converter 42a for AD converting the audio signal of the right microphone 41a is connected to the recording unit 4, and the AD data is converted by the AD converter 42a, and the audio data recorded in the recording unit 4 is added to the adder 43a, 43c and 43d. Also, the output end of the AD converter 42b for AD converting the audio signal of the left microphone 41b is connected to the recording unit 4, and AD conversion is performed by the AD converter 42b, and the audio data recorded in the recording unit 4 is added to the adder 43a, 43d and 43f.

  In the first embodiment described above, the sound recording unit 7 performs emphasized sound collection at the time of shooting and changes the sound collection range. However, in this embodiment, at the time of shooting, the sound signal from the stereo microphone 7a is AD. It is digitized by a converter and recorded in the recording unit 4 as it is without changing the sound collection range of the audio data. Then, based on the audio data read from the recording unit 4 during reproduction, the adder / multiplier 43 controls the left / right balance during audio reproduction. That is, the gain applied to the multipliers 43b and 43e is changed in accordance with the movement of the screen, and sound reproduction is performed as if there is a sound source in the direction of the image at the center of the panoramic image.

  Next, the speech enhancement subroutine in the direction of the panoramic central image in step S222 will be described with reference to FIGS. As described above, in the first embodiment, when detecting the image direction of the center portion of the panoramic image, the image (tree 55) at the end in the screen is used as a reference. In the present embodiment, the image direction of the central portion of the panoramic image can be detected even when there is no such edge image.

  FIG. 13A shows a state in which images are continuously captured in order to generate a panoramic image. That is, first, shooting is started at the timing of the camera 10a (timing T1). At this time, when the image is taken by the camera 10, an image 61a having an angle of view θ can be obtained as shown in FIG. Subsequently, when the image is taken at the timing T2 in the middle, the target tree 59 is reflected and an image 61b can be obtained. When the end is reached and the last image is taken at the position of the camera 10b (timing T3), an image 61c can be obtained. In addition, these images 61a-61c are acquired and recorded in step S206 (refer FIG. 11).

  When images 61a to 61c continuously acquired by the camera 10 are arranged, the result is as shown in FIG. 13B, and a panoramic image 62 can be obtained by combining these images. In the center of this panoramic image, there is a tree 59 as a target image. In the present embodiment, even if there is no common image (tree 59) in the first and last panoramic images, there is a tree 59 in the center of the panoramic image obtained by combining. Because of the difference in timing until the image enters the field of view, it is possible to correct the direction during audio reproduction.

  That is, as shown in FIG. 13 (a), the camera 10a starts to capture an image having an angle of view θ at timing T1, and the last of the panoramic image having the angle of view θ at the position of the camera 10b (timing T3). Get the image. At the timing T2 between these, the center image of the panoramic image is obtained. The angle Φ moved between the timing T1 and the timing T2 can be determined from the angle of view θ and the speed of the image moving within the angle of view.

  When the image moves from the end of the angle of view to the end in the time of ΔT, it can be seen how much angle θ the camera 10 has been moved based on the angular velocity v acquired by the angular velocity sensor. This angular velocity v is determined as a motion feature in step S208 during shooting, and is recorded together with the image. Therefore, the sound collection angle to be corrected can be determined by multiplying the difference T2-T1 between the timing T1 and the timing T2 by the angular velocity v.

  Next, a speech enhancement subroutine in the direction of the panorama central image will be described with reference to the flowchart shown in FIG. If this flow is entered, first, determination of the center frame when the panoramic image is synthesized is performed (S501). In this step, in the example of FIG. 13, the image 61b corresponding to the center of the panoramic image is determined as the center frame. Subsequently, the central frame shooting timing is set to T2 (S502).

  Next, the voice obtained at T is read (S503). In this step, audio data corresponding to the read image data is read from the recording unit 4, and the timing of this audio is T. Subsequently, a correction angle is obtained from (T2−T) × θ / ΔT (S504). In this step, as described with reference to FIG. 13, the correction angle is obtained by multiplying the difference (T2−T) from the central portion at each timing T by the angular velocity (v = θ / ΔT).

  Once the correction angle is calculated, the correction angle speech enhancement is performed using the correction angle (S505). In this step, the gain applied to the multipliers 43b and 43e of the left and right audio recording unit 7 shown in FIG. 12 is changed according to the correction angle. As a result, it is possible to perform enhanced reproduction such that the sound source position is in the direction of the substantially central portion of the panoramic image.

  Once the correction angle speech enhancement has been performed, it is next determined whether or not it is finished (S506). Here, whether or not the reproduction is finished is determined according to the operation state of the operation member such as the reproduction button. If the result of this determination is that it has not ended, processing returns to step S503, and audio reproduction by audio enhancement is continued. On the other hand, if the result of determination in step S506 is end, processing returns to the original flow.

  As described above, in the second embodiment of the present invention, stereo recording is performed at the time of shooting, and playback is performed by voice enhancement according to the movement of the screen during playback, so that the position of the sound source is at the center. Audio playback is in progress. For this reason, when a panoramic image is reproduced and displayed as a still image, it is possible to prevent the position of the sound source from moving and unnatural sound reproduction.

  Further, in the present embodiment, the correction angle can be obtained by calculation even if there is no end image (tree 55) in the first and last screens. For this reason, even during panoramic shooting in a wider range than in the first embodiment, the correction angle can be calculated, and even if the shooting direction changes greatly, changes in sound are not anxious.

  Note that the speech enhancement is performed so that the sound source is in the direction of the image at the center, but this direction may be changed as appropriate even when the direction is other than the center. In this case, the timing of T2 may be set manually.

  Next, modifications of the first and second embodiments of the present invention will be described with reference to FIGS. 15 and 16. That is, in the first and second embodiments, continuous shooting is performed, a panoramic image is synthesized using a plurality of images obtained thereby, and is displayed as a still image. On the other hand, in this modified example, it is assumed that the continuously shot images are sequentially reproduced and reproduced like a moving image. According to this modification, it is possible to reduce unnecessary changes in sound due to a change in the visual field of the camera.

  With reference to FIG. 15, photographing and sound recording by the camera 10 in the present modification will be described. FIG. 15A shows a state where shooting and audio recording are performed by the camera 10. The user 15 first starts photographing at the position of the camera 10b and moves the camera 10 toward the position of the camera 10a. At this time, when an image is taken at the position of the camera 10a, an image as shown in FIG. 15B is obtained, and when an image is taken at the position of the camera 10b, an image as shown in FIG. 15C is obtained. .

  When a wide seaside scene as shown in FIGS. 15B and 15C is recorded together with continuous shooting or moving image shooting, a narrow range corresponding to the angle of view ahead of the position of each camera 10 is recorded. Will be recorded. However, in this range, it is possible to move only the human eyes 15a and 15b without moving the face. In other words, the camera 10 picks up the sound with a forward sound according to the screen change, but the photographer's ear 15 c may be listening to the sound in the audible range 35.

  When the sound recorded in such a situation is reproduced, if the sound changes in accordance with the movement of the camera 10, it is not suitable for enjoying the image and sound. Therefore, in this modification, even if the screen moves, there is a sound source at a predetermined location in the screen so that sound can be heard from there.

  Although this modification can be applied at the time of photographing as in the first embodiment, an example applied to the second embodiment will be described here. In this case, during the camera control flow shown in FIG. 11, audio playback may be performed when playing back a moving image instead of a still image panoramic image in steps S221 and S222. In other words, while reproducing and displaying a moving image, reproduction is performed as if the sound was recorded when the camera was facing the center of the screen during panorama.

  For example, as shown in FIG. 20, consider a case where a moving image is photographed while slowly looking up from the bottom to the top along the trunk of a large tree. In this case, if the heel stops and sings on the trunk, when shooting the bottom, you will hear the voice of the heel from the center of the screen, but as the screen heads to the top, Voice-enhanced playback is performed so that the voice can be heard. As a result, since the voice of the fox is heard from the same sound source position, it does not give an unnatural feeling. Of course, similar to the first and second embodiments, the same effect can be obtained even in continuous shooting that moves left and right.

  By adopting such a configuration and performing the operation, it is possible to stably reproduce the environmental sound and to record and reproduce the sound having a rich atmosphere. The user of the camera 10 does not necessarily listen to the sound in the range of the viewing direction. From the viewpoint of reproduction of memory, audio reproduction that can indulge in memories is preferable to accurate reproduction of sound. It is desired to collect sound that can easily reproduce the sound remaining in the memory. Therefore, in this modification, emphasis is placed on the recording and reproduction of environmental sound that the photographer listens at the time of shooting and remains in memory while performing optimum sound collection. As a result, the sound does not change over quickly, and it is possible to calm down and memorize the memory, and it is possible to reproduce images and sound with a healing effect.

  As described above, in each embodiment of the present invention, the sound collection range is obtained during continuous shooting so that the position of the sound source is constant when a synthesized still image or video is reproduced based on a plurality of images. The sound recording which becomes the sound effect without a sense of incongruity is enabled. In addition, when playing back a composite still image or video based on multiple images, the left / right balance of the sound has been changed so that the position of the sound source is constant, making it possible to play sound with a sound effect that does not feel strange. Yes.

  In each embodiment of the present invention, the movement of the screen is detected based on the image data. However, the present invention is not limited to this. For example, the camera 10 may be detected by an angular velocity sensor or an acceleration sensor provided in the camera 10. Of course, it may be possible to detect the movement of the camera directly.

  In each embodiment of the present invention, the left and right two-channel stereo recording has been described. However, the present invention is not limited to this and can be applied to a multi-channel recording.

  Furthermore, in each embodiment of the present invention, when reproducing and displaying, transmission is performed from the display unit 8 of the camera 10 or the camera 10 to the external device 20. However, the present invention is not limited to this. For example, the recording medium recorded by the recording unit 4 may be directly loaded into a television or personal computer.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone or a personal digital assistant (PDA). In any case, the present invention can be applied to any device for photographing that can record sound together with an image.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a block diagram which shows the structure of the camera concerning 1st Embodiment of this invention, and an external apparatus. It is a figure explaining the use condition of the camera which concerns on 1st Embodiment of this invention, (a) shows the mode at the time of imaging | photography, (b) is a figure which shows a mode that an image is transferred to external apparatuses, such as a television. is there. It is a figure explaining recording of an image and a sound in the camera concerning a 1st embodiment of the present invention. In the first embodiment of the present invention, it is a diagram showing a state where a panoramic image is generated from continuously captured images, (a) is a diagram in which a panoramic image is generated from images of three frames, (b) is each The sound reproduction when directivity sound collection is performed at the center part of the frame is shown, and (c) is a diagram showing the case where sound is picked up in the direction of the image at the center part of the panoramic image. It is a flowchart which shows the operation | movement of the camera control in the camera concerning 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of imaging | photography and sound recording in 1st Embodiment of this invention. It is a figure which shows an image in case of being a person as a main subject in 1st Embodiment of this invention. It is a block diagram which shows the structure of the left-right audio | voice recording part in the camera concerning 1st Embodiment of this invention. 2A and 2B are diagrams illustrating a state where image edge image recording is performed in the camera according to the first embodiment of the present invention, in which FIG. 1A illustrates the positional relationship between the camera and a subject, and FIG. It is a figure which shows each frame of an image. It is a flowchart which shows the operation | movement of right emphasis to left emphasis recording in 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the camera control in 2nd Embodiment of this invention. It is a block diagram which shows the structure of the left-right audio | voice recording part in the camera concerning 2nd Embodiment of this invention. It is a figure which shows a mode that the audio | voice emphasis reproduction | regeneration of the direction of a panoramic center image is performed in the camera concerning 2nd Embodiment of this invention, (a) shows the positional relationship of a camera and a to-be-photographed object, (b) is. It is a figure which shows the panoramic image synthesize | combined with each frame of the continuous image image | photographed with the camera. It is a flowchart which shows the audio | voice emphasis operation | movement of the direction of the panoramic center part image in 2nd Embodiment of this invention. It is a figure explaining recording of an image and sound in a camera concerning the modification of the 1st and 2nd embodiment of the present invention, and (a) shows signs that photography and voice recording are performed by the camera, b) shows an image taken at the position of the camera 10a, and (c) shows an image taken at the position of the camera 10b. It is a figure which shows a mode that a continuous shooting is carried out, moving a camera to a longitudinal direction in the camera concerning the modification of 1st, 2nd embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Signal processing and control part, 2 ... Imaging part, 3 ... Change determination part, 4 ... Recording part, 6 ... Operation determination part, 7 ... Left-right audio | voice recording part, 8 Display unit 9 Clock unit 10 Camera 10a Camera 10b Camera 12 Communication unit 15 User (photographer) 15a .. Eye, 15b ... Eye, 15c ... Ear, 20 ... External device, 21 ... Signal processing and control unit, 22 ... Communication unit, 23 ... Display / playback unit, 24 ... Display priority part, 25 ... Remote control receiving part, 31a ... Angle of view, 31b ... Angle of view, 33a ... Sound collection range, 33b ... Sound collection range, 35 ... Audible Range, 41a ... right microphone, 41b ... left microphone, 42a ... AD converter, 42b ... AD converter, 43a ... add 43b ... adder, 43c ... multiplier, 43d ... adder, 43e ... multiplier, 43f ... adder, 51a-51c ... image, 52a-52c ... Sound source position, 53a to 53c ... sound source, 55 ... tree, 56a to 56c ... image, 57 ... person, 58a-58c ... sound collection range, 59 ... tree, 61a- 61c ... image, 62 ... panoramic image

Claims (13)

An imaging unit for continuously shooting a subject;
A sound collection changer that can change the sound collection range of the sound from the subject direction;
An image combining unit that combines a plurality of images continuously obtained by the imaging unit and generates a combined image;
A control unit that changes a sound collection range of the sound collection change unit when shooting the plurality of images;
A camera characterized by comprising:   The control unit sets the sound collection range from left to right or right according to the case where each image for generating the composite image is obtained from right to left and from left to right, respectively. The camera according to claim 1, wherein the camera is changed from left to right. An imaging unit that captures continuous images while moving the camera field of view to the left and right;
An audio acquisition unit that records audio in multiple directions at the time of shooting,
An image combining unit that combines a plurality of images continuously obtained by the imaging unit and generates a combined image;
When synthesizing the plurality of images, a control unit that changes synthesis of audio in a plurality of directions obtained by the audio acquisition unit;
A camera characterized by comprising:   The camera according to claim 3, wherein the control unit changes voice synthesis in accordance with a change in position of a predetermined subject in each image that generates the synthesized image. An imaging unit for continuously shooting a subject;
A sound collection changer capable of changing the sound collection range of the sound from the direction of the subject;
A control unit that changes a sound collection range according to a case where a plurality of images continuously obtained by the imaging unit are obtained from right to left and a case where the images are obtained from left to right;
A camera characterized by comprising: An imaging unit for continuously shooting a subject;
A sound collection changer capable of changing the sound collection range of the sound from the direction of the subject;
An image combining unit that combines a plurality of images continuously obtained by the imaging unit and generates a combined image;
A control unit that changes the sound collection range of the sound collection change unit in the direction of the image obtained by the imaging unit when photographing the plurality of images;
A camera characterized by comprising: An imaging unit for continuously shooting a subject;
A sound collection changer capable of changing the sound collection range of the sound from the direction of the subject;
A motion determination unit for determining the motion of the camera;
A control unit that changes a sound collection range of the sound collection change unit based on a determination result by the motion determination unit when the continuous shooting is performed;
A camera characterized by comprising:   The camera according to claim 7, wherein the motion determination unit determines based on image data output from the imaging unit.   And a face detection unit that determines whether or not the subject's face exists based on the image data output from the imaging unit. The control unit detects the face by the face detection unit. 8. The camera according to claim 7, wherein a sound collection range of the sound collection change unit is controlled based on the face. A storage unit for storing continuously captured image data and stereo sound data recorded in stereo during the continuous shooting;
A display unit for reproducing and displaying an image based on the image data;
Based on the stereo audio data, an audio playback unit that can be played by changing the left and right balance;
A motion determination unit for determining the motion of the camera;
A control unit that controls the left-right balance of the stereo audio data based on the movement of the camera when reproducing the image data and the stereo audio data;
A playback apparatus comprising:   The control unit obtains a correction angle from the angular velocity of the camera and the timing of each frame of continuous shooting, and controls the left-right balance of the stereo audio data according to the correction angle. The reproducing apparatus according to 10. Stores continuously captured image data and stereo audio data recorded in stereo during this continuous shooting.
Judge the camera movement,
Controlling the left / right balance of the stereo audio data based on the movement of the camera when reproducing the image data and the stereo audio data,
A reproduction method characterized by the above. Stores continuously captured image data and stereo audio data recorded in stereo during this continuous shooting.
Judge the camera movement,
Controlling the left / right balance of the stereo audio data based on the movement of the camera when reproducing the image data and the stereo audio data,
A program characterized by causing a computer to execute the above.