JP5750668B2 - Camera, playback device, and playback method - Google Patents

Description

  The present invention relates to a camera and a playback apparatus, and more particularly, to a camera capable of recording ambient environmental sound during shooting, and a playback apparatus and playback method for a captured image shot with this camera.

  In recent years, large screen televisions have become widespread, and captured images are played back and displayed on large screen televisions. Also, since the image quality of televisions has been improved and the power consumption has been reduced, captured images are displayed like posters and the images can be enjoyed as interiors. In addition, digital photo frames for displaying digital images have become widespread. Thus, recently, life has been colored with photographed images.

  When displaying in an interior sensation, images are not intrusive, and there is a need for something that can heal, such as majestic landscapes and beautiful flower-and-bird styles. It has become.

In addition, when displaying a photographed image with an interior sensation, if the sound recorded at the time of photographing is reproduced, it is further healed. There have been various proposals for recording audio during shooting. For example, Patent Document 1 discloses a video camera recording apparatus that focuses a microphone directivity on a subject in synchronization with zoom lens focusing in order to increase the sense of reality when recording with a video camera. ing. Further, in Patent Document 2, using information in an image knowledge database, an object in an image, a movement (position) of the object, an operation of a camera, and the like are analyzed from a divided image, and the object is emitted. An apparatus and a method are disclosed in which a sound source considered to be separated from sound information and the separated sound source is rearranged in a sound field space suitable for video.
Japanese Patent Laid-Open No. 5-308553 JP 2000-295700 A

  Conventional cameras are supposed to be used for recording voices of children, for example, in athletic meet or school performance. The recording device for a video camera disclosed in Patent Document 1 includes a microphone that controls directivity, and adjusts the directivity of the microphone in synchronization with focusing on the subject so that the sound is matched to the subject. In addition, the apparatus disclosed in Patent Document 2 is audio reproduction for enhancing the sense of reality, and does not record or reproduce environmental sound that is healed.

  It is not always necessary that the image and the sound source coincide with each other in a healing image or sound reproduction that makes you feel something. I often listen to a wide range of environmental sounds even when I look close. For example, in a situation where you are listening to the sea while looking at the sea, even if the viewpoint changes and the image changes, it is not necessary for the sound to change according to the orientation of the face. Also, when watching seashells at the beach, you don't want to hear the sound in the direction of the shells. The user may move his face trying to look over a wide image, but his auditory directivity is much wider and he doesn't have to look at the direction of the sound source every time, unless there is something wrong.

  The present invention has been made in view of such circumstances, and in consideration of the difference between the range that the photographer is viewing and the range that the photographer is listening to, a camera and a playback device that have a rich atmospheric effect It is an object to provide a reproduction method.

In order to achieve the above object, a camera according to a first aspect of the invention captures an object existing in a first direction and outputs first image data, and a sound collection range of sound from the image capturing direction of the object. A sound collection changing unit that can change the image, a second imaging unit that captures an image of a photographer present in a second direction different from the first direction, and outputs second image data; and the second imaging unit. Based on the obtained second image data , the photographer is gazing inside the screen of the display unit that displays the subject based on the first image data, or gazing outside the screen of the display unit. And a control unit that changes the sound collection range of the sound collection change unit according to a gaze state of the photographer with respect to the display unit .

A camera according to a second invention is the face detection according to the first invention, wherein the camera uses the image data output from the imaging unit to determine whether a human face image is included in the image. The face detection unit detects the gaze state of the photographer according to the direction in which the photographer is viewing.

The camera according to a third invention is characterized in that, in the first invention, the control unit further changes the sound collection range of the sound collection change unit depending on whether or not a person is present as the subject. .

The camera according to a fourth invention is the camera according to the first invention, wherein the control unit changes the sound collection range in the sound collection change unit narrowly when the photographer is gazing inside the screen of the display unit. It is characterized by doing.
The camera according to a fifth aspect of the present invention is the camera according to the first aspect, wherein the control unit widely changes the sound collection range in the sound collection change unit when the photographer is not gazing inside the screen of the display unit. It is characterized by doing.

The camera according to a sixth aspect is the camera according to the first aspect , wherein the control unit determines that the viewpoint of the photographer has shifted from the first part to the second part as the gaze state of the photographer. In this case, the sound collection range by the sound collection change unit is once expanded and then narrowed.

According to a seventh aspect of the present invention, there is provided a reproduction method for imaging a subject existing in a first direction, outputting and recording first image data, and collecting a sound collecting range of the subject from the imaging direction. Based on the step of changing by the sound changing unit, the second imaging step of imaging a photographer who exists in a second direction different from the first direction, and outputting the second image data, and the second image data And determining whether the photographer is gazing inside the screen of the display unit displaying the subject or gazing outside the screen of the display unit based on the first image data. Recording a sound collection range based on a gaze state of the photographer with respect to the display unit based on a determination result ; imaging a recording result of the subject; and reproducing a recording result of the sound collection range. With features That.

According to an eighth aspect of the present invention, there is provided a camera according to an eighth aspect that captures an image of a subject existing in a first direction and outputs first image data, an audio recording unit, and a second direction different from the first direction. The second imaging unit obtained by the second imaging unit in order to perform a volume balance control when capturing an image of an existing photographer and outputting second image data and reproducing the imaging result of the imaging unit. Based on two image data, whether the photographer is gazing inside the screen of the display unit that displays the subject based on the first image data or gazing outside the screen of the display unit And a recording unit that records a gaze state of the photographer with respect to the display unit based on the determination result .

Reproducing apparatus according to the ninth invention is a reproducing apparatus for reproducing the imaging result and the sound collecting results of the subject, the photographer the photographer at the time of imaging is obtained from the second imaging result obtained by imaging the object, the display unit or not watching the screen, and a recording unit for recording the gaze condition if it had been watching the outside of the screen of the display unit, according to the gaze condition recorded in the recording unit is reproduced at the time of the imaging result reproduced And a control unit for performing volume balance control of the sound.

According to a tenth aspect of the present invention, there is provided a reproducing method for imaging a subject existing in a first direction and outputting first image data, and a photographer present in a second direction different from the first direction. The photographer displays the subject based on the first image data, based on the second imaging step for capturing the image and outputting the second image data, and the second image data obtained in the second imaging step. The gaze state of whether the user is gazing inside the screen of the display unit or the outside of the screen of the display unit is determined, and the gaze state of the photographer with respect to the display unit is recorded based on the determination result to the recording step, characterized in that it and a control step of performing volume balance control at the time of reproduction of the image data obtained by the imaging step in accordance with the recorded the gaze state in the recording step of

  According to the present invention, it is possible to provide a camera, a playback device, and a playback method that have a rich sound effect in consideration of the difference between the range that the photographer is viewing and the range that the photographer is listening.

  Hereinafter, preferred embodiments using a digital camera to which the present invention is applied will be described with reference to the drawings. The digital camera according to the present embodiment is configured to record sound having a rich sound effect in addition to images. As described above, the camera user does not always listen to the sound in the range of viewing. In the reproduction of the memory, it is desirable to collect the sound so that the sound can be reproduced so that it can be remembered, and the sound in the memory can be reproduced without difficulty, rather than the reproduction of the exact sound direction. In the present embodiment, optimum sound collection is performed according to the shooting scene, and environmental sound that the photographer listens to and remains in memory during shooting is recorded and reproduced. In addition, the voice does not switch quickly, it is possible to calm down and memorize the memory, and it is possible to reproduce images and sounds having a healing effect. Furthermore, in order to convey the atmosphere of the environment at the time of shooting, the difference between the directivity of the photographer's eyes and ears is taken into consideration so that the scene at that time can be remembered both visually and auditorily.

  FIG. 1 is a block diagram showing the configuration of a camera 10 and an external device 20 according to the first embodiment of the present invention. The camera 10 is a digital camera, and includes a signal processing and control unit 1, an imaging unit 2, a face detection unit 3, a recording unit 4, an operation determination unit 6, an audio recording unit 7, a display unit 8, a clock unit 9, and a communication unit. Twelve.

  The signal processing and control unit 1 in the camera 10 includes a signal processing LSI dedicated to the camera 10 and controls the entire camera 10 and performs image processing of image data output from the imaging unit 2. The imaging unit 2 includes a photographing lens 2a (see FIG. 2A), an imaging element that converts a subject image formed by the photographing lens 2a into image data, and the like.

  The recording unit 4 records the image data output from the imaging unit 2 after being subjected to image processing and compression processing by the signal processing and control unit 1. The face detection unit 3 uses the image data output from the imaging unit 2 to determine whether a human face image is included in the image. Further, when a face image is included, the face detection unit 3 can also determine the facial expression of a person by detecting a change in the shadow pattern of the face.

  The audio recording unit 7 has a stereo microphone 7a and records audio around the front. The audio recording unit 7 can process an audio signal from the stereo microphone and change the audio recording range. The audio data output from the audio recording unit 7 is recorded in the recording unit 4 together with the image data after being subjected to signal processing by the signal processing and control unit 1.

  The operation determination unit 6 includes an operation member such as a release button and a switch linked to the operation member. The operation state determined by the operation determination unit 6 is sent to the signal processing and control unit 1, and the signal processing and control unit 1 executes a process according to the operation state. The clock unit 9 has a calendar / time keeping function such as date and time, and outputs shooting date and time information at the time of shooting. The shooting date / time information is recorded in the recording unit 4 together with the image data.

The display unit 8 performs live view display of the subject image for framing based on the image data output from the imaging unit 2, and reproduces and displays the image data recorded in the recording unit 4.
The communication unit 12 performs transmission and reception with an external device 20 such as a television. As communication means, wireless LAN, proximity wireless communication, infrared communication, wired communication using a USB cable or the like can be used, and image data and audio data captured by the camera 10 can be transmitted. In recent years, HDMI and the like tend to be used for transmitting images and sounds to a high-definition display, and the communication unit 12 may include an HDMI terminal and may perform wired communication.

  The external device 20 such as a television or a photo stand includes a signal processing and control unit 21, a communication unit 22, a display / playback unit 23, a display priority unit 24, and a remote control reception unit 25. Similar to the signal processing and control unit 1 of the camera 10, the signal processing and control unit 21 is configured by a signal processing LSI dedicated to the external device 20, etc., and controls the entire external device 20 and received via the communication unit 22. Controls playback and display of image data and audio data.

  The communication unit 22 communicates with the camera 10 and receives image data and audio data from the camera 10. Similar to the communication unit 12 of the camera 10, communication by wireless LAN, proximity wireless communication, infrared communication, USB cable, HDMI cable, or the like is possible. The display priority unit 24 determines the priority of the image. That is, it is determined whether or not the image is a priority image to be displayed first on the display unit 8 built in the camera 10.

  The display / playback unit 23 includes a thin large screen monitor and a speaker, and plays back and displays image data and audio data received from the camera 10. In reproduction, the signal processing and control unit 21 performs reproduction control according to the determination result of whether or not the image is the priority image in the display priority unit 24. In addition, when the external device 20 is a television, a normal television broadcast or the like is also displayed.

  The remote control receiving unit 25 receives an instruction signal from the remote control device by infrared communication. For example, the remote controller can receive, reproduce, or interrupt a designated image or sound from the camera 10.

  Next, a method of using the camera 10 will be described with reference to FIG. As shown in FIG. 2A, the user 15 holds the camera 10 and shoots a subject image through the photographic lens 2a, and can also record audio from the front by the stereo microphone 7a.

  As shown in FIG. 2B, the image and the sound thus captured are transmitted to the external device 20 via the communication unit 12 of the camera 10 and the communication unit 22 of the external device 20. The external device 20 reproduces and displays the received image and sound on the display / reproduction unit 23. The image displayed in this way is not an appreciation like viewing a conventional album, but an image displayed as if it were an interior, as shown in FIG.

  Next, referring to FIG. 3, photographing and sound recording by the camera 10 in the present embodiment will be described. FIG. 3A shows a state in which shooting and audio recording are performed by the camera 10. The user 15 first starts photographing at the position of the camera 10b and moves the camera 10 toward the position of the camera 10a. At this time, when an image is taken at the position of the camera 10a, an image as shown in FIG. 3B is obtained, and when an image is taken at the position of the camera 10b, an image as shown in FIG. 3C is obtained. .

  When a wide seaside scene as shown in FIGS. 3B and 3C is recorded together with continuous shooting or moving image recording, sound in the sound collection ranges 33a and 33b in front of the camera 10 is focused. Will be recorded. However, in this range, it is possible to move only the human eyes 15a and 15b without moving the face. In other words, the camera 10 picks up the sound with a forward sound according to the screen change, but the photographer's ear 15 c may be listening to the sound in the audible range 35.

If the sound recorded under such circumstances changes in a calm manner according to the movement of the camera 10, it is not suitable for enjoying the image and sound as an interior as shown in FIG.
Therefore, in the present embodiment, unless there is a target to be picked up, the directivity of picking up the sound is as wide as possible, and the camera emphasizes environmental sound. Only when the position of the camera 10a in FIG. 3 (a) or the subject person in FIG. 3 (b) is talking, the range of sound collection is narrowed and the voice is recorded.

  Next, the operation in the present embodiment will be described with reference to the flowchart shown in FIG. This flowchart is controlled by the signal processing and control unit 1 of the camera 10.

  If the camera control flow shown in FIG. 4 is entered, it is first determined whether or not the camera is in shooting mode (S101). The camera 10 has a shooting mode and a playback mode. If the result of determination in step S101 is shooting mode, image capture is performed and face detection is performed (S102). In this step, the image data output from the imaging unit 2 for live view display is acquired, and the face detection unit 3 performs face detection using the acquired image data. Subsequently, image display is performed (S103). Here, the subject image is displayed on the display unit 8 based on the image data acquired in step S102. The photographer can perform framing while viewing the subject image.

  Once image display has been performed, it is next determined whether or not a face has been detected (S104). In step S102, face detection is performed. At this time, it is determined in this step whether or not a face portion has been detected from the image. If it is determined that a face has been detected as a result of this determination, the position and expression of the face are determined (S105). Here, the detected face position is used for focusing and exposure control. Further, the sound collection range performed in steps S111 to S113 may be initially set according to the face position.

  If the face position or expression determination is performed, or if it is determined in step S104 that there is no face, it is next determined whether or not to start recording (S106). Here, the operation state of the release button is detected, and it is determined whether to start moving image shooting, panoramic shooting, or the like. If the result of this determination is that recording has not started, processing returns to step S101 and the aforementioned operation is executed.

  If the result of determination in step S106 is that recording has started, it is determined whether or not there is a face larger than a predetermined size or a predetermined number of faces in the center of the screen (S111). Here, in the present embodiment, the face having a predetermined size or more is set to a size of 1/5 or more of the screen width, but may be a size other than this. Since the sound collection range is switched according to the size of the face and the number of faces, the determination value may be any value that can determine whether or not the subject is the main subject.

  If the result of determination in step S111 is that there is a face larger than a predetermined size or a predetermined number of faces in the center of the screen, the sound collection range is narrowed (S112). In this case, the sound collection range is narrowed so that the image can be recorded when the person is the subject and the person speaks something. On the other hand, if the result of determination in step S111 is No, the sound collection range is widened. Here, sound collection is performed with emphasis on surrounding environmental sounds.

  When the sound collection initial setting in steps S111 to S113 is performed, shooting and sound collection are performed (S114). In this subroutine, image and sound are continuously recorded and face detection is performed as needed, and the sound collection range is changed in accordance with the face detection result. The photographing / sound collecting operation continues until an end determination is made in this subroutine. This shooting / sound recording subroutine will be described later using the flow shown in FIG.

  If the result of determination in step S101 is that shooting mode has not been set, it is determined whether or not playback mode has been set (S121). If the result of this determination is that playback mode has not been set, processing returns to step S101. On the other hand, if the result of determination in step S121 is that playback mode has been set, playback is performed (S122). In this step, the captured image recorded from the recording unit 4 is read out, and the image is displayed on the display unit 8 in the thumbnail format. When an image is selected, the image is enlarged and displayed. Further, when the audio data is recorded together with the display of the image, it is reproduced. If no speaker is provided in the camera 10, only image playback is performed and audio playback is not performed.

  Once playback has been performed, it is next determined whether or not to perform transmission (S141). Here, it is determined whether or not an operation member for transmission instruction has been operated in order to transmit an image to the external device 20 such as a television. If the result of this determination is transmission, a display image is transmitted (S142). In this step, the image being displayed in step S122 is transmitted to the external device 20. When a plurality of images are selected, these images may be transmitted together. When the display image is transmitted, if the result of determination in step S141 is not transmission, or when shooting / sound recording is completed in step S114, the flow of camera control is terminated. Returning to, the above-described operation is executed.

  Next, the photographing / sound recording subroutine in step S114 will be described with reference to the flowchart shown in FIG.

  When this flow is entered, it is first determined whether or not a face exists in the center of the screen (S1). If the result of this determination is that a face is present at the center of the screen, it is determined whether or not a predetermined time has elapsed since the face at the center of the screen was recognized (S2). In steps S1 and S2, it is determined whether or not there is a person's face in the center of the screen for a predetermined time. If both determinations are satisfied, the person is considered to be the main subject, and therefore processing for narrowing the sound collection range is performed in step S3 and subsequent steps.

  If the predetermined time has passed as a result of the determination in step S2, it is determined whether or not the current sound collection range is wide (S3). If the sound collection range is wide as a result of this determination, it is next determined whether or not the sound collection range has reached the limit on the narrow range side (S4). If the result of this determination is that the limit has not been reached, it is determined whether or not there has been a change in facial expression (S5). Since the face detector 3 also detects facial expression changes, in this step, it is determined whether or not there is a facial expression change in the face of the person at the center of the screen.

  If the result of determination in step S5 is facial expression change, the sound collection range is narrowed (S7). On the other hand, if the result of determination is that there is no change in facial expression, the sound collection range is gradually narrowed (S6). If there is a change in facial expression such as opening the mouth, the person who is the main subject may speak something, so the sound collection range is quickly narrowed. On the other hand, when there is no change in facial expression, nothing may be spoken, and conversely something may be spoken, so the sound collection range is gradually narrowed.

  If the result of determination in step S1 is that there is no human face in the center of the screen, it is determined whether or not a predetermined time has passed (S11). Here, a scene is assumed in which the subject is matched with another person outside the center of the screen even if no face is present at the center of the screen, and the sound collection range from the audible range to the screen range after the elapse of a predetermined time in step S11. To narrow.

  If the predetermined time has passed as a result of the determination in step S11, it is next determined whether or not the current sound collection range is narrow (S12). If the sound collection range is narrow as a result of this determination, it is determined whether or not the wide-range limit has been reached (S13). If the result of this determination is that the limit has not been reached, the sound collection range is gradually widened (S14). The reason why it is gradually expanded is to suppress a sudden change in sound and to enable sound reproduction suitable for displaying an image as an interior.

  When the result of determination in steps S2, S3, S11, and S12 is No, or when the result of determination in steps S4 and S13 is Yes, or the processing in steps S6, S7, and S14 is performed. Next, it is determined whether or not to finish shooting / sound recording (S10). As described above, in step S106, shooting is started by operating the release button. In this step, it is determined whether or not the operation of the release button is finished.

  If the result of determination in step S10 is not end, processing returns to step S1 and the aforementioned operation is executed. On the other hand, if the result of determination is that the processing has ended, the flow returns to the original flow.

  As described above, the camera 10 according to the present embodiment attaches importance to the collection of environmental sound in an audible range wider than the shooting screen range, and an image that can withstand repeated viewing by suppressing a large change in sound.・ It is possible to shoot audio content. In addition, when a person's face is present on the screen, the direction of sound collection is reduced from the environmental sound so that the person's speech can be recorded. Furthermore, facial expressions are also determined, and if there is a change in facial expressions, the directionality of sound collection is reduced, and recording is possible even when suddenly speaking.

  In the present embodiment, when a person becomes a subject, priority is given to the person's face for statistical reasons that the person often becomes the main subject. However, the present invention is not limited to this, and it is of course possible to determine the pet's face, the state of the birds singing, etc., and in this case, the sound collection directivity may be limited.

  In addition, sound collection that emphasizes environmental sound may be turned on and off. By enabling it to turn on and off, for example, in a scene where the train passes and leaves a lingering sound, the setting for emphasizing environmental sound is canceled and the microphone directivity is narrowed so that the train can be recorded as it passes. Also, the stereo feeling can be emphasized.

  Next, the configuration and operation of the sound recording unit 7 for changing the sound collection range will be described. The audio recording unit 7 includes a stereo microphone 7a, an AD converter 42, and an adder / multiplier 43, as shown in FIG.

  The stereo microphone 7a includes a right microphone 41a and a left microphone 41b, and is disposed on the front side of the camera body 10. The stereo microphone 7a is connected to the AD converter 42, and the audio signal is digitized. That is, the right microphone 41a is connected to the AD converter 42a, and the left microphone 41b is connected to the AD converter 42b to output digital audio data.

  The output terminal of the AD converter 42 is connected to an adder / multiplier 43 to calculate the difference between the left and right sounds. That is, the AD converter 42a that outputs the audio data of the right microphone 41a is connected to the minus input terminal of the adder 43a and the plus input terminal of the adder 43d. The AD converter 42b that outputs the audio data of the left microphone 41b is connected to the plus side input terminal of the adder 43a and the minus side input terminal of the adder 43d.

  The output of the adder 43a is connected to the input terminal of the multiplier 43b, and the output terminal of the adder 43d is connected to the input terminal of the multiplier 43e. Control ends of the multiplier 43b and the multiplier 43e are connected to the signal processing and control unit 1, and input gains of the multipliers 43b and 43e. The input terminal of the adder 43c is connected to the output terminal of the AD converter 42a and the output terminal of the multiplier 43b. The input terminal of the adder 43f is connected to the output terminal of the AD converter 42b and the output terminal of the multiplier 43e.

  The output terminal of the adder / multiplier 43 is an output unit as the audio recording unit 7 and is connected to the recording unit 4. That is, the output terminal of the adder 43c and the output terminal of the adder 43f output right audio data and left audio data, respectively, and each audio data is recorded in the recording unit 4 via these output terminals.

  Thus, the audio recording unit 7 is configured, and the left / right balance of the audio data input in stereo can be controlled, and the directivity of the audio can be narrowed or widened. Audio signals input by the two microphones 41a and 41b of the audio recording unit 7 are converted into digital audio data by the AD converters 42a and 42b, and (left audio data) − (right audio data) is converted by the adder 43a. Then, the adder 43d calculates (right audio data) − (left audio data). That is, the difference between the left and right audio data is calculated by the adders 43a and 43b. Here, the calculated difference is a difference between left and right sounds, and by reducing this difference, the sound at the center can be emphasized, and this addition calculation is a preprocessing for that purpose.

  The differences obtained by the adders 43a and 43d are multiplied by multipliers 43b and 43e based on the gain from the signal processing and control unit 1, respectively, and the multiplication results are added to the right audio data by the adders 43c and 43f. And the left audio data are added respectively. Since the outputs of the adders 43a and 43d are negative, the subtraction is practically performed. For this reason, the left and right audio data output from the adders 43c and 43f is an audio output in which the left and right spread is suppressed. Here, if the gains in the multipliers 43b and 43e are increased, the feeling of spreading can be eliminated, and if the gain is reduced, the feeling of spreading can be increased. The signal processing and control unit 1 can change the sense of spread by controlling the gain for the multipliers 43b and 43e at the timings of steps S6, S7, and S14.

  Thus, in this embodiment, the range of sound collection can be expanded or narrowed using a pair of microphones having the same performance. When the directivity is wide, abundant ambient environmental sounds can be taken in abundantly, and when the directivity is narrow, sound focused on a specific subject can be recorded.

  In addition to providing a pair of microphones having the same performance, for example, a microphone having a wide directivity and a microphone having a narrow directivity may be provided. Further, it is not necessarily stereo at the time of reproduction, and the sound in the center of the screen may be simply emphasized or switched so as not to be emphasized. Further, the recording is not limited to the two-channel stereo on the left and right, and 5.1 ch or the like may be recorded. Further, not only the right and left but also a vertical microphone may be provided.

  As described above, according to the first embodiment of the present invention, it is possible to provide a camera capable of shooting image / sound content that can suppress an unnecessary change in sound accompanying a change in field of view and can reproduce and display calmly. it can. In addition, in order to convey the atmosphere of the environment at the time of shooting, the difference between the directivity of the photographer's eyes and ears is taken into consideration so that the scene at that time can be remembered both visually and auditorily.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment of the present invention, the face in the subject is detected from the image of the camera 10 and the sound is recorded over a wide range, which is sound recording with an emphasis on environmental sound, or the sound of the main subject is recorded. It narrowed the directivity. In the second embodiment, a sub camera (rear camera) that images the photographer is also arranged on the back of the camera, and the directivity is switched in consideration of the gaze direction of the photographer. Since the configuration of the present embodiment largely overlaps with the configuration of the first embodiment, the description will focus on the differences, and the same reference numerals will be given to the same configuration, and description thereof will be omitted.

FIG. 7 is a block diagram illustrating a configuration of the camera 10 according to the second embodiment. The difference from the configuration of the first embodiment shown in FIG. 1 is that it has an imaging unit 2a (rear camera).
The imaging unit 2a includes an optical system and an imaging element, and is disposed on the back surface of the camera 10 as shown in FIG. The imaging unit 2 a captures an image of the face of the photographer 16 and outputs the image to the signal processing and control unit 1 and the face detection unit 3. The face detection unit 3 receives the image data from the imaging unit 2 a and detects the gaze direction of the photographer 16.

  In the second embodiment, as shown in FIG. 8, for example, when a bird's song is heard in another direction 38 instead of the direction 37 in which the camera 10 is directed, the sound collection at the center of the screen is weakened. It is also possible to record the bird's voice effectively with emphasis on environmental sounds. That is, when the photographer 16 detects the face of the photographer 16 by the imaging unit (rear camera) 2a and the face detection unit 3 and the photographer 16 looks at another direction 38, the environmental sound is emphasized. When the photographer 16 wants to see the direction 39 of the display unit 8, the audio recording range is controlled by the audio recording unit 7 so that the directivity of sound collection is narrowed.

  Next, the operation of the present embodiment will be described using the flowchart shown in FIG. The camera control flow shown in FIG. 4 in the first embodiment is common, and only the shooting / sound recording subroutine in step S114 is replaced with FIG. In the flow of FIG. 9, the imaging unit (rear camera) 2a determines where the photographer 16 is interested and switches the sound collection range and direction.

  When the shooting / sound recording flow shown in FIG. 9 is entered, it is first determined whether or not the user is gazing forward (S1a). In this step, since the face detection unit 3 detects the gaze direction of the photographer 16 based on the image data from the imaging unit 2a, the determination is made according to the detection result. As a result of this determination, if the photographer 16 is gazing forward, that is, gazing at the direction 39 of the display unit 8, it is next determined whether or not a predetermined time has elapsed ( S2). In this step, it is determined whether or not a predetermined time has elapsed since the photographer 16 began to gaze at the display unit 8 of the camera 10. In some cases, the photographer 16 merely temporarily looks at the front display unit 8, so it is determined whether or not the user has been gazing for a predetermined time.

  If the result of determination in step S2 is that a predetermined time has elapsed, the photographer 16 was interested in the display unit 8 of the camera 10 for a predetermined time. If there is, narrow the sound collection range according to the field of view. First, it is determined whether or not the current sound collection range is wide (S3). If the sound collection range is wide as a result of this determination, it is next determined whether or not the sound collection range has reached the wide-range limit (S4). If the limit is not reached as a result of this determination, the sound collection range is gradually narrowed to the center of the screen (S6a).

  If the sound collection range is not wide as a result of the determination in step S3, it is determined whether or not the sound collection range is the periphery (S8). For example, when a bird is watched, if the bird flies away and disappears, the photographer looks at the screen monitor of the display unit 8 again. In this case, the determination in step S3 is the sound collection range. Is determined to be narrow. Thereafter, the determination in step S8 is performed. As a result, the sound collection range becomes ambient, and the process proceeds to step S13 described later. In step S14, the sound collection range is gradually widened, and then the sound collection range is gradually narrowed. . On the other hand, if the result of determination in step S8 is that the sound collection range is not surrounding, the process proceeds to step S10.

  If the result of determination in step S1a is that the front has not been watched, that is, if the photographer 16 has not viewed the display unit 8, it is next determined whether or not the sound collection range is narrow (S12). ). When the front is not watched, it indicates that the photographer is interested in the outside of the screen. In step S13 and subsequent steps, the sound collection range is expanded and recording is performed with emphasis on environmental sound. If the result of determination in step S12 is that the sound collection range is narrow, it is determined whether or not it is a wide range limit (S13). If the limit is not reached as a result of this determination, the sound collection range is gradually expanded (S14).

  If the result of determination in step S13 is that the limit has been reached, it is determined whether or not the gaze direction is constant (S21). Here, the face of the photographer 16 is monitored by the imaging unit 2a, and it is determined whether or not the face detection unit 3 continues to look in the same direction as a result of detection. If the gaze direction is constant as a result of this determination, the gaze direction is picked up (S22). In this step, the sound collection in the direction of gaze is emphasized. That is, if the gaze direction is the right side, the sound collection of the right microphone 41a is emphasized, and if the gaze direction is the left side, the sound collection of the left microphone 41b is emphasized.

  If the determinations in Steps S2, S4, S8, S12, and S21 are No, or if the processes in Steps S13 and S22 are executed, it is next determined whether or not the process is finished (S10). Here, as in the first embodiment, the operation state of the release button is detected and a determination is made based on this. If the result of this determination is not end, the process returns to step S1a to perform the above-described operation. On the other hand, if the result of determination is end, the process returns to the original flow.

As described above, in the present embodiment, it is possible to separate the image and the sound and record the image that the photographer has shown and the sound that the photographer wanted to hear. Therefore, even in a scene where the photographer is watching and what is being listened to can be photographed according to the photographer's intention. For example, there is often a situation where a bird standing on another tree is being listened to while the leaves of the tree are shining, and such a situation can be accurately recorded.
It should be noted that the present embodiment and the first embodiment are combined, and for example, even when there is a person in the center of the screen and the photographer is looking at another place, sound collection that emphasizes the center can be performed. Conceivable.

  Next, a third embodiment of the present invention will be described with reference to FIGS. 10 and 11. In the first and second embodiments of the present invention, at the time of shooting, sound is collected between a sound collected in a wide range including ambient sounds (environmental sound emphasis) or a sound collected with a narrow directivity toward the subject. Was controlling. In this third embodiment, recording is performed in stereo at the time of shooting, and at the time of sound playback, whether the playback is performed with emphasis on environmental sound or playback with a narrow directivity toward the subject is determined based on the scene on the screen and the user's gesture. , Switching.

  FIG. 10 is a block diagram showing the configuration of the audio recording unit 7. This audio recording unit 7 adjusts the balance between left and right audio reproduction during reproduction. Compared with the configuration in the first embodiment shown in FIG. 6, the point that the recording unit 4 is connected between the AD converter 42 and the adder / multiplier 43 is different, but the configuration in each circuit is as follows. This is the same as the sound recording unit 7 in the first embodiment.

  That is, the output end of the AD converter 42a that AD converts the audio signal of the right microphone 41a is connected to the recording unit 4, and the audio data AD-converted by the AD converter 42a is output to the adders 43a, 43c, and 43d. . The output end of the AD converter 42b that AD converts the audio signal of the left microphone 41b is connected to the recording unit 4, and the audio data AD-converted by the AD converter 42b is output to the adders 43a, 43d, and 43f. .

  In the first and second embodiments described above, the sound recording unit 7 changes the sound collection range at the time of shooting. In this embodiment, at the time of shooting, the sound signal from the stereo microphone 7a is digitally converted by an AD converter. And the sound data is recorded in the recording unit 4 as it is without changing the sound collection range. The balance of audio reproduction is controlled by the adder / multiplier 43 based on the audio data read from the recording unit 4 during reproduction.

  Next, the operation in this embodiment will be described with reference to the flowchart shown in FIG. Upon entering this flow, steps S101 to S104 are the same as the camera control flow shown in FIG. 4 and will not be described in detail. However, in the shooting mode, image data is captured (S102). Then, live view display is performed using this image data (S103), and face determination is performed (S104).

  As a result of the face determination in step S104, if a face exists, the position of the face is determined (S105b). When the face position is determined, or when the result of the face determination is that there is no face, the release button determines the operation state and determines whether to start recording, as in the first embodiment. Perform (S106). If the result of this determination is that recording has not started, processing returns to step S101 and the aforementioned operation is executed.

  If the result of determination in step S <b> 106 is that recording has started, stereo recording is started with front shooting (S <b> 115). The shooting in this step is a front shooting in which the front direction of the camera 10 is shot, and image data obtained by continuous shooting such as moving image shooting or panoramic shooting is recorded in the recording unit 4. In addition to the image recording, the left and right sounds are separately recorded in the recording unit 4 by the stereo microphone 7a.

  Along with the front photographing and the stereo recording, the rear image feature recording is also performed (S116). Here, based on the image data of the photographer on the back side of the camera 100 output from the imaging unit 2a, the characteristics of the photographer's change, for example, where the user is looking, and the like are detected, and the detected change of the photographer is detected. The feature is recorded in the recording unit 4.

  Subsequently, it is determined whether or not the recording is finished (S119). In this step, the operation state of the release button is detected and a determination is made based on this. If the result of this determination is that recording has not ended, processing returns to step S115 and imaging is continued. On the other hand, if the result of determination in step S119 is that recording has been completed, the camera control flow is terminated, and processing is executed again from step S101.

  If the result of determination in step S101 is not shooting mode, it is determined whether or not playback mode is in effect (S121). If the result of this determination is that playback mode has not been set, processing returns to step S101. On the other hand, if the playback mode is set, it is determined whether or not the photographer is gazing (S131). In step S116, the photographer's gaze direction is recorded using the back image. In this step, whether or not the photographer is gazing at the monitor screen of the display unit 8 based on the recorded gaze direction. Determine whether.

  As a result of the determination in step S131, when the photographer is not gazing, a wide range of sounds is reproduced together with the reproduction of the image (S133). In this step, a wide range of sounds picked up by the stereo microphone 7a is reproduced with emphasis on environmental sounds. On the other hand, if the photographer is gazing as a result of the determination in step S131, sound reproduction with emphasis on the gaze direction is performed along with image reproduction (S132). That is, the left and right volume balance is controlled by the sound recording unit so that the sound source is in the direction of the screen on which the photographer is gazing. With such a method of adjusting the sound during reproduction, it is possible to prevent a problem that the collected sound is not in time when the central person speaks and is delayed.

  Once the audio reproduction is performed in step S132 or S133, it is next determined whether or not the reproduction is finished (S134). If the result of this determination is that playback has not ended, processing returns to step S131 and playback continues. On the other hand, if the result of determination is that playback has ended, it is next determined whether or not to transmit (S141). In this step, it is determined whether or not the image reproduced and displayed in step S121 and after is transmitted to be reproduced and displayed on the external device 20.

  If the result of determination in step S141 is transmission, a display image is transmitted (S142). Here, the selected display image is transmitted to the external device 20 such as a television or a photo stand via the communication unit 12. Thereby, in the situation shown in FIG. 2 (c), it is possible to obtain a content that can decorate an image or sound as an interior. This content also reproduces audio, making it suitable for recollection of memories.

  It should be noted that the user may be able to correct the sound reproduction before transmitting the image, such as by reducing the directivity of the environmental sound or by adjusting the left / right balance of the sound. If the display image is transmitted or if the result of determination in step S141 is not transmission, the camera control flow is terminated, and the process is executed again from step S101.

  As described above, in the third embodiment of the present invention, the sound is recorded in stereo as it is at the time of shooting, and the balance between the left and right volumes and the position of the sound source are controlled during playback. For this reason, even if there is a sudden change in the situation at the shooting location, it is possible to adjust appropriately after shooting.

  Next, a fourth embodiment of the present invention will be described with reference to FIGS. In the first to third embodiments, based on whether or not a face portion is included in the subject image, or the position where the photographer is gazing, the sound collection range, the sound source position at the time of reproduction, etc. Voice control was performed. In the present embodiment, the direction in which the subject is gazing is detected, and voice control is performed based on the detection result.

  For example, assume a scene as shown in FIG. In this scene, the subject 17 is watching the sea and listening to the sound of breaking waves. An image captured at the position of the camera 10a becomes a captured image 32 in consideration of the angle of view 31c. In such a situation, it is desirable to record the sound that the subject 17 is listening to, in this example, the sound of breaking waves.

  Therefore, in this embodiment, the sound that the photographer and the subject 17 are listening to is recorded. If the camera 10a, the user's (photographer's) ear 15c, the subject 17, and the wave 18 being listened to by the subject are in the positional relationship shown in FIG. 12, and the sound collection range 33c is the range shown in the figure, This sound collection range 33c is a range in which the user (photographer) 15 and the subject 17 are listening in common. In the example of FIG. 12, in the case of stereo recording, the right side sound (direction in which the subject 17 is viewed) is emphasized and recorded, or the right side sound is emphasized during reproduction.

  The configuration of the present embodiment may be the same as that of the camera 10 shown in FIG. 1, and the face detection unit 3 has a function of determining the face direction of the subject 17. Since other configurations are the same as those in FIG. 1, detailed description thereof is omitted.

  Next, the operation in the present embodiment will be described with reference to the flowchart of camera control shown in FIG. When this flow is entered, as in the first embodiment, it is first determined whether or not the camera is in the shooting mode (S101). If the camera is in the shooting mode, then the image is captured and face detection is performed (step S101). S102). Subsequently, live view display is performed on the display unit 8 using the captured image (S103), and face determination is performed (S104a).

  If the face is present as a result of the face determination in step S104a, the face position is determined (S105b). In these face detections at the time of shooting, the position of the face of the subject 17 is determined, so that it is easy to determine which face of the subject is facing immediately after shooting, and where the exposure and focus are determined. Make it possible to determine immediately whether to match. For example, if the subject 17 is looking at the camera 10 before shooting, it is easy to detect the face, but if the camera turns to the opposite side of the camera 10 after starting shooting, it becomes difficult to detect the face, so recording starts. It is determined from the front, and the orientation of the face can be recorded even after the recording is started.

  If the face position is determined or if the result of determination in step S104a is that there is no face, it is next determined whether or not recording is started, as in the first embodiment (S106). If the result of this determination is that recording has not started, processing returns to step S101 and the aforementioned operation is executed.

  On the other hand, if the result of determination in step S106 is that recording has started, shooting is then started, stereo recording is performed, and the gaze direction of the subject is recorded (S117). Subsequently, voice-enhanced recording is performed based on the gaze direction of the subject detected by the face detection unit 3 (S118). Although only stereo recording may be used, in the present embodiment, the result of changing the direction of sound collection based on the gaze direction of the subject is also recorded simultaneously. As a result, at the time of reproduction, either audio can be selectively reproduced.

  Next, it is determined whether or not the recording is finished (S119). The start of recording is performed by operating the release button, and the end of recording is also determined based on the operating state of the release button. If the result of this determination is that recording has not ended, shooting, that is, recording of images and sounds is continued. On the other hand, if the result of determination is that recording has ended, shooting is ended, and the process returns to step S101 again to execute the above-described operation.

  If the result of determination in step S101 is not shooting mode, it is next determined whether or not playback mode is in effect (S111). If the result of this determination is not playback mode, processing returns to step S101 and mode determination is repeated. On the other hand, if the result of determination is that it is in playback mode, it is next determined whether or not there is data in the gaze direction (S131). In this step, the reproduction of the image data is started, and it is determined whether or not there is a recording in the gaze direction recorded together with the image in step S117.

  If the result of determination in step S131 is that there is no gaze direction data, audio playback is performed over a wide range, that is, environmental sound is emphasized (S133). On the other hand, when there is data on the gaze direction, reproduction is performed with emphasis on the sound in the gaze direction (S132). As a result, when the person who is the subject recalls the situation at that time, the sound he / she listened to is reproduced, so that the person can easily soak in the memories.

Once audio playback has been carried out, it is next determined whether or not playback has ended (S134). If the result of this determination is that playback has not ended, processing returns to step S131 and playback of images and sounds continues.
On the other hand, if the reproduction has ended, it is next determined whether or not to transmit an image to the external device 20 (S141). If the result of this determination is transmission, a display image is transmitted (S142). As described with reference to the flow of FIG. 9 in the second embodiment, this allows the external device 20 such as a television to enjoy a photographed image like an interior. If the display image is transmitted, or if the result of determination in step S141 is not transmission, the flow of camera control is terminated, and the process is executed again from step S101.

  Next, the configuration of the audio recording unit 7 in the present embodiment will be described with reference to the block diagram shown in FIG. The audio recording unit 7 in this embodiment is the same as FIG. 6 except that the polarities of the adders 43a and 43d are reversed compared to the audio recording unit 7 in the first embodiment shown in FIG. Therefore, the difference will be mainly described. That is, the output end of the AD converter 42a that AD converts the audio signal from the right microphone 41a is connected to the plus side end of the adder 43a and the minus side end of the adder 43d. The output end of the AD converter 42b that AD converts the audio signal from the left microphone 41b is connected to the minus side end of the adder 43a and the plus side end of the adder 43d. The other configuration is the same as that in FIG.

  Because of such a configuration, the difference output from the adders 43a and 43d becomes positive, and the degree of emphasis increases as the gain applied to the multipliers 43b and 43e increases. That is, it is possible to obtain an audio output that emphasizes the left or right spread. Therefore, the left or right sound collection can be increased by increasing the gain of the multiplier 43a or the multiplier 43e.

  In step S118 in the camera control flow in FIG. 13 described above, sound collection according to the gaze direction can be performed by performing control to change the gain of the multiplier 43b or the multiplier 43e according to the gaze direction. it can. Thus, in this embodiment, a pair of microphones having the same performance are provided, and the range of sound collection can be easily biased left and right.

  In the present embodiment, the sound that the subject is listening to at the time of shooting is detected, and this sound is recorded. For this reason, it becomes possible to record and reproduce the sound suitable for the image.

  As described above, in the embodiment of the present invention, the sound that the photographer or the subject is listening to is detected, and this sound is recorded or reproduced. For this reason, an image having a rich sound effect can be reproduced in consideration of the difference between the range that the photographer is viewing and the range that the photographer is listening. In addition, it is possible to calmly remember the state at the time of shooting based on the voice. Furthermore, in order to convey the atmosphere of the environment at the time of shooting, the difference between the directivity of the photographer's eyes and ears is taken into consideration so that the scene at that time can be remembered both visually and auditorily.

  In each embodiment of the present invention, reproduction display is performed by transmitting from the display unit 8 of the camera 10 or the camera 10 to the external device 20. However, the present invention is not limited to this. For example, the recording medium recorded by the recording unit 4 may be directly loaded into a television or personal computer.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone or a personal digital assistant (PDA). In any case, the present invention can be applied to any device for photographing that can record sound together with an image.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a block diagram which shows the structure of the camera concerning 1st Embodiment of this invention, and an external apparatus. It is a figure explaining the use condition of the camera which concerns on 1st Embodiment of this invention, (a) shows a mode at the time of imaging | photography, (b) shows a mode that an image is transferred to external apparatuses, such as a television, (c) is a diagram showing a state in which a transferred image is reproduced and displayed. FIG. 4 is a diagram for explaining recording of images and sound in the camera according to the first embodiment of the present invention, where (a) shows a state where shooting and sound recording are performed by the camera, and (b) shows the camera 10a. The image image | photographed in the position is shown, (c) is a figure which shows the image image | photographed in the position of the camera 10b. It is a flowchart which shows the operation | movement of the camera control in the camera concerning 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of imaging | photography and sound recording in 1st Embodiment of this invention. It is a block diagram which shows the structure of the audio | voice recording part 7 in the camera concerning 1st Embodiment of this invention. It is a block diagram which shows the structure of the camera concerning 2nd Embodiment of this invention. It is a figure which shows the state which uses a camera in 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of imaging | photography and sound recording in 2nd Embodiment of this invention. It is a block diagram which shows the structure of the audio | voice recording part 7 in the camera concerning 3rd Embodiment of this invention. It is a block diagram which shows the structure of the camera concerning 3rd Embodiment of this invention. In the camera concerning 4th Embodiment of this invention, it is a figure explaining recording of an image and an audio | voice, (a) shows a mode that imaging | photography and audio | voice recording are performed with the camera, (b) has a to-be-photographed object. It is a figure which shows the image image | photographed in the position of the camera 10a in the scene. It is a block diagram which shows the structure of the camera concerning 4th Embodiment of this invention. It is a block diagram which shows the structure of the audio | voice recording part 7 in the camera concerning 4th Embodiment of this invention.

DESCRIPTION OF SYMBOLS 1 ... Signal processing and control part, 2 ... Imaging part, 3 ... Face detection part, 4 ... Recording part, 6 ... Operation determination part, 7 ... Audio | voice recording part, 8. ..Display unit, 9 ... clock unit, 10 ... camera, 10a ... camera, 10b ... camera, 12 ... communication unit, 15 ... user (photographer), 15a ... Eyes, 15b ... eyes, 15c ... ears, 16 ... photographer, 17 ... subject, 18 ... wave that subject is listening to, 20 ... external device, 21 ... Signal processing and control unit, 22 ... communication unit, 23 ... display / playback unit, 24 ... display priority unit, 25 ... remote control receiving unit, 31a ... angle of view, 31b ... image Angle, 31c ... Angle of view, 32 ... Captured image, 33a ... Sound collection range, 33b ... Sound collection range, 33c ... Sound collection range, 35 ... Audible range , 37 ... direction, 38 ... direction, 39 ... direction, 41a ... right microphone, 41b ... left microphone, 42a ... AD converter, 42b ... AD converter, 43a ... Adder, 43b ... adder, 43c ... multiplier, 43d ... adder, 43e ... multiplier, 43f ... adder

Claims (10)

An imaging unit that images a subject existing in a first direction and outputs first image data;
A sound collection changer capable of changing a sound collection range of the sound from the imaging direction of the subject;
A second imaging unit that captures an image of a photographer present in a second direction different from the first direction and outputs second image data;
Based on the second image data obtained by the second imaging unit, the photographer is gazing inside the screen of the display unit that displays the subject based on the first image data, or of the display unit. A control unit that determines a gaze state as to whether or not the user is gazing outside the screen, and changes the sound collection range of the sound collection change unit according to a gaze state of the photographer with respect to the display unit ;
A camera characterized by comprising: The camera includes a face detection unit that determines whether a human face image is included in an image using image data output from the imaging unit,
The camera according to claim 1, wherein the face detection unit detects the gaze state of the photographer according to a direction in which the photographer is looking.   The camera according to claim 1, wherein the control unit further changes the sound collection range of the sound collection change unit depending on whether a person is present as the subject. The control unit narrowly changes the sound collection range in the sound collection change unit when the photographer is gazing inside the screen of the display unit .
The camera according to claim 1 . The control unit widely changes the sound collection range in the sound collection change unit when the photographer is not gazing inside the screen of the display unit .
The camera according to claim 1 .   When the control unit determines that the photographer's viewpoint has moved from the first part to the second part as the gaze state of the photographer, the sound collection range by the sound collection change unit is temporarily The camera according to claim 1, wherein the camera is narrowed after being widened. An imaging step of imaging a subject existing in a first direction and outputting and recording first image data;
Changing the sound collection range of the sound from the imaging direction of the subject by a sound collection change unit;
A second imaging step of imaging a photographer present in a second direction different from the first direction and outputting second image data;
Whether the photographer is gazing inside the screen of the display unit that displays the subject based on the first image data, or is gazing outside the screen of the display unit based on the second image data Determining a gaze state, and recording a sound collection range based on the gaze state of the photographer with respect to the display unit based on the determination result ;
Replaying the recording and recording result of the subject and the recording result of the sound collection range ;
A reproduction method characterized by comprising: An imaging unit that images a subject existing in a first direction and outputs first image data;
An audio recording unit;
A second imaging unit that captures an image of a photographer present in a second direction different from the first direction and outputs second image data;
In order to perform volume balance control when reproducing the imaging result of the imaging unit, based on the second image data obtained by the second imaging unit, the photographer can use the subject based on the first image data. Whether or not the user is gazing inside the screen of the display unit or gazing outside the screen of the display unit, and based on the determination result, the gaze state of the photographer with respect to the display unit A recording section for recording
A camera characterized by comprising: In a playback device that plays back the imaging results and sound collection results of a subject,
Records a gaze state indicating whether the photographer was gazing inside the screen of the display unit or gazing outside the screen of the display unit, which was obtained from the second imaging result obtained by imaging the photographer during the imaging of the subject. A recording section ;
A control unit that performs volume balance control of audio reproduced when reproducing the imaging result according to the gaze state recorded in the recording unit ;
A playback apparatus comprising: An imaging step of imaging a subject existing in a first direction and outputting first image data;
A second imaging step of imaging a photographer present in a second direction different from the first direction and outputting second image data;
Based on the second image data obtained in the second imaging step, the photographer is gazing inside the screen of the display unit that displays the subject based on the first image data, or the screen of the display unit A step of determining whether or not the user is gazing outside, and, based on the determination result, a recording step of recording the gazing state of the photographer on the display unit ;
A control step for performing volume balance control during reproduction of the image data obtained in the imaging step according to the gaze state recorded in the recording step ;
A reproduction method characterized by comprising: