JP6167483B2 - Reproduction processing device, imaging device, and reproduction processing program - Google Patents

Description

  The present invention relates to a reproduction processing device, an imaging device, and a reproduction processing program.

  Conventionally, the direction of the sound source of the sound can be acquired from the sound recorded using a plurality of microphones.

  For example, a digital camera having a plurality of microphones captures still images and moving images and records audio, estimates the direction of each sound source and the position in the image, and obtains sound data and position data for each sound source by direction. There is a technique of generating and recording in association with image data (see Patent Document 1).

JP 2009-239348 A

  However, according to the conventional technology, at the same time as reproducing and displaying an image, the sound data of each sound source is simply reproduced by reproducing all the sound sources or sound data for each sound source according to the scene of the image, the photographer or the viewpoint of the subject, etc. It cannot be played back with effects and effects.

  In view of the problems of the above-described conventional technology, an object of the present invention is to provide a technology capable of reproducing audio data for each sound source with effects and effects.

An aspect of a reproduction processing apparatus illustrating the present invention includes an input unit that reads image data and audio data recorded during image data generation processing, a subject detection unit that detects a main subject from the image data, and a detected main component Obtains information about the distance to the subject from the image data, and generates playback control data that controls the volume of the audio data during playback based on the information about the distance , the direction of the sound source, and whether the sound source is within the angle of view. A control unit.

In addition, the control unit obtains the sound source sound data by obtaining the direction of at least one sound source from the sound data, and determines the sound source sound data based on the sound source direction and distance information and whether the sound source is within the angle of view. The reproduction control data may be generated by weighting.

In addition, the control unit determines the direction of each of the plurality of sound sources from the sound data, generates sound data of each sound source, and determines each sound source based on information on the direction and distance of each sound source and whether the sound source is within the angle of view. The reproduction control data may be generated by weighting.

The image processing apparatus further includes a scene determination unit that determines a scene of the image data, and the control unit weights the sound source based on information on the direction and distance of the sound source and whether or not the sound source is within the angle of view and the determination result of the scene determination unit. You may do.

The image data includes a plurality of frames that are continuously captured in time series, and the control unit detects a motion associated with panning of the imaging device when capturing the image data, and calculates the motion vector of the frame. The control unit may further include a motion vector calculation unit, and the control unit may weight the sound source in each frame based on the information on the direction and distance of the sound source and whether the sound source is within the angle of view and the motion vector of each frame. The image data includes a plurality of frames that are continuously captured in time series, and the control unit further includes a motion vector calculation unit that detects a motion of the main subject and calculates a motion vector of the main subject. The information about the direction and distance of the sound source, whether the sound source is within the angle of view, and the motion vector of the main subject The sound source may be weighted.

  In addition, the control unit may perform weighted averaging of the sound source in the frame and the weight of the sound source in the adjacent frame.

  In addition, a speaker unit that outputs voice data to the outside may be provided, and the control unit may control the volume of the voice data based on the reproduction control data and output the voice to the speaker unit.

  Moreover, you may provide the display part which displays image data.

  An aspect of an imaging device illustrating the present invention includes an imaging unit that captures an object scene and generates image data, a microphone unit that receives audio and generates audio data, and a reproduction processing device of the present invention. Is provided.

An aspect of a reproduction processing program illustrating the present invention includes an input procedure for reading image data and audio data recorded during the image data generation process, a subject detection procedure for detecting a main subject from the image data, and a detected main subject. Control to generate playback control data that controls the volume of audio data during playback based on distance information , the direction of the sound source, and whether or not the sound source is within the angle of view Make the computer execute the procedure.

  According to the present invention, audio data for each sound source can be reproduced with effects and effects.

The figure which shows the structure of the computer which concerns on one Embodiment of this invention. The flowchart which shows the reproduction | regeneration processing by the computer which concerns on one Embodiment A figure showing an example of the object scene The figure which shows the structure of CPU in the computer which concerns on other embodiment of this invention. The flowchart which shows the reproduction processing by the computer which concerns on other embodiment. Diagram showing an example of continuous frames of a video Diagram showing an example of a frame motion vector The figure which shows the list of the weighting factor which is set according to the playback mode The figure which shows an example of the motion vector of the main subject

<< One Embodiment >>
FIG. 1 is a diagram showing a configuration of a computer 100 that is operated as a reproduction processing apparatus according to an embodiment of the present invention.

  A computer 100 shown in FIG. 1A includes a CPU 10, a storage unit 11, an input / output interface (input / output I / F) 12, and a bus 13. The CPU 10, the storage unit 11, and the input / output I / F 12 are connected via the bus 13 so that information can be transmitted. The computer 100 is connected to an output device 14 for displaying the progress of image processing and processing results, and an input device 15 for receiving input from the user, via the input / output I / F 12. As the output device 14, a general liquid crystal monitor, a printer, or the like can be used. As the input device 15, a keyboard, a mouse, or the like can be appropriately selected and used. Note that the computer 100 of the present embodiment also has a speaker as the output device 14. The input / output IF 12, together with the output device 14 and the input device 15, connects a digital camera (not shown) with a universal serial bus (USB) cable or a memory card (not shown) attached to the digital camera (not shown). It shall have a slot for inserting (not shown).

  An image to be processed by the computer 100 of the present embodiment is captured by a digital camera (not shown) or the like, and a still image in which audio data obtained by stereo-recording audio around the scene at the time of imaging is added to the header area or It is a video. In the still image or moving image header area, the subject distances of at least all main subjects imaged in the still image or moving image, together with the exposure conditions (focal length, aperture value, shutter speed, ISO value, etc.) at the time of image capturing are included. Assume that the information of D is included.

  The CPU 10 is a processor that comprehensively controls each unit of the computer 100. For example, the CPU 10 reads a reproduction processing program stored in the storage unit 11 based on an instruction from the user received by the input device 15. The CPU 10 operates as the subject detection unit 20 and the scene determination unit 21 by executing the reproduction processing program (FIG. 1B), and performs reproduction processing on the processing target image. The CPU 10 reproduces and displays still images and moving images on the liquid crystal monitor of the output device 14 and outputs the sound of the added sound data to the speaker.

  The subject detection unit 20 performs subject detection processing on each frame of a still image or moving image to be processed, and detects a human face region or an image region such as a building or a car. This subject detection process is performed by a known algorithm. For example, the subject detection unit 20 uses a template of various patterns such as a person or a building stored in the storage unit 11 to perform pattern matching processing on a still image or a frame to detect an image area of the main subject. To do. The CPU 10 acquires the size and position of the detected image area of the main subject as subject information.

  The scene determination unit 21 determines a scene in the scene captured in a still image or a frame using a known method. In addition, when the scene determination unit 21 has scene mode information at the time of imaging in the header area of the read still image or moving image, the scene determination unit 21 determines a scene based on the scene mode information.

  The storage unit 11 records a control program, a reproduction processing program, and the like, and an image read from a digital camera (not shown). Programs, images, and the like stored in the storage unit 11 can be appropriately referred to from the CPU 10 via the bus 13. A storage device such as a general hard disk device or magneto-optical disk device can be selected and used for the storage unit 11. Although the storage unit 11 is incorporated in the computer 100, it may be an external storage device. In this case, the storage unit 11 is connected to the computer 100 via the input / output I / F 12.

  Next, playback processing by the computer 100 of the present embodiment will be described with reference to the flowchart of FIG. In this embodiment, the image to be processed is a still image in which the object scene 40 including the person 30 as shown in FIG. However, FIG. 3B shows the object scene 40 after the still image is captured, and the car 32 is outside the object scene 40 at the time of image capturing, but is approaching the person 30. .

  The user uses the input device 15 to input a command for the playback processing program or double-click the program icon displayed on the output device 14 to instruct the CPU 10 to start the playback processing program. The CPU 10 receives the instruction via the input / output I / F 12 and reads and executes the reproduction processing program stored in the storage unit 11. CPU10 starts the process from step S101.

  Step S101: The CPU 10 reads a still image designated by the user via the input device 14 from a digital camera (not shown) via the storage unit 11 or the input / output I / F 12.

  Step S102: The subject detection unit 20 detects the face area of the person 30 and the image area of the building 31 from the read still image. The CPU 10 acquires, as subject information, the size and position of the detected image area of each main subject together with the subject distance D of each main subject added to the header area of the still image.

  Step S103: The CPU 10 obtains the direction of the sound source of each sound included in the sound data from the sound data of the stereo recording added to the still image using a known method such as Patent Document 1, and the sound for each sound source. The signal is extracted to generate audio data for each sound source. In this embodiment, there are sound sources of the sound of the person 30, music flowing from the building 31, and engine sound of the approaching automobile 32 as the sound source, and the CPU 10 generates sound data of these sound sources. Further, the CPU 10 specifies where the sound sources are located on the still image based on the subject information and the direction of each sound source, and generates position information including the direction of each sound source and the specified position. The CPU 10 records audio data and position information for each sound source in an internal memory (not shown) in association with the subject information.

  As shown in FIG. 3A, since the automobile 32 is not within the angle of view of the still image, the size and position of the image area of the subject information of the automobile 32 and the subject distance D (i) are the angle of view. It is assumed that it is set to a predetermined value indicating outside, infinity, or the like.

  Step S104: The CPU 10 determines whether or not each sound source exists within the angle of view of the still image in order to set the sound volume at the time of reproducing the sound data of each sound source according to the scene captured in the still image described later. Is determined based on the angle of view of the still image and the position information of each sound source. When the sound source is within the angle of view, the CPU 10 sets a flag UseFlag (i) of the sound source to 1. On the other hand, when the sound source is outside the angle of view, the CPU 10 sets the flag UseFlag (i) to 0. Here, the coefficient i indicates each sound source, and in this embodiment, the person 30 is i = 0, the building 31 is i = 1, and the car 32 is i = 2.

Step S105: The scene determination unit 21 determines a scene captured in a still image, and the CPU 10 weights each sound source during reproduction according to the determination result of the scene determination unit 21. The scene determined by the scene determination unit 21 according to the present embodiment is any one of “snap”, “portrait”, and “landscape”. The setting of the weighting of each sound source by the CPU 10 in each scene will be described.
A) In the case of “snap” The CPU 10 weights each sound source at the time of reproduction according to the distance L on the still image from the in-focus area at the time of capturing the still image, for example, the face area of the person 30. The coefficient WT (i) is set as α / L. The coefficient α is set to a predetermined value. When the sound source is outside the angle of view of the still image, that is, the distance L of the automobile 32 is set to a predetermined value, infinity, or the like.
B) “Portrait” The CPU 10 sets the sound source weight coefficient WT (i) from the person to the largest value. For example, the CPU 10 sets the weighting factor WT (i) of the person 30 to 1, and sets the weighting factor WT (i) of the building 31 and the automobile 32 to 0.5.
C) In the case of “scenery” The CPU 10 sets the weight coefficient WT (i) of the sound source from a person other than the person to a large value. For example, the CPU 10 sets the weighting factor WT (i) of the person 30 to 0.2, and sets the weighting factor WT (i) of the building 31 and the automobile 32 to 1.

  Note that the value and setting method of the weighting factor WT (i) in each scene are examples, and may be set by other values or other setting methods. For example, in the case of “portrait”, the weight coefficient WT (i) of the building 31 or the car 32 may be set so as to be inversely proportional to the distance to the person 30.

Step S106: The CPU 10 calculates the amplification factor AMP (i) for determining the volume of each sound source at the time of reproduction based on the following equation (1) using the weight coefficient WT (i) of each sound source set in step S105. And generated as reproduction control data.
AMP (i) = UseFlag (i) × WT (i) / (β × D (i)) (1)
Here, the coefficient β is a coefficient that normalizes the subject distance D (i) of each main subject.

  Step S107: The CPU 10 calculates the product of the sound data and the amplification factor AMP (i) of the reproduction control data for each sound source, and generates sound data for reproduction of each sound source. The CPU 10 reproduces and displays the still image on the liquid crystal monitor of the output device 14 and outputs the sound data for reproduction of each sound source as sound to the speaker of the output device 14. The CPU 10 ends a series of processes.

  When the image to be processed is a moving image, the computer 100 performs the reproduction process shown in FIG. 2 on each frame of the moving image. That is, the computer 100 performs the processing from step S102 to step S105 on all the frames of the moving image, and then proceeds to step S106, calculates the amplification factor AMP for each sound source in each frame, and generates reproduction control data. To do.

As described above, in this embodiment, by setting the weighting coefficient of each sound source according to the scene of the captured image, it is possible to reproduce the sound data of each sound source with effects and effects.
<< Other embodiments >>
A computer according to another embodiment of the present invention is the same as the computer 100 according to one embodiment shown in FIG. 1, and a detailed description of each component will be omitted.

  The difference between the computer 100 of this embodiment and that of the one embodiment is that 1) the image to be processed is only a moving image, and 2) the computer 100 is a photographer viewpoint or a subject viewpoint in reproducing a moving image. Accordingly, a reproduction mode for setting the weighting coefficient WT of the sound data of each sound source is provided. Here, the photographer viewpoint mode in the reproduction mode is a mode in which the sound data of each sound source is reproduced with sound heard by the photographer of the digital camera (not shown), and the subject viewpoint mode is the position of the main subject. In this mode, the sound data of each sound source is reproduced with sound that can be heard. Further, as the image to be processed is only a moving image, the CPU 10 of the present embodiment executes a reproduction program so that the motion vector calculation unit 22 is combined with the subject detection unit 20 as shown in FIG. Operate.

  The vector calculation unit 22 detects a motion associated with panning of a digital camera (not shown) that has captured a moving image as a frame motion, calculates a motion vector of the frame, detects a motion of the main subject, and calculates a motion vector of the main subject. To do. Specifically, the motion vector calculation unit 22 performs a known correlation process on two adjacent frames of the moving image. Based on the correlation result, for example, the motion vector calculation unit 22 detects the frame motion from the amount of deviation between two frames in the background image region excluding the main subject image region detected by the subject detection unit 20. The motion vector of the frame is calculated. On the other hand, the motion vector calculation unit 22 detects the movement of the main subject based on the amount of deviation of the background image area and the amount of deviation of the image area of the main subject, and calculates the motion vector of the main subject.

  Next, the reproduction process of the computer 100 of this embodiment will be described with reference to the flowchart of FIG. Note that the moving image to be processed in the present embodiment is an image of the scene 40 shown in FIG. FIGS. 6A to 6C show three consecutive frames as an example among the frames of the moving image. That is, FIGS. 6A to 6C show the k-th, k + 1-th, and k + 2-th frames (k is a natural number). However, FIGS. 6A and 6B are frames obtained by panning a digital camera (not shown) so as to follow the person 30. Further, FIG. 6C shows a frame obtained by panning a digital camera (not shown) so that the automobile 33 appearing from the left side becomes the center of the image.

  The user uses the input device 15 to input a command for the playback processing program or double-click the program icon displayed on the output device 14 to instruct the CPU 10 to start the playback processing program. The CPU 10 receives the instruction via the input / output I / F 12 and reads and executes the reproduction processing program stored in the storage unit 11. CPU10 starts the process from step S201.

  Step S201: The CPU 10 reads a moving image designated by the user via the input device 14 from a digital camera (not shown) via the storage unit 11 or the input / output I / F 12. Note that the CPU 10 preferably accepts designation of a reproduction mode along with designation of a moving image to be reproduced.

  Step S202: The subject detection unit 20 detects an image region such as the person 30 from each frame of the read moving image. The CPU 10 acquires, as subject information, the size and position of the detected image area of each main subject together with the subject distance D of each main subject added to the header area of the moving image.

  Step S203: The CPU 10 obtains the direction of the sound source of each sound included in the sound data from the sound data of the stereo recording added to the moving image using a known method such as Patent Document 1, and the sound signal for each sound source. To generate audio data of each sound source. Further, the CPU 10 specifies where the sound sources are located on each frame based on the subject information and the direction of each sound source, and position information including the direction of each sound source and the specified position in each frame. Generate. The CPU 10 records audio data and position information for each sound source in an internal memory (not shown) in association with the subject information.

  Step S204: The motion vector calculation unit 22 performs correlation processing on two adjacent frames, detects the frame motion from the amount of deviation in the background image region, and calculates the motion vector of the frame. The motion vector calculation unit 22 calculates a motion vector of each main subject from the amount of deviation of the background image area and the amount of deviation of the main subject in the image area. The CPU 10 records the calculated frame and the motion vector of each main subject in an internal memory (not shown) in association with each frame.

  Step S205: The CPU 10 determines whether or not the photographer viewpoint mode is set as the reproduction mode. When the photographer viewpoint mode is set, the CPU 10 proceeds to step S206 (YES side), and when the subject viewpoint mode is set, the CPU 10 proceeds to step S207 (NO side).

  Step S206: In the case of the photographer viewpoint mode, the CPU 10 reproduces the sound data of each sound source with the sound heard by the photographer of the digital camera (not shown), for example, the motion vector of the frame in the mth frame Based on the position information of each sound source, the weight coefficient WT (m, i) of each sound source in the mth frame is set. Specifically, it is set as follows.

  FIGS. 7A to 7C show the directions of the motion vectors of the frames in the kth, k + 1th, and k + 2th frames shown in FIGS. 6A to 6C by arrows at the centers of the respective frames. FIG. 8A shows a list of weighting factors WT (m, i) of each sound source set in each frame based on the direction of the motion vector of the frame and the position information of each sound source. In the photographer viewpoint mode, a larger weight coefficient is set for a sound source that matches the direction of the motion vector of the frame and is closer to the center of the frame.

  That is, the kth and (k + 1) th frames are captured so that the person 30 comes to the center of the frame, so that the weight coefficient of the person 30 is the largest value as shown in FIG. Set to Further, since the building 31 and the car 32 are on the side where the motion vector of the frame is directed and approaches the person 30, the weight coefficient of the building 31 and the car 32 is larger in the k + 1th frame than in the kth. Set to On the other hand, since the car 33 is outside the angle of view in the k-th and k + 1-th frames and on the side opposite to the direction of the motion vector of the frame, the weight coefficient is set to a small value.

  On the other hand, the k + 2th frame is an image obtained by panning a digital camera (not shown) so that the car 33 is at the center of the frame, so the weight coefficient of the car 33 is set to the largest value. The On the other hand, since the person 30, the building 31, and the car 32 are on the opposite side of the direction of the motion vector of the frame, the respective weighting factors are set to be smaller values than those of the kth and k + 1th frames.

  Note that the value of the weighting factor WT (m, i) for each sound source in each frame shown in FIG. 8A and the method of setting the value are examples, and the number of main subjects, the size and direction of the motion vector of the frame, and the like. It is preferable to set appropriately.

  Step S207: In the subject viewpoint mode, for example, in order to reproduce the sound data of each sound source with sound that the person 30 listens, based on the direction of the motion vector of the person 30 and the position information of each sound source, The weight coefficient WT (m, i) of each sound source in the mth frame is set. Specifically, it is set as follows.

  9A to 9C show the direction of the motion vector of the person 30 in the k-th, k + 1-th, and k + 2-th frames shown in FIGS. An arrow is shown at the center of each frame. FIG. 8B shows a list of weighting factors WT (m, i) of each sound source set based on the direction of the motion vector of the person 30 and the position information of each sound source in each frame. In the subject viewpoint mode, a larger weight coefficient is set for a sound source that matches the direction of the motion vector of the person and is closer to the person 30.

  That is, the motion vectors of the person 30 in the kth to (k + 2) th frames are in the same direction, and the building 31 and the car 32 approach the person 30. Therefore, as shown in FIG. The weighting factor is set to a large value according to the kth to k + 2th frames. On the other hand, since the car 33 is outside the angle of view in the k-th and k + 1-th frames, the weighting coefficient is set to a small value of 0.1. The car 33 is close to the person 30 within the angle of view of the (k + 2) th frame, but is opposite to the direction of the motion vector of the person 30 and is therefore set to a value smaller than the weight coefficient of other sound sources. Is done.

  In the subject viewpoint mode, the sound data of each sound source is reproduced with sound that the person 30 hears, so that the sound of the person 30 is reproduced at a low volume as shown in FIG. For example, a weighting factor having a predetermined value such as 0.5 is set in advance.

  Further, the value of the weighting coefficient WT (m, i) for each sound source in each frame shown in FIG. 8B and the method of setting the value are examples, and the number of main subjects, the size and direction of the motion vector of the main subject, etc. It is preferable to set appropriately according to the above.

  Step S208: The CPU 10 uses the following equation (2) to calculate the weight coefficient WT (m, i) of each sound source in the mth frame set in Step S206 or Step S207, and the adjacent N including the mth frame. The weighting coefficients in the frames (<total number of frames of moving image) are weighted and averaged in the time axis direction.

Here, the coefficient ε (j) indicates the degree to which the weight coefficient of the sound source in the j-th frame contributes to the m-th frame, and is set to contribute most when j = m. Further, the number N of frames to be used is about 10 or less, and the weighted average range j is appropriately selected from m to m + N-1, m-N + 1 to m, m-N / 2 to m + N / 2, and the like. It is preferable. With this weighted average, the CPU 10 can smoothly change the volume of each sound source.

Step S209: The CPU 10 uses the weighting coefficient <WT (m, i)> weighted and averaged in step S208 to calculate the amplification factor AMP (m, i) that determines the volume of each sound source in each frame at the time of reproduction. Reproduction control data is generated by calculation using equation (3).
AMP (m, i) = <WT (m, i)> / (β × D (m, i)) (3)
Here, the coefficient β is a coefficient that normalizes each subject distance D (m, i) in the m-th frame.

  Step S210: The CPU 10 calculates the product of the sound data and the amplification factor AMP (m, i) of the reproduction control data for each sound source, and generates sound data for reproduction of each sound source.

  Step S211: The CPU 10 reproduces and displays on the liquid crystal monitor of the output device 14, and outputs the sound data for reproduction of each sound source to the speaker of the output device 14 as sound. The CPU 10 ends a series of processes.

As described above, in this embodiment, by setting the weighting coefficient of each sound source in accordance with the frame in each captured frame or the motion vector of the main subject, the sound data of each sound source is reproduced with effects and effects. be able to.
<< Additional items of embodiment >>
(1) Although the reproduction processing apparatus of the present invention is realized by causing the computer 100 to execute a reproduction processing program, the present invention is not limited to this. The present invention can also be applied to a reproduction processing program for realizing the processing in the reproduction processing apparatus according to the present invention by the computer 100 and a medium on which the program is recorded.

  The present invention can also be applied to a digital camera having the reproduction processing program of the present invention. When the digital camera operates as the image processing apparatus of the present invention, the CPU 10 may realize each process of the subject detection unit 20, the scene determination unit 21, and the motion vector calculation unit 22 by software, or ASIC. Each of these processes may be realized by hardware using In this case, it is preferable that subject information, audio data, position information, and reproduction control data are added to the header region of a still image or moving image captured by a digital camera along with the exposure conditions. Note that the audio data added to the header area may be extracted for each sound source or may be recorded in stereo before extraction.

  (2) In the above embodiment, the computer 100 reproduces and displays still images and moving images as well as the sound data of each sound source. However, the present invention is not limited to this. Good.

  (3) In the above embodiment, the weighting coefficient of each sound source is set according to the scene determination result or the playback mode, and the playback control data and the playback audio data are generated. However, the present invention is not limited to this. For example, the CPU 10 may generate the reproduction control data and the reproduction audio data in consideration of the Doppler effect and the like based on the frame and the motion vector of the main subject.

  (4) In the other embodiment described above, the motion vector calculation unit 22 calculates the motion vector of the frame and each main subject based on the correlation processing for two adjacent frames, but the present invention is not limited to this. For example, in the case of a moving image compressed in a moving image format such as H.264, a motion vector is calculated in motion compensation in inter-frame prediction in order to increase compression efficiency. Therefore, the motion vector calculation unit 22 may obtain the motion vector of the frame and each main subject based on the motion vector.

  Alternatively, the motion vector may be calculated using the motion of the subject detected by the subject tracking function.

  When the digital camera (not shown) includes a sensor such as an acceleration sensor or an electronic gyro, the motion vector calculation unit 22 may calculate a frame motion vector based on the output value of the sensor.

  (5) In the above-described other embodiments, the main subject to be followed is the person 30 in the subject viewpoint mode. However, the present invention is not limited to this, and the main subject such as the building 31 or the car 32 may be followed. .

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

10 CPU, 11 storage unit, 12 input / output I / F, 13 bus, 14 output device, 15 input device, 20 subject detection unit, 21 scene determination unit, 22 motion vector calculation unit, 100 computer

Claims (11)

An input unit for reading image data and audio data recorded during the image data generation process;
A subject detection unit for detecting a main subject from the image data;
Information about the detected distance to the main subject is acquired from the image data, and based on the information about the distance , the direction of the sound source, and whether the sound source is within the angle of view, A control unit for generating playback control data for controlling the volume;
A reproduction processing apparatus comprising: The reproduction processing apparatus according to claim 1,
The control unit, the seeking direction of the sound source of at least one of the audio from the audio data to generate audio data of the sound source, the direction of the sound source, the information and the sound source regarding the distance whether the angle A reproduction processing apparatus that generates the reproduction control data by weighting the sound source based on the reproduction control data. The reproduction processing apparatus according to claim 2, wherein
Wherein the control unit generates an audio data from the seeking direction of each of the plurality of the sound source of each of the sound source audio data, the direction of each sound source, the information and the sound source regarding the distance whether the angle A reproduction processing apparatus that generates the reproduction control data by weighting each sound source based on the reproduction control data. In the reproduction | regeneration processing apparatus of Claim 2 or Claim 3,
A scene determination unit for determining a scene of the image data;
The reproduction processing apparatus, wherein the control unit weights the sound source based on the direction of the sound source, information on the distance, and whether or not the sound source is within an angle of view and a determination result of the scene determination unit. In the reproduction | regeneration processing apparatus of Claim 2 or Claim 3,
The image data consists of a plurality of frames imaged continuously in time series,
The controller is
A motion vector calculating unit that detects a motion associated with panning of the imaging device when capturing the image data and calculates the motion vector of the frame;
The control unit weights the sound source in each frame based on the direction of the sound source, information on the distance, and whether or not the sound source is within an angle of view, and a motion vector of each frame. In the reproduction | regeneration processing apparatus of Claim 2 or Claim 3,
The image data consists of a plurality of frames imaged continuously in time series,
The controller is
A motion vector calculating unit for detecting a motion of the main subject and calculating a motion vector of the main subject;
The reproduction processing apparatus weights the sound source in each frame based on the direction of the sound source, information on the distance, and whether or not the sound source is within an angle of view and a motion vector of the main subject. In the reproduction | regeneration processing apparatus of Claim 5 or Claim 6,
The control unit is a reproduction processing apparatus that weights and averages the weight of the sound source in the frame with the weight of the sound source in an adjacent frame. The reproduction processing apparatus according to any one of claims 1 to 7, further comprising a speaker unit that outputs the audio data to the outside as audio.
The reproduction processing apparatus, wherein the control unit controls a volume of the audio data based on the reproduction control data and outputs sound to the speaker unit. In the reproduction | regeneration processing apparatus of any one of Claim 1 thru | or 8,
A reproduction processing apparatus comprising a display unit for displaying the image data. An imaging unit that images the object scene and generates image data;
A microphone unit that receives sound and generates sound data;
A reproduction processing apparatus according to claim 1;
An imaging apparatus comprising: An input procedure for reading image data and audio data recorded during the image data generation process;
A subject detection procedure for detecting a main subject from the image data;
Information about the detected distance to the main subject is acquired from the image data, and based on the information about the distance , the direction of the sound source, and whether the sound source is within the angle of view, A control procedure for generating playback control data for controlling the volume;
A reproduction processing program for causing a computer to execute.

Images