JP6525029B2 - Reproduction processing apparatus, imaging apparatus and reproduction processing program - Google Patents

Description

  The present invention relates to a reproduction processing device, an imaging device, and a reproduction processing program.

  Conventionally, the direction of the sound source of the voice can be acquired from the voice recorded using a plurality of microphones.

  For example, a digital camera having a plurality of microphones picks up a still image or a moving image and records a voice to estimate the direction of each voice source and the position in the image, and the directional voice data and position data of each voice source There is a technique of generating and recording in association with image data (see Patent Document 1 etc.).

JP, 2009-239348, A

  However, in the prior art, the sound data of each sound source may be selected according to the scene of the image or the viewpoint of the photographer or the subject by simply reproducing the sound source for all sound sources or sound sources simultaneously with reproducing and displaying the image. It can not be reproduced with effects or effects.

  SUMMARY OF THE INVENTION In view of the problems of the prior art, an object of the present invention is to provide a technology capable of reproducing sound data by applying effects and effects to sound data for each sound source.

In order to solve the above problems, according to one aspect of a reproduction processing apparatus exemplifying the present invention, an input unit for reading audio data recorded at the time of processing image data generated by imaging a subject, and a sound source are displayed or within the angle of view whether the images are, determined on the basis of the angle of view and the position information of the sound source of the image, based on whether the angle of view, the reproduction control data for controlling the sound volume of the audio data during reproduction And a control unit that generates the

  In addition, the control unit determines the direction of the sound source of at least one sound from the sound data to generate sound data of the sound source, and weights the sound source based on the direction of the sound source and whether the sound source is within the angle of view. Reproduction control data may be generated.

  Also, the control unit determines the direction of each of the plurality of sound sources from the sound data to generate sound data of each sound source, and weights each sound source based on the direction of each sound source and whether the sound source is within the angle of view. Playback control data may be generated.

  The image processing apparatus further includes a scene determination unit that determines a scene of the image data, and the control unit weights the sound source based on the determination result of the scene determination unit as well as the direction of the sound source and whether the sound source is within the angle of view. Good.

  Further, the input unit inputs a plurality of image data captured continuously, and the control unit detects a motion associated with the panning of the imaging device at the time of imaging a subject, and a motion vector based on the plurality of image data The control unit may weight the sound source in each frame based on the direction of the sound source, whether the sound source is within the angle of view, and the motion vector.

  In addition, a display unit may be provided to display an image based on image data.

  One aspect of an imaging device exemplifying the present invention includes an imaging unit that captures an object scene and generates image data, a microphone unit that receives audio and generates audio data, and a reproduction processing device of the present invention; Equipped with

One aspect of the reproduction processing program exemplifying the present invention is an input procedure for reading audio data recorded at the time of processing image data generated by imaging a subject, and whether or not the sound source is within the angle of view of the displayed image. or a determination based on the image view angle and position information of the sound source, based on whether the angle of view, a control procedure for generating reproduction control data for controlling the sound volume of the audio data during reproduction, the Make it run on a computer.

  According to the present invention, sound data for each sound source can be reproduced with effects and effects.

The figure which shows the structure of the computer which concerns on one Embodiment of this invention. Flow chart showing reproduction processing by the computer according to one embodiment Figure showing an example of the subject field A diagram showing a configuration of a CPU in a computer according to another embodiment of the present invention Flow chart showing reproduction processing by a computer according to another embodiment A diagram showing an example of continuous frames of a moving image Diagram showing an example of a frame motion vector Diagram showing a list of weighting factors set according to the playback mode Diagram showing an example of the motion vector of the main subject

<< One embodiment >>
FIG. 1 is a diagram showing the configuration of a computer 100 operated as a reproduction processing apparatus according to an embodiment of the present invention.

The computer 100 illustrated in FIG. 1A includes a CPU 10, a storage unit 11, an input / output interface (input / output I / F) 12, and a bus 13. The CPU 10, the storage unit 11, and the input / output I / F 12 are connected via the bus 13 so as to be able to transmit information. Further, the computer 100 is connected with an output device 14 for displaying a progress of image processing and a processing result, and an input device 15 for receiving an input from a user via the input / output I / F 12. A general liquid crystal monitor or printer can be used as the output device 14, and the input device 15 can be
A keyboard, a mouse, etc. can be appropriately selected and used. The computer 100 according to this embodiment also includes a speaker as the output device 14. Further, the input / output IF 12, together with the output device 14 and the input device 15, is a connection unit for connecting a digital camera (not shown) with a Universal Serial Bus (USB) cable or a memory card attached to the digital camera (not shown) (Not shown) shall be inserted.

  An image to be processed by the computer 100 according to the present embodiment is captured by a digital camera (not shown) or the like, and a still image in which audio data in which audio around the object scene is stereo-recorded at the time of imaging is added It is a video. In the header area of the still image or moving image, at least the subject distance of all the main subjects imaged in the still image or moving image, together with the exposure conditions (focal length, aperture value, shutter speed, ISO value, etc.) It shall have the information of D.

  The CPU 10 is a processor that controls each part of the computer 100 in an integrated manner. For example, the CPU 10 reads the reproduction processing program stored in the storage unit 11 based on the instruction from the user accepted by the input device 15. The CPU 10 operates as the subject detection unit 20 and the scene determination unit 21 by executing the reproduction processing program (FIG. 1B), and performs reproduction processing on the image to be processed. The CPU 10 reproduces and displays a still image or a moving image on the liquid crystal monitor of the output device 14, and outputs the sound of the added sound data to the speaker.

  The subject detection unit 20 performs subject detection processing on each frame of a still image or a moving image to be processed, and detects a face area of a person or an image area of a building, a car, or the like. This subject detection process is performed by a known algorithm. For example, the subject detection unit 20 performs pattern matching processing or the like on a still image or frame using the templates of various patterns such as a person or a building stored in the storage unit 11 to detect the image area of the main subject Do. The CPU 10 acquires the size, the position, and the like of the detected image area of the main subject as subject information.

  The scene determination unit 21 determines a scene of a scene captured in a still image or a frame using a known method. In addition, when the scene determination unit 21 has scene mode information at the time of imaging in the header area of the read still image or moving image, the scene determination unit 21 determines a scene based on the scene mode information.

  The storage unit 11 records a control program, a reproduction processing program, and the like, and an image read from a digital camera (not shown). The programs, images, and the like stored in the storage unit 11 can be appropriately referenced from the CPU 10 via the bus 13. A storage device such as a general hard disk drive or a magneto-optical disk drive can be selected and used as the storage unit 11. Although the storage unit 11 is incorporated in the computer 100, it may be an external storage device. In this case, the storage unit 11 is connected to the computer 100 via the input / output I / F 12.

  Next, reproduction processing by the computer 100 of the present embodiment will be described with reference to the flowchart of FIG. In the present embodiment, the image to be processed is a still image in which an object scene 40 including a person 30 as shown in FIG. 3A is captured. However, FIG. 3B shows the field 40 after capturing of the still image, and it is assumed that the automobile 32 is outside the field 40 at the time of capturing, but is approaching the person 30. .

The user uses the input device 15 to input a command of the reproduction processing program or double-click the icon of the program displayed on the output device 14 to instruct the CPU 10 to start the reproduction processing program. The CPU 10 inputs / outputs the instruction.
The program is accepted via F12, and the reproduction processing program stored in the storage unit 11 is read and executed. The CPU 10 starts the process from step S101.

  Step S101: The CPU 10 reads a still image specified by the user via the input device 15 from the digital camera (not shown) via the storage unit 11 or the input / output I / F 12.

  Step S102: The subject detection unit 20 detects the face area of the person 30 and the image area of the building 31 from the read still image. The CPU 10 acquires the size, the position, and the like of the detected image area of each main subject as subject information D together with the subject distance D of each main subject added to the header area of the still image.

  Step S103: The CPU 10 obtains the direction of the sound source of each sound included in the audio data from the audio data of stereo recording added to the still image using a known method such as Patent Document 1 etc. The signal is extracted to generate voice data of each sound source. In this embodiment, there are sound sources of the voice of the person 30, music flowing from the building 31, and engine sound of the approaching car 32 as sound sources, and the CPU 10 generates voice data of those sound sources. Further, the CPU 10 specifies where the sound sources are located on the still image based on the subject information and the direction of each sound source, and generates position information including the direction of each sound source and the specified position. The CPU 10 records voice data and position information for each sound source in an internal memory (not shown) in association with the subject information.

  As shown in FIG. 3A, since the car 32 is not within the angle of view of the still image, the size and position of the image area of the subject information of the car 32 and the subject distance D (i) It is assumed that it is set to a predetermined value indicating the outside or infinity.

Step S104: The CPU 10 determines whether or not each sound source is present within the angle of view of the still image in order to set the volume at the time of reproducing the sound data of each sound source according to the scene captured in the still image described later. Is determined based on the angle of view of the still image and the position information of each sound source. When the sound source is within the angle of view, the CPU 10 sets the flag UseFlag (i) of the sound source to 1. Meanwhile, CP
U10 sets the flag UseFlag (i) to 0 when the sound source is out of the angle of view. Where the coefficient i
Represents each sound source, and in this embodiment, the person 30 is i = 0, the building 31 is i = 1, and the automobile 32 is i = 2.

Step S105: The scene determination unit 21 determines a scene captured in a still image, and the CPU 10 weights each sound source at the time of reproduction according to the determination result of the scene determination unit 21. Note that the scene determined by the scene determination unit 21 according to the present embodiment is either “snap”, “portrait” or “landscape”. The setting of weighting of each sound source by the CPU 10 in each scene will be described.
A) In the case of "snap" The CPU 10 changes the weighting of each sound source at the time of reproduction according to the in-focus area at the time of capturing a still image, for example, the distance L on the still image from the face region of the person 30 Set as coefficient WT (i) = α / L. The coefficient α is set to a predetermined value. When the sound source is outside the angle of view of the still image, that is, the distance L of the car 32 is set to a predetermined value or infinity.
B) "Portrait" The CPU 10 sets the weighting factor WT (i) of the sound source from the person to be the largest value. For example, the CPU 10 sets the weight coefficient WT (i) of the person 30 to 1 and sets the weight coefficient WT (i) of the building 31 and the automobile 32 to 0.5.
C) In the case of "landscape"
The CPU 10 sets the weighting factor WT (i) of the sound source from a person other than the person to a large value. For example, the CPU 10 sets the weight coefficient WT (i) of the person 30 to 0.2, and sets the weight coefficient WT (i) of the building 31 and the automobile 32 to 1.

  Note that the value and the setting method of the weighting factor WT (i) in each scene are an example, and may be set by other values or other setting methods. For example, in the case of “portrait”, the weighting factor WT (i) of the building 31 or the automobile 32 may be set in inverse proportion to the distance to the person 30.

Step S106: The CPU 10 calculates the amplification factor AMP (i) that determines the volume of each sound source at the time of reproduction based on the following equation (1) using the weighting factor WT (i) of each sound source set in step S105. And generates as reproduction control data.
AMP (i) = UseFlag (i) × WT (i) / (β × D (i)) (1)
Here, the coefficient β is a coefficient that normalizes the subject distance D (i) of each main subject.

  Step S107: The CPU 10 calculates the product of the audio data and the amplification factor AMP (i) of the reproduction control data for each sound source, and generates reproduction audio data of each sound source. The CPU 10 reproduces and displays the still image on the liquid crystal monitor of the output device 14 and outputs the reproduction audio data of each sound source as sound to the speaker of the output device 14. The CPU 10 ends the series of processing.

  When the image to be processed is a moving image, the computer 100 performs the reproduction process shown in FIG. 2 on each frame of the moving image. That is, after performing the processing of steps S102 to S105 on all the frames of the moving image, the computer 100 proceeds to step S106, calculates amplification factor AMP for each sound source in each frame, and generates reproduction control data. Do.

As described above, in the present embodiment, by setting the weight coefficient of each sound source according to the scene of the captured image, the sound data of each sound source can be reproduced with an effect or an effect.
Other Embodiments
A computer according to another embodiment of the present invention is the same as the computer 100 according to one embodiment shown in FIG. 1, and detailed description of each component is omitted.

  The difference between the computer 100 of the present embodiment and that of the one embodiment is that 1) the image to be processed is only a moving image, and 2) the computer 100 is a photographer's viewpoint or a subject viewpoint in playback of a moving image And the reproduction mode for setting the weighting factor WT of the sound data of each sound source according to. Here, the photographer viewpoint mode in the playback mode is a mode in which the voice data of each sound source is reproduced with a voice heard by the photographer of the digital camera (not shown), and the subject viewpoint mode is the position of the main subject In this mode, the sound data of each sound source is reproduced with a sound that can be heard in the. Further, as the image to be processed is only a moving image, the CPU 10 according to this embodiment executes the reproduction program, and as shown in FIG. Operate.

The vector calculation unit 22 detects a motion associated with panning of a digital camera (not shown) that has captured a moving image as a frame motion, calculates a motion vector of the frame, detects a motion of the main subject, and calculates a motion vector of the main subject Do. Specifically, the motion vector calculation unit 22 performs known correlation processing on two adjacent frames of the moving image. The motion vector calculation unit 22 detects frame motion from the amount of displacement between two frames in the background image area excluding the image area of the main subject detected by the subject detection unit 20 based on the correlation result, for example Calculate the motion vector of the image frame. On the other hand, the motion vector calculation unit 22 moves the main subject based on the amount of shift of the background image area and the amount of shift of the main subject image area.
To calculate the motion vector of the main subject.

  Next, the reproduction process of the computer 100 according to the present embodiment will be described with reference to the flowchart of FIG. The moving image to be processed in the present embodiment is obtained by imaging the object scene 40 shown in FIG. FIGS. 6A to 6C show, as an example, three consecutive frames among the frames of the moving image. That is, FIGS. 6A to 6C show the k-th, k + 1-th and k + 2-th frames (k is a natural number). 6 (a) and 6 (b) are taken as frames captured by panning a digital camera (not shown) so as to follow the person 30. FIG. Further, FIG. 6C shows a frame captured by panning a digital camera (not shown) so that the car 33 appearing from the left side becomes the image center.

  The user uses the input device 15 to input a command of the reproduction processing program or double-click the icon of the program displayed on the output device 14 to instruct the CPU 10 to start the reproduction processing program. The CPU 10 receives the instruction via the input / output I / F 12 and reads and executes the reproduction processing program stored in the storage unit 11. The CPU 10 starts the process from step S201.

  Step S201: The CPU 10 reads a moving image specified by the user via the input device 15 from a digital camera (not shown) via the storage unit 11 or the input / output I / F 12. In addition, it is preferable that the CPU 10 accepts the specification of the reproduction mode as well as the specification of the moving image to be reproduced.

  Step S202: The subject detection unit 20 detects an image area of the person 30 or the like from each frame of the read moving image. The CPU 10 acquires the size, the position, and the like of the detected image area of each main subject as subject information D together with the subject distance D of each main subject added to the header area of the moving image.

  Step S203: The CPU 10 obtains the direction of the sound source of each sound included in the audio data from the audio data of stereo recording added to the moving image using a known method such as Patent Document 1 etc. To generate voice data of each sound source. Further, the CPU 10 specifies where on the respective frames the sound sources are located based on the subject information and the direction of each sound source, and position information including the direction of each sound source in each frame and the specified position is Generate The CPU 10 records voice data and position information for each sound source in an internal memory (not shown) in association with the subject information.

  Step S204: The motion vector calculation unit 22 performs correlation processing on two adjacent frames, detects frame motion from the amount of displacement in the background image area, and calculates a motion vector of the frame. Further, the motion vector calculation unit 22 calculates the motion vector of each main subject from the amount of shift of the image area of the background and the shift amount of the image area of each main subject. The CPU 10 records the calculated frame and the motion vector of each main subject in an internal memory (not shown) in association with each frame.

  Step S205: The CPU 10 determines whether or not the photographer viewpoint mode is set as the reproduction mode. The CPU 10 proceeds to step S206 (YES side) when the camera viewpoint mode is set, and proceeds to step S207 (NO side) when the camera viewpoint mode is set.

Step S206: In the case of the photographer viewpoint mode, the CPU 10 reproduces, for example, the voice data of each sound source with a voice heard by the photographer of the digital camera (not shown). And the position information of each sound source
The weighting factor WT (m, i) of each sound source in the frame is set. Specifically, it sets as follows.

  7A to 7C show the directions of the motion vectors of the frames in the k-th, k + 1-th and k + 2-th frames shown in FIGS. 6A to 6C by arrows at the centers of the respective frames. Further, FIG. 8A shows a list of weighting factors WT (m, i) of each sound source, which are set based on the direction of the motion vector of the frame and the position information of each sound source in each frame. In the photographer's viewpoint mode, the direction of the motion vector of the frame matches the direction of the frame, and a weighting factor with a larger value is set as the sound source is closer to the center of the frame.

  That is, since the k-th and k + 1-th frames are captured so that the person 30 is at the center of the frame, as shown in FIG. 8A, the weight coefficient of the person 30 is the largest value. Set to In addition, since the building 31 and the car 32 are on the opposite side of the motion vector of the frame and approach the person 30, the weight coefficients of the building 31 and the car 32 are larger in the k + 1st frame than the kth Set to a value. On the other hand, since the automobile 33 is outside the angle of view in the k-th and k + 1-th frames and on the side opposite to the direction of the motion vector of the frame, the weighting factor is set to a small value.

  On the other hand, the weight coefficient of the car 33 is set to the largest value because the k + 2 th frame is obtained by panning and imaging the digital camera (not shown) so that the car 33 is at the center of the frame. Ru. On the other hand, since the person 30, the building 31, and the car 32 are on the side opposite to the direction of the motion vector of the frame, the respective weighting coefficients are set to small values compared to the k-th and k + 1-th frames.

  The setting method of the value and value of the weighting coefficient WT (m, i) for each sound source in each frame shown in FIG. 8A is an example, and the number and the size of the motion vector of the frame, the direction, etc. It is preferable to appropriately set accordingly.

  Step S207: In the case of the object viewpoint mode, the CPU 10 reproduces, for example, the voice data of each sound source with a voice heard by the person 30, based on the direction of the motion vector of the person 30 and the position information of each sound source. The weighting factor WT (m, i) of each sound source in the m-th frame is set. Specifically, it sets as follows.

  9 (a) to 9 (c) show the direction of the motion vector of the person 30 in the k-th, k + 1-th and k + 2-th frames shown in FIGS. 6 (a) to 6 (c) as in FIG. Indicated by an arrow at the center of each frame. Further, FIG. 8B shows a list of weighting factors WT (m, i) of each sound source set based on the direction of the motion vector of the person 30 and the position information of each sound source in each frame. Note that, in the subject viewpoint mode, the direction of the motion vector of the person is matched, and a weighting factor having a larger value is set as the sound source is closer to the person 30.

  That is, since the motion vectors of the person 30 in the kth to k + 2th frames are in the same direction, and the building 31 and the car 32 approach the person 30, as shown in FIG. The weighting factor is set to a large value in accordance with the kth to k + 2th frames. On the other hand, since the automobile 33 is out of the angle of view in the k-th and k + 1-th frames, it is set to a weighting factor as small as 0.1. In addition, the automobile 33 approaches the person 30 within the angle of view of the (k + 2) th frame, but since it is opposite to the direction of the motion vector of the person 30, it is set to a smaller value than the weighting factor of other sound sources. Be done.

In the subject viewpoint mode, since the voice data of each sound source is reproduced with a voice that the person 30 himself / herself hears, the voice of the person 30 itself is reproduced at a small volume, as shown in FIG.
For example, a weighting factor of a predetermined value, such as 0.5, is preset.

  Further, the setting method of the value and value of the weighting coefficient WT (m, i) for each sound source in each frame shown in FIG. 8B is an example, and the number of main subjects, the size and direction of motion vectors of main subjects, etc. It is preferable to set appropriately according to

  Step S208: The CPU 10 sets the weighting factor WT (m, i) of each sound source in the mth frame set in step S206 or step S207 to the adjacent N including the mth frame using the following equation (2). Weighted averaging is performed in the time axis direction with respect to the weighting factors in the frames (<total number of frames of moving image).

Here, the coefficient ε (j) indicates the degree to which the weighting coefficient of the sound source in the j-th frame contributes to the m-th frame, and is set so as to most contribute when j = m. In addition, the number of frames N to be used is about 10 or less, and the range j of the weighted average is appropriately selected and performed from m to m + N−1, m−N + 1 to m, or m−N / 2 to m + N / 2, etc. Is preferred. By this weighted averaging, the CPU 10 can smoothly change the volume of each sound source.

Step S209: The CPU 10 uses the weight coefficient <WT (m, i)> weighted at step S208 to set the amplification factor AMP (m, i) for determining the volume of each sound source in each frame at the time of reproduction next. It calculates using Formula (3) and produces | generates reproduction | regeneration control data.
AMP (m, i) = <WT (m, i)> / (β × D (m, i)) (3)
Here, the coefficient β is a coefficient that normalizes each subject distance D (m, i) in the m-th frame.

  Step S210: The CPU 10 calculates the product of the audio data and the amplification factor AMP (m, i) of the reproduction control data for each sound source, and generates reproduction audio data of each sound source.

  Step S211: The CPU 10 reproduces and displays on the liquid crystal monitor of the output device 14, and outputs the reproduction audio data of each sound source as a voice to the speaker of the output device 14. The CPU 10 ends the series of processing.

As described above, in the present embodiment, the sound data of each sound source is reproduced by applying effects and effects by setting the weight coefficient of each sound source according to the motion vector of the frame or main subject in each frame captured. be able to.
<< Supplementary items of the embodiment >>
(1) The reproduction processing apparatus of the present invention is realized by causing the computer 100 to execute the reproduction processing program, but the present invention is not limited to this. The present invention is also applicable to a reproduction processing program for realizing the processing in the reproduction processing apparatus according to the present invention by the computer 100 and a medium having the recording program recorded thereon.

The present invention is also applicable to a digital camera having the reproduction processing program of the present invention.
. When the digital camera operates as the image processing apparatus of the present invention, the CPU 10 may realize each processing of the subject detection unit 20, the scene determination unit 21 and the motion vector calculation unit 22 as software, or an ASIC Each of these processes may be realized in hardware using In this case, it is preferable that subject information, audio data, position information, and reproduction control data be added to the header region of a still image or a moving image captured by a digital camera, together with the exposure condition. The audio data added to the header area may be extracted for each sound source or may be stereo-recorded before extraction.

  (2) In the above embodiment, the computer 100 reproduces and displays the still image and the moving image together with the reproduction of the sound data of each sound source. However, the present invention is not limited to this. Good.

  (3) In the above embodiment, according to the scene determination result or the reproduction mode, the weighting factor of each sound source is set to generate reproduction control data and reproduction sound data, but the present invention is not limited to this. For example, the CPU 10 may generate reproduction control data and reproduction sound data in consideration of the Doppler effect or the like based on a frame or a motion vector of a main subject.

(4) In the other embodiment described above, the motion vector calculation unit 22 calculates the motion vector of the frame and each main subject based on the correlation processing for two adjacent frames, but the present invention is not limited to this. For example, in the case of a moving image compressed in the moving image format such as H.264, a motion vector is calculated in motion compensation in inter-frame prediction in order to improve compression efficiency. Therefore, the motion vector calculation unit 22 may obtain the motion vector of the frame and each main subject based on the motion vector.

  Also, the motion vector may be calculated using the motion of the subject detected by the subject tracking function.

  When a digital camera (not shown) includes a sensor such as an acceleration sensor or an electronic gyro, the motion vector calculation unit 22 may calculate a motion vector of a frame based on the output value of the sensor.

  (5) In the other embodiment described above, the main subject to be followed is the person 30 in the subject viewpoint mode, but the present invention is not limited to this and the main subject such as the building 31 or the car 32 may be followed .

  The features and advantages of the embodiments will be apparent from the foregoing detailed description. This is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the invention. In addition, those skilled in the art should be able to easily conceive of any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above, and the embodiments With appropriate modifications and equivalents which fall within the scope disclosed in.

10 CPU, 11 storage units, 12 input / output I / F, 13 buses, 14 output devices, 15
Input device 20 subject detection unit 21 scene determination unit 22 motion vector calculation unit 100 computer

Claims (8)

An input unit for reading audio data recorded at the time of processing of image data generated by imaging an object;
Whether or not the sound source is within the angle of view of the displayed image is determined based on the angle of view of the image and the position information of the sound source, and based on whether or not it is within the angle of view, A control unit that generates reproduction control data that controls the volume of audio data;
A reproduction processing apparatus comprising: In the reproduction processing apparatus according to claim 1,
The control unit determines a direction of a sound source of at least one sound from the sound data to generate sound data of the sound source, and generates the sound data of the sound source based on the direction of the sound source and whether the sound source is within an angle of view. A reproduction processing apparatus that generates the reproduction control data by weighting. In the reproduction processing apparatus according to claim 2,
The control unit obtains the direction of each of the plurality of sound sources from the sound data to generate sound data of each of the sound sources, and based on whether or not the direction of each sound source and the sound source is within an angle of view A reproduction processing apparatus that generates the reproduction control data by weighting a sound source. In the reproduction processing apparatus according to claim 2 or 3,
A scene determination unit that determines a scene of the image data;
The control processing unit is configured to weight the sound source based on the determination result of the scene determination unit together with the direction of the sound source and whether or not the sound source is within an angle of view. In the reproduction processing apparatus according to claim 2 or 3,
The input unit inputs a plurality of continuously captured image data;
The control unit
The image processing apparatus further comprises a motion vector calculation unit that detects a motion associated with panning of the imaging device when capturing an object, and calculates a motion vector based on the plurality of image data.
The control unit performs weighting on the sound source in each frame based on the direction of the sound source, whether or not the sound source is within an angle of view, and the motion vector. The reproduction processing apparatus according to any one of claims 1 to 5.
A reproduction processing apparatus comprising a display unit for displaying an image based on the image data. An imaging unit that captures an object scene and generates image data;
A microphone unit that receives voice and generates voice data;
A reproduction processing apparatus according to any one of claims 1 to 6.
An imaging device comprising: An input procedure for reading audio data recorded at the time of processing of image data generated by imaging an object;
Whether or not the sound source is within the angle of view of the displayed image is determined based on the angle of view of the image and the position information of the sound source, and based on whether or not it is within the angle of view, A control procedure for generating reproduction control data for controlling the volume of audio data;
A playback processing program that causes a computer to execute.

Images