JP2018019295A - Information processing system, control method therefor, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology capable of generating sound having presence matching arbitrary viewpoint video.SOLUTION: An information processing system for processing a picture and sound corresponding to any viewpoint on the basis of a plurality of picture signals obtained by imaging by a plurality of imaging apparatuses and a plurality of sound collection signals obtained by collecting sound at a plurality of sound collection points comprises: acquisition means for acquiring a viewpoint position and a direction of a line of sight to an imaging object; selection means for selecting, from the plurality of sound collection points, depending on the viewpoint position and the direction of the line of sight, a sound collection point for a sound collection signal to be used for generating a sound signal corresponding to a picture that depends on the viewpoint position and the direction of the line of sight and is based on the plurality of picture signals; and sound generation means for generating a sound signal using a sound collection signal collected at the sound collection point selected by the selection means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, a control method therefor, and a computer program, and more particularly to a technique for generating an audio reproduction signal suitable for an arbitrary viewpoint video.

  In recent years, by developing computer graphics technology, etc., a system has been developed that generates images from arbitrary viewpoints by appropriately processing images shot by multiple cameras installed so as to surround a wide area such as a sports stadium. Has been.

  In order to give a sense of reality to an arbitrary viewpoint video generated by such a system, it is required to generate and reproduce an audio signal corresponding to the video.

  Therefore, it is known to specify an arbitrary position in a stadium or the like and generate an acoustic signal suitable for the place (Patent Document 1). In this configuration, arbitrary viewing points and viewing angles are input in addition to the directivity and location of multiple microphones installed in the stadium, and the signal distribution ratio for each channel and the effect of the distance between each microphone and the viewing point. Is automatically calculated, and surround sound is automatically mixed.

Japanese Patent Laid-Open No. 2005-223771

  However, in the configuration of Patent Document 1, the surround signal is generated using all the microphone signals regardless of the position at which the listening point is specified in the stadium. For this reason, there is a problem that even if the listening point moves, there is little change in the sound field feeling.

  Therefore, an object of the present invention is to provide a technique capable of generating a realistic sound suitable for an arbitrary viewpoint video.

In order to achieve the above object, an information processing system according to the present invention comprises the following arrangement. That is,
An information processing system that processes an image and sound corresponding to an arbitrary viewpoint based on a plurality of image signals photographed by a plurality of photographing devices and a plurality of sound collection signals collected at a plurality of sound collection points. There,
An acquisition means for acquiring the viewpoint position and the direction of the line of sight with respect to the imaging target;
A sound collection point of a sound collection signal used to generate an acoustic signal corresponding to an image based on the plurality of image signals, the image being in accordance with the viewpoint position and the direction of the line of sight, from the plurality of sound collection points. Selecting means for selecting according to the viewpoint position and the direction of the line of sight;
Sound generating means for generating an acoustic signal using the collected sound signal picked up at the sound collecting point selected by the selecting means.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which can produce | generate the sound with the realistic feeling suitable for arbitrary viewpoint images can be provided.

The block diagram which shows the example of 1 structure of the arbitrary viewpoint video generation systems. The schematic diagram which shows the arrangement | positioning condition of the sound collection point in a stadium. The flowchart which shows the process sequence of a main process. The figure which shows the data structure of the information used with an arbitrary viewpoint image | video production | generation system. The flowchart which shows the process sequence of listening range determination processing. The schematic diagram which shows the relationship between a viewpoint, listening range, listening point, and listening direction. 7 is a flowchart showing a processing procedure for subject position detection processing. The flowchart which shows the process sequence of a sound collection point selection process. The flowchart which shows the process sequence of the sound collection point selection process in a listening range. The flowchart which shows the process sequence of reproduction | regeneration signal generation processing. The flowchart which shows the process sequence of a stereo reproduction signal production | generation process. The flowchart which shows the process sequence of a surround reproduction signal production | generation process. The flowchart which shows the process sequence of a headphone reproduction signal generation process. The flowchart which shows the process sequence of the sound collection point selection process in a listening range.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential for the implementation of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

<< Embodiment 1 >>
(Arbitrary viewpoint video generation system)
An arbitrary viewpoint video generation system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of an arbitrary viewpoint video generation system according to the present embodiment. The arbitrary viewpoint video generation system according to the present embodiment can be used for an arbitrary viewpoint based on a plurality of image signals photographed by a plurality of photographing devices and a plurality of sound collection signals collected at a plurality of sound collection points. It operates as an information processing system that outputs corresponding images and sounds.

  In FIG. 1, reference numeral 1 denotes a sound pickup signal input unit, which inputs sound pickup signals from a plurality of microphones installed at sound pickup points that are scattered all over the stadium to be photographed by this system. Amplification and noise removal. Further, additional information indicating the characteristics of the sound collection point is added to each sound collection signal, and is output to the sound collection point selection unit 3 as sound collection point information.

  FIG. 2 is a diagram schematically illustrating a setting example of sound collection points in a shooting target stadium. In FIG. 2, 101 is one of the sound pickup points, 102 is the auditorium, 103 is the track, and 104 is the ground. In the present embodiment, as shown in FIG. 2, an example will be described in which sound collection points are set evenly in every place of the stadium and sound at the sound collection points is always collected.

  Reference numeral 2 denotes a listening range determination unit, which is based on the viewpoint information specified by the viewpoint information specifying unit 6, the viewpoint video output by the viewpoint video generation unit 8, and the subject position output by the subject position detection unit 9. Determine the listening range, listening point, and listening direction. Reference numeral 3 denotes a sound collection point selection unit. The sound collection signal input unit outputs the sound collection points used for generating the sound reproduction signal according to the listening range, the listening point, and the listening direction output from the listening range determination unit 2. Select from point information as appropriate. Reference numeral 4 denotes an acoustic signal generation unit, which performs acoustic generation for generating a reproduction acoustic signal of an arbitrary reproduction format based on the sound collection signal at the sound collection point selected by the sound collection point selection unit 3. And it outputs to the sound reproduction part 11 and MUX15.

  Reference numeral 5 denotes an operation unit, which accepts user operation instructions for the system. A viewpoint information designation unit 6 generates viewpoint information based on a user operation instruction transmitted via the operation unit 5, and outputs the viewpoint information to the listening range determination unit 2 and the viewpoint video generation unit 8. The operation unit 5 is realized by a keyboard, a pointing device such as a mouse, a touch panel, or the like. Thus, in the present embodiment, the arbitrary viewpoint video generation system acquires viewpoint information via the viewpoint information specifying unit 6.

  Reference numeral 7 denotes a video signal input unit that inputs video signals shot by a plurality of cameras installed in a stadium that is the shooting target of this system, and performs amplification and noise removal of the video signals. Further, camera parameters at the time of shooting are added to each sound pickup signal and output to the viewpoint video generation unit 8 as camera shooting information. In the present embodiment, an example in which a video that is a moving image is generated as an arbitrary viewpoint image will be described, but a still image may be a target.

  Reference numeral 8 denotes a viewpoint video generation unit, which generates an arbitrary viewpoint video by appropriately processing videos from a plurality of cameras in accordance with the viewpoint information specified by the viewpoint information specifying unit 6, and generates a listening range determination unit 2 and subject position detection. Unit 9, video playback unit 10, and MUX 15. Reference numeral 9 denotes a subject position detection unit that detects the position of the subject in the viewpoint video based on the viewpoint video and viewpoint information generated by the viewpoint video generation unit 8. As will be described later, the subject includes a person or a specific object other than the person. Reference numeral 10 denotes a video playback unit, which plays back the viewpoint video generated by the viewpoint video generation unit 8 and outputs it to the display unit 19.

  Reference numeral 11 denotes an acoustic reproduction unit which reproduces the acoustic signal generated by the acoustic signal generation unit 4 according to the reproduction environment. A stereo speaker set 12 amplifies a stereo sound signal generated by the sound signal generation unit 4 as appropriate, and outputs the sound instead of sound. Reference numeral 13 denotes a surround speaker set, which appropriately amplifies the surround sound signal generated by the sound signal generation unit 4 and outputs the sound instead of sound. Reference numeral 14 denotes a headphone, which outputs the headphone signal generated by the acoustic signal generation unit 4 by converting it into sound. In the present embodiment, an example in which the sound of the acoustic signal is output by any one of the stereo speaker set 12, the surround speaker set 13, and the headphones 14 will be described. However, the sound reproduction environment is not limited to that exemplified here. .

  Reference numeral 15 denotes a MUX (multiplexer) that superimposes the arbitrary viewpoint video signal generated by the viewpoint video generation unit 8 and the audio signal generated by the audio signal generation unit 4 to create one video stream data. And output to the output unit 18. Reference numeral 16 denotes a communication unit, which appropriately transmits video stream data output from the MUX 15 to the communication network 17. Reference numeral 17 denotes a communication network, which indicates a public communication network such as the Internet or a public line network. An output unit 18 includes an output terminal and outputs video stream data output from the MUX 15 to an external device connected to the output terminal. Reference numeral 19 denotes a display unit that displays an arbitrary viewpoint video reproduced by the video reproduction unit 10. The display unit 19 is realized by a liquid crystal panel, an organic EL display, or the like.

  These components are connected to a CPU (Central Processing Unit) (not shown) via a control bus, and their operations are integrated and controlled in accordance with a command instruction from the CPU. The CPU reads a computer program from a storage device (not shown) and controls the entire apparatus according to the computer program.

(Main process)
In the configuration of the present embodiment shown in FIG. 1, a process of determining a listening range, a listening point, and a listening direction according to viewpoint information and generating a sound field based on the listening range will be described below with reference to a flowchart. FIG. 3 is a flowchart showing a processing procedure of the main processing of the present embodiment. The following steps are executed based on CPU control.

  S101 is a process in which the viewpoint information specifying unit 6 changes the viewpoint information in accordance with the instruction temporarily stored in the internal change instruction buffer, and outputs it to the listening range determination unit 2 and the viewpoint video generation unit 8.

  FIG. 4A shows a data structure of viewpoint information in the present embodiment. As shown in FIG. 4A, the viewpoint information of the present embodiment includes a viewpoint position, a viewpoint depression angle, a viewpoint direction, and a field angle. Among these, the viewpoint position is a three-dimensional coordinate indicating the position of the viewpoint in the stadium that is the subject of photographing. As an example, in this embodiment, a three-dimensional coordinate system is set in which the east-west direction is the eastward X-axis, the north-south direction is the north-facing Y-axis, and the top-bottom direction is the Z-axis. Explain the case. The viewpoint depression angle is the depression angle to which the viewpoint is directed, and is specified within a range of ± 90 ° with the horizontal direction being 0 °. The viewpoint direction is the direction of the horizontal plane in which the viewpoint is also facing, and in this embodiment, clockwise (clockwise) with the absolute direction with true north as 0 ° (that is, the positive Y-axis direction) as the reference front. Positive and counterclockwise (counterclockwise) are shown as negative. The angle of view is a value indicating the vertical and horizontal widths of the viewpoint video viewed from the viewpoint. In addition, the three-dimensional direction of the line of sight observed from the viewpoint is hereinafter referred to as the direction of the line of sight. The direction of the line of sight corresponds to the sum of the viewpoint depression angle (viewpoint elevation angle) and the viewpoint direction. In the system of this embodiment, it is possible to generate a video viewed from an arbitrary viewpoint specified by this information.

  When the process of S101 is completed, the process flow of generating an acoustic signal from S102 to S105, the process flow of generating a video from S106 to S108, and the process of S109 are performed in parallel. Hereinafter, the flow of these processes will be described.

  In S102, the sound acquired at the sound collection point is acquired, and the sound collection signal input unit 1 performs amplification and noise removal of the sound collection signals of a plurality of microphones. Further, header information indicating the characteristics of the sound collection points is added to each sound collection signal and output to the sound collection point selection unit 3 as sound collection point information. FIG. 4B shows a data structure of sound collection point information in the present embodiment. As shown in FIG. 4B, the sound collection point information in the present embodiment includes a sound collection point ID, sound collection point coordinates, and a sound collection signal. Among these, the sound collection point ID is a number (identification information) for identifying the sound collection point. The sound collection point coordinates are coordinates indicating the position of the sound collection point, and in the present embodiment, the sound collection point coordinates are horizontal plane coordinates in the stadium. The collected sound signal is an acoustic signal itself collected by a microphone installed at the sound collection point.

  Next, in S103, the listening range determination unit 2 determines the listening range, listening position, and listening direction based on the viewpoint information transmitted from the viewpoint information specifying unit 6 in S101. Details of this process (listening range determination process) will be described later with reference to FIGS. Next, in S104, the sound collection point selection unit 3 selects a sound collection point that has collected the sound collection signal used to generate an acoustic signal suitable for the viewpoint video. Details of this processing (sound pickup point selection processing) will be described later with reference to FIGS. Next, in S105, the acoustic signal generator 4 generates an acoustic signal corresponding to each acoustic reproduction environment. Details of this processing (reproduction signal generation processing) will be described later with reference to FIGS. 10, 11, 12, and 13. FIG. The generated acoustic signal is output to the sound reproducing unit 11 and the MUX 15. When the process is finished, the process proceeds to S110.

  On the other hand, in S106, video signals photographed by a plurality of cameras are acquired, and the video signal input unit 7 performs noise removal and brightness adjustment of the video signals. Further, the shooting parameters of each camera are added to the video signal as header information and output to the sound pickup point selection unit 3 as camera shooting information. FIG. 4C shows a data structure of camera shooting information in the present embodiment. As shown in FIG. 4C, the camera shooting information of this embodiment includes a camera position, a camera depression angle, a camera direction, a field angle, a focal length, and a video signal. Among these, the camera position is a three-dimensional coordinate indicating the position of the camera in the stadium that is the subject of photographing. The camera depression angle is the depression angle to which the viewpoint is directed, and is specified in a range of ± 90 ° with the horizontal direction being 0 °. Similarly, the camera direction is the direction of the horizontal plane that the camera is facing. In this embodiment, the camera is clockwise (clockwise) with the absolute direction with true north (that is, the positive Y-axis direction) being 0 ° as the reference front. Positive and counterclockwise (counterclockwise) are shown as negative. The angle of view is a value indicating the width of the captured video in angle. The focal length is a value indicating the distance from the optical center of the camera lens to the imaging surface. The video signal is a signal of the video itself taken by this camera.

  Next, in S107, the viewpoint video generation unit 8 appropriately processes and synthesizes the multiple camera videos received in S106 based on the viewpoint information transmitted from the viewpoint information designation unit 6 in S101, and generates an arbitrary viewpoint video. . That is, image generation for generating an image corresponding to the viewpoint position and the direction of the line of sight is performed based on a plurality of image signals. Such a method for synthesizing a video of an arbitrary viewpoint from a plurality of camera videos is well known and is generally performed in this field, and thus will not be described in detail.

  Next, in S108, the subject position detection unit 9 analyzes the video of the plurality of cameras received in S106 and the arbitrary viewpoint video generated in S107, so that the stadium that actually exists in the arbitrary viewpoint video exists. Detect the position at. Details of this processing (subject position detection processing) will be described later with reference to FIG. When the process is finished, the process proceeds to S110.

  Further, in S109, the viewpoint information designation unit 6 accepts a viewpoint change instruction input by the user via the operation unit 5, converts it into a viewpoint information change command, and primarily stores it in the internal change command buffer. When the process is finished, the process proceeds to S110.

  S110 is a process of superimposing and synthesizing the sound reproduction signal generated in S105 and the arbitrary viewpoint video signal generated in S107 in the MUX 15 and combining them as one video stream data and outputting it to the communication unit 16 and the output unit 18. is there. When the process is finished, the process proceeds to S111.

  S111 is processing for determining an output destination of the system of the present embodiment in a CPU (not shown). If the output destination is a playback device, the process proceeds to S112. If the output destination is a communication network, the process proceeds to S113. If the output destination is an external device, the process proceeds to S114.

  S112 is a process of outputting the audio reproduction signal generated in S105 and the arbitrary viewpoint video signal generated in S107 in the audio reproduction unit 11 and the video reproduction unit 10 in synchronization with the audio reproduction environment and the display unit 19, respectively. is there. Such a process is generally performed in a general video output apparatus and is well known, and thus will not be described in detail. By this processing, an arbitrary viewpoint video and an audio signal corresponding to the arbitrary viewpoint video are played back in synchronization with each other, so that it is possible to enhance a sense of reality when playing back the video. When the process is finished, the process proceeds to S115.

  In step S113, the communication unit 16 transmits the video stream data created in step S110 to the outside via the communication network 17. When the process is finished, the process proceeds to S115.

  In step S114, the output unit 18 outputs the video stream created in step S110 to an external device connected to the external output terminal. When the process is finished, the process proceeds to S115.

  S115 is processing for determining whether or not to terminate the main processing performed in the entire flow in a CPU (not shown). As a result of this determination, when the process is terminated (YES in S115), the process of this flow is terminated. If not finished (NO in S115), the process returns to S101.

(Listening range determination process)
FIG. 5 is a flowchart showing a detailed processing procedure of the listening range determination processing in S103 in the present embodiment. All processes in this flowchart are performed by the listening range determination unit 2.

  First, in S201, listening point information stored in an internal RAM (not shown) of the listening range determination unit 2 is initialized. Here, the data structure of the listening point information is shown in FIG. The listening point information in the present embodiment includes a listening range, a listening point, and a listening direction. Among these, the listening range is data indicating a range in which a sound that will be heard in the immediate vicinity when the viewpoint video is immersed in the stadium to be photographed is generated. In the present embodiment, the coordinates of four points on the horizontal plane are stored, and a range surrounded by a rectangle formed by connecting them is defined as a listening range. As will be described later, the listening range functions as a local range serving as a reference for selecting a sound collection point of a sound collection signal used for generating an acoustic signal. The listening point is a point used as a reference when arranging the collected sound signal when the reproduction signal is generated in the later processing, and the coordinates on the horizontal plane are also stored. Similarly, the listening direction is a direction used as a reference when arranging the collected sound signal when the reproduction signal is generated in the later processing. In the present embodiment, the front direction viewed from the listening point is indicated by an absolute angle in the same manner as the viewpoint direction. In S201, all the data of the listening point information are initialized.

  Specifically, in S202, it is determined whether or not the elevation angle of the viewpoint information specified by the viewpoint information specifying unit 6 in S101 is below -10 °. The purpose of this process is to determine whether the specified viewpoint information is an overhead viewpoint or a horizontal viewpoint. The bird's-eye view is when the viewpoint is at a position where the shooting target such as a stadium is observed from an overhead view from the information, and the horizontal viewpoint is when the viewpoint is at a position where the shooting target is observed almost horizontally from the side. Say. Therefore, in the present embodiment, the viewpoint switching determination criterion is set to an elevation angle of −10 °, but this is an example, and another determination criterion may be used depending on the implementation status. For example, if the subject to be photographed is a basketball court in a gymnasium, the elevation angle, which is a criterion for determination, may be made deeper, such as −20 °. By doing so, when the observation range is large in the vertical direction due to the nature of the observation target, it is possible to appropriately set the case of determining as a bird's-eye view viewpoint, appropriately determine the listening range, and express a realistic sound signal It becomes possible. Also, the horizontal plane projection range of the angle of view calculated in S203, which is the subsequent process, may be calculated in advance, and if the projection range falls within a predetermined range, an overhead view viewpoint may be obtained, and if not, a horizontal viewpoint may be obtained. As described above, in this embodiment, it is determined whether the elevation angle of the line of sight is below a predetermined negative value, and the process for determining the listening range is branched according to the determination result. It is possible to appropriately determine a listening range that is a basis for generating an acoustic signal.

  As a result of such determination, if it is below −10 °, that is, in the case of an overhead viewpoint (YES in S202), the process proceeds to S203. Otherwise, that is, in the case of a horizontal viewpoint (NO in S202), the process proceeds to S205.

  In S203, a range when the angle of view of this viewpoint information is projected onto a competition surface such as a shooting target stadium, that is, a horizontal plane is calculated, and this range is set as a listening range. As an example, a viewpoint video of a viewpoint having a viewpoint position (15, 0, 10), a viewpoint depression angle of −45 °, a viewpoint direction of 0 ° (that is, a positive Y-axis direction), a horizontal field angle of 20 °, and a vertical field angle of 14 °, The case where it projects on the 0 height competition surface is demonstrated. At this time, the horizontal plane coordinate of the center point of the projection plane (Z = 0) is (15, 10), and the projection plane has an upper end Y coordinate of 10 × tan 52 ° ≈12.8 and a lower end Y coordinate of 10 ×. It becomes a trapezoid of tan 38 ° ≈7.8. Here, half of the vertical angle of view 14 ° is 14 ° / 2 = 7 °, 52 ° = 45 ° + 7 °, and 38 ° = 45 ° -7 °.

  The distances between the center points of the upper and lower sides of the trapezoid and the viewpoint are 10 / cos 52 ° ≈16.2 and 10 / cos 38 ° ≈12.7, respectively. Therefore, when the left and right sides are opened 10 ° from each other (half of the horizontal angle of view 20 °), the length of the upper edge of the trapezoid is 16.2 × tan 10 ° × 2≈5.7, and the length of the lower edge is 12.7. Xtan10 ° × 2≈4.5.

  Therefore, the listening range is (12.15, 12.8), (17.85, 12.8), (12.75, 7.8), (17.25, 7) on the projection plane with Z = 0. .8) is an area surrounded by four points. Here, the x coordinates of both vertices of the upper edge of the trapezoid relating to the projection plane are 15−5.7 / 2 = 12.15 and 15 + 5.7 / 2 = 17.85. The x coordinates of both vertices of the lower edge of the trapezoid relating to the projection plane are 15−4.5 / 2 = 12.75 and 15 + 4.5 / 2 = 17.25. The listening range calculated in this way is stored in the listening information stored in the internal RAM of the listening range determination unit 2.

  Next, in S204, in the listening range determined in S203, the direction on the projection plane (horizontal plane) corresponding to the upward direction at the angle of view is set as the listening direction. In the case of the previous example, the angle is 0 ° (that is, the Y axis positive direction). When the process is finished, the process proceeds to S208.

  On the other hand, in S205, the subject position coordinates detected by the subject position detection unit 9 in S108 are acquired. Next, in S206, a range surrounding the acquired subject position is calculated and set as a listening range. For example, if the number of object positions acquired in S205 is three and the horizontal plane coordinates are (2, 2), (6, 3), (5, 6), the listening range surrounding this is (1, 1). , (1, 7), (7, 1), and (7, 7) are set as an area surrounded by four points. In other words, in this example, if the minimum value of the X coordinate at all subject positions is Xmin, the maximum value of the X coordinate is Xmax, the minimum value of the Y coordinate is Ymin, and the maximum value of the Y coordinate is Ymax, the listening range is It becomes as follows. That is, the listening range is a quadrangle having (Xmin-1, Ymin-1), (Xmin-1, Ymax + 1), (Xmax + 1, Ymin-1), and (Xmax + 1, Ymax + 1) as vertex coordinates. However, the listening range does not have to be a rectangle as long as it is a region having a necessary minimum size including all the acquired subject positions.

  Next, in S207, the viewpoint direction of the viewpoint information transmitted in S101 is stored in the listening point information as the listening direction as it is. In this way, in the case of a horizontal viewpoint, the viewpoint direction in the viewpoint video and the sound direction in the reproduction signal coincide with each other. When the process is finished, the process proceeds to S208.

  S208 is processing to store the center point of the listening range determined in S203 and S206 as listening points in the listening point information. In the example of the present embodiment, since the listening range is a quadrangle, a point where the diagonal lines intersect is calculated and stored as the listening point in the listening point information of the internal RAM. Note that the coordinates of the listening point may be the average coordinates of the vertices of a rectangle that defines the listening range. Next, in S209, the listening point information stored in the internal RAM is output to the sound collection point selection unit 3, and the listening range determination process is completed and the process returns.

  FIG. 6 is a schematic diagram illustrating the relationship between the viewpoint, the listening range, the listening point, and the listening direction in the present embodiment. In FIG. 6, the listening range is indicated by a dotted rectangle. The listening point is indicated by an octagonal star, and the listening direction is indicated by a black arrow. FIG. 6A shows the case of a bird's eye view, where the range of projection of the angle of view on the horizontal plane is the listening range, the intersection of the diagonals is the listening point, and the Y axis positive direction corresponding to the upward direction of the angle of view is listened to. The direction. On the other hand, FIG. 6B shows the case of the horizontal viewpoint, where the position of the subject in the viewpoint video is detected and the listening range is set so as to surround the position. The listening point is the intersection of the diagonal lines of the listening range, and the viewing direction (Y-axis positive direction) is the listening direction.

  As described above, in the listening range determination process according to the present embodiment, the listening range, listening point, and listening position corresponding to the arbitrary viewpoint video are automatically determined from the viewpoint information and the position of the subject. That is, a listening point serving as a reference for generating an acoustic signal corresponding to the image generated by the viewpoint video generation unit 8 is determined according to the viewpoint position and the direction of the line of sight, and a plurality of acoustic signals corresponding to the listening point are collected. Generated based on the sound signal. In this way, the sound of an arbitrary viewpoint is automatically generated by automatically determining the listening point, the listening range, and the listening direction according to the position of the viewpoint, the direction of the line of sight, the angle of view, the position of the subject, etc. Therefore, it is possible to faithfully express the change of sound according to the change of viewpoint. In the present embodiment, the listening range and the listening position are determined by detecting the projection range of the angle of view and the position of the subject. However, the center of the arbitrary viewpoint video may be always handled as the listening point.

  In the present embodiment, when the elevation angle of the line of sight falls below a predetermined negative value and is determined to be a bird's-eye view viewpoint, the range in the shooting target corresponding to the angle of view viewed from the viewpoint is determined as the listening range. To do. When it is determined that the elevation angle of the line of sight does not fall below a predetermined negative value and is a horizontal viewpoint, the range including the position of the subject in the photographing target is determined as the listening range. In this way, by branching the method for determining the listening range according to the elevation angle of the line of sight, it is possible to reproduce realistic sound according to the elevation angle of the line of sight.

  In the present embodiment, the listening direction indicating the listening direction at the listening point is determined based on the direction of the line of sight, and an acoustic signal corresponding to the listening direction is generated when generating the acoustic signal. Specifically, in the case of a bird's-eye view viewpoint, the upper direction of the angle of view is set as the listening direction, and in the case of a horizontal viewpoint, the viewpoint direction is determined as the listening direction. For this reason, the acoustic signal corresponding to the direction of the line of sight can be generated. In addition, the present invention can be carried out without departing from the gist of the present invention.

(Subject position detection processing)
FIG. 7 is a flowchart showing a detailed processing procedure of the subject detection processing in S108 in the present embodiment. All processes in this flowchart are executed by the subject position detection unit 9.

  First, in S301, all data temporarily stored in the internal RAM of the subject position detection unit are initialized. Next, in S302, the viewpoint video generated by the viewpoint video generation unit 8 in S107 is analyzed, and the subject in focus in the viewpoint video is detected and extracted. For example, by converting the viewpoint video into a contrast image, an edge image, or the like, a subject with a clear outline, that is, a subject in focus is detected. The subject to be extracted is not limited to a person such as a player but may be an object such as a car or a motorcycle. Moreover, the number may be one or plural. In S302, all such in-focus subjects are extracted, and the characteristics of each extracted image are temporarily stored in the internal RAM as subject information.

  Next, from S303 to S306, a loop process is performed on the individual subject information extracted in S302. First, loop processing is started in S303. In S304, among a plurality of camera images used for generating an arbitrary viewpoint video, a plurality of camera images obtained by capturing an image showing a subject to be processed are specified, and the camera position coordinates and the subject direction are specified. Ask for. Next, in S305, the position coordinates of the subject to be processed are calculated by the triangulation method from the plurality of camera position coordinates and the subject direction obtained in S304. The calculated coordinates are stored as subject position coordinates in the internal RAM of the subject position detection unit.

  In step S306, it is determined whether or not the processing has been completed for all subject information. If the processing has been completed, the process exits the loop, ends the subject position detection processing, and returns. Note that the subject position coordinates stored in the internal RAM are output to the listening range determination unit 2 as appropriate in response to a request from the listening range determination unit 2.

  Thus, in the present embodiment, the arbitrary viewpoint image generated by the viewpoint video generation unit is analyzed, and the position of the subject reflected in the image is detected. Therefore, the listening range can be appropriately determined by detecting the position of the subject without providing a dedicated sensor or the like for detecting the position of the subject. Note that the position of the subject may be detected using a position sensor or the like.

(Sound collection point selection processing)
FIG. 8 is a flowchart showing a detailed processing procedure of the sound collection point selection processing of S104 in the present embodiment. Note that all the processing in this flowchart is executed by the sound pickup point selection unit 3.

  First, in S401, a selected sound pickup point information list stored in the internal RAM of the sound pickup point selection unit 3 is initialized. The selected sound collection point information list is an area for storing information on the selected sound collection point. FIG. 4E shows an example of the data structure of the selected sound pickup point information in this embodiment. As shown in FIG. 4E, the selected sound collection point information includes a sound collection point ID, a corresponding sound source arrangement direction ID, and a direction viewed from the listening point. Among these, the sound collection point ID is an ID (identification information) for identifying the sound collection point, and is common to the data of the same name stored in the sound collection point information described with reference to FIG. Data is used. The corresponding sound source arrangement direction ID is a number (identification information) indicating the sound source arrangement direction covered by this sound collection point. The sound source arrangement direction will be described later. The direction viewed from the listening point is calculated from the direction of the sound collection point viewed from the listening point with reference to the listening direction.

  Next, in S402, based on the listening point information determined in S103, a rough arrangement direction of sound sources arranged around during reproduction is determined. In the present embodiment, the sound source placement direction is set as the sound source placement direction, with the listening direction as 0 ° as a starting point, and eight directions that make one round of the horizontal plane every 45 °.

  Next, from S403 to S410, a loop process is performed for each sound source arrangement direction set in S402. In S403, loop processing is started.

  In S404, an area within an angle range of ± 22.5 ° with respect to the target sound source arrangement direction as viewed from the listening point is set as a sound collection point search range. The sound collection point corresponding to the sound source arrangement direction is searched from this search range.

  In step S405, it is determined whether there is a sound collection point within the search range set in step S404. If there is a sound collection point within the search range (YES in S405), the process proceeds to S406. If not (NO in S405), it is determined not to assign a sound collection point in the target sound source arrangement direction, and the process proceeds to S410.

  S406 is processing for determining whether or not there is a sound collection point within the angle of the search range and outside the listening range. As a result of this determination, if there is a sound collection point (YES in S406), the process proceeds to S407. If not (NO in S406), the process proceeds to S408.

  S407 is processing for selecting a sound collection point closer to the listening point within the search range and outside the listening range as a sound source in this sound source arrangement direction. When a sound collection point is selected, a new element is added to the selected sound collection point information list stored in the internal RAM of the sound collection point selection unit 3, and the corresponding sound collection point ID of the selected sound collection point is associated. The sound source arrangement direction ID is stored. When the process is finished, the process proceeds to S409.

  On the other hand, in S408, the sound collection point farthest from the listening point within the search range and the listening range is selected as the sound source in this sound source arrangement direction. In this case as well, an element for storing information on the selected sound collection point is created and added to the selected sound collection point information list. When the process is finished, the process proceeds to S409.

  S409 is a process of calculating the direction seen from the listening direction of the sound collection point selected in S407 or S408, and storing it in the selected sound collection point information newly added in the preprocessing. For example, when the coordinates of the listening point are (1, 1) and the sound collecting point coordinates are (2, 1 + √3) and the listening direction is 60 °, the direction from the listening point to the sound collecting point is −30 °. Therefore, this angle is stored in the selected sound collection point information. When the process is finished, the process proceeds to S410.

  In S410, it is determined whether or not the processing for all sound source arrangement directions has been completed. If all the processing has been completed, the loop is terminated. Then, the process proceeds to S411.

  S411 is a process of selecting a sound collection point used for reproduction from among the sound collection points in the listening range and determining the arrangement direction. Details of this processing (listening point selection processing within the listening range) will be described later with reference to FIG.

  In S <b> 412, the selected sound collection point information list created by the processing so far is output to the acoustic signal generation unit 4. Then, the sound collection point selection process is terminated and the process returns.

  As described above, in the present embodiment, based on the listening range determined according to the viewpoint position and the direction of the line of sight, a sound collection point used for generating an acoustic signal is selected from a plurality of sound collection points, An acoustic signal is generated using the collected sound signal collected at the sound collection point. In this way, by selecting the sound collection points necessary for generating the reproduction signal before the process of generating the reproduction signal in S105, the processing necessary for generating the reproduction signal can be reduced. Furthermore, by automatically selecting a sound collection point that matches the range of interest in the viewpoint video, it is possible to generate a reproduced sound signal that is more immersive than the viewpoint video. Therefore, it is possible to generate a realistic sound suitable for an arbitrary viewpoint video with a small processing amount.

  Further, in the present embodiment, the shooting target viewed from the listening point determined according to the viewpoint position and the direction of the line of sight is divided into a plurality of areas, and the sound pickup points are determined based on the listening range from each of the plurality of areas. select. For this reason, the sound collection points around the listening point can be selected evenly based on the listening point.

  Further, when a sound collection point exists within the listening range in the region viewed from the listening point, a sound collecting point farthest from the listening point is selected from among the sound collecting points existing within the listening range. On the other hand, when there is no sound collection point within the listening range in the region viewed from the listening point, the sound collecting point closest to the listening point is selected from the sound collecting points present in the region. For this reason, it is possible to appropriately select a sound collection point corresponding to the expansion of the listening range and appropriately generate a reproduction signal.

  In the present embodiment, an example has been described in which sound sources arranged around in the reproduced sound signal are selected as eight directions, but the direction of the sound source is not limited to eight directions, and may be more or less than this. Further, in this embodiment, an example in which the periphery of the listening point is equally divided as the sound source arrangement direction has been described. However, it is not equally divided, for example, divided and selected in a direction corresponding to the channel direction in the sound reproduction environment. May be.

(Sound collection point selection processing within the listening range)
FIG. 9 is a flowchart showing a detailed processing procedure of the sound collection point selection processing within the listening range in S411 in the present embodiment. Note that all the processing in this flowchart is also executed by the sound pickup point selection unit 3.

  First, in S501, the sound collection points included in the listening range are listed and temporarily stored in the internal RAM of the sound collection point selection unit 3. Next, from S502 to S505, a loop process is performed for each sound collection point listed in S501. In step S502, loop processing is started.

  In S503, it is determined whether or not the sound collection point to be processed is included in the selected sound collection point information list stored in the internal RAM. If the target sound collection point is not included in the selected sound collection point list (NO in S503), the process proceeds to S504. If it is included (YES in S503), since it has already been selected as a sound collection point used for reproduction, the process proceeds to S505 to end the loop processing.

  In S504, a new element is added to the selected sound collection point list stored in the internal RAM, and the sound collection point ID of this sound collection point and 0 ° as the direction viewed from the listening point are stored. Thereby, in this embodiment, in the subsequent reproduction signal generation processing, the signal collected at the sound collection point within the listening range is reproduced so as to be localized in front of the listener. When the process is finished, step S505 follows.

  In S505, it is confirmed whether or not the processing has been completed for all the sound collection points listed in S501. If all the sound collection points have been completed, the loop processing is exited, the sound collection point selection processing in the listening range is terminated, and the process returns.

  As described above, in this embodiment, since all the sound collection points existing in the listening range are selected and the reproduction signal is generated, it is possible to generate a realistic sound according to the listening range. In addition, in each of the plurality of areas viewed from the listening point, the sound collecting point closest to the listening point is selected, so that it is possible to generate a realistic sound signal based on the positional relationship between the listening point and the sound collecting point. It becomes.

(Playback signal generation processing)
FIG. 10 is a flowchart showing a detailed processing procedure of the reproduction signal generation processing of S105 in the present embodiment. In the reproduction signal generation process, an acoustic signal corresponding to the listening point and the listening range is generated based on the plurality of collected sound signals. Note that all the processes in this flowchart are executed by the acoustic signal generation unit 4.

  S601 is a process of initializing and clearing the output buffer in the acoustic signal generation unit 4. The output buffer is a buffer for each output channel of the reproduced sound signal, and stores the generated sound signal for each output channel. When the process is finished, step S602 follows.

  S602 is processing for determining an environment in which an acoustic signal to be generated will be reproduced. As described above, in the example of the present embodiment, a stereo playback environment, a surround playback environment, and a headphone playback environment are provided as environments for playing back an audio signal, and the sound playback format is set to one of these environments. Is set.

  If it is a stereo reproduction environment, the process proceeds to S603. If it is a surround playback environment, the process proceeds to S604. If it is a headphone playback environment, the process proceeds to S605.

  In step S603, a stereo reproduction signal is generated using the sound collection signal at the sound collection point selected in step S104. Details of this processing will be described later with reference to FIG. When the process is finished, step S606 follows.

  S604 is processing for generating a surround reproduction signal using the sound collection signal at the sound collection point selected in S104. Details of this processing will be described later with reference to FIG. When the process is finished, step S606 follows.

  S605 is a process of generating a headphone playback signal using the sound collection signal at the sound collection point selected in S104. Details of this processing will be described later with reference to FIG. When the process is finished, step S606 follows.

  S606 is a process of outputting the reproduced sound signal generated in the immediately preceding process to the sound reproducing unit 11 or the MUX 15. When the process is finished, the reproduction signal generation process is finished and the process returns. Note that the flowchart of FIG. 10 shows an example in which any one playback format is selected and generated, but all of these formats may be generated sequentially.

  As described above, in the present embodiment, the listening direction indicating the listening direction at the listening point is further determined based on the direction of the line of sight, and the sound collection signal collected at the sound collection point selected in S104 is used. Thus, an acoustic signal that can be heard from the front in the listening direction is generated. For this reason, the sound corresponding to the arbitrary viewpoint image can be expressed with good reproducibility in consideration of the direction.

(Stereo playback signal generation processing)
FIG. 11 is a flowchart showing a detailed processing procedure of the stereo reproduction signal generation processing of S603 in the present embodiment. Note that all the processing in this flowchart is also executed by the acoustic signal generation unit 4.

  From S701 to S709, a loop process is performed on each selected sound collection point information stored in the selected sound collection point information list output from the sound collection point selection unit 3 in S104.

  In S701, loop processing is started. In S702, whether or not the direction viewed from the listening point of the selected sound collection point information to be processed is within a range of −90 ° to 90 °, that is, the target sound collection point is positioned forward from the side as seen from the listening point. Determine whether to do. If the result of this processing is not within the range, that is, if the sound collection point is behind (NO in S702), the processing proceeds to S703. If not, that is, if it is ahead (YES in S702), the process proceeds to S707.

  In S703, the collected sound signal stored in the selected selected sound collection point information is inverted in phase. Thereby, although a listener does not feel a sound image behind, the production which can hear the sound from the back unlike a normal sound can be performed. Next, in S704, it is determined whether the direction of the sound collection point viewed from the listening point is positive. If not positive, that is, if the sound collection point is on the left as viewed from the listening point (NO in S704), the process proceeds to S705. If it is positive, that is, if the sound collection point is on the right side when viewed from the listening point (YES in S704), the process proceeds to S706.

  In S705, the sign is inverted by adding 180 ° in the direction seen from the listening point. On the other hand, in S706, the sign is inverted by subtracting 180 ° from the direction seen from the listening point. By the processing of S705 and S706, the rear half of the circle centered on the listening point can be folded forward to convert the rear direction to the front direction. When the process is finished, step S707 follows.

In S707, stereo panning calculation is performed in the direction viewed from the listening point in the range of −90 ° to 90 °, and the collected sound signal is distributed to the L and R channels according to the obtained amplitude distribution ratio of the L and R channels. In general, in the standard stereo playback environment, the left and right speakers are placed at ± 30 °, so the stereo panning calculation is performed by linearly projecting the direction seen from the listening point in the ± 90 ° range to the ± 30 ° range. I do. In this embodiment, this panning calculation is performed as follows using a sine rule, where θ is the direction viewed from the listening point.
wL = (sin30 ° −sin (θ * 30/90)) / 2sin30 ° = 1 / 2−sin (θ / 3)
wR = (sin30 ° + sin (θ * 30/90)) / 2sin30 ° = 1/2 + sin (θ / 3) (1)
Here, wL is the amplitude distribution ratio for the left channel, and wR is the amplitude distribution ratio for the right channel.

  Next, in S708, the channel signal distributed in S707 is added to the output buffer of each channel. In S709, it is confirmed whether or not the processing of all selected sound collection point information included in the selected sound collection point information list has been completed. When all the processing is completed, the loop processing is exited, the stereo reproduction signal generation processing is terminated, and the process returns.

(Surround playback signal generation processing)
FIG. 12 is a flowchart showing a detailed processing procedure of the surround reproduction signal generation processing of S604 in the present embodiment. Note that all the processing in this flowchart is also executed by the acoustic signal generation unit 4.

  From S801 to S807, a loop process is performed for each selected sound collection point information included in the selected sound collection point information list.

  In step S801, loop processing is started. In S802, it is determined whether or not the direction of the sound collection point viewed from the listening point is the default channel arrangement direction. For example, if the playback environment is a 5.1 channel surround playback environment, the default channel arrangement angles are 0 °, ± 30 °, and ± 110 ° to 130 °. When the direction seen from the listening point of the sound collection point indicated by the target selected sound collection point information corresponds to an angle within this range, it is determined that the channel arrangement direction is set. In this case (YES in S802), the process proceeds to S806. If not (NO in S802), the process proceeds to S803.

  S803 is processing for selecting two channels in a direction that sandwiches the angle of the direction of the sound collection point viewed from the listening point. For example, if the direction of the sound collection point as viewed from the listening point is 50 °, the 30 ° R channel and the 120 ° SR channel are selected as the channels sandwiching it.

Next, in step S804, amplitude panning calculation is performed between the channels selected in step S803, and the sound collection signals at the sound collection points are distributed to the two channels. In the present embodiment, amplitude panning calculation is performed by a sine rule. In the above example, the central direction of the R and SR directions is 75 °, and the opening angle between the central direction and the direction of each channel is 45 °. Further, when viewed from the center direction between the channels, 50 ° is 50 ° −75 ° = −25 °. The distribution ratios wR and wSR to the R and SR channels can be obtained by the following equations from the sine rule.
wR = (sin45 ° −sin (−25 °)) / 2sin45 ° ≈0.649
wSR = (sin45 ° + sin (−25 °)) / 2sin45 ° ≈0.351 (2)
Next, in S805, the signal of each channel distributed in S804 is added to the output buffer for each channel. On the other hand, in S806, the collected sound signal is added as it is to the output buffer of the channel having the same direction determined in S802.

  In S807, it is confirmed whether or not the processing for all the selected sound collection point information included in the selected sound collection point information list is completed. When all the processes are finished, the loop process is finished and the process proceeds to S808.

  In S808, an LFE (Low Frequency Element) signal is generated by adding a low-pass filter (LPF) to the acoustic signal accumulated in the output buffer of each channel, and adding the result. The LFE signal is a low-frequency signal, and usually a signal of 80 Hz or less is extracted by a low-pass filter. This signal is reproduced by a subwoofer included in the surround speaker set. The generated LFE signal is accumulated in the output buffer for the LFE channel. When the process is finished, the surround reproduction signal generation process is finished and the process returns.

(Headphone playback signal generation processing)
FIG. 13 is a flowchart showing a detailed processing procedure of the headphone reproduction signal generation processing in S605 in the present embodiment. In addition, all the processes in this flowchart are also performed by the four acoustic signal generation units.

  From S901 to S904, a loop process is performed on each selected sound collection point information included in the selected sound collection point information list.

  In step S901, loop processing is started. In S902, the binaural signal is calculated by convolving the HRIR in the direction seen from the listening point with the collected sound signal. HRIR is a head related impulse response. HRIR is an impulse response for both ears by measuring the wraparound of sound by the human head and pinna, which changes depending on the sound source direction. By convolution of the collected sound signal, a stereophonic sound signal in which the collected sound signal is localized in the direction viewed from the listening point when listening with headphones can be created. In the present embodiment, a database storing the HRIR for each direction is stored in the internal ROM of the acoustic signal generation unit 4, and the HRIR for both ears is read by inputting an arbitrary direction and searching. Can be used.

  Next, in S903, the binaural signal generated in S902 is added to the output buffer for each of the L and R output channels.

  In S904, it is confirmed whether or not the processing for all the selected sound collection point information included in the selected sound collection point information list is completed. When all the processes are completed, the process exits the loop, ends the headphone reproduction signal generation process, and returns.

  In the present embodiment, the reproduction signal is generated using the collected sound signals of the omnidirectional sound collection points even in the stereo reproduction process. For example, in the case of stereo, only the collected sound signal of the sound collection point in the front is used. A reproduction signal may be generated using

  In this embodiment, a microphone for sound collection is installed at the position of the sound collection point, but the method for realizing sound collection is not limited to such a mode. For example, by using multiple microphones that can pick up far-reaching sounds, picking up and processing the sound corresponding to the sound pickup point and picking up the sound at the sound pickup point aimed from a long distance. You may sound.

  In this embodiment, the sound collection signal and the captured video signal are immediately processed to generate and reproduce the arbitrary viewpoint video and the sound signal corresponding thereto. However, the sound collection signal and the captured video signal are temporarily stored in the storage device. It may be processed later.

  As described above, the configuration according to the present embodiment automatically changes the listening range, the listening point, and the listening position according to the arbitrary viewpoint video from the viewpoint information, thereby changing according to the movement of the arbitrary viewpoint. Sound field reproduction with a feeling can be realized.

  Also, by determining the listening range according to the arbitrary viewpoint video from the viewpoint information, selecting the minimum sound collection point according to the listening range and arranging it appropriately in the playback sound field, it is possible to reduce the processing amount to the video. Sound field reproduction with a sense of realism can be realized. That is, by selecting a sound collection point to be used for generating an acoustic signal based on the listening range and generating an acoustic signal, there is a sense of presence corresponding to an arbitrary viewpoint video by selecting the minimum necessary sound collection signal. An acoustic signal can be automatically generated.

<< Other Embodiments >>
In the first embodiment, the reproduction signal is generated using all the collected sound signals from the sound collection points in the listening range. However, it is also possible to select and use an important sound collection signal among the sound collection signals in the listening range. it can. Here, an example of selecting a sound collection signal including a human voice (hereinafter referred to as “voice”) will be described as an example of an important sound collection signal. Hereinafter, an embodiment in this case will be described.

  The difference between the present embodiment and the first embodiment is the sound collection point selection processing within the listening range in S411 in FIG. 8, and the others are the same, so the description thereof will be omitted, and a brief description will be given focusing on the differences from the first embodiment. .

  FIG. 14 is a flowchart illustrating a detailed processing procedure of the sound collection point selection processing within the listening range in the present embodiment. The processing from S1001 to S1003 is the same as the processing from S501 to S503 in FIG.

  S1004 is a process of analyzing the sound collection signal of the target sound collection point information and determining whether or not sound is included. By performing pitch detection processing, formant detection processing, or the like on the collected sound signal, it is determined whether or not sound is included in the collected sound signal. As a result of the determination, if sound is included, the process proceeds to S1006. Otherwise, the process proceeds to S1005.

  S1005 is processing for determining whether or not the average amplitude of the collected sound signal exceeds a predetermined value (threshold value). As a result of the determination, if the threshold value is exceeded, the process proceeds to S1006. Otherwise, the process proceeds to S1007.

  Since S1006 and S1007 are the same processes as S504 and S505 in FIG.

  By performing the processing control described above, the signal collected at the sound collection point within the listening range is mixed with the voice that is estimated to contain important information, and the source of the sound source. Only signals with a large average amplitude that can be estimated to be close can be selected and reproduced. Therefore, it is possible to select and reproduce only important sounds while further reducing the amount of processing required for the reproduction signal generation processing.

  In the above configuration, an example in which an acoustically important sound collection point such as a human voice is detected when a sound collection point within the listening range is selected has been described. Furthermore, it is also possible to analyze a viewpoint video, specify a place where a predetermined event such as kicking a ball or forming a scrum occurs, and select a sound pickup point closest to the place. The occurrence of an event can be detected by user designation, use of a sensor, or the like.

  In the first embodiment, the sound in the upward direction of the screen is arranged forward in the case of the overhead view point. However, in the case of outputting to a playback format in which channels are arranged in the upper layer in addition to the horizontal plane, the upper layer channel You may make it arrange | position to. As described above, the sound collection point is generated by synthesizing the sound collection signals collected at the sound collection point based on the positional relationship viewed from the viewpoint of the sound collection point selected in S104 and generating the acoustic signal. It is possible to reproduce the realistic sound according to the arrangement. In addition, the present invention can be carried out without departing from the gist of the present invention.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Sound collection signal input unit: 1, listening range determination unit: 2, sound collection point selection unit: 3, sound signal generation unit: 4, operation unit: 5, viewpoint information designation unit: 6, video signal input unit: 7, viewpoint Video generation unit: 8, subject position detection unit: 9, video playback unit: 10, sound playback unit: 11, stereo speaker set: 12, surround speaker set: 13, headphones: 14, display unit: 19

Claims (15)

An information processing system that processes an image and sound corresponding to an arbitrary viewpoint based on a plurality of image signals photographed by a plurality of photographing devices and a plurality of sound collection signals collected at a plurality of sound collection points. There,
An acquisition means for acquiring the viewpoint position and the direction of the line of sight with respect to the imaging target;
A sound collection point of a sound collection signal used to generate an acoustic signal corresponding to an image based on the plurality of image signals, the image being in accordance with the viewpoint position and the direction of the line of sight, from the plurality of sound collection points. Selecting means for selecting according to the viewpoint position and the direction of the line of sight;
An information processing system comprising: an acoustic generation unit that generates an acoustic signal using a collected sound signal collected at a sound collection point selected by the selection unit. Determination to determine a listening range, which is a reference range for selecting a sound pickup point of a sound pickup signal used for generating an acoustic signal corresponding to the image, according to the viewpoint position and the direction of the line of sight Means and
The information processing system according to claim 1, wherein the selection unit selects a sound collection point to be used for generating the acoustic signal from the plurality of sound collection points based on the listening range. The determining means further determines a listening point as a reference for generating the acoustic signal according to the viewpoint position and the direction of the line of sight,
The said selection means divides the said imaging | photography object seen from the said listening point into several area | region, and selects the said sound collection point from each of the said several area | region based on the said listening range. Information processing system described in 1.   When there is a sound collection point within the listening range in the region, the selection means selects a sound collection point farthest from the listening point among the sound collection points existing within the listening range. The information processing system according to claim 3.   When the sound collection point does not exist within the listening range in the area, the selection means selects a sound collection point closest to the listening point from among the sound collection points existing in the area. The information processing system according to claim 3 or 4.   6. The information processing system according to claim 3, wherein the selection unit selects a sound collection point closest to the listening point in each of the plurality of regions. The determining means further determines a listening direction indicating a listening direction at the listening point based on the direction of the line of sight;
The sound generation means generates an acoustic signal that can be heard from the front in the listening direction by using a sound pickup signal picked up at a sound pickup point selected by the selection means. The information processing system according to any one of the above.   The information processing system according to any one of claims 2 to 7, wherein the selection unit selects all sound collection points existing in the listening range.   The information processing system according to any one of claims 1 to 8, wherein the selection unit selects a sound collection point where a sound collection signal including a human voice is collected.   10. The information processing system according to claim 1, wherein the selection unit selects a sound collection point where a sound collection signal having an average amplitude exceeding a predetermined threshold is collected. .   The information processing system according to any one of claims 1 to 10, wherein the selection unit selects a sound pickup point closest to a place where occurrence of a predetermined event is detected.   The sound generation means generates the sound signal by synthesizing the sound pickup signals collected at the sound pickup points based on the positional relationship of the sound pickup points selected by the selection means as viewed from the viewpoint. The information processing system according to any one of claims 1 to 11, wherein: An information processing system that processes an image and sound corresponding to an arbitrary viewpoint based on a plurality of image signals photographed by a plurality of photographing devices and a plurality of sound collection signals collected at a plurality of sound collection points. There,
An acquisition means for acquiring the viewpoint position and the direction of the line of sight with respect to the imaging target;
An listening point serving as a reference for generating an acoustic signal corresponding to the image based on the plurality of image signals, which is an image according to the viewpoint position and the direction of the line of sight, is determined according to the viewpoint position and the direction of the line of sight. A determination means;
Selection means for selecting a sound collection point used for generating the acoustic signal from the plurality of sound collection points based on a positional relationship between the listening point and the sound collection point;
An information processing system comprising: an acoustic generation unit that generates an acoustic signal using a collected sound signal collected at a sound collection point selected by the selection unit. An information processing system for processing an image and sound corresponding to an arbitrary viewpoint based on a plurality of image signals photographed by a plurality of photographing devices and a plurality of sound collection signals collected at a plurality of sound collection points A control method,
An acquisition step in which the acquisition unit acquires the viewpoint position and the direction of the line of sight with respect to the imaging target;
The selection means is an image according to the viewpoint position and the direction of the line of sight, and the sound collection points of the sound collection signal used for generating an acoustic signal corresponding to the image based on the plurality of image signals are the plurality of sound collection points. A selection step of selecting from the sound collection point according to the viewpoint position and the direction of the line of sight;
A method for controlling an information processing system, comprising: a sound generation unit that generates a sound signal using a collected sound signal picked up at a sound collection point selected in the selection step.   The computer program for functioning a computer as each means with which the information processing system of any one of Claim 1 to 13 is provided.

Images