JP2022007108A - Information processor, information processing method, and program - Google Patents

Abstract

To estimate the position of a sound source relating to a virtual viewpoint image.SOLUTION: An information processing system 10 acquires viewpoint information showing the transition of a virtual viewpoint corresponding to a virtual viewpoint image generated on the basis of a plurality of picked-up images obtained by imaging an imaging area from a plurality of directions. The information processing system 10 estimates the position of a sound source in the imaging area on the basis of acquired viewpoint information.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for estimating the position of a sound source.

There is a technique in which a plurality of photographing devices are installed at different positions to perform synchronous shooting from multiple viewpoints, and a virtual viewpoint image in which the viewpoint can be arbitrarily changed is generated by using a plurality of images obtained by the shooting. For example, a user can watch a game from various viewpoints by generating a virtual viewpoint image corresponding to a viewpoint designated by the user based on a plurality of images of a game such as soccer or basketball.

In addition, it is being studied to make the acoustic signal reproduced together with the virtual viewpoint image highly realistic. The virtual viewpoint corresponding to the virtual viewpoint image can be set at any position in the field where the competition to be shot is held, but a microphone for collecting sound should be brought into the field to follow the virtual viewpoint. Is difficult. Therefore, it is conceivable to install multiple microphones around the field and select and mix the sound pickup signals obtained from those microphones to generate an acoustic signal to be reproduced together with the virtual viewpoint image. .. In this case, the presence of the acoustic signal can be improved by selecting and mixing the sound pickup signal based on the position of the sound source in the field.

Patent Document 1 discloses that a sound collecting position is selected based on a position of a subject included in a field of view corresponding to a virtual viewpoint in order to generate an acoustic signal to be reproduced together with a virtual viewpoint image. Further, Patent Document 1 discloses a method of analyzing a virtual viewpoint image and a method of using a position sensor as a method of detecting the position of a subject.

Japanese Unexamined Patent Publication No. 2018-19295

However, when an acoustic signal is generated based on the position of a sound source detected by analyzing the generated virtual viewpoint image as in the technique described in Patent Document 1, from the generation of the virtual viewpoint image to the generation of the acoustic signal. There is a risk that the delay will increase. In addition, it may be difficult to attach a position sensor to the subject depending on the competition or event to be photographed.

In view of the above problems, it is an object of the present invention to provide a new method for estimating the position of a sound source related to a virtual viewpoint image.

In order to solve the above problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, the acquisition means for acquiring the viewpoint information representing the transition of the virtual viewpoint corresponding to the virtual viewpoint image generated based on the plurality of captured images obtained by photographing the shooting area from a plurality of directions, and the acquisition means. It has an estimation means for estimating the position of a sound source in the photographing region based on the acquired viewpoint information.

According to the present invention, the position of the sound source related to the virtual viewpoint image can be estimated.

It is a figure which shows the configuration example of the system which concerns on embodiment. It is a flowchart which shows the process in a learning phase. It is a flowchart which shows the process in an application phase. It is a flowchart which shows the process about relearning. It is a figure which shows the example of a microphone position and a sound source position. It is a figure which shows the configuration example of the system which concerns on the modification. It is a flowchart which shows the process in a learning phase. It is a flowchart which shows the process in an application phase.

[System configuration]
FIG. 1A is a block diagram showing a configuration of an information processing system 10 according to the present embodiment. Further, FIG. 1B is a block diagram showing a configuration of the sound generation system 20 according to the present embodiment. The information processing system 10 performs learning using the camera path corresponding to the virtual viewpoint image and the detected sound source position, and generates a trained model. The sound generation system 20 estimates the sound source position by inputting the input camera path into the trained model and processing it. In the present embodiment, the phase in which the learned model is generated and updated by the information processing system 10 is called the learning phase, and the phase in which the learning model is applied by the sound generation system 20 and the sound source position is estimated is the application phase. Called.

The virtual viewpoint image is generated based on a plurality of images taken by a plurality of image pickup devices and a designated virtual viewpoint, and represents a scene from the designated virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images are also included in the virtual viewpoint video. The virtual viewpoint may be specified by a user operation, or the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may include a still image. It can be said that the virtual viewpoint image is an image simulating the captured image obtained by the camera, assuming that the camera exists at the position of the virtual viewpoint set in the space. In the present embodiment, the viewpoint information indicating the contents of the transition of the virtual viewpoint over time is referred to as a camera path.

The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. The camera path contains multiple parameter sets corresponding to each of the multiple times. For example, the camera path may have a plurality of parameter sets corresponding to a plurality of frames constituting the moving image of the virtual viewpoint video, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. ..

As shown in FIG. 1A, the information processing system 10 includes a sound source detection unit 111, a video data storage unit 112, a camera path generation unit 130, and a model generation unit 100. The model generation unit 100 includes a camera path receiving unit 101, a teacher data generation unit 102, a learning unit 103, a sound source information storage unit 104, a sound source type storage unit 105, and a re-learning unit 106.

The camera path receiving unit 101 receives the camera path generated by the camera path generating unit 130. As a method of transmitting and receiving the camera path, for example, the viewpoint information for each frame may be transmitted / received during the generation of the camera path, or the viewpoint information for all frames may be transmitted / received collectively after the generation of the camera path is completed. good.

The teacher data generation unit 102 acquires the camera path received by the camera path receiving unit 101, the sound source position corresponding to the camera path, the sound source generation time, and the sound source type ID from the sound source information storage unit. Then, the acquired camera path, sound source position, sound source generation time, and sound source type ID are transmitted to the learning unit 103 as a set of teacher data. The sound source position corresponding to the camera path is the position of the sound source related to the virtual viewpoint image generated based on the camera path. For example, consider a case where a virtual viewpoint image is generated based on a plurality of captured images and a camera path obtained by photographing a field in which a game is played from a plurality of directions in a certain shooting period. In this case, the position of the sound source that emits sound in the field (in the shooting area) during the shooting period is the sound source position corresponding to the camera path.

The learning unit 103 creates and updates a sound source calculation model, which is a learned model, by machine learning based on the received teacher data. At this time, learning is performed by treating the sound source position, the sound source generation time, and the sound source type ID of the teacher data as correct answer data. As a specific algorithm for machine learning, linear regression, logistic regression, support vector machine, neural network, or the like can be used.

The sound source information storage unit 104 stores sound source information related to the sound source generated during the match. For example, the sound source information is managed by a table as shown in Table 1 below. These data are generated by the sound source detection unit 111 and are treated as correct answer data. Further, these data are referred to by the teacher data generation unit 102.

In Table 1, column 104-1 stores a match ID for uniquely identifying which match the sound source is in. Column 104-2 stores the time when the sound source was generated. Column 104-3 stores a sound source type ID for identifying the type of the generated sound source. Column 104-4, column 104-5, and column 104-6 store the X-coordinate, Y-coordinate, and Z-coordinate of the generated sound source position, respectively. In the example of Table 1, the unit is meters.

The sound source type storage unit 105 stores sound source type information. For example, the sound source type is managed by a table as shown in Table 2 below.

In Table 2, column 105-1 stores a sound source type ID for identifying the sound source type. Column 105-2 stores the sound source type name corresponding to the sound source type ID. In the example of Table 2, the type names of sound sources that are expected to occur during a rugby game are stored.

The re-learning unit 106 determines the validity of the result of calculating the sound source position by the sound source position calculation unit 140 of the sound generation system 20 in the application phase, and feeds back the result to the model generation unit 100 to perform re-learning. The determination process is performed based on the data stored in the sound source history storage unit 129. This determination process will be described later with reference to FIG.

The sound source detection unit 111 detects a sound source based on the video data acquired from the video data storage unit 112, determines the match ID, the sound source generation time, the sound source type, and the sound source position, and stores those data in the sound source information storage unit 104. Store. In the present embodiment, it is assumed that the sound source is detected by performing the video recognition process on the video data. However, the sound source detection method is not limited to this. For example, the operator may manually input data of some or all sound sources while checking the video. Further, for example, the sound source position may be detected manually or automatically by using the GPS information output from the position sensor attached to the person or the object as the sound source. The video data storage unit 112 stores video data taken by the bird's-eye view camera. The bird's-eye view camera is a camera that shoots from a position that gives a bird's-eye view of the field.

The camera path generation unit 130 generates a camera path for generating a virtual viewpoint image. As a method of generating a camera path, for example, there is a method of generating a camera path from an input corresponding to the operation based on an operation in which a user moves a virtual viewpoint in a virtual space using a joystick controller. The camera path generated here is transmitted to the camera path receiving unit 101.

As shown in FIG. 1B, the sound generation system 20 has a sound generation unit 120 and a camera path generation unit 131. The sound generation unit 120 includes a camera path reception unit 121, a sound source information acquisition unit 122, a microphone selection unit 123, a competition sound generation unit 124, a cheer sound generation unit 125, and a mix unit 126. Further, the sound generation unit 120 includes a sound pickup signal storage unit 127, a camera path history storage unit 128, a sound source history storage unit 129, and a sound source position calculation unit 140.

The sound source position calculation unit 140 inputs the camera path generated by the camera path generation unit 131 into the sound source calculation model generated by the model generation unit 100 in the learning phase, and processes the sound source position and sound source generation time. And the sound source type ID are calculated. In the present embodiment, the input data to the sound source calculation model is used as the camera path, but the present invention is not limited to this. For example, a photographed image or a parameter obtained from the photographed image may be included in the input data. In this case, the teacher data for generating the sound source calculation model also includes the captured image and the parameters obtained from the captured image.

Here, the parameter obtained from the captured image is information such as the number and density of objects for each region in the field. By including such data in the teacher data and the input data, there is a high possibility that the sound source position can be calculated correctly in a situation where athletes are crowded and competition sounds are likely to occur, such as scrum in rugby. Further, the sound source position calculation unit 140 may perform correction processing on the sound source position output from the sound source calculation model in the application phase. As the correction process, for example, the sound source position may be corrected based on a sound pick-up signal picked up by one or more microphones or a parameter calculated from the sound pick-up signal.

Here, a supplementary explanation will be given as to why the sound source position and the sound source generation time can be estimated based on the camera path. The camera path is created by the user who operates the virtual viewpoint so that the subject he / she wants to pay attention to is within the field of view of the virtual viewpoint. On the other hand, competition sounds that the viewer wants to hear in order to improve the sense of presence are often generated at the position of the subject or in the vicinity of the subject. Therefore, there is a correlation between the position of the subject chased by the virtual viewpoint and the sound source position and the sound source generation time, and it becomes possible to estimate the sound source position and the sound source generation time from the camera path by machine learning.

Here, a supplementary explanation will be given as to why the sound source type can be inferred based on the camera path. For example, when generating a camera path for a place kick scene in rugby, a camera path is generated in which the virtual viewpoint shows the ball before being kicked up, and the virtual viewpoint chases the ball flying in the air immediately after being kicked. Often done. As in this example, the pattern of movement of the virtual viewpoint represented by the camera path has a correlation with the content of the play, and naturally there is a correlation between the content of the play and the sound source type. Therefore, there is a correlation between the camera path and the sound source type, and it is possible to estimate the sound source type from the camera path.

The camera path receiving unit 121 receives the camera path generated by the camera path generating unit 131. The sound source information acquisition unit 122 transmits the camera path acquired by the camera path reception unit 121 to the sound source position calculation unit 140, and acquires the sound source generation time, sound source type, and sound source position as return values.

The microphone selection unit 123 selects one or more microphones for collecting the competition sound included in the acoustic signal for reproduction according to the sound source position acquired by the sound source information acquisition unit 122. As a selection method, for example, a microphone may be selected so that the distance from the estimated sound source position to the microphone is equal to or less than a predetermined threshold value. Further, for example, a microphone having directivity toward the estimated sound source position may be selected, or a microphone having a distance from the sound source position less than or equal to the threshold value and having directivity toward the sound source position may be selected. ..

The competition sound generation unit 124 generates an acoustic signal to be used as a competition sound by using the sound collection signal picked up by the microphone selected by the microphone selection unit 123. The cheer sound generation unit 125 generates an acoustic signal to be used as a cheer sound by using a sound pick-up signal picked up by a cheer sound pick-up microphone installed separately from the microphone for picking up the competition sound. .. The mixing unit 126 mixes the sound source generated by the competition sound generation unit 124 and the sound source generated by the cheering sound generation unit 125, and generates an acoustic signal for reproduction to be reproduced together with the virtual viewpoint image.

The sound collection signal storage unit 127 stores the sound collection signal collected by the microphone installed in the stadium. In the present embodiment, the sound collection signal is stored in a format that allows a desired sound collection signal to be extracted by designating the microphone used for sound collection and the sound collection time. For example, the pick-up signal may be stored in the form of a database and the data may be referenced by SQL. Alternatively, the sound collection signal may be managed by a WAVE format file group, and the data may be referred to by specifying the file name.

The camera path history storage unit 128 stores the history of the camera path received by the camera path reception unit 121. For example, the history of the camera path is managed by a table as shown in Table 3 below.

Column 128-1 stores a camera path ID for uniquely identifying the camera path. Column 128-2 stores the contents of the camera path. As the storage format of the camera path, for example, the time information included in the camera path and the information on the position and orientation of the viewpoint may be expressed in JSON format and stored as one data. Further, for example, time information and information on the position and orientation of the viewpoint may be managed in the database.

The sound source history storage unit 129 stores the history of sound source information calculated by applying the sound source calculation model. For example, the history of sound source information is managed by a table as shown in Table 4 below.

Column 129-1 stores a match ID for uniquely identifying which match the sound source information is. Column 129-2 stores the ID associated with the camera path used as an input by the sound source position calculation unit 140 to calculate the sound source position. Column 129-3 stores the time when the sound source was generated. Column 129-4 stores a sound source type ID for identifying the type of the generated sound source. Columns 129-5, 129-6, and 129-7 store the X-coordinate, Y-coordinate, and Z-coordinate of the generated sound source position, respectively. In the example of Table 4, the unit of coordinates is meters.

Column 129-8 stores information as to whether or not the determination has been performed by the re-learning unit 106. In the present embodiment, when the TRUE is stored, the re-learning unit 106 has already determined the validity of the sound source position calculation result for the corresponding data, and the feedback of the result is transmitted to the sound source position calculation unit 140. It shows that.

Like the camera path generation unit 130, the camera path generation unit 131 generates a camera path for generating a virtual viewpoint image. The camera path generated here is transmitted to the camera path receiving unit 121.

FIG. 1C shows the hardware configuration of the model generation unit 100 and the sound generation unit 120. The model generation unit 100 can be realized by an information processing device having a CPU 161, a ROM 162, a RAM 163, an auxiliary storage device 164, a display unit 165, an operation unit 166, a communication I / F 167, a GPU 168, and a bus 169. The sound generation unit 120 can be realized by an information processing device having a CPU 171, a ROM 172, a RAM 173, an auxiliary storage device 174, a display unit 175, an operation unit 176, a communication I / F 177, a GPU 178, and a bus 179. Further, the model generation unit 100 and the sound generation unit 120 can communicate with each other via the network 180.

The CPU 161 realizes each function of the model generation unit 100 shown in FIG. 1A by controlling the entire model generation unit 100 using computer programs and data stored in the ROM 162 and the RAM 163. The model generation unit 100 may have one or more dedicated hardware different from the CPU 161 and the dedicated hardware may execute at least a part of the processing by the CPU 161. Examples of such dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

The ROM 162 stores programs and the like that do not require changes. The RAM 163 temporarily stores programs and data supplied from the auxiliary storage device 164, data supplied from the outside via the communication I / F 167, and the like. The auxiliary storage device 164 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and acoustic data. In the present embodiment, the sound source information storage unit 104, the sound source type storage unit 105, and the video data storage unit 112 are configured by the auxiliary storage device 164, but the present invention is not limited to this. For example, it may be configured by an external device connected via communication I / F 167.

The display unit 165 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the model generation unit 100. The operation unit 166 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 161 in response to an operation by the user. The CPU 161 operates as a display control unit that controls the display unit 165 and an operation control unit that controls the operation unit 166.

The communication I / F 167 is used for communication with an external device of the model generation unit 100. For example, when the model generation unit 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 167. When the model generation unit 100 has a function of wirelessly communicating with an external device, the communication I / F 167 includes an antenna.

Since GPU 168 can perform efficient calculation by processing more data in parallel, it is effective when learning is performed a plurality of times using a learning model such as deep learning. Therefore, in the present embodiment, the GPU 168 is used in addition to the CPU 161 for the processing by the learning unit 103. Specifically, when a learning program is executed using a learning model, learning is performed by the CPU 161 and the GPU 168 collaborating to perform calculations. However, the process of the learning unit 103 may be executed by either the CPU 161 or the GPU 168.

The bus 169 connects each part of the model generation unit 100 to transmit information.

In this embodiment, it is assumed that the display unit 165 and the operation unit 166 exist inside the model generation unit 100, but at least one of the display unit 165 and the operation unit 166 is another device outside the model generation unit 100. May exist as.

The hardware configuration of the sound generation unit 120 is the same as the hardware configuration of the model generation unit 100. However, the GPU 178 may be used for processing when the sound source position calculation unit 140 calculates the sound source position.

[Operation flow]
FIG. 2 is a flowchart showing the processing of the model generation unit 100 in the learning phase. The process shown in FIG. 2 is realized by the CPU 161 and the GPU 168 expanding the program stored in the ROM 162 into the RAM 163 and executing the program. It should be noted that at least a part of the processing shown in FIG. 2 may be realized by one or a plurality of dedicated hardware different from the CPU 161 and the GPU 168. The same applies to the processing of the flowcharts shown in FIGS. 4 and 7, which will be described later. In the process shown in FIG. 2, video data is stored in the video data storage unit 112, a camera path is generated by the camera path generation unit 130, and a sound source corresponding to the camera path is detected by the sound source detection unit 111, and then learning is performed. The start of the phase is started at the instructed timing. However, the start timing of the process shown in FIG. 2 is not limited to this.

In S201, the camera path receiving unit 101 receives the camera path generated by the camera path generating unit 130. In S202, the teacher data generation unit 102 takes a picture corresponding to the start time and end time of the camera path (that is, the virtual viewpoint image generated based on the camera path) based on the time information included in the camera path received in S201. Calculate the start time and end time of the period). In S203, the teacher data generation unit 102 acquires the match ID included in the camera path received in S201.

In S204, the teacher data generation unit 102 acquires sound source information including the sound source position, the sound source generation time, and the sound source type ID from the sound source information storage unit 104. In the sound source information acquired at this time, the match ID in Table 1 matches the match ID acquired in S203, and the sound source generation time in Table 1 is between the start time and the end time of the camera pass calculated in S202. Information on the sound source.

The processes of S205 to S208 are executed for each of the sound source information acquired in S204. In S206, the teacher data generation unit 102 generates a set of teacher data using the camera path received in S201 as input data and the sound source position, sound source generation time, and sound source type ID acquired in S202 as correct answer data. do. In S207, the learning unit 103 updates the sound source calculation model using the teacher data generated in S206. A sound source calculation model as a trained model is generated by repeating the update of the sound source calculation model for each sound source information.

FIG. 3 is a flowchart showing the processing of the sound generation unit 120 in the application phase. The process shown in FIG. 3 is realized by the CPU 171 and the GPU 178 expanding the program stored in the ROM 172 into the RAM 173 and executing the program. It should be noted that at least a part of the processing shown in FIG. 3 may be realized by one or a plurality of dedicated hardware different from the CPU 171 and the GPU 178. The same applies to the processing of the flowchart shown in FIG. 8 to be described later. In the process shown in FIG. 3, a sound source calculation model as a trained model is generated by the process of the learning phase shown in FIG. 2, and after the camera path is generated by the camera path generation unit 131, the start of the application phase is instructed. It starts at the timing. However, the start timing of the process shown in FIG. 3 is not limited to this.

In S301, the camera path receiving unit 121 receives the camera path generated by the camera path generating unit 131. In S302, the sound source information acquisition unit 122 acquires the sound source position, the sound source generation time, and the sound source type ID estimated by inputting the camera path acquired in S301 into the sound source calculation model. At this time, two or more sets of sound source information may be acquired. In S303, the sound source position, the sound source generation time, and the sound source type ID acquired in S302 are stored in the sound source history storage unit 129. The data stored here is referred to by the re-learning unit 106, and is used to determine whether or not the result calculated by the sound source calculation model is correct. This determination process will be described later with reference to FIG.

In S304, the microphone selection unit 123 selects the microphone to be used based on the sound source position calculated in S303. As a method of selecting a microphone, for example, as shown in FIG. 5, the distance between each of the microphones 501 to 512 and the sound source position 520 may be obtained, and a microphone whose distance is equal to or less than a certain threshold value may be selected. Further, for example, the microphone may be selected in consideration of the directivity of the microphone. Further, the microphone selection unit 123 determines not only the selection of the microphone but also the coefficient of the ratio of the sound pick-up signal picked up by each microphone to be mixed with the acoustic signal for reproduction as the competition sound for each microphone. You may.

In S305, the sound pick-up signal picked up by the microphone selected in S304 is acquired from the sound pick-up signal storage unit 127, and the competition sound is generated. As a method of generating a competition sound, for example, when the sound source type name indicated by the sound source type ID is a place kick, even if only the impact sound of the kick is cut out from the sound collection signal in a short time based on the sound source generation time. good. In S306, a cheering sound corresponding to the camera path is generated. As a method for generating cheers, for example, a stereo channel may be generated by arranging the pick-up signals picked up by a plurality of cheers microphones in a well-balanced manner on the L channel and the R channel. Further, a stereo channel or a multi-channel sound generation such as 5.1 channel may be generated so as to follow the change of the virtual viewpoint.

In S307, the competition sound generated in S305 and the cheering sound generated in S306 are mixed to generate an acoustic signal to be reproduced together with the virtual viewpoint image. As a mixing method, for example, a method of calculating the average level of the competition sound and the cheering sound and automatically adjusting the volume so as to have the same level and mixing can be specified. The generated acoustic signal for reproduction is output from the acoustic generation unit 120 to an external speaker, a network, or a storage device. In this way, by synthesizing the sound pickup signal based on the sound source position estimated from the camera path to generate an acoustic signal for reproduction, the height suitable for reproduction together with the virtual viewpoint image corresponding to the camera path. Realistic acoustic signals can be generated.

Next, with reference to FIG. 4, a process in which the re-learning unit 106 determines whether or not the estimation result of the sound source position in the application phase is correct and causes the sound source calculation model to relearn the determination result will be described. The process shown in FIG. 4 is started at the timing when the process of the application phase shown in FIG. 3 is performed. However, the start timing of the process shown in FIG. 4 is not limited to this.

In S401, among the data stored in the sound source history storage unit 129, a part or all of the data in which the determined flag of the column 129-8 is FALSE is acquired. The processing of S402 to S407 is executed for each data acquired in S401. In S403, the match ID of the data acquired in S401 is acquired. In S404, the bird's-eye view camera image of the match corresponding to the match ID acquired in S403 is acquired from the video data storage unit 112. However, if there is no bird's-eye view camera image corresponding to the match ID, it may be replaced with data that can determine the sound source position calculation result, such as broadcast data at the time of TV broadcasting. If the video data capable of the above determination cannot be obtained, the determination may be omitted.

In S405, it is determined whether or not the sound source position, the sound source generation time, and the sound source type ID in the history data acquired in S401 are correct based on the image of the bird's-eye view camera. As a method of determination, for example, at the sound source generation time acquired by S401, the sound source position is marked on the image of the bird's-eye view camera, and the judge visually determines whether the answer is correct or incorrect. May be good. In that case, the re-learning unit 106 has a UI for receiving the determination result, and the input information (TRUE / FALSE) is used as the determination result.

In S406, the sound source calculation model is updated by feeding back the determination using the determination result acquired in S405 and the data acquired in S401. By such re-learning, the sound source calculation model is improved, and the accuracy of sound source position estimation in the subsequent application phase is improved.

[Modification example]
In the above-described embodiment, a configuration is described in which a sound source calculation model that outputs a sound source position by inputting a camera path is generated by machine learning. However, the input to the sound source calculation model is not limited to the camera path. In the configuration described below, instead of inputting the camera path, a sound source calculation model that estimates the sound source position is generated by machine learning by inputting the position and orientation of the microphone and the sound collection signal collected by the microphone. ..

FIG. 6A is a block diagram showing a configuration example of the information processing system 11 that performs the processing of the learning phase in this modification. FIG. 6B is a block diagram showing a configuration example of the sound generation system 21 that processes the application phase in this modification. In the following, the description of the portion where the same processing as that of the configuration described with reference to FIGS. 1 (A) and 1 (B) is performed will be omitted, and the description will be focused on the difference. The hardware configuration is the same as that described with reference to FIG. 1 (C).

The microphones 641-1 to 641-M are microphones installed around the field for collecting sounds (for example, cheering sounds or competition sounds in the field) in the sound collecting area. The field is also a shooting area to which a shooting device for acquiring a plurality of shot images used for generating a virtual viewpoint image is directed. The sound pick-up signal picked up by the microphones 641-1 to 641-M is stored in the sound pick-up signal storage unit 632.

The microphone information storage unit 631 stores microphone information indicating the position and orientation of the installed microphone. For example, microphone information is managed by a table as shown in Table 5 below.

The column 631-1 stores a match ID for uniquely identifying the information of the microphone installed in which match. Column 631-2 stores an ID for uniquely identifying the installed microphone. Columns 631-3, 631-4, and 631-5 store the X, Y, and Z coordinates of the microphone position, respectively. In the example of Table 5, the unit of coordinates is meters. Columns 631-6 and 631-7 are two angles for expressing the direction of the microphone in the spherical coordinate system, respectively. The column 631-6 is an angle formed by the Z axis and the radius, and takes a value from 0 degrees to 90 degrees. Also, 90 degrees represents the horizontal direction. Columns 631-7 are angles formed by the X-axis and the projection of the radius onto the XY plane, and take values from 0 degrees to 360 degrees. Further, 0 degree represents the X-axis direction.

The sound collection signal storage unit 632 stores the sound collection signal collected by the microphone installed in the field. The sound collecting signal storage unit 632 is arranged at a location accessible from the model generation unit 600.

The teacher data generation unit 601 acquires microphone information from the microphone information storage unit 631 and acquires a sound collection signal from the sound collection signal storage unit 632. Further, the teacher data generation unit 601 acquires the sound source position, the sound source generation time, and the sound source type ID from the sound source information storage unit 603 for all the competition sounds generated in the sound collection period corresponding to the sound collection signal. Then, the teacher data generation unit 601 transmits the acquired microphone information, sound pickup signal, sound source position, sound source generation time, and sound source type ID to the learning unit 602 as teacher data. The learning unit 602 creates and updates a sound source calculation model based on the received teacher data. At this time, the sound source position, the sound source generation position, and the sound source type ID, which are teacher data, are treated as correct answer data.

The sound source position calculation unit 660 inputs the position of the microphone, the direction of the microphone, and the sound collection signal picked up by the microphone into the sound source calculation model, and estimates the sound source position, the sound source generation time, and the sound source type ID. In the learning phase of the sound source calculation model, the model is generated and updated by the model generation unit 600. In the application phase of the sound source calculation model, the position and orientation of the microphone stored in the microphone information storage unit 651 and the sound collection signal stored in the sound collection signal storage unit 652 are used as production data, and the sound source position and sound source generation time are used. , And the sound source type ID are calculated.

In this modification, the input data to the sound source calculation model is the position of the microphone, the direction of the microphone, and the sound collection signal collected by the microphone, but the data input to the sound source calculation model is not limited to this. .. For example, the microphone type may be included in the input data in view of the fact that the sound collection characteristic changes depending on the type of microphone. Alternatively, considering that the temperature and humidity affect the sound quality, the temperature and humidity may be included in the input data. Further, in view of the fact that the reverberation characteristic of the sound changes depending on the structure of the space to be picked up, the input data may include the place information indicating the place where the sound is picked up. Further, the captured image and the parameters obtained from the captured image may be included in the input data.

Here, a supplementary explanation will be given as to why the sound source position and the sound source generation time can be estimated based on the position of the microphone, the direction of the microphone, and the sound pick-up signal picked up by the microphone. For example, when a shoot occurs in soccer, there is a correlation between the timing of the kick sound included in the sound pick-up signal picked up by the microphone and the position of the microphone and the position where the shoot sound is generated (sound source position). In addition, there is a correlation between the orientation of the microphone having directivity and the pick-up level of the shooting sound. As described above, since there is a correlation between the position of the microphone, the direction of the microphone, and the sound source signal picked up by the microphone, the sound source position can be estimated by machine learning. Furthermore, the reason for using machine learning in addition to mere distance calculation is that, for example, by detecting the excitement of the spectator after shooting, the estimation accuracy of the event that the shooting has occurred is improved. This estimation reduces the possibility of confusing the shoot sound with other sounds (for example, the sound of the drum of a cheering party), and improves the estimation accuracy of the sound source position.

In addition, a supplementary explanation will be given as to why the sound source type can be estimated based on the position of the microphone, the direction of the microphone, and the sound pick-up signal picked up by the microphone. For example, as described above, by detecting the excitement of the spectators, it is possible to estimate whether a shot has been taken or a corner kick has been taken. In this way, since there is a correlation between the sound pick-up signal picked up by the microphone and the sound source type, it is possible to estimate by using machine learning.

The sound source information acquisition unit 622 acquires the camera path received by the camera path reception unit 621. Further, the sound source information acquisition unit 622 acquires the sound collection signal corresponding to the period from the start time to the end time of the camera path from the sound collection signal storage unit 652. Further, the sound source information acquisition unit 622 acquires microphone information from the microphone information storage unit 651. By inputting these acquired data into the sound source calculation model, the sound source position, the sound source generation time, and the sound source type ID can be acquired.

FIG. 7 is a flowchart showing the processing of the model generation unit 600 in the learning phase of this modification. The process shown in FIG. 7 is started at the timing when the sound collection signal and the microphone information are stored in the data storage unit 630, the sound source is detected by the sound source detection unit 111, and then the start of the learning phase is instructed. However, the start timing of the process shown in FIG. 2 is not limited to this.

In S701, the match ID is acquired. As an acquisition method, for example, the operator selects a match to be learned via an external operating PC, and the match ID corresponding to the selected match is sent to the teacher data generation unit 601. Further, for example, a match ID may be automatically determined for a match or a sound source position that has not yet been learned.

In S702, the microphone information corresponding to the match ID acquired in S701 is acquired from the microphone information storage unit 631. In S703, the sound collection signal corresponding to the match ID acquired in S701 is acquired from the sound collection signal storage unit 632. In S704, the teacher data generation unit 601 acquires sound source information including the sound source position, the sound source generation time, and the sound source type ID from the sound source information storage unit 604. The sound source information acquired here is information about a sound source whose match ID in Table 1 matches the match ID acquired in S701.

The processes of S705 to S709 are executed for each sound source information acquired in S704. In S706, the sound pick-up signal in the range of several seconds before and after the acquired sound source generation time is cut out from the sound pick-up signal acquired in S703. The number of seconds to cut out depends on the machine learning algorithm in the sound source calculation model. In particular, when learning is performed including how the cheers rise before and after the sound source generation time, it may be cut out within a range of 10 seconds or more before and after.

In S707, the teacher data generation unit 601 uses the microphone information acquired in S702 and the sound pick-up signal cut out in S706 as input data, and the sound source position, sound source generation time, and sound source type ID acquired in S704 as correct data. Generate a set of teacher data. In S708, the learning unit 602 updates the sound source calculation model using the teacher data generated in S707. A sound source calculation model as a trained model is generated by repeating the update of the sound source calculation model for each sound source information.

FIG. 8 is a flowchart showing the processing of the sound generation unit 620 in the application phase of this modification. In the process shown in FIG. 8, a sound source calculation model as a learned model is generated by the process of the learning phase shown in FIG. 7, and after the microphone information and the sound pickup signal are stored in the data storage unit 650, the application phase is started. It will start at the indicated timing. However, the start timing of the process shown in FIG. 8 is not limited to this.

In S801, the camera path receiving unit 621 receives the camera path generated by the camera path generating unit 643. In S802, the match ID is acquired from the camera pass acquired in S801. In S803, the microphone information corresponding to the match ID acquired in S802 is acquired from the microphone information storage unit 651. In S804, the sound collection signal corresponding to the match ID acquired in S802 and corresponding to the period from the start time to the end time of the camera pass acquired in S801 is acquired from the sound collection signal storage unit 652. In S805, the sound source position calculation unit 660 inputs the microphone information acquired in S803 and the sound collection signal acquired in S804 into the sound source calculation model, and calculates the sound source position, the sound source generation time, and the sound source type ID. At this time, two or more sets of sound source information may be acquired.

Since the processing of S806 to S809 is the same as the processing of S304 to S307 described with reference to FIG. 3, the description thereof will be omitted. In this way, by synthesizing the sound pickup signal based on the sound source position estimated from the microphone information and the sound collection signal to generate an acoustic signal for reproduction, the sound is reproduced together with the virtual viewpoint image corresponding to the sound collection period. It is possible to generate a highly realistic acoustic signal suitable for.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

10 Information processing system 20 Sound generation system 100 Model generation unit 120 Sound generation unit

Claims (18)

An acquisition means for acquiring viewpoint information indicating the transition of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of shot images obtained by shooting a shooting area from a plurality of directions.
An information processing apparatus comprising: an estimation means for estimating the position of a sound source in the photographing region based on the viewpoint information acquired by the acquisition means. The estimation means is
The viewpoint information acquired by the acquisition means is applied to a trained model obtained by machine learning that outputs data related to a sound source in the photographing region according to input data including the viewpoint information. type in,
The information processing apparatus according to claim 1, wherein the position of a sound source in the photographing area is estimated based on the data output from the trained model. The input data includes a photographed image obtained by photographing the photographing area, information indicating the position of a microphone that collects the sound of the photographing area, and a sound obtained by collecting the sound of the photographing area. The information processing apparatus according to claim 2, wherein at least one of the collected sound signals is included. The information processing apparatus according to claim 2 or 3, wherein the machine learning uses at least one of linear regression, logistic regression, a support vector machine, and a neural network as an algorithm. Any of claims 2 to 4, further comprising a re-learning means for re-learning the trained model using the data output from the trained model and the determination result regarding the validity of the data. The information processing apparatus according to item 1. The information processing apparatus according to any one of claims 1 to 5, wherein the viewpoint information represents a transition between a position and an orientation of a virtual viewpoint corresponding to the virtual viewpoint image in a predetermined period. The information processing apparatus according to claim 6, wherein the estimation means estimates the position of a sound source in the photographing region in the predetermined period. The estimation means is characterized in that at least one of the generation time of the sound source in the shooting area and the type of the sound source in the shooting area is estimated based on the viewpoint information acquired by the acquisition means. The information processing apparatus according to any one of claims 1 to 7. A generation means for generating an acoustic signal to be reproduced together with the virtual viewpoint image by processing the sound collection signal obtained by collecting the sound in the photographing region based on the estimation result by the estimation means. The information processing apparatus according to any one of claims 1 to 8, wherein the information processing apparatus has. A learning means for generating the trained model by using the viewpoint information indicating the transition of the virtual viewpoint corresponding to the virtual viewpoint image and the sound source information indicating the position of the sound source as teacher data.
An information processing system comprising the information processing apparatus according to any one of claims 2 to 5. An acquisition means for acquiring microphone information indicating the positions of a plurality of microphones for collecting sound in the sound collection region, and a sound collection signal acquired based on sound collection by the plurality of microphones.
The acquired model is a trained model obtained by machine learning and outputs data related to a sound source in the sound collecting region according to input data including the microphone information and the sound collecting signal by the acquisition means. An estimation means that inputs the acquired microphone information and the sound collection signal and estimates the position of the sound source in the sound collection area based on the data output from the trained model.
An information processing device characterized by having. The information processing apparatus according to claim 11, wherein the input data includes information indicating at least one of the directivity of the microphone, the type of the microphone, the temperature, the humidity, and the sound collecting place. An acquisition process for acquiring viewpoint information indicating the transition of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of shot images obtained by shooting a shooting area from a plurality of directions, and an acquisition process.
An information processing method comprising an estimation step of estimating the position of a sound source in the photographing region based on the viewpoint information acquired in the acquisition step. The estimation process is
The viewpoint information acquired in the acquisition step is applied to a trained model obtained by machine learning, which outputs data related to a sound source in the photographing region according to input data including the viewpoint information. type in,
The information processing method according to claim 13, wherein the position of the sound source in the photographing area is estimated based on the data output from the trained model. A generation step of generating an acoustic signal to be reproduced together with the virtual viewpoint image by processing the sound pick-up signal obtained by picking up the sound in the shooting area based on the estimation result in the estimation step. The information processing method according to claim 13, wherein the information processing method is characterized by having. An acquisition process for acquiring microphone information indicating the positions of a plurality of microphones for collecting sound in the sound collection region and sound collection signals acquired based on sound collection by the plurality of microphones.
In the acquisition process, a trained model obtained by machine learning that outputs data related to a sound source in the sound collection region according to input data including the microphone information and the sound collection signal. An estimation process in which the acquired microphone information and the sound collection signal are input, and the position of the sound source in the sound collection area is estimated based on the data output from the trained model.
An information processing method characterized by having. The information processing method according to claim 16, wherein the input data includes information indicating at least one of the directivity of the microphone, the type of the microphone, the temperature, the humidity, and the sound collecting place. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 9, 11 and 12.

Images