JP2023048531A - Information processing device, information processing method, program, and acoustic automatic synthesis system - Google Patents

Abstract

To efficiently synthesize virtual viewpoint image data and acoustic data acquired in imaging spaces different from each other.SOLUTION: An information processing device initially acquires data on a first attitude model and data on a virtual viewpoint image, generated on the basis of a first multi-viewpoint image obtained from a plurality of imaging devices for imaging a first imaging space. Next, the device acquires data on a second attitude model generated on the basis of a second multi-viewpoint image obtained from the plurality of imaging devices for imaging a second imaging space different from the first imaging space. Moreover, the device acquires data on sound collected when the second multi-viewpoint image is taken in the second imaging space. Furthermore, the device evaluates a coincidence between the first attitude model and the second attitude model, and generates the data including the virtual viewpoint image and the sound, on the basis of the coincidence.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a technique for synthesizing acoustic data with virtual viewpoint image data.

Synchronous multi-viewpoint imaging is performed using imaging devices such as cameras placed at a plurality of different positions, and a virtual viewpoint image is generated based on a plurality of captured images (hereinafter referred to as "multi-viewpoint images") obtained by the imaging. There is technology to generate. According to the virtual viewpoint image, an object existing in the imaging space can be viewed from various angles. In a system for generating virtual viewpoint images, sound data (hereinafter referred to as "acoustic data") collected when multiple viewpoint images are captured is combined with virtual viewpoint image data (hereinafter referred to as "virtual viewpoint image data"). there is something to do In such a system, sound data is combined with virtual viewpoint image data by synchronizing the time of the virtual viewpoint image with the time of the sound collected when capturing the multiple viewpoint images. In the field of game systems, Japanese Patent Laid-Open No. 2003-200000 discloses a technique for outputting a sound corresponding to the motion of a character according to the distance between the joints in the joint model of the character that moves in the virtual space.

JP 2020-195672 A

Virtual viewpoint image data generated based on multi-viewpoint images captured in a first imaging space such as a studio, and sound collected in a second imaging space different from the first imaging space such as a stage. Synthesis with data can be realized by a user's manual synthesis technique. Specifically, a user who provides a virtual viewpoint image (hereinafter, simply referred to as a “user”) manually synchronizes the timing of synchronizing acoustic data and virtual viewpoint image data acquired in advance, so that the acoustic data and the virtual viewpoint image data are synchronized. Synthesize with the viewpoint image. However, in such a method, the user needs to manually synthesize the virtual viewpoint image data with various sound data that the user wishes to synthesize while creating the virtual viewpoint image. Therefore, in such a method, there is a problem that it takes a long time to edit for synthesizing the acoustic data and the virtual viewpoint image data.

The present disclosure is intended to solve such problems, and aims to provide an information processing apparatus that efficiently synthesizes virtual viewpoint image data and sound data acquired in different imaging spaces. .

An information processing device according to the present disclosure is a first model that acquires data of a first posture model generated based on first multi-viewpoint images obtained from a plurality of imaging devices that capture images of a first imaging space. From an acquisition means, an image acquisition means for acquiring data of a virtual viewpoint image generated based on a first multi-viewpoint image, and a plurality of imaging devices for imaging a second imaging space different from the first imaging space. second model acquisition means for acquiring data of a second posture model generated based on the obtained second multi-viewpoint image; and when the second multi-viewpoint image is captured in the second imaging space. sound acquisition means for acquiring data of sound collected by the virtual viewpoint image; evaluation means for evaluating the degree of matching between the first posture model and the second posture model; and data generation means for generating data including

According to the present disclosure, it is possible to efficiently synthesize virtual viewpoint image data and sound data acquired in imaging spaces different from each other.

1 is a block diagram showing an example of the configuration of an information processing system according to Embodiment 1; FIG. 2 is a block diagram showing an example of a hardware configuration of a first information processing device according to Embodiment 1; FIG. FIG. 4 is an explanatory diagram for explaining an example of a standard shape model and a standard posture model according to Embodiment 1; 8 is a flow chart showing an example of the flow of processing in the second information processing apparatus according to the first embodiment; 4 is a flow chart showing an example of the flow of processing in the first information processing apparatus according to the first embodiment; FIG. 5 is an explanatory diagram for explaining an example of matching processing in the matching unit according to the first embodiment; FIG. 4 is an explanatory diagram for explaining an example of a configuration of information indicating correspondence between acoustic data and a posture model by a correspondence unit according to the first embodiment; 10 is a flow chart showing an example of the flow of processing in the second information processing apparatus according to the second embodiment; 10 is a flow chart showing an example of the flow of processing in the first information processing apparatus according to the second embodiment; FIG. 11 is an explanatory diagram for explaining an example of matching processing in a matching unit according to the second embodiment;

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the scope of the present disclosure is not limited only to those configurations.

(Embodiment 1)
[composition]
An information processing system 1 according to the first embodiment will be described with reference to FIGS. FIG. 1 is a block diagram showing an example of the configuration of an information processing system 1 according to the first embodiment. The information processing system 1 includes a plurality of imaging devices 11 , a sound collector 15 , a plurality of imaging devices 16 , a first information processing device 100 , a second information processing device 150 and an output device 19 .

The information processing system 1 is a system for synthesizing virtual viewpoint image data and sound data obtained in different imaging spaces. There are various effects that sound has on a viewer viewing an image. For example, in the field of performing arts, stage sounds such as applause or footsteps caused by movements of performers have a great impact on audiences. Also, in the field of indoor sports such as sports, the sound in spaces such as gymnasiums is essential for giving viewers a sense of presence in the games. These sounds are affected by reverberations in an imaging space such as a stage or a gymnasium, or by structures such as walls and floors forming the imaging space. Therefore, the echo or structure in an imaging space such as a studio in which a system for generating a virtual viewpoint image is installed is different from the echo or structure in a stage or a gymnasium. Therefore, sound collection in an imaging space such as a studio cannot acquire sound data necessary to give a viewer a sense of performance or realism. The information processing system 1 in this embodiment is a system for solving this problem.

Each of the plurality of imaging devices 11 is configured by a digital video camera, a digital still camera, or the like, and is installed around a first imaging space (hereinafter referred to as "first imaging space") such as a studio. Each of the plurality of imaging devices 11 captures an image of the first imaging space and outputs captured image data obtained by the imaging (hereinafter referred to as “captured image data”) to the first information processing device 100 . Each of the plurality of imaging devices 16 is composed of a digital video camera, a digital still camera, or the like, and is installed around a second imaging space (hereinafter referred to as "second imaging space") such as a stage or a gymnasium. Each of the plurality of imaging devices 16 captures an image of the second imaging space and outputs captured image data obtained by the imaging to the second information processing device 150 .

The sound collector 15 is composed of a microphone or the like, and collects sound in the second imaging space, specifically, the sound generated when an object existing in the second imaging space moves. The generated sound is converted into an acoustic signal and output to the second information processing device 150 . Hereinafter, captured image data output from each of the plurality of imaging devices 11 is collectively referred to as multi-viewpoint image data. Similarly, the captured image data output from each of the plurality of imaging devices 16 is collectively referred to as multi-viewpoint image data (hereinafter referred to as “multi-viewpoint image data”).

The second information processing device 150 acquires the acoustic signal output by the sound collector 15 and the multi-viewpoint image data output by the plurality of imaging devices 16 . The second information processing device 150 outputs the sound indicated by the acquired sound signal to the first information processing device 100 as sound data. In addition, the second information processing apparatus 150 generates posture model data corresponding to an object appearing in the captured image based on a plurality of pieces of captured image data forming the acquired multi-viewpoint image data, and generates posture model data. is output to the first information processing device 100 . The second information processing device 150 includes an audio acquisition unit 151 , a second image group acquisition unit 152 , a second foreground acquisition unit 153 , a second model generation unit 154 , a model output unit 155 and an audio output unit 156 . Details of each unit included in the second information processing apparatus 150 will be described later.

Here, the posture model data is data representing the positions of the joints forming the object, the connection relationship between the joints, the distance between the joints, the angles of the joints, and the like. Hereinafter, it is assumed that the acoustic data includes information indicating the time when the second information processing device 150 acquired the audio signal. It is also assumed that the imaging devices 16 are synchronized in time, and the second information processing device 150 and the imaging devices 16 are synchronized in time. In addition, it is assumed that the captured image data output by each imaging device 16 includes information indicating the capturing time of the captured image (hereinafter referred to as “capturing time information”). Note that the method for synchronizing the time between devices is well known, so the description thereof will be omitted.

The first information processing device 100 acquires the acoustic data and posture model data output by the second information processing device 150 and the multi-viewpoint image data output by the plurality of imaging devices 16 . The first information processing apparatus 100 converts data of a virtual viewpoint image (hereinafter referred to as “virtual viewpoint image data”) and an object appearing in the captured image based on a plurality of pieces of captured image data forming the acquired multiple viewpoint image data. and corresponding pose model data. Hereinafter, the posture model data generated by the first information processing device 100 will be referred to as first posture model data, and the posture model data generated by the second information processing device 150 will be referred to as second posture model data. The first information processing apparatus 100 evaluates the degree of matching between the first posture model and the second posture model, and generates data including the generated virtual viewpoint image and the acquired sound based on the degree of matching.

Specifically, the first information processing apparatus 100 synthesizes the generated virtual viewpoint image data with the acquired acoustic data to generate virtual viewpoint image data with acoustic data. generating data including the acquired sound; Further, the first information processing device 100 outputs the synthesized virtual viewpoint image data with the acoustic data to the output device 19 . The first information processing apparatus 100 includes a first image group acquisition unit 101, a first foreground acquisition unit 102, an image generation unit 103, a first model generation unit 104, a model acquisition unit 105, a sound acquisition unit 106, an association unit 107, An evaluation unit 108 and a data generation unit 109 are provided. Details of each unit included in the first information processing apparatus 100 will be described later. Hereinafter, it is assumed that the imaging devices 11 are synchronized with each other in time. In addition, it is assumed that the captured image data output by each imaging device 11 includes information indicating the capturing time of the captured image (capturing time information). Note that the method for synchronizing the time between devices is well known, so the description thereof will be omitted.

The output device 19 has a display output unit such as an LCD and an audio output unit such as a speaker, and renders the virtual viewpoint image data output by the first information processing device 100 to produce a virtual viewpoint It outputs images and sounds so that they can be viewed. The destination to which the first information processing device 100 outputs the synthesized virtual viewpoint image data is not limited to the output device 19. The first information processing device 100 outputs the synthesized virtual viewpoint image data to the image shown in FIG. may be output to a storage device (not shown). In this case, the first information processing apparatus 100 writes the synthesized virtual viewpoint image data in the storage device, and causes the storage device to store the synthesized virtual viewpoint image data.

A hardware configuration of the first information processing apparatus 100 will be described. The processing of each unit included in the first information processing apparatus 100 is performed by hardware such as an ASIC (Application Specific Integrated Circuit) incorporated in the first information processing apparatus 100 . The processing may be performed by hardware such as FPGA (Field Programmable Gate Array). Further, the processing may be performed by software using a CPU (Central Processor Unit) or GPU (Graphic Processor Unit) and memory.

With reference to FIG. 2, the hardware configuration of the first information processing apparatus 100 when each unit included in the first information processing apparatus 100 operates as software will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the first information processing device 100 according to the first embodiment. The first information processing apparatus 100 is composed of a computer, and as shown in FIG. and bus 208 .

The CPU 201 controls the computer using programs or data stored in the ROM 202 or the RAM 203, thereby causing the computer to function as each unit included in the first information processing apparatus 100 shown in FIG. Note that the first information processing apparatus 100 may have one or a plurality of dedicated hardware different from the CPU 201 , and the dedicated hardware may execute at least part of the processing by the CPU 201 . Examples of dedicated hardware include ASICs, FPGAs, and DSPs (digital signal processors). The ROM 202 stores programs and the like that do not require modification. The RAM 203 temporarily stores programs or data supplied from the auxiliary storage device 204, data supplied from the outside via the communication unit 207, or the like. The auxiliary storage device 204 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The display unit 205 is configured by, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate or view the first information processing apparatus 100 . The operation unit 206 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 201 in response to user's operations. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206 .

The communication unit 207 is used for communication with devices external to the first information processing device 100 . For example, when the first information processing device 100 is wired to an external device, a communication cable is connected to the communication unit 207 . If the first information processing device 100 has a function of wirelessly communicating with an external device, the communication unit 207 has an antenna. A bus 208 connects each unit provided in the first information processing apparatus 100 and transmits information. In the first embodiment, the display unit 205 and the operation unit 206 are described as existing inside the first information processing apparatus 100 . It may exist outside as another device.

A hardware configuration of the second information processing device 150 will be described. As with the first information processing apparatus 100 , the processing of each unit included in the second information processing apparatus 150 is performed by hardware such as ASIC and FPGA built into the second information processing apparatus 150 . Also, the processing may be performed by software using a CPU or GPU and memory. When each part provided in the second information processing device 150 operates as software, the second information processing device 150 is configured by the computer shown in FIG. may function as

[Processing of each part]
Processing of each unit included in the second information processing device 150 will be described. The sound acquisition unit 151 acquires the sound signal output by the sound collector 15, digitizes the sound signal by AD conversion, and generates sound data. When generating the audio data, the audio acquisition unit 151 generates the audio data such that, for example, information indicating the time when the audio signal was acquired is included in the audio data. The voice acquisition unit 151 causes the auxiliary storage device 204 included in the second information processing device 150 to store the generated acoustic data, and causes the auxiliary storage device 204 to hold the acoustic data. The sound output unit 156 outputs the sound data generated by the sound acquisition unit 151 to the first information processing device 100 . Specifically, the audio output unit 156 reads the audio data held in the auxiliary storage device 204 from the auxiliary storage device 204 and outputs the read audio data to the first information processing device 100 .

The second image group acquisition unit 152 acquires captured image data output from each of the plurality of imaging devices 16 . That is, the second image group acquisition unit 152 acquires multi-viewpoint image data from the plurality of imaging devices 16 . The second foreground acquisition unit 153 extracts an image region corresponding to an object appearing in the captured image for each of a plurality of captured images forming the multi-viewpoint image acquired by the second image group acquisition unit 152, and extracts the extracted image region. Take as foreground region. Furthermore, the second foreground acquisition unit 153 generates a foreground image representing the acquired foreground area. Here, the object corresponding to the image area extracted as the foreground area is generally a dynamic object that moves or whose position can change when the image is captured from the same direction in time series. (hereinafter referred to as "moving object"). A moving object is, for example, a natural person such as a player or a referee in the field where the game is played, or a ball used in a ball game. Also, in events such as concerts and entertainment, singers, musicians, performers, moderators, and the like are moving objects. Since the method of generating the foreground image from the captured image is well known, the description thereof is omitted.

The second model generation unit 154 generates a posture model (second posture model) corresponding to the object captured in the captured image based on the foreground regions corresponding to each of the plurality of captured images acquired by the second foreground acquisition unit 153. Generate data. A method by which the second model generation unit 154 generates the data of the second posture model will be described. The second model generating unit 154 first generates data of a standard three-dimensional shape model imitating a standard human shape (hereinafter referred to as “standard shape model”) and a standard posture corresponding to the standard shape model. Acquire the data of the model (hereinafter referred to as "standard posture model"). Data of the standard shape model and the standard posture model are stored in advance in the auxiliary storage device 204 included in the second information processing device 150, for example, and the second model generation unit 154 reads these data from the auxiliary storage device 204. Obtained by

A standard shape model 301 and a standard posture model 302 will be described with reference to FIG. FIG. 3 is an explanatory diagram for explaining an example of the standard shape model 301 and the standard posture model 302 according to the first embodiment. The standard shape model 301 is a model represented by a three-dimensional mesh, and the data of the standard shape model 301 includes coordinates indicating the position of each vertex and IDs (identifiers) of vertices forming surfaces such as triangles or quadrilaterals. ) and other identification information. The standard shape model 301 may be represented by a set of points called voxels.

The data of the standard posture model 302 includes information (hereinafter referred to as “joint information 303”) indicating positions in the standard posture model 302 corresponding to joint parts of the human body such as the head, shoulders, elbows, wrists, or limbs. is In addition to the joint information 303, the data of the standard posture model 302 includes information indicating the connection relationship between the joint parts in the standard posture model 302 (hereinafter referred to as "connection information 304"). The connection information 304 is, for example, information indicating distances between joint parts in the standard posture model 302 . In addition to the joint information 303 and the connection information 304, the data of the standard posture model 302 also includes information indicating the angles of the joints in the standard posture model 302 (hereinafter referred to as "angle information"). The angle information is, for example, line segments connecting adjacent joint parts in the standard posture model 302, and is information indicating the angle formed by the adjacent line segments.

The second model generation unit 154 transforms the standard shape model 301 corresponding to the standard posture model 302 shown as an example in FIG. 3 so as to match the foreground region acquired by the second foreground acquisition unit 153 . The second model generation unit 154 estimates the standard posture model 302 that is most consistent as a result of deformation as a posture model (second posture model) corresponding to the object, and generates data of the second posture model. . However, the method of generating the second posture model is not limited to the method described above. For example, the second model generation unit 154 estimates a two-dimensional posture of an object on a two-dimensional image, and estimates a three-dimensional posture model based on the position, imaging direction, angle of view, etc. of each imaging device 16. may be used to generate the data of the second posture model.

The second model generating unit 154 causes the auxiliary storage device 204 included in the second information processing device 150 to store the data of the generated second posture model, and causes the auxiliary storage device 204 to hold the data. When storing the data of the second posture model in the auxiliary storage device 204, the second model generating unit 154 stores information (imaging time information) indicating the capturing time of the captured image used when generating the second posture model. , is stored in the auxiliary storage device 204 in association with the data of the second posture model. The model output unit 155 outputs the data of the second posture model generated by the second model generation unit 154 and the imaging time information associated with the data of the second posture model to the first information processing apparatus 100 . Specifically, the model output unit 155 reads the data of the second posture model and the imaging time information from the auxiliary storage device 204, and sends the read data of the second posture model and the imaging time information to the first information processing device 100. Output.

Processing of each unit included in the first information processing apparatus 100 will be described. The model acquisition unit 105 acquires the data of the second posture model output by the model output unit 155 of the second information processing device 150 and the imaging time information associated with the data of the second posture model. The sound acquisition unit 106 acquires sound data output by the sound output unit 156 of the second information processing device 150 . The associating unit 107 associates the data of the second posture model with the acoustic data. Details of the association performed by the associating unit 107 will be described later with reference to FIGS. 5 and 6. FIG.

The first image group acquisition unit 101 acquires captured image data output from each of the plurality of imaging devices 11 . That is, the first image group acquisition unit 101 acquires multi-viewpoint image data from a plurality of imaging devices 11 . A first foreground acquisition unit 102 extracts an image region corresponding to an object appearing in each of a plurality of captured images constituting the multi-viewpoint image acquired by the first image group acquisition unit 101, and extracts the extracted image region. Take as foreground region. Furthermore, the first foreground acquisition unit 102 generates a foreground image representing the acquired foreground area. Here, the object refers to a moving object, and the moving object is a natural person or the like existing in the first imaging space such as a studio. The first model generation unit 104 generates a posture model (first posture model) corresponding to the object captured in the captured image based on the foreground regions corresponding to each of the plurality of captured images acquired by the first foreground acquisition unit 102. Generate data. Specifically, for example, the first model generation unit 104 generates the data of the first posture model using the same method as the method of generating the data of the second posture model by the second model generation unit 154 described above. .

Based on the data of the first posture model generated by the first model generation unit 104 and the data of the second posture model acquired by the model acquisition unit 105, the evaluation unit 108 compares the first posture model and the second posture model. Evaluate the degree of agreement between Specifically, for example, the evaluation unit 108 evaluates the degree of matching by comparing the data of the first posture model and the data of the second posture model with the data of the second posture model as correct data. More specifically, the evaluation unit 108 compares at least one of joint information 303, connection information 304, and angle information included in the data of the first posture model and the data of the second posture model. Evaluate the degree of agreement. For example, when at least one of the difference value between the joint information 303, the difference value between the connection information 304, and the difference value between the angle information is equal to or less than a predetermined threshold, the evaluation unit 108 determines that the first posture It is determined that the model and the second posture model match.

The evaluation unit 108 may combine two or more of the joint information 303, the connection information 304, and the angle information to evaluate the degree of matching between the first posture model and the second posture model. . In addition, the evaluation unit 108 may compare the joint information 303 with information indicating the positions of predetermined joint parts such as the joints of the tips of the head and limbs in the standard posture model 302 . Similarly, the evaluation unit 108 determines connection information between predetermined joint parts in the standard posture model 302 out of the connection information 304, or Angles may be compared. The method of evaluating the degree of matching between the first posture model and the second posture model by the evaluation unit 108 is not limited to the one described above.

The image generation unit 103 generates a virtual Generate a viewpoint image. Specifically, for example, the image generation unit 103 generates a virtual viewpoint image by the following method. First, the image generation unit 103 generates data of a three-dimensional shape (hereinafter referred to as a “foreground model”) corresponding to an object by a visual hull method using a foreground image representing a foreground region in each captured image. Generate. Since the method of generating three-dimensional shape data by Visual Hull is well known, the explanation is omitted.

Next, the image generation unit 103 colors the foreground model by performing texture mapping on the foreground model using at least one of the plurality of captured images forming the multi-viewpoint image. The image generation unit 103 also colors the background model by performing texture mapping on the background model, which is the background of the foreground model, using a background image prepared in advance. Here, for example, the background image is a captured image obtained by capturing an image of a stadium, a stage, or the like. Finally, the image generation unit 103 generates a virtual viewpoint image by performing rendering according to the position of a viewpoint (hereinafter referred to as “virtual viewpoint”) in a three-dimensional virtual space specified by a user or the like. The method of generating the virtual viewpoint image in the image generation unit 103 is not limited to the method described above. may generate a virtual viewpoint image.

The data generation unit 109 generates data including the virtual viewpoint image generated by the image generation unit 103 and the sound indicated by the sound data acquired by the sound acquisition unit 106, based on the degree of matching, which is the evaluation result of the evaluation unit 108. do. Specifically, the data generation unit 109 combines the sound data acquired by the sound acquisition unit 106 with the virtual viewpoint image data generated by the image generation unit 103 to generate virtual viewpoint image data with sound data. That is, the data including the virtual viewpoint image and the sound generated by the data generation unit 109 is virtual viewpoint image data with sound data. More specifically, first, the data generation unit 109 acquires the time corresponding to the second posture model determined by the evaluation unit 108 to match the first posture model and the second posture model. Here, the time corresponding to the second orientation model is the imaging time of the captured image that acquired the foreground region used when generating the second orientation model.

Next, the data generation unit 109 causes the evaluation unit 108 to determine whether the first posture model and the second posture model match among the acoustic data associated with the data of the second posture model by the association unit 107. Acquisition of the acoustic data at the time corresponding to the second posture model. Finally, the data generation unit 109 adds the data of the acquired time at the virtual viewpoint image corresponding to the first posture model determined by the evaluation unit 108 to match the first posture model and the second posture model. Acoustic data is synthesized to generate virtual viewpoint image data with acoustic data. Here, the virtual viewpoint image corresponding to the first orientation model is a virtual viewpoint image generated using the same foreground area as the foreground area used when generating the first orientation model. The data generation unit 109 outputs the generated virtual viewpoint image data with acoustic data to the output device. Note that the data generating unit 109 uses the sound acquisition unit 106 to obtain the data of the virtual viewpoint image corresponding to the first posture model determined by the evaluation unit 108 that the first posture model and the second posture model do not match. The acquired acoustic data is output as it is to the output device without being synthesized.

[Operation flow]
The operation of the second information processing device 150 will be described with reference to FIG. FIG. 4 is a flow chart showing an example of the flow of processing in the second information processing device 150 according to the first embodiment. In addition, in the description of FIG. 4, the symbol "S" means a step. First, in S401, the sound acquisition unit 151 acquires a sound signal output by the sound collector 15 and generates acoustic data. Also, the second image group acquisition unit 152 acquires captured image data output from each of the plurality of imaging devices 16, that is, multi-viewpoint image data. Next, in S402, the second foreground acquisition unit 153 acquires a foreground region corresponding to an object appearing in the captured image for each of a plurality of captured images forming the multi-viewpoint image acquired in S401, and obtains the foreground region. generates a foreground image showing Next, in S403, the second model generation unit 154 estimates the second posture model and generates data of the second posture model.

Next, in S<b>404 , the model output unit 155 outputs the data of the second posture model and the imaging time information associated with the data of the second posture model to the first information processing apparatus 100 . Specifically, for example, when receiving an output instruction from the first information processing apparatus 100, the model output unit 155 outputs the data of the second posture model and the imaging time information to the first information processing apparatus 100. . Also, the sound output unit 156 outputs sound data to the first information processing device 100 . Specifically, for example, the sound output unit 156 outputs sound data to the first information processing device 100 when receiving an output instruction from the first information processing device 100 . After S404, the second information processing apparatus 150 ends the processing of the flowchart shown in FIG.

The operation of the first information processing apparatus 100 will be described with reference to FIG. FIG. 5 is a flow chart showing an example of the flow of processing in the first information processing apparatus 100 according to the first embodiment. In addition, in the description of FIG. 5, the symbol "S" means a step. First, in S<b>501 , the model acquisition unit 105 acquires the data of the second posture model and the imaging time information associated with the data of the second posture model from the second information processing apparatus 150 . Also, the sound acquisition unit 106 acquires sound data from the second information processing device 150 .

Next, in S502, the associating unit 107 analyzes the acoustic data acquired in S501. Specifically, for example, the associating unit 107 analyzes the magnitude of the sound volume indicated by the sound data, that is, the magnitude of the amplitude of the audio signal corresponding to the sound, Search for a value point. The volume analysis in the association unit 107 may be, for example, analysis of the volume of sound corresponding to a predetermined frequency such as 48 kilohertz (kHz) among the frequencies of the sound. The associating unit 107 also cuts out the acoustic data based on the analysis result of S502. Extraction of the acoustic data based on the analysis result may be, for example, extraction of a portion corresponding to the landing sound when the natural person as the object jumps on the stage from the acoustic data acquired in S501.

In this case, sounds other than the landing sound produced from the floor of the stage upon landing are unnecessary sounds. Therefore, by using a method such as deleting a period before and after the point in time when the sound volume reaches a maximum value in the sound data when the sound volume is equal to or less than a predetermined threshold value, the sound other than the period desired to be combined with the virtual viewpoint image data is extracted. Delete the data and cut it out as sound data for synthesis. The method of cutting out sound data for synthesis is not limited to the one described above.

Next, in S503, the associating unit 107 associates the acoustic data for synthesis with the data of the second posture model. Specifically, first, based on the analysis result in step S502, the associating unit 107 converts the captured image captured at the same time as the time corresponding to the point in time when the sound volume of the sound data for synthesis reaches the maximum value. The data of the second posture model generated based on this is specified. In the following description, posture model data generated based on an image captured at a certain time is referred to as frame data of a posture model frame. That is, based on the result of the analysis in step S502, the associating unit 107 generates a captured image captured at the same time as the time corresponding to the point in time when the sound volume of the sound data for synthesis reaches the maximum value. Identify the frame data of the second posture model frame. Next, the associating unit 107 associates the time corresponding to the point in time when the sound volume of the sound data for synthesis reaches the maximum value, and the frame data of the specified second posture model frame. In this manner, the associating unit 107 associates the sound data for synthesis with the data of the second posture model.

The association processing in the association unit 107 will be described with reference to FIG. FIG. 6 is an explanatory diagram for explaining an example of matching processing in the matching unit 107 according to the first embodiment. In FIG. 6, the upper portion shows second posture model frames 601a to 601e arranged in time series, and the lower portion shows acoustic data 602a to 602c represented by time-series audio signals. In addition, the lower part of FIG. 6 shows, as an example, audio data corresponding to 48 kHz among the audio data in the form of an audio signal. Here, the sound data 602b indicates the sound data for synthesis extracted by the extraction processing in S502. Also, acoustic data 602a and 602c indicate audio data deleted by the extraction processing in S502 from the audio data acquired in S501. In FIG. 6, the horizontal axis is the time axis, and the vertical axis in the lower part of FIG. 6 indicates the amplitude of the audio signal.

The acoustic data 602b is associated with one of the second posture model frames 601a to 601e as synthesizing acoustic data for synthesizing with the virtual viewpoint image. The analysis in S502 identifies the point in time when the volume of the acoustic data 602b reaches its maximum, that is, the point in time when the amplitude of the acoustic data 602b reaches its maximum. After the identification, the second posture model frame generated based on the captured image captured at the same time as the time is identified. In the example shown in FIG. 6, the time corresponding to the second posture model frame 601d coincides with the time corresponding to the time when the acoustic data 602b reaches its maximum amplitude. are associated by the associating unit 107 .

However, even if the second posture model frame and the acoustic data are time-synchronized with each other, the frame rate of the second posture model frame and the sampling rate of the acoustic data may differ from each other. In such a case, there may not be a second posture model frame at the same time as the time when the acoustic data reaches the maximum amplitude. Therefore, in such a case, the second posture model frame at the time closest to the time point at which the acoustic data reaches its maximum amplitude is identified, and the time point corresponding to the time point at which the acoustic data reaches its maximum amplitude is specified; All that is necessary is to associate it with the specified second posture model frame. The method of associating the acoustic data with the second posture model is not limited to the one described above, and any method that can synchronize the acoustic data with the second posture model may be used.

The configuration of information indicating the correspondence between the acoustic data and the posture model by the associating unit 107 will be described with reference to FIG. FIG. 7 is an explanatory diagram for explaining an example of a configuration of information indicating correspondence between acoustic data and a posture model by the association unit 107 according to the first embodiment. As shown in FIG. 7, for example, according to the number of patterns of acoustic data to be combined with a virtual viewpoint image, pattern No. is allocated. Here, it is assumed that the pattern is set with, for example, acoustic data to be combined with the virtual viewpoint image for each piece of performing arts or for each shooting scene. The number of sounds shown in FIG. The number of pieces of sound data to be combined with the virtual viewpoint image, that is, the number of pieces of sound data for synthesis cut out in S502 is input. Acoustic data for synthesis is stored in each of the acoustic information shown in FIG. 7, and the posture estimation model data shown in FIG. Stores the frame data of the model frame.

After S503, in S511, the first image group acquisition unit 101 acquires captured image data output from each of the plurality of imaging devices 11, that is, multi-viewpoint image data. Next, in S512, the first foreground acquisition unit 102 acquires a foreground region corresponding to an object appearing in the captured image for each of a plurality of captured images forming the multi-viewpoint image acquired in S511, and obtains the foreground region. generates a foreground image showing Next, in S513, the image generation unit 103 generates a virtual viewpoint image. Next, in S514, the first model generation unit 104 estimates the first posture model and generates data of the first posture model.

Next, in S515, the evaluation unit 108 evaluates the degree of matching between the first posture model and the second posture model, and determines whether or not the first posture model and the second posture model match. . If it is determined in S515 that they match, in S516 the data generation unit 109 adds sound for synthesis associated with the data of the second posture model in S503 to the virtual viewpoint image data generated in S513. Synthesize data. After that, the data generation unit 109 outputs the synthesized virtual viewpoint image data to the output device. If it is determined in S515 that the first orientation model and the second orientation model do not match, the data generation unit 109 outputs the virtual viewpoint image data generated in S513 as it is to the output device.

The first information processing apparatus 100 determines whether or not the termination condition is satisfied in S520. Here, the termination condition is, for example, a case where an operation signal for termination instruction from the user is received. The first information processing apparatus 100 repeatedly executes the processing from S511 to S516 until the termination condition is satisfied, and terminates the processing of the flowchart shown in FIG. 5 when the termination condition is satisfied.

As described above, according to the first information processing apparatus 100, it is possible to efficiently synthesize the virtual viewpoint image data and the sound data acquired in different imaging spaces. As a result, sound that cannot be reproduced in a studio or the like can be efficiently synthesized with the virtual viewpoint image, so that the workload for synthesizing the acoustic data and the virtual viewpoint image data can be reduced.

In the first embodiment, the first information processing apparatus 100 is described as including the first image group acquisition unit 101, the first foreground acquisition unit 102, the image generation unit 103, and the first model generation unit 104. It is not limited to this. For example, the first information processing apparatus 100 has a first posture model acquisition unit shown in FIG. If so, the first information processing device 100 does not need to have the first model generation unit 104 . Further, for example, when the first information processing apparatus 100 has the image acquisition unit illustrated in FIG. 1 Information processing apparatus 100 does not need to have image generation unit 103 . Further, the first information processing device 100 may have each unit included in the second information processing device 150 . That is, the first information processing device 100 may have the functions that the second information processing device 150 has. If the first information processing device 100 has all the configurations of the second information processing device 150 , the information processing system 1 may not have the second information processing device 150 .

(Embodiment 2)
An information processing system 1 according to the second embodiment will be described with reference to FIGS. The configuration of the information processing system 1 according to the second embodiment is the same as the configuration of the information processing system 1 according to the first embodiment shown in FIG. 1 as an example. That is, the information processing system 1 includes a plurality of imaging devices 11 , a sound collector 15 , a plurality of imaging devices 16 , a first information processing device 100 , a second information processing device 150 , and an output device 19 .

The information processing system 1 according to the first embodiment was as follows. First, in the second information processing device 150, based on captured image data and audio signals obtained by capturing and collecting sound in a second capturing space such as a stage, data of a second posture model and acoustic data are generated in advance. is generated. Next, in the first information processing apparatus 100, the data of the first posture model and the virtual viewpoint image data are generated based on the imaged image data obtained by imaging in the first imaging space such as a studio. Further, the first information processing device 100 evaluates the degree of matching between the data of the first posture model and the data of the second posture model, and based on the degree of matching, the virtual viewpoint image data and the acoustic data are generated. are synthesized, and the synthesized virtual viewpoint image data is output.

On the other hand, an information processing system 1 (hereinafter simply referred to as "information processing system 1") according to the second embodiment is as follows. First, in the second information processing device 150, based on captured image data and audio signals obtained by capturing and collecting sound in a second capturing space such as a stage, data of a second posture model and acoustic data are generated in advance. is generated. Also, physical information is generated in advance by analyzing the data of the second posture model. Physical information will be described later. Next, in the first information processing apparatus 100, the data of the first posture model and the virtual viewpoint image data are generated based on the imaged image data obtained by imaging in the first imaging space such as a studio. Also, physical information is generated by analyzing the data of the first posture model. Furthermore, the first information processing device 100 evaluates the degree of matching between the data of the first posture model and the physical information corresponding to the data and the data of the second posture model and the physical information corresponding to the data. . Finally, the first information processing device 100 synthesizes the virtual viewpoint image data and the sound data based on the degree of matching, and outputs the synthesized virtual viewpoint image data.

[composition]
The configuration of the functional blocks of the first information processing apparatus 100 according to the second embodiment (hereinafter simply referred to as the "first information processing apparatus 100") is the first information processing apparatus 100 according to the first embodiment shown as an example in FIG. Since it is the same as the functional block of , the explanation is omitted. That is, the first information processing device 100 includes: That is, the first information processing apparatus 100 includes a first image group acquisition unit 101, a first foreground acquisition unit 102, an image generation unit 103, a first model generation unit 104, a model acquisition unit 105, a sound acquisition unit 106, an association unit 107 , an evaluation unit 108 and a data generation unit 109 . Further, the configuration of the functional blocks of the second information processing apparatus 150 according to the second embodiment (hereinafter simply referred to as the "second information processing apparatus 150") is similar to that of the second information processing apparatus according to the first embodiment shown as an example in FIG. Since the functional blocks are the same as those of the device 150, description thereof is omitted. That is, the second information processing device 150 includes an audio acquisition unit 151 , a second image group acquisition unit 152 , a second foreground acquisition unit 153 , a second model generation unit 154 , a model output unit 155 and an audio output unit 156 . Also, the hardware configurations of the first information processing apparatus 100 and the second information processing apparatus 150 are the same as those of the first information processing apparatus 100 and the second information processing apparatus 150 according to the first embodiment, so the description thereof will be omitted.

[Processing of each part]
Differences between the information processing system 1 and the information processing system 1 according to the first embodiment will be described below. First, processing of each unit included in the second information processing device 150 will be described. The audio acquisition unit 151, the second image group acquisition unit 152, the second foreground acquisition unit 153, and the sound output unit 156 are the same as the audio acquisition unit 151, the second image group acquisition unit 152, and the second foreground acquisition unit according to the first embodiment. 153 and the sound output unit 156, the description is omitted.

In addition to the function of generating the data of the second posture model, the second model generation unit 154 analyzes the generated data of the second posture model, and performs physical information (hereinafter referred to as “second posture model data”) corresponding to the data of the second posture model. 2 physical information”). Here, the physical information is information indicating the velocity or acceleration of the joint part corresponding to the object in the second posture model frame. Specifically, the second model generation unit 154 generates the second physical information by calculating the velocity or acceleration of the joint part based on the frame data of the plurality of second posture model frames. The second model generation unit 154 generates the generated second physical information in addition to the data of the second orientation model and the information indicating the imaging time of the captured image used when generating the second orientation model (imaging time information). is stored in the auxiliary storage device 204 included in the second information processing device 150 . Specifically, the second model generation unit 154 stores the second physical information in the auxiliary storage device 204 included in the second information processing device 150 in association with the frame data of the second posture model frame.

The model output unit 155 outputs the second physical information generated by the second model generation unit 154 to the first information processing, in addition to the data of the second posture model and the imaging time information associated with the data of the second posture model. Output to device 100 . Specifically, the model output unit 155 reads the data of the second posture model, the imaging time information, and the second physical information from the auxiliary storage device 204, and reads the read data of the second posture model, the imaging time information, and the second physical information. 2 Physical information is output to the first information processing device 100 .

Next, processing of each unit included in the first information processing apparatus 100 will be described. The first image group acquisition unit 101, the first foreground acquisition unit 102, the image generation unit 103, the sound acquisition unit 106, and the data generation unit 109 are the same as the corresponding units according to the first embodiment, and descriptions thereof are omitted. . In addition to the function of generating the data of the first posture model, the first model generation unit 104 analyzes the generated data of the first posture model and performs physical information (hereinafter referred to as “first posture model data”) corresponding to the data of the first posture model. 1 physical information”). Specifically, the first model generation unit 104 generates the first physical information by calculating the velocity or acceleration of the joint part based on the frame data of the plurality of first posture model frames. The generated first physical information is associated with the frame data of the first posture model frame.

In addition to the data of the second orientation model output by the model output unit 155 of the second information processing device 150 and the imaging time information associated with the data of the second orientation model, the model acquisition unit 105 obtains a second physical model. Get information. The acquired second physical information is associated with the frame data of the second posture model frame. The associating unit 107 has a function of extracting acoustic data for synthesis from acoustic data, and a second physical model corresponding to the acoustic data for synthesis, the frame data of the second posture model frame, and the frame data of the second posture model frame. It has the function of associating information with each other. Among the functions of the associating unit 107, the function of extracting sound data for synthesis from sound data has been described in the first embodiment, and therefore the description thereof will be omitted. Further, since the method of associating the acoustic data for synthesis with the frame data of the second posture model frame has been described in the first embodiment, the description thereof will be omitted. Further, since the second physical information is associated with the frame data of the second posture model frame, description thereof will be omitted.

The evaluation unit 108 has a function of evaluating the degree of matching between the first posture model and the second posture model based on the data of the first posture model and the data of the second posture model. Specifically, the evaluation unit 108 compares at least one of the joint information 303, the connection information 304, and the angle information included in the data of the first posture model and the data of the second posture model. It has a function to evaluate the degree of matching. In addition to this function, the evaluation unit 108 has a function of evaluating the degree of matching between the first posture model and the second posture model based on the first physical information and the second physical information. Specifically, the evaluation unit 108 calculates information indicating the velocity or acceleration of the joint part corresponding to the object in the first posture model and the velocity of the joint part corresponding to the object in the second posture model. Or compare with information indicating acceleration. For example, when at least one of the difference value between velocities and the difference value between accelerations is equal to or less than a predetermined threshold value, it is determined that the first posture model and the second posture model match.

More specifically, for example, the evaluation unit 108 first evaluates the degree of matching between the first posture model and the second posture model based on the data of the first posture model and the data of the second posture model. do. Next, when it is determined that the first posture model and the second posture model match based on the data of the first posture model and the data of the second posture model, the evaluation unit 108 evaluates the first physical information and the second posture model. The degree of matching between the first pose model and the second pose model is evaluated based on the two physical information. Such stepwise evaluation can improve the accuracy of evaluating the degree of matching between the first posture model and the second posture model. Note that when it is determined that the first posture model and the second posture model match based on the data of the first posture model and the data of the second posture model, the evaluation unit 108 generates the first model generation unit 104 and the The model acquisition unit 105 may be instructed to generate physical information.

[Operation flow]
The operation of the second information processing device 150 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of processing in the second information processing device 150 according to the second embodiment. In addition, in the description of FIG. 8, the symbol "S" means a step. Further, in FIG. 8, descriptions of the components denoted by the same reference numerals as those in FIG. 4 are omitted. First, the second information processing device 150 executes the processes from S401 to S403.

After S403, in S804, the second model generating unit 154 generates second physical information. Next, in S<b>805 , the model output unit 155 outputs the data of the second posture model, the imaging time information associated with the data of the second posture model, and the second physical information to the first information processing apparatus 100 . do. Specifically, for example, when receiving an output instruction from the first information processing apparatus 100, the model output unit 155 outputs the data of the second posture model, the imaging time information, and the second physical information to the first information processing apparatus. Output to device 100 . Also, the sound output unit 156 outputs sound data to the first information processing device 100 . Specifically, for example, the sound output unit 156 outputs sound data to the first information processing device 100 when receiving an output instruction from the first information processing device 100 . After S805, the second information processing apparatus 150 ends the processing of the flowchart shown in FIG.

The operation of the first information processing apparatus 100 will be described with reference to FIG. FIG. 9 is a flowchart showing an example of the flow of processing in the first information processing apparatus 100 according to the second embodiment. In addition, in the description of FIG. 9, the symbol "S" means a step. Further, in FIG. 9, the description of the components denoted by the same reference numerals as in FIG. 5 will be omitted. First, the first information processing apparatus 100 executes the processes of S501 and S502.

After S502, in S903, the associating unit 107 associates the acoustic data for synthesis, the data of the second posture model, and the second physical information with each other. Specifically, first, based on the analysis result in step S502, the associating unit 107 converts the captured image captured at the same time as the time corresponding to the point in time when the sound volume of the sound data for synthesis reaches the maximum value. The data of the second posture model generated based on this is specified. That is, based on the result of the analysis in step S502, the associating unit 107 generates a captured image captured at the same time as the time corresponding to the point in time when the sound volume of the sound data for synthesis reaches the maximum value. Identify the frame data of the second posture model frame. Next, the associating unit 107 has a function of extracting sound data for synthesis from the sound data, and corresponds to the sound data for synthesis, the frame data of the second posture model frame, and the frame data of the second posture model frame. Associate the second physical information with each other. In this manner, the associating unit 107 associates the acoustic data for synthesis, the data of the second posture model, and the second physical information with each other.

Referring to FIG. 10, association processing in association unit 107, in particular, association processing between acoustic data for synthesis and frame data of a plurality of second posture model frames will be described. FIG. 10 is an explanatory diagram for explaining an example of matching processing in the matching unit 107 according to the second embodiment. In FIG. 10, the upper portion shows second posture model frames 1001a to 1001e arranged in time series, and the lower portion shows acoustic data 1002a to 1002c represented by time-series audio signals. In addition, the lower part of FIG. 10 shows, as an example, audio data corresponding to 48 kHz among the audio data in the form of an audio signal. Here, the sound data 1002b indicates the sound data for synthesis extracted by the extraction processing in S502. Also, acoustic data 1002a and 1002c indicate audio data deleted by the extraction processing in S502 from the audio data acquired in S501. In FIG. 10, the horizontal axis is the time axis, and the vertical axis in the lower part of FIG. 10 indicates the amplitude of the audio signal.

The acoustic data 1002b is associated with one of the second posture model frames 1001a to 1001e as synthesizing acoustic data for synthesizing with the virtual viewpoint image. The analysis in S502 identifies the point in time when the volume of the sound data 1002b reaches its maximum, that is, the point in time when the sound data 1002b reaches its maximum amplitude. After the identification, the second posture model frame generated based on the captured image captured at the same time as the time is identified. In the example shown in FIG. 10, the time corresponding to the second posture model frame 1001d and the time corresponding to the maximum amplitude of the acoustic data 1002b match. Therefore, the associating unit 107 associates the acoustic signal 1002b with the second posture model frame 1001d. If there is no second posture model frame at the same time as the time corresponding to the time when the acoustic data reaches its maximum amplitude, first, as with the associating unit 107 according to the first embodiment, at the time when the acoustic data reaches its maximum amplitude, A second posture model frame at a time closest to the corresponding time is identified. Next, the time corresponding to the point in time when the acoustic data reaches its maximum amplitude may be associated with the specified second posture model frame.

After the acoustic signal 1002b and the second posture model frame 1001d are associated with each other, one or more second posture model frames included in a predetermined period before and after the time of the second posture model frame 1001d are combined with the acoustic signal. and the signal 1002b. Note that it is not necessary to associate the acoustic signal 1002b with all the second posture model frames included in the period. For example, among the second posture model frames included in the period, the acoustic signal 1002b may be associated only with the second posture model frames having a predetermined time interval between the second posture model frames. good. Note that the method of determining the second posture model frame adjacent to the second posture model frame 1001d with which the acoustic signal 1002b is associated is not limited to the one described above. Also, the method of associating the acoustic data with the second posture model is not limited to the one described above, and any method that can synchronize the acoustic data with the second posture model may be used.

After S503, the first information processing apparatus 100 executes the processes from S511 to S514. After S514, in S911, the first model generation unit 104 generates first physical information. Next, in S<b>912 , the evaluation unit 108 evaluates the degree of matching between the frame data of the first posture model frame to be evaluated and the frame data of the second posture model frame. and the second posture model frame match. If it is determined that they match in S515, the evaluation unit 108 executes the process of S913. In S913, the evaluation unit 108 compares the frame data and first physical information of the first posture model frame and the frame data and second physical information of the second posture model frame after the first posture model frame to be evaluated. Determine whether or not they match.

If it is determined in S913 that they match, in S516 the data generation unit 109 adds the sound for synthesis associated with the data of the second posture model in S503 to the virtual viewpoint image data generated in S513. Synthesize data. After that, the data generation unit 109 outputs the synthesized virtual viewpoint image data to the output device. If it is determined in S912 or S913 that they do not match, the data generation unit 109 outputs the virtual viewpoint image data generated in S513 as it is to the output device. The first information processing apparatus 100 determines whether or not the termination condition is satisfied in S520. The first information processing apparatus 100 repeatedly executes the processing from S511 to S516 until the termination condition is satisfied, and terminates the processing of the flowchart shown in FIG. 9 when the termination condition is satisfied.

As described above, according to the first information processing apparatus 100, it is possible to efficiently synthesize the virtual viewpoint image data and the sound data acquired in different imaging spaces. As a result, sound that cannot be reproduced in a studio or the like can be efficiently synthesized with the virtual viewpoint image, so that the workload for synthesizing the acoustic data and the virtual viewpoint image data can be reduced. Further, the first information processing apparatus 100 calculates the degree of matching between the data of the first posture model and the data of the second posture model, and also the first physical information corresponding to the first posture model and the data corresponding to the second posture model. It evaluates the degree of matching with the second physical information. According to the first information processing apparatus 100 as described above, it is possible to improve the accuracy of automatic synthesis of the virtual viewpoint image data and the sound data acquired in different imaging spaces. As a result, it is possible to prevent sound data at an incorrect time from being combined with the virtual viewpoint image data.

(Other embodiments)
The present disclosure supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in the computer of the system or device read and execute the program. processing is also feasible. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

In addition, within the scope of the disclosure, the present disclosure allows free combination of each embodiment, modification of arbitrary constituent elements of each embodiment, or omission of arbitrary constituent elements in each embodiment.

100 information processing device 103 image generation unit 104 first model generation unit 105 model acquisition unit 106 sound acquisition unit 108 evaluation unit 109 data generation unit

Claims (14)

a first model acquiring means for acquiring data of a first posture model generated based on first multi-viewpoint images obtained from a plurality of imaging devices imaging a first imaging space;
image acquisition means for acquiring data of a virtual viewpoint image generated based on the first multi-viewpoint image;
A second model for acquiring data of a second posture model generated based on second multi-viewpoint images obtained from a plurality of imaging devices imaging a second imaging space different from the first imaging space. acquisition means;
sound acquisition means for acquiring sound data collected when the second multi-viewpoint image is captured in the second imaging space;
evaluation means for evaluating a degree of matching between the first posture model and the second posture model;
data generation means for generating data including the virtual viewpoint image and the sound based on the degree of matching;
An information processing device characterized by comprising: further comprising an associating means for analyzing the acoustic data and associating the second posture model data with the analyzed acoustic data;
The data generating means specifies the first posture model corresponding to the second posture model based on the degree of matching, the virtual viewpoint image corresponding to the specified first posture model, and the 2. The information processing apparatus according to claim 1, wherein data is generated in which the acoustic data associated with the data of the second posture model identified as corresponding to one posture model are associated with each other. The associating means analyzes the sound data, cuts out sound data for synthesis from the sound data based on the result of the analysis, and cuts out the sound data for synthesis; making a correspondence with the data of the second posture model corresponding to the period of the sound data for synthesis,
3. The information processing apparatus according to claim 2, wherein the data generating means generates data including the virtual viewpoint image and the synthesis sound. 4. The information processing apparatus according to claim 3, wherein the associating means cuts out the sound data for synthesis based on the volume level of the sound data. 5. The information according to claim 4, wherein the associating means cuts out the sound data for synthesis based on a volume level corresponding to a predetermined frequency in the sound data. processing equipment. The evaluation means includes information indicating positions of joints of the first posture model, information indicating positions of joints of the second posture model, and information indicating connection relationships between joints in the first posture model. and information indicating a connection relationship between joints in the second posture model, and evaluating the degree of matching between the first posture model and the second posture model. The information processing apparatus according to any one of claims 1 to 5. physical information acquiring means for acquiring first physical information that is physical information of the first posture model and second physical information that is physical information of the second posture model;
The evaluation means evaluates the degree of matching between the first posture model and the second posture model based on the first physical information and the second physical information. Item 7. The information processing apparatus according to any one of Items 1 to 6. The first physical information and the second physical information are the velocity and acceleration of joint parts in the first posture model and the second posture model, and the first posture model and the second posture. 8. The information processing apparatus according to claim 7, wherein the information indicates at least one of the angles of parts in the model. The first posture model is a three-dimensional model representing joints and connection relationships between joints in an object existing in a first imaging space, and the second posture model exists in a second imaging space. 9. The information processing apparatus according to any one of claims 1 to 8, wherein the model is a three-dimensional model showing joints in an object and connection relationships between the joints. a first image group acquiring means for acquiring the first multi-viewpoint images;
a first foreground obtaining means for obtaining, as a foreground area, an image area corresponding to an object in the captured image for each of a plurality of captured images that constitute the first multi-viewpoint image;
a first model generation means for generating the first posture model corresponding to the object based on the foreground region acquired for each of a plurality of captured images;
image generation means for generating the virtual viewpoint image based on the first multi-viewpoint image and the foreground region obtained for each of a plurality of captured images that constitute the first multi-viewpoint image;
further having
The first model acquisition means acquires data of the first posture model generated by the first model generation means,
The information processing apparatus according to any one of claims 1 to 9, wherein the image obtaining means obtains data of the virtual viewpoint image generated by the image generating means. a second image group acquiring means for acquiring the second multi-viewpoint images;
a second foreground obtaining means for obtaining, as a foreground area, an image area corresponding to an object in the captured image for each of a plurality of captured images forming the second multi-viewpoint image;
a second model generation means for generating the second posture model corresponding to the object based on the foreground region acquired for each of a plurality of captured images;
sound acquisition means for acquiring a signal of sound collected by the sound collecting device from the sound collecting device installed in the second imaging space, and digitizing the signal of the sound to acquire the data of the sound; ,
further having
the second model acquisition means acquires data of the second posture model generated by the second model generation means;
The information processing apparatus according to any one of claims 1 to 10, wherein the sound acquisition means acquires the sound data acquired by the sound acquisition means. a first information processing device that generates first posture model data and virtual viewpoint image data based on first multi-viewpoint images obtained from a plurality of imaging devices imaging a first imaging space; ,
generating data of a second posture model based on second multi-viewpoint images obtained from a plurality of imaging devices imaging a second imaging space different from the first imaging space; a second information processing device that generates acoustic data based on an acoustic signal obtained by sound collection by a sound collector installed in the imaging space;
has
The first information processing device acquires the second posture model generated by the second information processing device and the acoustic data, and generates the first posture model and the second posture model. an information processing system that generates data including the virtual viewpoint image and the sound based on the degree of matching between the virtual viewpoint image and the sound. a first model acquisition step of acquiring data of a first posture model generated based on first multi-viewpoint images obtained from a plurality of imaging devices imaging a first imaging space;
an image acquisition step of acquiring data of a virtual viewpoint image generated based on the first multi-viewpoint image;
A second model for acquiring data of a second posture model generated based on second multi-viewpoint images obtained from a plurality of imaging devices imaging a second imaging space different from the first imaging space. an acquisition step;
a sound acquisition step of acquiring sound data collected when the second multi-viewpoint image is captured in the second imaging space;
an evaluation step of evaluating a degree of matching between the first posture model and the second posture model;
a data generation step of generating data including the virtual viewpoint image and the sound based on the degree of matching;
An information processing method characterized by having A program for operating a computer as the information processing apparatus according to any one of claims 1 to 11.

Images