WO2022048113A1

WO2022048113A1 - Collaborative performance method and system, terminal device, and storage medium

Info

Publication number: WO2022048113A1
Application number: PCT/CN2021/076155
Authority: WO
Inventors: 段新; 段拙然
Original assignee: 佛山创视嘉科技有限公司
Priority date: 2020-09-07
Filing date: 2021-02-09
Publication date: 2022-03-10
Also published as: CN112203114A; CN112203114B

Abstract

The present application is suitable for use in the technical field of computers, and provides a collaborative performance method and system, a terminal device, and a storage medium, comprising: a first user sending collaborative performance invitation information by means of a first terminal, the collaborative performance invitation information comprising performance scene information; when acquiring invitation acceptance information returned by a second terminal, the first terminal and the second terminal together loading a target virtual reality scene corresponding to the performance scene information from a server; displaying the target virtual reality scene, and instructing the first user to implement collaborative performance with a second user in the target virtual reality scene. The embodiments of the present application enable people to implement remote music communication.

Description

Collaborative performance method, system, terminal device and storage medium

This application claims the priority of the Chinese patent application with the application number 202010927502.4 and the invention title "Cooperative performance method, system, terminal device and storage medium" filed in the China Patent Office on September 7, 2020, the entire contents of which are approved by Reference is incorporated in this application.

technical field

The present application belongs to the field of computer technology, and in particular, relates to a collaborative performance method, system, terminal device and storage medium.

Background technique

With the development of society and the improvement of living standards, people's entertainment methods are becoming more and more abundant, among which music entertainment such as singing and musical instrument performance is a popular entertainment method. However, in real life, the performance of musical works often requires everyone's cooperation, and it is often difficult for people in different regions to reach the same place to collaborate on musical works, which makes it difficult to achieve collaborative performance of musical works.

technical problem

Embodiments of the present application provide a collaborative performance method, system, terminal device, and storage medium to solve the problem in the prior art of how to easily and effectively enable people in different regions to perform music collaboratively.

technical solutions

In order to solve the above-mentioned technical problems, the technical solutions adopted in the embodiments of the present application are:

A first aspect of the embodiments of the present application provides a collaborative performance method, where the method is applied to a first terminal corresponding to a first user, including:

Sending collaborative performance invitation information, where the collaborative performance invitation information includes performance repertoire and performance scene information;

If the acceptance invitation information returned by the second user through the second terminal is obtained, load the target virtual reality scene corresponding to the performance scene information from the server together with the second terminal; the target virtual reality scene is the service The virtual reality scene determined by the terminal according to the collaborative performance invitation information;

The target virtual reality scene is displayed, and the first user is instructed to perform the performance piece together with the second user in the target virtual reality scene.

The second aspect of the embodiments of the present application provides another collaborative performance method, the method is applied to the second terminal corresponding to the second user, and includes:

Receive the collaborative performance invitation information, and return the acceptance invitation information to the first terminal corresponding to the first user; the collaborative performance invitation information includes performance repertoire and performance scene information;

Loading the target virtual reality scene corresponding to the performance scene information from the server together with the first terminal; the target virtual reality scene is the virtual reality scene determined by the server according to the collaborative performance invitation information;

The target virtual reality scene is displayed, and the second user is instructed to perform the performance piece together with the first user in the target virtual reality scene.

A third aspect of the embodiments of the present application provides a collaborative performance system, where the system includes a first terminal corresponding to a first user, a second terminal corresponding to a second user, and a server;

The first terminal is configured to execute the collaborative performance method as described in the first aspect;

The second terminal is configured to execute the collaborative performance method as described in the second aspect;

The server is used for receiving the collaborative performance invitation information, and determining a target virtual reality scene according to the collaborative performance invitation information; and for transmitting data of the target virtual reality scene to the first terminal and the second terminal.

A fourth aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program At the time, the terminal device is made to implement the steps of the collaborative performance method described in the first aspect or the second aspect.

A fifth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, enables a terminal device to implement the first aspect or The steps of the collaborative performance method described in the second aspect.

A sixth aspect of the embodiments of the present application provides a computer program product that, when the computer program product runs on a terminal device, enables the terminal device to execute the steps of the collaborative performance method described in the first aspect or the second aspect.

beneficial effect

In this embodiment of the present application, the first user sends a collaborative performance invitation through the first terminal, and after obtaining the invitation acceptance information returned by the second user through the second terminal, the first user loads the collaborative performance invitation information from the server together with the second terminal. The determined virtual reality scene (that is, the target virtual reality scene), and then displaying the target virtual reality scene and instructing the first user to perform the repertoire in collaboration with the second user in the target virtual reality scene. Since the second user can be invited according to the collaborative performance invitation information and the target virtual reality scene can be loaded from the server, the first user and the second user can collaboratively play the specified performance piece in the target virtual reality scene, which can be convenient and effective. Realize remote music entertainment interaction, so that people in different regions do not need to arrive at the same place, and based on virtual reality technology, they can easily and effectively realize immersive remote music collaborative performance.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are required to be used in the description of the embodiments or the prior art.

1 is a schematic diagram of an application scenario of a collaborative performance method provided by an embodiment of the present application;

Fig. 2 is the realization flow schematic diagram of the first collaborative performance method provided by the embodiment of the present application;

3 is a schematic diagram of a virtual electronic striking pad provided by an embodiment of the present application;

4 is a schematic diagram of a virtual MIDI keyboard provided by an embodiment of the present application;

5 is a schematic diagram of a virtual string provided by an embodiment of the present application;

6 is a schematic diagram of a virtual hole provided by an embodiment of the present application;

7 is a schematic diagram of the present application providing a kind of musical instrument operation prompt information corresponding to a virtual hole according to an embodiment;

Fig. 8 is the extraction schematic diagram of rhythm tone and melody tone provided by the embodiment of the present application;

Fig. 9 is the realization flow schematic diagram of the second kind of collaborative performance method provided by the embodiment of the present application;

10 is a schematic diagram of a collaborative performance system provided by an embodiment of the present application;

11 is a schematic diagram of another collaborative performance system provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of a first terminal provided by an embodiment of the present application.

FIG. 13 is a schematic diagram of a second terminal provided by an embodiment of the present application.

FIG. 14 is a schematic diagram of a terminal device provided by an embodiment of the present application.

Embodiments of the present invention

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

In order to illustrate the technical solutions described in the present application, the following specific embodiments are used for description.

In the current music entertainment method, it is necessary for multiple music lovers to arrive at the agreed place to realize it, and it is difficult for music lovers in different regions and far away to communicate with each other in music entertainment synchronously. In order to solve this technical problem, the embodiments of the present application provide a collaborative performance method, system, terminal device, and storage medium. The collaborative performance method includes the first user sending a collaborative performance invitation through the first terminal, and after obtaining the second user After receiving the invitation information returned by the second terminal, load the target virtual reality scene constructed according to the collaborative performance invitation information from the server together with the second terminal, and then display the target virtual reality scene and instruct the first user to play in the target virtual reality scene. and perform the repertoire in collaboration with the second user. Since the second user can be invited according to the collaborative performance invitation information and the target virtual reality scene can be loaded from the server, the first user and the second user can collaboratively play the specified performance piece in the target virtual reality scene, which can be convenient and effective. Realize remote music entertainment interaction, so that people in different regions can realize remote music entertainment communication based on virtual reality technology.

Exemplarily, FIG. 1 is a schematic diagram of an application scenario of a collaborative performance method provided by an embodiment of the application, including a server, multiple users and their corresponding terminals and interactive devices (which may include a head-mounted display device, an earphone, a microphone, a handle, etc.). /data gloves, etc.). When multiple users need to realize music entertainment exchange through collaborative performance, one of the users acts as the first user (that is, the inviter of the collaborative performance), and sends the collaborative performance invitation information through the terminal corresponding to the first user (referred to as the first terminal). to the server, so that the collaborative performance invitation information is directly or indirectly transmitted through the server to the terminal (referred to as the second terminal) corresponding to at least one second user (that is, other users invited by the first user); then, if The second user accepts the invitation and sends the invitation acceptance information through the second terminal. After receiving the invitation acceptance information, the first terminal loads the target virtual reality scene from the server together with the second terminal; The user shows the target virtual reality scene, and instructs the first user to play the preset performance repertoire in collaboration with the second user in the target virtual reality scene (the first terminal may specifically output the information and instruction information of the target virtual reality scene to the first user Wearing a head-mounted display device to display the target virtual reality scene and instruct the first user), similarly, the second terminal displays the target virtual reality scene to the second user, and instructs the second user to interact with the target virtual reality scene in the target virtual reality scene. The first user performs the piece of performance collaboratively, thereby completing the remote collaborative performance of the first user and the second user, so that users in different regions can realize remote music entertainment communication based on the virtual reality technology.

Example 1:

2 shows a schematic flowchart of a first collaborative performance method provided by an embodiment of the present application. The method is applied to a first terminal, and the details are as follows:

In the embodiment of the present application, the first user is the inviter of the current collaborative performance method, the second user is the user invited by the first user; the terminal device used by the first user is the first terminal, and the terminal device used by the second user is the first terminal. The terminal device is the second terminal. The first user, the second user, the first terminal, and the second terminal are only described as differences. Any user can be the first user when he wants to become an inviter, and perform the following steps from S201 to S201 to his corresponding terminal device as the first terminal. The step of S203: any one or more users other than the first user can be used as the second user, and accept the collaborative performance invitation information sent by the first user through the first terminal through its corresponding second terminal, so as to realize the first Remote collaborative performance of a user and a second user.

In S201, collaborative performance invitation information is sent, where the collaborative performance invitation information includes performance pieces and performance scene information.

The first user can log in to the client program preset by the first terminal through his own user account, so that the first terminal can access the collaborative performance system. After that, the first user can set the collaborative performance on the first terminal, generate collaborative performance invitation information, and send the collaborative performance invitation information to the server or the second terminal corresponding to the second user.

Specifically, the collaborative performance invitation information includes at least performance pieces and performance scene information, where the performance scene information is information for setting a virtual environment of a virtual reality scene, and the virtual environment may include K hall, grassland, seaside, and the like. Specifically, the first terminal can load the repertoire library and the virtual environment library pre-stored by the server from the server, select the current performance repertoire from the repertoire library, and select the current virtual environment from the virtual environment library, thereby generating the collaborative performance invitation information. Optionally, the collaborative performance invitation information may also include performance difficulty information, account information of the invited second user, the number of users for this collaborative performance, the authority of the second user, the user's avatar setting information, and performance form settings. information, etc.

In S202, if the acceptance invitation information returned by the second user through the second terminal is obtained, load the target virtual reality scene corresponding to the performance scene information from the server together with the second terminal; the target virtual reality scene It is the virtual reality scene determined by the server according to the collaborative performance invitation information.

In this embodiment of the present application, the second terminal corresponding to the second user may directly establish a communication connection with the first terminal, and directly receive the collaborative performance invitation information sent by the first terminal, or the first terminal may send the collaborative performance invitation information to the server. After that, it is forwarded by the server to the second terminal. After the second terminal obtains the collaborative performance invitation information, if the second user accepts the collaborative performance invitation, the second terminal directly sends the acceptance invitation information to the first terminal, or indirectly returns the acceptance invitation information to the first terminal through the server.

If the first terminal obtains the invitation acceptance information returned by the second user through the second terminal, the first terminal loads the target virtual reality scene corresponding to the above-mentioned performance scene information from the server together with the second terminal. Specifically, after acquiring the invitation acceptance information, the first terminal instructs the second terminal to load the target virtual reality scene while loading the target virtual reality scene from the server; or, after the second terminal sends the invitation acceptance information, The target virtual reality scene is automatically loaded from the server, and the first terminal loads the target virtual reality scene from the server after acquiring the invitation acceptance information.

Specifically, the target virtual reality scene is a virtual reality scene determined by the server according to the collaborative performance invitation information. The server pre-stores a variety of virtual reality scenes including K hall, grassland, seaside and other virtual environments, and the server selects a virtual reality scene from the pre-stored multiple virtual reality scenes according to the performance scene information in the collaborative performance invitation information, and The virtual avatar and virtual performance device corresponding to the first user, and the virtual avatar and virtual performance device corresponding to the second user are added, thereby generating a target virtual reality scene. In one embodiment, the collaborative performance invitation information includes, in addition to the performance piece and performance scene information, the user's avatar setting information and performance form setting information, and the user's avatar setting information includes the virtual avatar selected for the first user. Avatar and information of the virtual avatar selected for the second user, the performance form setting information includes the performance form selected for the first user and the information of the corresponding virtual performance device, and the performance form selected for the second user and the corresponding virtual performance equipment information, wherein the virtual performance device may include any one or more of a virtual microphone, a virtual musical instrument, and a virtual simple performance device; after the first terminal sends the collaborative performance invitation information to the server through step S201, the service The terminal selects the current virtual reality scene from the pre-stored virtual reality scene library according to the performance scene information contained in the collaboration invitation information, and adds the corresponding virtual avatar in the current virtual reality scene according to the user's avatar setting information, and according to the performance The form setting information adds a corresponding virtual performance device to the current virtual reality scene, so as to obtain the target virtual reality scene, and after receiving the loading request from the first terminal and the second terminal, the data of the target virtual reality scene is sent. to the first terminal and the second terminal, thereby completing the loading of the target virtual reality scene by the first terminal and the second terminal. In another embodiment, the collaborative performance invitation information only includes performance pieces and performance scene information. After the first terminal sends the collaborative performance invitation information to the server through step S201, the server sends the collaborative performance invitation information to the server according to the collaborative performance invitation information. The current virtual reality scene is selected from the pre-stored virtual reality scene library; after the first terminal obtains the acceptance invitation information returned by the second terminal, the first user sends the server through the first terminal the set by himself. The information of the virtual avatar and the virtual performance device, the second user sends the information of the virtual avatar and the virtual performance device set by himself to the server through the second terminal, and then the server correspondingly adds from the selected current virtual reality scene. The virtual avatar and virtual performance device corresponding to the first user, and the virtual avatar and virtual performance device corresponding to the second user, thereby obtain the target virtual reality scene for subsequent loading by the first terminal and the second terminal.

In S203, the target virtual reality scene is displayed, and the first user is instructed to perform the performance piece in collaboration with the second user in the target virtual reality scene.

In this embodiment of the present application, the first terminal and the second terminal are both connected to corresponding interactive devices such as a head-mounted display device, an earphone, a microphone, a handle, and a data glove. According to the loaded data of the target virtual reality scene, the first terminal outputs the visual image information of the target virtual reality scene to the head display device worn by the first user to show the first user that the virtual environment, the virtual performance device, the first The target virtual reality scene of the virtual avatars of the user and the second user, and by adding instruction information for instructing the first user to play in the visual image information, instructing the first user to perform the performance through an interactive device such as a handle, a data glove, or a microphone The performance action is performed, so that the first user and the second user can perform the performance piece collaboratively in the target virtual reality scene.

Specifically, the collaborative performance invitation information further includes performance form setting information for setting the performance form of the first user and the performance form of the second user, and the performance forms include singing and musical instrument performance;

Correspondingly, the step S203 includes:

S20301: Display the target virtual reality scene, and display performance prompt information according to the performance form setting information and the performance repertoire;

S20302: Acquire feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and output the feedback information, so that the first user can perform a performance in the target virtual reality scene. The target virtual reality scene cooperates with the second user to perform the repertoire.

In the embodiment of the present application, the collaborative performance invitation information specifically includes performance form setting information, and the performance form setting information specifically sets the performance form of the first user and the performance form of the second user. Specifically, the performance form may include singing and musical instrument performance. Exemplarily, when both the first user's performance form and the second user's performance form are singing, the current collaborative performance form is specifically a chorus form; when both the first user's performance form and the second user's performance form are musical instrument performances , the current form of collaborative performance is specifically the ensemble form; when the first user's performance form and the second user's performance form include both singing and musical instrument performance, the current form of collaborative performance is specifically the accompaniment form.

In step S20301, after the first terminal presents the target virtual reality scene to the first user, the corresponding performance prompt information is determined and displayed according to the performance form setting information and the performance repertoire. If the performance form of the user is singing, the lyrics information corresponding to the performance piece is obtained as the current performance prompt information; if the performance form of the first user set in the performance form setting information is musical instrument performance, then the performance piece corresponding to the performance piece is obtained. The musical instrument operation prompt information is used as the current performance prompt information, thereby instructing the first user to perform a corresponding performance action.

In step S20302, the first terminal obtains the performance action performed by the first user according to the performance prompt information through the interactive device, and generates feedback information corresponding to the performance action of the first user through the calculation of the first terminal itself or with the help of the calculation of the server. output, and receive and output feedback information generated by the server or the second terminal according to the performance action performed by the second user. The feedback information includes at least visual feedback information, auditory feedback information, and may also include force feedback information. Specifically, the feedback information can be output through a head-mounted display device, the auditory feedback information can be output through an earphone, and the force feedback information can be output through a data glove or a handle. information, so as to feed back to the first user in real time the visual, auditory, and force perception effects generated in the target virtual reality scene after performing the performance action by the first user or the second user, so as to enhance the performance of the first user when performing collaborative performance with the second user. Realism and immersion.

Optionally, after the first terminal acquires the information of the performance movements of the first user, and the second terminal acquires the information of the performance movements of the second user, both are uploaded to the server; position, gesture, acceleration and other information) to generate corresponding feedback information and send it to the first terminal and the second terminal, and then the first terminal outputs the feedback information to the interactive device worn by the first user, and the second terminal outputs the feedback information to the interactive device worn by the first user. The feedback information is output to the interactive device worn by the second user. The steps for the server to generate feedback information according to the information of the performance action may be as follows: (1) According to the information of the performance action, perform collision detection and calculation by the spatial decomposition method or the hierarchical bounding box method, and obtain the collision detection result. The result includes the contact position of the user's virtual hand and the virtual object in the virtual reality scene calculated according to the position of the playing action, and the force on the virtual object calculated according to the gesture and acceleration of the playing action; (2) According to the collision The deformation calculation of the virtual object is performed on the detection result to obtain visual feedback information; (3) the user's playing effect on the virtual performance device is determined according to the collision detection result, and the corresponding audio information is obtained as the auditory feedback information; (4) According to the collision detection result, Combined with Newton's third law and the physical properties of virtual objects, force feedback calculations are performed to obtain force feedback information.

In the embodiment of the present application, since the corresponding performance prompt information can be accurately displayed according to the performance form setting information and the performance repertoire in the collaborative performance invitation information, even if the user does not understand music theory and the professional performance method of musical instruments, the performance prompt information can be displayed according to the performance prompt information. Accurately execute the corresponding performance actions, thereby reducing the professional performance requirements for users, improving the universal applicability and ease of operation of the collaborative performance method; and, because the feedback information generated by the performance actions performed by the first user and the second user can be obtained. and output, so that the user can obtain the feedback effect of the current collaborative performance in time, thus enhancing the sense of realism and immersion of the first user when performing the collaborative performance with the second user.

Optionally, if it is determined according to the performance form setting information that the performance form of the first user includes singing, the performance prompt information includes lyrics prompt information; the step S20302 includes:

A1: Acquire the first response data generated by the first user singing in the target virtual reality scene according to the lyrics prompt information and transmit it to the second terminal, and acquire the second user sent by the second terminal second response data generated by performing the performance action;

A2: Generate feedback information according to the first response data and the second response data, and output the feedback information; the feedback information includes auditory feedback information and visual feedback information.

In the implementation of this application, the performance form of the first user specifically set in the performance form setting information is singing, and the currently displayed performance prompt information includes lyrics prompt information. Specifically, the first terminal loads the lyrics data corresponding to the current performance piece from the server. , and output the lyric data to the head display device worn by the first user in time sequence according to the current performance progress. The lyric data includes at least lyric text information, and may also include prompt information such as vocalization, pitch, and rhythm corresponding to the lyrics.

Correspondingly, step S20302 includes steps A1 to A2. specifically:

In A1, the first user sings the above-mentioned performance piece according to the prompt of the lyrics prompt information, and the first terminal captures the sound of the first user singing the performance piece through the microphone, and generates corresponding singing audio data. The pre-stored singing dynamic image data for displaying the virtual avatar of the first user is acquired during the sound, and the singing audio data and the dynamic image data are used as the first response data. In addition, the first terminal also receives second response data sent by the second terminal and generated according to the performance action of the second user, the second response data may include audio data generated by the second user singing or playing the virtual musical instrument, and the second response data Video data of singing of the user's avatar or video data of musical instrument performance.

In A2, the first terminal generates and outputs corresponding feedback information according to the acquired first response data and the second response data. Specifically, according to the acquired audio data in the first response data and the audio data in the second response data, the auditory feedback information is generated and output to the earphone worn by the first user; according to the acquired first response data of the first user In the virtual avatar's singing dynamic image data and the second response data, the second user's singing dynamic image data or musical instrument performance dynamic image data is generated, and visual feedback information is generated and output to the head display device worn by the first user.

In the embodiment of the present application, when the performance form is specifically singing, the user can be guided to sing and output corresponding feedback information through the lyrics prompt information in the target virtual reality scene, so as to accurately and effectively realize the remote control of the user in the virtual reality scene. chorus, accompaniment.

Optionally, if it is determined according to the performance form setting information that the performance form of the first user includes musical instrument performance, the target virtual reality scene includes a virtual musical instrument or a virtual simple performance device corresponding to the target musical instrument, and the performance prompt information Including the musical instrument operation prompt information identified in the virtual musical instrument; the step S20302 includes:

B1: Acquire first response data generated by the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information, and transmit it to the second terminal , and obtain the second response data generated by the second user performing the performance action sent by the second terminal;

B2: Generate feedback information according to the first response data and the second response data, and output the feedback information, where the feedback information includes auditory feedback information, visual feedback information, and force feedback information.

In the embodiment of the present application, the performance form of the first user specifically set by the performance form setting information is musical instrument performance. Specifically, the performance form setting information also sets the target musical instrument to be played by the first user. The target musical instruments may include percussion instruments such as gongs, drums, and drums; keyboard instruments such as piano and electronic organ; stringed instruments such as violin, cello, guqin, and zither; Correspondingly, the current target virtual reality scene includes a virtual musical instrument or a virtual simple performance device corresponding to the target musical instrument. Specifically, the virtual musical instrument is a virtual object having the form of a target musical instrument in the target virtual reality scene (that is, a three-dimensional model constructed by imitating the actual target musical instrument). For example, if the current target musical instrument is a gong and drum, then in the target virtual reality scene Virtual gongs and drums in the form of gongs and drums may be set. Specifically, the virtual simple performance device is a virtual object in the form of a simple performance device for playing the target musical instrument in the target virtual reality scene. The simple performance device can simplify the user's operation on the target musical instrument, so that users who do not understand musical instruments can also Simple and effective implementation of musical instrument performance. Wherein, if the target musical instrument is a complex percussion instrument, the simple performance device can be an electronic percussion pad, and the virtual simple performance device is a virtual electronic percussion pad, the schematic diagram of which is shown in Figure 3; Exemplarily, when the target musical instrument selected by the user is When a drum kit is used, multiple hitting positions on the drum kit are mapped to the virtual electronic pad, so that the user can tap the virtual electronic pad in the target virtual reality scene to produce the same sound effects as operating the drum kit. Compared with the drum kit, the paddle has fewer operating positions and lower requirements on the striking action, so the complexity of the user's operation of the musical instrument can be reduced. If the target musical instrument is a keyboard musical instrument, the simple performance device may be a Musical Instrument Digital Interface (MIDI) keyboard, and the virtual simple performance device is a virtual MIDI keyboard, the schematic diagram of which is shown in FIG. 4 ; When the target instrument is a piano, by mapping multiple playing positions on the piano to the virtual MIDI keyboard, the user can play the virtual MIDI keyboard in the target virtual reality scene to produce sound effects consistent with playing the piano. The virtual MIDI keyboard has fewer keys than a piano, which can reduce the complexity of the user's operation of the musical instrument. If the target musical instrument is a stringed musical instrument, the virtual simple performance device may be composed of virtual strings in the form of several strings; exemplarily, if the current virtual stringed instrument is specifically a virtual violin, the virtual simple performance device is shown in FIG. 5 , It is composed of four strings corresponding to the four tone names of G, D, A, and E respectively; exemplarily, when the virtual stringed instrument is specifically a guitar, the virtual simple performance device is composed of four strings corresponding to E, A, D, G, B, E respectively. It consists of six strings with six note names. If the target musical instrument is a wind instrument, the virtual simple performance device may be a virtual hole in the form of several round holes, including blow holes and finger holes, as shown in FIG. 6 . Optionally, the blow hole is only used for indication, not for user operation, and the position of the finger hole can be pressed by the user through an interactive device such as a data glove. When the user presses at the position of the finger hole, the default The blowing hole has already taken effect; that is, the user can perform the blowing and pressing coordination effect equivalent to the actual wind instrument only by instructing the pressing operation, thereby further simplifying the user's musical instrument performance operation. Further, because the operation of stringed instruments and wind instruments is more complicated, its corresponding virtual simple performance device can be a virtual sound board, and the form of this virtual sound board is similar to the virtual electronic percussion board shown in Figure 3, and the user only needs to tap or press. The sound board can show the sound effects of stringed instruments and wind instruments in the virtual reality scene.

Correspondingly, the performance prompt information in the embodiment of the present application includes the musical instrument operation prompt information marked on the virtual musical instrument or the virtual simple performance device, and the musical instrument operation prompt information specifically includes the time point and operation mode of the operation, and the operation time point is when the repertoire is played. The time point when the sound effect of the target instrument appears in the middle, the operation mode can be hitting, pressing or plucking, etc. Exemplarily, when the virtual simple performance device is specifically a virtual electronic percussion pad as shown in FIG. 3 , then the musical instrument operation prompt information can be the percussion instruction information displayed on the virtual electronic percussion pad in time sequence, for example, at 1 to 6. Graphical information such as highlights, or sparks, is displayed on one of the number plates to instruct the user to strike the plate. Exemplarily, when the virtual simple performance device is specifically a virtual MIDI keyboard as shown in FIG. 4, then the musical instrument operation prompt information can be the pressing instruction information displayed on the virtual MIDI keyboard in time sequence, for example, in the MIDI keyboard. A highlighted or raised graphic message is displayed on a keyboard to instruct the user to press the keyboard. Exemplarily, when the virtual simple performance device is specifically a virtual string as shown in FIG. 5 , then the musical instrument operation prompt information can be the strumming/plucking/pressing instruction information displayed on the virtual string in time sequence, for example, the current need A virtual string to be operated is highlighted, and the text prompt information of "play", "pluck" or "press" is displayed to instruct the user to strum/pick/press the corresponding virtual string. Exemplarily, when the virtual simple performance device is specifically made up of a blow hole and a finger hole as shown in Figure 6, then the musical instrument operation prompt information can specifically be that some original state of the hollow finger hole is transformed into a solid state in time sequence, To instruct the user to press the corresponding finger hole, exemplarily, as shown in FIG. 7(si) Schematic diagram of the finger hole pressing corresponding to the seven roll calls.

In step B1, the first user operates the virtual musical instrument in the target virtual reality scene through the interactive device (handle or data glove) according to the musical instrument operation prompt information marked on the virtual musical instrument or the virtual simple performance device, so as to realize the musical instrument performance. At this time, the first terminal captures the action information of the first user operating the virtual musical instrument or the virtual simple performance device through the interactive device, and calculates the corresponding first response data from the action information through the response data determination algorithm of the first terminal itself , or send the action information to the server, and the server calculates the corresponding first response data through the response data determination algorithm and returns it to the first terminal. And the first terminal also receives second response data generated by the second user performing the performance action and sent by the second terminal or the server.

In step B2, the first terminal combines the first response data generated by the first user operating the virtual musical instrument and the second response data generated by the second terminal performing the performance action to determine and output the feedback information that needs to be fed back to the first user. Specifically, in addition to the auditory feedback information and the visual feedback information, the feedback information also includes information of the reaction force of the first user operating the virtual musical instrument or the virtual simple performance device—force feedback information. Specifically, the first terminal outputs the auditory feedback information to the headset worn by the first user, outputs the visual feedback information to the head display device worn by the first user, outputs the force feedback information to the handle held by the first user, or Wearing data gloves, so as to accurately feed feedback information to the first user.

In the embodiment of the present application, when the performance form is the performance of a musical instrument, the user can be instructed to operate the virtual musical instrument or the virtual simple instrument in the target virtual reality scene through the interactive device through the musical instrument operation prompt information marked on the virtual musical instrument or the virtual simple performance device. The performance device enables non-professional users to conveniently and accurately perform musical instrument performance in the target virtual reality scene through the musical instrument operation prompt information; and, since auditory feedback information, visual feedback information and The force feedback information is fed back to the user, so it is possible to enhance the sense of realism and immersion of the first user when performing collaborative performance with the second user. Further, when using virtual simple performance devices such as virtual electronic percussion pads, virtual MIDI keyboards, and virtual sound effect boards as virtual performance equipment in the target virtual reality scene, the user's operation can be further simplified, so that users who do not understand musical instruments can also simply Effectively implement instrumental performance in virtual reality scenarios.

Optionally, obtaining the first response data generated by the first user operating the virtual musical instrument or the virtual simple performance device in the virtual reality scene according to the musical instrument operation prompt information, and transmitting it to the second user. terminal, including:

B11: Acquire the action information of the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information;

B12: Generate the first visual response data and the first force response data according to the action information;

B13: Acquire corresponding pre-stored sound source information according to the action information, and generate first sound response data; the pre-stored sound source information is the sound effect performed by the target musical instrument, and the sound effect includes a rhythm sound or a melody sound;

B14: Transmit the first visual response data and the first sound response data to the second terminal;

Correspondingly, the step B2 includes:

generating auditory feedback information according to the first sound response data and the second sound response data in the second response data;

generating visual feedback information according to the first visual response data and the second visual response data in the second response data;

generating force feedback information according to the first force response data;

The auditory feedback information, the visual feedback information, and the force feedback information are output.

In this embodiment of the present application, the first response data specifically includes first sound response data, first visual response data, and first force-sensing response data. Specifically, in step B11, the first terminal obtains the sensor data on the interactive device to determine the action information of the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene through the interactive device according to the musical instrument operation prompt information . Specifically, the motion information includes motion position, motion gesture, and motion acceleration information. In step B12, the first terminal determines the performance of the first user and the virtual musical instrument or virtual instrument in the target virtual reality scene through the collision detection algorithm of itself or the server (for example, the spatial decomposition method or the hierarchical bounding box method) according to the action information. Collision detection results generated by virtual objects such as a simple performance device; then, according to the collision detection results and the deformation algorithm, determine the first visual response data, which is used to indicate that the first user operates the virtual musical instrument or the virtual simple performance device to the target virtual reality scene The deformation result brought by the virtual object in the virtual reality can be drawn with the NURBS interface provided in the virtual reality construction language OpenGL API; and, according to the collision detection result and the force feedback algorithm (such as the mass-spring model algorithm or the finite element method) , determine the first force sense response data, the first force sense response data is data representing the reaction force of the virtual musical instrument obtained after the first user operates the virtual musical instrument to bring force to the virtual musical instrument. In step B13, specifically according to the action position in the action information, it is determined whether the current user's operation matches the position indicated by the musical instrument operation prompt information. . Specifically, the pre-stored sound source information is a pre-stored sound effect played by the target musical instrument, and the sound effect may be a rhythm sound of a percussion instrument, or a melody sound of a keyboard instrument, a stringed instrument, a wind instrument, or the like. Optionally, the pre-stored sound source information is to extract the rhythm sound and/or the melody sound in the pre-stored audio file corresponding to the performance piece according to the time sequence, and the pre-stored audio file is to collect in advance the sound of at least one actual musical instrument playing the performance piece. The resulting audio file, the at least one actual instrument includes at least the target instrument. The server stores the pre-stored audio files of the currently played repertoire, and extracts the rhythm sounds or melody sounds played by each instrument at each moment in the pre-stored audio files according to the time sequence, and obtains the pre-stored audio source information arranged in time sequence; , when the action information performed by the user matches the musical instrument operation prompt information, the pre-stored sound source information corresponding to the target musical instrument at this moment is acquired as the current first sound response data. Exemplarily, as shown in Figure 8, it is assumed that the pre-stored audio file of the current performance includes the rhythm sound of the drum playing the performance and the melody tone of the piano playing the performance, and extract the music from the pre-stored audio file respectively. The performance time of the repertoire is the rhythm sound and melody sound of the 1'20" (1 minute 20 seconds), which are saved separately, and the rhythm sound that appears on the 1'20" when the drum is playing the repertoire, and the piano sound is obtained. The melody sound that appears at the 1'20" when the performance is played; after that, when the user selects the performance and selects the drum kit as the target instrument, if the user's action information at the 1'20" and the instrument operation prompt information indicate The striking position matches, then take the 1'20" rhythm sound stored in advance as the current pre-stored sound source information to generate the first sound response data; Or, when the user selects the piano as the target The action information of 20" is consistent with the flicking position indicated by the musical instrument operation prompt information, then the melody tone of the 1'20" stored in advance is used as the current pre-stored sound source information to generate the first sound response data. Similarly, Fig. 8 The process of extracting and saving the rhythm sound and melody sound of the 1'30" (1 minute and 30 seconds) shown in Fig. 1 is the same as that of the 1'20". It should be understood that Fig. 8 only illustrates the rhythm sound and Melody tone, the extraction and preservation process of the rhythm tone of other performance moments not shown in the figure, the melody tone are similar to the extraction and preservation process of the two performance moments of the example in the figure.This extracts the rhythm tone from the pre-stored audio file of the performance repertoire , The melody sound is saved as pre-stored sound source information, and the method of using the pre-stored sound source information as the generated first sound response information in the follow-up can be called a "keying restoration" method, through which the actual musical instrument performance current can be accurately restored. The sound of playing the track. Optionally, the above-mentioned pre-stored sound source information can also be downloaded from the sound source database on the network, or can be downloaded through, for example, professional audio editing software (Adobe Audition, AE), fruit music editing software (Fruity Loops Studio, FL studio) and other music editing software are obtained by synthesizing the sound effect of the target instrument playing the performance. In step B14, since the second user performing collaboratively with the first user needs to acquire the visual and auditory effects of the virtual musical instrument played by the first user to enhance the realism of the collaborative performance, the above-generated first visual response data and The first sound response data is directly or indirectly transmitted to the second terminal through the server, so that the second terminal generates corresponding feedback information to feed back to the second user, and the first force response data is received by the first user operating the virtual musical instrument. The reaction force data only needs to be fed back to the first user in the follow-up, and does not need to be transmitted to the second user.

Correspondingly, in step B2, the first terminal combines the first sound response data determined in step B13 with the second sound response data contained in the second response data (that is, the sound response data generated by the second user performing the performance action) Obtain the auditory feedback information, and output it to the earphone worn by the first user, so that the user can obtain the auditory feedback information; combine the second visual response data included in the second response data of the first visual response data set determined in step B12 Obtain the visual feedback information, and output it to the head display device worn by the second user, so that the user can obtain the visual feedback information; output the first force sense response data determined in step B12 directly to the first user as force sense feedback information Operate the interactive device or wear the data glove, so that the user can obtain the force feedback information.

In the embodiment of the present application, since the acquisition of the action information of the first user operating the virtual musical instrument through the interactive device accurately generates the corresponding first visual response data, the first force response data, and the first sound response data generated in combination with the pre-stored sound source information, And then combined with the second response data received by the first terminal to accurately output auditory feedback information, visual feedback information and force feedback information respectively, so it can improve the accuracy of feedback information output, and further enhance the first user's ability to communicate with the second user. Realism when users perform collaborative performances. Further, when the pre-stored sound source information is specifically the rhythm sound or melody sound extracted in time sequence from the pre-stored audio file corresponding to the performance piece, the sound of the target musical instrument playing the current piece of performance can be accurately restored through the "keying restoration" method.

Optionally, after the step S203, it also includes:

Acquire and output performance evaluation data.

In this embodiment of the present application, after the first user and the second user perform collaboratively, the first terminal statistically generates and outputs performance evaluation data of the first user according to the performance information recorded during the collaborative performance of the first user. Alternatively, the first terminal uploads the recorded information on the performance of the first user when performing collaboratively to the server, so that the server records the information on the performance of the first user recorded by the first terminal and the information on the performance recorded by the second terminal. 2. Perform statistical analysis on the information of the user's performance actions to obtain the overall performance evaluation data of the collaborative performance, and then the first terminal and the second terminal respectively obtain and output the overall performance evaluation data from the server. Specifically, when the performance action includes singing, the recorded performance action information includes the pitch information and rhythm information recorded in time sequence when the user sings. The pre-stored pitch information and pre-stored rhythm information of the repertoire are compared, and the user's singing score is obtained as the performance evaluation data. Specifically, when the performance action includes operating a virtual musical instrument, the recorded performance action information includes the action information and action frequency when the user operates the virtual musical instrument, and the statistical operation accuracy (the action information of the user operating the virtual musical instrument is consistent with the musical instrument operation prompt information) percentage), and use this accuracy rate as performance evaluation data. Optionally, in this embodiment of the present application, after acquiring the performance evaluation data, the first terminal may convert the performance evaluation data into image information and output it to the head-mounted display device worn by the first user, or convert it into audio information and output it to the first terminal. An earphone worn by the user, so that the first user can obtain the performance evaluation data.

In the embodiment of the present application, since the performance evaluation data can be obtained and output after the collaborative performance, the first user can obtain the evaluation feedback information of the collaborative performance in time, and the user can grasp the results of each collaborative performance in time so as to Make corresponding improvements and strengthen training to improve the intelligence and user experience of collaborative playing.

In this embodiment of the present application, the first user sends a collaborative performance invitation through the first terminal, and after obtaining the invitation acceptance information returned by the second user through the second terminal, the first user loads the collaborative performance invitation information from the server together with the second terminal. The determined virtual reality scene (that is, the target virtual reality scene), and then displaying the target virtual reality scene and instructing the first user to perform the repertoire in collaboration with the second user in the target virtual reality scene. Since the second user can be invited according to the collaborative performance invitation information and the target virtual reality scene can be loaded from the server, the first user and the second user can collaboratively play the specified performance piece in the target virtual reality scene, which can be convenient and effective. Realize remote music entertainment interaction, so that people in different regions can realize remote music entertainment communication based on virtual reality technology.

Embodiment 2:

FIG. 9 shows a schematic flowchart of a second collaborative performance method provided by an embodiment of the present application. The method is applied to a second terminal, and the details are as follows:

In this embodiment of the present application, the definitions of the first user, the second user, the first terminal, and the second terminal are exactly the same as those in the previous embodiment, and will not be repeated here.

In S901, the collaborative performance invitation information is received, and the acceptance invitation information is returned to the first terminal corresponding to the first user; the collaborative performance invitation information includes performance pieces and performance scene information.

The second terminal directly receives the collaborative performance invitation information sent by the first terminal, or indirectly receives the collaborative performance invitation information sent by the first terminal through the server, and after the second user inputs the information confirming the reception of the invitation, generates the invitation acceptance information and sending, so that the invitation acceptance information is directly sent to the first terminal corresponding to the first user or indirectly returned to the first terminal through the server. The collaborative performance invitation information includes at least performance repertoire and performance scene information, and may also include performance difficulty information, account information of the invited second user, the number of users for this collaborative performance, the permissions of the second user, and the user's avatar settings. Information, performance form setting information, etc. The specific meaning of the collaborative performance invitation information is the same as the collaborative performance invitation information in the first embodiment, and for details, please refer to the relevant description in the first embodiment.

In S902, load the target virtual reality scene corresponding to the performance scene information from the server together with the first terminal; the target virtual reality scene is the virtual reality scene determined by the server according to the collaborative performance invitation information .

After returning to accept the invitation information, the second terminal and the first terminal together load the target virtual reality scene corresponding to the performance scene information in the collaborative performance invitation information from the server. The specific meaning of the target virtual reality scene is the same as that in the first embodiment. The collaborative performance invitation information is the same, and for details, please refer to the relevant description in Embodiment 1.

In S903, the target virtual reality scene is displayed, and the second user is instructed to perform the performance piece together with the first user in the target virtual reality scene.

After loading the target virtual reality scene, the second terminal outputs the data of the target virtual reality scene to the interactive device worn by the second user, and instructs the second user to perform a performance action through the interactive device. Specifically, the second terminal outputs the visual image information of the target virtual reality scene to the head display device worn by the second user according to the loaded data of the target virtual reality scene, so as to show the second user that the virtual environment and the virtual performance device are included. , the target virtual reality scene of the virtual avatars of the first user and the second user, and by adding instruction information for instructing the second user to perform in the visual image information to instruct the second user to pass the handle, data glove or microphone, etc. The interactive device performs the performance action, so that the first user and the second user can perform the performance piece together in the target virtual reality scene.

Optionally, the collaborative performance invitation information further includes performance form setting information for setting the performance form of the first user and the performance form of the second user, and the performance forms include singing and musical instrument performance;

Correspondingly, the step S903 includes:

Displaying the target virtual reality scene, and displaying performance prompt information according to the performance form setting information and the performance repertoire;

Acquire feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and output the feedback information, so that the first user can perform the performance in the target virtual reality scene. Cooperate with the second user to perform the piece in the target virtual reality scene.

Optionally, if it is determined according to the performance form setting information that the performance form of the second user includes singing, the performance prompt information includes lyrics prompt information;

The acquiring feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and outputting the feedback information, includes:

Acquiring second response data generated by the second user singing in the target virtual reality scene according to the lyrics prompt information and transmitting it to the first terminal, and acquiring the performance performed by the second user sent by the first terminal second response data generated by the action;

Generate feedback information according to the first response data and the second response data, and output the feedback information; the feedback information includes auditory feedback information and visual feedback information.

Optionally, if it is determined according to the performance form setting information that the performance form of the second user includes musical instrument performance, the target virtual reality scene includes a virtual musical instrument or a virtual simple performance device corresponding to the target musical instrument, and the performance prompt information Including musical instrument operation prompt information identified on the virtual musical instrument or the virtual simple performance device;

acquiring first response data generated by the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information, and transmitting it to the second terminal, and Acquiring second response data sent by the second terminal and generated by the second user performing the performance action;

Generate feedback information according to the first response data and the second response data, and output the feedback information, where the feedback information includes auditory feedback information, visual feedback information, and force feedback information.

Optionally, the second response data includes second sound response data, second visual response data, and second force-sensing response data, and the acquisition of the first user according to the musical instrument operation prompt information in the target virtual The first response data generated by operating the virtual musical instrument or the virtual simple performance device in the real scene and transmitted to the second terminal, including:

Acquiring the action information of the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information;

generating the second visual response data and the second force response data according to the motion information;

Acquire corresponding pre-stored sound source information according to the action information, and generate first sound response data; the pre-stored sound source information is the pre-stored sound effect played by the target musical instrument, and the sound effect includes a rhythm sound or a melody sound;

transmitting the second visual response data and the second sound response data to the first terminal;

Correspondingly, the generating feedback information according to the first response data and the second response data, and outputting the feedback information, includes:

outputting the auditory feedback information according to the second sound response data and the first sound response data in the first response data;

outputting the visual feedback information according to the second visual response data and the first visual response data in the first response data;

According to the second force sense response data, the force sense feedback information is output.

Optionally, after the step S903, it also includes:

Acquire and output performance evaluation data.

In this embodiment of the present application, the specific process of the second terminal performing the above steps is similar or the same as the related steps of the first terminal in Embodiment 1. For details, please refer to the relevant description in Embodiment 1, which will not be repeated here.

In the embodiment of the present application, because the second terminal can obtain the collaborative performance invitation information sent by the first terminal, load and display the corresponding target virtual reality scene, and instruct the second user to perform collaborative performance with the first user in the target virtual reality scene. The predetermined performance repertoire can effectively realize the collaborative performance of the second user and the first user through the interaction between the second terminal and the first terminal, so that the first user and the second user can perform collaborative performance in the target virtual reality scene. Therefore, it is convenient and effective to realize remote music entertainment interaction, so that music lovers in different regions can realize remote music entertainment communication based on virtual reality technology.

Embodiment three:

FIG. 10 shows a schematic structural diagram of a collaborative performance system provided by an embodiment of the present application. For the convenience of description, only the part related to the embodiment of the present application is shown:

The collaborative performance system includes a first terminal, at least one second terminal and a server. The first terminal is used to execute the collaborative performance method described in the first embodiment, and the second terminal is used to execute the collaborative performance method described in the second embodiment. For details, please refer to the relevant descriptions of the first and second embodiments. I won't go into details.

Specifically, the server is used for receiving the collaborative performance invitation information, determining a target virtual reality scene according to the collaborative performance invitation information; and for transmitting the data of the target virtual reality scene to the first terminal and the the second terminal.

In the embodiment of the present application, the server is a terminal device that constructs and stores a virtual reality scene. The server can be built in advance through tools such as Virtual Reality Modeling Language (VRML), Java 3D (a set of application programming interfaces extended by Java language in the field of 3D graphics), or Open Graphics Library (OpenGL). And store various virtual reality scenes including K hall, grassland, seaside and other virtual environments, as well as pre-stored virtual performance equipment database and virtual avatar database. After receiving the collaborative performance invitation information, the server determines the corresponding virtual reality scene according to the performance scene information contained in the collaborative performance invitation information, and selects the virtual performance device and the virtual avatar from the virtual reality scene according to the selection of the first user and/or the second user. The performance equipment database selects the corresponding virtual performance equipment, and selects the corresponding virtual avatar from the virtual avatar database to add to the virtual reality scene, thereby obtaining the target virtual reality scene. After that, when the server receives the loading request from the first terminal and the second terminal, it transmits the data of the target virtual reality scene (specifically, the VRML file of the target virtual reality scene) to the first terminal and the second terminal, so that the data of the target virtual reality scene can be transmitted to the first terminal and the second terminal. The first terminal and the second terminal display the target virtual reality scene to the corresponding first user and the second user.

Optionally, in this embodiment of the present application, in addition to constructing, storing and transmitting virtual reality scenarios, the server can also be used as a data transmission medium between the first terminal and the second terminal, and the interaction between the first terminal and the second terminal can also be used. All are realized through the data transfer of the server.

Optionally, the server is also used to monitor the information of the performance of the first user recorded by the first terminal and the information of the performance of the second user recorded by the second terminal, and calculate and generate the corresponding performance according to the information of these performances. feedback information, and then return the feedback information to the first terminal and the second terminal, so that the first terminal outputs the feedback information to the first user, and the second terminal outputs the feedback information to the second user, so that the first terminal outputs the feedback information and feeds back the feedback information to the second user. The user and the second user can timely and accurately obtain the execution effect of their own and each other's performance actions, so as to enhance the sense of realism and immersion of the first user when performing collaborative performance with the second user.

Optionally, the server is also used to calculate and generate the performance evaluation data of the collaborative performance according to the information of the performance of the first user and the information of the performance of the second user, and output it to the first terminal and the second terminal for feedback. to the corresponding first user and second user, so that the user can grasp the evaluation feedback information of the current collaborative performance in time, and improve the intelligence and user experience of the collaborative performance.

As an example and not a limitation, the terminal collaborative performance system in this embodiment of the present application can be implemented based on a Web3D virtual reality network platform, and the first terminal, the second terminal (collectively referred to as user terminals), and the server can establish a communication connection through a 5G network. Specifically, as shown in FIG. 11 , the server may include a collaboration server and a web server, wherein the web server stores the files of the virtual reality scene in advance, specifically including VRML files with the suffix .wrl and java with the suffix .class (an object-oriented programming language) file. The user can log in to the corresponding user account through the browser on the user terminal, establish a communication connection with the server, and load the file of the corresponding target virtual reality scene, and display the target virtual reality scene according to the VRML file output of the target virtual reality scene. , according to the java file of the virtual reality scene, generate the Java used to realize the interaction between the inside and the outside of the target virtual reality scene Applet (a small application written in the Java language that can be directly embedded in a web page). After that, the monitoring thread in the collaborative server on the server side monitors the change information inside the target virtual reality scene recorded by the Java Applet on each user terminal in real time (it can be the collision detection result information brought by the user performing the performance action in the target virtual reality scene) , and pass the change information to the Java Applet programs of other user terminals through the communication thread of the collaborative server, so that the Java Applet programs of the other user terminals act the change information on the target virtual reality scene displayed respectively, and generate corresponding The feedback information is fed back to the corresponding user, so that each user can obtain the change effect produced by other users acting on the target virtual reality scene. Exemplarily, FIG. 11 shows the process that the user terminal a loads the VRML file with the suffix of .wrl and the java file with the suffix of .class from the Web server, and the server monitors the change information on the user terminal a through the monitoring thread And the process of feeding back to other user terminals through the communication thread.

The collaborative performance system of the embodiment of the present application realizes the transmission of collaborative performance invitation information, the determination, loading and display of the target virtual reality scene, and the output of feedback information through the interaction of the first terminal, the second terminal and the server, so that different users can Through the target virtual reality scene, the remote collaborative performance can be realized immersively, and the remote music entertainment interaction can be realized conveniently and effectively.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Embodiment 4:

FIG. 12 shows a schematic structural diagram of a first terminal provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown:

The first terminal includes: a collaborative performance invitation information sending unit 121 , an invitation acceptance information obtaining unit 122 , and a first displaying unit 123 . in:

The collaborative performance invitation information sending unit 121 is configured to send collaborative performance invitation information, where the collaborative performance invitation information includes performance pieces and performance scene information.

The acceptance invitation information obtaining unit 122 is configured to, if the acceptance invitation information returned by the second user through the second terminal is obtained, load the target virtual reality scene corresponding to the performance scene information from the server together with the second terminal; The target virtual reality scene is a virtual reality scene determined by the server according to the collaborative performance invitation information.

The first display unit 123 is configured to display the target virtual reality scene, and instruct the first user to perform the performance piece together with the second user in the target virtual reality scene.

FIG. 13 shows a schematic structural diagram of a second terminal provided by the present application. For the convenience of description, only the parts related to the embodiments of the present application are shown:

The second terminal includes: a collaborative performance invitation information receiving unit 131 , a loading unit 132 , and a second displaying unit 133 . in:

The collaborative performance invitation information receiving unit 131 is configured to receive collaborative performance invitation information, and return the acceptance invitation information to the first terminal corresponding to the first user; the collaborative performance invitation information includes performance repertoire and performance scene information;

The loading unit 132 is configured to load the target virtual reality scene corresponding to the performance scene information from the server together with the first terminal; the target virtual reality scene is the virtual reality determined by the server according to the collaborative performance invitation information. real scene;

The second display unit 133 is configured to display the target virtual reality scene, and instruct the second user to perform the performance piece together with the first user in the target virtual reality scene.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

Embodiment 5:

FIG. 14 is a schematic diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 14 , the terminal device 14 of this embodiment includes: a processor 140 , a memory 141 , and a computer program 142 stored in the memory 141 and executable on the processor 140 , such as a collaborative performance program. When the processor 140 executes the computer program 142, the steps in each of the above-mentioned embodiments of the collaborative performance method are implemented, such as steps 201 to S203 shown in FIG. 2 or steps S901 to S903 shown in FIG. 3 . Alternatively, when the processor 140 executes the computer program 142, the functions of the modules/units in the above device embodiments are implemented, for example, the functions of the units 121 to 123 shown in FIG. 12 or the functions of the units 131 to 133 shown in FIG. 13 . Function.

Exemplarily, the computer program 142 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 141 and executed by the processor 140 to complete the this application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 142 in the terminal device 14 . For example, the computer program 142 can be divided into a collaborative performance invitation information sending unit, an invitation acceptance information acquisition unit, and a first presentation unit; or, the computer program 142 can be divided into a collaborative performance invitation information receiving unit, a loading unit, Second display unit.

The terminal device 14 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device may include, but is not limited to, the processor 140 and the memory 141 . Those skilled in the art can understand that FIG. 14 is only an example of the terminal device 14, and does not constitute a limitation on the terminal device 14, and may include more or less components than those shown, or combine some components, or different components For example, the terminal device may further include an input and output device, a network access device, a bus, and the like.

The so-called processor 140 may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, Digital Signal Processors (Digital Signal Processors, DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (Field-Programmable Gate Arrays) Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 141 may be an internal storage unit of the terminal device 14 , such as a hard disk or a memory of the terminal device 14 . The memory 141 may also be an external storage device of the terminal device 14, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the terminal device 14 card, flash memory card (Flash Card), etc. Further, the memory 141 may also include both an internal storage unit of the terminal device 14 and an external storage device. The memory 141 is used to store the computer program and other programs and data required by the terminal device. The memory 141 may also be used to temporarily store data that has been output or will be output.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims

A collaborative performance method, the method being applied to a first terminal corresponding to a first user, characterized in that the method includes:

Sending collaborative performance invitation information, where the collaborative performance invitation information includes performance repertoire and performance scene information;

If the acceptance invitation information returned by the second user through the second terminal is obtained, load the target virtual reality scene corresponding to the performance scene information from the server together with the second terminal; the target virtual reality scene is the service The virtual reality scene determined by the terminal according to the collaborative performance invitation information;

The target virtual reality scene is displayed, and the first user is instructed to perform the performance piece together with the second user in the target virtual reality scene.
The collaborative performance method according to claim 1, wherein the collaborative performance invitation information further comprises performance form setting information for setting the performance form of the first user and the performance form of the second user, The performance forms described include singing and musical instrument performance;

Correspondingly, the displaying the target virtual reality scene and instructing the first user to perform the repertoire in collaboration with the second user in the target virtual reality scene includes:

Displaying the target virtual reality scene, and displaying performance prompt information according to the performance form setting information and the performance repertoire;

Acquire feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and output the feedback information, so that the first user can perform the performance in the target virtual reality scene. Cooperate with the second user to perform the performance piece in the target virtual reality scene.
The collaborative performance method according to claim 2, wherein if it is determined according to the performance form setting information that the performance form of the first user includes singing, the performance prompt information includes lyrics prompt information;

The acquiring feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and outputting the feedback information, includes:

Acquiring first response data generated by the first user singing in the target virtual reality scene according to the lyrics prompt information and transmitting it to the second terminal, and acquiring the performance performed by the second user sent by the second terminal second response data generated by the action;

Generate feedback information according to the first response data and the second response data, and output the feedback information; the feedback information includes auditory feedback information and visual feedback information.
The collaborative performance method according to claim 2, wherein if it is determined according to the performance form setting information that the performance form of the first user includes musical instrument performance, the target virtual reality scene includes a virtual musical instrument corresponding to the target musical instrument Or a virtual simple performance device, the performance prompt information includes musical instrument operation prompt information identified on the virtual musical instrument or the virtual simple performance device;

The acquiring feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and outputting the feedback information, includes:

acquiring first response data generated by the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information, and transmitting it to the second terminal, and Acquiring second response data sent by the second terminal and generated by the second user performing the performance action;

Generate feedback information according to the first response data and the second response data, and output the feedback information, where the feedback information includes auditory feedback information, visual feedback information, and force feedback information.
The collaborative performance method according to claim 4, wherein the first response data comprises first sound response data, first visual response data and first force-sensing response data, and the acquiring the first user according to The musical instrument operation prompt information is first response data generated by operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene, and transmitted to the second terminal, including:

Acquiring the action information of the first user operating the virtual musical instrument or the virtual simple performance device in the target virtual reality scene according to the musical instrument operation prompt information;

generating the first visual response data and the first force response data according to the motion information;

Acquire corresponding pre-stored sound source information according to the action information, and generate first sound response data; the pre-stored sound source information is the pre-stored sound effect played by the target musical instrument, and the sound effect includes a rhythm sound or a melody sound;

transmitting the first visual response data and the first sound response data to the second terminal;

Correspondingly, the generating feedback information according to the first response data and the second response data, and outputting the feedback information, includes:

outputting the auditory feedback information according to the first sound response data and the second sound response data in the second response data;

outputting the visual feedback information according to the first visual response data and the second visual response data in the second response data;

According to the first force sense response data, the force sense feedback information is output.
The collaborative performance method according to claim 1, characterized in that when the target virtual reality scene is displayed, and the first user is instructed to perform collaboratively with the second user in the target virtual reality scene After reciting the repertoire, it also includes:

Acquire and output performance evaluation data.
A collaborative performance method, the method being applied to a second terminal corresponding to a second user, comprising:

Receive the collaborative performance invitation information, and return the acceptance invitation information to the first terminal corresponding to the first user; the collaborative performance invitation information includes performance repertoire and performance scene information;

Loading the target virtual reality scene corresponding to the performance scene information from the server together with the first terminal; the target virtual reality scene is the virtual reality scene determined by the server according to the collaborative performance invitation information;

The target virtual reality scene is displayed, and the second user is instructed to perform the performance piece together with the first user in the target virtual reality scene.
The collaborative performance method according to claim 7, wherein the collaborative performance invitation information further comprises performance form setting information for setting the performance form of the first user and the performance form of the second user, The performance forms described include singing and musical instrument performance;

Correspondingly, the displaying the target virtual reality scene and instructing the second user to perform the repertoire in collaboration with the first user in the target virtual reality scene includes:

Displaying the target virtual reality scene, and displaying performance prompt information according to the performance form setting information and the performance repertoire;

Acquire feedback information generated by the first user and the second user performing performance actions in the target virtual reality scene according to the performance prompt information, and output the feedback information, so that the first user can perform the performance in the target virtual reality scene. Cooperate with the second user to perform the piece in the target virtual reality scene.
A collaborative performance system, comprising a first terminal corresponding to a first user, a second terminal corresponding to a second user, and a server;

the first terminal, configured to execute the method according to any one of claims 1 to 6;

the second terminal, configured to execute the method of claim 7;

The server is used for receiving the collaborative performance invitation information, and determining a target virtual reality scene according to the collaborative performance invitation information; and for transmitting data of the target virtual reality scene to the first terminal and the second terminal.
The collaborative performance system according to claim 9, wherein the server is further configured to monitor the performance information of the first user recorded by the first terminal and the information recorded by the second terminal. the information of the performance movements of the second user; calculate and generate corresponding feedback information according to the information of the performance movements of the first user and the performance movements of the second user; return the feedback information to the first user a terminal and the second terminal.
The collaborative performance system according to claim 10, wherein the server is further configured to calculate and generate the collaborative performance according to the information of the performance of the first user and the information of the performance of the second user and output the performance evaluation data to the first terminal and the second terminal.
A first terminal, wherein the first terminal is a terminal corresponding to a first user, characterized by comprising:

a collaborative performance invitation information sending unit, configured to send collaborative performance invitation information, where the collaborative performance invitation information includes performance repertoire and performance scene information;

an invitation acceptance information acquisition unit, configured to load the target virtual reality scene corresponding to the performance scene information from the server together with the second terminal if the acceptance invitation information returned by the second user through the second terminal is obtained; the The target virtual reality scene is a virtual reality scene determined by the server according to the collaborative performance invitation information;

The first display unit is configured to display the target virtual reality scene, and instruct the first user to perform the performance piece in collaboration with the second user in the target virtual reality scene.
A second terminal, wherein the second terminal is a terminal corresponding to a second user, characterized by comprising:

a collaborative performance invitation information receiving unit, configured to receive the collaborative performance invitation information, and return the acceptance invitation information to the first terminal corresponding to the first user; the collaborative performance invitation information includes performance repertoire and performance scene information;

a loading unit, configured to load the target virtual reality scene corresponding to the performance scene information from the server together with the first terminal; the target virtual reality scene is the virtual reality determined by the server according to the collaborative performance invitation information Scenes;

The second display unit is configured to display the target virtual reality scene, and instruct the second user to perform the performance piece together with the first user in the target virtual reality scene.
A terminal device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that when the processor executes the computer program, the terminal device realizes The steps of the method of any one of claims 1 to 8.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, a terminal device is made to realize the implementation of any one of claims 1 to 8 steps of the method.