CN113596353B

CN113596353B - Somatosensory interaction data processing method and device and somatosensory interaction equipment

Info

Publication number: CN113596353B
Application number: CN202110917645.1A
Authority: CN
Inventors: 周跃兵; 尹志伟; 范广祥; 刘丹勇
Original assignee: Aimyunion Technology Ltd
Current assignee: Aimyunion Technology Ltd
Priority date: 2021-08-10
Filing date: 2021-08-10
Publication date: 2024-06-14
Anticipated expiration: 2041-08-10
Also published as: CN113596353A

Abstract

The application relates to a somatosensory interaction data processing method, a somatosensory interaction data processing device and somatosensory interaction equipment, wherein the method comprises the steps of obtaining audio data to be played and corresponding matched first somatosensory image data, outputting the audio data to a sound system for sound effect processing and playing; receiving second motion image data fed back in real time by an image processing system; wherein the second motion image data is image data associated with a user, which is acquired by the image processing system based on the audio data when the audio data is played; and comparing and analyzing the second somatosensory image data according to the first somatosensory image data, and generating an interactive data processing result of the user based on the comparison and analysis to display. According to the technical scheme, an intelligent somatosensory interaction feedback auxiliary function can be provided for a user in time in an entertainment scene of limb movement.

Description

Somatosensory interaction data processing method and device and somatosensory interaction equipment

Technical Field

The application relates to the technical field of somatosensory interaction, in particular to a somatosensory interaction data processing method and device, somatosensory interaction equipment, computer equipment and a computer storage medium.

Background

In the existing somatosensory interaction entertainment equipment, a television, an audio equipment sound box or an intelligent terminal is generally adopted for use, for example, the television is used for providing a video display function, the sound box is generally used for providing an audio playing function, along with the development of technology, some intelligent functions and networking application functions are added on the traditional audio equipment of the television and the audio equipment, but the television adopts a television box with network camera shooting, the sound box adopts intelligent voice technology such as voice audio equipment, and voice interaction between a user and the sound box of the audio equipment can be realized; some sound equipment is also connected to the television and the sound box, and the functions of video call, photo taking and the like are realized by the camera through the camera; the intelligent technology provides assistance for users in aspects of music entertainment, shopping experience, information acquisition and the like.

However, in practical application, the applicant finds that in most of somatosensory entertainment scenes with limb movements, especially in entertainment scenes such as games, body building, dance training and the like, the requirements of the scenes on somatosensory feedback are high, and when the existing television, sound box or intelligent terminal and the like are adopted for construction and use, professional application aiming at the scenes is lacking, so that an intelligent somatosensory interaction feedback auxiliary function is difficult to provide for users in time.

Disclosure of Invention

The application provides a somatosensory interaction data processing method which can provide an intelligent somatosensory interaction feedback auxiliary function for a user in time in an entertainment scene of limb movement.

A somatosensory interaction data processing method comprises the following steps:

acquiring audio data to be played and corresponding matched first somatosensory image data, outputting the audio data to a sound system for sound effect processing and playing;

Receiving second motion image data fed back in real time by an image processing system; wherein the second motion image data is image data associated with a user, which is acquired by the image processing system based on the audio data when the audio data is played;

And comparing and analyzing the second somatosensory image data according to the first somatosensory image data, and generating an interactive data processing result of the user based on the comparison and analysis to display.

In one embodiment, the image processing system includes a camera device and an image recognition module; wherein,

The camera device is configured to shoot real-time images of somatosensory interaction of a user and output image data in a specified format;

The image recognition module is configured to recognize the real-time image, extract limb motion data and obtain the second motion image data for output.

In one embodiment, a method of acquiring first motion image data includes:

acquiring a section of audio and video data;

Playing the audio-video data, and identifying each time point of the audio part;

extracting limb action data corresponding to the image part of the audio and video data according to the time point;

And correlating the limb action data with the corresponding time point, and adding the correlated limb action data into the audio and video data to obtain first motion image data.

In one embodiment, a method of comparing and analyzing the second motion image data from the first motion image data includes:

Acquiring a time point of audio data being played;

Reading second limb movement data from the second motion image data according to the time point information;

Reading first limb motion data in the first somatosensory image data at the same time point;

and comparing the similarity of the first limb motion data with the similarity of the second limb motion data, and obtaining a comparison result according to a preset judgment rule.

In one embodiment, the method for identifying the real-time image and extracting limb motion data by the image identification module comprises the following steps:

receiving a video stream of the real-time image output by the camera device through the image recognition module, and converting the video stream into a frame picture with a set frame rate;

And according to the playing time progress information of the audio data, invoking a motion capture algorithm to calculate the motion trail of the user from the frame picture, and forming limb motion data of the user.

In one embodiment, the comparing and analyzing the second motion sensing image data according to the first motion sensing image data, and generating the interactive data processing result of the user based on the comparing and analyzing, and displaying the interactive data processing result comprises the following steps:

acquiring reference limb movements according to the first somatosensory image data, and differentiating the limb movement data with the reference limb movements to obtain movement deviation data;

Generating error correction prompt information of the limb actions of the user according to the action deviation data;

and outputting the error correction prompt information to a sound system for playing, or outputting the error correction prompt information to a display device for displaying.

In one embodiment, the somatosensory interaction data processing method further includes:

Receiving the step action data of the user somatosensory interaction detected by the action capturing equipment; wherein, the audio data also corresponds to the matched standard step action;

performing comparison analysis on the second motion image data according to the first motion image data, and generating an interactive data processing result of the user based on the comparison analysis for display, wherein the method comprises the following steps:

Acquiring reference limb movements according to the first somatosensory image data, and respectively judging the limb movement data by using the reference limb movements and judging the step movement data by using the standard step movements;

obtaining a grading value of the somatosensory interaction of the user according to the judging result;

and outputting the grading value to a sound system for playing through voice prompt information, or outputting the grading value to display equipment for displaying.

Calculating the minimum resolution and the minimum coding rate meeting the image parameter requirements according to the image parameter requirements of the comparison analysis of the second motion sensing image data;

respectively calculating the output resolution and the output coding rate of the camera device according to the lowest resolution and the lowest coding rate;

Sending a notification message to the camera device according to a preset network protocol, wherein the notification message comprises an image data output format switching instruction, the output resolution and the output coding rate;

controlling the image pickup device to output image data only containing gray component format based on the image data output format switching instruction;

And controlling the camera module to output image data according to the output resolution and the output coding rate.

In one embodiment, acquiring audio data to be played, and outputting the audio data to a sound system for sound effect processing, including:

acquiring audio data selected by a user, and identifying the music category style to which the audio data belong according to attribute information;

Determining an audio mode matched with the audio data according to the music category style; wherein, each music category style is matched with at least one sound effect mode in advance;

Acquiring sound effect configuration parameters according to the sound effect mode; wherein each sound effect mode corresponds to a set of sound effect configuration parameters, the sound effect configuration parameters including: one or more of volume adjustment parameters, filter parameters, compression ratio, start/resume time;

And sending the sound effect configuration parameters to the sound system, and carrying out corresponding sound effect processing on the played audio data by the sound system according to the sound effect configuration parameters.

Receiving pre-displacement data sent by the image recognition module; the image recognition module is configured to calculate an absolute spatial relationship between the motion trail of the user and the real-time action image, and predict the pre-displacement data of the user according to the absolute spatial relationship;

And controlling the camera device to track the movement position of the user.

A somatosensory interactive data processing apparatus comprising:

the audio processing unit is used for acquiring audio data to be played and corresponding matched first somatosensory image data, and outputting the audio data to the sound system for sound effect processing and playing;

The image processing unit is used for receiving second motion image data fed back in real time by the image processing system; wherein the second motion-sensing image data is image data associated with a user captured by an image processing system based on playing the audio data;

and the data analysis unit is used for comparing and analyzing the second somatosensory image data according to the first somatosensory image data, and generating an interactive data processing result of the user based on the comparison and analysis for displaying.

A somatosensory interactive apparatus comprising: an image processing system, a system host and a sound system; the image processing system is connected with the system host, and the sound system is connected with the system host;

The image processing system is used for acquiring image data associated with a user when the audio data are played;

the system host is used for acquiring audio data to be played, outputting the audio data to a sound system after sound effect processing, receiving second motion sensing image data fed back by the image processing system in real time, comparing and analyzing the second motion sensing image data according to the first motion sensing image data, and generating interactive data processing result output of the user based on the comparison and analysis for display; wherein the audio data is correspondingly matched with the first somatosensory image data;

The sound system is used for receiving the audio data output by the system host and processing and playing the audio data.

A computer apparatus, comprising:

one or more processors;

A memory;

One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: executing the somatosensory interaction data processing method.

A computer readable storage medium storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or instruction set being loaded by the processor and performing the somatosensory interaction data processing method described above.

The technical scheme of the application has the following beneficial effects:

The technical scheme of the application can be suitable for most entertainment scenes with limb movements, especially for games, body building, dance training and other entertainment scenes, can give users real-time and rapid somatosensory interaction feedback, especially for judging the accuracy of various high-speed movements of the users, guiding training, correcting action errors and the like, and improves the entertainment application experience of the users.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Drawings

The foregoing and/or additional aspects and advantages of the application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic diagram of a somatosensory interaction apparatus according to an embodiment;

FIG. 2 is a schematic electrical structure of a somatosensory interaction device according to an embodiment;

FIG. 3 is an exploded view of the hardware architecture of a somatosensory interaction device according to one embodiment;

FIG. 4 is a flow chart of a somatosensory interaction data processing method according to the present application;

FIG. 5 is a state diagram of a somatosensory interaction data processing method;

FIG. 6 is a schematic diagram of an exemplary alignment analysis and presentation interactive data processing flow;

FIG. 7 is a flowchart of sound effect processing;

FIG. 8 is a schematic diagram of an interactive data processing flow for alignment analysis and presentation in accordance with another embodiment;

fig. 9 is a flowchart of control of the image data format output by the image pickup apparatus;

fig. 10 is a schematic diagram of image data format control output by the image pickup device;

Fig. 11 is a flowchart of the image pickup apparatus output video format control;

FIG. 12 is a schematic diagram of a multi-person application scenario for a single somatosensory interactive device;

FIG. 13 is a schematic diagram of a multi-person application scenario for a multi-somatosensory interactive device;

FIG. 14 is a schematic diagram of a somatosensory interaction data processing apparatus according to the present application.

Detailed Description

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, but do not preclude the presence or addition of one or more other features, integers, steps, operations.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a somatosensory interaction device 100 according to an embodiment, and the technical scheme of the somatosensory interaction data processing method of the present application may be applied to the somatosensory interaction device 100 shown in fig. 1, where the device includes: an image processing system 01, a system host 13, and a sound system 03; wherein the image processing system 01 is connected with the system host 13, and the sound system 03 is connected with the system host 13.

Wherein, the image processing system 01 is used for acquiring image data associated with a user when playing audio data; the system host 13 is configured to obtain audio data to be played, perform audio processing on the audio data, output the audio data to the sound system 03, receive second motion image data fed back by the image processing system 01 in real time, perform comparison analysis on the second motion image data according to the first motion image data, and generate an interactive data processing result output of the user for display based on the comparison analysis; wherein the audio data is correspondingly matched with the first motion image data; the sound system 03 is used for receiving audio data output by the system host 13, and performing sound effect processing on the audio data and playing the audio data.

As shown in fig. 1, the image processing system 01 includes an image pickup device 12 and an image recognition module 19, and the sound system 03 includes an audio processing circuit 14 and a speaker unit 15, wherein the image pickup device 12 is configured to pick up a real-time image of a user's somatosensory interaction and output image data in a specified format; the image recognition module 19 is configured to recognize the real-time image, extract limb movement data output. The system host 13 outputs audio data to the audio processing circuit 14, and the audio processing circuit 14 outputs the audio data to the speaker unit 15 for playing after performing audio processing.

Exemplary, as shown in fig. 1, the somatosensory interaction apparatus 100 specifically includes: a housing 11, an imaging device 12, an image recognition module 19, an audio system 03, and a system main unit 13; the image processing system 01 comprises an image pickup device 12 and an image recognition module 19, wherein the image pickup device 12 acquires image data through the image pickup device 12, the image pickup device 12 is connected with a system host 13 through the image recognition module 19, and the system host 13 is connected with the loudspeaker unit 15 through the audio processing circuit 14; the image pickup device 12 picks up a real-time motion image of the user in front of the case 11, sends the image to the image recognition module 19, and the image recognition module 19 recognizes the image data by a motion capture algorithm, captures the user's limb motion data, etc., and then sends the relevant limb motion data to the system host 13.

The image recognition module 19, the system host 13 and the audio processing circuit 14 are mutually matched, the system host 13 realizes related functions such as system control, analysis processing and the like, the image recognition module 19 is focused on capturing high-speed image processing of user limb actions, the audio processing circuit 14 is mainly responsible for processing audio data, and the image recognition module and the system host are mutually matched in a labor division manner, so that performance optimization combination is achieved, higher image data processing efficiency and audio processing effect are achieved, higher response speed is achieved, and user application experience is improved.

Exemplary, referring to fig. 2, fig. 2 is an electrical schematic diagram of a somatosensory interaction apparatus 100 according to an embodiment; as illustrated, the audio processing circuit 14 includes a DSP chip 141, a digital-to-analog conversion chip 142, and a power amplifier module 143, which are sequentially connected, and a wireless microphone receiving module 144 connected to the DSP chip 141; the DSP chip 141 is connected to the system host 13, and the power amplifier module 143 is connected to the speaker unit 15. The operating principle of the audio processing circuit 14 is as follows: the DSP chip 141 performs sound effect processing on the audio data output from the system host 13; the digital-to-analog conversion chip 142 can convert digital signals to analog signals; the power amplification module 143 amplifies the power of the analog audio data and outputs the amplified analog audio data to the speaker unit 15 for playing; the voice signal of the wireless microphone may be directly processed by the DSP chip 141 and then sent to the speaker unit 15 for playing, or may be input as a control signal to the system host 13 for processing.

For example, as shown in fig. 2, in order to enrich the peripheral functions of the somatosensory interaction device 100, a video signal interface 131a, a microphone peripheral interface 131b, and the like may be further provided on the system host 13; the video signal interface 131a may be connected to an external display device such as a television, a projector, or the like. The microphone peripheral interface 131b may be connected to an external microphone; the user can input operation instructions through the keys of the microphone, and the system host 13 executes functions corresponding to the operation instructions, for example, the microphone volume is increased or decreased through the keys on the microphone; increasing and decreasing the volume of music; cutting songs; the upper curve and the lower curve; switching original singing and accompaniment; mute, etc. Meanwhile, the system host 13 may also be connected with an infrared receiving module 132, a bluetooth module 133, a voice recognition module 134, etc.; the infrared receiving module 132 may receive a control signal of the infrared remote controller, the bluetooth module 133 may receive a control signal of the bluetooth remote controller (such as a wristwatch, a bracelet, a U-section microphone with integrated bluetooth transmission, etc.), and the voice recognition module 134 may recognize a voice signal input by a user through the pickup microphone 134A.

For example, as shown in fig. 2, the system host 13 may further be provided with a motion capture peripheral interface 135 for connecting with at least one external motion capture device 135A, where the motion capture device 135A may be a dance mat, a dance pedal, etc. for detecting the step motion data of the user and sending the step motion data to the system host 13; the connection mode may be a wired connection or a wireless connection. For example, a USB connection mode, a wireless mode, such as Bluetooth, WIFI, 2.4G/5.8G connection mode, and the like, can be adopted. Preferably, the somatosensory interaction device 100 may be connected to a plurality of motion capture devices 135A, and the camera 12 captures real-time motion images of a plurality of users at the same time.

Illustratively, referring to fig. 3, fig. 3 is an exploded view of a hardware structure of a somatosensory interaction apparatus according to an embodiment; the front part of the box 11 is provided with a perforated panel 110, in order to avoid vibration of the camera device 12 caused by vibration of the loudspeaker unit 15 during operation, the camera device 12 is fixed on the perforated panel 110 through a damping structure 111, and the perforated panel 110 and the box 11 can be flexibly connected through a third flexible damping member 111 e.

As shown in fig. 3, the speaker unit 15 of the present application preferably adopts a 2.1 dual-channel design, specifically adopts a "2.1 channel + inverter tube" scheme, and as shown in the drawing, specifically includes a dual-voice coil mid-bass channel speaker 154 and a left channel tweeter 157L and a right channel tweeter 157R, and an inverter tube 155 is disposed on the back plate 112 of the housing 11. Preferably, the tweeter in the horn unit 15 is mounted on the hole site 110a; the perforated panel 110 is provided with a plurality of circular holes 110a, and the image pickup device 12, the speaker unit 15 (tweeter) can be fitted to any circular hole 110a on the perforated panel 110; a plurality of image pickup devices 12 can be fitted on the circular hole 110a; in addition, in combination with the selection of the speaker unit 15, a plurality of tweeters are adapted to the panel 110 with holes, so as to meet different requirements of different functions and different costs. Preferably, a speaker mesh 15A is provided between the speaker unit 15 and the perforated panel 110; a pickup microphone 134A and a voice recognition module 134 are provided at the top of the housing 11 to facilitate capturing of voice signals.

Illustratively, as shown in FIG. 3, an infrared receiving module 132 is disposed at the front of the housing 11 for receiving infrared signals; the power module 16 and the power amplifier module 143 are arranged on a PCB card, and a shielding cover 113 is arranged between the system host 13 and the power amplifier module 143 to avoid mutual interference; an active capture peripheral interface 135, a microphone peripheral interface 131b, an external power interface, and the like may be provided at the rear of the back plate 112. In addition, an electric rotating base 18 may be provided under the case 11, and the case 11 is rotated by the rotating base 18 to control the shooting direction of the imaging device 12, so as to track the movement position of the user for shooting.

The embodiments of the somatosensory interaction data processing method according to the present application will be described below with reference to the somatosensory interaction apparatus provided by the present application as an example.

Referring to fig. 4, fig. 4 is a flowchart of a somatosensory interaction data processing method according to the present application, which may include:

s10, acquiring audio data to be played and corresponding matched first somatosensory image data, and outputting the audio data to a sound system for sound effect processing and playing.

In this step, the system host 13 may acquire the audio data to be played, and after the system host 13 acquires the audio data, the system host 13 stores the first motion image data and sends the audio data to the sound system 03 for playing.

The audio data and the corresponding matched first motion image data can be two types of data matched in advance during content production, wherein the first motion image data can be limb motion related data and the like; for example, dance music, a piece of dance audio data corresponds to a set of body motion data of dance, and for example, taiji, the guiding music of each taiji corresponds to a piece of motion data of taiji boxing.

The audio data and the first motion image data may be rock, disco, square dance, or the like, or may be motion game, or the like.

For the manner of acquiring the audio data and the corresponding matched first motion image data by the system host 13, the audio data and the corresponding matched first motion image data may be searched from the locally stored data or downloaded from the cloud through the network according to the selection operation of the user.

In one embodiment, the first motion image data may be set in advance in a matching manner, or may be acquired through existing audio and video. Accordingly, the method of acquiring the first motion image data may include:

1) Acquiring a section of audio and video data;

2) Playing the audio-video data, and identifying each time point of the audio part;

3) Extracting limb action data corresponding to the image part of the audio and video data according to the time point;

4) And correlating the limb action data with the corresponding time point, and adding the correlated limb action data into the audio and video data to obtain first motion image data.

By the method, any audio and video data can be manufactured into the first somatosensory image data, so that the somatosensory interaction device can be used, and the method is suitable for the somatosensory interaction data processing method provided by the application, and can provide real-time and rapid somatosensory interaction feedback for users.

S20, receiving second motion image data fed back in real time by the image processing system; wherein the second motion-sensing image data is image data associated with the user, which is acquired by the image processing system when the audio data is played.

In this step, when the system host 13 plays the audio data through the sound system 03, the image processing system 01 simultaneously shoots and acquires the image data associated with the user, and performs somatosensory motion recognition to obtain the second somatosensory image data of the user.

As described in the foregoing embodiments, the image processing system 01 includes the image pickup device 12 and the image recognition module 19; accordingly, the image capturing apparatus 12 may be configured to capture a real-time image of the somatosensory interaction of the user and output image data in a specified format; the image recognition module 19 may be configured to recognize the real-time image, extract limb movement data, and obtain a second motion image data output.

In a specific operation, the camera device 12 shoots a real-time action image of a user and generates a video stream, the video stream is transmitted to the image recognition module 19, and the image recognition module 19 can perform quick recognition by utilizing embedded action capture; preferably, the image recognition module 19 is realized based on an independent chip and a circuit, and an embedded motion capture algorithm realizes a limb motion recognition function, and through independently arranged hardware, the computing capacity can be improved, the data processing capacity of the system host 13 can be reduced, the limb motion of a user can be captured at a high speed, and the user application experience can be improved.

In one embodiment, the process of the image recognition module 19 for recognizing real-time images and extracting limb motion data may include the following:

(1) And receiving the video stream of the real-time image output by the image pickup device through the image recognition module, and converting the video stream into a frame picture with a set frame rate.

(2) And according to the playing time progress information of the audio data, invoking a motion capture algorithm to calculate the motion trail of the user from the frame picture, and forming limb motion data of the user.

Specifically, the image recognition module 19 first converts the video stream into a frame picture with a set frame rate (for example, into a frame rate not greater than 120 frames/s), and calculates a motion track of the user from the frame picture by using a motion capture algorithm, so as to form limb motion data of the user.

S30, comparing and analyzing the second somatosensory image data according to the first somatosensory image data, and generating an interactive data processing result of the user based on the comparing and analyzing to display.

In this step, the system host 13 performs a comparison analysis on the second motion image data based on the feedback, and determines an interaction result by the comparison analysis on the second motion image data stored in advance, and then displays the interaction result.

In one embodiment, for the above comparison analysis method, the following may be included:

1) Acquiring a time point of the audio data being played in step S10;

2) Reading second limb movement data from the second motion image data according to the time point information;

3) Reading first limb motion data in the first somatosensory image data at the same time point;

4) And comparing the similarity of the first limb motion data with the similarity of the second limb motion data, and obtaining a comparison result according to a preset judgment rule.

According to the technical scheme, when the audio data are played, the first limb motion data in the first somatosensory image data are extracted according to the pre-associated time points and then are compared with the second limb motion data acquired in real time, so that an intelligent somatosensory interaction feedback auxiliary function is provided for a user.

Referring to fig. 5, fig. 5 is a state diagram of a somatosensory interaction data processing method, firstly, the system host 13 downloads audio data and corresponding matched first somatosensory image data, and then the system host 13 outputs the audio data to the sound system 03 for audio processing and playing; simultaneously, the first motion image data is stored locally, and the second motion image data fed back by the image processing system 01 in real time is synchronously received; and comparing and analyzing the first somatosensory image data with the second somatosensory image data to obtain a somatosensory interaction data processing result. By the somatosensory interaction data processing method, an intelligent somatosensory interaction feedback auxiliary function can be provided for a user in time in an entertainment scene of limb movement.

In one embodiment, referring to fig. 6, fig. 6 is a schematic diagram of an interactive data processing flow for the comparison analysis and display of the interactive data processing result in step S30, which may include the following steps:

S301a, acquiring a reference limb motion according to the first motion sensing image data, and performing difference between limb motion data of the second motion sensing image data and the reference limb motion to acquire motion deviation data.

S302a, generating error correction prompt information of the limb action of the user according to the action deviation data.

S303a, outputting the error correction prompt information to the sound system 03 for playing, or outputting the error correction prompt information to the display device for displaying.

For example, when playing Taiji music, the user exercises Taiji actions along with music rhythm and guiding information, and the Taiji actions are compared through limb action data, then a comparison result between standard actions is provided for the user in a voice prompt mode, and error correction voice prompts are generated, so that the user can be prompted to correct error actions, real-time interaction feedback is given to the user, and entertainment application experience of the user is improved.

In order to improve the audio data playing effect, the application sets specific sound effect processing for each style of audio data, thereby automatically identifying the type of the audio data and automatically adjusting the audio effect in the playing process.

Accordingly, referring to fig. 7, fig. 7 is a flowchart of an audio processing, and the process of performing the audio processing on the audio data by the audio system 03 in step S10 may include:

S101, acquiring audio data selected by a user, and identifying the music category style to which the audio data belongs according to attribute information.

S102, determining an audio mode matched with the audio data according to the music category style; wherein each music category style is pre-matched with at least one sound effect mode.

S103, obtaining sound effect configuration parameters according to the sound effect mode; wherein each sound effect mode corresponds to a set of sound effect configuration parameters, the sound effect configuration parameters including: volume adjustment parameters, filter parameters, compression ratio, start/resume time.

And S104, the sound effect configuration parameters are sent to the sound system 03, and the sound system 03 carries out corresponding sound effect processing on the played audio data according to the sound effect configuration parameters.

For example, in a dance scene, dance music selected by different users has different styles, and common dance music has various dance styles such as Chinese dance, rock roll, disco, square dance, hip hop, HI-POP, taiji and the like; in order to facilitate a user to quickly search for the needed dance music and set the sound effect modes, the somatosensory interaction data processing method of the application enables different song categories to automatically match with corresponding sound effect modes through automatic identification and configuration by embedding sound effect configuration parameters of multiple sound effect modes.

In the following, the motion-sensing interaction device 100 provided by the present application will be described in detail as an example, in this example, the system host 13 obtains audio data selected by a user, and simultaneously identifies a music category style to which the audio data belongs, and determines a corresponding sound effect mode according to the music category style; then, according to the sound effect mode, obtaining sound effect configuration parameters and sending the sound effect configuration parameters to a DSP chip; and the DSP chip carries out corresponding sound effect processing on the audio data according to the sound effect configuration parameters. For sound effect configuration parameters, mainly including: volume adjustment parameters, filter parameters, compression ratio, start/resume time, etc.

Taking 2.1 binaural sound system 03 as an example, two dance style auto-matching schemes of "disco" and "taiji" will be described below:

(1) When the user selects the "disco" style, the system host 13 sends sound effect configuration parameters (for example, the bass sound volume increases by 6dB, the 80hz increases by 6dB, the q value is 1, the sound volume of the full-frequency sound box channel (left and right speakers) attenuates by 1dB, a HISHIFT filter is opened, the parameters are 3KHZ, the amplitude is 2, the q value is 1, meanwhile, a bass channel limiter is opened, the parameters are configured to be threshold value-3 dB, the compression ratio is 10:1, the starting time is 10ms, and the recovery time is 200 ms), and the DSP chip carries out corresponding sound effect processing on the played music according to the sound effect configuration parameters.

Through the sound effect configuration parameters, the music playing device can be suitable for the characteristics of strong disco dance music, no light and heavy, and simple lyrics and melody, which are loud like a metronome, so that the music playing device can automatically match the dance music style, and a user can obtain better application experience.

(2) When the user selects the "taiji" style song category, the system host 13 sends an audio configuration parameter (e.g., turn on a high pass filter and set the frequency to 80Hz order to second order, the bass channel volume decays by 3dB, the bass channel low pass filter frequency is modified from default 120Hz to 500Hz, the full-frequency speaker channel (left and right speakers) turn on an EQ frequency point parameter set to 250Hz, q 1 is +3dB in amplitude, turn on a second EQ frequency point, the parameter set to 600Hz, q 0.5dB in amplitude +2dB, turn on a third EQ frequency point, the parameter set to 6.3khz, q 1 is +3dB in amplitude).

Through the sound effect configuration parameters, the method can be suitable for the characteristics of the 'Taiji' style songs, and the somatosensory interaction data processing method can send out full, pleasant, clear and transparent Taiji music, so that a user can obtain better application experience.

In one embodiment, the somatosensory interaction data processing method of the present application may further set a karaoke mode automatic matching scheme, and specifically includes the following steps:

when the system host 13 detects that the microphone is on, it automatically switches to the karaoke mode, sends an audio profile to the DSP chip, opens the microphone channel switch inside the DSP chip, and opens the reverberation effect.

Meanwhile, the system host 13 may also receive the control information sent by the microphone and perform corresponding operations, for example: the user can make a voice song request through the microphone.

For example: the user can realize quick song requesting operation by the keys on the microphone of the user, and can also realize the volume increase and decrease of the microphone by the keys on the microphone; increasing and decreasing the volume of music; cutting songs; the upper curve and the lower curve; switching original singing and accompaniment; mute and other operating functions. When the microphone function is used, the microphone function can be used after the wireless microphone receiving module is matched.

In order to improve the interaction effect of somatosensory interaction, in one embodiment, the application can further compare and analyze through the footstep action data of the somatosensory interaction of the user, so that the training process, or game score judgment and the like can be realized.

Accordingly, the somatosensory interaction data processing method of the present application may further include receiving step motion data of the somatosensory interaction of the user detected by the motion capture device in step S20; wherein, the audio data also corresponds to the process of matching standard footstep actions.

Correspondingly, in step S30, the second motion image data is compared and analyzed according to the first motion image data, and the interactive data processing result of the user generated based on the comparison and analysis is displayed, and referring to fig. 8, fig. 8 is a schematic diagram of a comparison, analysis and display interactive data processing flow in another embodiment, which may specifically include:

S301b, acquiring reference limb movements according to the first motion image data, and respectively judging the limb movement data by using the reference limb movements and judging the step movement data by using the standard step movements.

And S302b, obtaining the grading value of the somatosensory interaction of the user according to the judging result.

And S303b, outputting the grading value to a sound system for playing through voice prompt information, or outputting the grading value to display equipment through an image output interface for displaying.

Specifically, the motion capture device is used for capturing the footstep motion of the user in somatosensory interaction, pre-stored reference limb motion and standard footstep motion are used for respectively judging detected limb motion data and footstep motion data, the scoring value of the user is obtained through judging the two data, and then the scoring value is displayed through voice or images according to actual needs.

For example, when dance music is played, a user finishes dance work in the music playing process, and the dance training process of the user can be scored through limb action data comparison, so that the training result of the user can be accurately and rapidly fed back; further, in the game, the score points of the user can be determined by comparing the limb motion data, and the score points can be quickly prepared. Three-dimensional feedback detection is carried out on the motion process of the user by combining the two motion data; the judgment accuracy of games and dances is greatly improved, and the experience of games and dances is improved.

In addition, the somatosensory interaction data processing method of the application can also acquire the foot motion data of a plurality of motion acquisition devices, can shoot real-time motion images of a plurality of users simultaneously through the camera device 12, and then respectively compare the real-time motion images of the users; for example, in a game, multiple players play simultaneously, scoring teams, greatly increasing interactivity and interest.

In order to improve the processing speed and the feedback speed of the somatosensory interaction data, the somatosensory interaction data processing method of the application can further comprise the step of switching and controlling the output image data format of the camera device.

Specifically, the system host 13 may control the data output format of the image capturing apparatus 12 according to the access state of the external display device or the video display state of the user, so that, in one embodiment, in the non-display image scenario, referring to fig. 9, fig. 9 is a flowchart of controlling the image data format of the image capturing apparatus output; the specific control method comprises the following steps:

s401, calculating the lowest resolution and the lowest coding rate meeting the image parameter requirements according to the image parameter requirements of the comparison analysis on the second motion sensing image data.

Here, the minimum resolution and minimum coding rate required are calculated mainly based on the image data parameter requirements of the system host 13.

And S402, respectively calculating the output resolution and the output coding rate of the image pickup device 12 according to the lowest resolution and the lowest coding rate.

Specifically, the output resolution and the output code rate of the image pickup device 12 are calculated correspondingly according to the minimum resolution and the minimum code rate required by the system host 13.

S403, sending a notification message to the camera device 12 according to a preset network protocol, wherein the notification message comprises an image data output format switching instruction, the output resolution and the output coding rate;

s404, based on the image data output format switching instruction, controlling the image pickup device to output image data only containing gray component format;

and S405, controlling the camera module to output image data according to the output resolution and the output coding rate.

Referring to fig. 10, fig. 10 is a schematic diagram illustrating control of the output image data format of the image capturing device, in which the system host calculates relevant parameters in real time and notifies the image capturing device to change the output format and resolution of the video captured video image.

Referring to fig. 11, fig. 11 is a flowchart of control of the output video format of the image capturing apparatus, in which the modified output video format is used as an improvement format, and the output normal video format before modification is used as a normal format, the switching control scheme is as follows:

s1, starting the image pickup device to pick up video images.

S2, judging whether the user video is required to be displayed when the external display device is accessed currently; if yes, s3 is executed, otherwise s4 is executed.

And s3, informing the camera device in real time through a network protocol to output video data in a conventional format.

And s4, informing the camera device in real time through a network protocol to change the camera device into an output improved format, so that the resolution, coding and the like of the video output by the camera device are reduced to only meet the current application function.

S5, running the camera application program to output video data until the camera 12 application program is closed.

In addition, during outputting of video data in a modified format, if the system host accesses an external display device and user video is required to be displayed, the image pickup apparatus is notified to output video data in a conventional format.

For current application functions, including but not limited to the tracking user motion trail or position functions required in the currently running application, the output video in the improved format is modified as follows:

The video acquisition format is changed into YUV400 format with only gray component Y;

(II) the resolution is reduced to only meet the current application function;

And (III) the video coding rate is reduced to only meet the application function of the current system host.

According to the scheme of the embodiment, when the external display equipment is connected and the user video is required to be displayed in a scene where the image is required to be displayed, the video camera device outputs video data in a conventional format through the real-time notification of the network protocol, and when the external display equipment is not connected or the external display equipment is connected and the user video is not required to be displayed, the video camera device is notified to output an improved format through the real-time notification of the network protocol, so that the resolution, the coding and the like of the video output by the video camera device are reduced to only meet the current application function, the intelligent control of the video format is achieved, the data processing capacity is reduced, and the bandwidth occupation and the transmission delay are reduced.

In order to ensure the integrity of the image data shot by the image pickup device, in one embodiment, the somatosensory interaction equipment 100 of the application is further provided with an electric rotating base 18, an electric part of the rotating base 18 comprises a rotating motor and a motor control circuit board, the shooting direction of the image pickup device 12 can be controlled through the rotating base, in the control process, a system host 13 sends a driving signal to the motor control circuit board, and the motor control circuit board drives the rotating motor to rotate after receiving the driving signal, so that the function of tracking the movement position of a user is realized; the user is always in the center of the real-time moving image, so that the limb actions of the user can be better shot, the situation that the user cannot acquire the motion beyond the shooting range in the motion process of the user is avoided, the omnibearing shooting function is improved, and the somatosensory interaction data processing effect is ensured.

Preferably, the method for tracking the motion position of the user for the system host comprises the following steps:

S501, receiving pre-displacement data sent by the image recognition module; the image recognition module is configured to calculate an absolute spatial relationship between the user motion trail and the real-time motion image, and predict the pre-displacement data of the user according to the absolute spatial relationship.

S502, controlling the image pickup device to track the movement position of the user.

Specifically, when the user performs positioning tracking shooting, the image recognition module 19 calculates the absolute spatial relationship between the motion trail of the user and the real-time motion image, predicts the pre-displacement data of the user according to the absolute spatial relationship, sends the pre-displacement data to the system host 13, and the system host 13 outputs a driving signal according to the pre-displacement data to the rotating base to drive the rotating base to rotate.

In the control process, firstly, a user shoots real-time video through the camera device 12 (the wide-angle camera device 12 can be adopted), video streams are obtained and output to the image recognition module 19, the independent processing capacity of the motion capture algorithm processing module is utilized, firstly, frame rate conversion is carried out on the video streams, frame pictures are utilized to calculate human skeleton motion tracks, then absolute spatial relations between the human skeleton motion tracks of the user and real-time motion images are calculated, pre-displacement data of the user are predicted, the pre-displacement data are sent to the system host 13, the system host 13 compiles into driving signals through processing of the pre-displacement data, the driving signals are sent to the motor control circuit board, and the motor control circuit board drives the rotating motor to run, so that the camera device 12 is controlled to track the motion position of the user, meanwhile, the loudspeaker unit 15 can be guaranteed to always point to the user, and the audio playing effect is guaranteed.

In addition, for the rotation state of the rotating base, the user movement track can be identified through the camera device, the motor for driving the rotating base rotates at the angular speed of not more than 0.1rad/s, and the user position is automatically tracked, so that the user is ensured to move at the central position of a picture in the speed of being smaller than 1 m/s transversely when the distance from the user to the product is 1.5m, in addition, the loudspeaker unit also tracks the user position along with the rotating base, so that the loudspeaker sounding axis is always directed to the user, and the user always obtains the best sound experience.

In order to ensure the audio playing effect, in one embodiment, the somatosensory interaction data processing method of the present application may further include:

The distance between the user and the somatosensory interaction equipment 100 is measured through the camera device 12, and the system host 13 automatically adjusts the volume of the sound system 03 according to the measured distance, for example, the user always keeps 80-85dBSPL of comfortable listening sound pressure for dance, body building, game and the like through measuring the distance and adjusting the volume.

Several embodiments of the somatosensory interaction apparatus of the present application are described below.

Based on the description related to the above embodiments, the somatosensory interaction apparatus 100 provided by the present application may be provided with a single or multiple image capturing devices 12, and the recognition rate is improved through the cooperation of the multiple image capturing devices 12; meanwhile, the somatosensory interaction data processing method can also support linkage cooperation of the multi-equipment multi-camera device 12, and combined with the connected external action acquisition equipment, the accurate identification of multiple users in the scene is captured in all directions, meanwhile, the multiple sound boxes form full scene immersion type sound hearing, and more shocking sound experience is brought to the user.

Referring to fig. 12, fig. 12 is a schematic diagram of a multi-user application scenario of the single motion-sensing interactive apparatus 100; in the figure, each motion interactive device 100 is provided with one camera device 12 or a plurality of camera devices 12, and a plurality of people can use the motion interactive device 100 to perform motion entertainment motions at the same time and can be interconnected with motion acquisition devices of the plurality of people, so that the motion entertainment motions can be performed.

Referring to fig. 13, fig. 13 is a schematic diagram of a multi-person application scenario of the multi-person somatosensory interaction apparatus 100; in the figure, each motion interactive device 100 may be provided with one camera device 12 or a plurality of camera devices 12, and a plurality of people use a plurality of motion interactive data processing methods to perform motion entertainment motions at the same time, and each motion interactive device 100 may be interconnected with motion acquisition devices of a plurality of people, so as to perform motion entertainment motions.

In summary, according to the embodiments of the present application, the somatosensory interaction data processing method and the somatosensory interaction device 100 can complete high-speed exercise AI decision games, body-building training, dance guidance, and the like, and can provide excellent application experience for users in entertainment activities.

Referring to fig. 14, fig. 14 is a schematic structural diagram of a somatosensory interaction data processing device according to the present application, which includes:

The audio processing unit 10 is configured to obtain audio data to be played and corresponding matched first somatosensory image data, and output the audio data to the sound system for audio processing and playing;

an image processing unit 20 for receiving second motion image data fed back in real time by the image processing system; wherein the second motion-sensing image data is image data associated with a user captured by an image processing system based on playing the audio data;

and the data analysis unit 30 is used for comparing and analyzing the second somatosensory image data according to the first somatosensory image data, and generating an interactive data processing result of the user based on the comparison and analysis for displaying.

The motion of each module in the motion-sensing interactive data processing device in each embodiment of the present disclosure corresponds to the steps in the motion-sensing interactive data processing method in each embodiment of the present disclosure, and detailed functional descriptions of each module in the motion-sensing interactive data processing device may be referred to the descriptions in the corresponding motion-sensing interactive data processing method shown in the foregoing, which are not repeated herein.

An embodiment of the computer device of the present application is set forth below, comprising:

one or more processors;

A memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: the somatosensory interaction data processing method according to any of the above embodiments is performed.

Embodiments of a computer readable storage medium storing at least one instruction, at least one program, code set, or instruction set loaded by the processor and performing the somatosensory interactive data processing method of any of the above embodiments are described below.

According to the technical scheme, an intelligent somatosensory interaction feedback auxiliary function can be provided for a user in time in an entertainment scene of limb movement.

The foregoing is only a partial embodiment of the present application, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present application, and such modifications and adaptations are intended to be comprehended within the scope of the present application.

Claims

1. A somatosensory interaction data processing method is characterized by comprising the following steps:

Receiving second motion image data fed back in real time by an image processing system; calculating the minimum resolution and the minimum coding rate meeting the image parameter requirements according to the image parameter requirements of the comparison analysis of the second motion sensing image data; respectively calculating the output resolution and the output coding rate of the camera device according to the lowest resolution and the lowest coding rate; controlling the camera device to output image data only containing gray component format according to the output resolution and the output coding rate, and outputting the image data according to the output resolution and the output coding rate; wherein the second motion image data is image data associated with a user, which is acquired by the image processing system based on the audio data when the audio data is played; the image processing system comprises an image pickup device and an image recognition module, wherein the image pickup device is configured to pick up real-time images of somatosensory interaction of a user and output image data in a specified format;

2. The somatosensory interaction data processing method according to claim 1, wherein the image recognition module is configured to recognize the real-time image, extract limb motion data, and obtain the second somatosensory image data output.

3. The somatosensory interaction data processing method according to claim 2, wherein the method for identifying the real-time image by the image identification module and extracting limb motion data comprises the following steps:

4. A somatosensory interaction data processing method according to claim 3, wherein the comparison analysis of the second somatosensory image data is performed according to the first somatosensory image data, and the interactive data processing result of the user is generated based on the comparison analysis and displayed, comprising:

5. The somatosensory interactive data processing method according to claim 3, further comprising:

6. The somatosensory interactive data processing method according to claim 2, further comprising:

and controlling the image pickup device to output the image data only containing the gray component format and output the image data at the output resolution and the output coding rate based on the image data output format switching instruction.

7. The somatosensory interaction data processing method according to claim 1, wherein acquiring audio data to be played, outputting the audio data to a sound system for sound effect processing, comprises:

8. The somatosensory interactive data processing method according to claim 2, further comprising:

And controlling the camera device to track the movement position of the user.

9. A somatosensory interactive data processing apparatus, comprising:

The image processing unit is used for receiving second motion image data fed back in real time by the image processing system; calculating the minimum resolution and the minimum coding rate meeting the image parameter requirements according to the image parameter requirements of the comparison analysis of the second motion sensing image data; respectively calculating the output resolution and the output coding rate of the camera device according to the lowest resolution and the lowest coding rate; controlling the camera device to output image data only containing gray component format according to the output resolution and the output coding rate, and outputting the image data according to the output resolution and the output coding rate; wherein the second motion-sensing image data is image data associated with a user captured by an image processing system based on playing the audio data; the image processing system comprises an image pickup device and an image recognition module, wherein the image pickup device is configured to pick up real-time images of somatosensory interaction of a user and output image data in a specified format;

10. A somatosensory interactive apparatus, comprising: an image processing system, a system host and a sound system; the image processing system is connected with the system host, and the sound system is connected with the system host; the system host executes the somatosensory interaction data processing method according to any one of claims 1 to 8;