CN113453048A - Audio and video tactile feedback setting method, audio and video playing method and computer equipment - Google Patents

Audio and video tactile feedback setting method, audio and video playing method and computer equipment Download PDF

Info

Publication number
CN113453048A
CN113453048A CN202010219219.6A CN202010219219A CN113453048A CN 113453048 A CN113453048 A CN 113453048A CN 202010219219 A CN202010219219 A CN 202010219219A CN 113453048 A CN113453048 A CN 113453048A
Authority
CN
China
Prior art keywords
haptic feedback
audio
video
instruction
feedback
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010219219.6A
Other languages
Chinese (zh)
Inventor
单德军
向彬
耿田琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Bilibili Technology Co Ltd
Original Assignee
Shanghai Bilibili Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Bilibili Technology Co Ltd filed Critical Shanghai Bilibili Technology Co Ltd
Priority to CN202010219219.6A priority Critical patent/CN113453048A/en
Publication of CN113453048A publication Critical patent/CN113453048A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42201—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42224—Touch pad or touch panel provided on the remote control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/47217—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks

Abstract

The invention discloses a method for setting tactile feedback of audio and video, a playing method, a device and computer equipment, and belongs to the technical field of video playing. The audio and video tactile feedback setting method comprises the following steps: acquiring a tactile feedback setting instruction triggered by a user, wherein the tactile feedback setting instruction comprises an audio or video clip to be set with tactile feedback and type information of the audio or the video; generating a haptic feedback instruction corresponding to the audio or video clip according to the type information and sound information contained in the audio or video clip, and storing the haptic feedback instruction in association with the audio or video clip, wherein the haptic feedback instruction comprises at least one haptic feedback event, and each haptic feedback event comprises time information and haptic information of haptic feedback. The method and the device can enable the user to have more immersive sensory experience when the audio or video file is played through the player of the mobile terminal.

Description

Audio and video tactile feedback setting method, audio and video playing method and computer equipment
Technical Field
The invention relates to the technical field of video playing, in particular to a tactile feedback setting method of audio and video, an audio and video playing method and computer equipment.
Background
With the rapid development of mobile terminals, the requirements of users on obtaining entertainment in the process of using the mobile terminals are also improved, and users can obtain audio-visual entertainment through vision and hearing in the entertainment process. This kind of information acquisition mode is single relatively, can not satisfy user's variety demand, and can not make the user have more immersive sense organ experience when using the smart mobile phone.
Disclosure of Invention
In view of this, a method, an apparatus, and a computer device for setting haptic feedback of audio and video are provided to solve the problem that a user cannot obtain more immersive sensory experience when using a smart phone.
The invention provides a method for setting tactile feedback of audio and video, which comprises the following steps:
acquiring a tactile feedback setting instruction triggered by a user, wherein the tactile feedback setting instruction comprises an audio or video clip to be set with tactile feedback and type information of the audio or the video;
generating a haptic feedback instruction corresponding to the audio or video segment according to the type information and sound information contained in the audio or video segment, and storing the haptic feedback instruction in association with the audio or video segment, wherein the haptic feedback instruction comprises at least one haptic feedback event, each haptic feedback event comprises time information and haptic information of haptic feedback, the time information comprises a start time and an end time, and the haptic information comprises at least one of intensity and sharpness.
Optionally, the generating a haptic feedback instruction corresponding to the audio or video clip according to the type information and the sound information included in the audio or video clip includes:
determining a haptic feedback instruction template according to the type information, wherein the haptic feedback instruction template comprises a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback;
detecting the loudness of sounds contained in the audio or video segment in chronological order;
generating a haptic feedback event whenever it is detected that the loudness satisfies the first rule, wherein a time corresponding to the audio or video segment when the loudness satisfies the first rule is taken as a start time of haptic feedback included in the generated haptic feedback event; taking the moment corresponding to the audio or video segment when the loudness does not meet the first rule as the end time of the generated tactile feedback contained in the tactile feedback event; determining a strength and/or sharpness of haptic feedback included in the generated haptic feedback event according to a change in loudness of a time period between the start time and the end time and the second rule.
Optionally, the audio-video haptic feedback setting method further includes:
and when a haptic feedback modification instruction is received, modifying the corresponding haptic feedback event according to the haptic feedback modification instruction.
The invention also provides an audio and video playing method, which comprises the following steps:
detecting whether the current playing progress reaches a preset position or not in the process of playing the audio or the video;
if the current playing progress reaches the preset position, acquiring a tactile feedback event related to the preset position, wherein the tactile feedback event comprises time information and tactile information of tactile feedback, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and/or sharpness;
generating corresponding haptic feedback according to the haptic feedback event.
Optionally, the haptic feedback event is generated based on the loudness of the sound satisfying a preset rule.
Optionally, the audio/video playing method further includes:
detecting whether a haptic feedback switch is in an on state, the haptic feedback switch for controlling generation of the haptic feedback.
Optionally, the audio/video playing method further includes:
and when a haptic feedback intensity adjusting instruction triggered by a user is received, adjusting the intensity of the haptic feedback according to the adjusting instruction.
The invention also provides a tactile feedback setting device of audio and video, comprising:
the system comprises an acquisition module, a display module and a control module, wherein the acquisition module is used for acquiring a tactile feedback setting instruction triggered by a user, and the tactile feedback setting instruction comprises an audio or video clip to be set with tactile feedback and the type information of the audio or the video;
and the generating module is used for generating a tactile feedback instruction corresponding to the audio or video segment according to the type information and the sound information contained in the audio or video segment, and storing the tactile feedback instruction in association with the audio or video segment, wherein the tactile feedback instruction comprises at least one tactile feedback event, each tactile feedback event comprises time information and tactile information of tactile feedback, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and/or sharpness.
The invention also provides an audio and video playing device, comprising:
the detection module is used for detecting whether the current playing progress reaches a preset position or not in the process of playing the audio or the video;
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring a tactile feedback event related to a preset position if the current playing progress reaches the preset position, the tactile feedback event comprises time information and tactile information of tactile feedback, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and/or sharpness;
a generating module for generating corresponding haptic feedback according to the haptic feedback event.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
The beneficial effects of the above technical scheme are that:
in the technical scheme, after a haptic feedback setting instruction triggered by a user is acquired, a haptic feedback instruction corresponding to an audio or video clip is generated according to type information contained in the haptic feedback instruction and sound information contained in the audio or video clip, and the haptic feedback instruction is stored in association with the audio or video clip so that the audio or video file carries the haptic feedback instruction. In addition, by applying the haptic feedback technology to playing audio or video, application scenes of the haptic feedback technology can be increased, and thus, the diversity requirements of users can be met.
Drawings
FIG. 1 is a block diagram of one embodiment of a system block diagram of a haptic feedback arrangement for audio and video in accordance with the present invention;
FIG. 2 is a flow chart of one embodiment of a method for haptic feedback setting of audio and video in accordance with the present invention;
FIG. 3 is a flowchart illustrating a detailed process of generating haptic feedback commands corresponding to the audio or video clips according to the type information and the sound information contained in the audio or video clips;
fig. 4 is a flowchart of an embodiment of an audio/video playing method according to the present invention;
FIG. 5 is a block diagram of a program for an embodiment of the audio/video haptic feedback setting apparatus according to the present invention;
fig. 6 is a program module diagram of an embodiment of an audio/video playing apparatus according to the present invention;
fig. 7 is a schematic diagram of a hardware structure of a computer device that executes a haptic feedback setting method or an audio/video playing method for audio/video according to an embodiment of the present invention.
Detailed Description
The advantages of the invention are further illustrated in the following description of specific embodiments in conjunction with the accompanying drawings.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
In the description of the present invention, it should be understood that the numerical references before the steps do not identify the order of performing the steps, but merely serve to facilitate the description of the present invention and to distinguish each step, and thus should not be construed as limiting the present invention.
Referring to fig. 1, fig. 1 is a schematic diagram illustrating an application environment according to an embodiment of the present application. The system of the application environment may include an authoring terminal 102, a playback terminal 106, and a server 104.
Wherein the authoring terminal 102 communicates with the server 104 over a network. The cast terminal 106 communicates with the server 104 via a network. The authoring terminal 102 and the playback terminal 106 may be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server 104 may be implemented as a stand-alone server or as a server cluster comprised of multiple servers. An audio and video tactile feedback setting tool is run on the authoring terminal 102. The authoring personnel can set the audio and video tactile feedback instructions by utilizing the audio and video tactile feedback setting tool at the authoring terminal 102.
Fig. 2 schematically shows a flowchart of a haptic feedback setting method of an audio-video according to a first embodiment of the present application. The method is applied to computer equipment, and the computer equipment takes a mobile terminal as an example for explanation. It is to be understood that the flow charts in the embodiments of the present method are not intended to limit the order in which the steps are performed. The following description is made by taking a mobile terminal as an execution subject.
As shown in fig. 2, the audio-video haptic feedback setting method includes steps S20 to S21, in which:
step S20, obtaining a haptic feedback setting instruction triggered by a user, wherein the haptic feedback setting instruction comprises an audio or video clip to be set with haptic feedback and the type information of the audio or the video;
specifically, when the user needs to set the haptic feedback instruction for audio or video, the user first selects the audio or video segment for which the haptic feedback instruction needs to be set through the haptic feedback setting tool, that is, selects which time period of audio or video the haptic feedback instruction needs to be set for.
There are many ways for a user to select an audio or video segment to be set with a haptic feedback instruction through a haptic feedback setting tool, for example, selecting an audio or video segment to be set with a haptic feedback instruction by using a mouse to frame an audio or video segment for a time period in an interface of a mobile terminal, or selecting an audio or video segment to be set with a haptic feedback instruction by using a touch mode to frame an audio or video segment for a time period in an interface of a mobile terminal.
When the user selects the audio or video segment needing to set the tactile feedback instruction through the tactile feedback setting tool, or after that, the user also needs to select the type information of the audio or video through the tactile feedback setting tool.
After receiving the audio or video clip to be set with the tactile feedback and the set audio or video type information selected by the user, the mobile terminal triggers and generates a tactile feedback setting instruction according to the setting information.
The tactile feedback is feedback that can reproduce a tactile sensation for a user through a series of actions such as an applied force, vibration, and the like. In this embodiment, the tactile feedback is preferably vibratory feedback.
The type information is information representing the type of the audio or video, for example, the type of the audio or video is "raindrop" type audio or video, "firework" type audio or video, "weather" type audio or video, "wrestling" type audio or video, and the like. The haptic feedback setting tool is pre-deployed with a plurality of different audio or video types for the user to set the type of the audio or video to be set currently.
Step S21, generating a haptic feedback instruction corresponding to the audio or video segment according to the type information and the sound information contained in the audio or video segment, and storing the haptic feedback instruction in association with the audio or video segment, where the haptic feedback instruction includes at least one haptic feedback event, each haptic feedback event includes time information and haptic information of haptic feedback, where the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness.
In particular, the different types of audio or video generated haptic feedback instructions may or may not be the same. The different audio or video generated haptic feedback may or may not be the same. After the audio or video clip to be generated is acquired, the mobile terminal automatically analyzes sound information contained in the audio or video clip, and then generates a corresponding tactile feedback instruction according to the change condition and the type information of the sound contained in the audio or video clip. In this embodiment, the haptic feedback command includes at least one haptic feedback event, each haptic feedback event includes time information and haptic information of haptic feedback, wherein the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness. Wherein, the strength is a vibration amplitude value for representing the tactile feedback, the sharpness is a value of stimulation characteristics brought by vibration for representing the tactile feedback, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the feeling is brought by the lower the value is.
In an embodiment, in order to generate a haptic feedback instruction corresponding to an audio or video clip according to type information and sound information included in the audio or video clip when a haptic feedback setting instruction is acquired, haptic feedback instruction templates corresponding to various types of audios and videos need to be created in advance, and then the created haptic feedback instruction templates are deployed in a haptic feedback setting tool, so that after the type information is acquired, a haptic feedback instruction template corresponding to a current type of audio and video can be acquired according to a mapping relationship between the type information and the haptic feedback instruction template, and then a haptic feedback instruction can be generated by using the haptic feedback template and the sound information.
In one example, suppose there is a haptic feedback template a corresponding to a "raindrop" type audio or video, with rules for generating haptic feedback commands in the haptic feedback command template a; the firework-like audio or video corresponds to a tactile feedback template B, and the tactile feedback instruction template B also has rules for generating tactile feedback instructions; the 'weather' audio or video corresponds to a tactile feedback template C, and the tactile feedback instruction template C also has rules for generating tactile feedback instructions; the "wrestling" type audio or video corresponds to a haptic feedback template D, which also has rules for generating haptic feedback commands. After obtaining the type information, analyzing the type information, and if the type information is found to be a raindrop type, generating a tactile feedback instruction according to the tactile feedback template a and sound information contained in the audio or video clip; if the type information is found to be the firework type, generating a tactile feedback instruction according to the tactile feedback template B and the sound information contained in the audio or video clip; if the type information is found to be of the "weather" type, a haptic feedback instruction may be generated according to the haptic feedback template C and the sound information included in the audio or video clip; if the type information is found to be a "wrestling" type, a haptic feedback command may be generated from the haptic feedback template D and the sound information included in the audio or video clip. It should be noted that the rules in the haptic feedback command template are used to determine the start time, the end time, and the strength and/or sharpness of the generated haptic feedback command.
In one embodiment, the haptic feedback event further includes information on the type of audio or video clip.
In this embodiment, after the setting of the haptic feedback command is completed, the haptic feedback command needs to be stored in association with the audio or video clip. In one embodiment, when the haptic feedback instruction is saved in association with an audio or video clip, the haptic feedback instruction may be saved to an audio or video file for a corresponding time period. In another embodiment, the haptic feedback command may be saved to a separate file, and then the file is mapped to the audio or video clip.
In the embodiment of the invention, after the haptic feedback setting instruction triggered by the user is acquired, the haptic feedback instruction corresponding to the audio or video clip is generated according to the type information contained in the haptic feedback instruction and the sound information contained in the audio or video clip, and the haptic feedback instruction is stored in association with the audio or video clip so as to enable the audio or video file to carry the haptic feedback instruction. In addition, by applying the haptic feedback technology to playing audio or video, application scenes of the haptic feedback technology can be increased, and thus, the diversity requirements of users can be met.
In an embodiment, after the haptic feedback instruction corresponding to the audio or video clip is generated, an audio or video file containing the audio or video clip needs to be uploaded to a server, so that other terminal devices can acquire the audio or video file from the server to play.
Further, referring to fig. 3, in an embodiment, the generating a haptic feedback instruction corresponding to the audio or video clip according to the type information and the sound information included in the audio or video clip includes:
step S30, determining a haptic feedback instruction template according to the type information, wherein the haptic feedback instruction template comprises a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback;
specifically, a plurality of haptic feedback instruction templates are pre-deployed in the haptic instruction setting tool, and the haptic feedback instruction templates corresponding to different types of audio or video are different. Each haptic feedback instruction template contains a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback, i.e. the first rule is used for determining under what circumstances a haptic feedback event needs to be generated, and the second rule is used for determining specific values of the strength and sharpness of the generated haptic feedback.
Step S31, detecting the loudness of the sounds contained in the audio or video segment in chronological order.
Specifically, the loudness of the sound contained in each time point is sequentially detected according to the time sequence of the audio or video segment, for example, the loudness of the sound contained in the audio or video segment is 20db at 0 second, and the loudness of the sound contained in the audio or video segment is 25db at 0.1 second.
In this embodiment, it is necessary to sequentially detect the loudness of the sound included in each time point in chronological order until the loudness of the sound included in the audio or video at all time points is detected.
In step S32, a haptic feedback event is generated whenever the loudness is detected to satisfy the first rule.
Specifically, when detecting the loudness of the sounds included in the audio or video segment in time sequence, if it is detected that the loudness value of the sound corresponding to the current time point satisfies the first rule, for example, it is detected that the loudness value of the sound corresponding to 0.1 second is greater than or equal to 20db, a haptic feedback event may be generated, and the time point (0.1 second) may be used as the start time of the generated haptic feedback included in the haptic feedback event.
After detecting that the loudness value of the sound corresponding to 0.1 second meets a first rule (for example, the loudness value is greater than or equal to 20db), continuing to detect the loudness of the sounds corresponding to other time points in time sequence, and when detecting that the loudness of the sound corresponding to the current time (for example, 1 second) does not meet the first rule, that is, the loudness value is less than 20db, taking the current time (1 second) as the end time of the generated haptic feedback included in the haptic feedback event.
After obtaining the start time and the end time of the haptic feedback, determining the strength and/or sharpness of the haptic feedback included in the generated haptic feedback event according to the change of the loudness of the sound in the time period between the start time and the end time and the second rule may be continued. In this embodiment, the intensity may be positively correlated to the amplitude of the sound, and the sharpness is changed based on a preset value in the haptic feedback command template, for example, the sharpness may be fixed based on the preset value to increase the change value of the loudness of the sound, or the sharpness may be based on the preset value and positively correlated to the change value of the loudness of the sound. In a particular scenario, the sharpness value may be set larger when a large change in loudness is found in the time period between the start time and the end time, and may be set smaller when a small change in loudness is found in this time period. The value of the sharpness value is used for representing the stimulation characteristics brought by vibration, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the touch is brought by the lower the value is. When the maximum value of the loudness in the time period between the start time and the end time is found, the vibration amplitude corresponding to the maximum loudness value may be used as the intensity of the haptic feedback, or the vibration amplitude corresponding to the loudness value is reduced according to a preset rule, so as to obtain the intensity of the haptic feedback.
In this embodiment, the time period between the start time and the end time is taken as the duration of the haptic feedback.
After generating the haptic feedback event, continuously detecting the loudness of the sounds corresponding to other time points according to the time sequence, when finding the next time period meeting the first rule, generating a second haptic feedback event according to the method, and repeating the process until the detection of the sounds corresponding to all the time points is completed.
In this embodiment, the corresponding feedback instruction template is acquired according to the type information, and then the haptic feedback event is generated according to the acquired feedback instruction template and the loudness information of the sound, so that different types of audio or video can generate different haptic feedback events, the diversity of haptic feedback is improved, the diversity requirement of a user can be met, and the user experience is improved.
Further, in an embodiment, the method for setting the haptic feedback of the audio/video further includes:
and when a haptic feedback modification instruction is received, modifying the corresponding haptic feedback event according to the haptic feedback modification instruction.
Specifically, after the haptic feedback instruction corresponding to the audio or video clip is automatically generated, the user may preview the generated haptic feedback effect, that is, an audio or video file carrying the haptic feedback instruction may be played through the player, and after the audio or video file is played, if the user is not satisfied with the haptic feedback effect corresponding to the set haptic feedback event, the user may modify the haptic feedback event. In an embodiment, a user may modify the haptic feedback events through a provided haptic feedback modification interface, for example, the user may modify the strength or sharpness parameter of the haptic feedback corresponding to each haptic feedback event through the haptic feedback modification interface, and after the user completes the modification of the strength or sharpness parameter of the haptic feedback, trigger a haptic feedback modification instruction based on the modified parameter, where the haptic feedback modification instruction carries the modified strength or sharpness parameter, and the mobile terminal may modify the corresponding parameter in the corresponding haptic feedback event according to the strength or sharpness parameter carried in the haptic feedback modification instruction.
According to the embodiment of the invention, when the haptic feedback modification instruction is received, the corresponding haptic feedback event can be modified, so that the parameters contained in the haptic feedback event can be flexibly changed.
Further, referring to fig. 4, in an embodiment, an audio and video playing method is provided, where the audio and video playing is applied to the playing terminal shown in fig. 1, and the method includes:
in step S40, during playing audio or video, it is detected whether the current playing progress reaches a preset position.
In particular, the preset position is a point in time with a haptic feedback event. During the playing of the audio or video, whether the playing progress of the current playing reaches the time point with the tactile feedback event can be detected in real time.
It should be noted that there is at least one preset position in this embodiment.
Step S41, if the current playing progress reaches the preset position, obtaining a haptic feedback event associated with the preset position, where the haptic feedback event includes time information and haptic information of haptic feedback, where the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness.
Step S42, generating corresponding haptic feedback according to the haptic feedback event.
Specifically, when the current playing progress reaches the preset position, a haptic feedback event associated with the preset position is acquired. Since the haptic feedback event includes time information and haptic information of the haptic feedback, wherein the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness, after the haptic feedback event is acquired, the corresponding haptic feedback can be generated according to the data.
In this embodiment, the haptic feedback event is generated based on the loudness of the sound satisfying a preset rule, which is a rule for determining under what circumstances a haptic feedback event needs to be generated. In a specific example, whether to generate the haptic feedback event may be determined by detecting the loudness of the sound contained in the audio or video segment corresponding to the current time point, for example, detecting that the loudness of the sound contained in the audio or video segment is 20db at 0 seconds. After detecting the loudness value of the sound corresponding to 0 second, it is determined whether the loudness value satisfies the preset rule (for example, the loudness value is greater than or equal to 20db), and if so, a haptic feedback event may be generated. Meanwhile, the time point (0 second) may be set as the start time of the haptic feedback included in the generated haptic feedback event.
After detecting that the loudness value of the sound corresponding to 0 second meets a preset rule, continuously detecting the loudness of the sounds corresponding to other time points according to a time sequence, and when detecting that the loudness of the sound corresponding to the current time (for example, 1 second) does not meet the first rule, that is, the loudness value is less than 20db, taking the current time (1 second) as the end time of the generated haptic feedback contained in the haptic feedback event.
After obtaining the start time and the end time of the haptic feedback, determining the strength and/or sharpness of the haptic feedback included in the generated haptic feedback event according to the change of the loudness of the sound in the time period between the start time and the end time and a preset second rule, wherein the preset second rule is a specific value for determining the strength and sharpness of the generated haptic feedback. In this embodiment, the intensity may be positively correlated to the amplitude of the sound, and the sharpness is changed based on a preset value in the haptic feedback command template, for example, the sharpness may be fixed based on the preset value to increase the change value of the loudness of the sound, or the sharpness may be based on the preset value and positively correlated to the change value of the loudness of the sound. In a particular scenario, the sharpness value may be set larger when a large change in loudness is found in the time period between the start time and the end time, and may be set smaller when a small change in loudness is found in this time period. The value of the sharpness value is used for representing the stimulation characteristics brought by vibration, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the touch is brought by the lower the value is. When the maximum value of the loudness in the time period between the start time and the end time is found, the vibration amplitude corresponding to the maximum loudness value may be used as the intensity of the haptic feedback, or the vibration amplitude corresponding to the loudness value is reduced according to a preset rule, so as to obtain the intensity of the haptic feedback.
In one embodiment, after the data contained in the haptic feedback event is acquired, the data needs to be converted into corresponding driving parameters for driving the haptic feedback device, so that the haptic feedback device can perform the corresponding haptic feedback operation according to the driving parameters. In this embodiment, the haptic feedback device may be a Linear Resonant Actuator (LRA), a rotating eccentric mass (ERM), a piezoelectric actuator, an electroactive polymer (EAP), or other actuators.
In this embodiment, when playing an audio or video file, whether the current playing progress reaches the preset position is detected, and when reaching the preset position, a tactile feedback is generated, so that the user can have more immersive sensory experience when playing the audio or video file through the player of the mobile terminal.
Further, in an embodiment, the audio/video playing method further includes:
detecting whether a haptic feedback switch is in an on state, the haptic feedback switch for controlling generation of the haptic feedback.
Specifically, to facilitate control of the haptic feedback, a haptic feedback switch may be provided to control the activation and deactivation of the haptic feedback.
In this embodiment, before generating the haptic feedback, it is required to detect whether the haptic feedback switch is in the on state, and only when the haptic feedback switch is in the on state, the haptic feedback is generated according to the haptic feedback event.
In one embodiment, the user may perform the on and off operations on the haptic feedback switch by touching a haptic feedback switch control displayed in the interface of the mobile terminal, and in another embodiment, the user may also perform the on and off operations on the haptic feedback switch by pressing one of the physical keys in the mobile terminal.
Further, in an embodiment, the audio/video playing method further includes:
and when a haptic feedback intensity adjusting instruction triggered by a user is received, adjusting the intensity of the haptic feedback according to the adjusting instruction.
In particular, a haptic feedback intensity adjustment interface may be provided in a mobile terminal interface for a user to adjust haptic feedback intensity. When the user needs to adjust the intensity of the tactile feedback, the intensity of the tactile feedback can be adjusted through the tactile feedback intensity adjusting interface. After the user completes the adjustment of the strength of the tactile feedback, an adjustment instruction is generated, and the mobile terminal can adjust the strength of the tactile feedback according to the strength adjustment proportion carried in the adjustment instruction or the adjusted strength value.
In this embodiment, the strength of the haptic feedback is adjusted, so that the adjusted haptic feedback can better meet the requirements of the user.
Fig. 5 schematically illustrates a program module diagram of an audiovisual tactile feedback setting apparatus 500 according to the present application, where the audiovisual tactile feedback setting apparatus 500 may be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to implement the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of performing specific functions, and are more suitable for describing the execution process of the video processing system in the storage medium than the program itself.
As shown in fig. 5, the audio-video haptic feedback setting apparatus 500 may include a position obtaining module 501 and a generating module 502, where:
the obtaining module 501 is configured to obtain a haptic feedback setting instruction triggered by a user, where the haptic feedback setting instruction includes an audio or video segment to be set with haptic feedback, and type information of the audio or the video.
When a user needs to set a haptic feedback instruction for audio or video, the user first selects an audio or video segment for which the haptic feedback instruction needs to be set through the haptic feedback setting tool, that is, selects which time period of audio or video the haptic feedback instruction needs to be set for.
There are many ways for a user to select an audio or video segment to be set with a haptic feedback instruction through a haptic feedback setting tool, for example, selecting an audio or video segment to be set with a haptic feedback instruction by using a mouse to frame an audio or video segment for a time period in an interface of a mobile terminal, or selecting an audio or video segment to be set with a haptic feedback instruction by using a touch mode to frame an audio or video segment for a time period in an interface of a mobile terminal.
When the user selects the audio or video segment needing to set the tactile feedback instruction through the tactile feedback setting tool, or after that, the user also needs to select the type information of the audio or video through the tactile feedback setting tool.
After receiving the audio or video clip to be set with haptic feedback and the set audio or video type information selected by the user, the mobile terminal generates a haptic feedback setting instruction according to the setting information by the obtaining module 501.
The tactile feedback is feedback that can reproduce a tactile sensation for a user through a series of actions such as an applied force, vibration, and the like. In this embodiment, the tactile feedback is preferably vibratory feedback.
The type information is information representing the type of the audio or video, for example, the type of the audio or video is "raindrop" type audio or video, "firework" type audio or video, "weather" type audio or video, "wrestling" type audio or video, and the like. The haptic feedback setting tool is pre-deployed with a plurality of different audio or video types for the user to set the type of the audio or video to be set currently.
A generating module 502, configured to generate a haptic feedback instruction corresponding to the audio or video segment according to the type information and sound information contained in the audio or video segment, and store the haptic feedback instruction in association with the audio or video segment, where the haptic feedback instruction includes at least one haptic feedback event, each haptic feedback event includes time information and haptic information of haptic feedback, where the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness.
In particular, the different types of audio or video generated haptic feedback instructions may or may not be the same. The different audio or video generated haptic feedback may or may not be the same. After the audio or video clip to be generated is acquired, the mobile terminal automatically analyzes sound information contained in the audio or video clip, and then generates a corresponding tactile feedback instruction according to the change condition and the type information of the sound contained in the audio or video clip. In this embodiment, the haptic feedback instruction includes at least one haptic feedback event, each haptic feedback event including a start time of the haptic feedback. End time, intensity, and sharpness. Wherein, the intensity is a value of the loudness of the vibration for representing the tactile feedback, the sharpness is a value of the stimulation characteristics brought by the vibration for representing the tactile feedback, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the feeling is brought by the lower the value is.
In an embodiment, in order to generate a haptic feedback instruction corresponding to an audio or video clip according to type information and sound information included in the audio or video clip when a haptic feedback setting instruction is acquired, haptic feedback instruction templates corresponding to various types of audios and videos need to be created in advance, and then the created haptic feedback instruction templates are deployed in a haptic feedback setting tool, so that after the type information is acquired, a haptic feedback instruction template corresponding to a current type of audio and video can be acquired according to a mapping relationship between the type information and the haptic feedback instruction template, and then a haptic feedback instruction can be generated by using the haptic feedback template and the sound information.
In one example, suppose there is a haptic feedback template a corresponding to a "raindrop" type audio or video, with rules for generating haptic feedback commands in the haptic feedback command template a; the firework-like audio or video corresponds to a tactile feedback template B, and the tactile feedback instruction template B also has rules for generating tactile feedback instructions; the 'weather' audio or video corresponds to a tactile feedback template C, and the tactile feedback instruction template C also has rules for generating tactile feedback instructions; the "wrestling" type audio or video corresponds to a haptic feedback template D, which also has rules for generating haptic feedback commands. After obtaining the type information, analyzing the type information, and if the type information is found to be a raindrop type, generating a tactile feedback instruction according to the tactile feedback template a and sound information contained in the audio or video clip; if the type information is found to be the firework type, generating a tactile feedback instruction according to the tactile feedback template B and the sound information contained in the audio or video clip; if the type information is found to be of the "weather" type, a haptic feedback instruction may be generated according to the haptic feedback template C and the sound information included in the audio or video clip; if the type information is found to be a "wrestling" type, a haptic feedback command may be generated from the haptic feedback template D and the sound information included in the audio or video clip. It should be noted that the rules in the haptic feedback command template are used to determine the start time, the end time, and the strength and/or sharpness of the generated haptic feedback command.
In one embodiment, the haptic feedback event further includes information on the type of audio or video clip.
In this embodiment, after the setting of the haptic feedback command is completed, the haptic feedback command needs to be stored in association with the audio or video clip. In one embodiment, when the haptic feedback instruction is saved in association with an audio or video clip, the haptic feedback instruction may be saved to an audio or video file for a corresponding time period. In another embodiment, the haptic feedback command may be saved to a separate file, and then the file is mapped to the audio or video clip.
In the embodiment of the invention, after the haptic feedback setting instruction triggered by the user is acquired, the haptic feedback instruction corresponding to the audio or video clip is generated according to the type information contained in the haptic feedback instruction and the sound information contained in the audio or video clip, and the haptic feedback instruction is stored in association with the audio or video clip so as to enable the audio or video file to carry the haptic feedback instruction. In addition, by applying the haptic feedback technology to playing audio or video, application scenes of the haptic feedback technology can be increased, and thus, the diversity requirements of users can be met.
In an embodiment, after the haptic feedback instruction corresponding to the audio or video clip is generated, an audio or video file containing the audio or video clip needs to be uploaded to a server, so that other terminal devices can acquire the audio or video file from the server to play.
Further, in an embodiment, the generating module 502 is further configured to determine a haptic feedback instruction template according to the type information, where the haptic feedback instruction template includes a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback;
specifically, a plurality of haptic feedback instruction templates are pre-deployed in the haptic instruction setting tool, and the haptic feedback instruction templates corresponding to different types of audio or video are different. Each haptic feedback instruction template contains a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback, i.e. the first rule is used for determining under what circumstances a haptic feedback event needs to be generated, and the second rule is used for determining specific values of the strength and sharpness of the generated haptic feedback.
The generating module 502 is further configured to detect loudness of sounds included in the audio or video segment in a time sequence.
Specifically, the loudness of the sound contained in each time point is sequentially detected according to the time sequence of the audio or video segment, for example, the loudness of the sound contained in the audio or video segment is 20db at 0 second, and the loudness of the sound contained in the audio or video segment is 25db at 0.1 second.
In this embodiment, it is necessary to sequentially detect the loudness of the sound included in each time point in chronological order until the loudness of the sound included in the audio or video at all time points is detected.
The generating module 502 is further configured to generate a haptic feedback event whenever the loudness is detected to satisfy the first rule.
Specifically, when detecting the loudness of the sounds included in the audio or video segment in time sequence, if it is detected that the loudness value of the sound corresponding to the current time point satisfies the first rule, for example, it is detected that the loudness value of the sound corresponding to 0.1 second is greater than or equal to 20db, a haptic feedback event may be generated, and the time point (0.1 second) may be used as the start time of the generated haptic feedback included in the haptic feedback event.
After detecting that the loudness value of the sound corresponding to 0.1 second meets a first rule (for example, the loudness value is greater than or equal to 20db), continuing to detect the loudness of the sounds corresponding to other time points in time sequence, and when detecting that the loudness of the sound corresponding to the current time (for example, 1 second) does not meet the first rule, that is, the loudness value is less than 20db, taking the current time (1 second) as the end time of the generated haptic feedback included in the haptic feedback event.
After obtaining the start time and the end time of the haptic feedback, determining the strength and/or sharpness of the haptic feedback included in the generated haptic feedback event according to the change of the loudness of the sound in the time period between the start time and the end time and the second rule may be continued. In this embodiment, the intensity may be positively correlated to the amplitude of the sound, and the sharpness is changed based on a preset value in the haptic feedback command template, for example, the sharpness may be fixed based on the preset value to increase the change value of the loudness of the sound, or the sharpness may be based on the preset value and positively correlated to the change value of the loudness of the sound. In a particular scenario, the sharpness value may be set larger when a large change in loudness is found in the time period between the start time and the end time, and may be set smaller when a small change in loudness is found in this time period. The value of the sharpness value is used for representing the stimulation characteristics brought by vibration, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the touch is brought by the lower the value is. When the maximum value of the loudness in the time period between the start time and the end time is found, the vibration amplitude corresponding to the maximum loudness value may be used as the intensity of the haptic feedback, or the vibration amplitude corresponding to the loudness value is reduced according to a preset rule, so as to obtain the intensity of the haptic feedback.
In this embodiment, the time period between the start time and the end time is taken as the duration of the haptic feedback.
After generating the haptic feedback event, continuously detecting the loudness of the sounds corresponding to other time points according to the time sequence, when finding the next time period meeting the first rule, generating a second haptic feedback event according to the method, and repeating the process until the detection of the sounds corresponding to all the time points is completed.
In this embodiment, the corresponding feedback instruction template is acquired according to the type information, and then the haptic feedback event is generated according to the acquired feedback instruction template and the loudness information of the sound, so that different types of audio or video can generate different haptic feedback events, the diversity of haptic feedback is improved, the diversity requirement of a user can be met, and the user experience is improved.
Further, in an embodiment, the audio-visual tactile feedback setting apparatus 500 further includes: and modifying the module.
And the modification module is used for modifying the corresponding tactile feedback event according to the tactile feedback modification instruction when receiving the tactile feedback modification instruction.
Specifically, after the haptic feedback instruction corresponding to the audio or video clip is automatically generated, the user may preview the generated haptic feedback effect, that is, an audio or video file carrying the haptic feedback instruction may be played through the player, and after the audio or video file is played, if the user is not satisfied with the haptic feedback effect corresponding to the set haptic feedback event, the user may modify the haptic feedback event. In an embodiment, a user may modify the haptic feedback events through a provided haptic feedback modification interface, for example, the user may modify the strength or sharpness parameter of the haptic feedback corresponding to each haptic feedback event through the haptic feedback modification interface, and after the user completes the modification of the strength or sharpness parameter of the haptic feedback, trigger a haptic feedback modification instruction based on the modified parameter, where the haptic feedback modification instruction carries the modified strength or sharpness parameter, and the mobile terminal may modify the corresponding parameter in the corresponding haptic feedback event according to the strength or sharpness parameter carried in the haptic feedback modification instruction.
According to the embodiment of the invention, when the haptic feedback modification instruction is received, the corresponding haptic feedback event can be modified, so that the parameters contained in the haptic feedback event can be flexibly changed.
Fig. 6 schematically shows a program module diagram of an av playback device 600 according to the present application, where the av playback device 600 may be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to implement the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of performing specific functions, and are more suitable for describing the execution process of the video processing system in the storage medium than the program itself.
As shown in fig. 6, the audio/video playing apparatus 600 may include a detecting module 601, an obtaining module 602, and a generating module 603, where:
the detecting module 601 is configured to detect whether a current playing progress reaches a preset position in a process of playing audio or video.
In particular, the preset position is a point in time with a haptic feedback event. During the playing of the audio or video, whether the playing progress of the current playing reaches the time point with the tactile feedback event can be detected in real time.
It should be noted that there is at least one preset position in this embodiment.
An obtaining module 602, configured to obtain a haptic feedback event associated with the preset position if the current playing progress reaches the preset position, where the haptic feedback event includes time information and haptic information of haptic feedback, where the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness.
A generating module 603 configured to generate corresponding haptic feedback according to the haptic feedback event.
Specifically, when the current playing progress reaches the preset position, a haptic feedback event associated with the preset position is acquired. Since the haptic feedback event includes time information and haptic information of the haptic feedback, wherein the time information includes a start time and an end time, and the haptic information includes at least one of intensity and/or sharpness, after the haptic feedback event is acquired, the corresponding haptic feedback can be generated according to the data.
In this embodiment, the haptic feedback event is generated based on the loudness of the sound satisfying a preset rule, which is a rule for determining under what circumstances a haptic feedback event needs to be generated. In a specific example, whether to generate the haptic feedback event may be determined by detecting the loudness of the sound contained in the audio or video segment corresponding to the current time point, for example, detecting that the loudness of the sound contained in the audio or video segment is 20db at 0 seconds. After detecting the loudness value of the sound corresponding to 0 second, it is determined whether the loudness value satisfies the preset rule (for example, the loudness value is greater than or equal to 20db), and if so, a haptic feedback event may be generated. Meanwhile, the time point (0 second) may be set as the start time of the haptic feedback included in the generated haptic feedback event.
After detecting that the loudness value of the sound corresponding to 0 second meets a preset rule, continuously detecting the loudness of the sounds corresponding to other time points according to a time sequence, and when detecting that the loudness of the sound corresponding to the current time (for example, 1 second) does not meet the first rule, that is, the loudness value is less than 20db, taking the current time (1 second) as the end time of the generated haptic feedback contained in the haptic feedback event.
After obtaining the start time and the end time of the haptic feedback, determining the strength and/or sharpness of the haptic feedback included in the generated haptic feedback event according to the change of the loudness of the sound in the time period between the start time and the end time and a preset second rule, wherein the preset second rule is a specific value for determining the strength and sharpness of the generated haptic feedback. In this embodiment, the intensity may be positively correlated to the amplitude of the sound, and the sharpness is changed based on a preset value in the haptic feedback command template, for example, the sharpness may be fixed based on the preset value to increase the change value of the loudness of the sound, or the sharpness may be based on the preset value and positively correlated to the change value of the loudness of the sound. In a particular scenario, the sharpness value may be set larger when a large change in loudness is found in the time period between the start time and the end time, and may be set smaller when a small change in loudness is found in this time period. The value of the sharpness value is used for representing the stimulation characteristics brought by vibration, the higher the value is, the more sharp and mechanical the feeling is brought, and the softer and organic the touch is brought by the lower the value is. When the maximum value of the loudness in the time period between the start time and the end time is found, the vibration amplitude corresponding to the maximum loudness value may be used as the intensity of the haptic feedback, or the vibration amplitude corresponding to the loudness value is reduced according to a preset rule, so as to obtain the intensity of the haptic feedback.
In one embodiment, after the data contained in the haptic feedback event is acquired, the data needs to be converted into corresponding driving parameters for driving the haptic feedback device, so that the haptic feedback device can perform the corresponding haptic feedback operation according to the driving parameters. In this embodiment, the haptic feedback device may be a Linear Resonant Actuator (LRA), a rotating eccentric mass (ERM), a piezoelectric actuator, an electroactive polymer (EAP), or other actuators.
In this embodiment, when playing an audio or video file, whether the current playing progress reaches the preset position is detected, and when reaching the preset position, a tactile feedback is generated, so that the user can have more immersive sensory experience when playing the audio or video file through the player of the mobile terminal.
Further, in an embodiment, the audio/video playing apparatus 600 further includes: and a switch detection module.
The switch detection module is used for detecting whether a tactile feedback switch is in an on state or not, and the tactile feedback switch is used for controlling the generation of the tactile feedback.
Specifically, to facilitate control of the haptic feedback, a haptic feedback switch may be provided to control the activation and deactivation of the haptic feedback.
In this embodiment, before generating the haptic feedback, it is required to detect whether the haptic feedback switch is in the on state, and only when the haptic feedback switch is in the on state, the haptic feedback is generated according to the haptic feedback event.
In one embodiment, the user may perform the on and off operations on the haptic feedback switch by touching a haptic feedback switch control displayed in the interface of the mobile terminal, and in another embodiment, the user may also perform the on and off operations on the haptic feedback switch by pressing one of the physical keys in the mobile terminal.
Further, in an embodiment, the audio/video playing apparatus 600 further includes: and an adjusting module.
The adjusting module is used for adjusting the strength of the tactile feedback according to the adjusting instruction when receiving the tactile feedback strength adjusting instruction triggered by the user.
In particular, a haptic feedback intensity adjustment interface may be provided in a mobile terminal interface for a user to adjust haptic feedback intensity. When the user needs to adjust the intensity of the tactile feedback, the intensity of the tactile feedback can be adjusted through the tactile feedback intensity adjusting interface. After the user completes the adjustment of the strength of the tactile feedback, an adjustment instruction is generated, and the mobile terminal can adjust the strength of the tactile feedback according to the strength adjustment proportion carried in the adjustment instruction or the adjusted strength value.
In this embodiment, the strength of the haptic feedback is adjusted, so that the adjusted haptic feedback can better meet the requirements of the user.
Fig. 7 schematically shows a hardware architecture diagram of a computer device suitable for implementing the haptic feedback setting method of audio and video according to an embodiment of the present application. In the present embodiment, the computer device 7 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a command set or stored in advance. For example, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. As shown in fig. 7, the computer device 7 includes at least, but is not limited to: memory 710, processor 720, and network interface 730 may be communicatively linked to each other by a system bus. Wherein:
the memory 710 includes at least one type of computer-readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 710 may be an internal storage module of the computer device 7, such as a hard disk or a memory of the computer device. In other embodiments, the memory 710 may also be an external storage device of the computer device 7, such as a plug-in hard disk provided on the computer device 7, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Of course, the memory 710 may also include both internal and external memory modules of the computer device 7. In this embodiment, the memory 710 is generally used to store an operating system installed in the computer device 7 and various types of application software, such as a program code of an audio-video haptic feedback setting method or a program code of an audio-video playing method. In addition, the memory 710 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 720 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 720 is generally used for controlling the overall operation of the computer device 7, such as performing control and processing related to data interaction or communication with the computer device 7. In this embodiment, processor 720 is configured to execute program codes stored in memory 710 or process data.
Network interface 730 may include a wireless network interface or a wired network interface, with network interface 730 typically being used to establish communication links between computer device 7 and other computer devices. For example, the network interface 730 is used to connect the computer device 7 with an external terminal via a network, establish a data transmission channel and a communication link between the computer device 7 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), or Wi-Fi.
It is noted that FIG. 7 only shows a computer device having components 710-730, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the audio/video haptic feedback setting method or the audio/video playing method stored in the memory 710 may be further divided into one or more program modules and executed by one or more processors (in this embodiment, the processor 720) to complete the present invention.
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the haptic feedback setting method or the audio-video playing method of the audio-video in the embodiments.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various types of application software installed in the computer device, for example, a program code of an audio and video haptic feedback setting method or a program code of an audio and video playing method in the embodiment, and the like. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (10)

1. A haptic feedback setting method of an audio/video, characterized by comprising:
acquiring a tactile feedback setting instruction triggered by a user, wherein the tactile feedback setting instruction comprises an audio or video clip to be set with tactile feedback and type information of the audio or the video;
generating a haptic feedback instruction corresponding to the audio or video segment according to the type information and sound information contained in the audio or video segment, and storing the haptic feedback instruction in association with the audio or video segment, wherein the haptic feedback instruction comprises at least one haptic feedback event, each haptic feedback event comprises time information and haptic information of haptic feedback, the time information comprises a start time and an end time, and the haptic information comprises at least one of intensity and sharpness.
2. The audio-video haptic feedback setting method according to claim 1, wherein the generating of the haptic feedback instruction corresponding to the audio or video clip according to the type information and the sound information included in the audio or video clip comprises:
determining a haptic feedback instruction template according to the type information, wherein the haptic feedback instruction template comprises a first rule for generating a haptic feedback event and a second rule for determining the strength and/or sharpness of the generated haptic feedback;
detecting the loudness of sounds contained in the audio or video segment in chronological order;
generating a haptic feedback event whenever it is detected that the loudness satisfies the first rule, wherein a time corresponding to the audio or video segment when the loudness satisfies the first rule is taken as a start time of haptic feedback included in the generated haptic feedback event; taking the moment corresponding to the audio or video segment when the loudness does not meet the first rule as the end time of the generated tactile feedback contained in the tactile feedback event; determining a strength and/or sharpness of haptic feedback included in the generated haptic feedback event according to a change in loudness of a time period between the start time and the end time and the second rule.
3. The audio-visual haptic feedback setting method according to claim 1, further comprising:
and when a haptic feedback modification instruction is received, modifying the corresponding haptic feedback event according to the haptic feedback modification instruction.
4. An audio and video playing method is characterized by comprising the following steps:
detecting whether the current playing progress reaches a preset position or not in the process of playing the audio or the video;
if the current playing progress reaches the preset position, acquiring a tactile feedback event related to the preset position, wherein the tactile feedback event comprises time information and tactile information of tactile feedback, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and sharpness;
generating corresponding haptic feedback according to the haptic feedback event.
5. The audiovisual playback method of claim 4, wherein the haptic feedback event is generated based on a loudness of a sound meeting a preset rule.
6. The audio-video playing method according to claim 4, wherein the audio-video playing method further comprises:
detecting whether a haptic feedback switch is in an on state, the haptic feedback switch for controlling generation of the haptic feedback.
7. The audio-video playing method according to claim 4, wherein the audio-video playing method further comprises:
and when a haptic feedback intensity adjusting instruction triggered by a user is received, adjusting the intensity of the haptic feedback according to the adjusting instruction.
8. An audio-visual tactile feedback setting device, comprising:
the system comprises an acquisition module, a display module and a control module, wherein the acquisition module is used for acquiring a tactile feedback setting instruction triggered by a user, and the tactile feedback setting instruction comprises an audio or video clip to be set with tactile feedback and the type information of the audio or the video;
and the generating module is used for generating a tactile feedback instruction corresponding to the audio or video segment according to the type information and the sound information contained in the audio or video segment, and storing the tactile feedback instruction in association with the audio or video segment, wherein the tactile feedback instruction comprises at least one tactile feedback event, each tactile feedback event comprises tactile feedback time information and tactile information, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and sharpness.
9. An audio-video playback device, comprising:
the detection module is used for detecting whether the current playing progress reaches a preset position or not in the process of playing the audio or the video;
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring a tactile feedback event related to a preset position if the current playing progress reaches the preset position, the tactile feedback event comprises time information and tactile information of tactile feedback, the time information comprises a start time and an end time, and the tactile information comprises at least one of intensity and sharpness;
a generating module for generating corresponding haptic feedback according to the haptic feedback event.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the haptic feedback setting method of audio-video according to any one of claims 1 to 3 or implementing the steps of the audio-video playing method according to any one of claims 4 to 7 when executing the computer program.
CN202010219219.6A 2020-03-25 2020-03-25 Audio and video tactile feedback setting method, audio and video playing method and computer equipment Pending CN113453048A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010219219.6A CN113453048A (en) 2020-03-25 2020-03-25 Audio and video tactile feedback setting method, audio and video playing method and computer equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010219219.6A CN113453048A (en) 2020-03-25 2020-03-25 Audio and video tactile feedback setting method, audio and video playing method and computer equipment

Publications (1)

Publication Number Publication Date
CN113453048A true CN113453048A (en) 2021-09-28

Family

ID=77806856

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010219219.6A Pending CN113453048A (en) 2020-03-25 2020-03-25 Audio and video tactile feedback setting method, audio and video playing method and computer equipment

Country Status (1)

Country Link
CN (1) CN113453048A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024036708A1 (en) * 2022-08-19 2024-02-22 瑞声开泰声学科技(上海)有限公司 Generation method and system for tactile feedback effect, and related device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120315605A1 (en) * 2011-06-08 2012-12-13 Jin-Soo Cho System and method for providing learning information for visually impaired people based on haptic electronic board
CN104182036A (en) * 2013-05-24 2014-12-03 意美森公司 Method and apparatus to provide haptic feedback based on media content and one or more external parameters
CN104735545A (en) * 2015-03-31 2015-06-24 努比亚技术有限公司 Audio/video file playing method and system
CN106255942A (en) * 2014-05-06 2016-12-21 高通股份有限公司 For optimizing the system and method for sense of touch feedback
CN107015639A (en) * 2017-01-24 2017-08-04 瑞声科技(新加坡)有限公司 Dynamic range control method, haptic feedback system and the electric terminal of touch feedback
CN110703903A (en) * 2018-07-09 2020-01-17 意美森公司 System and method for providing automatic haptic generation for video content
CN110874139A (en) * 2018-08-30 2020-03-10 意美森公司 Systems, devices, and methods for streaming haptic effects

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120315605A1 (en) * 2011-06-08 2012-12-13 Jin-Soo Cho System and method for providing learning information for visually impaired people based on haptic electronic board
CN104182036A (en) * 2013-05-24 2014-12-03 意美森公司 Method and apparatus to provide haptic feedback based on media content and one or more external parameters
CN106255942A (en) * 2014-05-06 2016-12-21 高通股份有限公司 For optimizing the system and method for sense of touch feedback
CN104735545A (en) * 2015-03-31 2015-06-24 努比亚技术有限公司 Audio/video file playing method and system
CN107015639A (en) * 2017-01-24 2017-08-04 瑞声科技(新加坡)有限公司 Dynamic range control method, haptic feedback system and the electric terminal of touch feedback
CN110703903A (en) * 2018-07-09 2020-01-17 意美森公司 System and method for providing automatic haptic generation for video content
CN110874139A (en) * 2018-08-30 2020-03-10 意美森公司 Systems, devices, and methods for streaming haptic effects

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024036708A1 (en) * 2022-08-19 2024-02-22 瑞声开泰声学科技(上海)有限公司 Generation method and system for tactile feedback effect, and related device

Similar Documents

Publication Publication Date Title
US9946348B2 (en) Automatic tuning of haptic effects
US9454881B2 (en) Haptic warping system
US10162416B2 (en) Dynamic haptic conversion system
US9891714B2 (en) Audio enhanced simulation of high bandwidth haptic effects
CN104932681B (en) The adjust automatically of haptic effect
CN105141587A (en) Virtual doll interaction method and device
CN107172502B (en) Virtual reality video playing control method and device
WO2020031497A1 (en) Preemptive driving of tactile feedback presentation device
CN113453048A (en) Audio and video tactile feedback setting method, audio and video playing method and computer equipment
CN107133072B (en) Operation execution method and device
CN107133030B (en) Dynamic event execution method and device
US10133408B2 (en) Method, system and computer program product
CN114760274B (en) Voice interaction method, device, equipment and storage medium for online classroom
CN108319444B (en) Music drumbeat-based control terminal vibration method, storage device and computer device
WO2014024255A1 (en) Terminal and video playback program
CN113568548A (en) Animation information processing method and apparatus
CN112399259B (en) Progress bar control method of player
CN111028598A (en) Virtual experiment guidance method and system based on operation behavior information
CN112188290B (en) Pressure relieving method, television, integrated system and storage medium
CN106372846A (en) Method and device for determining state of course
CN116560545A (en) Interaction method, system, device, storage medium and computing equipment
CN111930280A (en) Progress change response method, system and computer storage medium
CN117880602A (en) Video playing, video configuring method, computer device and computer program product
CN117041676A (en) Video clip switching method, device, equipment and storage medium
KR20200038048A (en) Method for providing configuration information log and apparatus thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210928

RJ01 Rejection of invention patent application after publication