CN104932681B - The adjust automatically of haptic effect - Google Patents
The adjust automatically of haptic effect Download PDFInfo
- Publication number
- CN104932681B CN104932681B CN201510238832.1A CN201510238832A CN104932681B CN 104932681 B CN104932681 B CN 104932681B CN 201510238832 A CN201510238832 A CN 201510238832A CN 104932681 B CN104932681 B CN 104932681B
- Authority
- CN
- China
- Prior art keywords
- haptic effect
- filter
- tracking
- automatically
- selectable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B27/038—Cross-faders therefor
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
This application involves the adjust automaticallies of haptic effect.More specifically, it is related to a kind of system for generating haptic effect, receives input medium, which includes audio data, video data or sensing data.The system automatically generates haptic effect tracking according to input medium.At least one filter is applied to the haptic effect automatically generated and tracked by the subsequent system, to generate adjusted haptic effect tracking.
Description
Cross reference to related applications
This application claims the preferential of the temporary patent application sequence number No.61/968,419 submitted on March 21st, 2014
Power, content are herein incorporated by reference.
Technical field
One embodiment more specifically, automatically generates for one kind generally directed to haptic effect and adjusts tactile
The equipment of effect.
Background technique
Haptics is a kind of tactile and force feedback technique, passes through the tactile for applying such as power, vibration and movement to user
The touch perception of feedback effects (i.e. " haptic effect ") Lai Liyong user.Such as mobile device, touch panel device and individual calculus
The equipment of machine can be configurable to generate haptic effect.In general, can in the operating system (" OS ") of equipment programming to can
Generate the calling of the embedded hardware (such as actuator) of haptic effect.These call specified which haptic effect of execution.For example,
When for example button, touch screen, control-rod, control stick, steering wheel or some other control pieces interact user's use with equipment, if
Standby OS can execute order to embedded hardware transmission by control circuit.It is then inserted into formula hardware and generates tactile effect appropriate
Fruit.
Equipment can be configured as coordinating the output of haptic effect and the output of other content, such as game or other matchmakers
Body, so that haptic effect is comprised in other content.For example, in a gaming environment, when developing game, audio developer
It can develop associated with game and indicate the audio of movement occurred in gaming, such as gun shooting, explosion or collision.
In general, in the later period of development of games process, such as when game developer completes the exploitation of game application, or in game
When the game application of completion is transplanted to new platform by developer, haptic effect is added to game.This is often resulted in
The phenomenon that haptic effect is added after having been developed all audios.Because usually just adding haptic effect very late in the process, make
It is usually fallen on the body of haptic effect developer or some other developers about by haptic effect decision associated with audio.
In addition, audio developer, which does not have usually, selects the related input of haptic effect appropriate with for audio.This will cause final packet
Quality containing haptic effect in the content reduces.This quality reduces to become and high quality haptic effect is included in this
Obstruction in appearance.
Summary of the invention
One embodiment is a kind of system for generating haptic effect.The system receives input coal body, which includes
Audio data, video data or sensing data.The system automatically generates haptic effect tracking according to input medium.The system with
At least one filter is applied to the haptic effect automatically generated afterwards to track, to generate adjusted haptic effect tracking.
Detailed description of the invention
Fig. 1 shows the block diagram of system according to an embodiment of the invention.
Fig. 2 is block diagram according to the system in the embodiment of the present invention, and it includes audio, video and any and sensing which, which receives,
Multimedia scene including the relevant data of device (for example, acceleration, rotation, speed etc.), will data relevant to scene it is automatic
Haptic effect is converted to, and then adjust/filter haptic effect.
Fig. 3 is the diagram according to two scenes of fusion (" scene 1 " and " scene 2 ") of one embodiment.
Fig. 4 is shown to be adjusted according to the computer vision (" CV ") implemented by " object of interest " filter of one embodiment
It is whole.
Fig. 5 shows the CV implemented by " background/foreground " filter according to another embodiment of the invention and adjusts.
Fig. 6 shows the CV tune implemented by " proximity with movement " filter according to another embodiment of the invention
It is whole.
Fig. 7 is the process of function of the system of Fig. 1 according to the embodiment in adjust automatically/filtering haptic effect tracking
Figure.
Specific embodiment
One embodiment is a kind of system, in given source data, such as audio or video source data, or and multimedia
It is automatically generated in the case where the relevant any sensing data (such as angular speed, rotation, speed of object etc.) of the content of editing
And adjust/filter one or more haptic effects.Received source data can be automatically converted to tactile data by embodiment, and with
Adjustment or filtering are executed afterwards, consider user preference, positioning (i.e. based on regional preference), output to adjust the output automatically generated
Device parameter etc..Filtering can also be applied to the haptic effect created.Therefore, the quality of haptic effect is improved.
Fig. 1 shows the block diagram of system 10 according to an embodiment of the invention.In one embodiment, system 10 is to set
Standby part, and system 10 provides automatic haptic effect adjustment function for equipment.In another embodiment, system 10 with set
Back-up is from and providing automatic haptic effect adjustment function remotely for equipment.Although shown as triangular web, but the function of system 10
It can be implemented as distributed system.System 10 includes bus 12 or other communication agencies for being used for transmission information, and is coupled to
Bus 12 is to handle the processor 22 of information.Processor 22 can be any kind of general or specialized processor.System 10
It further comprise memory 14, for storing information and instruction to be executed by processor 22.Memory 14 can include random
Access memory (" RAM "), read-only memory (" ROM "), such as disk or CD static storage device or any other
Any combination of the computer-readable medium of type.
Computer-readable medium can be any usable medium, can be accessed by processor 22 and may include volatibility
With non-volatile media, removable and nonremovable medium, communication media and storage medium.Communication media may include such as carrying
Computer readable instructions, data structure, program module or other data in the modulated data signal of wave or other transmission mechanisms,
It and may include any other form of information-delivery media as known in the art.Storage medium may include RAM, flash memory,
ROM, Erasable Programmable Read Only Memory EPROM (" EPROM "), electrically erasable programmable read-only memory (" EEPROM "), deposit
Device, hard disk, removable disk, compact disk read-only memory (" CD-ROM ") or any other form as known in the art
Storage medium.
In one embodiment, memory 14 stores software mould and determines, and provides function when being executed by processor 22.One
In embodiment, module includes operating system 15, and the remainder for system 10 and equipment provides operation system function.The mould
Block further comprises automatic haptic effect adjustment module 16, and adjust automatically haptic effect is such as disclosed more fully below.At certain
In a little embodiments, automatic haptic effect adjustment module 16 can include multiple modules, wherein each individual module provides use
In the specific individual function of adjust automatically haptic effect.For example, automatic haptic effect adjustment module 16 may include modulus of conversion
Block, receive include at least one of audio or sensing data media, and according to media automatically generate haptic effect with
Track.Automatic haptic effect adjustment module 16 can also include filtering module, at least one selectable filter is applied to
The haptic effect tracking automatically generated, to generate adjusted haptic effect tracking.System 10 would generally include one or more
Additional application modules 18, to include additional function, such as " integrator " application of Immersion Corp., according to audio/view
Feel that input automatically generates corresponding haptic effect.
It sending and/or is being received in the embodiment of data from remote source, system 10 further comprises communication equipment 20, such as
Network interface card, to provide mobile wireless network communication, such as infrared, radio, Wi-Fi or cellular network communication.At other
In embodiment, communication equipment 20 provides cable network connection, such as Ethernet connection or modem.
Processor 22 is further coupled to display 24, such as liquid crystal display (" LCD ") via bus 12, for
Family shows graphical representation or user interface.Display 24, which can be, touches sensitizing input equipment, such as touch screen, be configured as from
Processor 22 sends and receives signal, and can be multi-point touch touch screen.Processor 22 can be further coupled to permission
Keyboard or the cursor control 28 that user interacts with system 10, such as mouse or stylus.
In one embodiment, system 10 further comprises actuator 26.Processor 22 can to actuator 26 send with
The associated haptic signal of the haptic effect of generation, and then actuator 26 exports haptic effect, such as the tactile effect of vibrating tactile
Fruit, electrostatic friction haptic effect or deformation haptic effect.Actuator 26 includes actuator driving circuit.Actuator 26 for example may be used
To be motor, electromagnetic actuators, voice coil, marmem, electroactive polymer, electromagnet, eccentric rotating mass motor
(" ERM "), Linear Resonant Actuator (" LRA "), piezoelectric actuator, high bandwidth actuator, electroactive polymer (" EAP ") actuating
Device, electrostatic friction display or ultrasonic vibration generator.In alternative embodiments, other than actuator 26, system 10
It can include one or more additional actuator (not shown in figure 1)s.Actuator 26 is the example of haptic output devices,
In, haptic output devices are configured as the equipment for exporting haptic effect in response to driving signal, such as the tactile of vibrating tactile
Effect, electrostatic friction haptic effect or deformation haptic effect.In alternative embodiments, actuator 26 can by it is some its
The haptic output devices of his type replace.A little outer, in other alternate embodiments, system 10 can not include actuator
26, and the equipment isolated with system 10 includes actuator, or generates other haptic output devices of haptic effect, and be
System 10 sends the haptic effect signal generated to the equipment by communication equipment 20.
System 10 can be coupled to database 30 in further operating, wherein database 30 can be configured as storage by
The data that module 16 and 18 uses.Database 30 can be operational database, analytic type database, data warehouse, distribution
Database, end-user database, external data base, navigation-type database, memory database, Oriented Documents database, in real time
Database, relevant database, object-oriented database or any other database as known in the art.
In one embodiment, system 10 further comprises one or more speakers 32.Processor 22 can be to loudspeaking
Device 32 sends audio signal, and then loudspeaker 32 exports audio.Loudspeaker 32 for example can be dynamic loudspeaker, electro dynamic loudspeaking
Device, piezoelectric speaker, magnetostriction loudspeaker, electrostatic loudspeaker, belt and plane magnetic speaker, are put down at bending wave loudspeaker
Plate loudspeaker, Haier's (heil) pneumatic type loudspeaker, plasma-arc loudspeaker and digital speaker.
In one embodiment, system 10 further comprises sensor 34.Sensor 34 can be configured as detecting one kind
The energy of form or other physical characteristics are such as but not limited to acceleration, bio signal, distance, flow, power/pressure/and answer
Change/bending, humidity, linear position, direction/inclination angle, radio frequency, rotation position, rotation speed, the operation of switch, temperature, vibration or
Visual intensity.Sensor 34 can be configured to the energy that will test or other physical characteristics be converted to electric signal or
Any signal of person's expression virtual-sensor information.Sensor 34 can be any equipment, be such as but not limited to, accelerometer,
Electrocardiograph, electroencephalograph, electromyograph(EMG, electroculogram instrument, palategraph, electrodermal reaction sensor, capacitance sensor, suddenly
Ear effect sensor, infrared sensor, ultrasonic sensor, pressure sensor, fibre optical sensor, curvature sensor (or bending
Sensor), force-sensing sensor, dynamometer (load cell), LuSense CPS2155, miniature pressure cell, piezoelectric sensing
Device, strain gauge, hygrometer, linear position touch sensor, linear potentiometer (or slider), linear variable differential transformer, compass,
Inclinometer, magnetic labels (or RFID tag), rotary encoder, rotating potentiometer, gyroscope, on-off switch, temperature pass
Sensor (such as thermometer, thermocouple, resistance temperature meter, thermistor, temperature transition integrated circuit), microphone, photometer, height
Degree meter, biological monitor or photo resistance.
In general, generating oneself of haptic effect according to the sensor multi-medium data including audio and/or video data
Dynamic tactile transfer algorithm lacks the quality human art touch that the haptic effect of creation can be improved by adjustment and fine tuning
(touch).Some this expertises are introduced into automatically using various filtering/adjustment technologies by embodiment in the conceived case
In conversion work stream.
The part that embodiment can be used as automatic conversion process executes adjustment, or can be used as user and can automatically generate
The optional filter selected after haptic effect.In one embodiment, filter is tracked applied to haptic effect, for example is had
Sample value -1 arrives+1 pulse code modulation (" PCM ") signal.In one embodiment, disclosed more fully below to use filtering
Device includes " aggravating collision ", " aggravating last ", " scene fusion ", " dynamic range compression/extension ", " actuator capabilities ", " sense is emerging
Interesting object ", " background/foreground " and " proximity with movement " filter.In one embodiment, haptic effect is being automatically generated
When, some or all available filters can be implemented simultaneously, so that the output of filter may be selected and then " immediately " can
With.
Fig. 2 is block diagram according to the system in the embodiment of the present invention, receive multimedia scene, including audio, video and appoint
Data relevant to scene are automatically converted to touching by what data (for example, acceleration, rotation, speed etc.) relevant to sensor
Feel effect, and then adjust/filter haptic effect.It, will be from multi-medium data using known system and method in Fig. 2
Audio, video and/or the sensor signal of each scene 1-n is automatically converted to haptic effect in 201-203.For automatic conversion
A kind of known system and method disclosed in U.S. Patent application No.13/785,166, the disclosure of which by reference simultaneously
Enter herein.In general, automatic conversion algorithm analyte sensors data or voiceband data signal, and identify some features, for example wrap
Network, frequency composition, peak value, noise etc., and then it is based on these feature-modeling haptic trackings.
210, one or more filters are applied to the haptic effect of automatic conversion.Filter can be pre-selection simultaneously certainly
Dynamic application, or the haptic tracking generated can be applied in any time after automatic conversion.For example, available filter
List can be presented to the user in the drop-down menu of user interface, and user can select one or more filters to answer
With.When having selected multiple filters, they can use pipelining and be applied with any selection or scheduled sequence.
Filter or multiple filters generate final haptic effect tracking 212 after application.
In one embodiment, by multiple filters, filter can with such as filter 1, then
The order of filter 2 etc. is applied, wherein filter 1 obtains original haptic tracking as input, and export new tactile with
Track, the new haptic tracking are fed as input to filter 2 etc., until in the final haptic effect tracking of 212 outputs.Using
Sequence/order of filter can change the input that result obtains.For example, if being sequentially filter in one embodiment
1, filter 2 and filter 3, and in another embodiment, it is sequentially filter 2, filter 1 and filter 3, then final touching
Feel that effect tracking can be different, even if applying identical three filters in order in two examples.
It in one embodiment, can be with Parallel application filter by multiple filters, wherein filtering
Device 1, filter 2 etc., all original haptic tracking having the same is as input, and the output of each filter is then
It is fed as input to summation filter, the summation filter is in the final haptic effect tracking of 212 outputs.Summing filter can be with
Including weighted sum scheme, wherein the weight of (i.e. predetermined value) each input can be specified or specified automatically by user.?
In another embodiment, all weights are all equal (that is, no user intervention ground is automatic specified).For example, if there is Parallel application
Five filters, each output tracking will summation filter in multiplied by 0.2 (=1/5) weight.
In addition, in one embodiment, by multiple filters, it is possible to implement " mixing " mode, wherein
Some filters can be applied in order, and can be by filter Parallel application in the output of multiple sequences.For example, filter 1,
Filter 3 and filter 4 can be in order using with formation sequences 1.Filter 2, filter 3 and filter 5 can be by another
A sequence is applied with formation sequence 2.The output of sequence 1 and sequence 2 can then be fed to summation filter, in 212 outputs
Final haptic effect tracking.
In embodiment, any Parallel application of any sequence or filter of any filter or filter is defeated
Enter can be the haptic tracking of original conversion, the output of any filter, filter any sequence output or filtering
One in the output of any Parallel application of device.
By multiple filters, the selection of filter and application model (that is, parallel or in order) are based on
Design style/hobby of user, and can for media clip part and change.For example, in first five minute, filter 1
It can be applied in order to 5, and in next two minutes, filter 2 and 3 can be with Parallel application etc..Each filter
It can be applied (such as advanced level user's mode) according to style/hobby of user with different parameters, or be come with default parameters
Using (i.e. no user intervention).
In one embodiment, an available filter is " aggravate collision " filter, by it is following come to again
It wants or gives more attention in relation to the relevant very brief effect (" colliding ") sharply but strongly of event, that is, reduce in the effect
The amplitude of effect and effect immediately after before, sometimes by the short decaying into carryover effect.It is touched to give
More attention are hit, need to feel collision different from back to back in preceding effect.And then embodiment, which reduces, " collides "
The intensity/amplitude in preceding effect.Embodiment is greater than specific threshold or " collision threshold for example, by finding the absolute value of haptic tracking
The sample of value " (such as 0.9), to find the collision in haptic tracking.Once finding " collision ", embodiment just checks haptic tracking
Time of the expression before collision " searching window " (such as 500ms) sample.In these samples, it is small to find absolute value
In specific threshold or " exclude threshold value " (such as 0.3) and covering be greater than specific threshold or " excluding duration threshold " (such as
The block of the connected sample of time 50ms).Then make these samples invalid (i.e. range value is reduced to 0).
In one embodiment, an available filter is " aggravating last " filter.In general, in editing/tracking
Last haptic effect should be usually it is strong, to mark the end of editing, thus answer automatic weight increasing it (such as compared with high-amplitude
Degree).Therefore, filter " is aggravated last " to find the net effect in haptic tracking and amplify its amplitude.This can be by will most
The sample value of effect is realized multiplied by the gain factor for being greater than 1 (such as 1.5) afterwards.Embodiment finds in haptic tracking indicate first
The sample (i.e. the last sample of net effect) of the end of net effect.For this purpose, finding absolute value in tracking is greater than specific threshold
Or the last sample of " silent threshold value " (such as 0.1).Embodiment, which then passes through, considers owning before the end of net effect
Sample, and find the last block of " tactile silent " in these samples, to find beginning (i.e. the of net effect of net effect
One sample).Net effect is initially the first sample followed after this last " tactile is silent "." tactile is silent " is exhausted
Is less than by specific threshold or " silent threshold value " (such as 0.1) and is covered for value and is greater than specific threshold or " silence durations threshold value "
The connected tactile eye track sample of the time of (such as 100ms) is determined.
In one embodiment, an available filter is " scene fusion " filter, is suitable for that user is made to experience matchmaker
Transition in body editing between a scene and next scene.Embodiment " will fade out " strategy applied in preceding scene tactile
It tracks and/or strategy " will be faded in " and be applied to current scene haptic tracking.Fig. 3 is two scenes of fusion according to one embodiment
The diagram of (" scene 1 " and " scene 2 ").Embodiment be come by distinguish related (in each scene) haptic effect can
It aggravates or prominent in the transition in editing between different scenes.Embodiment implements " fading out " and " fades in " scheme then to carry out
Filter.
In one embodiment, the function is only just applied in following scene:
● in the final time window in first scene/tracking or in " window is searched in fusion " (such as 500ms)
Haptic effect and the haptic effect in first time window in subsequent scene/tracking or in " merge and search window " are all strong
Strong (the tactile sample i.e. in each with the maximum value greater than such as 0.5)；Or
● if two effects are all that slight effect (has the touching of the maximum value less than such as 0.5 i.e. in each
Feel sample) but with similar intensity (i.e. two lookup window each in maximum value differ only by such as 0.1).
In addition, last " search window " (" window of the filter applied to the haptic effect of first scene that will fade out as follows
1 "), and first " search window " (" window 2 ") of the filter applied to the haptic effect of subsequent scene will be faded in:
● embodiment finds the maximum value in the first half of window 1, and constructs signal " signal 1 " identical with window 1, with
Modify it as follows afterwards: the sample for arriving maximum value position from the beginning will all have value=1；So that subsequent sample
(from maximum sample until signal 1 ends up) value is reduced to the mode of 0 (in the ending of signal 1) linearly from the maximum value of identification to count
Calculate these samples.1 sample of window is then multiplied by 1 sample of signal (element is multiplied by element), and result replaces window 1 to be added to original
In beginning haptic tracking.
● embodiment then finds the maximum value in window 2, and constructs signal " signal 2 " identical with window 2, then with
As under type modifies it: from maximum value position until the sample that signal 2 ends up all has value=1；So that preceding sample
The mode that (starting from signal 2 until this maximum sample) value linearly increases from 0 (in the beginning of signal 2) to the maximum value of identification
To calculate these samples.2 sample of window is then multiplied by 2 sample of signal (element is multiplied by element), and result is added instead of window 2
Into original haptic tracking.
In one embodiment, an available filter is " dynamic range compression/extension " filter, is adapted so that
Weak effect is weaker and/or makes potent fruit stronger.Embodiment improves the amplitude of powerful effect, and reduces the width of effect by a small margin
Degree.Tactile sample value is transformed to logarithmic scale (in terms of decibel dB) first by filter.Value is greater than specific threshold " thr1 " (example
As -30dB) tactile sample can have the following new value calculated using yield value " gain " (such as 1.5):
New_value=((old_value-thr1) * gain)+thr1
Institute tactile sample of the value less than specific threshold " thr2 " (such as -50dB) can have uses pad value as follows
The new value that " att " (such as 2) calculate:
New_value=thr2- ((thr2-old_value) * att)
In one embodiment, an available filter is " actuator capabilities " filter, what filtering automatically generated
Haptic tracking, to adapt to the ability of the actuator in expected playback apparatus.In general, the equipment of the actuator according to playback apparatus and
Quality, changed by making partly or entirely preliminary haptic effect extend or shorten specific, predetermined values (such as 10ms) tactile with
Track.For example, for the haptic effect based on quality, adjustment may include changing the parameter of delimiting period value (i.e. effect being implemented
The length of time and unfruitful time).For example, the haptic effect of quality may include 10ms for the duration of effect
Effect and 10ms are silent.In one example, on " Galaxy S4 " smart phone of Samsung Corp. based on quality
The periodic quantity of haptic effect is 10ms, and on " G2 " smart phone of LG Corp., effect becomes the periodic quantity with 25ms
More preferably, although the two all has the LRA as haptic output devices.To this filter, a required input is actuator
Or the ability of other haptic output devices of playback apparatus, the position of type, haptic output devices including haptic output devices
It sets.This can provide manually or read automatically from the pre-stored value in look-up table.
In other embodiments, some filters can be applied to the haptic effect that adjustment generates, to add in tactile
The dependent event (i.e. " object of interest ") of special object in media clip is focused on, or otherwise reduces the important of these events
Property.This object can be selected in one embodiment by user/designer, or automatic by system in another embodiment
Detection.Another embodiment can be with adjustment effect in the case where given movement and the proximity of video camera.In general, these realities
Example is applied for improvement " being detached from context " problem, the meeting when automatically generating haptic effect using sensing data sometimes of this problem
Occur.More specifically, amplitude/intensity by effect can be clearly observed this problem.For example, when use is living from BMX
When the data captured in dynamic cycling obtain haptic effect, the haptic effect automatically generated is usually approached in bicycle and is taken the photograph
The amplitude having the same when camera and when it is far from video camera.As a result, haptic effect feels less true, because they do not have
Reflect close concept.
In addition, haptic effect relevant to object/movement exist in video with video camera apart from closer another person/
Viewer can be generated in the case that object is mobile and obscured.This, which is obscured, arises from the fact that viewer perceives effect
To be related to " movement " of generation (closest with video camera) in the foreground, and actually they may be obtained from background (with take the photograph
Camera is farthest) in " movement ".Solution in this concrete condition is in the conceived case (i.e. if can get sensing
Device data) haptic effect that is shown from " foreground action " is reduced by the effect for being obtained from " background movement " if not all right
Amplitude.
Since " object of interest ", " background/foreground " and " proximity with movement " filter are based on video input.In reality
It applies and has used computer vision scheme in example.
Fig. 4 is shown to be adjusted according to the computer vision (" CV ") implemented by " object of interest " filter of one embodiment
It is whole.In Fig. 4,401, user by object/people for starting in editing (i.e. user/designer think interested object or
People) two-dimensional representation on click (it is assumed that obtaining sensing data 405 for the object) to select tactile associated therewith
Object/people.The object in video can be tracked in 402, CV algorithm, and estimates itself and video camera in every N number of frame 403
Distance.It is then modulated to (in N number of respective frame) on the estimated distance 407 406 for the haptic effect of the Object Creation.Example
Such as, when object is close to video camera, effect can be made stronger (i.e. by sample value multiplied by the gain factor for being greater than 1), Huo Zhe
When object is far from video camera, so that effect is weaker (i.e. by sample value multiplied by the reduction factor less than 1).Gain/reduction factor
Value can linearly change with the estimated distance with video camera.
The tracking of object is carried out by tracking visual signature relevant to the object in each frame and estimating its movement.
The visual signature being able to use includes but is not limited to: color, Scale invariant characteristic conversion (" SIFT "), accelerates steadily and surely shape
Feature (" SURF "), histograms of oriented gradients (" HOG ") descriptor, Lis Hartel sign, Corner Detection (Shi-Tomasi) feature and
Wavelet transformation.Optical flow computation technology, Kalman filtering or particle filter scheme are able to use to execute tracking.
Fig. 5 shows the CV in accordance with another embodiment of the present invention implemented by " background/foreground " filter and adjusts.Scheming
In 5, the object in each scene (N number of frame) is automatically divided into foreground and background action object/thing in 502, CV algorithm
Part (that is, if object in the foreground, but does not move, does not just consider it).Another algorithm comes in 503 use pattern matching process
These objects are identified, and 505, if sensing data relevant to object in the foreground is identified as is available (with object class
Phenotypic marker), then haptic tracking is generated using the sensing data.If it is not, just using the sensor number of background object
According to creating tactile.It can also dynamically application background/prospect filtering.In entire media clip, object/people can change position
It sets, therefore object/people in the foreground can be different in different moments.Embodiment can be adapted to this constraint, and correspondingly change
Become haptic effect.Object/people based on identification the object/people identified can also be adjusted at a distance from video camera as described above
The haptic effect of creation.The distance can be calculated by tracking/identification step.
By the feature of object in detection image and then they are classified relative to the feature of well-known object to carry out pair
Matching process/property identification of elephant.The feature being able to use includes but is not limited to: color, shape, Scale invariant characteristic conversion
(" SIFT "), accelerate robust feature (" SURF "), histograms of oriented gradients (" HOG ") descriptor, Lis Hartel sign, Corner Detection
(Shi-Tomasi) feature, wavelet transformation.Neural network, support vector machine (" SVM "), adaptive enhancing can be used
(Adaboost) classifier, cascade classifier, K arest neighbors (" KNN ") and Bayesian network come into classification.
Fig. 6 shows the CV tune implemented by " proximity with movement " filter according to another embodiment of the invention
It is whole.In Fig. 6, in the ratio for the pixel that the estimation of 601, CV algorithm fast moves in each scene (N frame).If this intermediate value
Height, then video camera is just with " movement " in the scene close to (such as viewpoint (" POV ") video camera), and the tactile therefore generated
Effect should just have by a relatively large margin.Otherwise, haptic effect should just be felt weaker.One embodiment is estimated using optical flow computation
This proximity.
Fig. 7 is according to the embodiment in adjust automatically/filtering haptic effect tracking, the stream of the function of the system 10 of Fig. 1
Cheng Tu.In one embodiment, the function of the flow chart of Fig. 7 is situated between by being stored in memory or other computer-readable or entities
The software implementation executed in matter and by processor.In other embodiments, which can be by hardware (such as by using dedicated collection
At circuit (" ASIC "), programmable gate array (" PGA "), field programmable gate array (" FPGA ") etc.) or hardware and software
Any combination executes.
702, haptic effect is automatically generated in response to the input medium received, the input medium received can include
Audio data or video data, and may include the sensing data for objects one or more in scene.The result is that automatic
Haptic effect tracking, may include multiple editings or scene.
704, receives one kind of the haptic effect tracking for automatically generating or multiclass adjusts or the selection of filtering.One
In a embodiment, available filter includes " aggravating collision ", " aggravating last ", " scene fusion ", " dynamic range compression/expansion
Exhibition ", " actuator capabilities ", " object of interest ", " background/foreground " and " proximity with movement " filter.
706, tracked based in 704 selection or multiple selections to filter the haptic effect automatically generated, and generate most
Whole filtered haptic effect tracking.
As disclosed in, haptic effect tracking is automatically generated.Adjust the selection of preference based on one or more then to adjust
Or filter the tracking of this benchmark haptic effect.As a result the haptic effect obtained is thus adapted to the preference of user.
It should be noted that present disclosure can also be used with flowering structure.
(1) computer-readable medium, has the instruction that is stored thereon, described instruction when executed by the processor so that
Processor generates haptic effect, generates haptic effect and includes:
Receive input medium, the input medium include in audio data, video data or sensing data at least it
One；
Haptic effect tracking is automatically generated according to the input medium；And
By at least one filter be applied to automatically generate haptic effect tracking, with generate adjusted haptic effect with
Track.
(2) computer-readable medium according to (1), wherein the filter includes the tactile for aggravating to automatically generate
Collision in effect tracking.
(3) computer-readable medium according to (1), wherein the filter includes the tactile for aggravating to automatically generate
Net effect in one or more editings of effect tracking.
(4) computer-readable medium according to (1), wherein the filter includes the touching modified automatically generating
Feel the haptic effect between the scene of effect tracking.
(5) computer-readable medium according to (1), wherein the filter includes modification haptic effect with protrusion
Transition between the scene of the haptic effect tracking automatically generated.
(6) computer-readable medium according to (1), wherein the filter include modification haptic effect so that
The strong tactile weaker, or that the haptic effect automatically generated is tracked of the weak haptic effect of the haptic effect tracking automatically generated
Effect is stronger.
(7) computer-readable medium according to (1), wherein the filter includes playing back in response to haptic effect
The tactile fan-out capability of equipment modifies the haptic effect of the haptic effect tracking automatically generated.
(8) according to the computer-readable medium of (1), wherein the input medium includes video data, and the filtering
Device includes the haptic effect for the haptic effect tracking that modification automatically generates to aggravate the object or event of selection.
(9) computer-readable medium according to (1), wherein the input medium includes video data, and described
Filter include modification automatically generate haptic effect tracking haptic effect with aggravate background or foreground object or background or
Foreground event.
(10) computer-readable medium according to (1), wherein the input medium includes video data, and institute
Stating filter includes proximity of the haptic effect for the haptic effect tracking that modification automatically generates to aggravate with act in media.
(11) computer-readable medium according to (1) further comprises:
There is provided the selectable option that be applied to the filter of the haptic effect automatically generated tracking, wherein can be with can
The filter of the sequence application selection of selection.
Specifically illustrate and/or illustrate herein several embodiments.But it can be appreciated that, above teaching covers disclosed embodiment
Modifications and variations, in the case where not departing from spirit of the invention or desired extent, their models in the appended claims
In enclosing.
Claims (26)
1. a kind of method for generating haptic effect, comprising:
Input medium is received, the input medium includes at least one of audio data, video data or sensing data；
Haptic effect tracking is automatically generated according to the input medium；And
Selectable filter is applied to the haptic effect automatically generated tracking, by using from selectable filter
The first selectable filter output the first adjusted haptic effect tracking and from the in selectable filter
Second adjusted haptic effect tracking of two selectable filter outputs is as the input to summation filter, to generate warp
The haptic effect of adjustment tracks, and the summation filter includes that the weight of wherein each input can be specified by user or be specified automatically
Weighted sum scheme,
At least one the haptic effect parameter for the haptic effect tracking that wherein each selectable filter modification automatically generates.
2. according to the method described in claim 1, wherein, the filter includes in the haptic effect tracking for aggravating to automatically generate
Collision.
3. according to the method described in claim 1, wherein, the filter includes that the haptic effect for aggravating to automatically generate tracks
Net effect in one or more editings.
4. according to the method described in claim 1, wherein, the filter includes the haptic effect tracking modified automatically generating
Scene between haptic effect.
5. according to the method described in claim 1, wherein, the filter includes that modification haptic effect is being automatically generated with protrusion
Haptic effect tracking scene between transition.
6. according to the method described in claim 1, wherein, the filter includes modification haptic effect so that automatically generating
The weak haptic effect of haptic effect tracking is weaker, or the strong haptic effect that the haptic effect automatically generated is tracked is stronger.
7. according to the method described in claim 1, wherein, the filter includes the tactile in response to haptic effect playback apparatus
Fan-out capability modifies the haptic effect of the haptic effect tracking automatically generated.
8. according to the method described in claim 1, wherein, the input medium includes video data, and the filter packet
The haptic effect for the haptic effect tracking that modification automatically generates is included, to aggravate the object or event of selection.
9. according to the method described in claim 1, wherein, the input medium includes video data, and the filter packet
The haptic effect for the haptic effect tracking that modification automatically generates is included, to aggravate background or foreground object or background or prospect thing
Part.
10. according to the method described in claim 1, wherein, the input medium includes video data, and the filter packet
The haptic effect for the haptic effect tracking that modification automatically generates is included, to aggravate the proximity in the input medium with movement.
11. according to the method described in claim 1, further comprising:
There is provided the selectable option that be applied to the filter of the haptic effect automatically generated tracking, wherein can be may be selected
Sequence application selection filter.
12. according to the method described in claim 1, further comprising the haptic effect for being applied to automatically generate by multiple filters
Tracking.
13. according to the method for claim 12, wherein the sequence of the multiple filter of application is based on predesigned order or pre-
Determine priority.
14. according to the method for claim 12, wherein in order or the multiple filter of Parallel application.
15. a kind of haptic effect generates system, comprising:
Conversion module, the conversion module receive input medium and according to the input medium automatically generate haptic effect with
Track, the input medium include at least one of audio data, video data or sensing data；And
Filtering module, selectable filter is applied to the haptic effect automatically generated and tracked by the filtering module, to pass through
Using the exported from the first selectable filter in selectable filter first adjusted haptic effect tracking and
The adjusted haptic effect tracking of second exported from the second selectable filter in selectable filter is as to asking
With the input of filter, to generate adjusted haptic effect tracking, the summation filter includes the power of wherein each input
Beijing South Maxpower Technology Co. Ltd is specified by user or automatically specified weighted sum scheme；
At least one the haptic effect parameter for the haptic effect tracking that wherein each selectable filter modification automatically generates.
16. a kind of computer-readable medium, there is the instruction being stored thereon, described instruction when executed by the processor so that
Processor generates haptic effect, generates haptic effect and includes:
Input medium is received, the input medium includes at least one of audio data, video data or sensing data；
Haptic effect tracking is automatically generated according to the input medium；And
Selectable filter is applied to the haptic effect automatically generated tracking, by using from selectable filter
The first selectable filter output the first adjusted haptic effect tracking and from the in selectable filter
Second adjusted haptic effect tracking of two selectable filter outputs is as the input to summation filter, to generate warp
The haptic effect of adjustment tracks, and the summation filter includes that the weight of wherein each input can be specified by user or be specified automatically
Weighted sum scheme；
At least one the haptic effect parameter for the haptic effect tracking that wherein each selectable filter modification automatically generates.
17. computer-readable medium according to claim 16, wherein the filter includes the touching for aggravating to automatically generate
Feel the collision in effect tracking.
18. computer-readable medium according to claim 16, wherein the filter includes the touching for aggravating to automatically generate
Feel the net effect in one or more editings of effect tracking.
19. computer-readable medium according to claim 16, wherein the filter includes modifying automatically generating
Haptic effect between the scene of haptic effect tracking.
20. computer-readable medium according to claim 16, wherein the filter includes modification haptic effect with prominent
Transition between the scene of the haptic effect tracking automatically generated out.
21. computer-readable medium according to claim 16, wherein the filter include modification haptic effect so that
The strong touching weaker, or that the haptic effect automatically generated is tracked of the weak haptic effect for the haptic effect tracking that must be automatically generated
Feel that effect is stronger.
22. computer-readable medium according to claim 16, wherein the filter includes returning in response to haptic effect
The tactile fan-out capability for putting equipment modifies the haptic effect of the haptic effect tracking automatically generated.
23. computer-readable medium according to claim 16, wherein the input medium includes video data, and
The filter includes the haptic effect for the haptic effect tracking that modification automatically generates to aggravate the object or event of selection.
24. computer-readable medium according to claim 16, wherein the input medium includes video data, and
The filter includes the haptic effect for the haptic effect tracking that modification automatically generates to aggravate background or foreground object or back
Scape or foreground event.
25. computer-readable medium according to claim 16, wherein the input medium includes video data, and
The filter includes proximity of the haptic effect for the haptic effect tracking that modification automatically generates to aggravate with act in media.
26. computer-readable medium according to claim 16, further comprises:
There is provided the selectable option that be applied to the filter of the haptic effect automatically generated tracking, wherein can be may be selected
Sequence application selection filter.
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201461968419P | 2014-03-21 | 2014-03-21 | |
| US61/968,419 | 2014-03-21 | ||
| US14/661,396 | 2015-03-18 | ||
| US14/661,396 US9946348B2 (en) | 2014-03-21 | 2015-03-18 | Automatic tuning of haptic effects |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104932681A CN104932681A (en) | 2015-09-23 |
| CN104932681B true CN104932681B (en) | 2019-09-20 |
Family
ID=52807576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510238832.1A Expired - Fee Related CN104932681B (en) | 2014-03-21 | 2015-03-20 | The adjust automatically of haptic effect |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104932681B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102521953B1 (en) | 2014-09-02 | 2023-04-14 | 애플 인크. | Semantic framework for variable haptic output |
| US9858036B2 (en) | 2015-11-10 | 2018-01-02 | Google Llc | Automatic audio level adjustment during media item presentation |
| DK201670737A1 (en) | 2016-06-12 | 2018-01-22 | Apple Inc | Devices, Methods, and Graphical User Interfaces for Providing Haptic Feedback |
| DK179823B1 (en) | 2016-06-12 | 2019-07-12 | Apple Inc. | Devices, methods, and graphical user interfaces for providing haptic feedback |
| DK201670720A1 (en) | 2016-09-06 | 2018-03-26 | Apple Inc | Devices, Methods, and Graphical User Interfaces for Generating Tactile Outputs |
| EP3531250B1 (en) * | 2016-09-06 | 2021-02-24 | Apple Inc. | Devices, methods, and graphical user interfaces for haptic mixing |
| US10600290B2 (en) * | 2016-12-14 | 2020-03-24 | Immersion Corporation | Automatic haptic generation based on visual odometry |
| US10147460B2 (en) * | 2016-12-28 | 2018-12-04 | Immersion Corporation | Haptic effect generation for space-dependent content |
| DK201770372A1 (en) | 2017-05-16 | 2019-01-08 | Apple Inc. | Tactile feedback for locked device user interfaces |
| CN109491502B (en) * | 2018-11-07 | 2021-10-12 | Oppo广东移动通信有限公司 | Haptic rendering method, terminal device and computer-readable storage medium |
| EP3748474B1 (en) * | 2019-06-06 | 2022-09-28 | Goodix Technology (HK) Company Limited | Audio-haptic processor, method, system and computer program product |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103257783A (en) * | 2012-02-15 | 2013-08-21 | 伊默逊股份有限公司 | Interactivity model for shared feedback on mobile devices |
| EP2703951A2 (en) * | 2012-08-31 | 2014-03-05 | Immersion Corporation | Sound to haptic effect conversion system using mapping |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9019087B2 (en) * | 2007-10-16 | 2015-04-28 | Immersion Corporation | Synchronization of haptic effect data in a media stream |
| KR101328054B1 (en) * | 2011-08-09 | 2013-11-08 | 엘지전자 주식회사 | Apparatus and method for generating sensory vibration |
-
2015
- 2015-03-20 CN CN201510238832.1A patent/CN104932681B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103257783A (en) * | 2012-02-15 | 2013-08-21 | 伊默逊股份有限公司 | Interactivity model for shared feedback on mobile devices |
| EP2703951A2 (en) * | 2012-08-31 | 2014-03-05 | Immersion Corporation | Sound to haptic effect conversion system using mapping |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104932681A (en) | 2015-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104932681B (en) | The adjust automatically of haptic effect | |
| US20180196520A1 (en) | Automatic tuning of haptic effects | |
| JP6595686B2 (en) | Automatic adaptation of haptic effects | |
| CN104423593B (en) | The system and method for generating haptic effect associated with the transition in audio signal | |
| US9245429B2 (en) | Haptic warping system | |
| CN105388995A (en) | Systems and Methods for Shape Input and Output for a Haptically-Enabled Deformable Surface | |
| KR20150028737A (en) | Haptic conversion system using frequency shifting | |
| US20200218356A1 (en) | Systems and methods for providing dynamic haptic playback for an augmented or virtual reality environments | |
| CN108021228A (en) | Dynamic haptic based on the Video Events detected produces | |
| CN103513767A (en) | Haptic feedback control system | |
| KR20150028733A (en) | Automatic remote sensing and haptic conversion system | |
| CN109375767A (en) | System and method for generating haptic effect | |
| CN105556423A (en) | Systems and methods for pressure-based haptic effects | |
| KR20200050869A (en) | Systems and methods for providing customizable haptic playback | |
| CN104793737A (en) | Systems and methods for content authoring | |
| CN104423707A (en) | Haptic conversion system using segmenting and combining | |
| US11328469B2 (en) | Electronic device and method for providing drawing environment | |
| CN105955494A (en) | Game operation processing method, apparatus and device | |
| CN104869507A (en) | Music playing method applied to intelligent sound box, and intelligent sound box | |
| CN114154068A (en) | Media content recommendation method and device, electronic equipment and storage medium | |
| CN107106908A (en) | Made an amendment haptic effect for slow motion | |
| CN106060707A (en) | Reverberation processing method and device | |
| KR20180111242A (en) | Electronic device and method for providing colorable content | |
| US20200192480A1 (en) | Systems and methods for providing haptic effects based on a user's motion or environment | |
| CN105988602A (en) | Mouse simulating method and mouse simulating device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190920 |