CN110399040A - Multi-modal exchange method, ustomer premises access equipment, server and system - Google Patents

Multi-modal exchange method, ustomer premises access equipment, server and system Download PDF

Info

Publication number
CN110399040A
CN110399040A CN201910667966.3A CN201910667966A CN110399040A CN 110399040 A CN110399040 A CN 110399040A CN 201910667966 A CN201910667966 A CN 201910667966A CN 110399040 A CN110399040 A CN 110399040A
Authority
CN
China
Prior art keywords
interaction
packet
interactive
mode
interactive command
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910667966.3A
Other languages
Chinese (zh)
Other versions
CN110399040B (en
Inventor
徐伟刚
陈熙旻
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yutou Technology Hangzhou Co Ltd
Original Assignee
Yutou Technology Hangzhou Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yutou Technology Hangzhou Co Ltd filed Critical Yutou Technology Hangzhou Co Ltd
Priority to CN201910667966.3A priority Critical patent/CN110399040B/en
Publication of CN110399040A publication Critical patent/CN110399040A/en
Application granted granted Critical
Publication of CN110399040B publication Critical patent/CN110399040B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/445—Program loading or initiating
    • G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The present invention relates to a kind of multi-modal exchange method, ustomer premises access equipment, server and systems to be sent to server-side for the interaction request this method comprises: one or more equipment ends receive the interaction request of one or more mode of user's input；The server-side receives the interaction request issued by the equipment end, generates one or more interactive commands according to the interaction request, the interactive command is sent to the equipment end；At least some of described interactive command includes the information of interaction packet or the interaction packet；The equipment end receives and executes the interactive command, including interaction packet described in dynamically load is interacted with realizing with user.Using multi-modal exchange method, ustomer premises access equipment, server and system of the invention convenient for extension, upgrade cost is low, is provided simultaneously with the advantage in local and cloud.

Description

Multi-modal exchange method, ustomer premises access equipment, server and system
Technical field
The present invention relates to human-computer interaction technique fields, more particularly to a kind of multi-modal exchange method, ustomer premises access equipment, clothes Business device and system.
Background technique
In the PC epoch, the primary modal of human-computer interaction is the equipment such as mouse, keyboard, display, loudspeaker.In recent years, with Technological innovation, especially with the development of speech recognition and semantic understanding technology etc., existing interaction modalities have been obtained significantly It expands, the human-computer interaction of speech modality is more more and more universal.
Many manufacturers are proposed the smart machine of such as intelligent sound box, and the voice that smart machine receives user is inputted and uploaded To the voice service in the server-side of such as cloud server, voice command and data are handed down to after voice service processing and set It is standby, Software Development Tools collection (Software the Development Kit, abbreviation SDK, also known as Software Development Tools of equipment end Packet) explain and execute order.Wherein, SDK encapsulates some functions of equipment end and passes through application programming interface The form of (Application Programming Interface, abbreviation API) is called mutually with equipment end system.
The difference of smart machine and PC, in addition to introducing interactive voice, it is often more important that interaction content is by server-side Equipment end is managed and be issued to, the computing capability of server-side had both been utilized in this way, reaches and can transport further through interaction content is dynamically issued The purpose of battalion.
The existing exchange method based on equipment end and the existing exchange method based on server-side all have many lack Point.For example, being determined based on the interaction capabilities of equipment end by the hardware capabilities of equipment and the system capability of equipment, and interaction content Dynamic does not change.Although the existing exchange method based on server-side is utilized the computing capability in cloud, and can be under dynamic Equipment end is issued, but the existing exchange method based on server-side still has the disadvantages that
1, interaction capabilities render language competence by cloud and equipment end interaction SDK ability determines.What SDK was integrated into equipment is In system, upgrade cost is high, therefore the extension of SDK ability is inconvenient.
2, about graphic user interface (Graphical User Interface, abbreviation GUI, also known as graphical user interface) Equal displays service, since the GUI system at distinct device end is different, display layout is different with the behavior of control, renders GUI with cloud Mode be difficult and the style of equipment end keeps unified.
Summary of the invention
The purpose of the present invention is to provide a kind of new multi-modal exchange method, ustomer premises access equipment, server and systems.
The purpose of the present invention is realized using following technical scheme.Multi-modal exchange method proposed according to the present invention, packet Include following steps: one or more equipment ends receive the interaction request of one or more mode of user's input, by the interaction Request is sent to server-side；The server-side receives the interaction request issued by the equipment end, is asked according to the interaction One or more interactive commands are sought survival into, the interactive command is sent to the equipment end；In the interactive command at least Some includes the information of interaction packet or the interaction packet；The equipment end receives and executes the interactive command, including dynamic adds The interaction packet is carried to interact to realize with user.
The purpose of the present invention can also be further achieved by the following technical measures.
Multi-modal exchange method above-mentioned, the interaction packet include it is one or more based on the object of first language as the An object and instruction for first object；Interaction packet is described in the dynamically load to realize that interacting with user includes: to adopt With dynamically load mode and use bridge joint mode, according to first object and the instruction for first object come Second object of control and first object bridge joint；Wherein, second object is the equipment end local based on second The object of language.
Multi-modal exchange method above-mentioned receives one or more moulds of user's input in one or more of equipment ends Before the step of interaction request of state, further includes: developed to obtain the interaction packet in advance at exploitation end, the interaction is wrapped It is sent to the server-side；The server-side is received is wrapped by the interaction that the exploitation end uploads, and stores the interaction packet.
Multi-modal exchange method above-mentioned, second object include the one or more second based on the second language Attribute and one or more second methods；It is described to be developed to obtain the interaction to include: to pass through bridge at exploitation end in advance Mode is connect, maps to obtain the attribute based on the first language as the first attribute, according to described according to second attribute Two methods map to obtain the method based on the first language as first method, to obtain comprising first attribute and described First method based on the bridge joint object of the first language as first object；By one or more described first pairs As, script and resource data are packaged to obtain the interaction packet.
Multi-modal exchange method above-mentioned, the interaction packet include object data and script, and the object data includes using In adding the data of extension object for the equipment end, and/or for being updated for original object in the equipment end Data, the script include the instruction for the instruction of the extension object, and/or for original object；The dynamic The interaction is loaded to wrap to realize that interacting with user includes: the object for controlling one or more mode according to the interaction packet, It specifically includes according to the interaction packet and is updated original object, the addition extension object, to original object The one or more for being operated, updated original object is operated or the extension object being operated.
Multi-modal exchange method above-mentioned, the information of the interaction packet include the address of the interaction packet；The equipment end It receives and executes the interactive command further include: before carrying out interaction packet described in the dynamically load, wrapped according to the interaction Address obtain it is described interaction packet.
Multi-modal exchange method above-mentioned, at least some interactive commands include version number；Described according to the friendship The address mutually wrapped obtained before the step of interaction packet, further includes: is searched among the equipment end according to the version number Whether the interaction packet of identical version number is had existed, if it does not exist, then carrying out the address according to the interaction packet Obtain the interaction packet.
Multi-modal exchange method above-mentioned, one or more of equipment ends receive one or more mode of user's input Interaction request, by the interaction request be sent to server-side include: using the equipment end dynamic engine receive user it is defeated The interaction request of the first mode entered, and the interaction request of the first mode is sent to by the service by interactive engine End, or using the equipment end interaction SDK receive user's input second mode interaction request, and by second mould The interaction request of state is sent to the server-side；It is described to include: according to the one or more interactive commands of interaction request generation The server-side is judged according to the interaction request received, if the interaction request of the first mode, then generate The interactive command for including the information of interaction packet or the interaction packet is as the first interactive command, if the second mode Interaction request then generates the interaction number that one or more mode of output are interacted comprising the interaction SDK for the equipment end According to and/or the interaction data address and one or more mode instruction the second interactive command.
Multi-modal exchange method above-mentioned, the equipment end receive and execute the interactive command further include: to receiving The interactive command judged；If the interactive command is that have among first interactive command or the interactive command About the part of the first mode, then interaction packet described in the dynamically load is carried out, is specifically included: utilizing the interaction Engine obtains and decompresses the interaction packet, using the dynamic engine dynamically load and parses the interaction and wraps and pass through institute One of equipment end described in dynamic engine calling or multiple modalities engine are stated to operate the object of corresponding mode；If the interaction Order is the part having among second interactive command or the interactive command about the second mode, then will be described Second interactive command or part about the second mode are distributed to the interactive SDK, called by the interactive SDK described in One of equipment end or multiple modalities engine interact to execute second interactive command to realize with user.
Multi-modal exchange method above-mentioned, it is described the described the step of interactive command is sent to the equipment end it Before, further includes: one or more interactive commands are integrated into JavaScript object numbered musical notation format；It is described by the interaction It includes: that the interactive command of the JavaScript object numbered musical notation format is sent to the equipment that order, which is sent to the equipment end, End.
The interactive command of multi-modal exchange method above-mentioned, the JavaScript object numbered musical notation format includes: for remembering Record the key field of the modality type of the interactive command and the value field of the content for recording the interactive command；Wherein, The value field include in the form of key-value pair record the version number, it is described interaction packet address, the interaction data and/or The one or more of the address of the interaction data.
Also the following technical solution is employed for the purpose of the present invention to realize.Ustomer premises access equipment proposed according to the present invention, packet Include: user inputs receiving module, the interaction request of one or more mode for receiving user's input, by the interaction request It is sent to server；Interactive command execution module, for receiving and executing the interactive command issued by the server；Wherein, At least some interactive commands include the information of interaction packet or the interaction packet；Wherein, the interactive command execution module packet Dynamically load unit is included, is interacted with realizing with user for interaction packet described in dynamically load.
The purpose of the present invention can also be further achieved by the following technical measures.
Ustomer premises access equipment above-mentioned, the interaction packet is including one or more based on the object of first language as first pair As and for first object instruction；The dynamically load unit is specifically used for: using dynamically load mode and using Bridge joint mode is controlled according to first object and the instruction for first object and is bridged with first object The second object；Wherein, second object is the object based on second language of the ustomer premises access equipment local.
Ustomer premises access equipment above-mentioned, it is described interaction packet include object data and script, the object data include for for The ustomer premises access equipment addition extends the data of object, and/or for carrying out more for original object in the ustomer premises access equipment New data, the script include the instruction for the instruction of the extension object, and/or for original object；It is described Dynamically load unit is specifically used for: controlling the object of one or more mode according to the interaction packet, specifically includes according to institute Interaction packet is stated to be updated original object, the addition extension object, be operated, original object to update The one or more that original object afterwards is operated or operated to the extension object.
Ustomer premises access equipment above-mentioned, the information of the interaction packet include the address of the interaction packet；The interactive command is held Row module further includes interactive packet acquiring unit, is wrapped for obtaining the interaction according to the address of the interaction packet.
Ustomer premises access equipment above-mentioned, at least some interactive commands include version number；The ustomer premises access equipment further includes Whether version number's judging unit has existed identical version among the ustomer premises access equipment for searching according to the version number Number it is described interaction packet, if it does not exist, then notify the interactive packet acquiring unit carry out it is described according to it is described interaction packet ground Location obtains the interaction packet.
Ustomer premises access equipment above-mentioned, it includes first mode input receiving unit and the second mould that the user, which inputs receiving module, State input receiving unit；The first mode input receiving unit is used to receive the first mode of user's input using dynamic engine Interaction request, and the interaction request of the first mode is sent to by the server by interactive engine, for the clothes Device be engaged according to the interactive command for described in the generation of the interaction request of the first mode including interaction packet or the interactive information wrapped As the first interactive command；The second mode input receiving unit is used to receive the second mould of user's input using interaction SDK The interaction request of state, and the interaction request of the second mode is sent to the server, so that the server is according to institute The interaction request for stating second mode, which is generated, interacts the one or more of output comprising the interaction SDK for the ustomer premises access equipment Second interactive command of the instruction of the address and one or more mode of the interaction data of mode and/or the interaction data； The interactive command execution module further includes interactive command receiving unit, for receiving the interaction issued by the server Order, including receive first interactive command or second interactive command.
Ustomer premises access equipment above-mentioned, the interactive command execution module further include: judging unit, for the institute received It states interactive command to be judged, if the interactive command is that have to close among first interactive command or the interactive command In the part of the first mode, then notify that the dynamically load unit is handled, if the interactive command is described second There is part about the second mode among interactive command or the interactive command, then notify interactive SDK execution unit into Row processing；The dynamically load unit is specifically used for: the interaction packet is obtained and decompressed using the interactive engine, using described Dynamic engine dynamically load and parse it is described interaction packet and by one in ustomer premises access equipment described in the dynamic engine calling Kind or multiple modalities engine operate the object of corresponding mode；The interactive SDK execution unit is used for: second interaction is ordered Order or the part about the second mode are distributed to the interaction SDK of the ustomer premises access equipment, pass through the interactive SDK and call institute One of ustomer premises access equipment or multiple modalities engine are stated to execute second interactive command to realize and interact with user.
Ustomer premises access equipment above-mentioned, the interactive command execution module are specifically used for: receiving and execute JavaScript pairs As the interactive command of numbered musical notation format；Wherein, the interactive command of the JavaScript object numbered musical notation format includes for recording State the key field of the modality type of interactive command and the value field of the content for recording the interactive command；The value field Address, the interaction data and/or the interactive number including the version number, the interaction packet that are recorded in the form of key-value pair According to address one or more.
Also the following technical solution is employed for the purpose of the present invention to realize.Server proposed according to the present invention, comprising: hand over Mutual request receiving module, for receiving the interaction request of the one or more mode issued by one or more ustomer premises access equipments； Interactive command generation module, for generating one or more interactive commands according to the interaction request；Interactive command sending module, For the interactive command to be sent to the ustomer premises access equipment；Wherein, at least some interactive commands include interaction packet or The information of the interaction packet.
The purpose of the present invention can also be further achieved by the following technical measures.
Server above-mentioned, the interaction packet include it is one or more based on the object of first language as the first object, With the instruction for being directed to first object；Wherein, first object is the bridge joint object of the second object, and second object is The object based on second language of the ustomer premises access equipment local.
Server above-mentioned further includes interactive packet memory module, for receiving the interaction packet uploaded by exploitation end, is deposited Store up the interaction packet.
Server above-mentioned, the interaction packet include object data and script, and the object data includes for being described Data of ustomer premises access equipment addition extension object, and/or for being updated for original object in the ustomer premises access equipment Data, the script include the instruction for the instruction of the extension object, and/or for original object.
Server above-mentioned, the interaction request receiving module are specifically used for: what reception was issued by the ustomer premises access equipment The interaction request of first mode or the interaction request of second mode；Wherein, the interaction request of the first mode is by described The dynamic engine of ustomer premises access equipment and receive, second mode request is the interaction SDK by the ustomer premises access equipment And it receives；The interactive command generation module includes the first interactive command generation unit, the second interactive command generation unit One or more；The first interactive command generation unit is used to be requested according to the first mode, and generating described includes handing over Mutually the interactive command of the information of packet or the interaction packet is as the first interactive command；The second interactive command generation unit is used for It is requested according to the second mode, generates the one kind or more for interacting output comprising the interaction SDK for the ustomer premises access equipment Second interaction life of the instruction of the address and one or more mode of the interaction data and/or interaction data of kind mode It enables.
Server above-mentioned further includes unified format module, for being integrated into one or more interactive commands JavaScript object numbered musical notation format；The interactive command sending module is specifically used for the JavaScript object numbered musical notation lattice The interactive command of formula is sent to the ustomer premises access equipment；Wherein, the interactive command packet of the JavaScript object numbered musical notation format Include the value word of the key field of the modality type for recording the interactive command and the content for recording the interactive command Section；The value field includes the version number recorded in the form of key-value pair, the address of the interaction packet, the interaction data And/or the one or more of the address of the interaction data.
Also the following technical solution is employed for the purpose of the present invention to realize.Multi-modal interaction system proposed according to the present invention System, the system comprises at least one equipment ends and at least one server-side, realize the multi-modal interaction of any one above-mentioned Method.
The present invention has obvious advantages and beneficial effects compared with the existing technology.By above-mentioned technical proposal, the present invention It is proposed multi-modal exchange method, ustomer premises access equipment, server and system at least have following advantages and the utility model has the advantages that
(1) present invention is by the Dynamic loading technique using dynamics engines such as JavaScript, the multimode that will locally develop State interaction, which is wrapped, passes to server-side, and the instruction for being issued to equipment is the download address of this packet, and equipment can be from the download address Multi-modal interaction packet is downloaded, the application range of dynamically load is extended to equipment end and server-side knot from the interactive mode of equipment end The interactive mode of conjunction increases local exploitation cloud on the basis of multi-modal interaction has the two ways of pure local and pure cloud The mode of end administration, is provided simultaneously with the advantage in local and cloud；
(2) interaction packet of the invention is to develop environment exploitation in local, is docked by way of bridge joint with local system, thus Version updating of the extension of ability independent of SDK, convenient for extension, upgrade cost is low；
(3) present invention can get through different language, get through the running environment of different language by way of bridge joint, so that by The interaction content that server-side issues is docked with the local system of equipment end；By taking GUI as an example, the present invention is enabled to by under server-side The GUI control of hair can be docked with a variety of local control systems at multiple equipment end, unified with local control system style；
(4) present invention is that equipment end extends object or the original object of update by utilizing interaction packet, can extend interactive energy Power.
The above description is only an overview of the technical scheme of the present invention, in order to better understand technological means of the invention, and It can be implemented in accordance with the contents of the specification, and to allow above and other objects, features and advantages of the invention can be brighter Show understandable, it is special below to lift preferred embodiment, and cooperate attached drawing, detailed description are as follows.
Detailed description of the invention
Fig. 1 is the flow diagram of existing multi-modal exchange method；
Fig. 2 is the flow diagram of the multi-modal exchange method of one embodiment of the invention；
Fig. 3 is the flow diagram of the multi-modal exchange method of another embodiment of the present invention；
Fig. 4 is the structural schematic diagram of the ustomer premises access equipment of one embodiment of the invention；
Fig. 5 is the structural schematic diagram of the server of one embodiment of the invention；
Fig. 6 is the structural schematic diagram of the multi-modal interactive system of one embodiment of the invention.
Specific embodiment
It is of the invention to reach the technical means and efficacy that predetermined goal of the invention is taken further to illustrate, below in conjunction with Attached drawing and preferred embodiment, to the tool of multi-modal exchange method proposed according to the present invention, ustomer premises access equipment, server and system Body embodiment, structure, feature and its effect, detailed description is as follows.
Mode of this paper, also referred to as interaction modalities, including speech modality, video modality, display mode etc..
Fig. 1 is the schematic flow chart of the existing multi-modal exchange method based on server-side.Referring to Fig. 1, equipment end Interaction SDK obtain user input, be uploaded to the craft services of server-side；The craft services of server-side be based on business and scene into The unified decision-making treatment of row, requested including being inputted according to the user received the service of the multiple modalities such as voice, video, display into The respective handlings such as row speech recognition, video processing, to respectively obtain the data such as voice flow, video flowing, image, display layout, then These data returned to the service of multiple modalities are uniformly processed, and then issue to the interactive engine of equipment end for equipment Hold the order and data exported to user；The interactive engine of equipment end explains order, and is distributed to interactive SDK；Interaction SDK is called The engines such as voice, video, display in equipment end system, interact with end user.
Wherein, the ability interacted with the external world that interactive system, interactive application provide is known as interaction capabilities or is skill Energy.By taking sound-box device as an example, traditional speaker only has the input for receiving this mode of button operation and control audio broadcasting etc. and hands over Mutual ability；Compared with traditional speaker, intelligent sound box has more abilities, such as is generally also equipped with the input for receiving speech modality Ability, the control ability of other equipment such as desk lamp etc..
It may be noted that in the example depicted in fig. 1, what server-side issued is fixed protocol.Fixed protocol is also referred to as static association View refers to that the content of the programs such as executable program, the library that exploitation is completed, format are fixed.Ability as defined in fixed protocol by Equipment end interaction SDK ability determines.Fixed protocol is difficult to play the local special of every kind of mode of every kind of equipment to consider versatility Other ability, the extension of ability depend on the version updating of SDK.
Fig. 2 is the schematic flow block diagram of multi-modal exchange method one embodiment of the invention.Fig. 3 is of the invention more The schematic flow chart of another embodiment of mode exchange method.Please refer to Fig. 2 and Fig. 3, the exemplary multi-modal interaction side of the present invention Method mainly comprises the steps that
Step S11, one or more equipment ends receive the interaction request of one or more mode of user's input, by the friendship Mutually request is sent to server-side.It may be noted that the interaction request is during the user of human-computer interaction inputs to system The signal of transmitting, therefore the interaction request also referred to as interactively enters, equipment end is also referred to as client.
Step S12, server-side receive the interaction request issued by equipment end, are generated according to the interaction request one or more The interactive command is sent to equipment end by interactive command, so that equipment end interacts output according to the interactive command.It needs to infuse Meaning, which is in the system of human-computer interaction to the signal transmitted during user feedback, therefore the interactive command Referred to as interaction output.Wherein, at least some of interactive command includes the information of interaction packet or interaction packet.Optionally, the interaction The information of packet includes the download address of interaction packet.It optionally, further include right for recording interactive command institute among interactive command The field for the modality type answered.
Step S13, equipment end receive and execute the interactive command issued by server-side, interacted with realizing with user.Wherein, Equipment end executes interactive command and specifically includes: dynamically load interaction packet is interacted with realizing with user.
Wherein, the definition of dynamically load is that when program operation, computer system loading procedure itself is not deposited originally Executable file and run the code logic in these files.Optionally, it is wrapped using dynamic engine come dynamically load interaction It is interacted with realizing with user.Optionally, the dynamic engine is by calling such as speech engine, video in equipment end system to draw It holds up, one or more mode engines such as GUI engine (graphic user interface engine, alternatively referred to as display engine) are realized and user Interaction.
In some embodiments of the invention, interaction packet includes script, which includes the instruction for being issued to equipment end.It needs Note that the present invention does not limit the programming language of interaction packet, but usually scripting language, such as it can be JavaScript language (abbreviation JS), or it is also possible to Lua language.It is furthermore noted that the present invention does not limit the class of specifically used dynamic engine Type, but be usually engine corresponding with the programming language of interaction packet.
It should be noted that " the interaction packet " shown in the present invention is different with " interacting SDK ".Interaction SDK is in equipment The module for executing interactive instruction is static is installed in equipment.And interaction packet of the invention is interaction content, and is under dynamic It is dealt into equipment.A part that interaction packet can be considered as to SDK extends, and the advantage than SDK is that dynamically load dynamic updates.Example It can such as be realized by dynamic engine and the dynamically load and dynamic of interaction packet are updated.For this purpose, interaction packet can also be known as SDK kinematic insert.
In an optional example, the programming language of interaction packet is JavaScript language, and dynamic engine is JavaScript engine, then dynamic load capability is the ability of JavaScript engine dynamic operation JavaScript script.Thing In reality, existing mainstream JavaScript engine all has this ability.
Optionally, the input of client is that each equipment, every kind of mode are individually handled.All inputs upload to service End, the output that server-side carries out unified decision-making treatment and is packaged discharge device end.
The exemplary multi-modal exchange method of the present invention, by dynamically issuing technical ability, application using Dynamic loading technique, under Hair be no longer fixed protocol but dynamic agreement, that is, the format of the interactive command issued, content be not it is fixed, in turn Interaction capabilities are extended and updated by dynamically issuing interaction packet.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includes Resource data.The object data includes for adding the data of extension object for equipment end, and/or for for the original in equipment end There are the data that object is updated.The script includes the instruction for the instruction of extension object, and/or for original object.It can Selection of land, the script further include the generic logic for subject instructions and pure mathematics calculating etc. built in dynamic engine.Before Stating the dynamically load interaction in step S13 includes: the object that one or more mode are controlled according to interaction packet, is specifically included It is updated original object according to interaction packet, addition extension object, original object is operated, to updated original right As the one or more for being operated or being operated to extension object.Optionally, by call equipment end in wait operate Object corresponding mode engine operate the object.
The exemplary multi-modal exchange method of the present invention is that equipment end extends object or update is original right by utilizing interaction packet As interaction capabilities can be extended.
It may be noted that although Fig. 3 illustrates only " speech engine, video engine, display engine ", it is also possible to include other classes The mode engine of type.In fact, in the embodiment using interaction packet addition extension object of the invention, JavaScript engine It may call upon engine corresponding with extension object, to operate to the extension object.
In some embodiments of the invention, the address comprising interaction packet in interactive command.Aforementioned step S13 is also wrapped It includes: before carrying out dynamically load interaction packet, interaction packet being obtained according to the address of the interaction packet in interactive command.It is issued to equipment Interactive command can be the download address of interaction packet, be also possible to completely interaction packet, but be usually the downloading of interaction packet Location, in order to save flow.
Optionally, interactive command and downloading interaction packet are received and explained using the interactive engine of equipment end.
In some embodiments of the invention, at least some interactive commands include version number, which hands over for indicating The mutually version of packet, interactive command or protocol format.The step of interaction packet is downloaded in the address according to interaction packet in step s 13 Before, the exemplary multi-modal exchange method of the present invention further include: equipment end local is searched according to the version number in interactive command Among whether have existed the interaction packet of identical version number, if there is no or version number it is different, then carry out above-mentioned according to friendship The address mutually wrapped obtains downloading interaction packet.It is alternatively possible to execute this step using the interactive engine of equipment end.The present invention shows The multi-modal exchange method of example is convenient for subsequent modification or Extended Protocol, is improved by adding version number field in interactive command Compatibility.
In some embodiments of the invention, interaction packet is based on first language.And aforementioned step S13 is specifically wrapped It includes: using dynamically load mode and using bridge joint mode, to execute the interaction packet based on first language, to according to the friendship It mutually wraps to control the object based on second language of equipment end local, is interacted with realizing with user.Specifically, it can use support The engine of the first language of dynamically load and bridge joint realizes this step.In general, the ability of first language is by its engine It provides, provides the ability got through with local second language by the engine of first language, first language can have second The ability that language has.In practice, the JavaScript engine of mainstream all has the bridging capability.
Wherein, bridge joint mode be one kind two objects can be linked together and the two between close coupling Method gets up communication with one another by " bridge ", while allowing their respectively independent variations again.And the exemplary multi-modal friendship of the present invention Mutual method, realizes object extension in the way of bridge joint, can be object extension attribute, call directly nation method.
Further, in some embodiments of the invention, interaction packet includes one or more pair based on first language As the first object and for the instruction of the first object.Optionally, the first language should be also based on for the instruction of the first object Speech.Optionally, interaction packet further includes resource data.Dynamically load interaction packet in step S13 specifically includes: being added using dynamic Load mode simultaneously uses bridge joint mode, according to the first object and for the instruction of the first object, to control and the first object bridge joint Second object.Wherein, which is the object based on second language of equipment end local.It may be noted that the first object and Two objects can be original object above-mentioned, or be also possible to extension object above-mentioned.
In some instances, by JavaScript engine come in fact now according to JavaScript script and bridge joint JavaScript object controls the object based on native language.Specifically JavaScript engine provides registration function, The user of JavaScript engine registers the binding relationship of JavaScript object and native object.JavaScript engine is logical It crosses mapping table and safeguards all binding relationships.Wherein, which can be equipment end or multi-modal interactive system.
It may be noted that the present invention is not limited in its kind for the second language of the development language as equipment end native object, Such as be the situation of JavaScript in first language, second language can be C++ or Java etc..As an optional example, For showing mode, the object above-mentioned based on first language is JavaScript object, the base of equipment end local above-mentioned It can be the GUI control based on C++ or Java language of equipment end local in the object of second language.
The exemplary multi-modal exchange method of the present invention, different language can be got through in the way of bridge joint, gets through different language Running environment so that the interaction content issued by server-side is docked with the local system of equipment end.By taking GUI as an example, the present invention Enable to the GUI control issued by server-side that can dock with a variety of local control systems at multiple equipment end, with local control Part system style is unified.
In some embodiments of the invention, before step S11, the exemplary multi-modal exchange method of the present invention is also wrapped It includes:
Step S21 is developed at exploitation end in advance, is obtained interactive packet, which is sent to server-side.It is optional Ground, the exploitation end can be personal computer (personal computer, the letter of such as desktop computer, notebook computer Referred to as PC) etc. development platforms.
Step S22, server-side is received is wrapped by the interaction that exploitation end uploads, and stores the interaction packet.It is alternatively possible to pass through by Interaction packet is sent to the storage service of server-side to be stored.Optionally, in the situation that first language is JavaScript, The storage service is specifically JavaScript script storage service.
In some embodiments of the invention, the second object include the second attribute of one or more based on second language with And one or more second methods.And aforementioned step S21 is specifically included:
Step S31 maps to obtain the corresponding attribute conduct based on first language according to the second attribute by way of bridge joint First attribute, according to second method map to obtain it is corresponding based on the method for first language as first method, to be included The bridge joint object of first attribute and the second object based on first language of the first method is as the first object；
The first object, script and the resource data of one or more bridge joint are packaged to be interacted by step S32 Packet.
In some embodiments of the invention, such as in the situation that the second object is newly added extension object, first exist Exploitation end is developed to obtain second object, then carries out aforementioned step S31 and S32.
Optionally, the interaction packet of JavaScript script is to be generated completely by developing end.Server-side be merely responsible for storage and Forwarding does not need to generate using the service of the multiple modalities such as the voice service of server-side, Video service, display service The interaction packet of JavaScript script.
As a specific embodiment, it is directed to display mode and first language is the situation of JavaScript, it is above-mentioned Second object is the GUI control of equipment end local, and the first object above-mentioned is JavaScript pair with GUI control bridge joint As.Exploitation end develop GUI control, the GUI control include the resource datas such as layout attributes, picture resource and literal resource, with And exchange method.At exploitation end, by way of bridge joint, layout attributes is mapped as JavaScript object attribute of the same name, are incited somebody to action Exchange method is mapped as JavaScript object method of the same name.Again by the JavaScript object attribute and the JavaScript Object method is encapsulated as JavaScript object.Then, to one or more bridge joint control objects, resource data and JavaScript script is packaged and is compressed to obtain JavaScript interaction packet.Finally by JavaScript interaction packet hair It send to server-side, and is stored in the server-side.
The exemplary multi-modal exchange method of the present invention, by developing the object of bridge joint in advance, and in aforementioned step S13 The object of the bridge joint is loaded and held using the dynamic engine with dynamically load dynamic updating ability and bridging capability Row, so that the first language also has the ability that second language has.
In some embodiments of the invention, as shown in figure 3, interactive engine is used for: obtaining interaction packet, decompression obtains object Data, JavaScript script and resource data.JavaScript engine is used for: parsing and execution JavaScript script. If there is operating the object of some mode in JavaScript script, then the method for the mode engine is called.Mode engine is The actuator of second object, the second object are the contents being executed.
In some embodiments of the invention, can simultaneously a variety of different mode be carried out with the interaction of different modes.
Specifically, aforementioned step S11 specifically includes one or more below:
The interaction request of the first mode of user's input is received using the dynamic engine of the first language of equipment end, and is passed through The interaction request of first mode is sent to server-side by interactive engine；
The interaction request of the second mode of user's input is received using the interaction SDK of equipment end, and by the friendship of second mode Mutually request is sent to server-side.
Optionally, dynamic engine provides registration monitoring mechanism for developer, determines which input will be by dynamic by developer Engine processing.For example, based on scene or interaction modalities type can be based on, to select the user handled by dynamic engine to input.
The step of server-side in aforementioned step S12 generates one or more interactive commands according to interaction request is specifically wrapped It includes:
Server-side is judged according to the interaction request received；
If what is received is the interaction request of first mode, generates and output is interacted by dynamic engine for equipment end It is above-mentioned include interaction packet or the interaction packet information interactive command, as the first interactive command；Optionally, according to first The interaction request of mode simultaneously generates first interactive command by requesting storage service above-mentioned；
If what is received is the interaction request of second mode, the second interactive command is generated, wherein second interactive command Comprising for equipment end by interaction SDK interact one or more interaction datas of output and/or the address of interaction data, with And the instruction of one or more mode；Optionally, requested according to the interaction request of second mode and by craft services voice, One or more mode services such as video, display, to obtain one or more friendships such as voice flow, video flowing, image, display layout Mutual data, to obtain second interactive command.Optionally, which is the interactive command based on fixed protocol.
Further, aforementioned step S13 further include:
Equipment end judges the interactive command received, including judging that the interactive command received is the first interaction life Enable or the second interactive command, or the interactive command that receives of judgement among whether about first mode part or The no part about second mode；It is alternatively possible to carry out the judgement using the interactive engine of equipment end；
If the interactive command received is that have among the first interactive command or the interactive command received about the first mould The part of state then carries out above-mentioned wrap according to the interactive address downloading interaction wrapped and arrives locally, using dynamic engine come dynamically load Interaction packet is interacted with realizing with user, is specifically included: being obtained using the interactive engine of equipment end and decompress interaction packet, utilizes equipment The dynamic engine dynamically load at end simultaneously parses interactive wrap and by one of dynamic engine calling equipment end or multiple modalities Engine operates the object of corresponding mode；
If the interactive command received is that have among the second interactive command or the interactive command received about the second mould The part of state then should be distributed to second interactive command or about the part of second mode the interaction SDK of equipment end, by this Interaction SDK call equipment end system in such as speech engine, video engine, (graphic user interface engine, can also for GUI engine Referred to as display engine) etc. one or more mode engines interacted to execute the second interactive command to realize with user.
In some embodiments of the invention, the step that interactive command is sent to equipment end in aforementioned step S12 Before rapid, further includes: one or more interactive commands are integrated into the interactive command of unified format by server-side.Optionally, it will close In one or more interactive commands of one or more mode or by the first interactive command and the second interactive command, it is integrated into system The interactive command of one format.And interactive command be sent to equipment end specifically include in step S12: the unification that integration is obtained The interactive command of format is sent to equipment end.As an alternative embodiment, it is JavaScript object numbered musical notation that this, which unifies format, (JavaScript Object Notation, referred to as JSON) format.Specifically, the friendship of JavaScript object numbered musical notation format Mutually order is key-value pair form (or being key-value form), the key including the modality type for recording interactive order The value field of field and the content for recording interactive order.Further, which includes being recorded in the form of key-value pair : version number, interaction packet address, interaction data and/or interaction data address one or more.
It may be noted that JSON is a kind of the recognized standard data format, the overwhelming majority such as C++, java, Lua programming language is all Support JSON, and the parsing library of JSON is also very much, C++, java have the library JSON of many prevalences, real convenient for interactive engine It is existing.
As a specific example, a kind of interactive command of JSON reference format proposed by the present invention is as follows:
{
"type":data,
"type":data,
……
}。
Wherein, " type ": data is a key-value pair, for indicating an interactive command.Type therein is key field, For modality type (titles of interaction modalities) of recording interactive order, such as speech modality, video modality, display mode etc.. Optionally, type is string format.Data is value field, the data needed for recording interactive.Optionally, data is also JSON format, including one or more key-value pairs, a key-value pair in data are used to record the type and specifically of one data Value.
For example, a kind of format of the interactive command about display mode are as follows:
Wherein, display field is the key field of this order, and expression is handling display command.Square brackets " [] " it Middle content be all this order value field.The value field entirely ordered includes multiple key-value pairs again: this key assignments of version To expression version number；This key-value pair of width is display width, this key-value pair of height is display height, this is for side Just equipment end display system does the adaptation of screen layout；This key-value pair of url is the download address of GUI interaction packet.
For another example, a kind of format of the interactive command about speech modality are as follows:
Wherein, voice field expression is handling voice command；Version key-value pair indicates version number；Text key-value pair It is speech text；Url key-value pair is the download address of voice document.It may be noted that the voice document is different from the speech text , speech text is the text for recording the content of voice flow, and voice document is the voice flow of speech text.
It may be noted that the interactive command about display mode is the first interactive command above-mentioned, wherein remembering in aforementioned exemplary The download address of interactive packet is recorded；Interactive command about speech modality is the second interactive command above-mentioned, wherein what is recorded is The particular content of interaction data or the address of interaction data.
The exemplary multi-modal exchange method of the present invention, by the way that interactive command is integrated into JSON format, convenient for extension mode, The data that extension interaction needs.
It may be noted that it is a kind of preferred embodiment of the invention that interactive command, which is integrated into JSON format only, the present invention is not It must include the steps that integrating interactive command, and the format integrated also is not necessarily JSON format.Such as it can also be with It is realized with fixed bid value: for example indicating that equipment carries out different function using fixed integer, for example 1 representative is broadcast It puts the music on, 2 represent bright light etc..
It may be noted that the type at the present invention not limiting device end and server-side.Optionally, server-side above-mentioned is server. Equipment end above-mentioned is ustomer premises access equipment, including smart phone, intelligent sound box, set-top box, smart glasses, smart watches, movement Computer etc. has the product of multi-modal interaction.
Optionally, multi-modal exchange method proposed by the present invention can use application program (Application, referred to as ) etc. APP softwares are implemented.Equipment end and server-side are equipped with for realizing the soft of multi-modal exchange method proposed by the present invention Part, so as to realize each step shown in previous embodiment using the equipment end and server-side.
In some embodiments of the invention, it can use the interaction that web mode realizes GUI mode.Specifically, of the invention Interaction is realized by being embedded in webview (network view) in interaction SDK.Wherein, which it is clear to can be regarded as the page Look at device.
It may be noted that web page (also referred to as webpage) is generally made of html+css language, page browser is to show web The engine of the page.Html language is supported to be embedded in JavaScript language script with<script>label.JavaScript script fortune It goes in browser sandbox, the energy that can be provided by the JavaScript object built in operation browser come access browser Power, such as action pane.
Fig. 4 is the schematic diagram of ustomer premises access equipment 100 provided by one embodiment of the present invention.The ustomer premises access equipment 100 can be smart phone, intelligent sound box, set-top box, smart glasses, smart watches, removable computer etc. with multi-modal interaction Product.Fig. 5 is the schematic diagram of server 200 provided by one embodiment of the present invention.
Fig. 4 and Fig. 5 are please referred to, in some embodiments, the exemplary ustomer premises access equipment 100 of the present invention mainly includes that user is defeated Enter receiving module 110 and interactive command execution module 120.
Wherein, user inputs the interaction request that receiving module 110 is used to receive one or more mode of user's input, will The interaction request is sent to server 200.
Interactive command execution module 120 is for receiving and executing the interactive command issued by server 200.Wherein, at least Some interactive commands include the information of interaction packet or interaction packet.
Further, which includes dynamically load unit (not shown).The dynamically load Unit is interacted with realizing with user for dynamically load interaction packet.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includes Resource data.The object data includes for adding the data of extension object for ustomer premises access equipment 100, and/or for for user The data that original object in end equipment 100 is updated.The script includes for the instruction of extension object, and/or for original There is the instruction of object.Optionally, which further includes for the subject instructions built in dynamic engine and such as pure mathematics calculating Deng generic logic.Dynamically load unit above-mentioned is specifically used for: the object of one or more mode is controlled according to interaction packet, Specifically include according to interaction packet come be updated original object, addition extension object, original object is operated, to update after Original object operated or to the one or more that is operated of extension object.Optionally, by calling ustomer premises access equipment Mode engine corresponding with object to be operated in 100 operates the object.
In some embodiments of the invention, the information of interaction packet above-mentioned includes the address of interaction packet.Interactive command is held Row module 120 further includes interactive packet acquiring unit, for obtaining interaction packet according to the address of interaction packet.
In some embodiments of the invention, at least some interactive commands include version number.Ustomer premises access equipment 100 further includes Version number's judging unit, is used for: searching whether have existed identical version number among ustomer premises access equipment 100 according to version number Interaction packet, if there is no or version number it is different, then notify interactive packet acquiring unit above-mentioned to carry out the address according to interaction packet Obtain interaction packet.
In some embodiments of the invention, interaction packet is based on first language.Dynamically load unit above-mentioned is specific For: dynamic engine is utilized, using dynamically load mode and bridge joint mode is used, to execute interactive packet, to according to the interaction It wraps to control the local object based on second language of ustomer premises access equipment 100.Wherein, the dynamic engine be support dynamically load and The engine of the first language of bridge joint.
Further, in some embodiments of the invention, interaction packet includes one or more pair based on first language As the first object and for the instruction of the first object.Optionally, the first language should be also based on for the instruction of the first object Speech.Optionally, interaction packet further includes resource data.Dynamically load unit above-mentioned is specifically used for: using dynamically load mode, And bridge joint mode is used, according to the first object and for the instruction of the first object, to control second with first object bridge joint Object.Wherein, the second object is the local object based on second language of ustomer premises access equipment 100.
In some embodiments of the invention, user input receiving module 110 include first mode input receiving unit and Second mode input receiving unit.The first mode input receiving unit is used for: receiving the of user's input using dynamic engine The interaction request of one mode, and the interaction request of the first mode is sent to by server 200 by interactive engine, for service Device 200 generates the interactive command of the information above-mentioned for including interaction packet or interaction packet according to the interaction request of the first mode, makees For the first interactive command.The second mode input receiving unit is used for: receiving the second mode of user's input using interaction SDK Interaction request, and the interaction request of the second mode is sent to server 200, so that server 200 is according to the second mode Interaction request generate the second interactive command.Wherein, second interactive command include for ustomer premises access equipment 100 interaction SDK into The interaction data of one or more mode of row interaction output and/or the address of interaction data and one or more mode Instruction.Interactive command execution module 120 further includes interactive command receiving unit, is used for: receiving the interaction issued by server 200 Order, including receive the first interactive command or the second interactive command above-mentioned.
In some embodiments of the invention, interactive command execution module 120 further includes that judging unit and interaction SDK are executed Unit.
The judging unit is used for: being judged the interactive command received, is specifically included the interaction life that judgement receives Whether order is the first interactive command or the second interactive command or judges among the interactive command received about first mode Part or whether about second mode part；If the interactive command received is the first interactive command or interactive command Among there is part about first mode, then notify that interactive packet acquiring unit and/or dynamically load unit are handled；If connecing The interactive command received is the part having among the second interactive command or interactive command about second mode, then notifies to interact SDK execution unit is handled.
In some embodiments of the invention, interaction packet acquiring unit is specifically used for: utilizing the interaction of ustomer premises access equipment 100 Engine obtains and decompresses interactive packet.Dynamically load unit is specifically used for: utilizing the dynamic engine dynamically load of ustomer premises access equipment 100 And interactive packet is parsed, and operate by one of dynamic engine calling ustomer premises access equipment 100 or multiple modalities engine pair Answer the object of mode.
Interaction SDK execution unit is used for: the second interactive command or the part about second mode are distributed to user terminal The interaction SDK of equipment 100 calls one of ustomer premises access equipment 100 or multiple modalities engine to execute second by interaction SDK Interactive command is interacted with realizing with user.
In some embodiments of the invention, interactive command execution module 120 is specifically used for: receiving, explains and executes The interactive command of JavaScript object numbered musical notation format.
As an alternative embodiment, the interactive command of JavaScript object numbered musical notation format includes: for recording interactive The value field of the key field of the modality type of order and the content for recording interactive order.Wherein, the value field include with Key-value pair form record version number, interaction packet address, interaction data and/or interaction data address one or more.
Fig. 4 and Fig. 5 are please referred to, in some embodiments, the exemplary server 200 of the present invention mainly connects including interaction request Receive module 210, interactive command generation module 220 and interactive command sending module 230.
Wherein, interaction request receiving module 210 be used to receive the one kind issued by one or more ustomer premises access equipments 100 or The interaction request of multiple modalities.
Interactive command generation module 220 is used to generate one or more interactive commands according to interaction request.Wherein, at least one A little interactive commands include the information of interaction packet or interaction packet.Optionally, the information of the interaction packet includes the download address of interaction packet.
Interactive command sending module 230 is used to interactive command being sent to ustomer premises access equipment 100.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includes Resource data.The object data includes for adding the data of extension object for ustomer premises access equipment 100, and/or for for user The data that original object in end equipment 100 is updated.The script includes for the instruction of extension object, and/or for original There is the instruction of object.Optionally, which further includes for the subject instructions built in dynamic engine and such as pure mathematics calculating Deng generic logic.
In some embodiments of the invention, at least some interactive commands include version number.
In some embodiments of the invention, server 200 further includes interactive packet memory module.The interaction packet memory module Also referred to as storage service stores interaction packet for receiving the interaction packet uploaded by exploitation end.
In some embodiments of the invention, interaction packet is to be based on first based on first language, including one or more The object of language is as the first object and for the instruction of the first object.It optionally, should be also base for the instruction of the first object In first language.Wherein, which is the bridge joint object of the second object.Second object is ustomer premises access equipment 100 The object based on second language on ground.
In some embodiments of the invention, interaction request receiving module 210 is specifically used for: receiving by ustomer premises access equipment The interaction request of 100 first modes issued or the interaction request of second mode.Wherein, the interaction request of first mode is to pass through The dynamic engine of ustomer premises access equipment 100 and receive user input interaction request, second mode request be to pass through user terminal The interaction SDK of equipment 100 and receive user input interaction request.
Interactive command generation module 220 include the first interactive command generation unit, the second interactive command generation unit one It is a or multiple.It includes interaction packet or interaction that the first interactive command generation unit, which is used to be generated according to first mode request above-mentioned, The interactive command of the information of packet is as the first interactive command.The second interactive command generation unit according to second mode for requesting Generate the second interactive command.Wherein, which interacts output comprising the interaction SDK for ustomer premises access equipment 100 The interaction data of one or more mode and/or the address of interaction data and one or more mode instruction.
In some embodiments of the invention, server 200 further includes unified format module.This unifies format module use In: one or more interactive commands are integrated into the interactive command of unified format.It optionally, will be about one or more mode One or more interactive commands or by the first interactive command and the second interactive command, are integrated into the interactive command of unified format.Make For an alternative embodiment, it is JavaScript object numbered musical notation format that this, which unifies format,.Interactive command sending module 230 is specifically used In the interactive command for unifying format is sent to ustomer premises access equipment 100.
As an alternative embodiment, the interactive command of JavaScript object numbered musical notation format includes: for recording interactive The value field of the key field of the modality type of order and the content for recording interactive order.Wherein, the value field include with Key-value pair form record version number, interaction packet address, interaction data and/or interaction data address one or more.
Fig. 6 is the schematic diagram of 300 one embodiment of multi-modal interactive system of the invention.Referring to Fig. 6, this hair Bright exemplary multi-modal interactive system 300 includes mainly at least one equipment end and at least one server-side, realizes above-mentioned Multi-modal exchange method shown in any one embodiment.
The above described is only a preferred embodiment of the present invention, limitation in any form not is done to the present invention, though So the present invention has been disclosed as a preferred embodiment, and however, it is not intended to limit the invention, any technology people for being familiar with this profession Member, without departing from the scope of the present invention, when the technology contents using the disclosure above are modified or are modified For the equivalent embodiment of equivalent variations, but anything that does not depart from the technical scheme of the invention content, according to the technical essence of the invention Any simple modification, equivalent change and modification made to the above embodiment, all of which are still within the scope of the technical scheme of the invention.

Claims (26)

1. a kind of multi-modal exchange method, which is characterized in that the described method comprises the following steps:
One or more equipment ends receive the interaction request of one or more mode of user's input, and the interaction request is sent To server-side；
The server-side receives the interaction request issued by the equipment end, generates one or more according to the interaction request The interactive command is sent to the equipment end by a interactive command；At least some of described interactive command includes interaction packet Or the information of the interaction packet；
The equipment end receives and executes the interactive command, including interaction packet described in dynamically load is interacted with realizing with user.
2. multi-modal exchange method according to claim 1, it is characterised in that:
The interaction packet as the first object and is directed to first object based on the object of first language including one or more Instruction；
Described in the dynamically load interaction packet by realize interact with user include: using dynamically load in a manner of and use bridge joint side Formula controls second with first object bridge joint according to first object and the instruction for first object Object；Wherein, second object is the object based on second language of the equipment end local.
3. multi-modal exchange method according to claim 2, which is characterized in that received in one or more of equipment ends Before the step of interaction request of one or more mode of user's input, further includes:
It is developed to obtain the interaction packet at exploitation end in advance, the interaction packet is sent to the server-side；
The server-side is received is wrapped by the interaction that the exploitation end uploads, and stores the interaction packet.
4. multi-modal exchange method according to claim 3, it is characterised in that:
Second object includes the second attribute of one or more and one or more second party based on the second language Method；
It is described in advance exploitation end developed to obtain the interaction include:
By way of bridge joint, map to obtain the attribute based on the first language as the first attribute according to second attribute, Map to obtain the method based on the first language as first method, to obtain comprising described first according to the second method Attribute and the first method based on the bridge joint object of the first language as first object；
One or more first objects, script and resource data are packaged to obtain the interaction packet.
5. multi-modal exchange method according to claim 1, it is characterised in that:
The interaction packet includes object data and script, and the object data includes being used for as equipment end addition extension object Data, and/or data for being updated for original object in the equipment end, the script include being directed to the expansion Open up the instruction of object, and/or the instruction for original object；
Interaction packet described in the dynamically load is to realize that it includes: one or more to control according to the interaction packet for interacting with user The object of mode specifically includes according to the interaction packet and is updated original object, the addition extension object, to institute State what original object was operated, and operated to updated original object or operated to the extension object It is one or more.
6. multi-modal exchange method according to claim 1, it is characterised in that:
The information of the interaction packet includes the address of the interaction packet；
The equipment end receives and executes the interactive command further include: before carrying out interaction packet described in the dynamically load, The interaction packet is obtained according to the address of the interaction packet.
7. multi-modal exchange method according to claim 6, it is characterised in that:
At least some interactive commands include version number；
Before the address according to the interaction packet obtains the step of interaction is wrapped, further includes: according to the version number Come search whether had existed among the equipment end identical version number it is described interaction packet, if it does not exist, then carry out described in The interaction packet is obtained according to the address of the interaction packet.
8. multi-modal exchange method according to claim 1, which is characterized in that
One or more of equipment ends receive the interaction request of one or more mode of user's input, by the interaction request Being sent to server-side includes:
The interaction request of the first mode of user's input is received using the dynamic engine of the equipment end, and will by interactive engine The interaction request of the first mode is sent to the server-side,
Or using the equipment end interaction SDK receive user's input second mode interaction request, and by described second The interaction request of mode is sent to the server-side；
It is described to include: according to the one or more interactive commands of interaction request generation
The server-side judged according to the interaction request received,
If the interaction request of the first mode, then the interaction life of the information for including interaction packet or the interaction packet is generated It enables and is used as the first interactive command,
If the interaction request of the second mode, then generates and interact output comprising the interaction SDK for the equipment end The of the instruction of the address and one or more mode of the interaction data of one or more mode and/or the interaction data Two interactive commands.
9. multi-modal exchange method according to claim 8, which is characterized in that the equipment end receives and executes the friendship Mutually order further include:
The interactive command received is judged；
If the interactive command is that have among first interactive command or the interactive command about the first mode Part is then carried out interaction packet described in the dynamically load, specifically includes: being obtained using the interactive engine and decompress the friendship Mutually packet, using the dynamic engine dynamically load and parse it is described interaction packet and by being set described in the dynamic engine calling The object of corresponding mode is operated for one of end or multiple modalities engine；
If the interactive command is that have among second interactive command or the interactive command about the second mode Second interactive command or part about the second mode are then distributed to the interactive SDK, pass through the friendship by part Mutual SDK calls one of described equipment end or multiple modalities engine to execute second interactive command to realize and hand over user Mutually.
10. according to multi-modal exchange method described in claim 4,5,7 and 9, which is characterized in that
Before the described the step of interactive command is sent to the equipment end, further includes: by one or more friendships Mutually order is integrated into JavaScript object numbered musical notation format；
It is described that the interactive command is sent to the equipment end includes: by the interaction of the JavaScript object numbered musical notation format Order is sent to the equipment end.
11. multi-modal exchange method according to claim 10, which is characterized in that the JavaScript object numbered musical notation lattice The interactive command of formula includes: key field for recording the modality type of the interactive command and for recording the interaction life The value field of the content of order；Wherein, the value field includes the version number recorded in the form of key-value pair, the interaction packet Address, the interaction data and/or the interaction data address one or more.
12. a kind of ustomer premises access equipment, which is characterized in that the ustomer premises access equipment includes:
User inputs receiving module, and the interaction request of one or more mode for receiving user's input asks the interaction It asks and is sent to server；
Interactive command execution module, for receiving and executing the interactive command issued by the server；Wherein, at least some institutes State the information that interactive command includes interaction packet or the interaction packet；
Wherein, the interactive command execution module includes dynamically load unit, for described in dynamically load interaction packet with realize with User's interaction.
13. ustomer premises access equipment according to claim 12, it is characterised in that:
The interaction packet as the first object and is directed to first object based on the object of first language including one or more Instruction；
The dynamically load unit is specifically used for: using dynamically load mode and bridge joint mode is used, according to first object The second object with first object bridge joint is controlled with the instruction for first object；Wherein, described second Object is the object based on second language of the ustomer premises access equipment local.
14. ustomer premises access equipment according to claim 12, it is characterised in that:
The interaction packet includes object data and script, and the object data includes being used for as ustomer premises access equipment addition extension The data of object, and/or data for being updated for original object in the ustomer premises access equipment, the script includes needle To the instruction for extending object, and/or for the instruction of original object；
The dynamically load unit is specifically used for: the object of one or more mode is controlled according to the interaction packet, it is specific to wrap It includes according to the interaction packet and is updated original object, the addition extension object, original object is grasped The one or more made, updated original object is operated or the extension object is operated.
15. ustomer premises access equipment according to claim 12, it is characterised in that:
The information of the interaction packet includes the address of the interaction packet；
The interactive command execution module further includes interactive packet acquiring unit, for according to the address acquisition of the interaction packet Interaction packet.
16. ustomer premises access equipment according to claim 15, it is characterised in that:
At least some interactive commands include version number；
The ustomer premises access equipment further includes version number's judging unit, for searching the ustomer premises access equipment according to the version number Among whether have existed identical version number it is described interaction packet, if it does not exist, then notify the interactive packet acquiring unit into The row address according to the interaction packet obtains the interaction and wraps.
17. ustomer premises access equipment according to claim 12, which is characterized in that
It includes first mode input receiving unit and second mode input receiving unit that the user, which inputs receiving module,；
The first mode input receiving unit is used to receive the interaction request of the first mode of user's input using dynamic engine, And the interaction request of the first mode is sent to by the server by interactive engine, so that the server is according to The interaction request of first mode generates the interactive command of the information for including interaction packet or the interaction packet as the first interaction Order；
The second mode input receiving unit is used to receive the interaction request of the second mode of user's input using interaction SDK, And the interaction request of the second mode is sent to the server, so that the server is according to the friendship of the second mode Mutually request generates the interaction data that one or more mode of output are interacted comprising the interaction SDK for the ustomer premises access equipment And/or the second interactive command of the instruction of the address and one or more mode of the interaction data；
The interactive command execution module further includes interactive command receiving unit, described in receiving and being issued as the server Interactive command, including receive first interactive command or second interactive command.
18. ustomer premises access equipment according to claim 17, which is characterized in that
The interactive command execution module further include:
Judging unit, for judging the interactive command received, if the interactive command is first interaction There is the part about the first mode among order or the interactive command, then the dynamically load unit is notified to carry out Processing, if the interactive command is that have among second interactive command or the interactive command about the second mode Part, then notify that interactive SDK execution unit is handled；
The dynamically load unit is specifically used for: being obtained using the interactive engine and decompresses the interaction packet, moved using described State engine dynamically load and parse it is described interaction packet and by one of ustomer premises access equipment described in the dynamic engine calling Or multiple modalities engine operates the object of corresponding mode；
The interactive SDK execution unit is used for: second interactive command or the part about the second mode are distributed to The interaction SDK of the ustomer premises access equipment calls one of described ustomer premises access equipment or multiple modalities to draw by the interactive SDK It holds up to execute second interactive command to realize and be interacted with user.
19. ustomer premises access equipment described in 3,14,16 and 18 according to claim 1, which is characterized in that
The interactive command execution module is specifically used for: receiving and execute the interactive command of JavaScript object numbered musical notation format；
Wherein, the interactive command of the JavaScript object numbered musical notation format includes the mode class for recording the interactive command The value field of the key field of type and the content for recording the interactive command；The value field includes being remembered in the form of key-value pair The version number of record, the address of the interaction packet, one or more of address of the interaction data and/or the interaction data It is a.
20. a kind of server, which is characterized in that the server includes:
Interaction request receiving module, for receiving the interaction of the one or more mode issued by one or more ustomer premises access equipments Request；
Interactive command generation module, for generating one or more interactive commands according to the interaction request；
Interactive command sending module, for the interactive command to be sent to the ustomer premises access equipment；
Wherein, at least some interactive commands include the information of interaction packet or the interaction packet.
21. server according to claim 20, it is characterised in that: the interaction packet includes one or more based on first The object of language is as the first object and for the instruction of first object；Wherein, first object is the second object Object is bridged, second object is the object based on second language of the ustomer premises access equipment local.
22. server according to claim 20, which is characterized in that further include interactive packet memory module, for receive by The interaction packet that end uploads is developed, the interaction packet is stored.
23. server according to claim 20, it is characterised in that: the interaction packet includes object data and script, institute Stating object data includes for adding the data of extension object for the ustomer premises access equipment, and/or for setting for the user terminal The data that original object in standby is updated, the script include for the instruction of the extension object, and/or for described The instruction of original object.
24. server according to claim 20, which is characterized in that
The interaction request receiving module is specifically used for: receiving the interaction request of the first mode issued by the ustomer premises access equipment Or the interaction request of second mode；Wherein, the interaction request of the first mode is drawn by the dynamic of the ustomer premises access equipment It holds up and receives, the second mode request is received by the interaction SDK of the ustomer premises access equipment；
The interactive command generation module include the first interactive command generation unit, one of the second interactive command generation unit or It is multiple；
The first interactive command generation unit is used to be requested according to the first mode, and generating described includes interaction packet or described The interactive command of the information of interaction packet is as the first interactive command；
The second interactive command generation unit is used to be requested according to the second mode, generates comprising supplying the ustomer premises access equipment Interaction SDK interact output one or more mode interaction data and/or the interaction data address, Yi Jiyi Second interactive command of the instruction of kind or multiple modalities.
25. according to server described in claim 21,22,23 and 24, it is characterised in that:
It further include unified format module, for one or more interactive commands to be integrated into JavaScript object numbered musical notation lattice Formula；
The interactive command sending module is specifically used for for the interactive command of the JavaScript object numbered musical notation format being sent to The ustomer premises access equipment；
Wherein, the interactive command of the JavaScript object numbered musical notation format includes the mode class for recording the interactive command The value field of the key field of type and the content for recording the interactive command；The value field includes being remembered in the form of key-value pair The version number of record, the address of the interaction packet, one or more of address of the interaction data and/or the interaction data It is a.
26. a kind of multi-modal interactive system, which is characterized in that the system comprises at least one equipment ends and at least one service End, realizes the method according to claim 1 to any one of 11.
CN201910667966.3A 2019-07-23 2019-07-23 Multi-mode interaction method, user terminal equipment, server and system Active CN110399040B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910667966.3A CN110399040B (en) 2019-07-23 2019-07-23 Multi-mode interaction method, user terminal equipment, server and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910667966.3A CN110399040B (en) 2019-07-23 2019-07-23 Multi-mode interaction method, user terminal equipment, server and system

Publications (2)

Publication Number Publication Date
CN110399040A true CN110399040A (en) 2019-11-01
CN110399040B CN110399040B (en) 2023-05-12

Family

ID=68325872

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910667966.3A Active CN110399040B (en) 2019-07-23 2019-07-23 Multi-mode interaction method, user terminal equipment, server and system

Country Status (1)

Country Link
CN (1) CN110399040B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115220631A (en) * 2022-07-19 2022-10-21 东软睿驰汽车技术(大连)有限公司 Application control method and device based on in-vehicle interaction mode and electronic equipment

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004045154A1 (en) * 2002-11-13 2004-05-27 Intel Corporation Multi-modal web interaction over wireless network
CN1610346A (en) * 2003-10-20 2005-04-27 国际商业机器公司 Systems and methods for providing dialog localization and enabling conversational communication
CN1700692A (en) * 2004-05-18 2005-11-23 阿尔卡特公司 Method and server for providing a multi-modal dialog
US20060150082A1 (en) * 2004-12-30 2006-07-06 Samir Raiyani Multimodal markup language tags
CN102141893A (en) * 2011-05-11 2011-08-03 清华大学 Large-breadth interactive desktop-oriented multi-user window system
US8346563B1 (en) * 2012-04-10 2013-01-01 Artificial Solutions Ltd. System and methods for delivering advanced natural language interaction applications
CN103292815A (en) * 2006-03-08 2013-09-11 通腾科技股份有限公司 Navigation device and method for conveying information relationships
US20130268260A1 (en) * 2012-04-10 2013-10-10 Artificial Solutions Iberia SL System and methods for semiautomatic generation and tuning of natural language interaction applications
CN106210008A (en) * 2016-06-30 2016-12-07 北京世纪好未来教育科技有限公司 Data interactive method, client and system
US20170160813A1 (en) * 2015-12-07 2017-06-08 Sri International Vpa with integrated object recognition and facial expression recognition
CN107004408A (en) * 2014-12-09 2017-08-01 微软技术许可有限责任公司 For determining the method and system of the user view in spoken dialog based at least a portion of semantic knowledge figure is converted into Probability State figure
CN107632706A (en) * 2017-09-08 2018-01-26 北京光年无限科技有限公司 The application data processing method and system of multi-modal visual human
US20190087205A1 (en) * 2017-09-18 2019-03-21 Microsoft Technology Licensing, Llc Varying modality of user experiences with a mobile device based on context

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004045154A1 (en) * 2002-11-13 2004-05-27 Intel Corporation Multi-modal web interaction over wireless network
CN1610346A (en) * 2003-10-20 2005-04-27 国际商业机器公司 Systems and methods for providing dialog localization and enabling conversational communication
CN1700692A (en) * 2004-05-18 2005-11-23 阿尔卡特公司 Method and server for providing a multi-modal dialog
US20060150082A1 (en) * 2004-12-30 2006-07-06 Samir Raiyani Multimodal markup language tags
CN103292815A (en) * 2006-03-08 2013-09-11 通腾科技股份有限公司 Navigation device and method for conveying information relationships
CN102141893A (en) * 2011-05-11 2011-08-03 清华大学 Large-breadth interactive desktop-oriented multi-user window system
US8346563B1 (en) * 2012-04-10 2013-01-01 Artificial Solutions Ltd. System and methods for delivering advanced natural language interaction applications
US20130268260A1 (en) * 2012-04-10 2013-10-10 Artificial Solutions Iberia SL System and methods for semiautomatic generation and tuning of natural language interaction applications
CN107004408A (en) * 2014-12-09 2017-08-01 微软技术许可有限责任公司 For determining the method and system of the user view in spoken dialog based at least a portion of semantic knowledge figure is converted into Probability State figure
US20170160813A1 (en) * 2015-12-07 2017-06-08 Sri International Vpa with integrated object recognition and facial expression recognition
CN106210008A (en) * 2016-06-30 2016-12-07 北京世纪好未来教育科技有限公司 Data interactive method, client and system
CN107632706A (en) * 2017-09-08 2018-01-26 北京光年无限科技有限公司 The application data processing method and system of multi-modal visual human
US20190087205A1 (en) * 2017-09-18 2019-03-21 Microsoft Technology Licensing, Llc Varying modality of user experiences with a mobile device based on context

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115220631A (en) * 2022-07-19 2022-10-21 东软睿驰汽车技术(大连)有限公司 Application control method and device based on in-vehicle interaction mode and electronic equipment

Also Published As

Publication number Publication date
CN110399040B (en) 2023-05-12

Similar Documents

Publication Publication Date Title
US7756905B2 (en) System and method for building mixed mode execution environment for component applications
US20210012059A1 (en) Mobile device resource provisioning system and method
EP3191942B1 (en) Interactive web application editor
US6996800B2 (en) MVC (model-view-controller) based multi-modal authoring tool and development environment
US7900214B2 (en) System and method for adaptable provisioning of generic application content
US7509649B2 (en) System and method for conversion of generic services&#39; applications into component based applications for devices
Huang et al. Programming situational mobile web applications with cloud-mobile convergence: an internetware-oriented approach
US7814404B2 (en) System and method for applying workflow of generic services to component based applications for devices
US7216351B1 (en) Systems and methods for synchronizing multi-modal interactions
US7747683B2 (en) Method and system for operating applications for remote terminal devices
US20060251047A1 (en) System and method of representing data entities of standard device applications as built-in components
US20110119601A1 (en) Method and apparatus for presenting a web application instance to multiple user interfaces
US8387039B2 (en) System and method for customized provisioning of application content
EP1698984A1 (en) System and method for conversion of WEB services&#39; applications into component based applications for mobile devices
CN110399040A (en) Multi-modal exchange method, ustomer premises access equipment, server and system
JP2011243213A (en) Managing multiple languages in data language
KR101599713B1 (en) Apparatus for transmitting and receiving the status for executing web application seamlessly in a terminal, method thereof and computer recordable medium storing the method
CN111290746A (en) Object access method, device, equipment and storage medium
EP1698975A1 (en) System and method for applying workflow of generic services to component based applications
EP4216052A1 (en) Method for developing mvvm architecture-based application, and terminal
US20240129402A1 (en) Customization framework for native mobile applications
EP1560114A1 (en) Computer system and method for customized provisioning of application content
EP1560115A1 (en) Computer system and method for adaptable provisioning of generic application content
KR20030006600A (en) A program library system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant