CN110851108A

CN110851108A - Electronic equipment operation method and device, electronic equipment and storage medium

Info

Publication number: CN110851108A
Application number: CN201911130236.6A
Authority: CN
Inventors: 董红光; 吴华; 范宏伟
Original assignee: Beijing Xiaomi Mobile Software Co Ltd
Current assignee: Beijing Xiaomi Mobile Software Co Ltd
Priority date: 2019-11-18
Filing date: 2019-11-18
Publication date: 2020-02-28

Abstract

The disclosure provides an electronic equipment operation method and device, electronic equipment and a storage medium, and belongs to the technical field of electronic equipment. The method comprises the following steps: displaying a first application interface of a first target application program; when an input voice signal is received, acquiring an execution event and event information corresponding to the voice signal through a first server; generating an operation instruction of a voice signal according to the execution event, the event information and the interface data of the first application interface; and executing the operation corresponding to the operation instruction in the first application interface. In the disclosure, the electronic device may generate the operation instruction of the voice signal only by acquiring the execution event and the event information corresponding to the voice signal through the first server and combining interface data of the first application interface currently displayed by the electronic device. Interface data of a first application interface currently displayed by the electronic equipment cannot be uploaded to the first server, so that privacy of the interface data of the first target application program can be protected.

Description

Electronic equipment operation method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of electronic devices, and in particular, to an electronic device operating method and apparatus, an electronic device, and a storage medium.

Background

The development of internet technology and electronic devices greatly facilitates the life of people. Applications on electronic devices can provide various services to people such as communication, entertainment, and shopping. People can obtain services required by themselves by installing application programs on the electronic equipment. For example, a person may install a take-away application on an electronic device, open an ordering interface of the take-away application, click on a desired meal item in the ordering interface, and add the meal item to a shopping cart.

In the related art, the electronic device obtains an installation package of an application program, and installs the application program using the installation package of the application program. When the electronic equipment runs the installed application program, displaying an interface of the application program; in the process of displaying the interface of the application program, when the electronic equipment receives the click operation of the control in the interface, corresponding operation is executed according to the control.

Disclosure of Invention

The embodiment of the disclosure provides an electronic device operation method and device, an electronic device and a storage medium, which can protect the privacy of interface data of a first target application program. The technical scheme is as follows:

according to an aspect of the embodiments of the present disclosure, there is provided an electronic device operating method, the method including:

displaying a first application interface of a first target application program;

when an input voice signal is received, acquiring an execution event and event information corresponding to the voice signal through a first server;

generating an operation instruction of the voice signal according to the execution event, the event information and the interface data of the first application interface;

and executing the operation corresponding to the operation instruction in the first application interface.

In a possible implementation manner, the generating the operation instruction of the voice signal according to the execution event, the event information, and the interface data of the first application interface includes:

matching a control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface;

and generating the operation instruction according to the execution event and the control identification.

In another possible implementation manner, the matching, in the first application interface, a control identifier corresponding to the event information according to the event information and the interface data of the first application interface includes:

the interface data of the first application interface comprises a plurality of interface elements and control identifications corresponding to the interface elements; searching a target interface element matched with the event information in the interface data of the first application interface;

and when the target interface element matched with the event information is found, determining a control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the method further includes:

when the target interface element matched with the event information is not found in the interface data of the first application interface, acquiring interface data of a second application interface which is displayed in a historical manner, wherein the interface data of the second application interface comprises a plurality of interface elements and control identifications corresponding to the interface elements;

searching a target interface element matched with the event information in the interface data of the second application interface;

In another possible implementation manner, the determining, according to the control identifier corresponding to the target interface element, the control identifier corresponding to the event information includes:

when the target interface element corresponds to a control identification, determining the control identification corresponding to the target interface element as the control identification corresponding to the event information;

and when the target interface element corresponds to the plurality of control identifications, determining the control identification corresponding to the execution event from the plurality of control identifications corresponding to the target interface element according to the execution event.

In another possible implementation manner, the displaying the first application interface of the first target application includes:

sending a first acquisition request to the first server, and sending a second acquisition request to a second server corresponding to the first target application program;

receiving style data of the first application interface returned by the first server, and receiving interface data of the first application interface returned by the second server;

and rendering the first application interface according to the style data and the interface data.

In another possible implementation manner, the method further includes:

displaying a third application interface of a second target application program which is installed locally, wherein the third application interface comprises an application interface of the first target application program which is not installed locally;

and starting the first target application program when detecting that the application interface of the first target application program is triggered.

In another possible implementation manner, the obtaining, by the first server, the execution event and event information corresponding to the voice signal includes:

sending the voice signal to a first server, wherein the first server is used for identifying an execution event and event information corresponding to the voice signal through a voice identification rule according to the voice signal;

and receiving the execution event and the event information returned by the first server.

According to another aspect of the embodiments of the present disclosure, there is provided an electronic device operating apparatus, the apparatus including:

a display module configured to display a first application interface of a first target application;

the acquisition module is configured to acquire an execution event and event information corresponding to an input voice signal through a first server when the input voice signal is received;

the generating module is configured to generate an operation instruction of the voice signal according to the execution event, the event information and the interface data of the first application interface;

and the execution module is configured to execute the operation corresponding to the operation instruction in the first application interface.

In a possible implementation manner, the generating module is further configured to match a control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface; and generating the operation instruction according to the execution event and the control identification.

In another possible implementation manner, the generating module is further configured to include, in the interface data of the first application interface, a plurality of interface elements and a control identifier corresponding to each interface element; searching a target interface element matched with the event information in the interface data of the first application interface; and when the target interface element matched with the event information is found, determining a control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the generation module is further configured to obtain interface data of a second application interface displayed in a history manner when a target interface element matched with the event information is not found in the interface data of the first application interface, where the interface data of the second application interface includes a plurality of interface elements and a control identifier corresponding to each interface element; searching a target interface element matched with the event information in the interface data of the second application interface; and when the target interface element matched with the event information is found, determining a control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the generating module is further configured to determine, when the target interface element corresponds to one control identifier, the control identifier corresponding to the target interface element as the control identifier corresponding to the event information; and when the target interface element corresponds to the plurality of control identifications, determining the control identification corresponding to the execution event from the plurality of control identifications corresponding to the target interface element according to the execution event.

In another possible implementation manner, the display module is further configured to send a first obtaining request to the first server, and send a second obtaining request to a second server corresponding to the first target application program; receiving style data of the first application interface returned by the first server, and receiving interface data of the first application interface returned by the second server; and rendering the first application interface according to the style data and the interface data.

In another possible implementation manner, the display module is further configured to display a third application interface of a second target application installed locally, where the third application interface includes an application interface of the first target application that is not installed locally; and starting the first target application program when detecting that the application interface of the first target application program is triggered.

In another possible implementation manner, the obtaining module is further configured to send the voice signal to the first server, where the first server is configured to identify, according to the voice signal, an execution event and event information corresponding to the voice signal through a voice recognition rule; and receiving the execution event and the event information returned by the first server.

According to another aspect of the embodiments of the present disclosure, there is provided an electronic device including:

one or more processors;

volatile or non-volatile memory for storing the one or more processor-executable instructions;

wherein the one or more processors are configured to:

displaying a first application interface of a first target application program;

According to another aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the electronic device operation method according to any one of the above-mentioned possible implementation manners.

In the embodiment of the disclosure, the electronic device can generate the operation instruction of the voice signal only by acquiring the execution event and the event information corresponding to the voice signal through the first server and combining the interface data of the first application interface currently displayed by the electronic device, so that the interface data of the first application interface currently displayed by the electronic device is not uploaded to the first server, and the privacy of the interface data of the first target application program can be protected.

In addition, as the electronic equipment searches the target interface element matched with the event information from the interface data of the currently displayed first application interface, the generated operation instruction is matched with the currently displayed interface, and the operation corresponding to the operation instruction executed by the electronic equipment is more accurate.

In addition, when the target interface element matched with the event information cannot be searched in the interface data of the first application interface by the electronic equipment, the target interface element matched with the event information can be searched in the second application interface displayed in the history by the electronic equipment, so that the electronic equipment can deal with more complex voice interaction scenes, and the experience degree of a user is improved.

And when the target interface element corresponds to the plurality of control identifications, the electronic device can determine the control identification corresponding to the event information from the plurality of control identifications according to the execution event, so that for the interface with more complex interface elements, the electronic device can also accurately determine the control identification corresponding to the event information, thereby coping with complex voice interaction scenes and improving the experience of users.

Moreover, the first server is only used for providing style data for the operation of the first target application program, and cannot acquire the interface data of the first target application program. Therefore, data privacy protection of the interface data of the first target application program can be achieved.

In addition, the user can operate the first target application program without downloading an installation package corresponding to the first target application program or installing the first target application program on the electronic equipment, so that the local storage space of the electronic equipment is saved, and the function of the first target application program can be used by directly triggering the application interface, so that the experience of the user can be improved.

Moreover, the first server can convert the voice signal into the execution event and the event information, and the voice recognition service is provided for the fast application developer, so that the developer can develop the fast application with the voice interaction function through the first server, and the application scene of the fast application is richer.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

FIG. 1 is a schematic diagram of an implementation environment shown in accordance with an exemplary embodiment;

FIG. 2 is a flow chart illustrating a method of operation of an electronic device in accordance with an exemplary embodiment;

FIG. 3 is an interaction flow diagram illustrating a method of operation of an electronic device in accordance with an exemplary embodiment;

FIG. 4 is a block diagram illustrating a fast application framework in accordance with an exemplary embodiment;

FIG. 5 is a flowchart illustrating the launching of a fast application, according to an exemplary embodiment;

FIG. 6 is a block diagram illustrating an electronic device operating apparatus in accordance with an exemplary embodiment;

FIG. 7 is a block diagram illustrating an electronic device in accordance with an example embodiment.

Detailed Description

To make the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, "a plurality" means two or more unless specifically limited otherwise.

Quick application: the application program is a novel application program and can run based on a native API (application programming Interface) provided by an operating system. The fast application is developed by adopting a unified standard, is rendered as native, and has the double advantages of an HTML5(Hyper Text Markup Language, 5 th generation) interface and native application. The framework of the fast application can be deeply integrated in an operating system, can operate at the level of the operating system, and realizes seamless connection with other application services. The developer can operate on different types of equipment by one-time development, and can use the equipment without downloading and installation by a user, namely, the equipment is used on point.

Due to the specificity of fast applications, fast applications are expected to perform more tasks and perform more functions. At present, while the function of the fast application is being gradually improved, the embodiment of the present disclosure provides a new function of the fast application, and by using the new function, the fast application can implement voice interaction with a user, thereby expanding the function of the fast application and improving the processing capability and the practicability of the fast application.

FIG. 1 is a schematic diagram illustrating one implementation environment in accordance with an example embodiment. Referring to fig. 1, the implementation environment includes an electronic device 101, a first server 102, and a second server 103; the electronic device 101 and the first server 102 are connected via a wireless or wired network. The electronic device 101 and the second server 103 are connected via a wireless or wired network. The first server 102 and the second server 103 may be connected via a wireless or wired network. The connection between the first server 102 and the second server 103 may not be established.

The electronic device 101 may be a computer, a cell phone, a smart television, a wearable device, a tablet, or other electronic device 101. A second target application may be installed on the electronic device 101, and the first server 102 is configured to provide a service for the second target application. The user of the electronic device 101 may implement functions such as data transfer, message interaction, etc. through the second target application.

The second target application may be an application preset in an operating system of the electronic device 101, or may be an application provided by a third party. For example, the second target application may be an application such as an application download center or a fast application center. The second target application may have an application interface of the first target application provided therein. The electronic device 101 may quickly enter the application interface of the first target application through the application interface of the first target application. The first target application may be any application that is not installed on the electronic device 101. The first target application may be a fast application. For example, the first target application program may be a shopping block application, a music block application, a take-away block application, a video block application, a social block application, a browser block application, or the like, which provides an arbitrary service to the user.

FIG. 2 is a flow chart illustrating a method of operation of an electronic device in accordance with an exemplary embodiment. Referring to fig. 2, the electronic device operating method includes the steps of:

in step S201, a first application interface of a first target application is displayed.

In step S202, when the input voice signal is received, an execution event and event information corresponding to the voice signal are acquired by the first server.

In step S203, an operation instruction of the voice signal is generated according to the execution event, the event information, and the interface data of the first application interface.

In step S204, an operation corresponding to the operation instruction is executed in the first application interface.

In one possible implementation manner, the generating an operation instruction of a voice signal according to the execution event, the event information, and the interface data of the first application interface includes:

and generating an operation instruction according to the execution event and the control identification.

In another possible implementation manner, matching a control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface includes:

and when the target interface element matched with the event information is found, determining the control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the method further includes:

when the target interface element matched with the event information is not found in the interface data of the first application interface, interface data of a second application interface which is displayed in a historical manner is obtained, wherein the interface data of the second application interface comprises a plurality of interface elements and control identifications corresponding to the interface elements;

In another possible implementation manner, determining a control identifier corresponding to the event information according to a control identifier corresponding to the target interface element includes:

when the target interface element corresponds to one control identification, determining the control identification corresponding to the target interface element as the control identification corresponding to the event information;

In another possible implementation manner, displaying a first application interface of a first target application includes:

sending a first acquisition request to a first server, and sending a second acquisition request to a second server corresponding to a first target application program;

receiving style data of a first application interface returned by a first server, and receiving interface data of the first application interface returned by a second server;

In another possible implementation manner, the method further includes:

In another possible implementation manner, acquiring, by a first server, an execution event and event information corresponding to a voice signal includes:

sending a voice signal to a first server, wherein the first server is used for identifying an execution event and event information corresponding to the voice signal through a voice identification rule according to the voice signal;

FIG. 3 is an interactive flow diagram illustrating a method of operation of an electronic device in accordance with an exemplary embodiment. Referring to fig. 3, the electronic device operating method includes the steps of:

in step S301, the electronic device displays a first application interface of a first target application.

The first target application is any fast application of the electronic device. The first application interface is any interface of the first target application program. The first application interface may be a main interface of the first target application program, and the first target application program may also be any sub-interface of the first target application program. For example, when the first target application program is a shopping block application, the first application interface may be a main interface of the shopping block application, that is, the first application interface may be a home page of the shopping block application; the first application interface may also be any sub-interface of the shopping mall application, for example, the first application interface may be any interface displaying item information.

The electronic equipment can acquire style data of the first application interface through the first server, acquire interface data of the first application interface through the second server, and render the first application interface according to the style data and the interface data.

Correspondingly, the step of displaying, by the electronic device, the first application interface of the first target application program may be: the electronic equipment sends a first obtaining request for obtaining the style data to a first server, and sends a second obtaining request for obtaining the interface data to a second server corresponding to the first target application program. Receiving style data of the first application interface returned by the first server, receiving interface data of the first application interface returned by the second server, and rendering the first application interface according to the style data and the interface data.

The first server may be configured to provide style data for rendering the first application interface for the first target application program. The first server may also be used to provide various capability interfaces for a developer of the first target application. The second server can be a background server corresponding to any fast application. The second server may be configured to provide interface data for rendering the first application interface for the first target application. The second server may also be used to provide services for the first target application. For example, when the first target application program is a shopping block application, the second server is a background server corresponding to the shopping block application. The second server provides item information data, order information data, etc. for the shopping mall application. The style data may be an interface framework for rendering the first application interface. The interface data may be content within an interface frame. For example, the first application interface displays an item information interface for a shopping mall application. The style data includes a fixed interface frame such as an item information display box, an immediate purchase button, or an add shopping cart button, etc. The interface data comprises an article image and information describing the article image, and the interface data can be displayed in the article information display frame.

In the embodiment of the present disclosure, the first server is only used to provide style data for the running of the first target application, and cannot acquire the interface data of the first target application. Therefore, data privacy protection of the interface data of the first target application program can be achieved.

It should be noted that before the electronic device displays the first application interface of the first target application, the first target application may also be started by the second target application. Correspondingly, the step of the electronic device starting the first target application program through the second target application program may be: and the electronic equipment displays a third application interface of the second target application program which is installed locally, wherein the third application interface comprises the application interface of the first target application program which is not installed locally. When the electronic equipment detects that the application interface of the first target application program is triggered, the first target application program is started.

The second target application program can be an application program installed locally on the electronic device. For example, the second target application is an application download center, a fast application center, or a desktop application installed on the electronic device. The third application interface may include an application interface of the first target application, and the application interface is used to launch the first target application. For example, the second target application is an application download center, the electronic device receives a search operation for the first target application, and displays a search result in the third application interface, where the search result includes an application interface of the first target application, and the application interface may include a "second on" identifier. And when the electronic equipment receives the trigger operation of the application interface, starting the first target application program and displaying the main interface of the first target application program. For another example, the second target application is a fast application center, and when the electronic device receives an operation that triggers the start of the second target application, a third application interface of the second target application is displayed, where the third application interface includes the application interface of the first target application. And when the electronic equipment receives the trigger operation of the application interface, starting the first target application program and displaying the main interface of the first target application program. For another example, the second target application program is a desktop application, the third application interface of the second target application program includes an application interface of the first target application program, and when the electronic device receives a trigger operation on the application interface, the first target application program is started, and the main interface of the first target application program is displayed.

In the embodiment of the disclosure, since the user does not need to download the installation package corresponding to the first target application program, and does not need to install the first target application program on the electronic device, the first target application program can be run, the local storage space of the electronic device is saved, and the function of the first target application program can be used by directly triggering the application interface, so that the experience of the user can be improved.

In step S302, when the electronic device receives an input voice signal, the voice signal is transmitted to a first server.

In an embodiment of the present disclosure, the first server is further capable of providing a voice recognition service for the first target application. When the electronic device receives the inputted voice signal, the voice signal may be transmitted to the first server.

The electronic device may transmit all received inputted voice signals to the first server. The electronic device may also start sending the voice signal to the first server upon detecting that the voice service is triggered. For example, when the electronic device receives an operation instruction for starting voice interaction, the electronic device starts to send a voice signal to the first server.

In step S303, the first server receives the voice signal sent by the electronic device, and identifies an execution event and event information corresponding to the voice signal according to the voice signal and through a voice identification rule.

In the embodiment of the present disclosure, the execution event is used to manipulate the electronic device that sends the voice signal, and the electronic device may execute a corresponding operation according to the execution event. For example, the execution event may be an operation event, such as a click event or an interface slide event. The event information may include information of an operation object corresponding to the execution event.

In a possible implementation manner, the first server may convert the received voice signal into text information, and then convert the text information into an execution event and event information corresponding to the voice signal. Correspondingly, the step of recognizing the execution event and the event information corresponding to the voice signal by the first server according to the voice signal and the voice recognition rule may be: the first server converts the received voice signal into text information through at least one of an ASR (Automatic Speech Recognition) model and an NLP (natural language Processing) model. The first server can identify the execution event and the event information corresponding to the voice signal according to the text information and the voice identification rule.

The speech recognition rules may include textual information and execution events. The first server can determine an execution event corresponding to the text message according to the voice recognition rule. For example, the voice signal received by the first server is "open file a", and the first server converts the voice signal into the text message "open file a". The first server recognizes that the execution event corresponding to the opening in the text information corresponding to the voice signal is 'click' and the event information corresponding to the file a in the text information corresponding to the voice signal is 'file a' through a voice recognition rule.

In another possible implementation manner, the first server can directly recognize the execution event and the event information corresponding to the voice signal according to the voice signal and the voice recognition rule. For clarity of description, the voice signal received by the first server is taken as the first voice signal. The speech recognition rule may include at least one second speech signal and an execution event corresponding to each second speech signal.

Correspondingly, the step of recognizing the execution event and the event information corresponding to the first voice signal by the first server according to the voice signal and the voice recognition rule may be: the first server determines a second voice signal matched with the first voice signal from a voice recognition rule according to the first voice signal, and determines an execution event corresponding to the second voice signal; the first server removes the second voice signal from the first voice signal, generates a third voice signal, and takes the third voice signal as event information. For example, the first server receives a first voice signal of "open file b", and the first server determines from the voice recognition rule that a second voice signal matching the first voice signal is "open". The second voice signal is opened, the corresponding execution event is "click", and the first server recognizes that the execution event corresponding to the first voice signal is "click". And if the third voice signal generated by removing the second voice signal from the first voice signal is 'file b', the server recognizes that the event information corresponding to the first voice signal is 'file b'.

In the embodiment of the disclosure, the first server can convert the voice signal into the execution event and the event information, and a voice recognition service is provided for a fast application developer, so that the developer can develop the fast application with the voice interaction function through the first server, and the application scene of the fast application is richer.

In step S304, the first server returns the execution event and the event information to the electronic device.

After recognizing the execution event and the event information corresponding to the voice signal, the first server may return the execution event and the event information to the electronic device. For example, the first server may return the execution event and the event information to the electronic device by wireless transmission.

In step S305, the electronic device receives the execution event and the event information returned by the first server, and generates an operation instruction of a voice signal according to the execution event, the event information, and the interface data of the first application interface.

In the embodiment of the disclosure, when the electronic device receives the execution event and the event information returned by the first server, the control identifier corresponding to the event information may be determined in the first application interface according to the event information, and the execution event and the control identifier are combined to generate the operation instruction. Correspondingly, the step of generating, by the electronic device, the operation instruction of the voice signal according to the execution event, the event information, and the interface data of the first application interface may be: the electronic equipment matches a control identification corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface; and generating an operation instruction according to the execution event and the control identification.

Wherein the control identifies an operable control for identifying interface data in the first application interface. For example, when the first target application is a takeaway application, the electronic device receives a voice signal of "i want package a", sends the voice signal to the first server, receives an execution event returned by the first server as "add", and sends the event information as "package a". The electronic equipment matches a control identifier corresponding to the package A in the first application interface to be 001 according to the event information and the interface data of the first application interface, and generates an operation instruction add 001 according to the execution event and the control identifier.

In the embodiment of the disclosure, because the electronic device receives the execution event and the event information returned by the first server, the operation instruction corresponding to the voice signal may be generated according to the interface data of the first application interface locally displayed by the electronic device, so that the interface data locally displayed does not need to be uploaded to the first server, and the operation instruction of the voice signal is generated by the first server, thereby protecting the data privacy of the interface data displayed by the first target application program.

In a possible implementation manner, the electronic device may search, according to the event information, a target interface element matched with the event information from the interface data of the first application interface, and determine a control identifier corresponding to the target interface element as the control identifier corresponding to the event information. Correspondingly, the step of matching, by the electronic device, the control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface may be: the interface data of the first application interface comprises a plurality of interface elements and control identifications corresponding to the interface elements; the electronic equipment searches a target interface element matched with the event information in the interface data of the first application interface; and when the target interface element matched with the event information is found, the electronic equipment determines the control identification corresponding to the event information according to the control identification corresponding to the target interface element. When the electronic device does not find the target interface element matched with the event information, the electronic device may display a prompt message or play a voice signal on the first application interface, where the prompt message and the voice signal are used to prompt the user that the event information cannot be found in the current interface.

For example, the interface data of the first application interface includes a plurality of interface elements such as "package a", "package B", and "package C", and also includes a control identifier "001" corresponding to "package a", a control identifier "002" corresponding to "package B", and a control identifier "003" corresponding to "package C". The electronic device searches a target interface element matched with the event information package A in the interface data of the first application interface, the target interface element is a first interface element in the first application interface, a control identifier corresponding to the first interface element is 001, and a control identifier corresponding to the event information package A is 001.

When finding the target interface element matched with the event information, the electronic device may determine an operable control corresponding to the target interface element from the interface data of the first application interface, and determine an identifier of the operable control as a control identifier corresponding to the event information. Correspondingly, when the electronic device finds the target interface element matched with the event information, the step of determining, by the electronic device, the control identifier corresponding to the event information according to the control identifier corresponding to the target interface element may be: when the electronic device finds the target interface element matched with the event information, the electronic device may determine, according to the target interface element, an operable control corresponding to the target interface element from the interface data of the first application interface, and determine a control identifier of the operable control as the control identifier corresponding to the event information.

For example, the interface data of the first application interface includes a user name input box, the voice signal received by the electronic device is "zhang san is input in the user name", the electronic device sends the voice signal to the first server, the execution event returned by the first server is received as "input", and the event information is "user name" and "zhang san". And the electronic equipment searches a target interface element matched with the user name from the interface data of the first application interface according to the event information and the interface data of the first application interface, and determines that the operable control corresponding to the interface element is an input box of the user name according to the interface element. And the electronic equipment determines that the control identification corresponding to the operable control is the control identification corresponding to the event information according to the operable control.

In the embodiment of the disclosure, the electronic device may determine the operable control corresponding to the target interface element according to the target interface element, and determine the control identifier of the operable control as the control identifier corresponding to the event information, so that the electronic device does not execute an invalid operation when executing a corresponding operation according to the control identifier in the operation instruction, thereby improving the efficiency of executing the operation.

In the embodiment of the disclosure, as the electronic device searches the target interface element matched with the event information from the interface data of the currently displayed first application interface, the generated operation instruction is matched with the currently displayed interface, and thus the electronic device executes the operation corresponding to the operation instruction more accurately.

In another possible implementation manner, if the target interface element corresponding to the event information cannot be found in the first application interface by the electronic device, the target interface element corresponding to the event information may be found from the interface data of the second application interface displayed in the history. Correspondingly, the step of matching the control identifier corresponding to the event information by the electronic device may further be: when the electronic device does not find the target interface element matched with the event information in the interface data of the first application interface, interface data of a second application interface displayed in a historical manner can be acquired, wherein the interface data of the second application interface comprises a plurality of interface elements and control identifications corresponding to the interface elements. The electronic device can search the interface data of the second application interface for the target interface element matched with the event information. When the electronic device finds the target interface element matched with the event information, the control identifier corresponding to the event information can be determined according to the control identifier corresponding to the target interface element.

The second application interface displayed in the history can be one or more interfaces displayed before the first application interface is displayed, and the second application interface can be converted to the first application interface through a jump or slide operation. The second application interface displayed in the history may also be one or more interfaces displayed within a preset time period before the first application interface is displayed. The preset time period may be 1 minute or 5 minutes, and the like, and is not limited in the embodiment of the present disclosure.

For example, when the user interacts with the electronic device through the voice signal, the user can control the interface to slide up and down or slide left and right through the voice signal sliding on the control interface. When the electronic equipment displays the second application interface, the electronic equipment receives a left-sliding voice signal, and slides the second application interface to the left to obtain the first application interface. Interface elements included in a first application interface currently displayed by the electronic equipment are 'a', 'b' and 'c', and interface elements included in a second application interface are 'b', 'e' and 'f'. If the electronic device receives a voice signal of "open b" while displaying the first application interface. The electronic device can search the target interface element 'b' in the interface data of the second application interface according to the event information 'b'.

In the embodiment of the disclosure, when the electronic device cannot find the target interface element matched with the event information in the interface data of the first application interface, the electronic device may also find the target interface element matched with the event information in the second application interface displayed in the history, so that the electronic device can cope with a more complex voice interaction scene, and thus, the experience of the user is improved.

In one possible implementation, the target interface element may correspond to one control identifier, and the target interface element may also correspond to a plurality of control identifiers. When the target interface element corresponds to a control identifier, the electronic device may determine the control identifier as a control identifier corresponding to the event information. Correspondingly, the step of determining, by the electronic device, the control identifier corresponding to the event information according to the control identifier corresponding to the target interface element may be: and when the target interface element corresponds to one control identification, the electronic equipment determines the control identification corresponding to the target interface element as the control identification corresponding to the event information.

For example, the voice signal received by the electronic device is "open file a", the execution event returned by the electronic device receiving the first server is "click", and the event information returned by the electronic device receiving the first server is "file a". The target interface element matched by the electronic device from the first application interface is 'file a', the target interface element is only corresponding to one control identifier '004', and the electronic device can use the control identifier '004' corresponding to the target element as the control identifier of the event information 'file a'.

In another possible implementation manner, when the target interface element corresponds to multiple control identifiers, the electronic device may determine, according to the execution event, a control identifier corresponding to the execution event from the multiple control identifiers corresponding to the target interface element. Correspondingly, the step of determining, by the electronic device, the control identifier corresponding to the event information according to the control identifier corresponding to the target interface element may further be: when the target interface element corresponds to the plurality of control identifiers, the electronic device may determine, according to the execution event, a control identifier corresponding to the execution event from the plurality of control identifiers corresponding to the target interface element.

For example, the interface data in the first application interface includes an interface element "package a", where the interface element corresponds to the first control identifier and the second control identifier. Wherein the first control identification is used for indicating that the number of the package A is increased by one, and the second control identification is used for indicating that the number of the package A is decreased by one. When the voice signal received by the electronic device is "i do not need package a", the execution event returned by the electronic device receiving the first server is "del (delete)", and the event information returned by the electronic device receiving the second server is "package a". The electronic equipment matches an interface element corresponding to a target interface element of package A from the interface data of the first application interface, wherein the target interface element corresponds to the first control identification and the second control identification. The electronic device may determine, from the control identifier corresponding to the target interface element, that the control identifier corresponding to the execution event is the second control identifier according to the execution event "del".

In the embodiment of the disclosure, when the target interface element corresponds to the plurality of control identifiers, the electronic device can determine the control identifier corresponding to the event information from the plurality of control identifiers according to the execution event, so that for an interface with a relatively complex interface element, the electronic device can also accurately determine the control identifier corresponding to the event information, thereby coping with a complex voice interaction scene and improving the experience of a user.

In step S306, the electronic device executes an operation corresponding to the operation instruction in the first application interface.

In the embodiment of the present disclosure, the operation instruction may include a control identifier corresponding to the execution event and the event information. The electronic equipment can automatically execute the operation corresponding to the operation instruction according to the operation instruction. Correspondingly, the steps can be as follows: the electronic equipment determines a control corresponding to the control identification from the operable controls in the first application interface according to the control identification in the operation instruction, and executes corresponding operation on the control corresponding to the control identification according to the execution event.

For example, the electronic device determines that the operation instruction is "click 004". The electronic equipment determines that the control corresponding to the control identifier "004" is the control corresponding to the "file a" from the operable controls in the first application interface according to the control identifier "004" in the operation instruction, and executes click operation on the control.

In the embodiment of the disclosure, since the electronic device receives the voice signal, and after the voice signal is converted into the operation instruction, the operation corresponding to the operation instruction can be automatically executed, so that the user does not need to perform manual operation, and the experience of the user is improved.

All the above optional technical solutions may be combined arbitrarily to form the optional embodiments of the present disclosure, and are not described herein again.

The fast Application related in the embodiments of the present disclosure is a novel Application form based on a hardware platform of an electronic device, and can run based on a native API (Application Programming Interface) provided by an operating system.

The fast application is developed by adopting a unified standard, and is rendered as native without installation, namely, the fast application is used as point, and has native application experience (performance, system integration, interaction and the like). The framework of the fast application can be deeply integrated in an operating system, can operate at the level of the operating system, and realizes seamless connection with other application services. Developers can operate on different types of electronic equipment through one-time development. The method is applied to the electronic equipment manufacturers at the beginning of birth, realizes standardization unification among the electronic equipment manufacturers on the aspects of development specifications, capability access, developer service and the like, and greatly reduces the adaptation cost of developers.

Compared with the traditional application, the quick application has the following characteristics:

instant: the user can use the method point by point, and the user does not need to wait;

everywhere (Everywhere visible): deeply integrated with the use scene of the electronic equipment, and everywhere has an entrance, such as a search engine, an intelligent assistant, intelligent recommendation, an application market, a browser and the like;

efficiency: the quasi-front-end development mode is high in efficiency.

FIG. 4 is a block diagram illustrating a fast application framework 400 according to an example embodiment, the fast application framework including: scenario portal 420, fast application engine 440, and Operating System (OS) infrastructure and hardware 460.

The scene portal 420 includes at least one of a negative one-screen, a global search, a lock screen, a desktop, an application marketplace, a browser, and a two-dimensional code. The appearance of the scene portal 420 may be in the form of a page and a card.

The fast application engine 440 includes a front end framework 441, a generic scenario 442, a lightweight scenario 443, an embedded SDK (Software Development Kit) 444, and business access 445.

The front-end framework 441 includes an MVVM (Model-View-Model ), a V-DOM (Virtual-Document Object Model), a route, an Application Programming Interface (API), a service API, a User Interface (UI) component, a route, and the like;

the general scene 442 and the lightweight scene 443 include a JavaScript (a programming language) engine, a standard rendering engine, an extreme speed rendering engine, end-cloud-core acceleration, a security mechanism, emerging scenes such as AI (Artificial Intelligence), AR (Augmented Reality), and the like, system integration (application management, rights management, and the like);

service access 445 includes push, account/payment, etc.

OS infrastructure and hardware 460 includes: a graphics library, a native control, a system service, and a GPU (graphics Processing Unit)/NPU (embedded Neural network processor), etc.

From the execution path level, there are a standard HTML (Hyper Text Markup Language) 5 mode to support a general Web scene, and a JS (JavaScript, a programming Language) and Native (Native) mode to support a lighter and faster experience.

The architecture of the fast application engine will be briefly described in 3 aspects below.

1) Application development:

the design of the front end of the fast application refers to and integrates the design thought of the mainstream front end framework: the application is built in a componentized mode, an MVVM design mode taking data binding as a core is adopted, the performance is improved in a V-DOM mode, and meanwhile, a simple and clear template of a class Vue (a virtual machine shelling engine) is selected. Meanwhile, the layout aspect is correspondingly simplified. From the perspective of new application form, native UI mapping and capability opening, a set of components and API specifications need to be defined, and the rapid development application is convenient to develop.

2) System integration:

fast applications, as a complete application modality, can be deeply integrated with the system, run as native applications, and interact with the system. There are currently two forms of rapid application: a full screen mode independent application mode and an embedded mode card mode. Under the independent application form, the experience of the user is similar to a native application program, and the method has complete life cycle management, page management, routing and the like. The fast application can be hosted in the Activity of android, the page is hosted in Fragment, and the instance is managed and controlled through independent background Service. The card form is another form, and dynamic content is displayed in a light weight manner by embedding the embedded SDK (Software Development Kit) as an independent local control into each corner of the system. In the aspect of security isolation, better security guarantee can be achieved through a sandbox mechanism, process isolation and authority control in combination with the support of an operating system layer.

3) Performance experience and emerging scenes, such as JavaScript engines, rendering engines, end-cloud-core acceleration, emerging scenes, and the like:

in the aspects of interactive experience, resource overhead, stability and the like, the fast application realizes effective combination of a front-end development mode, native rendering and platform capability by introducing a native rendering path.

Different from cross-platform frameworks of other application layers, the application is planted in an operating system of the electronic equipment, and deep integration of a slave chip, the operating system and the cloud can be realized. By taking the acceleration of the starting performance as an example by combining the end and the cloud, the optimization of the network link layer can greatly accelerate the starting speed of the application through the cooperative rendering of the cloud and the end. Meanwhile, the special capability of the hardware platform can be integrated, and the experience is further improved. For example, the computing power of an NPU (Neural-network processing Unit, network processor) may be integrated into a fast application engine in combination with an AI chip of an electronic device, so that an AI scene (face recognition, image super-resolution, and the like) may be executed at a low latency and high performance on an end side, and meanwhile, the privacy of a user is effectively protected, and a bandwidth is saved.

FIG. 5 is a flowchart illustrating the launching of a fast application, according to an example embodiment, including:

1) when the system is started for the first time, a user clicks a program package for triggering downloading of the fast application, and meanwhile, initialization related work of the fast application engine is carried out. After the whole fast application program package is downloaded and verified, the JavaScript file of the first page to be displayed is loaded and rendered.

2) The page rendering comprises JavaScript loading, execution of page and JavaScript frame logic, layout operation and drawing of the native UI control. When the logic in the page is executed, one or more network requests are generated, and the data returned by the network requests drive the page to be rendered again until the content of the page on the first screen is completely displayed.

The network request, JavaScript execution, typesetting and drawing are not in simple serial relation, but are parallelly interwoven, thereby affecting the rendering performance of the whole page and being strongly related to the logic and network conditions of page design and the running state of equipment.

Fig. 6 is a block diagram illustrating an electronic device operating apparatus according to an example embodiment. Referring to fig. 6, the apparatus includes:

a display module 601 configured to display a first application interface of a first target application;

an obtaining module 602, configured to, when an input voice signal is received, obtain, by a first server, an execution event and event information corresponding to the voice signal;

a generating module 603 configured to generate an operation instruction of the voice signal according to the execution event, the event information, and the interface data of the first application interface;

and the execution module 604 is configured to execute an operation corresponding to the operation instruction in the first application interface.

In a possible implementation manner, the generating module 603 is further configured to match a control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface; and generating an operation instruction according to the execution event and the control identification.

In another possible implementation manner, the generating module 603 is further configured to include, in the interface data of the first application interface, a plurality of interface elements and a control identifier corresponding to each interface element; searching a target interface element matched with the event information in the interface data of the first application interface; and when the target interface element matched with the event information is found, determining the control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the generating module 603 is further configured to, when a target interface element matched with the event information is not found in the interface data of the first application interface, obtain interface data of a second application interface displayed in history, where the interface data of the second application interface includes a plurality of interface elements and a control identifier corresponding to each interface element; searching a target interface element matched with the event information in the interface data of the second application interface; and when the target interface element matched with the event information is found, determining the control identification corresponding to the event information according to the control identification corresponding to the target interface element.

In another possible implementation manner, the generating module 603 is further configured to determine, when the target interface element corresponds to one control identifier, the control identifier corresponding to the target interface element as the control identifier corresponding to the event information; and when the target interface element corresponds to the plurality of control identifications, determining the control identification corresponding to the execution event from the plurality of control identifications corresponding to the target interface element according to the execution event.

In another possible implementation manner, the display module 601 is further configured to send a first obtaining request to a first server, and send a second obtaining request to a second server corresponding to the first target application; receiving style data of a first application interface returned by a first server, and receiving interface data of the first application interface returned by a second server; and rendering the first application interface according to the style data and the interface data.

In another possible implementation manner, the display module 601 is further configured to display a third application interface of the second target application installed locally, where the third application interface includes an application interface of the first target application not installed locally; and starting the first target application program when detecting that the application interface of the first target application program is triggered.

In another possible implementation manner, the obtaining module 602 is further configured to send a voice signal to a first server, where the first server is configured to identify, according to the voice signal and through a voice recognition rule, an execution event and event information corresponding to the voice signal; and receiving the execution event and the event information returned by the first server.

It should be noted that: in the electronic device operating apparatus provided in the above embodiment, when the electronic device is operated, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the electronic device may be divided into different functional modules to complete all or part of the functions described above. In addition, the electronic device operating apparatus provided in the above embodiments and the electronic device operating method embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.

Fig. 7 is a block diagram illustrating an electronic device 700 in accordance with an example embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 7, electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input/output (I/O) interface 712, a sensor component 714, and a communication component 716.

The processing component 702 generally controls overall operation of the electronic device 700, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to perform all or a portion of the steps of the above-described method of electronic device operation. Further, the processing component 702 may include one or more modules that facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.

The memory 704 is configured to store various types of data to support operations at the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 704 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 706 provides power to the various components of the electronic device 700. The power components 706 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 700.

The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 700 is in an operation mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 710 is configured to output and/or input audio signals. For example, the audio component 710 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 700 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 704 or transmitted via the communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

The I/O interface 712 provides an interface between the processing component 702 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 may detect an open/closed state of the electronic device 700, the relative positioning of components, such as a display and keypad of the electronic device 700, the sensor assembly 714 may also detect a change in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, orientation or acceleration/deceleration of the electronic device 700, and a change in the temperature of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a Near Field Communication (NFC) module to facilitate short-range communications.

In an exemplary embodiment, the electronic device 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described electronic device operation methods.

In an exemplary embodiment, a non-transitory computer readable storage medium comprising instructions, such as the memory 704 comprising instructions, executable by the processor 720 of the electronic device 700 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

The present disclosure also provides a computer program product comprising one or more computer programs for implementing the method of operation of the electronic device provided by the above-mentioned method embodiments when executed by a processor.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is intended to be exemplary only and not to limit the present disclosure, and any modification, equivalent replacement, or improvement made without departing from the spirit and scope of the present disclosure is to be considered as the same as the present disclosure.

Claims

1. A method of electronic device operation, the method comprising:

displaying a first application interface of a first target application program;

2. The method according to claim 1, wherein the generating the operation instruction of the voice signal according to the execution event, the event information and the interface data of the first application interface comprises:

3. The method according to claim 2, wherein the matching, in the first application interface, the control identifier corresponding to the event information according to the event information and the interface data of the first application interface includes:

4. The method of claim 3, further comprising:

5. The method according to claim 3 or 4, wherein the determining, according to the control identifier corresponding to the target interface element, the control identifier corresponding to the event information includes:

6. The method of claim 1, wherein displaying the first application interface of the first target application comprises:

7. The method of claim 6, further comprising:

8. The method of claim 1, wherein the obtaining, by the first server, the execution event and event information corresponding to the voice signal comprises:

9. An electronic device operating apparatus, characterized in that the apparatus comprises:

10. The apparatus according to claim 9, wherein the generating module is further configured to match a control identifier corresponding to the event information in the first application interface according to the event information and the interface data of the first application interface; and generating the operation instruction according to the execution event and the control identification.

11. The apparatus of claim 10, wherein the generating module is further configured to include a plurality of interface elements and a control identifier corresponding to each interface element in the interface data of the first application interface; searching a target interface element matched with the event information in the interface data of the first application interface; and when the target interface element matched with the event information is found, determining a control identification corresponding to the event information according to the control identification corresponding to the target interface element.

12. The apparatus according to claim 11, wherein the generating module is further configured to, when a target interface element matching the event information is not found in the interface data of the first application interface, obtain interface data of a second application interface displayed in history, where the interface data of the second application interface includes a plurality of interface elements and a control identifier corresponding to each interface element; searching a target interface element matched with the event information in the interface data of the second application interface; and when the target interface element matched with the event information is found, determining a control identification corresponding to the event information according to the control identification corresponding to the target interface element.

13. The apparatus according to claim 11 or 12, wherein the generating module is further configured to, when the target interface element corresponds to one control identifier, determine the control identifier corresponding to the target interface element as the control identifier corresponding to the event information; and when the target interface element corresponds to the plurality of control identifications, determining the control identification corresponding to the execution event from the plurality of control identifications corresponding to the target interface element according to the execution event.

14. The apparatus of claim 9, wherein the display module is further configured to send a first get request to the first server and send a second get request to a second server corresponding to the first target application; receiving style data of the first application interface returned by the first server, and receiving interface data of the first application interface returned by the second server; and rendering the first application interface according to the style data and the interface data.

15. The apparatus of claim 14, wherein the display module is further configured to display a third application interface of a second target application installed locally, the third application interface comprising an application interface of the first target application not installed locally; and starting the first target application program when detecting that the application interface of the first target application program is triggered.

16. The apparatus according to claim 9, wherein the obtaining module is further configured to send the voice signal to the first server, and the first server is configured to identify, according to the voice signal, an execution event and event information corresponding to the voice signal through a voice recognition rule; and receiving the execution event and the event information returned by the first server.

17. An electronic device, characterized in that the electronic device comprises:

one or more processors;

wherein the one or more processors are configured to:

displaying a first application interface of a first target application program;

18. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processor of an electronic device, enable the electronic device to perform the method of any one of claims 1-8.