CN116312514A

CN116312514A - Display apparatus and voice control method

Info

Publication number: CN116312514A
Application number: CN202310155184.8A
Authority: CN
Inventors: 付友苹; 付延松; 卢可敬
Original assignee: Hisense Visual Technology Co Ltd
Current assignee: Hisense Visual Technology Co Ltd
Priority date: 2023-02-22
Filing date: 2023-02-22
Publication date: 2023-06-23

Abstract

The application provides a display device and a voice control method, wherein the display device comprises a display, a sound collector and a controller respectively connected with the display and the sound collector; wherein the display is configured to display an image frame and a user interface; the sound collector is configured to collect voice control instructions of a user, and the controller is configured to: determining at least one scrolling control contained in the current display interface; constructing a voice scrolling control word list of a scrolling control in a current display interface; the voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control word; and responding to the voice control instruction of the user, and controlling the target scrolling control in the current display interface to execute scrolling operation based on the voice scrolling control word list and the voice control text corresponding to the voice control instruction. Therefore, voice control of the scrolling control is realized, flexibility and convenience of a control mode of the display equipment are improved, and user experience is improved.

Description

Display apparatus and voice control method

Technical Field

The present disclosure relates to the field of voice control technologies, and in particular, to a display device and a voice control method.

Background

With the development of voice control technology, voice functions are integrated in each display device (for example, smart televisions), and a user can jump to/open a designated application through voice instructions.

In general, the display interface of each display device is maintained with a plurality of controls capable of responding to the voice control instruction, and the controls can display corresponding interface descriptors to the outside. After receiving a voice control instruction of a user, the display equipment matches the voice control text corresponding to the voice control instruction with interface description words of all the controls in the current display interface one by one, and if the interface description words are matched, controls the controls corresponding to the interface description words to execute related operations. For example, if a resource library control is displayed in the current display interface, the display device may automatically execute the selected resource library control in response to a voice control instruction of "open the resource library", thereby opening the resource library to display multiple multimedia resources for selection by the user.

However, some controls without interface descriptors, such as scroll controls, may exist in some display interfaces. Because the scrolling control has no interface descriptor and the scrolling direction is not unique, the interface descriptor cannot be subjected to voice control in a matching mode.

Disclosure of Invention

The application provides a display device and a voice control method, which can effectively control a scroll control without an interface descriptor in a display interface.

In a first aspect, the present application provides a display device, including a display, a sound collector, and a controller connected to the display and the sound collector, respectively, wherein:

a display configured to display an image screen and a user interface;

a sound collector configured to collect a voice control instruction of a user;

the controller is configured to:

determining at least one scrolling control contained in the current display interface;

constructing a voice scrolling control word list of a scrolling control in a current display interface; the voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control word;

responding to a voice control instruction of a user, and controlling a target rolling control to execute rolling operation according to a rolling direction indicated by the voice control text based on a voice rolling control word list and a voice control text corresponding to the voice control instruction; the target scroll control is one of the scroll controls contained in the current display interface, which corresponds to the voice control text.

In some of these embodiments, the controller is further configured to:

monitoring the change condition of a display interface of the target application after the target application is started;

if the display interface changes, acquiring control information of a plurality of controls contained in the changed current display interface;

and determining at least one scrolling control contained in the current display interface according to the control information.

In some of these embodiments, the controller is further configured to:

acquiring display positions of a plurality of child controls controlled by the rolling control in a current display interface;

determining the rolling direction of the rolling control according to the display position of each sub-control;

and constructing a voice scrolling control word list of the scrolling control in the current display interface according to the scrolling direction.

In some of these embodiments, the controller is further configured to:

if the display heights of all the child controls in the current display interface are uniform, determining that the rolling direction of the rolling control is horizontal rolling;

if the display widths of all the sub-controls in the current display interface are the same, determining that the rolling direction of the rolling control is vertical rolling;

and if the display heights and/or the display widths of the child controls in the current display interface are different, determining that the rolling direction of the rolling control is waterfall flow rolling.

In some of these embodiments, the controller is further configured to:

acquiring a voice control text corresponding to the voice control instruction;

matching the voice control text with semantic control words in the voice scrolling control word list, and determining a target scrolling control of which the voice control text requests control in a current display interface and a target scrolling direction of the target scrolling control;

and controlling the target scrolling control to execute scrolling operation in the target scrolling direction according to the voice control text.

In some of these embodiments, the controller is further configured to:

acquiring a target rolling distance of a target rolling control according to the voice control text;

the control target scroll control performs a scroll operation in a target scroll direction in accordance with the target scroll distance.

In some of these embodiments, the voice control text comprises page-turning voice text, and the controller is further configured to:

if the target rolling direction is horizontal rolling, calculating a target rolling distance according to the display position of the rightmost child control and the display position of the leftmost child control controlled by the target rolling control;

if the target rolling direction is vertical rolling or waterfall flow rolling, calculating the target rolling distance according to the display position of the uppermost child control and the display position of the lowermost child control controlled by the target rolling control.

In some of these embodiments, if the current display interface includes a plurality of candidate scroll controls corresponding to the speech control text, the controller is further configured to:

acquiring control priority of each candidate rolling control;

a target scroll control is determined from the plurality of candidate scroll controls according to the control priority.

In some of these embodiments, the controller is further configured to:

the voice control instruction is sent to the cloud server to request the cloud server to analyze the voice control instruction;

and receiving a voice control text corresponding to the voice control instruction sent by the cloud server.

In a second aspect, the present application provides a voice control method, including:

In a third aspect, the present application also provides a computer readable storage medium having stored therein a computer program which, when executed by a controller in a display device, implements some or all of the steps of the speech control method provided herein.

In a fourth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a controller in a display device, implements some or all of the steps of the speech control method provided herein.

The technical scheme provided by the application can at least achieve the following beneficial effects:

in the display device and the voice control method provided by the application, a controller in the display device is respectively connected with a display and a voice collector, and in the process that the display displays an image picture and a user interface, the controller determines at least one rolling control included in the current display interface and builds a voice rolling control word list of the rolling control in the current display interface. The voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control words. Further, after detecting and collecting a voice control instruction of a user, the voice collection device sends the voice control instruction to the controller, so that the controller responds to the voice control instruction and controls a target scrolling control corresponding to the voice control instruction to execute scrolling operation in a current display interface based on a pre-constructed voice scrolling control word list and voice control text corresponding to the voice control instruction. Therefore, by recognizing the scrolling control in advance and constructing a voice scrolling control word list for the scrolling control, when the display device receives a voice control instruction of a user, the target scrolling control in the current display interface is controlled to execute scrolling operation according to the voice control instruction, the defect that the scrolling control cannot be directly controlled through voice is overcome, full voice control of each control in the display device is realized, flexibility and convenience of a control mode of the display device are improved, and user experience is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the implementation in the related art, a brief description will be given below of the drawings required for the embodiments or the related art descriptions, and it is apparent that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings for those of ordinary skill in the art.

FIG. 1 is a diagram illustrating an operational scenario between a display device and a control apparatus according to some embodiments of the present application;

FIG. 2 is a block diagram of a hardware configuration of a display device shown in some embodiments of the present application;

FIG. 3 is a software configuration block diagram of a display device shown in some embodiments of the present application;

FIG. 4 is a control distribution diagram of a display interface shown in some embodiments of the present application;

FIG. 5 is a flow chart illustrating a method for performing voice control by a display device according to some embodiments of the present application;

FIG. 6 is a schematic horizontal scrolling view of a scroll control as shown in some embodiments of the present application;

FIG. 7 is a schematic vertical scroll of a scroll control as shown in some embodiments of the present application;

FIG. 8 is a waterfall flow scrolling schematic of a scrolling control as shown in some embodiments of the present application;

FIG. 9 is a flow diagram of a voice-controlled scroll control performing a scroll operation as illustrated in some embodiments of the present application;

FIG. 10 is a schematic diagram of a multi-scroll control distribution of a display interface shown in some embodiments of the present application;

FIG. 11 is a schematic diagram of voice control logic for a scroll control in a display interface as illustrated in some embodiments of the present application.

Detailed Description

For the purposes of making the objects and embodiments of the present application more clear, the exemplary embodiments of the present application will be clearly and fully described below with reference to the accompanying drawings in which exemplary embodiments of the present application are shown, it being apparent that the exemplary embodiments described are only some, but not all, of the embodiments of the technical solutions of the present application.

It should be noted that the brief description of the terms in the present application is only for convenience in understanding the embodiments described below, and is not intended to limit the embodiments of the present application. Unless otherwise indicated, these terms should be construed in their ordinary and customary meaning.

The terms "comprising" and "having" and variations thereof, as used in the description of the present application and in the claims, are intended to cover a non-exclusive inclusion, such that a product/apparatus comprising a list of elements is not necessarily limited to all elements that are expressly listed or may include other elements not expressly listed or inherent to such product/apparatus.

In this embodiment, the display device may have various implementation forms, for example, may be a television, a smart television, a laser projection device, a display (monitor), an electronic whiteboard (electronic bulletin board), an electronic desktop (electronic table), or may be a personal computer, a notebook computer, a smart phone, a tablet computer, a portable wearable device, or the like.

The display device may be controlled by a control device, an intelligent control device, voice, motion, gesture, trigger motion, and the like.

Referring to fig. 1, in one exemplary control operation scenario between a display device and a control apparatus, a user may operate the display device 200 through the smart device 300 or the control apparatus 100.

In some embodiments, the control apparatus 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or bluetooth protocol communication, and other short-range communication modes, and the display device 200 is controlled by a wireless or wired mode. The user may control the display device 200 by inputting user instructions through keys on a remote control, voice input, control panel input, etc.

In some embodiments, a smart device 300 (such as a mobile terminal, tablet, computer, notebook, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on a smart device.

In some embodiments, the display device 200 may receive the instruction not using the smart device 300 or the control apparatus 100, but by touching or gesture, etc.

In some embodiments, the display device 200 may also be controlled in ways other than the control apparatus 100 and the smart device 300. For example, the voice command control of the user may be directly received through a module for acquiring voice commands configured inside the display device 200; the voice command control of the user may also be received through a voice control device provided outside the display device 200 device.

In some embodiments, the display device 200 is also in data communication with the server 400, which may allow the display device 200 to be communicatively connected via a local area network (Local Area Network, LAN), wireless local area network (Wireless Local Area Network, WLAN), or other network, etc.

Among other things, the server 400 may provide various contents and interactions to the display device 200.

As an example, the server 400 may be a stand-alone server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

Referring to fig. 3, the display device 200 may include: at least one of a modem 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory 280, a power supply, a user interface.

In some embodiments, the modem 210 receives broadcast television signals via wired or wireless reception and demodulates audio-video signals from a plurality of wireless or wired broadcast television signals. Such as electronic program guide (Electronic Program Guide, EPG) data signals.

In some embodiments, the modem 210 and the controller 250 may be located in separate devices, i.e., the modem 210 may also be located in an external device to the host device in which the controller 250 is located, such as an external set-top box or the like.

In some embodiments, communicator 220 is a component for communicating with external devices or servers 400 according to various communication protocol types. For example: the communicator may include at least one of a Wifi module, a bluetooth module, a wired ethernet module, or other network communication protocol chip or a near field communication protocol chip, and an infrared receiver.

In some embodiments, the display apparatus 200 may establish transmission and reception of control signals and data signals with the external control apparatus 100 or the server 400 through the communicator 220.

For example, the display device may communicate with the cloud server through the communicator, send data information to be processed to the cloud server, and obtain the data information processed by the cloud server.

In some embodiments, the detector 230 is used to collect signals of the external environment or the display device 200 interacts with the outside. For example, the detector 230 includes an image collector, such as a camera, that may be used to collect external environmental scenes, attributes of a user, or user interaction gestures; still alternatively, the detector 230 includes a sound collector, such as a microphone, for collecting external sound.

In the interactive scenario of the voice control display device shown in the application, the sound collector may be used to collect voice control instructions of a user.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: high definition multimedia interface (High Definition Multimedia Interface, HDMI), analog or data high definition component input interface (component), composite video input interface (Composite Video Broadcast Signal, CVBS), serial communication bus (Universal Serial Bus, USB) input interface, RGB port, or the like. The input/output interface may be a composite type input/output interface formed by a plurality of interfaces.

In some embodiments, the controller 250 includes: at least one of a central processing unit (Central Processing Unit, CPU), a video processor, an audio processor, a graphics processor (Graphics Processing Unit, GPU), a RAM (Random Access Memory, RAM), a ROM (Read-Only Memory), a first interface to an nth interface for input/output, a communication Bus (Bus), and the like.

Among them, the controller 250 may control the operation of the display device 200 and respond to the user's operations through various software control programs stored on the memory 280. The controller 250 controls the overall operation of the display apparatus 200. For example: in response to receiving a user command to select a UI object to be displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.

In the interactive scene of the voice control display device, the controller is configured to determine at least one scrolling control contained in the current display interface and construct a voice scrolling control word list of the scrolling control in the current display interface; and further, after the voice collector receives the voice control instruction of the user, based on the voice control word list and the voice control text corresponding to the voice control instruction, in the rolling control contained in the current display interface, controlling the target rolling control corresponding to the voice control text to execute the rolling operation according to the rolling direction indicated by the voice control text.

In some embodiments, display 260 includes a display screen assembly for presenting pictures, and a drive assembly to drive the display of images. The display 260 is used for receiving the image signals output by the controller 250, displaying video content, image content, and components of the menu manipulation interface, and manipulating the UI interface by the user.

As one example, the display 260 may be a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

Wherein a user may input a user command through a graphical user interface (Graphic User Interface, GUI) displayed on the display 260, the user interface receives the user input command through the GUI. Alternatively, the user may enter a user command by entering a specific sound or gesture, and the user interface receives the user input command by recognizing the sound or gesture through the detector.

In the interactive scenario of the voice-controlled display device shown herein, the display 260 is configured to display an image screen and a user interface.

Where a "user interface" is a media interface for interaction and exchange of information between an application or operating system and a user, it enables conversion between an internal form of information and a form acceptable to the user.

As one example, the user interface may be an interface element of an icon, window, control, etc., displayed in the display of the display device 200.

The controls may include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and the like.

Referring to fig. 3, taking an example of deploying an Android software platform in the display device 200, the display device 200 divides an Android operating system into four layers, namely, an application (application, abbreviated as "application layer"), an application framework (Application Framework), a An Zhuoyun-line (Android run) and a system library layer (abbreviated as "system runtime layer"), and a kernel layer from top to bottom.

In some embodiments, at least one application program is running in the application layer, and the application programs may be a Window (Window) program, a system setting program, a clock program or the like of the android operating system; or may be an application developed by a third party developer.

It should be appreciated that the applications included in the application layer are not limited to the above examples, but may include other applications.

The framework layer provides an application programming interface (application programming interface, API) and programming framework for the application, and the application framework layer includes some predefined functions.

In other words, the framework layer corresponds to a processing center that determines the application in the application layer to act upon. Through the API interface, the application program can access the resources in the android system and acquire the services of the system in the execution.

In some embodiments, the framework layer includes a manager (manager), a Content Provider (Content Provider), a View component system (View system), and the like.

Wherein the manager comprises at least one of the following modules: an Activity Manager (Activity Manager) is used to interact with all activities running in the system; a Location Manager (Location Manager) is used to provide system services or applications with access to system Location services; a Package Manager (Package Manager) for retrieving various information about an application Package currently installed on the device; a notification manager (Notification Manager) for controlling the display and clearing of notification messages; a Window Manager (Window Manager) is used to manage bracketing icons, windows, toolbars, wallpaper, and desktop components on the user interface.

In some embodiments, the activity manager is used to manage the lifecycle of the individual applications and the usual navigation rollback functions, such as controlling the exit, opening, fallback, etc. of the applications; the window manager is used to manage all window programs, such as obtaining the size of the display screen, judging whether a status bar exists, locking the screen, intercepting the screen, controlling the change of the display window (for example, reducing the display window to display, dithering, distorting, etc.), etc.

In some embodiments, the system runtime layer provides support for upper layers (i.e., the framework layer), and when the framework layer is in use, the android operating system runs the C/C++ libraries contained in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. The kernel layer contains at least one of the following drivers: audio drive, display drive, bluetooth drive, camera drive, WIFI drive, USB drive, HDMI drive, sensor drive (e.g., fingerprint sensor, temperature sensor, pressure sensor, etc.), and power supply drive, etc.

After the control interaction scene, the hardware structure and the deployed software platform of the display device are introduced, a detailed description will be given of a specific implementation process of executing the voice control method by the display device based on the display device 200.

With the development of intelligent informatization, the wider the voice control scene and coverage range, the larger the market share is. Voice functionality is also integrated into each display device, and a user can view various multimedia resources through voice control instructions and jump to/open a designated application.

Between applications in the display device, voice control may be achieved by the direct word to achieve direct jumps/switches between applications installed in the display device. That is, when the voice control text corresponding to the voice control instruction input by the user is a direct word of a certain application, the display device jumps from the current application to the target application indicated by the voice control text corresponding to the voice control instruction in response to the voice control instruction.

As one example, when a user needs to exit from a video application currently being browsed to a home page, the user may be instructed to switch from the currently running video application to the home page by entering a voice control instruction "back to home page" to facilitate the user's selection of other applications/multimedia resources of interest in the home page.

It should be understood that the home page is the user interface that presents and directly reaches the application interfaces. The user may browse the recommended multimedia assets and applications in the homepage to select the multimedia assets of interest therefrom for playback, or open the applications of interest to play/browse related content within the applications.

However, for a single application, because various scenes exist in the application, voice control cannot be realized among the scenes through the direct words, and control can only be realized through interface descriptors in different display interfaces. That is, when the voice control text corresponding to the voice control instruction input by the user is the interface description word of a control in the current display interface, the display device responds to the voice control instruction to open/select the target control indicated by the voice control text corresponding to the voice control instruction in the current display interface.

Because the scene in the application is relatively fixed, a voice interface control word list corresponding to each display interface in the application can be constructed in the first loading process after each application update, so that after a voice control instruction of a user is received, a target control which needs to be requested to be operated by the user is quickly determined in the current display interface based on the voice interface control word list.

As an example, after receiving a voice control instruction of a user, the display device matches a voice control text corresponding to the voice control instruction with interface descriptors of all controls in a voice interface control word list one by one, and if the voice control text is matched with the corresponding interface descriptors, opens/selects a target control corresponding to the interface descriptors in a current display interface.

In addition, in some display interfaces, in addition to the above-mentioned controls carrying interface descriptors, some controls without interface descriptors, such as scroll controls, are included in the display interfaces.

It should be noted that, the display interface including the scroll control in the display device may include at least one of a homepage, a web page, and a user interface in an application, which is not limited in the embodiment of the present application.

The scrolling control is used for realizing sliding and page turning operations. Because the scrolling control has no interface descriptor and the scrolling direction is not unique, it is difficult to perform voice control on the interface descriptor by constructing a voice interface control word list and matching the interface descriptor.

For easy understanding, referring to fig. 4, taking a display interface as an example of a device homepage, the display interface includes a plurality of controls such as searching, time, VIP, footprint, news, history, selection, multimedia resources (for short, media resources), etc., and a user may select a target control from the controls displayed in the display interface by using the control means 100, the intelligent device 300, or the voice control instruction, etc., so as to open a content display interface of the target control.

As an example, if the user needs to operate the a control in the display interface shown in fig. 4, the user inputs a voice control instruction for opening VIP to the display device, after receiving the voice control instruction, the display device traverses interface description words of the controls in a voice interface control word list of the display interface according to a voice control text corresponding to the voice control instruction, and after the control corresponding to the interface description word matched to "opening VIP" is the a control, the a control is operated to open a content display interface for opening VIP, so that the user can further execute related operations for opening VIP rights in the content display interface.

However, the scroll controls have no corresponding interface descriptor, the number of the scroll controls included in one display interface may be different, and the scroll direction of each scroll control is not unique, so that the target scroll control which needs to be operated by the user cannot be determined from the display interface in a manner of matching the interface descriptor according to the voice control instruction of the user.

With continued reference to FIG. 4, for a scroll control in the display interface that does not carry an interface descriptor: the control B and the control C can slide upwards or downwards, the control C can slide leftwards or rightwards, and the control B and the control C are displayed only by one sliding bar in the display interface and have no corresponding interface description words, so that it is difficult to determine that the target rolling control to be operated is the control B or the control C in the display interface according to a voice control instruction of a user.

Based on this, the present application provides a voice control method, so as to identify a scroll control in a current display interface, and construct a voice scroll control word list of the scroll control in the current display interface, so that after receiving a voice control instruction of a user, the display device 200 may determine, according to the voice scroll control word list and a voice control text corresponding to the voice control instruction, a target scroll control to be operated in the current display interface, and control the target scroll control to execute a scroll operation according to a scroll direction indicated by the voice control text. Thus, voice control of the scroll control in the display device is realized.

Referring to fig. 5, when the above-mentioned display apparatus 200 performs the voice control method provided in the present application, the controller 250 in the display apparatus 200 is configured to perform the steps of:

step 510: at least one scroll control contained in the current display interface is determined.

It should be appreciated that there may be no scroll control in the current display interface, either one scroll control or multiple scroll controls. If the scrolling control does not exist in the current display interface, the user can directly control the corresponding control to execute operations such as opening/selecting according to interface description words of the controls in the current display interface by direct voice.

If the rolling control exists in the current display interface, the technical scheme provided by the application is executed to determine at least one rolling control contained in the current display interface, and a voice rolling control word list of the rolling control in the current display interface is constructed to realize voice control of the rolling control in the current display interface to execute rolling operation.

As previously explained, the current display interface in embodiments of the present application includes at least one of a home page, a web page, and a user interface within an application.

In some embodiments, if the current display interface is a home page, when performing step 510 described above, the controller is further configured to: after the display device is powered on and turned on, the scrolling control included in the homepage is obtained, and the scrolling control in the homepage is controlled by voice to execute the scrolling operation through the following steps 520-530.

In some embodiments, if the current display interface is a web page, when performing step 510 described above, the controller is further configured to: when the skip operation based on the webpage link is detected, acquiring webpage related information of the skip webpage by adopting a web crawler technology, and acquiring webpage content of the webpage by analyzing the webpage related information, so as to judge whether a scroll control exists in the webpage according to the webpage content.

As one example, the total height of all content items in the web page is determined from the web page content. If the total height of the content item is larger than the window height value of the display interface, a scrolling control exists in the webpage, and the scrolling direction of the scrolling control is vertical scrolling; if the total height of the content item is not greater than the window height value of the display interface, no scroll control is present in the web page.

If it is determined that the scroll control exists in the web page, the following steps 520-530 are implemented to implement the scroll operation by using the scroll control in the voice-controlled web page.

In some embodiments, if the current display interface is a user interface within an application, when performing step 510 described above, the controller is further configured to: monitoring the change condition of a display interface of the target application after the target application is started; if the display interface changes, acquiring control information of a plurality of controls contained in the changed current display interface; and determining at least one scrolling control contained in the current display interface according to the control information.

It should be noted that, the target application may be a system application of the display device 200, an application installed in the display device 200 by a user, or a third party application started through a network link, and the embodiment of the present application does not limit the source and type of the target application.

That is, the target application in the present application is any application that the user triggers to start in the display device 200.

In one possible implementation, the target application in the display device, after being launched, registers an auxiliary function service (Accessibility Service) in the framework layer of the android system, which is able to receive some events triggered by the android system, such as notification status, view-related events, fingerprints, key clicks (touch), etc.

Therefore, when the application program of the target application starts to run, the auxiliary function service monitors the change condition of the display interface of the target application in real time after the target application is started. When the display interface changes, the callback interface of the auxiliary function service receives the interface change notification. Therefore, through the interface provided by the auxiliary function service, the Root view Node (Node Root) control of the changed current display interface and the child view Node controls corresponding to the Root view Node controls can be obtained, and finally all the controls contained in the current display interface are obtained.

Wherein, each control is correspondingly stored with control information such as type, function description information and the like in the display equipment. In the display device, control information of each control is stored in the form of a structure tree. Therefore, based on the characteristics, the control information of all the controls in the current display interface can be obtained through the structure tree of the current display interface.

Further, a control Node List (Node List) corresponding to the current display interface can be generated according to control information of all controls contained in the current display interface.

It should be noted that the control node list includes a control with an interface descriptor and a scroll control without an interface descriptor in the current display interface.

Therefore, the control Node List can be traversed, and each control in the control Node List is subjected to type judgment so as to determine a Scroll control Node List (Scroll Node List) corresponding to the current display interface.

In one possible implementation manner, a corresponding interface (such as an iscrol interface) is called through an auxiliary function service, whether the control corresponding to each node supports scrolling is judged in the structural tree of the current display interface, if yes, the control is determined to be a scrolling control, and the scrolling control is added to the scrolling control node list.

In some embodiments, if the scroll control included in the display interface is relatively fixed (for example, the display position, the type, the function information, and the like of the scroll control are not changed in a preset period of time), after determining the scroll control corresponding to the display interface for the first time, the correspondence between the display interface and the node list of the scroll control may be stored. When the display interface is opened later, the scroll control contained in the display interface can be rapidly determined according to the scroll control node list corresponding to the display interface.

Step 520: and constructing a voice scrolling control word list of a scrolling control in the current display interface.

The voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control words.

In some embodiments, when performing step 520 above, the controller is further configured to: acquiring display positions of a plurality of child controls controlled by the rolling control in a current display interface; determining the rolling direction of the rolling control according to the display position of each sub-control; and constructing a voice scrolling control word list of the scrolling control in the current display interface according to the scrolling direction.

The display position of the child control may be the display area coordinate of the child control in the current display interface.

In one possible implementation, a two-dimensional rectangular coordinate system is established with the top left corner/center point of the current display interface as the coordinate center. And determining the display position of each control (a scrolling control, a sub-control of the scrolling control and other controls carrying interface descriptors) in the current display interface based on the two-dimensional rectangular coordinate system.

As one example, a corresponding interface (e.g., a getsources interface) is invoked via an auxiliary function service to obtain the display position of each control in the current display interface. The display position may include boundary coordinates of the control, such as an upper boundary coordinate, a lower boundary coordinate, a left boundary coordinate, and a right boundary coordinate.

Further, according to the display area coordinates of each sub-control in the current display interface, determining the display height and the display width of each sub-control; and judging and controlling the rolling direction of the rolling control to which each sub-control belongs according to the display height and the display width.

In some embodiments, when determining the scroll direction of the scroll control, the controller is further configured to:

(1) If the display heights of all the child controls in the current display interface are uniform, determining that the rolling direction of the rolling control is horizontal rolling;

The display height is the difference between the upper boundary coordinates and the lower boundary coordinates of the control.

That is, when the display heights of the sub-controls are identical, which means that the plurality of sub-controls are horizontally distributed in the current display interface, the scrolling direction of the scrolling control to which the plurality of sub-controls belong is controlled to horizontally scroll, so that the corresponding sub-controls are browsed, selected and opened by sliding/turning pages left and right.

In one example, referring to fig. 6, taking a display interface as an example of a user interface within a fitness application, a fitness recommendation item in the display interface is a scroll control, where the scroll control may control sliding sub-controls corresponding to multiple fitness items, respectively.

Referring to fig. 6, icons of sub-controls such as course finding, plan finding, fat reducing, shaping training, yoga, AI game, screen throwing and body building, and icons of other sub-controls provided by the body building application but not displayed in the current display interface, such as icons corresponding to sub-controls of dance, gym, and the like, are displayed in the body building recommended item.

For convenience of explanation, only triangle icons are used to represent scroll controls corresponding to the exercise recommendation items in fig. 6; of course, the scroll control may also be displayed in other forms, which are not limited in this embodiment of the present application.

In the display interface of the exercise application, since the display heights of the icons of the exercise items contained in the exercise recommendation item are all h, that is, the display heights of the sub-controls contained in the scroll control corresponding to the exercise recommendation item are all the same, the scroll direction of the scroll control corresponding to the exercise recommendation item is horizontal scroll.

In this way, by controlling the scroll control corresponding to the exercise recommendation item to perform the scroll operation along the horizontal direction, the exercise application can be sequentially browsed to recommend a plurality of exercise items, so that the exercise item of interest is selected for exercise.

(2) If the display widths of all the sub-controls in the current display interface are the same, determining that the rolling direction of the rolling control is vertical rolling;

the display width is the difference between the left boundary coordinate and the right boundary coordinate of the control.

That is, when the display widths of the sub-controls are the same, which means that the plurality of sub-controls are vertically distributed in the current display interface, the scrolling direction of the scrolling control to which the plurality of sub-controls belong is controlled to be vertical scrolling, so that the corresponding sub-controls are browsed, selected and opened by sliding/turning pages up and down.

In one example, referring to fig. 7, taking a display interface as an example of a user interface within a music application, a song list (i.e., the XXX list in fig. 7) in the display interface is a scroll control, where the scroll control may control sliding sub-controls corresponding to a plurality of songs, respectively.

Referring to FIG. 7, a menu is shown with tabs for songs 01-09, as well as tabs for other sub-controls contained in the menu but not shown in the current display interface, such as tabs corresponding to songs 10-121.

For convenience of explanation, only the arrow shown on the right side in fig. 7 represents the scroll control corresponding to the song menu; of course, the scroll control may also be displayed in other forms, which are not limited in this embodiment of the present application.

In the display interface of the music application, since the display widths of the tabs of the songs contained in the song list are d, that is, the display widths of the sub-controls contained in the scroll control corresponding to the song list are the same, the scroll direction of the scroll control corresponding to the song list is vertical scroll.

Therefore, by controlling the scroll control corresponding to the song list to execute the scroll operation along the vertical direction, all songs contained in the song list can be browsed in turn, and the interested songs are selected to be played.

(3) And if the display heights and/or the display widths of the child controls in the current display interface are different, determining that the rolling direction of the rolling control is waterfall flow rolling.

That is, if the display heights of the sub-controls are different, or the display widths of the sub-controls are different, or the display heights and the display widths of the sub-controls are different, the sub-controls are distributed in a waterfall flow mode in the current display interface, and the rolling directions of the rolling controls to which the sub-controls belong are controlled to roll in the waterfall flow mode, so that the sub-controls are loaded by sliding up and down or turning pages, so that the user can browse, select and open the interesting sub-controls.

In one example, referring to fig. 8, taking a display interface as an example of a user interface of a picture viewing application, the display interface includes a scroll control, where the scroll control may control pictures corresponding to a plurality of sub-controls, and the sub-controls display the corresponding pictures in a waterfall stream manner in the display interface.

Referring to fig. 8, the display interface includes pictures corresponding to the pictures 1-9, and pictures corresponding to other sub-controls not shown in the display interface.

For ease of illustration, the scroll controls in the display page are represented in FIG. 8 with only the right scroll bar; of course, the scroll control may also be displayed in other forms, which are not limited in this embodiment of the present application.

In the display interface of the picture viewing application, because the pictures 1-9 are in different display areas in the current display interface, the boundary coordinates of the child controls corresponding to the pictures 1-9 are different, so that the display heights and/or the display widths of the child controls corresponding to the pictures 1-9 are different, and the rolling direction of the rolling control is waterfall flow rolling.

In this way, by controlling the scroll control in the display page to perform the scroll operation along the vertical direction, a plurality of pictures can be loaded at a time after each scroll operation, so that the picture of interest is selected to perform operations such as preview/download/processing.

In some embodiments, when constructing the list of voice scroll control words for the scroll control in the current display interface, the controller is further configured to: setting semantic control words corresponding to the rolling control according to the rolling direction of the rolling control; and generating a voice scrolling control word list of the scrolling control in the current display interface according to the scrolling direction and the semantic control word of the scrolling control in the current display interface.

The semantic control words comprise sliding semantic words and page turning semantic words.

As one example, for a scroll control whose scroll direction is horizontal scrolling, the set slide semantic words include "slide left" and "slide right"; for a rolling control with a vertical rolling direction, the set sliding semantic terms include "slide up" and "slide down", and for a rolling control with a waterfall flow rolling direction, the set sliding semantic terms include "slide up", "slide down", "slide left" and "slide right".

As one example, the page flip semantic words include "last page" and "next page".

In some embodiments, assume that the scroll controls in the current display interface include an M control, an N control, and a P control, and the scroll direction of the M control is horizontal scroll, the scroll direction of the N control is vertical scroll, and the scroll direction of the P control is waterfall flow scroll. An exemplary list of voice scrolling control words is given in table 1 below for this current display interface.

TABLE 1

Step 530: and responding to the voice control instruction of the user, and controlling the target scroll control to execute scroll operation according to the scroll direction indicated by the voice control text based on the voice control word list and the voice control text corresponding to the voice control instruction.

The target scroll control is one of scroll controls contained in the current display interface and corresponding to the voice control text.

In some embodiments, referring to fig. 9, in implementing step 530 above, the controller is further configured to perform the sub-steps of:

step 531: and acquiring a voice control text corresponding to the voice control instruction.

The display device can establish communication connection between the controller and the cloud server through the communicator so as to analyze the voice control instruction through the cloud server, and therefore the data processing amount and the algorithm storage resource consumption amount of the display device are reduced.

In some embodiments, when implementing step 531 above, the controller is further configured to: the voice control instruction is sent to the cloud server to request the cloud server to analyze the voice control instruction; and receiving a voice control text corresponding to the voice control instruction sent by the cloud server.

That is, in the embodiment of the present application, a voice analysis algorithm is stored in a cloud server, and after receiving a voice control instruction sent by a display device, the cloud server analyzes the voice control instruction to obtain a voice control text corresponding to the voice control instruction.

In some embodiments, if the cloud server parses the voice control instruction, it determines that the voice control instruction is an invalid instruction (for example, the voice control instruction is not parsed into related control word character information), then abnormal result feedback information is sent to the display device, so as to instruct the display device to temporarily not execute a control operation on any control, and continue to detect the voice control instruction input by the user.

Step 533: matching the voice control text with semantic control words in the voice scrolling control word list, and determining a target scrolling control of which the voice control text requests control in a current display interface and a target scrolling direction of the target scrolling control.

If only one matched scroll control exists in the current display interface, the scroll control is directly determined as the target scroll control, and the following step 535 is executed.

In some embodiments, if the current display interface includes a plurality of candidate scroll controls corresponding to the voice control text, the controller is further configured to, when performing step 533 above: acquiring control priority of each candidate rolling control; a target scroll control is determined from the plurality of candidate scroll controls according to the control priority.

In one possible implementation, the control priority of each candidate scroll control is determined according to the control relation among the candidate scroll controls and the display position of each candidate scroll control in the current display interface.

In the structural tree corresponding to the current display interface, the priority of the rolling control at the parent node is higher than that of the rolling control at the child node.

The priority of the scrolling control with the display position at the upper left side of the current display interface is higher than that of the scrolling control with the display position at the lower right side of the current display interface.

As an example, see the display interface shown in fig. 10, which includes a plurality of scroll controls therein: w control, X control, Y control and Z control. The W control can control the X control, the Y control and the Z control, and the priority of the W control is higher than that of the X control, the Y control and the Z control.

Regarding the control priorities of the X control, the Y control and the Z control, the control priorities can be further determined according to the display positions of the X control, the Y control and the Z control in the display interface, the X control is positioned at the uppermost side of the display interface, and the priorities of the X control and the Y control are highest; the Z control is positioned at the lowest side of the display interface, and the priority of the Z control is lowest, so that the priorities of the X control, the Y control and the Z control are as follows: control X > control Y > control Z.

Thus, the control priorities of the four scroll controls in fig. 10 are: control W > control X > control Y > control Z.

Thus, after the control priority of the scrolling control in the current interface is determined, a target scrolling control can be determined to execute scrolling operation according to the voice control instruction and the voice scrolling control word list.

It should be understood that after receiving the voice control instruction of the user, the controller may determine the control requested to be controlled by the user in the current display interface according to the order of "control word list of the first voice interface and then voice scroll control word list"; the controller may also determine, in the current display interface, a control that the user requests to control according to the order of "first voice scroll control word list and then voice interface control word list", which is not limited in this embodiment of the present application.

In some embodiments, if it is determined that the current display interface includes a scroll control, after the display device receives a voice control instruction of a user, it may first determine, based on the voice scroll control word list, whether a target scroll control that requests control exists in the current display interface in a voice control text corresponding to the voice control instruction.

Further, if the target scrolling control matched with the voice control text cannot be determined based on the voice scrolling control word list, whether the target control requiring control exists in the voice interface control word list of the current display interface is determined based on the voice control text continuously. The interface description words in the voice interface control word list of the target control are the same as the voice control text.

Step 535: and controlling the target scrolling control to execute scrolling operation in the target scrolling direction according to the voice control text.

In some embodiments, when implementing step 535 above, the controller is further configured to: acquiring a target rolling distance of a target rolling control according to the voice control text; and controlling the target rolling control to execute the rolling operation according to the target rolling distance in the target rolling direction.

The voice control text may include a slide voice text and a page-turning voice text, among others.

In one possible implementation, if the voice control text is a sliding voice text, the scrolling distance may be a preset distance value. For example, the distance values may be set according to the display heights, display widths, display position intervals, and the like of the plurality of sub-controls controlled by the scroll control, which is not limited in the embodiment of the present application.

As an example, if the voice control text is a sliding voice text, referring to fig. 6, the scroll distance of the scroll control may be a display position interval between the "find course" icon and the "find plan" icon; referring to fig. 7, the scroll distance of the scroll control may be the display width of the tab of song 01, i.e., d; referring to fig. 8, the scroll distance of the scroll control may be the display width or the display height of any picture in the display interface.

In some embodiments, if the voice control text is a page-flip voice text, the controller is further configured to, when calculating the scroll distance: if the scrolling direction is horizontal scrolling, calculating a scrolling distance according to the display position of the rightmost child control and the display position of the leftmost child control controlled by the target scrolling control; if the rolling direction is vertical rolling or waterfall flow rolling, calculating the rolling distance according to the display position of the uppermost child control and the display position of the lowermost child control controlled by the target rolling control.

As an example, if the voice control text is a page-turning voice text, referring to fig. 6, the scroll distance of the scroll control may be the total display width value between the "find lesson" icon to the "throw training" icon; referring to fig. 7, the scroll distance of the scroll control may be the total display height value between tabs of songs 01 through 09, i.e., 9*d; referring to fig. 8, the scroll distance of the scroll control may be a total display height value or a total display width value of a plurality of pictures in the current display interface, for example, the scroll distance may be a total display height value of picture 2 and picture 6 in the display interface.

Based on the above, the controller in the display device provided by the present application is connected to the display and the sound collector, respectively, and in the process of displaying the image frame and the user interface on the display, the controller determines at least one scroll control included in the current display interface, and constructs a voice scroll control word list of the scroll control in the current display interface. The voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control words. Further, after detecting and collecting a voice control instruction of a user, the voice collection device sends the voice control instruction to the controller, so that the controller responds to the voice control instruction and controls a target scrolling control corresponding to the voice control instruction to execute scrolling operation in a current display interface based on a pre-constructed voice scrolling control word list and voice control text corresponding to the voice control instruction. Therefore, by identifying the rolling control and constructing a voice rolling control word list for the rolling control, when the display equipment receives a voice control instruction of a user, the target rolling control in the current display interface is controlled to execute rolling operation according to the voice control instruction, the defect that the rolling control cannot be directly controlled through voice is overcome, full voice control of each control in the display equipment is realized, the flexibility and convenience of a control mode of the display equipment are improved, and the user experience is improved.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or steps.

In addition, the present application further provides a voice control method, where the voice control method may be applied to the display device 200 or other electronic devices, and the method includes:

Responding to a voice control instruction of a user, and controlling a target rolling control to execute rolling operation according to a rolling direction indicated by the voice control text based on a voice rolling control word list and a voice control text corresponding to the voice control instruction; the target scroll control is one of the scroll controls contained in the current display interface and corresponding to the voice control instruction.

In some embodiments, referring to fig. 11, the voice control logic of the scroll control is: and registering an auxiliary function service (Accessibility Service) which can monitor the page change condition after the target application is started, and monitoring and displaying the page change condition through the auxiliary function service. When the control in the display page changes, namely the display page changes, acquiring a root view node control and a child view node control controlled by the root view node control in the current display interface according to a structural tree of the control in the current display interface, and generating a control node list corresponding to the current display interface according to the root view node control and the child view node control.

Further, traversing the control node list of the current display interface, determining at least one rolling control contained in the current display interface, and generating a rolling control node list corresponding to the current display interface.

Further, for any one scrolling control, display positions of a plurality of sub-controls controlled by the scrolling control are obtained, for example, upper boundary coordinates, lower boundary coordinates, left boundary coordinates and right boundary coordinates of the control; determining the rolling direction of the rolling control according to the display position of each sub control controlled by the rolling control, and setting semantic control words corresponding to the rolling control; and generating a voice rolling control word list of the rolling control in the current display interface according to the rolling direction and the semantic control word of each rolling control.

If a voice control instruction of a user is received, when the display device executes voice control, traversing a voice scrolling control word list according to a voice control text corresponding to the voice control instruction, determining a target scrolling control corresponding to the voice control text, and further controlling the target scrolling control to execute scrolling operation.

It should be noted that, when the current display interface includes a plurality of scroll controls, the target scroll control in the current display interface is controlled to execute the scroll operation according to the control priority.

Regarding the implementation principle and the beneficial effects of the voice control method, reference may be made to the above specific limitation regarding each embodiment of the controller configuration content in the display device, which is not repeated herein.

In some embodiments, the present application also provides a computer-readable storage medium. The computer readable storage medium may store a computer program that, when invoked and executed by a controller in a display device or other electronic device, performs part or all of the steps of the voice control method provided herein.

As one example, the computer readable storage medium may be a magnetic disk, optical disk, read-only memory, random-access memory, or the like.

It should be understood that the technical solutions in the embodiments of the present application may be implemented by means of software plus necessary general hardware platforms. Thus, the aspects of embodiments of the present application, in essence or contributing to the prior art, may be embodied in the form of a software product, which may be stored in a computer readable storage medium.

In some embodiments, the present application also provides a computer program product. The computer program product comprises a computer program, and when the computer program is called and run by a controller in a display device or other electronic equipment, part or all of the steps of the voice control method provided by the application can be realized.

The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. However, the above discussion in some examples is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the disclosure and to enable others skilled in the art to best utilize the embodiments.

That is, any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art without departing from the technical concept of the present application are intended to be included in the scope of the present application.

Claims

1. A display device, characterized by comprising:

a display configured to display an image screen and a user interface;

a sound collector configured to collect a voice control instruction of a user;

and a controller respectively connected with the display and the sound collector, wherein the controller is configured to:

constructing a voice scrolling control word list of a scrolling control in the current display interface; the voice scrolling control word list is used for representing the corresponding relation between the scrolling direction of the scrolling control and the semantic control word;

Responding to a voice control instruction of a user, and controlling a target scrolling control to execute scrolling operation according to a scrolling direction indicated by the voice control text based on the voice scrolling control word list and the voice control text corresponding to the voice control instruction; and the target scroll control is one of the scroll controls contained in the current display interface, which corresponds to the voice control text.

2. The display device of claim 1, wherein the controller is further configured to:

after a target application is started, monitoring the change condition of a display interface of the target application;

3. The display device of claim 1, wherein the controller is further configured to:

acquiring display positions of a plurality of sub-controls controlled by the scrolling control in the current display interface;

determining the rolling direction of the rolling control according to the display position of each child control;

4. A display device according to claim 3, wherein the controller is further configured to:

if the display heights of the child controls in the current display interface are identical, determining that the rolling direction of the rolling control is horizontal rolling;

if the display widths of the child controls in the current display interface are the same, determining that the rolling direction of the rolling control is vertical rolling;

and if the display height and/or the display width of each child control in the current display interface are different, determining that the rolling direction of the rolling control is waterfall flow rolling.

5. The display device of any one of claims 1-4, wherein the controller is further configured to:

acquiring a voice control text corresponding to the voice control instruction;

matching the voice control text with semantic control words in the voice scrolling control word list, and determining a target scrolling control of which the voice control text requests to be controlled in the current display interface and a target scrolling direction of the target scrolling control;

6. The display device of claim 5, wherein the controller is further configured to:

acquiring a target rolling distance of the target rolling control according to the voice control text;

and controlling the target rolling control to execute rolling operation according to the target rolling distance in the target rolling direction.

7. The display device of claim 6, wherein the voice control text comprises a page-flip voice text, the controller further configured to:

if the target rolling direction is horizontal rolling, calculating the target rolling distance according to the display position of the rightmost child control and the display position of the leftmost child control controlled by the target rolling control;

and if the target rolling direction is vertical rolling or waterfall rolling, calculating the target rolling distance according to the display position of the uppermost sub-control and the display position of the lowermost sub-control controlled by the target rolling control.

8. The display device of claim 5, wherein if a plurality of candidate scroll controls corresponding to the voice-controlled text are included in the current display interface, the controller is further configured to:

Acquiring control priority of each candidate rolling control;

and determining the target scroll control from the plurality of candidate scroll controls according to the control priority.

9. The display device of claim 5, wherein the controller is further configured to:

the voice control instruction is sent to a cloud server, so that the cloud server is requested to analyze the voice control instruction;

10. A voice control method, comprising: