WO2023284870A1

WO2023284870A1 - Control method and control device

Info

Publication number: WO2023284870A1
Application number: PCT/CN2022/106080
Authority: WO
Inventors: 穆聪聪; 修建竹; 卢可敬; 杨香斌; 高娜; 颜贝
Original assignee: 海信视像科技股份有限公司
Priority date: 2021-07-15
Filing date: 2022-07-15
Publication date: 2023-01-19
Also published as: CN117616486A

Abstract

The present application discloses a control method and a control device. The control device comprises: a display used for displaying a user interface; a user input interface used for receiving an input signal; and a controller respectively connected to the display and the user interface. The controller is configured to: acquire a device list so as to control the display to display the device list; receive, via the user interface, a user operation command of a device control in the selected device list; broadcast, to a plurality of devices, a broadcast packet carrying a device identifier of a device corresponding to the device control, so that the device sends, on the basis of the device identifier, a Bluetooth pairing request; and in response to the Bluetooth pairing request received from the device, perform pairing with the device via Bluetooth.

Description

Control method and control equipment

Cross References to Related Applications

This application requires the application number 202110817639.9 submitted on July 20, 2021, the application number 202110799098.1 submitted on July 15, 2021, and the Chinese patent application with application number 202111041999.0 submitted on September 7, 2021 priority, the entire contents of which are incorporated by reference into this application.

technical field

The embodiments of the present application relate to the technical field of smart home appliances, and in particular, to a control method and a control device.

Background technique

In the related art, a user can use a control device to control an electronic device. For example, the control device can be a remote controller, and the electronic device can be a TV. The user can use the remote control to control the TV to achieve the purpose of switching programs. Before using the control device to control the device, it is necessary to carry out Bluetooth pairing between the control device and the electronic device, and the control device can be used to control the electronic device only after the pairing is successful. However, when Bluetooth pairing is performed between the control device and the electronic device in the related art, the user needs to perform Bluetooth pairing according to the prompts on the device. The Bluetooth pairing process is relatively complicated, which brings bad user experience to the user.

In addition, with the development of voice interaction technology, more and more electronic devices such as home terminal equipment have a voice interaction function. Using the voice interaction function, the user can voice control these terminal devices to perform corresponding operations, such as starting and stopping. For the voice interaction of the handheld control device, the way of raising and waking can be used to trigger the handheld control device to start the voice interaction function. For example, if the user picks up the remote control from the desktop, the voice interaction function of the remote control can be triggered.

However, in the way of raising to wake up, if the user just picks up the handheld control device without voice interaction intention. At this time, if the user is talking at the same time, it is easy to cause false reception, resulting in poor user experience.

Contents of the invention

The application provides a control device, including:

a display for displaying a user interface;

a user interface for receiving an input signal;

a controller, respectively connected to the display and the user interface, configured to:

Acquiring a device list, so as to control the display to display the device list;

receiving a user operation command for selecting a device control in the device list via the user interface;

broadcasting a broadcast packet carrying the device identifier of the device corresponding to the device control to multiple devices, so that the device sends a Bluetooth pairing request based on the device identifier; and

In response to receiving a Bluetooth pairing request from the device, pairing is performed with the device via Bluetooth.

The present application provides a method for controlling pairing between a control device and a device, including:

Receive the user operation command of the device control in the selected device list;

Broadcasting a broadcast packet carrying the device identifier of the device corresponding to the device control to multiple devices, so that the device sends a Bluetooth pairing request based on the device identifier;

Description of drawings

Embodiments of the application will be described below with reference to the accompanying drawings, wherein:

FIG. 1 shows a usage scenario of a display device according to some embodiments;

FIG. 2 shows a block diagram of a hardware configuration of a control device 100 according to some embodiments;

FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to some embodiments;

FIG. 4 shows a software configuration diagram in a display device 200 according to some embodiments;

Fig. 5 shows a schematic structural diagram of a terminal device according to some embodiments;

Fig. 6 shows a schematic diagram of a software architecture of a terminal device according to some embodiments;

FIG. 7 exemplarily shows a flow chart of a method for controlling a control device to perform pairing with a device according to some embodiments;

FIG. 8 exemplarily shows a schematic diagram of a user interface of a mobile terminal;

FIG. 9 exemplarily shows another schematic diagram of a user interface of a mobile terminal;

FIG. 10 exemplarily shows another schematic diagram of a user interface of a mobile terminal;

Fig. 11 exemplarily shows a schematic diagram of the principle of voice interaction according to some embodiments;

Fig. 12 exemplarily shows a schematic frame diagram of a sound receiving system of a control device according to some embodiments;

Fig. 13 exemplarily shows a schematic diagram of a sound source angle calculation method of a control device according to some embodiments;

Fig. 14 exemplarily shows a schematic diagram of a sound source angle scene of a control device according to some embodiments;

Fig. 15 exemplarily shows a signaling diagram of a sound collection method of a control device according to some embodiments;

Fig. 16 shows a schematic diagram of a user interface of a terminal device according to some embodiments;

Fig. 17 shows a schematic diagram of setting an identity authentication mode in a display device in some embodiments;

Fig. 18 shows a flowchart of a communication connection between a display device and a terminal device in some embodiments;

Fig. 19 shows a schematic diagram of a terminal interface of a terminal device in some embodiments;

Fig. 20 shows a flowchart of interaction between a terminal device and a server in some embodiments;

Fig. 21 shows a flow chart of the server obtaining a set of control instructions in some embodiments;

Figure 22 shows a schematic diagram of a terminal interface of a terminal device in some embodiments;

Figure 23a shows a schematic diagram of a control interface of a terminal device in some embodiments;

Fig. 23b shows a schematic diagram of another control interface of the terminal device in some embodiments;

Fig. 24 shows a flowchart of a terminal device controlling a display device in some embodiments;

Fig. 25 shows a schematic flowchart of an embodiment of a communication control method;

Fig. 26 shows a schematic flowchart of an embodiment of a communication control method.

detailed description

The following will describe the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments.

It should be noted that the brief description of terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.

Fig. 1 is a schematic diagram of a usage scenario of a display device according to an embodiment. As shown in FIG. 1 , an electronic device 200 such as a display device also performs data communication with a server 400 , and a user can operate the display device 200 through a terminal device 300 or a control device 100 . In addition, the control device 100 can also control other devices, such as electronic devices such as refrigerators, air conditioners, washing machines, and household lights.

In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, etc., and the display device 200 is controlled by wireless or other wired methods. Wherein, the wireless mode may be direct connection or non-direct connection, and may be routed or not routed. The user can control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

Device In some embodiments, the terminal device 300 may include any one of a mobile terminal, a tablet computer, a computer, a notebook computer, an AR/VR device, and the like.

In some embodiments, the terminal device 300 can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device. The app can be configured to provide users with various controls in an intuitive user interface (UI) on the screen associated with the smart device.

In some embodiments, the terminal device 300 and the display device 200 are capable of data communication.

In some embodiments, the display device 200 can also be controlled in a way other than the control device 100 and the terminal device 300, for example, the display device 200 can directly receive the user's voice command control through the module for acquiring voice commands configured inside the display device 200. , the user's voice command control can also be received through the voice control device installed outside the display device 200.

Fig. 2 is a configuration block diagram of the control device 100 according to some embodiments. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

In some embodiments, the communication interface 130 is used for communicating with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module.

In some embodiments, the user input/output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button or an alternative module.

FIG. 3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments.

In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.

In some embodiments the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.

Devices In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 may establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220 .

In some embodiments, the user input interface can be used to receive a control signal of the control device 100 (such as an infrared remote controller, etc.).

In some embodiments, the detector 230 is used to collect signals of the external environment or interaction with the outside. For example, the detector 230 includes a light receiver, which is a sensor for collecting ambient light intensity; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes or user interaction gestures, or , the detector 230 includes a sound collector, such as a microphone, for receiving external sound.

In some embodiments, the external device interface 240 may include but not limited to the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface ( Any one or more interfaces such as USB), RGB port, etc.

In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different split devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait.

In some embodiments, the control device 100 also includes a display.

Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are respectively the application (Applications) layer (abbreviated as "application layer"), application framework (Application Framework) layer (abbreviated as "framework layer") "), Android runtime (Android runtime) and system library layer (referred to as "system runtime layer"), and the kernel layer.

Fig. 5 is a block diagram of a hardware configuration of a terminal device 300 shown in some embodiments of the present application.

It should be understood that the terminal device 300 shown in FIG. 5 is only an example, and the terminal device 300 may have more or fewer components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.

As shown in Figure 5, the terminal device 300 includes: a radio frequency (radio frequency, RF) circuit 310, a memory 320, a display unit 130, a camera 340, a sensor 350, an audio circuit 360, and a wireless fidelity (Wireless Fidelity, Wi-Fi) module 370 , a controller such as a processor 380 , a Bluetooth module 381 , and a power supply 390 and other components.

Fig. 6 is a block diagram showing a software structure of a terminal device 300 according to some embodiments of the present application. Among them, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom. The application layer can consist of a series of application packages.

As shown in FIG. 6, the application package may include application programs such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The terminal device 300 in the embodiment of the present application may be a mobile phone, a tablet computer, a wearable device, a notebook computer, a TV, and the like. The above display device may be an example of a terminal device.

In the embodiment of the present application, in order to perform manipulation, the control device needs to establish a certain communication link with the home appliance for signal interaction, that is, the control device needs to perform communication pairing with the home appliance. In the embodiment of the present application, in the process of the user controlling the device, the method according to the embodiment of the present application may include: obtaining a device list to control the display to display the device list; receiving a user operation command for selecting a device control in the device list ; Broadcast a broadcast packet carrying the device identifier of the device corresponding to the device control to multiple devices, so that the device sends a Bluetooth pairing request based on the device identifier; in response to receiving the Bluetooth pairing request from the device, communicate with the device The above devices are paired via Bluetooth. Therefore, the control device and the device can automatically complete the Bluetooth pairing, so that the user is not aware of the pairing process between the control device and the device, and the user experience is improved.

Fig. 7 exemplarily shows a flow chart of a method for controlling a control device to perform pairing with a device according to some embodiments. As shown in Figure 7,

In step S100, the control device receives an instruction to display a device list to determine whether the control device is currently connected to the Internet. In some embodiments, the instruction for displaying the device list may be generated by controlling the device to start up. In some embodiments, a preset application is installed in the control device, and an instruction to display the device list is generated by opening the preset application. Since the device list needs to be delivered from the server, in this embodiment of the application, before obtaining the device list, it is necessary to determine whether the control device is currently connected to the Internet.

If yes, then in step S200, the control device acquires the device list, and controls the display to display the device list. It should be noted that the user can log in a user account on the control device, and the device list includes all devices bound with the user account, and the devices bound with the user account can be understood as devices to be controlled by the control device.

In some embodiments, the step of the control device obtaining the device list and controlling the display to display the device list includes: the control device sends a request for obtaining the device list to the server, the request carries a user account, and the server uses the user account to find corresponding device list, and send the device list to the control device, and the control device receives the device list sent by the server, and controls the display to display the device list.

In some embodiments, the device list includes device identifiers of devices, and the device identifiers are in a one-to-one correspondence with devices, and each device has a unique device identifier. Exemplarily, the device identifier can be composed of numbers, and the device identifier can also be composed of letters. In addition, the device identifier can also be composed of numbers and letters. In this embodiment of the application, the device is not limited to The specific form of identification.

Otherwise (that is, the control device is not connected to the network), at S300, the control display displays a notification that the device is not connected to the network. The user can connect the control device to the network according to the prompt. When the control device is connected to the network, obtain the device list, and control the display to display the device list. In the embodiment of the present application, the notification of not being connected to the Internet can be represented by text or by a picture, and the form of the notification of not being connected to the Internet is not limited in this application. In some embodiments, the control device is provided with a reminder light, and the notification of not being connected to the Internet can also be completed through the reminder light. For example, the color of the reminder light can be controlled to be red, and it can also be completed by controlling the reminder light to blink.

The following uses an example to illustrate how to bind a user account to a device.

The user interface of the mobile terminal is shown in FIG. 8 . As shown in FIG. 8 , an add device control 501 is set on the user interface, and the user clicks the add device control, and the page jumps to the user interface shown in FIG. 9 .

In some embodiments, when the display is a touch screen, the user can touch the position on the display interface corresponding to adding the device control to realize the operation of clicking to add the device control. In some other embodiments, the control device is provided with operation keys, for example, the operation keys include an up key, a down key, a left key, a right key and a confirmation key. The user can move the focus to the add device control by pressing the operation button, and press the confirmation key on the control device to complete the operation of clicking the add device control.

In the user interface shown in FIG. 9 , device categories are displayed, for example, categories such as air conditioners, refrigerators, washing machines, and smart door locks. Click the category corresponding to the device to be added. When the user selects an air conditioner category, the user interface displays air conditioner subcategories, including air conditioners of different brands and air conditioners of different uses. In some embodiments, when the refrigerator subcategory is clicked, the user interface jumps, as shown in FIG. 10 . In the user interface shown in Figure 10, all refrigerators under the refrigerator sub-category are displayed, and at the same time, the mac addresses corresponding to the refrigerators are displayed. For example, the mac address of smart refrigerator 1 is 228c7ffb, and the mac address of smart refrigerator 2 is 006b2a7b . Users can click the corresponding refrigerator control according to their own needs, and press the next step control set in the user interface to send the binding instruction. When the corresponding refrigerator receives the binding instruction, and after the binding is successful, the device identifier corresponding to the refrigerator is stored in the device list.

In the embodiment of the present application, other methods are also used to generate the device list, which is not limited here. For example, the device list can be pre-stored content, and the device identification of the device that can be controlled by the control device is preset. stored in the device list.

Next, in step S400, the control device may, in response to receiving a user operation command to select a device control in the device list, broadcast a broadcast packet carrying the device identifier of the device corresponding to the device control. It should be noted that in the embodiment of the present application, the user can directly select the device to be controlled, and select the device control of the device. In the process of using the control device to control the device, the control device and the device can be paired without additional operations. Improve user experience.

In some embodiments, the control device includes a bluetooth service, and uses the bluetooth service to broadcast a broadcast packet carrying a device identifier through a bluetooth broadcast (BLE broadcast).

In some embodiments, the list of devices displayed on the user interface includes a device control that can be configured with focus. In some embodiments, the display is a touch screen, and it is recognized that the user's touch position on the display is located at the corresponding position of the device control on the user interface, and the user operation of selecting the device control in the device list can be completed. In some embodiments, the user operation of the selected device control is completed through an operation button on the control device.

In step S500, the device that has received the broadcast packet may parse the broadcast packet to determine whether to send a Bluetooth pairing request to the control device. In some embodiments, any device whose system is the Android system can receive the broadcast packet.

In some embodiments, the device includes a Bluetooth service, and the Bluetooth service analyzes the broadcast packet to determine whether to send a Bluetooth pairing request to the control device.

In some embodiments, the determination of whether to send a Bluetooth pairing request may be based on device identification.

In some embodiments, the step of the device parsing the broadcast packet for determining whether to send a Bluetooth pairing request to the control device includes:

Parsing the broadcast packet to obtain the device identifier in the broadcast packet. The broadcast packet in the embodiment of the present application includes the device identifier of the device corresponding to the device control in the device list selected by the user.

Judging whether the device identifier in the broadcast packet is the same as the device identifier of the device that received the broadcast packet. Since the broadcast packet is broadcast to multiple devices, each device needs to determine whether the device identifier in the broadcast packet is the same as the device identifier of the device that received the broadcast packet, and determine whether to process the broadcast packet.

If they are the same, then send the Bluetooth pairing request to the control device. In the embodiment of the present application, if they are the same, the device processes the broadcast packet and sends a Bluetooth pairing request to the control device.

In this embodiment of the present application, the device includes a Bluetooth service, and the Bluetooth service is configured to send a Bluetooth pairing request to the control device.

If not, the Bluetooth pairing request is not sent to the control device. In the embodiment of the present application, if the device identifier in the broadcast packet is different from the device identifier of the device that received the broadcast packet, the device does not process the content of the broadcast packet, and does not send a Bluetooth pairing request to the control device.

In the embodiment of this application, although multiple devices can receive the broadcast packet, in fact the broadcast packet is only intended to be sent to the device with the device ID carried in the broadcast packet. When the device identifiers of the devices are the same, the devices send a Bluetooth pairing request to the control device.

In some embodiments, the device sends a Bluetooth pairing request to the controlling device. Finally, in step S600, the control device receives the Bluetooth pairing request, and performs pairing with the device. The method further includes: if the pairing is not successful, repeating the step of sending the broadcast packet carrying the device identification of the device corresponding to the device control to all devices in the device list until the number of repeated executions reaches a preset number of times.

In the embodiment of the present application, in order to avoid the pairing failure between the control device and the device due to special circumstances, the user is affected to control the device through the control device. When the pairing is not successful, repeat the operation of sending broadcast packets until the number of executions reaches the preset number of times, which can improve the error tolerance rate of the embodiment of the present application and avoid affecting the user's use of the control device to control the device due to accidental events. For example, The preset number of times may be three times.

In some embodiments, the method further includes: after repeating the step of broadcasting the broadcast packet carrying the device identifier of the device corresponding to the device control, controlling the display to display a pairing exception prompt.

In the embodiment of the present application, the abnormal pairing prompt can be expressed by text or by a picture, and the abnormal pairing prompt is not limited in this application. In some embodiments, the control device is provided with a reminder light, and the pairing abnormality reminder can also be completed through the reminder light. For example, the color of the reminder light can be controlled to be red, and it can also be completed by controlling the reminder light to blink.

In some embodiments, the method further includes: if the pairing is successful, establishing a connection between the control device and the device for controlling the device. In this embodiment of the application, when the Bluetooth pairing between the control device and the device is successful, the control device can control the device. For example, when the Bluetooth pairing between the control device and the refrigerator is successful, the user can adjust the internal temperature of the refrigerator through the control device. It can be understood that the embodiment of the present application automatically completes the Bluetooth pairing process between the control device and the device during the process of implementing the control device to operate the device, and the user is not aware of the Bluetooth pairing process, which improves the user experience.

In some embodiments, a control device for implementing the above control method is also provided, and the control device may include: a display for displaying a user interface; a user interface for receiving an input signal; a controller for communicating with the display and the The user interface is connected, and it is configured to: obtain a device list to control the display to display the device list; receive a user operation command for selecting a device control in the device list via the user interface; broadcast to multiple devices carrying all The device controls a broadcast packet corresponding to the device identification of the device, so that the device sends a Bluetooth pairing request based on the device identification; and in response to receiving the Bluetooth pairing request from the device, performs pairing with the device via Bluetooth.

In the above-mentioned embodiments, a method for pairing a control device with a device and a control device, in the embodiment of the present application, in the process of the user controlling the device, the control device and the device automatically complete the Bluetooth pairing, so that the user can pair the control device with the device. No perception, improve user experience.

The device in the embodiment of the present application may also be applied to speech recognition. In order to clearly illustrate the embodiment of the present application, a speech recognition network architecture provided by the embodiment of the present application is described below with reference to FIG. 11 .

FIG. 11 is a schematic diagram of a speech recognition network architecture provided by an embodiment of the present application. Referring to FIG. 11 , the smart device is used to receive input information and output processing results of the information. The speech recognition service device is an electronic device deployed with a speech recognition service, the semantic service device is an electronic device deployed with a semantic service, and the business service device is an electronic device deployed with a business service. The electronic device here may include a server, a computer, etc., and the speech recognition service, semantic service (also called a semantic engine) and business service here are web services that can be deployed on the electronic device, wherein the speech recognition service is used for audio Recognized as text, the semantic service is used for semantic analysis of the text, and the business service is used to provide specific services such as the weather query service of Moji Weather, the music query service of QQ Music, etc. In one embodiment, there may be multiple entity service devices deployed with different business services in the architecture shown in FIG. 23 , or one or more functional services may be integrated in one or more entity service devices.

In some embodiments, the process of processing the information input to the smart device based on the architecture shown in FIG. 11 is described below as an example. Taking the information input to the smart device as a query sentence input by voice as an example, the above process may include the following three processes :

[Speech Recognition]

After receiving the query sentence input by voice, the smart device can upload the audio of the query sentence to the voice recognition service device, so that the voice recognition service device can recognize the audio as text through the voice recognition service and return it to the smart device. In one embodiment, before uploading the audio of the query sentence to the speech recognition service device, the smart device may perform denoising processing on the audio of the query sentence, where the denoising processing may include steps such as removing echo and environmental noise.

[semantic understanding]

The smart device uploads the text of the query sentence recognized by the speech recognition service to the semantic service device, so that the semantic service device can perform semantic analysis on the text through the semantic service to obtain the business field and intention of the text.

[semantic response]

According to the semantic analysis result of the text of the query statement, the semantic service device sends a query instruction to the corresponding business service device to obtain the query result given by the business service. The smart device can obtain and output the query result from the semantic service device. As an embodiment, the semantic service device can also send the semantic analysis result of the query sentence to the smart device, so that the smart device can output the feedback sentence in the semantic analysis result.

It should be noted that the architecture shown in FIG. 11 is only an example, and does not limit the protection scope of the present application. In the embodiment of the present application, other architectures may also be used to implement similar functions. For example, all or part of the three processes may be completed by a smart terminal, which will not be described in detail here.

In some embodiments, the smart device shown in Figure 11 can be a display device, such as a smart TV, and the function of the speech recognition service device can be realized by the cooperation of the sound collector and the controller set on the display device, and the semantic service device and business service device The functions of can be realized by the controller of the display device, or by the server of the display device.

With the development of voice interaction technology, more and more home terminal devices have a voice interaction function. Using the voice interaction function, the user can voice control these terminal devices to perform corresponding operations, such as starting and stopping.

At present, the voice interaction for the control device usually requires the user to press a button, or the user needs to input a wake-up word to trigger the control device to start the voice interaction function. In order to improve the user experience, in addition to these two methods, a method of lifting to wake is also defined. For example, if the user picks up the remote control from the desktop, the voice interaction function of the remote control can be triggered.

However, in the way of raising to wake up, if the user just picks up the control device without voice interaction intention. At this time, if the user is talking at the same time, it is easy to generate a situation of false reception, resulting in poor user experience.

Fig. 12 is a schematic diagram of the framework of the radio system of the control device. In order to solve the above problems, as shown in FIG. 12 , the control device according to the embodiment of the present application may include a sound collection system, and the system includes at least a user layer, a sensing layer and a system layer. The user layer is the layer where the user performs an input operation to input a signal. The sensing layer is provided with various sensors for sensing the user's operation and converting the user's operation into a level of related signals. The system layer is the layer that receives signals and performs related operations on other components based on the received signals. The main chip in the system layer is the controller. The system layer may also include storage modules, power supplies and other components. The storage module can store preset parameters and other information.

It should be noted that the main chip in the embodiment of the present application may be set in the control device, or in other devices controlled by the control device. For example, when the control device is a remote control, the main chip can be set in the display device or in the remote control. In the embodiment of the present application, the main chip is set in the control device to illustrate the solution. In addition, it should be noted that the control device provided by this application can be mobile phone, wearable device, AR (Augmented Reality, augmented reality)/VR (Virtual Reality, virtual reality) device, tablet computer, notebook computer, UMPC (Ultra-mobile Personal Computer, super mobile personal computer), netbook, PDA (Personal Digital Assistant, personal digital assistant) and other arbitrary terminals, the embodiments of the present application do not make any restrictions on this.

In some embodiments, in the user layer, the information input by the user may include: picking up the operation of the control device, and inputting voice.

The sensing layer may include a sound collector and a screen lift sensor. The sound collector is used to collect the voice signal input by the user, and the lifting screen sensor is used to detect whether there is an operation of picking up the control device. The sound collector may be a microphone, and the screen lift sensor may be an acceleration sensor.

An accelerometer is a sensor capable of measuring acceleration. It usually consists of mass block, damper, elastic element, sensitive element and adjustment circuit. During the acceleration process, the sensor uses Newton's second law to obtain the acceleration value by measuring the inertial force on the mass block. According to different sensor sensitive elements, common acceleration sensors include capacitive, inductive and so on. The acceleration sensor used in this embodiment of the present application is not limited, for example, a three-axis acceleration sensor may be used.

If the control device is picked up by the user, the acceleration change can be detected by the acceleration sensor. The acceleration sensor generates a screen-up signal according to the acceleration change, and sends the screen-up signal to the main chip. After the main chip receives the screen-up signal, it controls the start of the sound collector. After the sound collector is started, it starts to collect the voice signal input by the user.

In some embodiments, the control device may also include a display. After the main chip receives the screen-up signal sent by the acceleration sensor, it controls the display to light up the screen. Prompts, such as "whether to perform voice interaction", can be displayed on the user interface that lights up the screen.

After the sound collector collects the voice signal input by the user, it sends the voice signal to the main chip. The main chip calculates the sound source angle between the user and the control device according to the voice signal. If the sound source angle between the user and the control device is within the preset sound source angle range, it means that the user intends to perform voice interaction, and voice recognition processing is performed on the collected voice signal. It should be noted that the speech recognition processing process belongs to the prior art, and this application does not describe it in detail. After the speech recognition processing, the process of judging the intention and executing the intention according to the recognition result also belongs to the prior art, and the present application will not describe it in detail.

If the sound source angle between the user and the control device is not within the preset sound source angle range, it means that the user does not intend to perform voice interaction, and no voice recognition processing is performed on the collected voice signal.

In some embodiments, the sound collector may be a microphone array, and the microphone array includes at least two microphones. Calculate the sound source angle between the user and the control device according to the voice signal, specifically:

Fig. 13 is a schematic diagram of a method for calculating a sound source angle. In FIG. 13 , the far-field incident signal is the voice signal input by the user, and the wave front refers to a spherical surface formed from the sound source to the propagation direction when the sound wave propagates. From a long distance, the wave front can be regarded as a plane, which is also convenient for calculation. In Fig. 13 p microphones 1, 2...p-2, p-1, p are shown. The formula for calculating the time difference between the arrival of the voice signal from the sound source at each microphone is:

Wherein, p is the number of microphones, d is the distance between two adjacent microphones, and is the sound source angle between the user and the control device. As shown in FIG. 26 , the sound source angle may specifically be the angle between the user and the central axis of the control device. The matrix can be constructed according to the above formula:

Among them, x1...xp is the X coordinate value of the microphone, y1...yp is the Y coordinate value of the microphone ... is the time difference between the arrival of the voice signal from the sound source to two adjacent microphones. According to the matrix, p sound source angle values can be obtained. Finally, the final sound source angle value can be calculated by using the least square method. The specific calculation process belongs to the prior art, and the present application does not elaborate on it in detail. In addition to the above-mentioned sound source localization method based on time difference of arrival, other sound source localization methods may also be adopted in this embodiment of the present application. For example, energy spectrum estimation methods in space based on high-resolution spectrum estimation. The basic principle is to use spatial spectrum scanning to obtain the corresponding received energy at each angle. Further, the angle of the maximum received energy is determined as the angle of arrival of the sound source.

The preset sound source angle range in this embodiment of the present application may be set by a user based on experience. For example, if the preset sound source angle is set to 90°±15°, if the calculated sound source angle is within the range of 90°±15°, it can be determined that the user intends to perform voice interaction. At this point, the user may pick up the control device, and place the microphone of the control device within a range of positions that are within 15° of left-right deviation directly in front of the user's mouth. If the calculated sound source angle is not within the range of 90°±15°, it can be determined that the user does not intend to perform voice interaction. At this time, it may be that the user just picks up the control device, but does not place the microphone of the control device directly in front of the user's mouth within a range of positions where the left and right deviation does not exceed 15°.

For example, as shown in FIG. 14 , if the user is at position B, the user is directly in front of the control device, and the sound source angle at position B is 90°. If the user is at position A, and the angle between the user and the central axis of the control device is 10°, the sound source angle is 90°±10°, and it can be determined that the user intends to perform voice interaction. If the user is at position C, and the angle between the user and the central axis of the control device is 50°, the sound source angle is 90°±50°, which is not within the preset sound source angle range. It may be determined that the user does not intend to perform voice interaction.

In some embodiments, if it is determined that the sound source angle is within the preset sound source angle range, it is further determined whether the result of the voice activity detection is end. If the voice activity detection result is incomplete, it means that the user is still inputting voice signals. The sound collector continuously collects the voice signal input by the user. The main chip finally performs speech recognition processing according to the complete speech signal. If the result of the voice activity detection is end, it means that the user does not continue to input the voice signal. The main chip performs speech recognition processing according to the currently collected speech signal.

Exemplarily, after the user picks up the remote controller, the user inputs a voice signal "turn on the TV". The main chip of the remote control determines that the sound source angle is within the preset sound source angle range, that is, it is determined that the user has a voice interaction intention. At this time, the user continues to input the voice signal "jump to the central set", and the result of the voice activity detection at this time is not completed. The sound collector continues to collect the voice signal input by the user, and obtains the complete voice signal "turn on the TV, jump to the central set". Finally, perform voice recognition processing on the completed voice signal "turn on the TV, jump to the central set". If the user does not continue to input the voice signal after it is determined that the user has a voice interaction intention, the result of the voice activity detection at this time is ended. Finally, the voice recognition process is performed according to the current voice signal "turn on the TV".

In some embodiments, if the sound source angle is not within the preset sound source angle range, regardless of whether the voice activity detection result is not finished or finished, the collection of the voice signal input by the user is stopped.

In some embodiments, if the sound source angle is within the preset sound source angle range, and the control device has a display, the result of performing speech recognition processing on the speech signal is finally displayed on the display.

In some embodiments, if the sound source angle is not within the preset sound source angle range, after it is determined that it is not within the preset sound source angle range, the screen of the control display is turned off.

In some embodiments, if a controller such as a processor receives a screen-up signal and controls the start of the sound collector, but the sound collector does not collect a voice signal, the prompt "Please say it again" can be displayed on the display, and at the same time Sound an alarm to alert the user. If the sound collector has not collected a voice signal within the preset time period, it is determined that the user has no voice interaction intention, and the display is controlled to turn off the screen. If the voice collector collects voice signals within the preset time period, it is determined that the user has voice interaction intentions, and voice recognition processing is performed on the collected voice signals.

Based on the foregoing embodiments, the present application further provides a sound collection method for a control device. As shown in FIG. 15 , from S1501 to S1502 , the user picks up the control device, and the screen lift sensor detects that the user picks up the control device, generates a screen lift signal, and sends the screen lift signal to the processor.

At S1503, the controller such as the processor controls the start of the sound collector after receiving the screen-up signal. From S1504 to S1505, the sound collector collects the voice signal input by the user, and sends the voice signal to the processor.

At S1506, the processor calculates the sound source angle between the user and the control device according to the voice signal.

Next, at S1507, it is judged whether the sound source angle is within the preset sound source angle range. If the sound source angle is within the preset sound source angle range, perform speech recognition processing on the speech signal at S1508.

If the sound source angle is not within the preset sound source angle range, no speech recognition processing is performed on the speech signal.

In some embodiments, the sound collector is a microphone array, and the microphone array includes at least two microphones. Calculate the sound source angle between the user and the control device according to the voice signal, the specific steps are:

The sound source angle between the user and the control device is calculated according to the distance between two adjacent microphones and the time difference between two adjacent microphones receiving voice signals. Among them, the sound source angle is the angle between the user and the central axis of the control device

In some embodiments, if the sound source angle is within the preset sound source angle range, continue to determine whether the voice activity detection result is over. If the voice activity detection result is incomplete, it means that the sound collector can still continue to collect the voice signal input by the user. Then the processor continues to receive the voice signal, and finally performs voice recognition processing on the complete voice signal. If the result of the voice activity detection is ended, it means that the sound collector cannot continue to collect the voice signal input by the user. Then the processor performs speech recognition processing on the current speech signal.

In some embodiments, if the sound source angle is not within the preset sound source angle range, no matter whether the voice activity detection result ends, the sound collector is controlled to stop collecting the voice signal input by the user.

In some embodiments, the control device also includes a display. When receiving the screen lift signal sent by the screen lift sensor, the processor also controls the display to turn on the screen. After the screen is turned on, if the sound source angle is within the preset sound source angle range, the result of performing speech recognition processing on the speech signal will be displayed on the display. If the sound source angle is not within the preset sound source angle range, the control display turns off the screen.

In the embodiment of the present application, the terminal device may also be used instead of the controller as the control device to control the household electrical appliances. In addition, the control device of the embodiment of the present application can not only use infrared signals to control household appliances, but also adopt AIOT (Artificial Intelligence & Internet of Things, artificial intelligence Internet of Things) cloud intelligent control method, through the cloud of the embodiment of the present application The server is used to control the home appliances to realize the corresponding functions, so that there is no need to combine the remote controllers for control, which improves the user experience.

Generally, the terminal device acquires corresponding data from the server by sending a data request to the server. For example, a media resource information acquisition request is made to the server to obtain the media resource information from the server, where the media resource information at least includes a media resource identifier, for example, may also include a media resource name. Based on the push function, users can select media assets on the media asset platform of terminal devices (such as mobile phones), and put the selected media assets on TV or other display devices for playback through push operations.

It should be noted that both the terminal device and the pushed display device are installed with a video playback application. For the convenience of description, in the following embodiments, the video playback application installed in the terminal device 300 is referred to as the first application, and the display device 300 is referred to as the first application. The video playing application installed in 200 is named as the second application. Wherein, the second application may be a system application or a third-party application.

Fig. 16 is a schematic diagram of a user interface of a terminal device (such as the terminal device 300 in Fig. 1 ) shown according to some embodiments of the present application. In some specific implementations, the user can open a corresponding application program by touching an application icon on the user interface, or open a corresponding folder by touching a folder icon on the user interface.

In the embodiments of the present application, a terminal device and a server are respectively provided. A terminal device includes a display unit, a communication unit, and a controller such as a processor. The display unit is used for displaying the terminal interface, and the communication unit is used for communicating with the display device and the server. The terminal device can be a mobile terminal, a notebook computer or a tablet computer. The server includes a communicator and a processor, and the communicator is configured to communicate with the terminal device and the display device.

In some embodiments, the user can use the terminal device to communicate with the display device, so as to realize information interaction between the terminal device and the display device.

For example, the user can manipulate the terminal device to download some media resources from the display device, or upload the media resources in the terminal device to the display device for playback. The user can also use the terminal device to send some control commands to the display device to control the display device to realize corresponding functions. At this time, the terminal device can be used to realize the function of the remote control.

In some embodiments, the display device may be provided with a communication mode. When the communication mode is off, the display device will not receive the communication connection request sent by the terminal device. At this time, the display device does not allow the terminal device to communicate with each other, and the display device will not exchange information with any terminal device at this time. When the communication mode is turned on, the display device will receive a communication connection request sent by the terminal device. At this time, the display device may allow the terminal device to communicate with the display device, so as to realize information interaction. The user can input a communication mode start command to the display device, and when the processor receives the communication mode start command, it can control the display device to enter the communication mode.

In some embodiments, the user may send a communication mode activation instruction to the display device by operating a designated key of the remote controller. In the process of actual application, the corresponding relationship between the command for starting the communication mode and the keys of the remote controller is bound in advance. For example, a communication mode button is set on the remote controller. When the user touches the button, the remote controller can send a communication mode start command to the processor, and the processor controls the display device to enter the communication mode. When the user touches the button again, the processor can control the display device to exit the communication mode.

In some embodiments, the correspondence between the communication mode command and multiple remote control keys may also be bound in advance, and when the user touches multiple keys bound to the communication mode command, the remote control sends out the communication mode command. In a feasible embodiment, the keys bound to the communication mode command are direction keys (left, down, left, down) in turn, that is, when the user touches the keys (left, down, left, down) continuously within a preset time In the case of , the remote controller sends the communication mode command to the processor. By adopting the above-mentioned binding method, it is possible to prevent the communication mode command from being issued due to a user's misoperation. The embodiment of the present application is only an example to provide the binding relationship between several communication mode commands and keys. In the process of actual application, the binding relationship between communication mode commands and keys can be set according to the user's habits. This is not too restrictive.

In some embodiments, the user can use a sound collector of the display device, such as a microphone, to send a communication mode instruction to the display device through voice input, so as to control the display device to enter the communication mode. An intelligent voice system may be installed in the display device, and the intelligent voice system may recognize the user's voice to extract the instruction content input by the user. The user can enter a preset wake-up word through the microphone to activate the intelligent voice system, so that the processor can respond to the user's input. And input the communication mode command within a certain period of time, so that the display device enters the communication mode. For example, the user can input "so-and-so classmate" to activate the intelligent voice system. Then enter "Enter Communication Mode" to send communication mode instructions to the display device.

The communication mode option can also be set in the UI interface of the display device, and when the user clicks on this option, the display device can be controlled to enter or exit the communication mode.

In some embodiments, when the display device is in the communication mode, an identity authentication mode may be further set in consideration of security. When the identity authentication mode is off, the display device will not perform identity authentication on the terminal device, and at this time the terminal device can directly communicate with the display device. That is, when the terminal device sends a communication connection request to the display device, the display device may directly perform a communication connection with the terminal device without verifying the communication connection request.

When the identity authentication mode is turned on, the display device will authenticate the terminal device. That is, when the display device receives the communication connection request sent by the terminal device, it will verify the communication connection request, and when the verification is passed, the communication connection request is allowed, so as to communicate with the terminal device.

Fig. 17 is a schematic diagram showing an identity authentication mode set in a device in some embodiments. Wherein, when the user chooses to turn on the communication mode, he may further choose to turn on the identity authentication mode or turn off the identity authentication mode.

Fig. 18 is a flowchart of a communication connection between a display device and a terminal device in some embodiments.

In some embodiments, when the display device is in the communication mode, the terminal device can communicate and interact with the display device. In step S1801, the terminal device needs to send a communication connection request to the display device, and the display device can further determine whether to establish a communication connection with the terminal device.

In some embodiments, the communication connection request may be a search command. The terminal device can send the search command to the network, specifically, the search command can be sent to the multicast address of the local area network.

All devices in the LAN can receive the search command through the multicast address. Therefore, when the terminal device and the display device are connected to a local area network, the display device can obtain the search instruction from the multicast address. When the display device acquires the search instruction, it can be considered that the terminal device has sent a communication connection request.

In steps S1802 to S1803, after the display device obtains the communication connection request, the display device determines its own network information and physical address, and can send its own network information to the terminal device. For example, the display device may send a Location URL (uniform resource locator, uniform resource locator) to the terminal device, and the UPL may include the IP address of the display device.

The display device can also send its own physical address (that is, the MAC address) to the terminal device, and the MAC address is used to identify the network card of the display device.

In step S1804, after receiving the network information and the physical address sent by the display device, the terminal device may generate a device information acquisition request according to the network information and the physical address. Wherein, the device information obtaining request is used to obtain device information of the display device. And at S1805, the terminal device may send a device information acquisition request to the display device.

In some embodiments, after receiving the device information acquisition request, the display device may send its own device information to the terminal device. At the same time, the display device may allow the communication connection request sent by the terminal device, that is, allow the terminal device and the display device to communicate and interact. Further, the display device can establish a communication connection with the terminal device.

When the terminal device receives the device information sent by the display device, it realizes the communication connection with the display device, and at this time, the terminal device can communicate and interact with the display device. For example, the terminal device may send some control instructions to the display device, so that the display device realizes corresponding functions.

The device information of the display device may include information such as DeviceId (device ID), brand name, model name, and FeatureCode (feature code). Wherein, the device ID represents the ID information of the display device, which is the unique identification attribute of each display device. The brand name and model name are the factory parameters of the display device, and are generally fixed names.

In some embodiments, the user can establish a communication connection between the terminal device and the display device by binding and associating the terminal device with the display device.

In some embodiments, the user can use the terminal device to directly search for all display devices in the local area network. Fig. 19 is a schematic diagram of a terminal interface of a terminal device in some embodiments.

The terminal device can include a "connect device" control, and the user can click the "connect device" control to instruct the terminal device to scan for connectable display devices in the local area network, and at the same time trigger to enter the corresponding device list page. When the device list page is displayed, the user can select the display device to be connected on this page.

After detecting that the user clicks on a certain display device, the terminal device may send a device information acquisition request to the display device. After receiving the device information acquisition request, the display device may send its own device information to the terminal device. At the same time, the display device can be bound and associated with the terminal device, that is, a communication connection can be established.

In some embodiments, the user can also log in his own account in the terminal device. After the user uses the terminal device to bind and associate with a certain display device, the user's account is also bound and associated with the display device.

In some embodiments, when the user uses the terminal device and the display device to establish a communication connection, different communication connection types may also be used.

The user can use the network to directly bind the terminal device and the display device, so as to realize the communication connection between the terminal device and the display device. The terminal device can directly search for the display device in the local area network, or send the search command to the local area network. For example, when the display device is an AIOT device, an AIOT network connection manner may be used to implement a communication connection between the terminal device and the display device. In the embodiment of the present application, the manner of these network connections is set as the first type of communication connection.

The user can also bind and associate the terminal device and the display device by means of an infrared connection, so as to realize the communication connection between the terminal device and the display device. The terminal device can transmit infrared signals to the display device, and the display device converts the infrared signal into an electrical signal, and further decodes and demodulates the corresponding information, so that the display device can establish a communication connection with the terminal device. Further, the terminal device can obtain the device information of the display device. In the embodiment of the present application, the infrared connection is set as the second type of communication connection.

In some embodiments, when it is detected that the terminal device establishes a communication connection with the display device, the user may use the terminal device to control the display device. At this time, the terminal device can function as a remote controller.

The terminal device can control the display unit to display a control interface, and the control interface includes a plurality of control instructions for controlling the display device. When the user clicks on a certain control instruction in the control interface, the terminal device can send the control instruction to the display device, so that the display device can realize a corresponding function.

It should be noted that when the terminal device and the display device interact using the AIOT cloud intelligent control method, due to the limited functions that AIOT can control, it is also necessary to integrate some control commands during the infrared connection in the control interface of the terminal device, so that This enables the terminal device to completely control the display device without using a remote controller for operation, so as to improve user experience.

Fig. 20 shows a flowchart of interaction between a terminal device and a server in some embodiments.

In some embodiments, at S2001, when it is detected that the terminal device establishes a communication connection with a certain display device, the processor may acquire device information of the display device. At the same time, the processor may further acquire the communication connection information of the communication connection between the terminal device and the display device. The communication connection information may include communication connection types of the terminal device and the display device. It can also include displaying device network information as well as physical address.

Specifically, it may be determined that the type of the communication connection is the first type of communication connection or the second type of communication connection.

After determining the communication connection information and the device information of the display device, at S2002 the processor may control the communication unit to send the communication connection information and device information to the server, so as to further determine the control commands currently supported by the display device.

In some embodiments, after receiving the communication connection information and device information sent by the terminal device, at S2003 the server may obtain the set of control instructions. The control command set includes all control commands supported by the display device and the terminal device when they are connected in communication. And from S2004 to S2006, the terminal device may interact with the server to obtain a set of control instructions, and generate a control interface according to the set of control instructions.

Specifically, the controller of the server may acquire the set of control instructions according to the communication connection information and device information. Fig. 21 is a flow chart of the server obtaining a set of control instructions in some embodiments.

It should be noted that when the communication connection types of the display device and the terminal device are different, the control instructions that the display device can support may also be different.

When the display device and the terminal device use the first type of communication connection for communication connection, that is, when the terminal device uses the AIOT cloud intelligent control method to control the display device, the terminal device can send the first type of control command to the display device, namely The AIOT control command can also send the second type of control command to the display device, that is, the infrared control command. At this time, in the first type of communication connection mode, the display device can support two kinds of control commands.

When the display device and the terminal device use the second type of communication connection for communication connection, that is, when the terminal device uses infrared control to control the display device, the terminal device can only send the second type of control command to the display device, that is, infrared Control commands, but the first type of control commands cannot be sent. At this time, the display device only supports the second type of control commands, but not the first type of control commands.

Therefore, the server needs to determine the control instructions supported by the display device according to the communication connection information. Specifically, the controller may detect the type of communication connection between the display device and the terminal device.

In some embodiments, if it is detected that the communication connection type is the first type of communication connection, it means that the display device supports both the first type of control instruction and the second type of control instruction under the current communication connection situation. At this point, the processor may obtain all first-type control instructions and all second-type control instructions supported by the display device from the preset instruction database, that is, all control instructions that the display device can support.

It should be noted that an instruction database is preset in the server. All control commands supported by each display device are stored in the command database.

The display device can support two kinds of control commands: AIOT control commands and infrared control commands. Wherein, the infrared control instruction is an instruction supported by the traditional remote controller. Generally speaking, each infrared control command corresponds to an operation of the display device, and the real-time control is strong, but the infrared control command cannot support some more complicated functions. For example, setting the volume or brightness value of the display device to a specific value, this function cannot be realized by infrared control commands.

The real-time controllability of AIOT control instructions is poor, and for some functions that require quick response, AIOT control instructions are difficult to implement. For example, the number keys and direction keys in the remote control, because the user generally touches these keys multiple times or continuously, requires multiple rapid interactions between the terminal device and the display device, resulting in AIOT control commands that cannot realize these functions. However, an AIOT control command can correspond to multiple operations of the display device, such as adjusting the volume of the display device from the current value to a specific value. If a traditional remote control is used, the user needs to issue multiple instructions to adjust the volume, but using AIOT control Instructions only need one instruction to realize, thus greatly improving the user's operation efficiency.

The embodiment of the present application comprehensively considers the advantages of the two kinds of control instructions, and integrates the two kinds of control instructions in the terminal device, so as to maximize the user experience.

Among the two types of instructions that the display device can support, the AIOT control instructions can be stored in the form of pure instructions. The infrared control commands can be stored in the form of infrared code values, and each infrared control command corresponds to an infrared code value.

Therefore, for a display device, all AIOT control commands that it can support can be formed into an AIOT command set. Form an infrared code library with all the infrared control commands it can support. That is, each display device will correspond to an AIOT instruction set and an infrared code library.

In the command database, according to the device ID of the display device, the AIOT command set and infrared code library corresponding to the display device can be queried.

In some embodiments, display devices of the same brand can support the same control commands, so display devices of the same brand can share the AIOT command set and infrared code library.

In some embodiments, if it is detected that the communication connection type is the first type of communication connection, after determining the AIOT command set and infrared code library corresponding to the display device, the AIOT command set and infrared code library can be integrated, and the All the control instructions of the first type and the control instructions of the second type are processed to obtain a complete set, so as to generate a set of control instructions. The control instruction set includes all the control instructions supported by the display device and the type of each control instruction under the current communication connection status.

In some embodiments, if it is detected that the communication connection type is the first type of communication connection, considering that the terminal device may not support infrared control, the server may also determine whether the display device supports the second type of communication connection according to the device information of the display device , that is, whether to support infrared control.

If it is detected that the display device supports the second type of communication connection, that is, it supports infrared control, the controller can simultaneously obtain the AIOT command set and infrared code library corresponding to the display device from the preset command database according to the device ID of the display device, and can All first-type control commands and all second-type control commands supported by the display device are obtained.

At this time, all the first-type control instructions and the second-type control instructions can be processed in a complete set, so as to obtain a set of control instructions.

If it is detected that the display device is not connected to the second type of communication, that is, it does not support infrared control, the controller can obtain only the AIOT command set corresponding to the display device in the preset command database according to the device ID of the display device, so as to obtain the display device. All first-class control commands supported by the device. At this time, all the first-type control instructions together constitute a control instruction set.

In some embodiments, if it is detected that the communication connection type is the second type of communication connection, it means that the terminal device cannot send the AIOT control command to the display device, and the display device only supports the second type of control command under the current communication connection condition. At this time, the processor can obtain the infrared code library corresponding to the display device from the instruction database according to the device ID, so as to obtain all second-type control instructions supported by the display device. At this time, all the second-type control instructions together constitute a control instruction set.

In some embodiments, after the control instruction set of the display device is determined, the processor may store the control instruction set.

A display device instruction library may be set in the server. Further, the processor may store the control instruction set of each display device into the display device instruction library.

When the communication connection type between a certain display device and the terminal device changes, the server can obtain the control command set under the current communication connection condition of the display device, and update the new control command set in the display device command library.

In some embodiments, after a communication connection is established between the display device and the terminal device, the user can use the terminal device to control the display device.

Fig. 22 is a schematic diagram of a terminal interface of a terminal device in some embodiments.

The terminal device may include a "control device" control, and the user may click the "control device" control to instruct the terminal device to confirm all display devices that have established communication connections, and at the same time trigger to enter the corresponding control list page. When the control list page is displayed, the user can select the display device to be controlled on the page.

In some embodiments, the user may click a certain display device in the control list page, so as to input an instruction to control the display device.

In response to the instruction, the terminal device may send a control instruction set acquisition request to the server. Wherein, the control instruction set acquisition request is used to acquire the control instruction set of the display device.

After receiving the control command set acquisition request sent by the terminal device, the server may send the control command set of the display device to the terminal device.

In some embodiments, after receiving the set of control instructions, the terminal device may generate a control interface according to the set of control instructions. At the same time, the display unit can be controlled to display the control interface. The control interface may include multiple control instructions in the control instruction set, so that the user can control the display device according to the control interface.

In some embodiments, when the control instruction set received by the terminal device includes both the first type of control instruction and the second type of control instruction, the two control instructions may have the same instruction. For example, the control instruction "mute the display device" may exist in the form of the first type of control instruction or in the form of the second type of control instruction, thus causing the same control instruction to appear repeatedly.

At this point, you can set the priority for the type of control command, for example: AIOT control command > infrared control command. That is, when a control command can be either an AIOT control command or an infrared control command, the AIOT control command is preferentially set. The priority can also be set as: infrared control command>AIOT control command, that is, the priority of the control command is set to the infrared control command. The type of each control instruction can also be set according to the needs of the user, which is not specifically limited in this embodiment of the present application.

After determining all the control commands supported by the display device under the current communication connection and the type of each control command, the terminal device can generate a control interface according to these control commands and display it on the display unit.

In some embodiments, when the control interface is generated according to the set of control instructions, the processor can analyze the set of control instructions to obtain all the control instructions that the display device can support and the value of each control instruction under the current communication connection. command type.

Further, the processor may convert each control instruction into an operation control, each operation control is a control instruction, and each operation control may have a separate icon. At this point, these icons can be generated into a control interface for the user to operate.

Fig. 23a is a schematic diagram of a control interface of a terminal device in some embodiments. As shown in Figure 23a, the control interface includes 9 kinds of operation controls: power control 1101, program source control 1102, return control 1103, main interface control 1104, keyboard switching control 1105, signal source control 1106, image recognition control 1107 and volume Settings interface 1108 . Wherein, the power control 1101 is used to control the turning on and off of the display device, and can be set as an AIOT control command or an infrared control command. The program source control 1102 is a numeric keyboard, which is used to adjust the program source currently played by the display device. The user can press a specific number key to switch to the corresponding TV program, or switch TV programs back and forth according to the sequence of the program source. It needs to be set to infrared Control instruction. The return control 1103 is used to return to the user's last operation, and can be set as an AIOT control command or an infrared control command. The main interface control 1104 is used to return to the main interface of the display device, and can be set as an AIOT control command or an infrared control command. The keyboard switching control 1105 is used to switch the displayed keyboard. The keyboard includes a numeric keypad and a direction keypad, which can be set as AIOT control commands or infrared control commands. The signal source control 1106 is used to display a list of signal sources of the display device, including signal sources such as HDMI, USB, ATV, etc., for the user to choose, and can be set as an AIOT control command or an infrared control command. The image recognition control 1107 is used to recognize the image displayed by the display device, and needs to be set as an AIOT control command. The volume setting interface 1108 is used to adjust the volume of the display device to a preset volume value, which can be adjusted to 50, and needs to be set as an AIOT control command. In the current control interface, when the user clicks the keyboard switching control 1105, the numeric keypad in the control interface will be switched to a direction keypad.

Fig. 23b is a schematic diagram of another control interface of a terminal device in some embodiments. As shown in Figure 23b, the focus control 1109 is a directional keyboard, and the user can use this control to adjust the focus position in the display device, which needs to be set as an infrared control command.

In some embodiments, the user may use the control interface to control the display device to control the display device according to the control instruction triggered on the control interface. For example, by clicking or touching a certain control in the control interface, a corresponding control instruction is input.

Fig. 24 is a flowchart of a terminal device controlling a display device in some embodiments.

As shown in FIG. 24, in step S2401, a controller such as a processor may detect the user's operation on the control interface, so as to determine the control command input by the user.

The terminal device can send the control instruction input by the user to the display device. In S2402, the display device receives the corresponding control instruction, and can analyze the control instruction, so as to determine the function corresponding to the control instruction, and implement it.

After determining the control instruction input by the user, the terminal device may detect the instruction type of the control instruction.

In some embodiments, when it is detected at S2403 that the control command input by the user is the first type of control command, that is, the AIOT control command, at S2404, the terminal device needs to first send the control command input by the user and the device information of the display device to To the server, specifically, the control instruction and the device ID of the display device may be sent to the server.

The server can determine which display device the user wants to control according to the device ID of the display device. At this time, the server may send the control instruction input by the user to the corresponding display device. Further, after receiving the control instruction, the display device can realize the function corresponding to the control instruction.

In some embodiments, at S2405, when it is detected that the control command input by the user is a second type of control command, that is, an infrared control command, at S2406 the terminal device may directly send the control command input by the user to the display device. Specifically, the infrared code value of the control command can be sent to the display device. After the display device converts the infrared code value into an electrical signal, it can decode the electrical signal, demodulate the corresponding control command, and realize the corresponding function.

In the embodiment of the present application, the terminal device may send the device information of the display device and the communication connection information between the terminal device and the display device to the server. The server can further integrate the AIOT control commands and infrared control commands that the display device can support, and send them to the terminal device. Therefore, the terminal device can send two types of control commands to the display device without the user using a remote controller, thereby improving user experience.

The embodiment of the present application provides a communication control method, which is applied to a terminal device. As shown in FIG. 25, the method includes the following steps:

S1301. Detect that the terminal device establishes a communication connection with the display device, and acquire communication connection information and device information of the display device, where the communication connection information includes a communication connection type.

S1302. Send the communication connection information and the device information to the server, so that the server obtains a control instruction set according to the communication connection information and the device information, and the control instruction set includes the communication connection between the display device and the terminal device All control commands supported at the time.

S1303. Send a request for acquiring a set of control instructions to the server in response to an instruction input by the user indicating to control the display device.

S1304. Generate a control interface according to the control instruction set sent by the server, and control the display unit to display the control interface, so that the user controls the display device according to the control instruction triggered on the control interface.

The embodiment of the present application provides a communication control method applied to a server, as shown in FIG. 26, the method includes the following steps:

S1401. Receive communication connection information and device information of the display device sent by the terminal device, where the communication connection information includes a communication connection type when the terminal device establishes a communication connection with the display device.

S1402. Acquire a set of control instructions according to the communication connection information and the device information, where the set of control instructions includes all control instructions supported by the display device and the terminal device when they are connected in communication.

S1403. In response to the request sent by the terminal device for acquiring a control instruction set, send the control instruction set to the terminal device, so that the terminal device generates a control interface according to the control instruction set, and displays the control interface.

The embodiment of the present application also provides a communication control method, which includes the following steps:

When the terminal device detects that a communication connection is established with the display device, it determines communication connection information and device information of the display device, where the communication connection information includes a communication connection type.

The terminal device sends the communication connection information and device information to the server.

The server obtains a control command set according to the communication connection information and the device information, and the control command set includes all control commands supported by the display device and the terminal device when they are connected in communication.

In response to an instruction input by the user indicating to control the display device, the terminal device sends a control instruction set acquisition request to the server, and the control instruction set acquisition request is used to acquire the control instruction set.

The server sends the set of control instructions to the terminal device.

The terminal device generates a control interface according to the set of control instructions, and controls the display unit to display the control interface, so that the user can control the display device according to the control interface.

For the same and similar parts among the various embodiments in this specification, only reference should be made to each other, and details will not be repeated here.

The above embodiments are only used to illustrate the implementation of the present application, not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still apply to the foregoing embodiments Modifications are made to the implementations described in the examples, or equivalent replacements are made to some or all of the technical features; and these modifications or replacements do not depart from the scope of the present disclosure.

Claims

A control device comprising:

a display for displaying a user interface;

The user input interface is used to receive input signals; the controller is connected to the display and the user interface respectively, and is configured to:

Acquiring a device list, so as to control the display to display the device list;

receiving a user operation command for selecting a device control in the device list via the user interface;

broadcasting a broadcast packet carrying the device identifier of the device corresponding to the device control to multiple devices, so that the device sends a Bluetooth pairing request based on the device identifier; and

In response to receiving a Bluetooth pairing request from the device, pairing is performed with the device via Bluetooth.
The control device according to claim 1, wherein the controller is further configured to: when the pairing is not successful, repeatedly broadcast the broadcast packet carrying the device identifier of the device corresponding to the device control until repeated execution The number of times reaches the preset number of times.
The control device according to claim 2, wherein the controller is further configured to: when the number of repeated executions reaches the predetermined number of times, control the display to display a pairing abnormal prompt.
The control device according to claim 1, wherein the controller is further configured to: when the pairing is successful, establish a communication connection between the control device and the device, so as to control the device.
The control device according to claim 1, wherein said device sending a Bluetooth pairing request based on a device identification further comprises:

Parsing the received broadcast packet to obtain the device identifier in the broadcast packet;

judging whether the device identifier carried in the broadcast packet is the same as the device identifier of the device; and

In the case that the device identifiers are the same, the Bluetooth pairing request is sent to the control device.
The control device according to claim 1, wherein the controller is further configured to:

Before obtaining the device list, determine whether the control device is currently connected to the Internet; and

When the control device is connected to the network, the device list is obtained.
The control device according to claim 1, wherein the control device further comprises: a sound collector configured to collect a voice signal input by a user; and a sensor configured to detect whether the control device is lifted;

Wherein, the controller is also configured as:

controlling the sound collector to start the collection of voice signals in response to a screen lift signal received from the screen lift sensor, the screen lift signal being used to indicate that the control device has been lifted;

calculating a sound source angle between the user and the control device based on the speech signal; and

determining whether to perform speech recognition on the speech signal based on the sound source angle and a preset angle range,

Wherein, when the sound source angle is within a preset sound source angle range, performing speech recognition processing on the speech signal.
The control device according to claim 7, wherein the sound collector includes at least two microphones, and the controller is further configured to:

Calculate the sound source angle between the user and the control device according to the distance between two adjacent microphones and the time difference between two adjacent microphones receiving the voice signal, wherein the sound source angle is the distance between the user and the control device. The angle between the central axes of the control devices described above.
The control device according to claim 7, wherein the controller is further configured to: when the sound source angle is within the preset sound source angle range and the voice activity detection result is not finished, continue to receive the the voice signal, and perform voice recognition processing on the received complete voice signal, wherein, when the voice activity detection result is not finished, the voice collector continues to collect the voice signal input by the user; or

When the sound source angle is within a preset sound source angle range, and the voice activity detection result is end, perform speech recognition processing on the currently collected voice signal, wherein the voice activity detection result is end , the sound collector cannot continue to collect the voice signal input by the user.
The control device according to claim 7, wherein the controller is further configured to:

When the sound source angle is not within the preset sound source angle range, stop collecting the voice signal input by the user.
The control device according to claim 7, wherein the controller is further configured to control the display to turn on a screen in response to a screen lift signal received from the screen lift sensor.
The control device according to claim 11, wherein the controller is further configured to perform speech recognition processing on the speech signal when the sound source angle is within the preset sound source angle range The results are shown on the display.
The control device according to claim 11, wherein the controller is further configured to: when the sound source angle is not within the preset sound source angle range, control the display to turn off the screen.
A method for controlling a device comprising:

Acquiring a device list, so as to control the display to display the device list;

Receive the user operation command of the device control in the selected device list;

Broadcasting a broadcast packet carrying the device identifier of the device corresponding to the device control to multiple devices, so that the device sends a Bluetooth pairing request based on the device identifier;

In response to receiving a Bluetooth pairing request from the device, pairing is performed with the device via Bluetooth.
The method according to claim 14, wherein the method further comprises: when the pairing is not successful, repeating the broadcasting of the broadcast packet carrying the device identification of the device corresponding to the device control until the number of repeated executions reaches a preset number of times.
The method according to claim 14, further comprising: when the pairing is successful, establishing a communication connection between the control device and the device to control the device.
The method of claim 14, wherein the device sending a Bluetooth pairing request based on the device identification further comprises:

Parsing the received broadcast packet to obtain the device identifier in the broadcast packet;

judging whether the device identifier in the broadcast packet is the same as the device identifier of the device; and

In the case that the device identifiers are the same, the Bluetooth pairing request is sent to the control device.
The method of claim 14, wherein,

When responding to the lifting screen signal received from the lifting screen sensor, the control sound collector starts the collection of the voice signal, and the lifting screen signal is used to indicate that the control device has been lifted;

calculating a sound source angle between the user and the control device based on the speech signal; and

determining whether to perform speech recognition on the speech signal based on the sound source angle and a preset angle range,

Wherein, when the sound source angle is within a preset sound source angle range, performing speech recognition processing on the speech signal.
The method according to claim 18, wherein the sound collector includes at least two microphones, and calculating the sound source angle between the user and the handheld device according to the voice signal further comprises:

Calculate the sound source angle between the user and the control device according to the distance between two adjacent microphones and the time difference between two adjacent microphones receiving the voice signal, wherein the sound source angle is the distance between the user and the control device. The angle between the central axes of the control devices described above.
The method according to claim 18, wherein the method further comprises: when the sound source angle is within the preset sound source angle range and the voice activity detection result is not over, continue to receive the voice signal , and performing voice recognition processing on the received complete voice signal, wherein, when the voice activity detection result is not finished, the voice collector continues to collect the voice signal input by the user; or

When the sound source angle is within a preset sound source angle range and the voice activity detection result is end, perform speech recognition processing on the currently collected voice signal, wherein the voice activity detection result is end , the sound collector cannot continue to collect the voice signal input by the user.